Graph Horizon Explained: a Proposal for the Evolution of the Protocol

Governance & GIPs
22

It’s hard to draw up a precise timeline for something this big, and it will likely become clearer as we write down the specs for each approach (working on it these days). But here’s what I think regarding scope of work and timelines:

  • Greenfield: Needs a completely new Staking contract, a different Indexer Agent, and changes in the gateway code. There are many open questions in the details of how that Staking contract would work, which we might be able to simplify if we default to something closer to the current protocol. So I’d estimate a couple months to finish the first iteration of the implementation, and then we’d need to do more than one audit and test it intensively. Once we deploy, we have a protocol that lives in parallel to the current one, with no Indexing Rewards. Then maybe we build tools for indexers to move stake between the two protocols? And this new protocol would have no UIs (e.g. Graph Explorer, Studio) unless those are implemented in parallel, and this would be completely new products to implement.
  • Brownfield: The changes in the Staking contract are significant but manageable. We also need updates to Indexer Agent and gateway code. We change the allocations interface, change delegation and slashing, and we need to be very careful about the transitions. We can do this in a few steps, first introducing permissionless data services and later adapting the subgraph data service to the new mechanism. This obviously also needs audits and testing, and while the transitions require lots of care it will likely be a lot less code. As soon as we deploy each change, everyone can immediately take advantage of the improvements. We’d also need some changes to Explorer and Studio to integrate with the new allocations format as this will involve a few breaking changes. Going through the GIP process at each step can take time, but I’m optimistic we can move forward quickly considering the positive feedback we’ve received so far.

So I think in terms of “time to mainnet” both approaches are similar, but in the greenfield case once we get to mainnet it will take time for people to start using it (and rewards in the current protocol will incentivize against it).

(Edit: note by “mainnet” here I mean Arbitrum One)

5 Likes
23

Thanks a ton for such a thorough response. After being torn initially, I’m now fully in favor of the Brownfield approach now.

Thanks again for all your work defining Horizon and fielding questions. It is greatly appreciated!

2 Likes
24

More IOH discussion on the topic (the link is timestamped):

3 Likes
25

Thanks for sharing @AbelsAbstracts.eth

Is there any reason why the gateway responsibilities proposed for the new DAO couldn’t just be assigned to the Core Devs as a collective?

I’m very skeptical of governance DAO’s in general, as most seem to be ineffective, highly political, captured, and/or decentralized in name only. One of my favorite things about The Graph is that protocol governance isn’t held captive by a DAO. Not sure why we’d invite that headache into the ecosystem.

Pinging @Pablo @chris @kw1k and @ellipfra for their input since they led most of this IOH conversation.

Thanks to all you guys for a great discussion on this. Greatly appreciate it.

6 Likes
26

Thanks for your thoughtful response @Mr.1776!

I totally agree with your sentiment regarding DAOs. In my mind, this organisation should be more akin to an industry association that sets standards for its members than a DAO that puts every governance decision to a vote.

This is something we’re thinking about at GraphOps and will likely share more in the coming months as third-party Gateways become a reality.

6 Likes
27

Hey Pablo (and Zac), thanks for putting this proposal together. It’s been great to see all the discussion so far, and I know a lot of great thinking by many people went into this.

I’m particularly encouraged to see so much momentum behind the “Brownfield” approach. Others have already made a good case for this, but I’ll add my strong support for the following reasons:

  • Unbundles higher priority work items from lower priority items, which gives the higher priority items a chance to ship as soon as possible.
  • Avoids another heavy migration—between hosted service migration and L2 migration, I think The Graph’s developers will have migration fatigue by now.
  • Gives each proposal an opportunity to be evaluated on it’s own merits and discussed by the community rather than “tagging along” with other proposals that have broad support.
  • Preserves optionality if other high priority work items arise in the meantime that we would to move towards the front of the queue.

Given the support there seems to be behind the Brownfield approach, I gather the next step will to be to post independent GIPs to the forums, so I will reserve much of my proposal-specific feedback until then. For now, I’ll share a high level reaction to some of the framing I’ve seen attached to the Horizon proposals.

I want to preface by saying that on a long enough time horizon (pun intended) I’m supportive of much, if not most, of what’s being proposed in Horizon bundle of proposals.

That said, I would like to challenge the idea that “permissionlessness” is a strategic objective our ecosystem should be orienting around in the short-to-medium term.

When this proposal was presented in Panama, permissionlessness featured heavily as one of the top three design goals, and it shows up again here as a stated benefit. Since I have not seen this reasoning questioned publicly anywhere yet, I would like to do so here.

To be clear, I’ve always thought building on The Graph (i.e., individual subgraphs or substreams modules) should be fully permissionless and censorship resistant. What I’m skeptical of is that building The Graph itself (i.e., unilaterally specifying the verifiability and economic properties of data services) should be fully permissionless at all, much less a top priority.

Rather, I believe The Graph should be focused on delivering the best possible service to as many developers and end-users as possible, while remaining true to the values of decentralization. To do that, it must:

  • Facilitate the development of new data services in response to market demand.
  • Achieve and maintain a high quality bar for all data service in the network, in terms of performance, liveness, and data correctness.
  • Avoid centralization vectors that would disintermediate the decentralized network and recreate the platform monopolies of web2.
  • Make sure rewards in the protocol flow to where value is actually being created, and establishes robust economic flywheels.

To address @ellipfra’s question above…

…there is still quite a bit of work required to achieve this, much of it involving on-chain upgrades to the protocol.

Not only do I believe permissionlessness as a strategic objective is orthogonal to the above goals and thus presents a real opportunity cost, but it actually could be counterproductive:

  • Removing any barriers for a data service to be considered part of The Graph potentially lowers the quality bar of The Graph, diluting The Graph’s brand and even harming the perceived quality of the more reliable data services.
  • Fragmenting governance of the protocol makes it harder for the protocol to be opinionated on how economic rewards or verifiability guarantees should flow across data services, which is very important for establishing economic flywheels in a world of composable data services.
  • Allowing any data service developer to define their own economics opens up vectors for unaligned actors to disintermediate the decentralized network. (You don’t need to believe these actors are malicious, just rational and self-interested).

Furthermore, I believe that orienting around this kind of “permissionlessness” as a path forward, does harm by creating the illusion of progress:

  1. It presents governance fragmentation as governance decentralization, without actually improving the underlying trust assumptions of using the protocol. Arbitration secured by some smaller DAO or centralized entity is not much more preferable, in my opinion, to arbitration guaranteed by The Graph Council. We should be investing in true verifiability using refereed games or ZKPs, both of which require on-chain work.
  2. For years, we have struggled as a core dev ecosystem to get data services that arguably are far simpler than Subgraphs, like Firehose or JSON-RPC onto the network, and I’ve often heard a lack of permissionlessness or on-chain functionality offered as explanations for why it cannot be done. As @Pablo alludes to above, as long as a data service can only be secured through arbitration or similar trust assumptions (which is the case for all of The Graph’s data services currently), then it can be initially supported by the V1 protocol without any on-chain changes. The on-chain protocol really isn’t even aware of the concept of a subgraph per se. Furthermore, there has never been an instance that I’m aware of where The Graph Council or a lack of permissionlessness has blocked a data service from being supported on the network.

What’s actually needed and has blocked supporting new data services on the network is designing and building the off-chain data services, the negotiation protocol, the service pricing, etc. that makes the data service market work off-chain. All the same things we had to do for subgraphs. This work has largely been neglected until recently, and while it’s great to see some progress finally happening, I feel like we would have been there months or years sooner had we as an ecosystem not let ourselves get distracted by the red herrings I’m describing.

I’ll reiterate that I support much, if not most, of what’s in the Horizon proposal, especially direct indexer payments (DIPs) and modular verifiability, both of which I feel are important to The Graph’s multi data service future.

My intention here is not to criticize the Horizon proposal as a whole, nor the great work the team has put into the bundle of proposals, but rather the “permissionless” meme that has been developing in our ecosystem around this proposal, which in my opinion has impeded and will continue to impede The Graph’s progress if left unquestioned.

Update: After sharing an earlier draft of this response with @Pablo, he indicated openness to reintroducing some opinionated criteria for data services around economics and verifiability through off-chain social conventions, Gateway standards, front-ends, use of The Graph’s brand, etc. He also explained that for him, permissionlessness was more about having a frictionless pathway for experimental data services to be deployed before they are able to implement the more opinionated economic and verifiability requirements of the network (please correct me @Pablo if I have not accurately conveyed your points). This is a far more nuanced and pragmatic view of permissionlessness as an objective than I’ve heard stated previously, and one which I’m firmly supportive of (an idea in the direction of experimental data services was actually already formally enshrined in GIP-0008).

5 Likes
28

Hi Brandon, thanks for your thoughtful response. It’s great to hear that you support most of the proposal. There is indeed a lot of momentum building behind the brownfield approach.
I would like to respond to this:

Gives each proposal an opportunity to be evaluated on it’s own merits and discussed by the community rather than “tagging along” with other proposals that have broad support.

Yes, in the brownfield approach we would have separate GIPs for each of the steps. However, for it to make sense, I would argue we would first have to come to a consensus that at least the rough outline of what we’re proposing (permissionless verifiers, slashable delegation, indexing fees in parallel to rewards) are worth pursuing, as I think there is a reason for these things to be bundled together. Permissionless verifiers without slashable delegation, for instance, would be an incomplete solution, as it would make little sense to distribute delegated stake across data services if only self stake is slashable. So I would hope that the community and the Council would agree to move forward, discussing implementation details and other aspects for each of the changes, but understanding that the broad aspects of this proposal work best when they are applied together.

Obviously, if there are specific aspects of the proposal that the community disagrees with, it would be good to discuss them and we can adapt the proposal as needed, but I would hope that if this is something major (e.g. permissionless verifiers vs. having an allowlist) we discuss those things now rather than wait for the specific GIPs. Based on the conversations we’ve had here, in IOH and other community calls, as well as the conversations I’ve had with the Council, I think all concerns raised so far have been answered, and we are at a point where we could say there is rough consensus for a brownfield Horizon.

To be clear, I’ve always thought building on The Graph (i.e., individual subgraphs or substreams modules) should be fully permissionless and censorship resistant. What I’m skeptical of is that building The Graph itself (i.e., unilaterally specifying the verifiability and economic properties of data services) should be fully permissionless at all, much less a top priority.

I’d like to explain my personal view for why I think permissionless is an important goal. I’m glad you agree that building things like subgraphs on The Graph should be fully permissionless and censorship-resistant. I would argue that unless data services themselves are fully permissionless, individual modules within data services will still suffer some level of permission. In the worst case, a compromised governance could theoretically shut the whole thing down. Sure, this is quite low risk in the short term, but if we’re to build the future of The Graph I would rather do it in a way that is robust against a varied set of attack vectors, and that includes compromised governance. For any mechanism that relies on a permissioned set of humans, it is impossible to predict how that set might evolve over time and whether it can be captured or compromised. Even a benign and well-meaning governing body could theoretically be pressured to censor a specific data service in the future.

Does this mean I believe The Graph should be a free-for-all, and data services should have no coherence or community-wide governance? Not at all. I still think the Council and our existing governance institutions play a super important role in the future of The Graph. I’m confident that the community can define standards for data services and requirements for these to allow composition and interoperability, preserving the “logical centralization” of The Graph as much as possible. The Council will still have the ultimate decision about what is done with token issuance, for instance, and that can play a huge role in incentivizing the development of data services that meet these standards.

“Permissionless” is also a core principle in crypto and web3 and I think it is worth pursuing at all levels. It is also important in keeping the protocol credibly neutral. It is worth noting, however, that the proposed change here that enables this permissionlessness is actually quite small: it simply allows service providers to set any verifier as the slasher. This does not mean that any verifier can automatically be considered “part of The Graph”, use The Graph brand, or be included in products built by core developers. It does mean, however, that core developers and external actors alike are free to use the Staking contract and GRT as a platform to innovate and build arbitrary data services. If we do things right, those actors will be incentivized to build things in an interoperable way and can eventually be part of the “canonical” parts of the protocol. But if anyone disagrees with specific requirements, they can still use this platform rather than having to deploy a completely new token and protocol, and their services might still be partially if not fully interoperable in ways that benefit the ecosystem as a whole. It also means that governance is unable to stop or censor these services even if pressured to do so. And for core developers, it provides much more freedom to experiment and find protocol-market-fit without long governance cycles at every step - obviously still going to governance for the things that are within the Council’s purview.

Arbitration secured by some smaller DAO or centralized entity is not much more preferable, in my opinion, to arbitration guaranteed by The Graph Council. We should be investing in true verifiability using refereed games or ZKPs, both of which require on-chain work.

I agree that we should very much strive for automated verifiability, and more decentralized arbitration is just a small step forward. I disagree that this will only give an illusion of progress - it is still a necessary step for a protocol that has many data services with different verifiability approaches, it opens the door for innovation, and it will be very impactful in making arbitration more scalable for some data services where automated verifiability may take several years.

What’s actually needed and has blocked supporting new data services on the network is designing and building the off-chain data services, the negotiation protocol, the service pricing, etc.

I also agree that much of the work needed to support new data services is actually on off-chain components, and this is happening in parallel right now. The permissionless verifiers in Horizon will simply allow these data services to specify their own arbitration or verifiability (the main thing missing from the current protocol for full support of arbitrary services), without introducing an unnecessary allowlist.

Finally, I’d say permissionless is a core part of the Horizon proposal but it’s certainly not its only feature. Modularity, improved economic security, and the many benefits of indexing fees are also very important. And I think we agree, Horizon is just one step, or a few steps, and we still have a lot of work to do after it to bring The Graph to its full potential.

Hope this clarifies my point of view a bit more. I think we’re more or less on similar pages about these things, and happy to keep discussing. In the past few weeks we’ve started writing the code for the first steps in the brownfield approach, and we’ll be working on the first GIP soon as well.

7 Likes