Recommendations for MIPs-V2

General
1

We are calling on the Graph Foundation to relaunch the MIPs program due to its numerous flaws, as outlined in our previous post. In general, the Foundation should

  • Clearly define the goals and objectives of the program, ensuring that they align with The Graph’s overall mission and vision.
  • Simplify the program’s structure and requirements, making it easier for indexers to participate and contribute, and reduce the workload of the program organizers.
  • Increase transparency and communication with program participants, including regular updates on progress and any changes to the program.
  • Establish effective channels for feedback and collaboration between organizers, developers, and indexers.
  • Review and address any issues with the scoring and evaluation process to ensure fairness and transparency.

We are specifically proposing a new rating system and program structure that will:

  • Prioritize openness and transparency.
  • Simplify the MIPs program to reduce the workload of organizers.
  • Accelerate the integration of new chains.
  • Incentivize high performance and quality of service.
  • Foster collaboration and leverage the expertise of indexers.

Our proposed rating system has three key criteria, corresponding to the three primary objectives of the MIPs:

  • C1. Community Participation
  • C2. Chain and subgraph coverage
  • C3. Performance and Quality of Service

Program Structure

The program is structured to allow for maximum parallelization of the integration of chains. It is flexible and removes dependencies between chains, so delays with one chain do not significantly impact the others. Multiple chains can be worked on simultaneously during testnet and mainnet stages, and indexers are encouraged to take initiatives on other chains.

Stage 0 (Pre-testnet, ongoing for all chains)

A rough roadmap listing potential future chains is shared with indexers, allowing them to properly evaluate hardware requirements and sync the chains in advance. This roadmap is not guaranteed, and indexers should expect that some chains may not make the final cut. Technically capable and motivated indexers can experiment with offchain indexation of subgraphs and identify pain points and raise issues to blockchain node developers and Graph core teams.

This stage is constantly active and indexers are free to participate or not, without impacting their score for Stage 1 and 2. However, indexers who engage in Stage 0 may be advantaged in future stages by achieving a greater level of readiness. Contributions to this stage will only be directly rewarded through the Community Participation criterion (C1).

Stage 1 (Testnet, one or more chain at a time)

Once the core developers determine that the maturity of the integration of the indexer stack with the new chain is acceptable, a formal testnet stage is launched. Indexers are given a two-week notice for integrating a new chain. The notice will indicate which subgraphs will be used for testing, giving indexers ample time to index them.

During the testnet stage, scoring will evaluate chain and subgraph coverage (C2), performance and quality of service (C3.1), and potentially an indexation speed and throughput competition may be held (C3.2). Shortly after the end of the stage, another testnet stage is launched for a different chain.

Stage 2 (Mainnet, one or mone chain at a time)

Once the testnet stage is completed, the Graph Council adopts the necessary GIPs to bring the chain to mainnet. To incentivize mainnet indexing, a supplemental GRT grant is distributed to indexers.

Only graph coverage is evaluated at this stage (C2).

Detailed scoring criteria

C1. Community Participation:

  • Grant GRT for notable contributions to the community made during the program. These grants are allocated by a committee of Indexers, Core devs and Foundation representatives on the following criteria:
    • Quality of the github issues raised (well-written, well-researched, and relevant issues that contribute to the improvement of the MIPs program)
    • Quality of PRs: Indexers who submit high-quality pull requests that are well-written, well-documented, and adhere to coding standards may be rewarded.
    • Participation on Discord and forums (active participation in Discord or forum discussions, provide helpful feedback, and contribute to the community
    • Documentation and guides published
    • Tooling shared with the community (scripts, dashboards, indexer software distribution, etc.)
    • Support provided to subgraph developers
    • Contributions to blockchain node development (e.g. working with node developers to fix issues affecting indexers, making snapshots available, etc.)
  • The committee meets on a regular basis (e.g. every month) to review contributions and award GRT.

C2. Chain and subgraph coverage

  • The same methodology should be used for both testnet and mainnet, with different objectives:
    • Testnet: Encourage broad participation
    • Mainnet: Ensure continuous support from indexers and guarantee proper quality of service to subgraph developers looking to migrate to the decentralized network.
  • Grants are proportional to the allocation period of the indexer, rather than allocation size, to level the playing field for new and smaller indexers. Allocation period is calculated after the first successful allocation closure with a valid Proof of Indexing (POI) to incentivize indexing speed. Indexers will be invited to perform a “reallocate” action to start the accumulation of MIPs rewards.
  • Grants are calculated using an exponential curve on the proportion of chain subgraphs that the indexer is supporting, to incentivize maximum coverage while subgraph developers are testing their subgraphs on testnet and migrating to mainnet.
  • The calculation formula is made public, allowing anyone to independently calculate the subgraph coverage score using only on-chain data, with the exception of POIs, which must be verified off-chain.
  • For mainnet, a longer evaluation period should be considered (months instead of days for testnet) to ensure proper migration support.

C3. Performance and Quality of Service:

  • C3.1: Data freshness and base latency
    • A simple testing harness will send a moderate volume of queries (1-5 qps) directly to indexers, bypassing the gateways.
    • Testing will happen over several days, for every subgraph deployment supported by the program.
    • The testing harness will calculate average query latency and block delay, and cross-check query results for accuracy.
    • The testing harness will run simultaneously from several worldwide locations, averaging latency from different geographic areas.
    • The code will be published, so indexers can run their own instance to test their setup before the official test. This will allow indexers to optimize their indexer and contribute to the improvement of the testing harness.
    • The harness is designed to mimic Gateway behavior as much as possible, minus the Indexer Selection Algorithm, so any performance optimization made by indexers will also improve regular mainnet usage.
    • Indexers will provide an authorization token to the MIPs organizers to grant them free queries directly through their indexer service endpoint.
    • Near-realtime metrics will be provided for transparency and to allow indexers to optimize and improve their setup on a continuous basis.
  • C3.2: High-performance and scale
    • Optional competitions will be organized to incentivize indexers to invest in high-performance and scalable architecture.
    • Competition C3.2.1: Throughput:
      • Using the testing harness of C3.1, a very high volume of queries will be sent to the indexers on a few moderately large subgraphs. Indexers will be ranked on the query volume they are able to sustain for a 15-minute period without generating an elevated error rate or hitting a cliff in query latency.
    • Competition C3.2.2: Indexation speed
      • A very complex and hard to index subgraph will be deployed at a predetermined time, and indexers will be ranked on their ability to rapidly index the new subgraph.
      • The first indexers to post their POI for a predetermined block in a Discord channel will win.

We hope that these recommendations will help to improve the MIPs program and foster a more collaborative and productive relationship between indexers and the Graph Foundation. We look forward to working with the Foundation to make these changes and create a more successful program for everyone involved.

15 Likes
2

This is clearly revolution rather than evolution. Given the momentum of the existing program, I do question if a complete pivot is possible. Interested in thoughts on that at a high level.

C3 is at the apex of many of the frustrations around feedback. Engineers, when left in the dark with no measurement methodology or data will literally lose their minds over not being able to optimise and fix problems. I am in strong support of C3 with one note on C3.2.2 - is there value in this or is it just for fun? Are we moving to a model where Indexers must presync a subgraph before allocation? I think the answer to that drives whether its worth incentivising that monetarily or if it’s just for fun, incentivising somehow else.

C1 I like this idea, but we would need a solid framework for measuring participation because people will compare apples to oranges all day long. Also need to consider conflicts of interest as the committee would be made up of alot of participants, presumably.

C2 says what some of us have been thinking, but without saying it out loud - this process of migration is going to take more time than I think we all are willing to recognise. This framework removes a lot of the linearity of the process which is currently causing quite a lot of stress in terms of deadlines across many contributors. Keen to hear from everyone across the board on this, especially if it has impacted your work.

4 Likes
3

On C3.2.2, the speed of indexation has been a significant issue for subgraph developers and indexers. The Graph has been focusing on making improvements in this area through the use of the firehose and substream products. However, the current protocol implementation does not incentivize indexation speed. While it is up for debate as to what can or should be done, a competition could help indexers invest in better nodes and potentially innovate in this area. Some indexers may find this type of competition fun, while others may appreciate the opportunity to showcase their expertise or powerful servers. Regardless of the motivation, this competition could improve the quality of service.

C1 also requires attention. Conflict of interest is a major problem, but attempting to solve it perfectly will only lead to opacity. C1 should be weighted just enough to ensure that contributors see that their work is being recognized, but it should be relatively lightweight compared to quantitative criteria.

1 Like
4

Hi ellipfra,

I appreciate the time you took to provide such a thorough list of general recommendations and scoring methodology. I will do my best to address each of these below.

Goals of the program and expectations

Before I dive deep into your feedback, I want to reiterate that the goal of the program is, indeed, to bring new chains to the network. As a consequence, we needed to be sure we’d have a good amount of Indexers on the network serving all of these chains with the desired quality of service. Assuming we could only do so with pre-MIPs Indexers would be too risky, which is why we decided to onboard many new stakeholders to the protocol in Phase 0.

Program structure and priorities

You make a good point about optimizing for maximum parallelization of chain integration. To your point, we do want to have more chains in parallel, and getting ready for it has been the focus for, at least, the last 6-7 weeks. However, we didn’t just want to announce the next batch of chains and have Indexers scrambling to get started without minimum guidance. While experienced Indexers - and infrastructure operators in general - could get away with little information, we wanted to ensure newcomers would have the minimum information required to get started. In hindsight, we could have communicated the rationale behind this approach more clearly.

Begining the program with one chain only (Gnosis) allowed us to focus on the most important thing at that time which was to ensure the network and protocol were multi-chain ready. We’re past this stage now (a great achievement!) and, moving forward, the Indexer community, core developers, Foundation, and the Council can work more efficiently when adding new chains to the network at a quicker pace. Furthermore, the program attracted a big number of Indexers, which meant we also had to quickly scale the operations team to handle applications. We’re past this stage, so we’re all focused on bringing new chains to the network and working more closely with participants.

Here’s a high-level list of things we’ve been actively working on with core contributors when gathering the minimum required information needed to onboard new Indexers to such new chains:

  • Minimum hardware requirements to urn an archive node.

  • Snapshots (ideally, as we are collaborating with protocol teams).

  • Guides on running archive nodes. This implies testing the new clients, working with the different teams (like we did with Nethermind who were extremely helpful), and writing new docs if the official ones are outdated or lack critical details, etc. In recent weeks we worked on getting Docker images, bare metal installation guides, and Helm charts for 4 different chains.

  • Validating that clients work reliably with Graph Node, exposing the required interface to support all subgraph features.

On the roadmap, we have been working on a way to share timelines for all future chains, phases, and missions. A plan for the next 6-8 weeks was released a few days ago when the four new chains were announced. It will be continuously updated, and we’ll make an effort to include new chains and phases weeks in advance, so Indexers can plan accordingly.

Also, in your original Forum post you called for clearer communication. Our current process has been to send updates through email and Discord, as information becomes available, plus a dedicated section during IOH every week. But, to address your point, we are going to add some scheduled touchpoints as well (such as weekly digest updates) to add more structure to communication. We are open to any other ideas you might have in this regard! We also try to avoid communication overload, as to avoid confusion.

Rating system

I want to start by saying we’re actually doing 90% of what’s recommended here under C2 and C3 of your post, both on how we conduct testing and monitoring progress on the two environments, as well as the proposed scoring criteria. I understand, however, participants had no visibility over the methodology. To this point, we’ve been working on a more transparent solution.

The framework adopted to score participants on how they fare on the network is quite simple and very QoS driven. While it is true we’re looking at Gateway-based metrics, we aren’t taking into account Indexers’ allocation size but merely the following metrics gathered at query time (regardless of how many are served individually):

  • Data freshness (number of blocks behind the head of the chain).
  • Query latency.
  • Success rate.

We’re essentially leveraging the already existing monitoring systems core developers had in place while removing the ISA (Indexer Selection Algorithm) from the equation.

Gateway-based public QoS data

To gather all data related to the metrics mentioned above, a subgraph was specifically built back in October: gateway-mips-qos-oracle. This data on QoS metrics has been our source for scoring since Gnosis Phase 1. Furthermore, a public real-time dashboard exposing all of these metrics (and more) has recently been made available here: https://mips-qos.streamlit.app/. The main focus of this dashboard is to allow mainnet participants to have better insight into Gateway-measured QoS across all chains. We hope MIPs participants (and mainnet Indexers in general) can use the dashboard to understand how QoS changes over time and how they rank against all others. The dashboard will be complementary to the public leaderboard, not a replacement. Please give it a try - we’d be happy to have your feedback on it. Also, if you build other dashboards on top of this QoS Oracle Subgraph, please do share it with the community.

Again, we appreciate all feedback shared thus far, and we hope this post helps clarify some aspects of the program. We’ll continue iterating while making progress with the program. Please don’t hesitate to reach out!

6 Likes
5

Thank you @Pedro for taking the time to respond to our proposal. We wrote it in the interest of constructive criticism, but also to show that there is a way to do things in a more transparent manner. We are advocating for a radically transparent rating system to better align with the crypto ethos, minimize the need for information requests to the organizing team, and allow for quick feedback to improve network quality.

While your response doesn’t fully align with the transparency we are pushing for, we recognize that publishing the inputs of the scoring methodology, such as the Gateway metrics through a subgraph, addresses most of our concerns. We are very excited by the publication of the Gateway subgraph and dashboard for both testnet and mainnet, as it is a game changer for improving QoS in the MIPs and mainnet. We understand that all future QoS metrics will be done through the DataEdge oracle, which will allow program participants to identify issues and improve their setup accordingly. However, we anticipate that publishing these metrics may trigger a wave of questions from indexers regarding certain measurements, and we hope that these questions will be perceived as legitimate attempts to improve the network and not as attempts to game the system. In the interest of transparency and to alleviate the load on the organizing team, we suggest that these conversations happen in the open (Discord, IOH, forum, etc.).

We also appreciate the clarification that the primary goal of the program is indeed to bring new chains, and that allocation size and stake size are not taken into account in the scoring, and that steps are taken to use only metrics that are not affected by those. While we wish things could be improved towards simplicity and even more transparency, we understand the constraints that the team has to work within. The clarification points and improvements announced are indeed moving in the right direction.

6 Likes