Join The Graph’s Core Devs meeting #9, happening next Wednesday, December 8 @ 7am PST. In lieu of an agenda and presentation, we will be covering a number of discussion topics that will be moderated by The Graph’s Core Dev teams to generate open community dialogues. The meeting link is: Launch Zoom Meeting
The Core Devs call not only covers graph node related topics, but every research & development work in the Graph ecosystem. Everyone involved in that space is welcome to propose a topic for the Core Devs call next week that you can introduce for discussions. This is a chance to connect and reflect on your work with other Core Devs in order to inform the community and gather feedback. If there is any content you would like us to add as discussion topics for next week, please feel to suggest them as a response in this thread. Thank you!
In this meeting, we listen to updates from core developers, receive updates from The Guild and Semiotic AI, discuss solutions to the amount of data Solana produces and learn how The Graph Protocol can solve Ethereum’s historical data conundrum.
Topics Include:
Developer Updates
Edge & Node
Figment
Streaming Fast
The Guild
LimeChain
EIP-4444 & The Graph
Ethereum Network Subgraph
Verifiability
Semiotic AI
Solana Integration Discussion
Detailed notes regarding each topic with timestamps can be found below.
Edge & Node Update (3:18)
Shipped:
Rust Gateway (better throughput and indexer selection and reliability improvements)
Graph Node 0.25.0 is live
Indexer 0.17.0 is live
GIP-0020 is implemented!
Allows indexers to stop executing a query in a way that avoids slashing risk and allows gateways to handle indexer issues gracefully
Indexers should update to the latest indexer-service and graph-node!
In Progress:
Planning for Filed Data Sources (Replacing IPFS Cat)
Support File Storage Networks Like IPFS and Arweave
Continuously check query results and versioning across indexers to make sure they generate identical data while indexing subgraphs. (Custom, network, P2P modes, indexing & Querying)
There’s been new graph-node releases and new indexer releases. Both of those are really important. If there are any indexers on the call today, I urge you to update your index service at least and graph node. The biggest change that I am personally excited about is GIP-0020. Unattestable indexer responses has been released as part of these versions and that’s why it's so important to update because this is a way that we can detect when certain indexers have issues that are, for instance, due to a corrupt database or something else. If we can detect this, that means an indexer doesn't get used for that particular query result and isn't on the hook for attestations for problematic results that they themselves don't have confidence in, but don't know that somethings up. -Jannis Pohlmann | Edge & Node
Figment Update (10:47)
Challenges:
Performance when syncing older block heights
Some issues related to merged block data
Resolved through tool created to verify merged file integrity
Implementing Tendermint within graph-cli
Tendermint packages the networking and consensus layers of a blockchain into a generic engine. It is the first step of integrating many of the networks of the Cosmos ecosystem.
Two-week multi node test currently running for testing purposes
Streaming Fast Update (13:23)
Indexers
Indexers who are able should please utilize the Firehose
Firehose utilization will accelerate the performance improvements and downstream for subgraph developers
Binance Smart Chain
The Firehose for BSC is in development
Ethereum
Team working on making sure the Firehose works with the call handlers from Ethereum
Solana
A larger discussion on Solana can be found near the end of these notes as it is covered in a separate portion of the call.
The Guild (16:34)
The guild is a group of open-source developers focused on Graph QL. As part of the Graph QL Foundation, The Guild maintains reference implementation of Graph QL, helps with the maintenance of Graph QL spec and builds open-source tools around Graph QL. The Guild were wave one grantees for their schema prototyper. They will now be focusing on API versioning and subgraph composition.
LimeChain (19:20)
LimeChain was previously featured in The Graphs Community Talk #2 for their development of a unit testing framework for The Graph. In this meeting, they shared about their subgraph debug tool that allows subgraph developers to speed up their debugging process. A recap on Matchstick from community talk #2 is included for review.
Matchstick
Matchstick is a unit testing framework for The Graph protocol. Matchstick allows developers to test their subgraphs logic before deployment, allowing them to find errors preemptively and ensure deployed subgraphs are functioning correctly. Before Matchstick, subgraph developers were required to deploy their subgraph, wait for it to sync and send it test queries to test functionality. If errors were found in the code, the process would then need to be repeated. This led to some subgraphs being deployed that were not functioning the way the developer intended.
A new release for Matchstick is coming in the next few days which will have a lot of bug fixes that have been pointed out in the discord channel.
Subgraph Debug Tool
The subgraph debug tool saves developers time during the debug process. Sometimes, when a subgraph is deployed, it fails sometime in the syncing process. Typically, a developer would need to attempt to fix the problem, redeploy and wait for syncing to see if the problem was fixed.
The subgraph debug tool allows developers to run their mappings using the store from the failed subgraph. It will run locally on the block the subgraph failed, so that developers can receive feedback on their attempts to fix the problem fast. The tool has been finalized and is part of graph-node.
EIP-4444 is an exciting opportunity for The Graph ecosystem. As part of the push for Ethereum 2.0, EIP-4444 introduces historical data pruning in Ethereum clients. Part of the Ethereum ethos is that the chain should be able to be validated by anyone with consumer grade hardware. The state required to run a client to verify the chain is currently over 400 gigabytes – which would require a one terabyte disk to run – and is pushing the limit of “consumer grade” hardware.
EIP-4444 addresses this issue by pruning any data older than one year that is no longer required to validate new blocks. This would reduce the hardware requirements drastically.
Once Ethereum clients no longer store historical data that’s older that one year, they will be unable to serve queries for that state. The Graph Protocol is mentioned in the text of EIP-4444 and has been publicly endorsed by prominent members of the Ethereum community and Vitalik Buterin as a solution to this problem.
For this to happen The Graph Protocol Needs:
Ethereum Network Subgraph
A subgraph would need to be build that will index all the relevant data and thereby be able to mimic the JSON RPC API.
Verifiability
The trust model for everything that would be used by the Ethereum Network Subgraph would need to have a 1 of N trust model. The Graph currently has four stages of trust levels. Most subgraph features are in the arbitration stage, where indexers sign attestations and can be slashed if their work was done incorrectly. The target in the short term is getting all indexing into the Fraud Proofs stage of verifiability. Then, in the future, the goal would be to support validity proofs for queries.
Experimental
Experimental features are not yet implemented in a way that the protocol can enforce all security guarantees, for example if the implementation is not deterministic or the API is unstable. Instead, the user must trust in the reputation of the Indexer (or Indexers, if the user cross-checks results). Consumers may mitigate risk by not relying on experimental features in production or selecting Indexers who are long-term incentive-aligned with the protocol and have demonstrated good behavior for an extended period of time.
Arbitration
The current process for dispute resolution in The Graph Network follows an Arbitration Charter that guides Indexers on expected slashing and dispute management. For features currently in Arbitration, the required level of trust moves to The Graph’s protocol governance: The Graph Council (6 of 10 multisig), and their elected Arbiters.
In this stage, Indexers sign their claims on their indexing. Claims thought to be in error (ie. query returns incorrect data) can be disputed, and the offending Indexer slashed. Here is an example of a dispute and arbitration on mainnet.
Even in this early stage, Indexers participate in adding security by sourcing data from their own Ethereum archive node to validate the work of other Indexers.
Fraud Proofs
Fraud proofs minimize the trust assumption to a 1 of N model. Refereed games replace the role of the Arbitrators, allowing any honest participant to slash a malicious Indexer and collect a reward if they can show bad behavior.
Validity Proofs
Validity proofs are zero-knowledge proofs that succinctly verify a given claim. This stage is the ultimate standard in verifiability.
Once a subgraph feature utilizes validity proofs fully, the consumer only needs to trust a block hash. The Indexer can provide all other required information to give the dapp developer and consumer complete confidence in the result.
Semiotic Update (37:40)
Semiotic AI was a wave one grantee focusing on reinforcement learning in the query layer of The Graph. They will be working with Zac and the Snark Force to design the new road runner snark. The three team members will be working full time to ensure The Graph can exceed trust standards using zero-knolwedge proofs.
Solana Integration Discussion (44:07)
Solana is a fast chain. It produces gigabytes of data per day, sometimes more. This produces a problem for indexers who can’t sync data from genesis and will need access for data already produced by other indexers. A discussion during the meeting covered possible solutions to this problem.
Possible Solutions
Warp Sync to State Snapshots
How do indexers ensure the provided data is accurate from a security standpoint?
How is the data offered?
Is it sold?
Is it provided through a decentralized service?
What are the incentives for providing snapshots?
How is the data verifiable?
Solutions to this problem will be explored further by members of The Graph Protocol development community.
Stay Tuned!
Join us next month for The Graph’s Core Devs Meeting #10!
Keep up to date by joining discussions in the forum, following The Graph on Twitter or joining the Discord server.
For further discussion on the topics covered in The Graph’s Core Devs meetings, attend Indexer Office hours in the Graph Stage voice channel every Tuesday.
