Summary
In December, our primary objective was to end 2024 with momentum and a base from which to accelerate into 2025. In part, this meant directing our engineering, research, and product efforts toward clearing our backlog so we could approach the new year with clear minds. In part, this meant strategizing with other core devs and The Foundation to ensure that our priorities for Q1 align with those of the broader ecosystem.
Indexer Stack
TAP
In Q1, we will adapt TAP to support the Subgraph Data Service on Graph Horizon - TAP v2. In order to give our entire attention to this, last month we pushed final improvements to TAP v1. These include small fixes, performance improvements, and more widespread testing. Testing in particular will be crucial for ensuring a stable transition to TAP v2. As per our current migration plan, there will be a period of overlap where both TAP v1 and TAP v2 will run in production simultaneously. This testing will help us ensure that we do not inadvertently break TAP v1 while developing TAP v2. TAP v1’s final performance improvements come from two sources. tap-aggregator now supports gRPC requests, which Edge & Node has put into their production setup. TAP also now uses rayon to verify signatures, which sped up the process by roughly 4.5x.
We will start January by finalizing TAP v1 testing and our TAP v2 design. In particular, we want to begin automated load testing TAP as the vast majority of TAP’s bugs only happened at load. This will quickly transition to developing TAP v2 with the aim of deploying on testnet as quickly as possible.
Verifiability
Verifiable Extraction
Over the past month, we continued advancing Verifiable Extraction (VE) from research to production. With the open-sourced vee library, we focused on addressing dependencies on unreleased upstream open-source libraries, continued working toward our long-term goals of supporting verifiable extraction for post-merge Ethereum, and began exploring broader multi-chain compatibility.
Key progress includes:
- Post-merge integration: We began integrating our post-merge Ethereum work into the
vee-mergerepository to prepare for production use. - Cross-chain research: We started exploring verification methods for other chains, including Arbitrum and, as our first non-EVM chain, Solana.
- Verifiable Indexing support: We are preparing to serve the verification requirements of Verifiable Indexing, aligning with our long-term goals for verifiable data integrity across diverse networks.
In the coming weeks, we plan to continue working on the vee library dependencies, take further steps toward its release as a standalone Rust crate, and explore how VE can best serve VI’s verification needs.
Verifiable Indexing
This past month we committed to a plan for enabling Verifiable Indexing (VI) within The Graph. This involves a protocol where Indexers cryptographically commit to indexed data, and these commitments can be challenged via an optimistic, refereed-games-based dispute mechanism. We developed this protocol during earlier research for The Graph, see the Github repo here for more details. These commitments then serve as the foundational reference point for downstream protocols, such as ensuring correct data indexing (analogous to PoI disputes) or verifying query results.
The complexity of VI varies based on the data processing methods—e.g., whether data is transformed during indexing, sourced from multiple blockchains, or includes off-chain data. To start, we are focusing on a limited scope: enabling verifiable indexing of non-transformed event data from Ethereum mainnet.
AI
This past month, we began work towards automating the generation of subgraph documentation using large language models. We look forward to collaborating with Edge & Node to bring this functionality to the Subgraph Studio, where we hope to improve both the direct developer experience and downstream utilization of deployed subgraphs. Additionally, we presented our work on synthetic data generation at the 3rd Table Representation Learning Workshop at NeurIPS 2024, a major AI conference.
In the coming weeks, we plan to continue our work on documentation generation, bringing this to production for the use of subgraph developers. We look forward to advancing this work stream and iterating with the community to further improve our AI offering.