GIP: XXX
Title: Substreams into Subgraphs
Authors: Adam Fuller
Created: 2022-07-19
Updated: 2022-08-31
Stage: Draft
Abstract
Substreams are a new streaming-first blockchain indexing technology developed for The Graph Network, focused on performance and composability.
A substreams package is a collection of interdependent modules, which combine to extract and transform blockchain events in a highly configurable and parallelisable way, producing a streaming output.
This GIP describes the simplest possible integration of Substreams with Subgraphs, to take that final streaming output and make it persistent and queryable.
Motivation
Adding Substream support to Graph Node is the fastest way to bring the performance and composability benefits of substreams to The Graph Network.
High Level Description
specVersion: 0.0.X
schema:
file: ./src/schema.graphql
dataSources:
- kind: substreams
name: Uniswap
network: mainnet
source:
package:
moduleName: output
file:
/: /ipfs/QmbHnhUFZa6qqqRyubUYhXntox1TCBxqryaBM1iNGqVJzT
# This IPFs path would be generated from a local path at deploy time
mapping:
kind: substreams/graph-entities
apiVersion: 0.0.X
This introduces a new substreams “kind” of dataSource. This dataSource is identified on the subgraph.yaml by a name and a filepath. This filepath will point locally to a Substreams package, which will be uploaded to IPFS on deployment.
Subgraphs with a substreams dataSource can only have that single dataSource.
Graph Node configuration
In order to support substream-based subgraphs, Graph Node will need to be Substream aware. An additional substream endpoint will be required for each network which Graph Node will want to support.
This can be an extension of the existing provider configuration in Graph Node:
[chains.mainnet]
shard = "main"
protocol = "ethereum"
provider = [
{ label = "substreams-provider-mainnet",
details = { type = "substreams",
url = "https://mainnet-substreams-url.grpc.substreams.io/",
token = "exampletokenhere" }},
]
Substreams may require dedicated error handling / retry logic, though a lot of the requirements should be covered by the Firehose cursor-based integration.
Mapping entities
Substreams have their own protobuf schemas (documentation). However to directly copy the raw substream schema would be to lose the ability to link and traverse entities in the resulting GraphQL schema.
Utilities should certainly be provided to auto-generate a .graphql file from a substream protobuf schema, with intelligent type mapping. However once generated, developers should be able to update the .graphql schema, to identify connections between entities, and to add derived fields. Therefore there is still value in having a schema file as part of the subgraph definition.
This proposal does assume a tight coupling between entity names in the substream, and entity names in the subgraph
Updating entities
In order to support performant querying, Graph Node stores full entities at every block where those entities change. It supports “upsert” type behaviour, where saved values are merged with existing entities.
Substreams currently provide entity “deltas”, which just capture the changed values. Therefore there is a requirement to fetch any prior entity values to create the full entity to store. This could be done on either the Substream or Graph Node side of the integration, so we should work through the trade offs & alternatives.
Graph Node will still need to “close” the block range for any existing entities, which could negatively impact indexing performance.
Deterministic indexing
A fundamental requirement is deterministic indexing, specifically the generation of a Proof of Indexing, which can then be cross-checked across indexers.
Graph Node makes the assumption that all upstream datasources are deterministic, and will make the same assumption about Substreams
Depending on the implementation in Graph Node, this could leverage the existing Proof of Indexing setup, or it may need a dedicated Proof of Indexing in Graph Node.
Handling re-orgs
Graph Node will still need to be re-org aware. The substreams connection should provide the relevant information (which blocks to remove). Graph Node will need to remove entities from those blocks, and re-open the block range for older entities. This functionality already exists in Graph Node, but might need to be updated for the new data source.
Linear process & batch back-filling
Graph node will need to support linear processing, e.g. when indexing at the chain head. There is a possibility that linear processing historical blocks becomes a significant performance bottleneck. In that case, a dedicated “batch” insert implementation might be required to load historical data into the subgraph.
Monitoring
Graph Node’s instrumentation & monitoring may need to be updated for the new type of indexing (e.g. tracking indexing time per block etc).
Graph CLI
Graph CLI will need to be updated for the new type of data source. Graph CLI could also include the helpers to auto-generate .graphql files for a given substream.
Backwards Compatibility
This functionality is purely additive. Graph Node may need a new specVersion given the new structure of the manifest, but this could also be handled in the same way as the introduction of new protocols, where Graph Node versioning provides the necessary guarantees.
Dependencies
- Substream support for a given network requires a Firehose implementation for that network.
- Requires determinism from the upstream Firehose & Substreams
- Substreams introduce additional infrastructural requirements for indexers
- This integration may require some changes as part of Substreams
Validation
Determinism - are indexing results consistent?
Performance - are the performance gains as expected?
Scaleability - can Graph Node support 1-2K concurrent subgraphs powered by substreams?
Rationale and Alternatives
- Alternative Postgres sink
- Less opaque integration
Copyright Waiver
Copyright and related rights waived via CC0.
