Substreams into Subgraphs: a simple integration


GIP: XXX
Title: Substreams into Subgraphs
Authors: Adam Fuller
Created: 2022-07-19
Updated: 2022-08-31
Stage: Draft


Abstract

Substreams are a new streaming-first blockchain indexing technology developed for The Graph Network, focused on performance and composability.

A substreams package is a collection of interdependent modules, which combine to extract and transform blockchain events in a highly configurable and parallelisable way, producing a streaming output.

This GIP describes the simplest possible integration of Substreams with Subgraphs, to take that final streaming output and make it persistent and queryable.

Motivation

Adding Substream support to Graph Node is the fastest way to bring the performance and composability benefits of substreams to The Graph Network.

High Level Description

specVersion: 0.0.X
schema:
  file: ./src/schema.graphql
dataSources:
  - kind: substreams
    name: Uniswap
    network: mainnet
    source:
      package:
        moduleName: output
        file:
          /: /ipfs/QmbHnhUFZa6qqqRyubUYhXntox1TCBxqryaBM1iNGqVJzT
          # This IPFs path would be generated from a local path at deploy time
    mapping:
      kind: substreams/graph-entities
      apiVersion: 0.0.X

This introduces a new substreams “kind” of dataSource. This dataSource is identified on the subgraph.yaml by a name and a filepath. This filepath will point locally to a Substreams package, which will be uploaded to IPFS on deployment.

Subgraphs with a substreams dataSource can only have that single dataSource.

Graph Node configuration

In order to support substream-based subgraphs, Graph Node will need to be Substream aware. An additional substream endpoint will be required for each network which Graph Node will want to support.

This can be an extension of the existing provider configuration in Graph Node:

[chains.mainnet]
shard = "main"
protocol = "ethereum"
provider = [
  { label = "substreams-provider-mainnet",
    details = { type = "substreams",
    url = "https://mainnet-substreams-url.grpc.substreams.io/",
    token = "exampletokenhere" }},
]

Substreams may require dedicated error handling / retry logic, though a lot of the requirements should be covered by the Firehose cursor-based integration.

Mapping entities

Substreams have their own protobuf schemas (documentation). However to directly copy the raw substream schema would be to lose the ability to link and traverse entities in the resulting GraphQL schema.

Utilities should certainly be provided to auto-generate a .graphql file from a substream protobuf schema, with intelligent type mapping. However once generated, developers should be able to update the .graphql schema, to identify connections between entities, and to add derived fields. Therefore there is still value in having a schema file as part of the subgraph definition.

This proposal does assume a tight coupling between entity names in the substream, and entity names in the subgraph

Updating entities

In order to support performant querying, Graph Node stores full entities at every block where those entities change. It supports “upsert” type behaviour, where saved values are merged with existing entities.

Substreams currently provide entity “deltas”, which just capture the changed values. Therefore there is a requirement to fetch any prior entity values to create the full entity to store. This could be done on either the Substream or Graph Node side of the integration, so we should work through the trade offs & alternatives.

Graph Node will still need to “close” the block range for any existing entities, which could negatively impact indexing performance.

Deterministic indexing

A fundamental requirement is deterministic indexing, specifically the generation of a Proof of Indexing, which can then be cross-checked across indexers.

Graph Node makes the assumption that all upstream datasources are deterministic, and will make the same assumption about Substreams

Depending on the implementation in Graph Node, this could leverage the existing Proof of Indexing setup, or it may need a dedicated Proof of Indexing in Graph Node.

Handling re-orgs

Graph Node will still need to be re-org aware. The substreams connection should provide the relevant information (which blocks to remove). Graph Node will need to remove entities from those blocks, and re-open the block range for older entities. This functionality already exists in Graph Node, but might need to be updated for the new data source.

Linear process & batch back-filling

Graph node will need to support linear processing, e.g. when indexing at the chain head. There is a possibility that linear processing historical blocks becomes a significant performance bottleneck. In that case, a dedicated “batch” insert implementation might be required to load historical data into the subgraph.

Monitoring

Graph Node’s instrumentation & monitoring may need to be updated for the new type of indexing (e.g. tracking indexing time per block etc).

Graph CLI

Graph CLI will need to be updated for the new type of data source. Graph CLI could also include the helpers to auto-generate .graphql files for a given substream.

Backwards Compatibility

This functionality is purely additive. Graph Node may need a new specVersion given the new structure of the manifest, but this could also be handled in the same way as the introduction of new protocols, where Graph Node versioning provides the necessary guarantees.

Dependencies

  • Substream support for a given network requires a Firehose implementation for that network.
  • Requires determinism from the upstream Firehose & Substreams
  • Substreams introduce additional infrastructural requirements for indexers
  • This integration may require some changes as part of Substreams

Validation

Determinism - are indexing results consistent?
Performance - are the performance gains as expected?
Scaleability - can Graph Node support 1-2K concurrent subgraphs powered by substreams?

Rationale and Alternatives

  • Alternative Postgres sink
  • Less opaque integration

Copyright Waiver

Copyright and related rights waived via CC0.

11 Likes

oh boi :eyes: :eyes: :eyes: :eyes: :eyes: :eyes: :eyes:

1 Like

The graph-node configuration has a provider section which includes a “token” element. In existing (legacy) public firehose/substreams implementations, this has historically been a JWT (json web token).

Is there plans to support generating a JWT from a given key, or existing public firehose/substream providers should switch to using a static key instead of JWT?

For further clarity, the JWT workflow looks like today:

  • register/login to provider web site
  • push buton to generate API key
  • request JWT by providing API key to providers’ JWT generator (using a well-known url)
  • provide JWT in graph-node configuration

JWTs are meant to be short-lived, so this manual work is not reasonable going forward. Either graph-node needs to know how to run this logic on its own, or need to get rid of it.

Hi! That is indeed a workaround. As you say, Graph Node providers can currently use a static key. @abourget interested in whether you feel strongly about the JWT on the Firehose / Substreams endpoint, vs moving to a static token? @lutter do you think it would make sense to add the JWT logic in Graph Node?

I am fine with static key, given that it my understanding that substreams are meant to be server-to-server based communication, not involving an application’s end-user via a web browser.

Really I just want a decision one way or another and there is a defined protocol for firehose/substreams providers.

1 Like