GIP: File data sources

device: rad:git:hnrkrhnth6afcc6mnmtokbp4h9575fgrhzbay
branch: adam/drafts

Preview: File data sources in Graph Node - HackMD

cc @Leo @That3Percent


The current mapping API for IPFS makes the overall indexing output depend on whether or not the index node was able to find a file at the moment was called. This is not deterministic and therefore not acceptable for the decentralized network. It also is not performant or reliable, as handler processing waits for the file to be found before proceeding. Finally, it creates a data-dependency between the data sourced from the chain, and the data sourced from IPFS, which means any change in file availability would require a full subgraph re-indexing from the block the file was first required.

The proposed solution consists of:

  • An Availability Chain , which tracks the availability over time of files referenced by Subgraphs.
  • Dynamic “file” data sources which correspond to content-addressed data (e.g. from IPFS). These are created by blockchain-based data sources, and will respect the availability definition of the Availability Chain. They are isolated, both in terms of being processed asynchronously from the main chain-based process, and in updating a discrete set of entities.

This GIP describes the implementation of file data sources in Graph Node, starting with IPFS as the first type of file data source.

The planning for the first phase of implemenation (File data sources in Graph Node, without an availability chain) is in progress.