The current mapping API for IPFS makes the overall indexing output depend on whether or not the index node was able to find a file at the moment ipfs.cat was called. This is not deterministic and therefore not acceptable for the decentralized network. It also is not performant or reliable, as handler processing waits for the file to be found before proceeding. Finally, it creates a data-dependency between the data sourced from the chain, and the data sourced from IPFS, which means any change in file availability would require a full subgraph re-indexing from the block the file was first required.
The proposed solution consists of:
An Availability Chain , which tracks the availability over time of files referenced by Subgraphs.
Dynamic “file” data sources which correspond to content-addressed data (e.g. from IPFS). These are created by blockchain-based data sources, and will respect the availability definition of the Availability Chain. They are isolated, both in terms of being processed asynchronously from the main chain-based process, and in updating a discrete set of entities.
This GIP describes the implementation of file data sources in Graph Node, starting with IPFS as the first type of file data source.
The planning for the first phase of implemenation (File data sources in Graph Node, without an availability chain) is in progress.
One use-case that recently came to my attention is the usage of IPNS: A developer might want to define a file data source pointing to an IPNS address (file/ipns), either directly in the subgraph manifest or by instantiate a template after observing an on-chain event. The difference between a IPNS and IPFS hash is that the IPNS content can change.
In order to support such changing data, graph-node needs to be able to know when such a file was changed (basically the IPNS reference changed) and run some logic accordingly. A very naive approach that might be good enough would be to just rerun the handler with the new content (e.g. handleFile).
What I do not know at the moment is how to know when such a IPNS pointer changes. Maybe there are some features in IPFS-nodes to listen to those events?
Thanks @schmidsi - that is an interesting extension. I am not sure that there is an easy way to listen for updates to IPNS pointers across peers, so I expect that the required approach for freshness would be similar to an HTTP endpoint (i.e. polling, or updates triggered somehow). I am also not sure how IPNS names are stored / pinned, and how they propagate (i.e. is it associated with one node, or is there some other registry)?
Hello. Any updates on the possibility to use different File Data Sources in the Graph Node? My particular case is to use IPNS within my smart contract so I would like to know if it is on the roadmap and how is the proper way to start implementing it and indexing it. Thank you for your support.