Core development notes and updates

2021-12-23

A few final notes from 2021 :christmas_tree:

Two new Core Developer announcements - The Guild and Semiotic have both received grants.

Firehose integration : the Firehose Blockingestor is now live in Graph Node on the Hosted service, which means full syncing progress support for NEAR subgraphs, and will unlock purely firehose-based syncing for Ethereum subgraphs in future. Validation and optimisation of the Ethereum Graph Node integration continues.

Work is in progress to pipeline different aspects of Graph Node’s execution (block extraction, writing to the database). In some cases this work also serves to unlock the aforementioned Firehose integration. Parallel receipt fetching was merged, which should improve Graph Node’s syncing the chain head for high throughput chains with large blocks.

There was quite a lot of discussion around indexes over the past couple of weeks, and the burden they can introduce (on indexing speed, and database size). At present, the subgraph developer has no tools to indicate what kind of queries they might be making, and indexers have relatively blunt instruments to change settings on the basis of the query load they are seeing.

StreamingFast have been working on a testing setup for Graph Node , with automatic testing over a consistent block range (requires introduction of a “stop block”). There is some discussion between teams over the best implementation ( graph man command, configuration etc.) Testing is in progress to get Firehose benchmarks - early runs indicate that as expected there is a far greater speedup for denser subgraphs.

NEAR testnet is now available on the Hosted Service, just deploy a near subgraph with network: near-testnet

There is a lot of motion on the subgraph GraphQL API, with _change_block filtering ready to merge (PR), and open PRs for filters by child entity, and introduction of validations into Graph Node. GraphQL versioning scoping is nearly complete, with some Proof-of-concepts in progress, and this will be picked up on next, as an important prerequisite to other GraphQL changes.

As part of testing a GraphQL execution refactor, a tool for comparing and validating queries from different Graph Node instances has been developed. This will be increasingly important, given the current iteration & versioning of the GraphQL API, and will be open sourced.

Multiblockchain: Figment shared an internal demo of Tendermint indexing, live in Graph Node, and there was some discussion around the GIP. The StreamingFast team are working on a patch of the Solana node, to significantly speed up restarts.

There is an ongoing project on The Graph’s documentation , including restructuring and open-sourcing the docs in the new year. Any feedback on the docs is very welcome at this time (in terms of missing aspects or areas of confusion!)

6 Likes