Core development notes and updates

Research & Development
1

Hello!

Starting a thread here for ongoing notes and updates from some of the teams working on core components. This aims to be informative and interesting, rather than comprehensive.

2021-10-19

GIP Process Review: given the increase in contributors in recent months, The Foundation have been reviewing the GIP process, and how it is tracked across teams and participants.

Unattestable Indexer Responses GIP shared by Jannis last week in the Forum, feedback welcome!

The Firehose has been integrated into Graph Node as a new Blockstream, with testing underway for Ethereum indexing. The initial priority is consistency in indexing output with the existing RPC-based implementation. We will be looking to work with indexers on testing the Firehose once that is done.

There are a few open threads on multiblockchain: NEAR integration and testing is ongoing across Graph Node, graph-ts and graph-cli. StreamingFast are actively working on Solana indexing, while Figment are integrating further chains with Graph Node via a Firehose.

Last week we discussed some of the Proof-of-indexing issues that have surfaced on the network for certain subgraphs. Ethereum client differences appear to be correlated, but more investigation is needed. New tooling for diagnosing divergences have recently been worked on by E&N, and a working group is planned to investigate further, as well as investment in more diverse integration tests, given the increasing complexity of the network.

A parallelism working group was kicked off last week, with participation from E&N, StreamingFast and Figment. This will be an ongoing effort, looking to optimise the existing Graph Node implementation to be more parallel, while also thinking about “step change” parallelism opportunities (research led by StreamingFast).

Comments, questions and feedback welcome!

12 Likes
3

2021-10-30

NEAR subgraph support is now in beta, in Graph Node and on the Hosted Service. Documentation is available here, and the corresponding GIP is here.

StreamingFast have been making progress with their Solana Firehose implementation, while Figment’s upcoming multiblockchain integration is nearly ready to demo.

Oliver shared an update on the Graph governance process, in particular giving more context on when the full governance process is required, and how core development work fits in.

Integration testing working group: a cross-functional group from several core development teams met this week, focused on unifying the existing integration testing setup used internally by Edge & Node with the tools used to investigate Proof of Indexing disputes on the network, extending both to support more multivariate cases & automation.

The Rust implementation of the Gateway has been undergoing performance testing & iteration. This will dramatically increase the Network’s potential throughput.

Some teams are planning improvements to the subgraph GraphQL schema, and to graph-ts, and are looking in particular for feedback and feature requests from the community on those components.

There are a range of meetings planned next week (w/c 1st November) for core contributors, please reach out if there are topics you would like to be discussed or considered!

Comments & questions are welcome.

5 Likes
4

We are happy to join discussions around “feedback and feature requests part” if needed.

3 Likes
5

2021-11-27

A few quick notes on ongoing work and research.

R&D Offsite: a number of teams contributing to core development met in Lisbon earlier this month, to discuss ongoing & planned work, spanning indexing, contracts & cryptography.

Graph Node 0.25.0: release scheduled for next week, this introduces some new fields in apiVersion 0.0.6, includes NEAR & Firehose Blockstream support, adds support for number_gte specification, as well as bug fixes (more details will be in the release notes!)

Ethereum Firehose: work is ongoing to finalise the Graph Node <> Firehose integration for Ethereum. Recent fixes include more elegant handling of chain re-orgs, and there is ongoing work to better support sparse subgraphs (those with relatively few triggers per block). This integration is crucial to unlock some initial performance enhancements, as well as parallel execution in future.

Multiblockchain work: Tendermint & Solana data extraction & integration work is ongoing. NEAR is being added to graph init, and NEAR block ingestion is in progress, to fully support sync status and pending versions for subgraphs.

Cross-chain PoIs: a thread on PoI recency evolved into a discussion of the correct architecture for supporting cross-chain POIs. This is an area of active research, with a range of proposed approaches.

File data sources: engineering planning for this work has started (see GIP here), which will improve the robustness of Graph Node’s IPFS integration, while preparing for general deterministic off-chain data indexing.

Rust Gateway is live: the new, more performant Gateway was rolled out last week, after extensive performance testing. This also included improvements in indexer selection algorithm, and other requested fixes.

Integration testing: work has begun on an improved open-source integration testing & network monitoring setup, for Graph Node and the indexer components.

GraphQL API Versioning: there is an ongoing discussion about feature support and versioning, specifically focusing on the GraphQL API, (which does not currently have versioning).

GraphQL API iteration: versioning is particularly relevant, as work has started on some new graphQL features - child entity filters, and the ability to filter on entities based on the block at which they changed.

Unit testing: Limechain continue to iterate on the Matchstick unit testing framework, as well as supporting the community in adopting unit testing as part of subgraph development, now part of graph-cli as graph test

graph debug : Limechain have also been working on a simple integration testing framework, described here, focused on solving a bug encountered at a specific block. This will require access to a graph-node, to run the required block.

9 Likes
6

2021-12-10

Some more notes on recent core development.

GIP-0020 Implemented: this is now supported across all core components, unlocking more graceful handling of unattestable queries by indexers and the gateway.

Indexer components 0.18.4: this was released yesterday, across indexer-service, indexer-agent and indexer-cli

Firehose integration & testing: this continues to be a key focus area, with progress in a few areas (ensuring sync status compatibility, addressing bugs in callhandlers, establishment of a dedicated block ingestor, and research into contract filtering for improved performance)

Graph Node performance and robustness: improving Graph Node’s performance is a key current focus area, with a range of optimisations in flight (skip block pointer updates, blockstream pipelining, automatic subgraph retries, parallel receipt ingestion and more). In addition there is work to establish an easier setup for performance benchmarking (e.g. introducing a stop block variable)

NEAR support: graph init was extended to support --protocol near, so NEAR developers can easily bootstrap their subgraphs. Meanwhile NEAR testnet indexing support on the Hosted Service is targeted for the coming weeks.

Tendermint integration: the GIP was published in the forum, for review & comment.

Change block filtering: when querying subgraphs, users will often want to fetch any new entities, since they last queried. Work is in progress to support this functionality for all subgraphs, via a new “_change_block” filter argument.

GraphQL API Versioning: specification of this feature is in progress, which will enable breaking API improvements, as well as guaranteeing client-defined query determinism

Subgraph forking: the subgraph debug feature by Limechain has evolved to improve the developer experience, PR in progress

6 Likes
7

2021-12-23

A few final notes from 2021 :christmas_tree:

Two new Core Developer announcements - The Guild and Semiotic have both received grants.

Firehose integration : the Firehose Blockingestor is now live in Graph Node on the Hosted service, which means full syncing progress support for NEAR subgraphs, and will unlock purely firehose-based syncing for Ethereum subgraphs in future. Validation and optimisation of the Ethereum Graph Node integration continues.

Work is in progress to pipeline different aspects of Graph Node’s execution (block extraction, writing to the database). In some cases this work also serves to unlock the aforementioned Firehose integration. Parallel receipt fetching was merged, which should improve Graph Node’s syncing the chain head for high throughput chains with large blocks.

There was quite a lot of discussion around indexes over the past couple of weeks, and the burden they can introduce (on indexing speed, and database size). At present, the subgraph developer has no tools to indicate what kind of queries they might be making, and indexers have relatively blunt instruments to change settings on the basis of the query load they are seeing.

StreamingFast have been working on a testing setup for Graph Node , with automatic testing over a consistent block range (requires introduction of a “stop block”). There is some discussion between teams over the best implementation ( graph man command, configuration etc.) Testing is in progress to get Firehose benchmarks - early runs indicate that as expected there is a far greater speedup for denser subgraphs.

NEAR testnet is now available on the Hosted Service, just deploy a near subgraph with network: near-testnet

There is a lot of motion on the subgraph GraphQL API, with _change_block filtering ready to merge (PR), and open PRs for filters by child entity, and introduction of validations into Graph Node. GraphQL versioning scoping is nearly complete, with some Proof-of-concepts in progress, and this will be picked up on next, as an important prerequisite to other GraphQL changes.

As part of testing a GraphQL execution refactor, a tool for comparing and validating queries from different Graph Node instances has been developed. This will be increasingly important, given the current iteration & versioning of the GraphQL API, and will be open sourced.

Multiblockchain: Figment shared an internal demo of Tendermint indexing, live in Graph Node, and there was some discussion around the GIP. The StreamingFast team are working on a patch of the Solana node, to significantly speed up restarts.

There is an ongoing project on The Graph’s documentation , including restructuring and open-sourcing the docs in the new year. Any feedback on the docs is very welcome at this time (in terms of missing aspects or areas of confusion!)

6 Likes
8

Some quick updates:

  • Figment’s Tendermint Pull Requests are in final review across graph-node, graph-cli and graph-ts.
  • A Pull Request for Solana support is open, and there has been significant progress on the Solana Firehose.
  • Ethereum Firehose support is at the final integration testing phase. In addition StreamingFast have added indexing of contracts and event signatures in the Firehose so they can be easily filtered, which will radically improve sync performance for sparse subgraphs.
  • Blockstream pipelining has been merged, and is live on the Hosted Service, reducing sync time for subgraphs which spend a lot of time scanning for events. Discussing optimisations to make this work better for dynamic data sources.
  • Store pipelining is in review, which will be especially beneficial for subgraphs which read and write a lot to the store as part of their mappings.
  • There is an open Pull Request for immutable entities, which will improve write and read times for entities that will never be updated.
  • graph man run, which runs a subgraph for a given range of blocks, has been adopted across teams for easily testing & comparing indexing performance. A range of other graph man commands are in progress, in particular to improve database interactions for indexers.
  • Limechain’s Subgraph Forking work (previously discussed as Simple Integration Testing) has been merged, and will be available in the next release of Graph Node. This should significantly improve the debug experience for subgraph developers.
  • Graph Node’s GraphQL execution has been refactored, making it significantly easier to add new features. The Guild have introduced GraphQL Validations, and are working on GraphQL versioning, which will unblock a lot of new GraphQL features (a lot of which are already ready to go!)
  • There is a proposal for a standard network configuration as part of graph-cli, to make it easier to update and add to subgraphs during development.
  • Significant R&D progress has been made on multiblockchain POI, and introducing more streaming into Graph Node - watch this space.

Far from comprehensive (as per the original message in this thread). Questions welcome!

3 Likes