Core development notes and updates

Hello!

Starting a thread here for ongoing notes and updates from some of the teams working on core components. This aims to be informative and interesting, rather than comprehensive.


2021-10-19

GIP Process Review: given the increase in contributors in recent months, The Foundation have been reviewing the GIP process, and how it is tracked across teams and participants.

Unattestable Indexer Responses GIP shared by Jannis last week in the Forum, feedback welcome!

The Firehose has been integrated into Graph Node as a new Blockstream, with testing underway for Ethereum indexing. The initial priority is consistency in indexing output with the existing RPC-based implementation. We will be looking to work with indexers on testing the Firehose once that is done.

There are a few open threads on multiblockchain: NEAR integration and testing is ongoing across Graph Node, graph-ts and graph-cli. StreamingFast are actively working on Solana indexing, while Figment are integrating further chains with Graph Node via a Firehose.

Last week we discussed some of the Proof-of-indexing issues that have surfaced on the network for certain subgraphs. Ethereum client differences appear to be correlated, but more investigation is needed. New tooling for diagnosing divergences have recently been worked on by E&N, and a working group is planned to investigate further, as well as investment in more diverse integration tests, given the increasing complexity of the network.

A parallelism working group was kicked off last week, with participation from E&N, StreamingFast and Figment. This will be an ongoing effort, looking to optimise the existing Graph Node implementation to be more parallel, while also thinking about “step change” parallelism opportunities (research led by StreamingFast).


Comments, questions and feedback welcome!

13 Likes

2021-10-30

NEAR subgraph support is now in beta, in Graph Node and on the Hosted Service. Documentation is available here, and the corresponding GIP is here.

StreamingFast have been making progress with their Solana Firehose implementation, while Figment’s upcoming multiblockchain integration is nearly ready to demo.

Oliver shared an update on the Graph governance process, in particular giving more context on when the full governance process is required, and how core development work fits in.

Integration testing working group: a cross-functional group from several core development teams met this week, focused on unifying the existing integration testing setup used internally by Edge & Node with the tools used to investigate Proof of Indexing disputes on the network, extending both to support more multivariate cases & automation.

The Rust implementation of the Gateway has been undergoing performance testing & iteration. This will dramatically increase the Network’s potential throughput.

Some teams are planning improvements to the subgraph GraphQL schema, and to graph-ts, and are looking in particular for feedback and feature requests from the community on those components.

There are a range of meetings planned next week (w/c 1st November) for core contributors, please reach out if there are topics you would like to be discussed or considered!


Comments & questions are welcome.

4 Likes

We are happy to join discussions around “feedback and feature requests part” if needed.

2 Likes

2021-11-27

A few quick notes on ongoing work and research.

R&D Offsite: a number of teams contributing to core development met in Lisbon earlier this month, to discuss ongoing & planned work, spanning indexing, contracts & cryptography.

Graph Node 0.25.0: release scheduled for next week, this introduces some new fields in apiVersion 0.0.6, includes NEAR & Firehose Blockstream support, adds support for number_gte specification, as well as bug fixes (more details will be in the release notes!)

Ethereum Firehose: work is ongoing to finalise the Graph Node <> Firehose integration for Ethereum. Recent fixes include more elegant handling of chain re-orgs, and there is ongoing work to better support sparse subgraphs (those with relatively few triggers per block). This integration is crucial to unlock some initial performance enhancements, as well as parallel execution in future.

Multiblockchain work: Tendermint & Solana data extraction & integration work is ongoing. NEAR is being added to graph init, and NEAR block ingestion is in progress, to fully support sync status and pending versions for subgraphs.

Cross-chain PoIs: a thread on PoI recency evolved into a discussion of the correct architecture for supporting cross-chain POIs. This is an area of active research, with a range of proposed approaches.

File data sources: engineering planning for this work has started (see GIP here), which will improve the robustness of Graph Node’s IPFS integration, while preparing for general deterministic off-chain data indexing.

Rust Gateway is live: the new, more performant Gateway was rolled out last week, after extensive performance testing. This also included improvements in indexer selection algorithm, and other requested fixes.

Integration testing: work has begun on an improved open-source integration testing & network monitoring setup, for Graph Node and the indexer components.

GraphQL API Versioning: there is an ongoing discussion about feature support and versioning, specifically focusing on the GraphQL API, (which does not currently have versioning).

GraphQL API iteration: versioning is particularly relevant, as work has started on some new graphQL features - child entity filters, and the ability to filter on entities based on the block at which they changed.

Unit testing: Limechain continue to iterate on the Matchstick unit testing framework, as well as supporting the community in adopting unit testing as part of subgraph development, now part of graph-cli as graph test

graph debug : Limechain have also been working on a simple integration testing framework, described here, focused on solving a bug encountered at a specific block. This will require access to a graph-node, to run the required block.

6 Likes