Hello Graph Community!
My name is Joseph, and I am excited to be leading our initiative to join The Graph as a part of their Core Dev Team. Building and enabling Web 3 has been our objective in all our initiatives since our inception in 2018.
To do it, we’ve built open source APIs, indexing services, provided enterprise-grade staking infrastructure and supported communities on multiple protocols.
We are extremely excited to join The Graph and build on that expertise around indexing different types of blockchains together with you all as a community.
If interested to learn more about the partnership check out the The Graph’s announcement or our recent GRTiQ podcast episode here.
We envision The Graph as a key element in Web 3, building upon our indexing expertise we have joined The Graph with a joint mission of providing indexing and API services in a decentralized way, and now it’s time to work together to make it happen.
Multi-Blockchain Indexing and The Graph
We’ve been researching different ways of bringing multi blockchain indexing to the graph, and since bringing many networks to the Graph ecosystem is not an easy task, we suggest a unified process across all chains. We strongly believe that it’s a key to fast adoption and maintainability throughout different networks.
Our unified process consist of defining:
-
Network Graph: an ordered, structured and versioned set of data extracted from 1st level blockchain networks. All data structures within the Network Graph will be strongly typed representations, as close as possible to the original network data representation. We could hence define a Subgraph as a derivative of the Network Graph.
-
Network Graph API: as an entity that returns Network Graph data and uses GraphQL queries as its primary method of communication. It should be written in a tight regime, with the enforcement of supporting every version of data ever stored on the Network Graph. Network Graph API will use an indexing mechanism. Although a database is not necessary, it might be that for some networks, it will be more optimal to store/read data from the filesystem or third party storage services like s3 or ipfs.
Introducing GraphQL to Subgraphs
Bringing many networks to the Graph ecosystem in the current way would require a deep integration within the core. Eventually integrating many networks would be problematic. This is why we needed some form of abstraction. The most natural, and perfect candidate was the graphQL interface widely adopted by the community.
By just adding a call() function that allows making a controlled graphQL call, it opens new ways for the subgraph processing.
- We can fetch data from many Network Graphs with just one integration. This form of processing shifts all the type mapping onto the subgraph developers. With just one change in the Graph Node’s core we can create an interface that would be able to integrate many networks.
- Adding graphQL interface into the subgraph processing also allows to create cross-subgraph connections. That’s a requested feature by the community.
- Querying other subgraphs to build additional data based on that wouldn’t be different from how dApps are using the subgraphs right now.
- Another type of the call that subgraph may make is the call to itself. Thanks to its own interface on which subgraph shares data for the dApps, now you can create more complex queries for data during its creation. What for? Detailed queries of historical entries should be extremely useful for statistical calculations on a dataset.
The flow of bringing different networks data onto subgraphs would be based on the simple idea of monitoring events. Events have the least information of what happened (like an event about a new block with just height). Afterwards, subgraphs would make a request to the Network Graph for desired information, and map the response into the known format.
The graphQL subgraph client interface is eligible for many various optimizations. In performance, parallel processing or resource usage.
We are mostly excited about using the graphQL ability to merge. For most of the currently deployed subgraphs, the indexer will be able to create one and only one data retrieval call. Afterwards the data would be transmitted to respective graphs. This should significantly lessen the network traffic, allowing the indexer to host much more subgraphs.
Although it would be great to bring cross-subgraph calls or extend the storage interface, our main focus for now is the new network adoption.
What’s Next? The “Manager / Worker” Model
The proposed process in our technical documents, which we’ll provide later, has proved to be the most efficient and resilient over the years - we call it Manager/Worker model.
In the most basic description, worker is a stateless implementation of the network’s api, while manager is responsible for maintaining ingestion progress, the storage of the data and serving the Network Graph API.
Both manager and worker are automatically horizontally scalable, for the sake of both data ingestion and Network Graph Api performance. The Graph node is supposed to subscribe to a manager using websocket connection, over which it would gradually receive next events, and retrieve the data.
As the Network Graph API definition says - nothing is forcing the mean of storage. It will always be the database with the indexing, however in various cases the data might be stored on the filesystem or even the 3rd party solution. It might be done for the best performance or to lower costs of maintenance.
@lukasz will soon release detailed technical documentation. Question, ideas, feedback? We would love to hear. We are looking forward to your feedback and questions.
Here to build!
The Figment Core Dev Team.