This Month in Graph Indexing - July 2022 Edition

:hammer: Graph-Node v0.27.0 has been released! :hammer:

A new graph-node version has appeared from the wilderness. This version is NOT yet recommended for the network, as the team is looking to iron out some kinks and bugs before being recommended for usage onto the decentralized protocol.

Here are the main highlights of this new release:

  • Store writes are now carried out in parallel to the rest of the subgraph process, improving indexing performance for subgraphs with significant store interaction. Metrics & monitoring was updated for this new pipelined process;
  • This adds support for apiVersion 0.0.7, which makes receipts accessible in Ethereum event handlers. Documentation link;
  • This introduces some improvements to the subgraph GraphQL API, which now supports filtering on the basis of, and filtering for entities which changed from a certain block;
  • Support was added for Arweave indexing. Tendermint was renamed to Cosmos in Graph Node. These integrations are still in “beta”;
  • Callhandler block filtering for contract calls now works as intended (this was a longstanding bug);
  • Gas costing for mappings is still set at a very high default, as we continue to benchmark and refine this metric;
  • A new graphman fix block command was added to easily refresh a block in the block cache, or clear the cache for a given network;
  • IPFS file fetching now uses files/stat, as object was deprecated;
  • Subgraphs indexing via a Firehose can now take advantage of Firehose-side filtering;
  • NEAR subgraphs can now match accounts for receipt filtering via prefixes or suffixes.

Read more about this new Graph Node version here:


:wrench: Indexer Stack v0.20.0 pre-release is ready for testing :wrench:

This release introduces a completely new paradigm for managing allocations using the action queue. The action queue facilitates direct control of allocation actions, batching of actions into a single transaction, oversight of indexer-agent proposed actions, and simple integration of 3rd party decision making tools. Please refer to the action queue guide for an explanation of the new features and how to use them.

A more detailed overview and demo of this new Release has been presented during Indexers Office Hours #70 by Ford. I highly recommend watching that one out!

Read more about this pre-release here:


:magnet: Actions Queue integrated into the Indexer-Agent v0.20.0 :magnet:

In the legacy paradigm the indexer-agent was the sole decision maker in the recommended indexer stack. It receives direction from the indexer in the form of indexing rules and uses that direction to take allocation management actions, sending transactions to Ethereum Mainnet to execute them. It uses the indexer management server as the source of indexer specific information (indexing rules, indexing deployments, cost models,…), directly queries the network subgraph on the graph-node for network information, and sends transactions directly to the Mainnet chain.

The action queue decouples action execution from decision-making, provide oversight of decisions to indexers, and provide a more clear delineation of concerns in the software. The indexer management server handles external interactions (data fetching and executing actions) while the indexer agent will be focused on managing the reconciliation loop decision-making process, and ongoing management of allocations. By moving transaction execution and data fetching to the indexer management server, providing the option to turn off the agent’s allocation management, and providing an allocation management interface hosted by the indexer management server we open up the design space for 3rd party decision-making software to replace or supplement the agent.

A demo of the Actions Queue was presented during the last Indexer Office Hours #70. It is highly recommended to be watched, to better understand how everything works, and how big of an improvement this is for the general indexer QoL.

Check out the Actions Queue Demo during Graph Hack:

Read the Actions Queue PR here:

Read read the Actions Queue docs here:

Watch it live, reducing the gas costs of allocations here:

https://thegraph.com/hosted-service/subgraph/fordn/allocation-multicall-gas-usage?query=Multicalls


:spider_web: Indexer Allocation Optimization Tool :spider_web:

The teams from GraphOps, Semiotic and Edge&Node have been working on releasing the Indexer Allocation Optimization Tool, and integrate it with the Indexer-Agent.

Indexers face an optimisation problem in deciding how to allocate their stake to various subgraphs. However, in practice, many indexers don’t have the time to properly solve this optimisation problem. As a result, not only do indexers receive sub-optimal indexing rewards, the network’s security is worsened.

You can read more about the Indexer Allocation Optimization Tool here:

Read the docs here:
https://graphprotocol.github.io/AllocationOpt.jl/dev/

Check out the repository on Github here:


:classical_building: Auto-Agora: A New Way For Query Pricing :classical_building:

For the entire lifetime of The Graph there have existed queries and query cost models. With the current state of the art, these models were written in an expressive language called Agora. These models are hand-written by the attentive indexer who wishes to charge more for more expensive and computationally complex queries. However, despite how powerful Agora is, it is limited by the ability of an indexer operator’s understanding of how a specific query to a specific subgraph impacts the operation.

Enter AutoAgora from Semiotic Labs. With AutoAgora, an indexer can glean understanding of query impacts automatically. AutoAgora monitors the existing indexer service for query structure, charges for the particular query (in GRT), and its execution time. It then abstracts the query into a “query skeleton” that allows for more general matching of genres of queries. Using this information, AutoAgora then assembles a set of initial cost models that enable an indexer to scale prices for each query according to their reported execution time.

Naturally, any indexer which wishes to compete in the marketplace must respond to changes in market prices. With AutoAgora, cost models are scaled up and down automatically. Using a continuous bandit, AutoAgora samples the market price using total collected query fees as a reward indicator to learn the ideal balance between query price and query volume. This unlocks the capability for an indexer to automatically select the right price to derive the highest revenue, with no intervention!

AutoAgora is still under development as engineers Semiotic Labs continue to work out the kinks in this solution, so it is still considered alpha! However, for those interested in trying AutoAgora, check out the tagged releases in our repositories:

AutoAgora:

AutoAgora Processor:

AutoAgora Indexer Service:

For a deeper dive into how AutoAgora works, check out the workshop from Semiotic Labs’ AutoAgora lead engineer Alexis Asseman:

Check out Semiotic’s blog post about AutoAgora:


:speech_balloon: The Graph Gossip Network :left_speech_bubble:

The Gossip Network is a decentralized peer-to-peer communication tool that GraphOps is developing. The tool allows Indexers to communicate across the network in real-time, report query and indexing analytics, and facilitate rapid sync negotiations.

You can imagine that the Gossip Network provides a standardized way of Indexers reporting any information to anyone who is listening. For example: the query fees that they are generating, when a subgraph that is being indexed experiences an error, which subgraphs they intend to allocate on in the future, how long particular types of queries take to execute, etc.

What’s cool about this is the "anyone listening” part. You could imagine that Curators “listening in” (or more realistically, looking at a website that shows data from the gossip network) could gain valuable insights around how query demand (and therefore query fees) is changing in the network in real-time. They might use this to inform curation decisions. Indexers listening to other Indexers can allocate their resources more efficiently.

Subgraph Developers could gain a better idea of query costs they should expect in the network.


:chains: Epoch Block Oracle :chains:

The GIP Introduces an “Epoch Block Oracle” to specify the currentEpochBlock on other chains, unlocking indexing rewards and network growth.

When indexers are indexing a subgraph, they allocate some of their stake towards that subgraph. This allocation indicates that they are indexing the subgraph, so they should be able to serve queries, and they will be entitled to collect query rewards.

In order to claim indexing rewards for a given epoch, they can close their allocation:

An allocation must be closed with a valid proof of indexing (POI) that meets the standards set by the arbitration charter in order to be eligible for rewards.

A POI for the first block of the current epoch must be submitted when closing an allocation for that allocation to be eligible for indexing rewards

Source

Information about the current Epoch, and the block that should be used for closing allocations, is provided by the EpochManager contract:

/**
     * @dev Return block where the current epoch started.
     * @return The block number when the current epoch started
     */
    function currentEpochBlock() public view override returns (uint256) {
        return lastLengthUpdateBlock.add(epochsSinceUpdate().mul(epochLength));
    }

Epochs are defined as 6646 blocks on Ethereum mainnet (approximately 24 hours).

If the network is to support indexing rewards for subgraphs indexing networks other than Ethereum mainnet, Indexers need to know which block POI to submit when closing an allocation.

Supporting additional indexing networks is a high priority for The Graph Protocol, given the growth of protocols and applications outside mainnet Ethereum.

You can read more on the Epoch Block Oracle here:


:eyes: Graph Day 2022 Recap :eyes:

Graph Day 2022 took place in June, and it was a full day focused on web3, dapps, protocols and the future of the internet. We’ve heard from leading protocol and dapp developers as they showcased how the web3 community is pioneering brand new forms of human coordination.

Watch the whole livestream here:


:astronaut:t3: The Graph core devs continue to grow! GraphOps was awarded a Core Dev Grant :astronaut:t3:

The Graph ecosystem continues to expand, with new core developers contributing to the development of the protocol and driving forward the mission of verifiable data in web3. At Graph Day on June 2nd, Eva Beylin, Director of The Graph Foundation, announced
that GraphOps has joined The Graph ecosystem as a core developer. The Graph Foundation awarded a grant to GraphOps for their contributions to The Graph ecosystem in key areas: Protocol Economics & Network Operations, Core Development of a Gossip Client and network, and Indexer Experience.

The GraphOps team brings together a wealth of experience inside and outside of The Graph ecosystem, spanning across protocol design, economics, infrastructure and systems orchestration. They are proven indexing experts, having been active members of the community since the testnet. Their work will expand the quality of service and capacity of The Graph Network and the capabilities of Indexers as the decentralized network scales.

Chris Wessels is joined by Juan Manuel Rodriguez Defango, Ana Maria Calin, Abel Tedros, and Petko Pavlovski. Their years of contribution to The Graph ecosystem gives them the depth of experience and firsthand knowledge of the protocol to provide best-in-class resources and tooling to optimize Indexer experience.

Read more about GraphOps and their mission here:


:fast_forward: Substreams: Massively Faster Indexing Performance for Subgraphs :fast_forward:

The Graph ecosystem has grown substantially over the past year, with five Core Developer teams now working full-time to enhance The Graph’s indexing and query capabilities for the world. StreamingFast, the first additional team to join as a Core Dev after Edge & Node, brings both an incredible pool of talent and powerful technology to further the protocol. One of the most exciting innovations is coming to fruition soon: substreams.

StreamingFast (formerly dfuse) was founded in 2018, providing high performance, cross-chain centralized indexing services. Interactions with the Edge & Node team convinced StreamingFast that decentralization is the most effective and scalable way to build for the future. Subsequently, StreamingFast accepted a grant from The Graph Foundation and, in June 2021, joined as a Graph Core Developer team to work full-time on The Graph ecosystem. This decentralized version of M&A was the first of its kind (but not the last).

In joining as a Graph Core Developer team, StreamingFast brought Firehose, a high-performance method of ingesting data from blockchains and started integrating it into to The Graph. At that time, an extremely complex subgraph could take weeks to sync, creating friction for developers building on The Graph. StreamingFast created a prototype called Sparkle, which helped decrease sync time on that subgraph from weeks to around six hours. Now, StreamingFast has evolved Sparkle’s capabilities and created substreams that can scale across all subgraphs on all chains.

Read more about Substreams here:


:shell: ShellProofs - a revolutionary Zero-Knowledge Proof for verifiable indexing and queries :shell:

Our vision for web3 is to build efficient institutions on a solid foundation of incentives and verifiable information. So we turn to snarks as a way to provide verifiability. What are snarks — to put it succinctly — a snark is a general-purpose tool for producing proofs of correct computation.

What does that mean — without snarks, you can ask questions like — given the latest data anchored on blockchain by the national centers for environmental prediction, what country will likely have the least rainfall next year, and you you get the response Egypt. Trust me.

But in the context of a global society, trust me is increasingly unsatisfying. With snarks we can replace trust with mathematics. The response can be — it is Egypt, and here is a mathematical proof that my answer is truthful.

Snarks are a wonder-kind technology. So why don’t we see ubiquitous use of snarks delivering on their promises? And the answer is that all snarks today both theoretical and in production have significant shortcomings. What those shortcomings are we’ll get into in a moment, but suffice it to say that these shortcomings are what prevented us from using any existing snark as our foundation for web3. Taking the long view, we decided to do something about it, and today we are thrilled to unveil an experimental snark that could forever change how snarks are built. We call it shellproofs.

This work is the culmination of three years of research and development that started with Jackson Blazensky, and has grown to become a collaboration between Edge&Node, Semiotic and the Graph Foundation.

Proof systems today fall into one of two camps, and the first camp are systems that rely on something called a trusted setup:

Groth16 and Plonks are snarks that are in this camp. This trusted setup infers that you can’t easily check the math, instead you have to rely on the integrity of a ceremony performed by a committee. And if you happen to be a committee member who aided in the trusted setup well, congratulations, because you can trust the snark. But for the rest of us, there’s no way to audit the process after the fact. Maybe we can get in on the next round, and we just aren’t comfortable with these trust requirements. So we rejected this entire category and focused exclusively on transparent proofs.

Transparent proofs like bulletproofs and starks carry no trust baggage. Transparent proofs are built upon open and auditable mathematics, but unfortunately by taking this route we are left with some less than ideal options. If you choose something like bulletproof you get long verifier times, and if you choose something like starks, then you get very large proof sizes. And the problem is that blockchain is a resource-constrained environment. Slow verifiers and large proof sizes have a measurable impact on the cost of using these systems.

So it seemed that we’re left with the difficult choice of either compromising our values by going back to an option in the trusted setup camp, or by choosing one of these and taking an enormous hit to efficiency for our on-chain verifier. What would be great is if we had a snark that was transparent AND efficient to verify on chain.

And that’s what shellproofs is. If the security proof wraps up as expected then we will have a best in-class prover and verifier and a world record proof size among transparent snarks.

Watch the recording of the presentation here:


:atom_symbol: The Graph’s Hosted Service Integrates First Cosmos Chain: Cosmos Hub :atom_symbol:

The Graph ecosystem continues to welcome web3 developers building applications across a spectrum of blockchains. As announced at Graph Day 2022, The Graph’s hosted service has completed a beta integration of its 35th chain and third non-EVM network: Cosmos Hub. This implementation, led by core dev Figment, marks The Graph’s expansion into the Cosmos ecosystem and paves the way for many more to come. In fact, beta support for the Osmosis zone in the Cosmos ecosystem is already underway as well!

The Cosmos Hub beta integration started in 2021 and now enables The Graph’s hosted service to index Cosmos Hub data. Thanks to speedy access to data recorded on the Cosmos Hub blockchain, it’s never been easier to build applications on Cosmos Hub. This integration is the latest milestone in creating a multi-chain future by facilitating interoperability between The Graph and the Cosmos ecosystem, empowering dapp developers building on Cosmos Hub to query on-chain data using subgraphs.

Read more about the Cosmos Hub integration here:


:city_sunrise: The Road to Sunsetting the Hosted Service :city_sunrise:

After nearly 4 years of supporting subgraphs for web3 dapps, the hosted service will be sunsetting in Q1 2023 with dapps migrating to the decentralized network. The hosted service was critical to The Graph’s decentralization journey, achieving product-market-fit and testing subgraph features with developers. Sunsetting the service is the greatest milestone since The Graph Network launched, and will enable all web3 applications to retrieve blockchain data in a truly decentralized way.

Moving to a fully decentralized network that supports querying across chains is the web3 future that The Graph has been building towards. At Graph Day, Yaniv Tal, CEO at Edge & Node announced plans to sunset the Hosted Service. The Graph Foundation has outlined a plan and three-phase schedule to sunset the Hosted Service, taking another massive step into the decentralized network.

The numerous builders in The Graph ecosystem, including five core dev teams, Indexers and many more individual contributors, have been collaborating to make web3 data as accessible as possible without centralized points of failure. Today The Graph Network supports subgraphs on Ethereum mainnet and will be supporting additional EVM and non-EVM chains on the network over the next several months.

Sunsetting of the Hosted Service

Phase 1: Cease new subgraph deployments to the Hosted Service Q3 2022

During this phase, developers will still be able to upgrade existing subgraphs on the Hosted Service but all new subgraphs will need to be deployed via Subgraph Studio to the The Graph Network.

Phase 2: Stop all upgrades from Hosted Service accounts Q4 2022

In this phase, all upgrades to subgraphs must be made through Subgraph Studio and subsequently published to the decentralized network. At this point the hosted service will only support subgraphs in their current form.

Phase 3: Fully phase out Hosted Service Subgraphs End of Q1 2023

At the end of Q1 2023, subgraphs will no longer be supported for any chain that is integrated with the decentralized network. Engineers across core developer teams and The Graph Foundation will support the phasing out of the Hosted Service APIs to make the transition as seamless as possible.

Read more about the sunsetting of the Hosted Service here:

For any questions, make sure you join our Graph Protocol Discord Server and ask your questions there! Our community is more than glad to help you migrate your subgraphs over, or answer any kind of questions you might have!


:small_red_triangle: Introducing Geo - a web3 browser built on Graph Protocol for creating, publishing and voting on graph pages :small_red_triangle:

Geo is a decentralized web browser built on the @graphprotocol with an on chain reputation system for voting on proposals. This is game changing for public goods and public policy.

Geo is stored on IPFS, anchored Polygon & indexed by Graph Protocol to solve the world’s problems in a decentralized way. Geo is indexed by a subgraph on the @graphprotocol’s decentralized network—which also powers the display of data from live queries from other subgraphs on the network.

Users can feed subgraph data into Geo pages, paying GRT query fees in the process. IPFS is used to store the content itself and thus resolves the content/proposals/votes connections without posting an overly large amount of information in the smart contract transactions.

Proposals and votes are posted in bulk actions, almost like filling up a shopping cart on Polygon.

Join the Beta Waitlist here:


:space_invader: The Graph Foundation Awards Messari $12.5mm in First-Ever Core Subgraph Developer Grant to Build and Standardize Subgraphs :space_invader:

The Graph Foundation has awarded Messari a $12.5mm* grant to become the first Core Subgraph Developer. The Graph Network is a decentralized indexing protocol for organizing blockchain data. Applications use GraphQL to query open APIs from subgraphs, to retrieve data indexed on the network.

With this grant, Messari will contribute its developer talent and domain expertise to develop and maintain high-quality, accurate, complex, and standardized protocol subgraphs. All subgraphs will be open source and available for the broader community. Each subgraph will be migrated to the decentralized network. Moreover, Messari will lead subgraph working groups, provide feedback to help inform core protocol efforts, share subgraph best practices, and help create community-wide standards for any protocol type (DEX, Lending, Yield Aggregator, Staking etc.).

Read more about the first ever Core Subgraph Dev Grant here:

https://messari.io/article/the-graph-foundation-awards-messari-usd12-5mm-in-first-ever-core-subgraph-developer-grant-to-build-and-standardize-subgraphs


:floppy_disk: The Graph’s Hosted Service has completed a beta integration with Arweave :floppy_disk:

This integration is a step forward for organizing all of web3’s storage data—& making it permanently searchable & accessible.

Now developers can query tx IDs with better support for file fetching coming soon!

Learn how to deploy an Arweave subgraph here:


:building_construction: The Graph’s Hosted Service has completed a beta integration with Harmony Protocol :building_construction:

Developers at Harmony can build and deploy their subgraphs. These subgraphs can be composed into a global graph of all the world’s public information, making data easily accessible.

Learn how to deploy subgraphs here:


:astronaut:t2: The Graph SubgraphDAO was announced :astronaut:t2:

The Graph SubgraphDAO was announced! It’s a a community body dedicated to supporting & sustaining subgraphs to index all of the world’s data.

The mission of SubgraphDAO is to support, fund & empower developers actively using subgraphs or in need of new subgraphs. SubgraphDAO will scale efforts in The Graph ecosystem & enable the community to support subgraph development, education & grants in a decentralized way.

SubgraphDAO will focus on stewarding:

:globe_with_meridians: Gathering & defining subgraph requirements
:globe_with_meridians: Subgraph grants & support
:globe_with_meridians: Standards working groups & education
:globe_with_meridians: Curation bootstrapping

The DAO aims to be the community-driven, one-stop-shop for all things subgraphs!

SubgraphDAO will help anyone go from 0 to subgraph developer. The DAO will educate new subgraph devs, set standards, award grants & match subgraph devs with teams. The DAO will also support subgraph maintenance until a protocol DAO is ready to take it over long-term.

Similar to The Graph AdvocatesDAO, SubgraphDAO is applying for a grant from The Graph Council to fund subgraph education, grants & DAO activities. Over the next few weeks, the DAO will bootstrap operations & launch a Charter with support from the community & the Foundation.

Welcome subgraph enthusiasts, GraphQL wizards & open data believers! Let’s bring subgraphs to the masses :astronaut: Join the conversation about SubgraphDAO, its mission and learn how to get involved in the DAO’s discord:


:bank: The Graph Foundation Allocates Nearly $1M in Wave 6 of Grants :bank:

It’s been a tremendous year so far for the growth of The Graph ecosystem. Over $9 million in grants has been paid out, with nearly $12 million allocated to date since the launch of the grants program in 2021. In the same time period, The Graph ecosystem has welcomed five additional core devs, including GraphOps and its first core subgraph developer, Messari, that were both announced at Graph Day in June. The Graph Foundation is maintaining and nurturing this momentum by onboarding the next cohort of grantees: over $800,000 in Wave 6 grants has been allocated towards protocol improvements, providing support to dapps building with subgraphs, creating new tools, and growing the community!

This wave also marks the end of an era and the beginning of a new chapter in decentralization: The Graph Foundation is officially passing responsibility of community grants to the Graph AdvocatesDAO! Anyone can apply for a grant from the AdvocatesDAO, as long as it follows the DAO’s goals for community growth and is aligned with the DAO’s Charter. The DAO received a $250,000 grant from The Graph Council in Q1 and will be serving community grants valued at $20,000 or under. Prospective grantees should expect to see diversification of opinions, votes, and funding with upcoming grant waves. Community grants will now be evaluated by Graph Advocates participating in community governance through the Graph AdvocatesDAO. Interested in being a part of the mission? Find out more about how you can contribute to The Graph ecosystem with the Graph Advocates program!

Another step bolstering decentralization is in the works: SubgraphDAO, a recently announced DAO supporting subgraph development. The SubgraphDAO will empower a community of subgraph developers, who aim to educate other subgraph developers and grow the pool of contributors who can work with dapps to help build out their subgraph needs. The DAO will also support subgraph migration, fund subgraph grants, and even maintain public goods subgraphs.

While SubgraphDAO takes shape, the Foundation will continue to evaluate grants for building subgraphs and migration grants to dapp teams publishing subgraphs onto The Graph Network. Moving forward, however, subgraph grants will be sized in GRT as opposed to past waves where grants were valued in USDC. Awarding grants to subgraph developers in GRT will help with bootstrapping the developers’ experience on the network, so they can be the first to signal on the subgraph and cover initial query fees. The focus on decentralization as the network moves toward sunsetting the hosted service makes subgraph grants a priority.

In addition to subgraphs, tooling and protocol R&D continue to be priorities. Follow progress on protocol improvements in The Graph Forum where you can also suggest new tooling, instrumentation, or research and receive feedback on your ideas!

If you have ideas or if you’d like to receive support ahead of Wave 7, make sure to apply for a grant. The application portal is always open!

Read more on the 6th wave of Grants here:


:bearded_person:t2: Soulbould Labs announced Soulbound Studio - A no-code Subgraph building GUI :bearded_person:t4:

Subgraphs are magical tools that unlock completely new paradigms previously unavailable to Web3 developers. They are the primitive that empowers DAOs, protocols, and Web3 communities to define, compute, and access meaningful data from decentralized networks.

However, subgraph development comes with a steep technical barrier to entry.
To build and deploy a functional subgraph, developers must familiarize themselves with Assembly Script, Graph CLI, GraphQL, Graph Nodes, subgraph config files, smart contract architecture, and The Graph protocol.

That’s enough friction to prevent many developers and non-technical Web3 contributors from accessing the power of subgraphs and fully participating in The Graph ecosystem. Even with the rich documentation and abundance of educational programs that The Graph provides, it usually takes two to four weeks to onboard a new subgraph developer.

Soulbound Labs decided a simpler solution should exist. In partnership with The Graph Foundation, Soulbound Labs is proud to announce the alpha release of Soulbound Studio–an online whiteboard for collaborative subgraph development. Soulbound Studio features a clean, drag-and-drop, no-code GUI with one-click subgraph deployment in under 30 seconds. The interface is designed to be immediately accessible for both technical and nontechnical Web3 contributors.

Alpha release update scheduled for Thursday, Aug 4th! Try it yourself here:

https://soulbound.xyz


:astronaut:t2: Catch up on Indexer Office Hours :astronaut:t2:

IOH Youtube Channel:

IOH Twitter Page:

https://twitter.com/TheGraphIOH


:astronaut:t3: Catch up on GRTiQ Podcasts :astronaut:t3:

Check out all the podcasts here:


:astronaut:t5: Catch up on The Index Podcasts :astronaut:t5:

Check out all the podcasts here:

1 Like

Fantastic overview :heart_eyes: :ok_hand:

1 Like