Hi Indexers!
Semiotic.ai is happy to announce the release of its Indexer Stress-Test tool: GitHub - semiotic-ai/indexer-stresstest.
Why?
Indexers respond to GraphQL queries originating from dApp end-users. It can be expensive for Indexers to maintain a high quality of service. In order to measure the impact of high query traffic on their indexing infrastructure, Semiotic.ai (operator of joystick.eth) created the Indexer Stress-Test tool. IST sends a continuous stream of random queries to a subgraph endpoint. IST can be used to estimate CPU, RAM, and egress costs for a given subgraph.
In more technical detail:
The Indexer Stress-Test tool sends a continuous stream of random queries to a subgraph endpoint. IST largely relies on IBM’s graphql-query-generator.
-
IST uses the endpoint’s schema to determine the Objects Types and Fields it can use. The schema is retrieved with get_schema.ts.
-
IST randomly uses ID arguments (only). IDs for all the Object Types have to be retrieved from the endpoint beforehand using get_providermap.ts.
-
stresstest.ts randomly composes queries using the information above, with repetitions, nesting, etc., and sequentially sends them to the endpoint. It waits for the query response after each query is sent. You can run multiple instances in parallel to stress the endpoint harder.
The main limitation is that the only argument it can use is ID. All the other arguments, such as first, skip, block and where are not used by graphql-query-generator.
Ongoing and future research:
Semiotic is engaged in ongoing research for The Graph Foundation that may result in Indexer Stress-Test 2.0. IST 2.0 would have an entirely new codebase (and GitHub repository), and it would support the goal of making query cost prediction and, ultimately, query fee selection, easier for Indexers. We are leveraging techniques from Natural Language Processing that enable the prediction of resource costs (CPU, RAM, and egress) directly from GraphQL queries. We have had initial success training NLP neural networks for resource cost prediction, but we need more unique data to achieve robust prediction models. The plan for IST 2.0 is to use Reinforcement Learning to adaptively generate queries that surprise our prediction models and therefore help them improve over time. This forum thread will be updated with any future announcements regarding IST 2.0.
The IST efforts are made possible by the support of The Graph Foundation and through technical collaboration and guidance from The Graph Core Devs / Core Devs.
