POIFIER - user-friendly tool for verification of indexing data consistency

Research & Development
1

Our mission

POIFIER - The Graph protocol infrastructure project made by Indexers for Indexers, which aims to help with verification of indexing data consistency.

Long story - short

Indexer is a node operator who provides indexing and query processing services for The Graph protocol consumers.

As the proof of data indexing Indexer submits POI (Proof Of Indexing) to the protocol and gets indexing rewards.

Submitted POI for a block is a digest for all entity store transactions for a specific subgraph deployment up to and including that block.

Since fair Indexer strives to provide service by delivering consistent and up-to-date data, the main challenge is verification of indexing data.

Data verification can be achieved by comparing the current POI generated by Indexer for a particular subgraph and ethereum block to POIs generated by other Indexers if they would generate POI for the same subgraph and ethereum block using referenced indexer ID.

We aim to build an end-to-end solution to implement this idea which comprises client and server (GUI) applications to help Indexer with verification of their indexing data.

Problem statement

For XYZ reason indexer’s node may experience an issue when node would produce “invalid” POI while no visible issue detected by node operator.

Invalid POI is a symptom of indexing data inconsistency and would lead to a case when the indexer would serve a query request with “wrong” data.

Which POI is valid

So far there is very little visibility and no tool to verify whether generated POI is valid or not, but it was supposed by design that POI generated by any indexer having referenced indexer ID for a particular subgraph and block would match.

There are could be multiple approaches, but we would focus on next 2:

  • Centralized, aka Oracle: where all POIs generated by indexers would test against POIs generated by single Oracle

  • Decentralized, aka consensus: where indexer’s POI would test against referenced POI which is being selected as a majority of submitted POIs for given subgraph/epoch/block

Solution

POIFIER is a client-server tool with deep analytics which aims to implement Decentralized approach.

  • On a regular basis Indexers would submit POIs generated for all subgraphs being indexed by their graph-node for a range of Epochs and Ethereum Blocks using referenced indexer ID (0x0…0)

  • POIfier-server would select POI based on major number of appearance for each subgraph/block/epoch and name it as a “Valid POI”

  • POIs would be assessed against “Valid POI”

Benefits

For Graph Developers:

  • On early stage to detect, identify and fix indexing issues
  • To research and investigate multiple indexer / cross ethereum nodes issue

For Indexer:

  • On early stage to detect, identify and fix single indexer issue
  • On early stage to detect, identify and flag developer on multiple indexer issue

GENERAL:

  • Add visibility into indexing data consistency
  • Improve protocol indexing data consistency

Design

POIFIER infrastructure consists of:

  • Poifier-client - python3 client which is deployed within Indexer infrastructure, queries indexer’s graph-node against subgraphs and POI per epoch/block and uploads report on poifier-server on regular basis

  • Poifier-server - user friendly GUI with analytics data, where registered indexer is able to submit to and analyze POIs

    Screenshot 2021-09-08 at 15.05.24
    Screenshot 2021-09-08 at 15.05.241272Ă—868 71.5 KB

Note

  • poifier-client:
    • uses qraphql queries and no db integration needed
    • does require API TOKEN to get authorized on POIfier server
  • POI is being generated based in referenced indexer id (0x0…0) and can not be used for closing allocations

How to use

Step by step guide

  1. Get API Token

WARNING

Do not share your API TOKEN with anyone

  1. Deploy poifier-client

  2. View submitted data on portal via GUI

  3. Now you are able to see POIs submitted by your node along with other Indexer’s POIs

NOTE
Worth to note, this is beta version, where some features may not be available

Best regards,
Grassets Tech & Ryabina

11 Likes
2

Great initiative! Looking forward to learning more at next week’s Office Hours.

1 Like
3

Very good job guys! We will test it soon.
Appreciate this initiative, all Indexers who still don’t check their POI should do that. And it seems that this solution will be really helpful!