# Indexer PoI discrepancy

**URL:** <https://forum.thegraph.com/t/indexer-poi-discrepancy/4011>\
**Category:** Research & Development\
**Tags:** indexer\
**Created:** [February 7, 2023, 8:41am UTC](https://forum.thegraph.com/t/indexer-poi-discrepancy/4011 "2023-02-07T08:41:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![warfollowsme](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.thegraph.com/warfollowsme/32/107_2.png) [@warfollowsme](https://forum.thegraph.com/u/warfollowsme)\
**Post date:** [February 7, 2023, 8:41am UTC](https://forum.thegraph.com/t/indexer-poi-discrepancy/4011/1 "2023-02-07T08:41:08Z")

</div>

I would like to discuss the topic of indexers PoI discrepancy.

Indexers have been noticing this discrepancy for a long time. By default, it is considered that if your PoI does not match, then you have a bad database and you need to resync the subgraph. But the question arises, who exactly has the error, you or the one with whom you are comparing? And then another question, why did this error occur at all?

Ryabina team, together with the Grassets.tech team, created the Poifier tool just to understand how large this problem is and who has it. Unfortunately, in order to conduct at least some analysis, it was necessary to gain a critical mass of users of this tool, which we could not achieve. But thanks to Payne, our tool was added to his docker compose, which greatly increased the number of users and now we can analyze at least some data.

Let’s look at a subgraph with the largest number of reports at the moment.

**subgraph** : [Connext Network - Gnosis](https://poifier.io/deployments/0x8845b7f8921bd96046df0736fc77d4aa3f31ae70c75e06c8d1ae9f535b727c92#epochs)  
**deployment id** : QmXWbpH76U6TM4teRNMZzog2ismx577CkH7dzn1Nw69FcV  
**reports count** (last epoch): 98  
**poi ratio** : 91/7  
This means that out of 98 reports, 91 indexers have the same PoI and only 7 do not. A good result, and now we can definitely answer the question which PoI is valid for this subgraph. But what is interesting is that the remaining 7 indexers, which are difficult to accuse of collusion and we can safely assume that they are not connected in any way, also have the same poi. It turns out that on several indexer servers independent of each other, one and the same error occurred, which led to the fact that the hash is now calculated incorrectly.

It is difficult to name the reason, but as for me, it is definitely necessary to start an investigation into why this happened. Find out what hardware and what software indexers use? Will resyncing this subgraph help them or will the bug reproduce itself?

It is important to make this investigation official by the Graph protocol development team. This is important, because the system of slashing indexers is tied precisely to the correctness of the Poi sent to the network. And if we do not understand the reason why there is a discrepancy between indexers, then it is not clear how non-malicious indexers can be sure that they will not be accused of incorrect poi when closing the allocation?

Here are some more interesting examples:

**subgraph** : [Connext Network - Gnosis](https://poifier.io/deployments/0xd44251bb751a42954ec77b4e8402027dc567b8c4f6b4789b47a72152b0586ff8#epochs)  
**deployment id** : QmcdDo39c8ztsCViQ5kQotjNj4iL3QVq9tnQEqVJXffcD5  
**reports count** : 14  
**poi ratio** : 7/7

**subgraph** : [Uniswap v3 Ethereum](https://poifier.io/deployments/0xd0b0e5b65df45a3fff1a653b4188881318e8459d3338f936aab16c4003884abf#epochs)  
**deployment id** : QmcPHxcC2ZN7m79XfYZ77YmF4t9UCErv87a9NFKrSLWKtJ  
**reports count** : 5  
**poi ratio** : 1/1/1/1/1

**subgraph** : [Radicle](https://poifier.io/deployments/0xeb449d43e21e518cdb0656cc31b1652dc177996b8dc2ddd9f4442cf30c95f12e#epochs)  
**deployment id** : QmeB3CddhtZqNW3FVvsbNeEt2Ek1VSjdwNBU9fJEYrsC3o  
**reports count** : 5  
**poi ratio** : 3/2  
This subgraph is notable for the fact that the poi ratio in it changes from epoch to epoch. In some epoch 5 poi are the same in some ratio is 3/2.

P.S. We have limited access to the Poifier dashboard and you have to be an indexer to enter. Login via Metamask at any address associated with indexers (operators, beneficiary, indexer address itself) If someone from the Graph team needs access, but he/she is not an indexer, let me know, we will add any address to the whitelist.

---

<div class="post-metadata">

**Author:** ![chris](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.thegraph.com/chris/32/657_2.png) [@chris](https://forum.thegraph.com/u/chris)\
**Post date:** [February 14, 2023, 7:30pm UTC](https://forum.thegraph.com/t/indexer-poi-discrepancy/4011/3 "2023-02-14T19:30:48Z")

</div>

@warfollowsme, thank you for raising this! Fully agree that this is an important issue that needs attention. Ensuring that POI discrepancies are minimised is important to making the protocol’s dispute and arbitration processes effective, and to ensuring that the security model of the network is upheld.

Your work at Ryabina (alongside the Grassets.tech team) on POIfier has been an exemplary effort in collecting data to understand the discrepancies that exist today. Thank you! From the data you have presented above, one can already observe a few different archetypes of nondeterminism emerging (e.g. POI forks vs total nondeterminism).

I think we need to prioritise an effort to understand and root-cause each instance of known non-determinism. The graph-node team will be essential to understanding and fixing possible determinism issues originating in graph-node. We will need a good way of separating those issues from ones arising from dependencies in the stack, like blockchain nodes or JSON-RPC load balancers. Aside from the technical work here, there’s likely a lot of operational work to do in collecting stack information and logs.

I also think it would be good for the Foundation or Council to define a set of “mainnet entry requirements” including data integrity KPIs (e.g. % of subgraphs displaying non-determinism issues) for a chain to meet before being eligible for the decentralised network mainnet and indexing rewards. This will create a clear target and incentives to focus efforts on squashing nondeterminism issues.

---

<div class="post-metadata">

**Author:** ![ari](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.thegraph.com/ari/32/2747_2.png) [@ari](https://forum.thegraph.com/u/ari)\
**Post date:** [February 14, 2023, 8:10pm UTC](https://forum.thegraph.com/t/indexer-poi-discrepancy/4011/4 "2023-02-14T20:10:53Z")

</div>

I agree with @chris - this is very important to the network’s health and quality of service and fundamental to ensuring that rewards are fairly distributed.

The data science team at Edge & Node expressed interest in accessing the dashboard. They are already consolidating some data about a few indexers, but it would be interesting to compare that to the POIfier. @warfollowsme please DM me on Discord so we chat about how to get access.

About the proof of indexing, but more generally, the reward distribution mechanism, we need to work on:

1. **Monitoring**  **the consensus of proofs** submitted by all indexing processes. There are multiple parties with tools for doing this today that we can iterate to have a more precise and global coverage. We can identify clusters where groups diverge, which can help find something they have in common, such as running a different version of an RPC node. Additionally, we may find single rogue indexers that any party observing the network can dispute.

2. **Commitment to determinism**

3. **Improving Arbitration Bandwidth**

4. **A transparent Subgraph Availability Oracle (SAO)**

Each of this items could really be GIPs. I’d like to see core devs and the community proposing improvements in these areas and Council working on grants to get those improvements going. Personally I have some ideas for 4 and 3 that will share soon.

---

<div class="post-metadata">

**Author:** ![warfollowsme](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.thegraph.com/warfollowsme/32/107_2.png) [@warfollowsme](https://forum.thegraph.com/u/warfollowsme)\
**Post date:** [February 17, 2023, 10:19am UTC](https://forum.thegraph.com/t/indexer-poi-discrepancy/4011/5 "2023-02-17T10:19:18Z")

</div>

Check discord DM please.
