Request for information about disputes #GDR-11

Governance & GIPs Arbitration
1

The arbitrators are contacting indexer address 0x4437e367cc68d87675148a9e4a1f052f0359c880 (mind-heart-soul) about disputes filed in the protocol.

To make the investigation as brief as possible, please provide the following information and any other relevant records about the open disputes:

  • Version of graph-node used.
  • Graph-node configuration in full.
  • Type and version of Ethereum node.
  • Table of PoIs generated by the affected subgraphs.
  • A dump of the call cache data from the database for the affected subgraphs.
  • Entity values as of the divergent block once identified.

This is not an all-inclusive list. Request for additional information may be made if considered necessary.

How to Get Relevant Data

You can use the following queries to extract information from the indexer software to be used in the evaluation.

# Get subgraphs schema
SELECT name FROM public.deployment_schemas where subgraph = 'QmWXF3jTLo6Wy9MxDZz6WKhET3GFcSFqn6vmrDDYPF7UHK';

# Move to the subgraphs schema
SET SEARCH_PATH TO <RESULT_FROM_ABOVE>;

# Dump PoI TABLE - (from outside of psql console but on server)
pg_dump --dbname="<YOUR_DB_NAME" --host=<DB_HOST> --port=5432 --username=<USERNAME>  --table='<SUBGRAPH_SCHEMA>."poi2$"' --file='<FILE_TO_SAVE_TO>'

# Dump call cache (on some setups this table may be in a different schema, check it with `select * from public.chains`)
pg_dump --dbname="<YOUR_DB_NAME" --host=<DB_HOST> --port=5432 --username=<USERNAME>  --table='public.eth_call_cache"' --file='<FILE_TO_SAVE_TO>'

Once a divergent block is identified:

# loop through all entity tables and get changes for that block
# for each entity table in subgraph deployment schema:
select * from <entity_table> where lower(<DIVERGENT_BLOCK>);

Purpose of the Requirement

This requirement is related to the following disputes:

Dispute (0xff4aeae335ee5fa0323435133055c4eeed29cd4433c8ec9181ea0c58668d1ff1)
├─ Type: Indexing
├─ Status: Undecided (26.72 days ago)
├─ Indexer: mind-heart-soul (0x4437e367cc68d87675148a9e4a1f052f0359c880)
├─ Fisherman: 0x93372d846b2f73cc34a03f44e0a9b09a037daa77
├─ SubgraphDeployment
│  └─ id: 0x79942149e2a7507a726b4c80d4319e4527c9c985cf4f8473e3b1f77d1fe6c3ce (QmWXF3jTLo6Wy9MxDZz6WKhET3GFcSFqn6vmrDDYPF7UHK)
├─ Economics
│  ├─ indexerSlashableStake: 102754.586839378639625538 GRT
│  └─ indexingRewardsCollected: 9281.002506722474031596 GRT
├─ Allocation
│  ├─ id: 0x8ec9f60c4c2f24b7be8daf5bad6f5100afc67c0d
│  ├─ createdAtEpoch: 378
│  ├─ createdAtBlock: 0xe9763aaa73072d96a9fab1fc111ed5d210a4d68efaaf5ecdfb0d81dc80e89b34
│  ├─ closedAtEpoch
│  │  ├─ id: 407
│  │  └─ startBlock: 0x800c2b817ac106023877125b06da6eb6c9463f83100d9ecaab4bcb68fc0b1231 (#14151690)
│  └─ closedAtBlock: 0x1ee52cb4220fcd396ed837e0853213e5104eed3de07178de07fed5efb25961dd (#14155003)
└─ POI
   ├─ submitted: 0xb1aee262759dc051dbbd20fa2ec6c08d8b9db7a12c809076e537b653f95b6291
   ├─ match: Not-Found
   ├─ previousEpochPOI: Not-Found
   └─ lastEpochPOI: Not-Found

About the Procedure

The Arbitration Charter regulates the arbitration process. You can find it in Radicle project ID rad:git:hnrkrhnth6afcc6mnmtokbp4h9575fgrhzbay

For communications, please use this forum. Additionally, please monitor Graph Protocol’s Discord #arbitration (https://discord.gg/9gg7XvfggW ) for any arbitrator reaching out for more information.

2

Note: I had already previously worked with @Koen obtaining the necessary information needed to address this dispute back in April and May, but an official forum post was never made.

Investigation

13380096 is the only observed failed block

Reference epoch and block info for POI:
Epoch: 290
Block: 13374108
Hash: 0xe8791dd112c51966cc37a424827c7f8d81c00c9182619f4dc6ba339d29e029b3

Error:

Error:    failed to process trigger: block #13380099 (0x38d4…b3aa), transaction ca32b1581b837c6ca539cf6c2fd542e9bcacbdc345c6a424382b4f0e48b96153: function handleSwapped not found

Query:

http post http://localhost:8030/graphql \
query='{
    proofOfIndexing(
        subgraph: "QmWXF3jTLo6Wy9MxDZz6WKhET3GFcSFqn6vmrDDYPF7UHK"
        blockHash: "0xe8791dd112c51966cc37a424827c7f8d81c00c9182619f4dc6ba339d29e029b3"
        blockNumber: 13374108
        indexer: "0x4437e367cc68d87675148a9e4a1f052f0359c880"
        )
}'

{
    "data": {
        "proofOfIndexing": "0x9a6c74603dfc19df5e2f09fc18f859bbcf4f598b778dc06ccea6999e294111f3"
    }
}

Indexer’s POI: 0xb1aee262759dc051dbbd20fa2ec6c08d8b9db7a12c809076e537b653f95b6291
My POI Result: 0x960dbfcc56fc6644f889d815f497deef9f30136d2339daefaa4a788f566f868a

Feedback

The latest release at the time this subgraph was indexed was v0.24.2 where future releases addressed determinism issues that were being caused by graph-node restarts. After reviewing the information provided and since the subgraph was indexed with v0.24.x, POI discrepancies are to be expected.

If @Koen or Fisherman 0x93372d846b2f73cc34a03f44e0a9b09a037daa77 would like to provide additional info, please try to do so in short order so the Arbitration Council can address these in a timely manner.

Thanks!

3

I’d like to add the following infos:
(For context, note that we talking about an allocation closed on Feb 6, 2022.)

  • we were running the at the time recommended production versions of graph-node and OpenEthereum respectively. As can be understood from POI / erigon channels on discord and i believe discussed during IOH, there have been discrepancies between OE and erigon, which i’d like to note as another plausible source of my on-chain POI not matching to fisherman.

  • we’ve nuked our entire indexer db, indexed every subgraph from scratch, both after switching from OpenEthereum to Erigon as well as after the skip-over-broken-block bug patch in graph-node was announced production-ready, to be extra diligent in avoiding any remnants that could affect generating matching POI’s.

Since indexer operations at Mind Heart Soul was (and is) running the software as provided, following best-practices and since we’ve responded more than adequately to discovered determinism challenges, it’s my sincere belief to be not at fault -let alone demonstrably exhibited malicious intent- and therefore the dispute should be ruled in our favor.

4

Wouldn’t this be beyond the Statute of Limitations stated in the Arbitration Charter?

  1. Statute of Limitations
    “Indexers should not be slashed for faults that occurred more than two thawing periods prior.
    For example, if the thawing period is set to 28 epochs, then an Indexer should not be slashed for any fault that occurred more than 56 epochs prior.
    The Arbitrator must decide any such dispute that is outside this statute of limitations as a Draw.”
5

Can we get down to a decision on this one? I’m concerned about the stress these disputes are having on Indexers and Fishermen due to the long period of no ruling.

6

Notes from past investigation:

This particular subgraph is a hard failure - it can’t be “bounced” past said failed block 13380096 to eventually reach a later block and new POI. After a couple reindexes, it still fails at the same block specified above. I’m assuming the POI mismatch is related to a divergent POI or a mistake in the block used to calculate it (checked the previous couple epoch blocks to no avail as well). We’ll need to request more information from the indexer to confirm.

Given information submitted by mind-heart-soul aka @Koen , the latest release at the time this subgraph was indexed was v0.24.2 where future releases addressed determinism issues that were being caused by graph-node restarts. After reviewing the information provided and since the subgraph was indexed with v0.24.x, POI discrepancies were to be expected.

The same Fisherman from GRD-10 submitted this dispute and for the same reasons. Both parties seemed to be operating in good faith at the time this dispute was submitted.

Combined with the statute of limitations mentioned by @DataNexus , this dispute was settled as a draw by the Arbiters:

1 Like