Request for information about disputes #GDR-13

[86b]

The arbitrators are contacting indexer address 0xb06071394531b63b0bac78f27e12dc2beaa913e4 (protofire) about disputes filed in the protocol.

To make the investigation as brief as possible, please provide the following information and any other relevant records about the open disputes:

• Version of graph-node used.
• Graph-node configuration in full.
• Type and version of Ethereum node.
• Table of PoIs generated by the affected subgraphs.
• A dump of the call cache data from the database for the affected subgraphs.
• Entity values as of the divergent block once identified.

This is not an all-inclusive list. Request for additional information may be made if considered necessary.

How to Get Relevant Data

You can use the following queries to extract information from the indexer software to be used in the evaluation.

# Get subgraphs schema
SELECT name FROM public.deployment_schemas where subgraph = 'QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6';

# Move to the subgraphs schema
SET SEARCH_PATH TO <RESULT_FROM_ABOVE>;

# Dump PoI TABLE - (from outside of psql console but on server)
pg_dump --dbname="<YOUR_DB_NAME" --host=<DB_HOST> --port=5432 --username=<USERNAME>  --table='<SUBGRAPH_SCHEMA>."poi2$"' --file='<FILE_TO_SAVE_TO>'

# Dump call cache (on some setups this table may be in a different schema, check it with `select * from public.chains`)
pg_dump --dbname="<YOUR_DB_NAME" --host=<DB_HOST> --port=5432 --username=<USERNAME>  --table='public.eth_call_cache"' --file='<FILE_TO_SAVE_TO>'

Once a divergent block is identified:

# loop through all entity tables and get changes for that block
# for each entity table in subgraph deployment schema:
select * from <entity_table> where lower(<DIVERGENT_BLOCK>);

Purpose of the Requirement

This requirement is related to the following disputes:

Dispute (0xa90f3240bd40c63359464063b4d8e76b67d1585f0f1e5303b588109feb0c559c)
├─ Type: Indexing
├─ Status: Undecided (2.98 days ago)
├─ Indexer: protofire (0xb06071394531b63b0bac78f27e12dc2beaa913e4)
├─ Fisherman: 0xc56961836857210e256d71c91a62e90865075380
├─ SubgraphDeployment
│  └─ id: 0xa1a64dc25cf8e6bb6adc4072b3027cd12efca9d323421f91859e1bc796cf2def (QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6)
├─ Economics
│  ├─ indexerSlashableStake: 572123.37197758264785916 GRT
│  └─ indexingRewardsCollected: 420.819777305358077637 GRT
├─ Allocation
│  ├─ id: 0x23989375ebdb245bbde333dace036ece75f16c0a
│  ├─ createdAtEpoch: 547
│  ├─ createdAtBlock: 0x4c8d25dc9a8d5c2f8988a5224ab64762e2426f8f66d1576cfebee884edc78f5c
│  ├─ closedAtEpoch
│  │  ├─ id: 548
│  │  └─ startBlock: 0xc1ee3e5f6a32d27c59ab250f70d1d0fbf08dd49bf69e2af2f7969bdee019a77c (#15088776)
│  └─ closedAtBlock: 0x6894bdd62ae52610fdf3881fe902f4af9b625cd5fad732421aa5f5abb591070c (#15090955)
└─ POI
   ├─ submitted: 0x4c2c1b705092d41efaeddea5d2aac6d2a844bf3d90976a8b4fb9e14976075f51
   ├─ match: Not-Found
   ├─ previousEpochPOI: Not-Found
   └─ lastEpochPOI: Not-Found

About the Procedure

The Arbitration Charter regulates the arbitration process. You can find it in Radicle project ID rad:git:hnrkrhnth6afcc6mnmtokbp4h9575fgrhzbay

For communications, please use this forum. Additionally, please monitor Graph Protocol’s Discord arbitration (https://discord.gg/9gg7XvfggW ) for any arbitrator reaching out for more information.

[bleiva.protofire]

@86b

0x23989375ebdb245bbde333dace036ece75f16c0a

Indexer agent version: v0.19.3
graph-node version: 0.26
Erigon version: Erigon1 [2022.04.02-beta]

Hi I’m Brian from Protofire. This is our first request about disputes we receive since after running our indexer cluster for over a year and half

After checking out why we received this request, we realized that we got an indexing error when we were indexing this subgraph.

We usually run some scrips for many operations tasks like open and close allocations and one of them run indexing stop rules when we detect any error in any subgraph using. It seems that the indexer agent recommended us to use a valid POI to close this allocation

And I don’t know why our script closed it automatically after detecting indexing error without verifying correctly that the subgraph was not indexing correctly.

I want it to be clear that there was no malicious intent on our side. We are still debugging this.

If you need more information please let us know

Thank you!

[bleiva.protofire]

[86b]

Previous Notes:

Notes & Findings

This particular subgraph is in a weird failed state… one that I haven’t personally seen before. It’s basically scanning blocks, adds a handful of entities, then restarts this process:

index-node-erigon       | Aug 02 18:06:03.121 INFO Scanning blocks [14800903, 14801402], range_size: 500, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: BlockStream
index-node-erigon       | Aug 02 18:06:03.132 INFO Scanning blocks [14801403, 14801902], range_size: 500, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: BlockStream
index-node-erigon       | Aug 02 18:10:02.011 INFO Scanning blocks [14762292, 14762292], range_size: 1, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: BlockStream
index-node-erigon       | Aug 02 18:10:02.032 INFO Create data source, params: 0x27ebc6019acb9c760483cc13e9256fbcc1ed8e85, name: ERC1155, block_hash: 0xb474d299a6cd6480b3b3d3da6ee49ff5e8b6f6f3805a22a63ae74fe37b83e72c, block_number: 14762292, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: SubgraphInstanceManager
index-node-erigon       | Aug 02 18:10:02.033 INFO Create data source, params: 0xe19b772ba1d7e0f11e82de80d28f59ad85eea10e, name: ERC1155, block_hash: 0xb474d299a6cd6480b3b3d3da6ee49ff5e8b6f6f3805a22a63ae74fe37b83e72c, block_number: 14762292, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: SubgraphInstanceManager
index-node-erigon       | Aug 02 18:10:02.034 INFO Create data source, params: 0x2f52c085daa8213e1577bb02ded6049b14a05a1c, name: ERC1155, block_hash: 0xb474d299a6cd6480b3b3d3da6ee49ff5e8b6f6f3805a22a63ae74fe37b83e72c, block_number: 14762292, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: SubgraphInstanceManager
index-node-erigon       | Aug 02 18:10:02.035 INFO Done processing trigger, gas_used: 476090559, data_source: TokenFactory, handler: handleBatchCollectionsAdded, total_ms: 1, transaction: 0x8ec7…dc1b, address: 0xf163…5154, signature: BatchCollectionsAdded(address[],string[]), block_hash: 0xb474d299a6cd6480b3b3d3da6ee49ff5e8b6f6f3805a22a63ae74fe37b83e72c, block_number: 14762292, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: SubgraphInstanceManager
index-node-erigon       | Aug 02 18:10:02.037 INFO Scanning blocks [14762293, 14762302], range_size: 10, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: BlockStream
index-node-erigon       | Aug 02 18:10:02.037 WARN no runtime hosted created, there is already a runtime host instantiated for this data source, address: 27ebc6019acb9c760483cc13e9256fbcc1ed8e85, name: ERC1155, block_hash: 0xb474d299a6cd6480b3b3d3da6ee49ff5e8b6f6f3805a22a63ae74fe37b83e72c, block_number: 14762292, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: SubgraphInstanceManager
index-node-erigon       | Aug 02 18:10:02.038 WARN no runtime hosted created, there is already a runtime host instantiated for this data source, address: e19b772ba1d7e0f11e82de80d28f59ad85eea10e, name: ERC1155, block_hash: 0xb474d299a6cd6480b3b3d3da6ee49ff5e8b6f6f3805a22a63ae74fe37b83e72c, block_number: 14762292, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: SubgraphInstanceManager
index-node-erigon       | Aug 02 18:10:02.039 WARN no runtime hosted created, there is already a runtime host instantiated for this data source, address: 2f52c085daa8213e1577bb02ded6049b14a05a1c, name: ERC1155, block_hash: 0xb474d299a6cd6480b3b3d3da6ee49ff5e8b6f6f3805a22a63ae74fe37b83e72c, block_number: 14762292, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: SubgraphInstanceManager
index-node-erigon       | Aug 02 18:10:02.041 INFO Scanning blocks [14762303, 14762402], range_size: 100, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: BlockStream
index-node-erigon       | Aug 02 18:10:02.048 INFO Scanning blocks [14762403, 14762902], range_size: 500, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: BlockStream
index-node-erigon       | Aug 02 18:10:02.051 INFO Applying 4 entity operation(s), block_hash: 0xb474d299a6cd6480b3b3d3da6ee49ff5e8b6f6f3805a22a63ae74fe37b83e72c, block_number: 14762292, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: SubgraphInstanceManager
index-node-erigon       | Aug 02 18:10:02.052 ERRO Subgraph failed with non-deterministic error: Error while processing block stream for a subgraph: store error: operator does not exist: bytea = text, retry_delay_s: 480, attempt: 2, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: SubgraphInstanceManager
index-node-erigon       | Aug 02 18:10:02.059 INFO Scanning blocks [14762903, 14763402], range_size: 500, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: BlockStream
index-node-erigon       | Aug 02 18:10:02.099 INFO Scanning blocks [14763403, 14763902], range_size: 500, sgd: 8, subgraph_id: QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6, component: BlockStream

The last good block to obtain a POI from is 14762291 . It appears the Fisherman’s indexer-agent recommended a latestValidPoi as could be seen in their logs provided.

I ran a script checking from the first to last good block on this subgraph (203 blocks total) and found that Protofire’s POI matches block 14762140 with hash 0xe66e70c2c3edda08159fd660186116a2947cf6b439a83be8c6827b4c11edb24c :

http post http://localhost:8030/graphql \
query='{
    proofOfIndexing(
        subgraph: "QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6"
        blockHash: "0xe66e70c2c3edda08159fd660186116a2947cf6b439a83be8c6827b4c11edb24c"
        blockNumber: 14762140
        indexer: "0xb06071394531b63b0bac78f27e12dc2beaa913e4"
        )
}'

{
    "data": {
        "proofOfIndexing": "0x4c2c1b705092d41efaeddea5d2aac6d2a844bf3d90976a8b4fb9e14976075f51"
    }
}

Given this, it looks like Protofire was operating in good faith, but we’ll have to establish clarity with the community on the failed-subgraph-POI-reference block which should always be the first block of the epoch where the failure took place. In this case if their last good block was 14762140 , then their Epoch-Reference block for manual POI generation should have been 14756476 which is the first block of Epoch 498, the epoch in which the failure took place.

The Fisherman in this dispute was submitting it due to a lack of clarity surrounding using unofficial graph-node releases and pre-releases, as was mentioned in GDR-12, however, the indexer in this case wasn’t on a newer release allowing Europa to successfully fully index.

All-in-all, the evidence suggests both parties were operating in good faith and the Arbiters settled this dispute as a draw:

Ethereum Transaction Hash: 0x51dfbd1743... | Etherscan