Request for information about disputes #GDR-10

86b · July 19, 2022, 4:57pm

The arbitrators are contacting indexer address 0x4437e367cc68d87675148a9e4a1f052f0359c880 (mind-heart-soul) about disputes filed in the protocol.

To make the investigation as brief as possible, please provide the following information and any other relevant records about the open disputes:

Version of graph-node used.
Graph-node configuration in full.
Type and version of Ethereum node.
Table of PoIs generated by the affected subgraphs.
A dump of the call cache data from the database for the affected subgraphs.
Entity values as of the divergent block once identified.

This is not an all-inclusive list. Request for additional information may be made if considered necessary.

How to Get Relevant Data

You can use the following queries to extract information from the indexer software to be used in the evaluation.

# Get subgraphs schema
SELECT name FROM public.deployment_schemas where subgraph = 'QmdoTSPsdt491REYXDheBJwLvNXUvKsNGkiHJZMYNSca22';

# Move to the subgraphs schema
SET SEARCH_PATH TO <RESULT_FROM_ABOVE>;

# Dump PoI TABLE - (from outside of psql console but on server)
pg_dump --dbname="<YOUR_DB_NAME" --host=<DB_HOST> --port=5432 --username=<USERNAME>  --table='<SUBGRAPH_SCHEMA>."poi2$"' --file='<FILE_TO_SAVE_TO>'

# Dump call cache (on some setups this table may be in a different schema, check it with `select * from public.chains`)
pg_dump --dbname="<YOUR_DB_NAME" --host=<DB_HOST> --port=5432 --username=<USERNAME>  --table='public.eth_call_cache"' --file='<FILE_TO_SAVE_TO>'

Once a divergent block is identified:

# loop through all entity tables and get changes for that block
# for each entity table in subgraph deployment schema:
select * from <entity_table> where lower(<DIVERGENT_BLOCK>);

Purpose of the Requirement

This requirement is related to the following disputes:

Dispute (0xdb90a863a2986a88c6653d537e1ceb225c544727cd5b50190e14239f800f4097)
├─ Type: Indexing
├─ Status: Undecided (26.74 days ago)
├─ Indexer: mind-heart-soul (0x4437e367cc68d87675148a9e4a1f052f0359c880)
├─ Fisherman: 0x93372d846b2f73cc34a03f44e0a9b09a037daa77
├─ SubgraphDeployment
│  └─ id: 0xe5bd3de08a2b9236582c78a990f5ed13f26519923172c5610de5c9aefac657c7 (QmdoTSPsdt491REYXDheBJwLvNXUvKsNGkiHJZMYNSca22)
├─ Economics
│  ├─ indexerSlashableStake: 102754.586839378639625538 GRT
│  └─ indexingRewardsCollected: 14901.4995700776049 GRT
├─ Allocation
│  ├─ id: 0xff90561c197db14b813433d802e25880720d3228
│  ├─ createdAtEpoch: 364
│  ├─ createdAtBlock: 0xa000ef8109e6e3680d9ac27a9311d874eb6936142284664c9a27d3cf4c7c6b17
│  ├─ closedAtEpoch
│  │  ├─ id: 394
│  │  └─ startBlock: 0x7568f92204f2f353574be2d5727c8122d4a4ec982a599ab3a69a9505874df054 (#14065292)
│  └─ closedAtBlock: 0x36a2b47326d082504608741d59a54e3a437ef34479568035ceb9e35cc8d1f057 (#14065945)
└─ POI
   ├─ submitted: 0x960dbfcc56fc6644f889d815f497deef9f30136d2339daefaa4a788f566f868a
   ├─ match: Not-Found
   ├─ previousEpochPOI: Not-Found
   └─ lastEpochPOI: Not-Found

About the Procedure

The Arbitration Charter regulates the arbitration process. You can find it in Radicle project ID rad:git:hnrkrhnth6afcc6mnmtokbp4h9575fgrhzbay

For communications, please use this forum. Additionally, please monitor Graph Protocol’s Discord #arbitration (https://discord.gg/9gg7XvfggW ) for any arbitrator reaching out for more information.

86b · July 23, 2022, 2:31am

Note: I had already previously worked with @Koen obtaining the necessary information needed to address this dispute back in April and May, but an official forum post was never made.

Investigation

12057776 was the first observed failed block

Error:

Critical error logged in mapping wasm backtrace: 0: 0xe7a - <unknown>!~lib/@graphprotocol/graph-ts/index/log.critical 1: 0xe9e - <unknown>!src/entities/SetToken/getSetToken 2: 0x13e3 - <unknown>!src/mappings/StreamingFeeModule/handleFeeActualized in handler `handleFeeActualized` at block #12057776 (846a448e56ab724fd0cca44f193c8c4bdf73c2cd06246d5764946e222d3ebd52)

Reference epoch and block info for POI:
Epoch: 91
Block: 12051554
Hash: 0x288b07e20b4c6aee8a5bd029b337980a5d342a70d2232be081cb72ee2a6fc4b1

POI Query:

http post http://localhost:8030/graphql \
query='{
    proofOfIndexing(
        subgraph: "QmdoTSPsdt491REYXDheBJwLvNXUvKsNGkiHJZMYNSca22"
        blockHash: "0x288b07e20b4c6aee8a5bd029b337980a5d342a70d2232be081cb72ee2a6fc4b1"
        blockNumber: 12051554
        indexer: "0x4437e367cc68d87675148a9e4a1f052f0359c880"
        )
}'

{
    "data": {
        "proofOfIndexing": "0x960dbfcc56fc6644f889d815f497deef9f30136d2339daefaa4a788f566f868a"
    }
}

Indexer’s POI: 0x960dbfcc56fc6644f889d815f497deef9f30136d2339daefaa4a788f566f868a
My POI Result: 0x960dbfcc56fc6644f889d815f497deef9f30136d2339daefaa4a788f566f868a

Feedback

POI match found. This subgraph error can be “bounced” past the failure block, but the subgraph will continue to fail with the same error at a later block. Because of this, there’s a chance the Fisherman wound up on a later epoch reference block than the Indexer, prompting a POI mismatch between the two when calculating the POI.

If Fisherman 0x93372d846b2f73cc34a03f44e0a9b09a037daa77 would like to provide additional info, please try to do so in short order so the Arbitration Council can address these in a timely manner.

Thanks!

pete · July 26, 2022, 2:10am

I opened this case because I noticed that 0x4437e367cc68d87675148a9e4a1f052f0359c880 closed 6 allocations (163 epochs duration) on subgraph QmdoTSPsdt491REYXDheBJwLvNXUvKsNGkiHJZMYNSca22 with the same POI. It is indeed known that this subgraph crashes during epoch 91. Yet the indexer continued to open and close allocations, and therefore collect rewards, while there is no proof that any indexing was done/attempted during that timespan. I would expect that like most indexers, this indexer would notice the subgraph failure, and stop allocating to it within a reasonable amount of time.

Note that after those 6 allocations, 3 more were made over a timespan of 103 epochs, with 2 different POIs, then finally the same POI as the first 6 allocations. These 2 new POI values were probably due to Errored subgraphs sometimes resume correctly after restart · Issue #3236 · graphprotocol/graph-node · GitHub.

Supporting material:

{
	allocations(
		where: {
			indexer: "0x4437e367cc68d87675148a9e4a1f052f0359c880"
			status: Closed
			subgraphDeployment: "0xe5bd3de08a2b9236582c78a990f5ed13f26519923172c5610de5c9aefac657c7"
		}
		orderBy: createdAtEpoch
	) {
		poi
		indexingRewards
		createdAtEpoch
		createdAtBlockNumber
		closedAtEpoch
		closedAtBlockNumber
	}
}

{
	"data": {
		"allocations": [
			{
				"poi": "0x960dbfcc56fc6644f889d815f497deef9f30136d2339daefaa4a788f566f868a",
				"indexingRewards": "12517824387947562000000",
				"createdAtEpoch": 255,
				"createdAtBlockNumber": 13144254,
				"closedAtEpoch": 282,
				"closedAtBlockNumber": 13325238
			},
			{
				"poi": "0x960dbfcc56fc6644f889d815f497deef9f30136d2339daefaa4a788f566f868a",
				"indexingRewards": "16943740849092654000000",
				"createdAtEpoch": 282,
				"createdAtBlockNumber": 13325273,
				"closedAtEpoch": 310,
				"closedAtBlockNumber": 13508947
			},
			{
				"poi": "0x960dbfcc56fc6644f889d815f497deef9f30136d2339daefaa4a788f566f868a",
				"indexingRewards": "19752864251862020000000",
				"createdAtEpoch": 310,
				"createdAtBlockNumber": 13508992,
				"closedAtEpoch": 337,
				"closedAtBlockNumber": 13687403
			},
			{
				"poi": "0x960dbfcc56fc6644f889d815f497deef9f30136d2339daefaa4a788f566f868a",
				"indexingRewards": "21919595483685160000000",
				"createdAtEpoch": 337,
				"createdAtBlockNumber": 13687420,
				"closedAtEpoch": 364,
				"closedAtBlockNumber": 13871826
			},
			{
				"poi": "0x960dbfcc56fc6644f889d815f497deef9f30136d2339daefaa4a788f566f868a",
				"indexingRewards": "22925383953965546000000",
				"createdAtEpoch": 364,
				"createdAtBlockNumber": 13871861,
				"closedAtEpoch": 394,
				"closedAtBlockNumber": 14065945
			},
			{
				"poi": "0x960dbfcc56fc6644f889d815f497deef9f30136d2339daefaa4a788f566f868a",
				"indexingRewards": "16965074387838774000000",
				"createdAtEpoch": 394,
				"createdAtBlockNumber": 14065958,
				"closedAtEpoch": 418,
				"closedAtBlockNumber": 14226350
			},
			{
				"poi": "0xbd53d2815974e43992abab12c5991968516e47a59c1cfe3f5233bd2db25d5c8e",
				"indexingRewards": "22071814855367954000000",
				"createdAtEpoch": 418,
				"createdAtBlockNumber": 14226358,
				"closedAtEpoch": 445,
				"closedAtBlockNumber": 14407548
			},
			{
				"poi": "0xf44a12e38ef734a158aa3a84dbb6746b1adc026f54b9bb5b65dcbdb6c73852a7",
				"indexingRewards": "17400990305538816101758",
				"createdAtEpoch": 446,
				"createdAtBlockNumber": 14417450,
				"closedAtEpoch": 473,
				"closedAtBlockNumber": 14590459
			},
			{
				"poi": "0x960dbfcc56fc6644f889d815f497deef9f30136d2339daefaa4a788f566f868a",
				"indexingRewards": "18132994460982098651968",
				"createdAtEpoch": 473,
				"createdAtBlockNumber": 14590467,
				"closedAtEpoch": 497,
				"closedAtBlockNumber": 14754555
			}
		]
	}
}

86b · December 22, 2022, 9:12pm

Previous Notes:

12057776 is the first observed failed block

Reference epoch and block info for POI:
Epoch: 91
Block: 12051554
Hash: 0x288b07e20b4c6aee8a5bd029b337980a5d342a70d2232be081cb72ee2a6fc4b1

Error:

Critical error logged in mapping wasm backtrace: 0: 0xe7a - <unknown>!~lib/@graphprotocol/graph-ts/index/log.critical 1: 0xe9e - <unknown>!src/entities/SetToken/getSetToken 2: 0x13e3 - <unknown>!src/mappings/StreamingFeeModule/handleFeeActualized in handler `handleFeeActualized` at block #12057776 (846a448e56ab724fd0cca44f193c8c4bdf73c2cd06246d5764946e222d3ebd52)

Query:

http post http://localhost:8030/graphql \
query='{
    proofOfIndexing(
        subgraph: "QmdoTSPsdt491REYXDheBJwLvNXUvKsNGkiHJZMYNSca22"
        blockHash: "0x288b07e20b4c6aee8a5bd029b337980a5d342a70d2232be081cb72ee2a6fc4b1"
        blockNumber: 12051554
        indexer: "0x4437e367cc68d87675148a9e4a1f052f0359c880"
        )
}'

{
    "data": {
        "proofOfIndexing": "0x960dbfcc56fc6644f889d815f497deef9f30136d2339daefaa4a788f566f868a"
    }
}

Indexer POI: 0x960dbfcc56fc6644f889d815f497deef9f30136d2339daefaa4a788f566f868a

Our POI Result: 0x960dbfcc56fc6644f889d815f497deef9f30136d2339daefaa4a788f566f868a

Match: Yes

Notes & Findings

POI match found. This subgraph error can be “bounced” past the failure block, but the subgraph will continue to fail with the same error at said later block. Because of this, there’s a chance the Fisherman was on a later block than the Indexer, prompting a POI mismatch between the two when the Fisherman was calculating the POI.

Fisherman comment:

I opened this case because I noticed that 0x4437e367cc68d87675148a9e4a1f052f0359c880 closed 6 allocations (163 epochs duration) on subgraph QmdoTSPsdt491REYXDheBJwLvNXUvKsNGkiHJZMYNSca22 with the same POI. It is indeed known that this subgraph crashes during epoch 91. Yet the indexer continued to open and close allocations, and therefore collect rewards, while there is no proof that any indexing was done/attempted during that timespan. I would expect that like most indexers, this indexer would notice the subgraph failure, and stop allocating to it within a reasonable amount of time.

Given the fisherman’s response, it’s clear that this dispute was filed because they were unaware of the rules surrounding broken subgraphs that still have valid curation signal.

Both parties seemed to be operating in good faith at the time this dispute was submitted, with minor negligence on the Fisherman’s behalf with regards to failed subgraph POI rules.

Along with the statute of limitations, this dispute has been settled as a draw by the Arbiters:

Topic		Replies	Views
Request for information about disputes #GDR-11 Arbitration	5	3760	December 22, 2022
Request for information about disputes #GDR-8 Arbitration	17	2842	January 14, 2022
Request for information about disputes #GDR-7 Arbitration	13	2744	January 19, 2022