Request for information about disputes #GDR-14

The arbitrators are contacting indexer address 0xd75c4dbcb215a6cf9097cfbcc70aab2596b96a9c (semiotic-indexer) about disputes filed in the protocol.

To make the investigation as brief as possible, please provide the following information and any other relevant records about the open disputes:

  • Version of graph-node used.
  • Graph-node configuration in full.
  • Type and version of Ethereum node.
  • Table of PoIs generated by the affected subgraphs.
  • A dump of the call cache data from the database for the affected subgraphs.
  • Entity values as of the divergent block once identified.

This is not an all-inclusive list. Request for additional information may be made if considered necessary.

How to Get Relevant Data

You can use the following queries to extract information from the indexer software to be used in the evaluation.

# Get subgraphs schema
SELECT name FROM public.deployment_schemas where subgraph = 'QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6';

# Move to the subgraphs schema
SET SEARCH_PATH TO <RESULT_FROM_ABOVE>;

# Dump PoI TABLE - (from outside of psql console but on server)
pg_dump --dbname="<YOUR_DB_NAME" --host=<DB_HOST> --port=5432 --username=<USERNAME>  --table='<SUBGRAPH_SCHEMA>."poi2$"' --file='<FILE_TO_SAVE_TO>'

# Dump call cache (on some setups this table may be in a different schema, check it with `select * from public.chains`)
pg_dump --dbname="<YOUR_DB_NAME" --host=<DB_HOST> --port=5432 --username=<USERNAME>  --table='public.eth_call_cache"' --file='<FILE_TO_SAVE_TO>'

Once a divergent block is identified:

# loop through all entity tables and get changes for that block
# for each entity table in subgraph deployment schema:
select * from <entity_table> where lower(<DIVERGENT_BLOCK>);

Purpose of the Requirement

This requirement is related to the following disputes:

Dispute (0xc2f035f63b01216acb12d66e9720b0242c4c678575ab9ec201b25d0d2b1ac72e)
β”œβ”€ Type: Indexing
β”œβ”€ Status: Undecided (2.97 days ago)
β”œβ”€ Indexer: semiotic-indexer (0xd75c4dbcb215a6cf9097cfbcc70aab2596b96a9c)
β”œβ”€ Fisherman: 0xc56961836857210e256d71c91a62e90865075380
β”œβ”€ SubgraphDeployment
β”‚  └─ id: 0xa1a64dc25cf8e6bb6adc4072b3027cd12efca9d323421f91859e1bc796cf2def (QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6)
β”œβ”€ Economics
β”‚  β”œβ”€ indexerSlashableStake: 250000.0025 GRT
β”‚  └─ indexingRewardsCollected: 34.652382949899126087 GRT
β”œβ”€ Allocation
β”‚  β”œβ”€ id: 0x55c86a7725d187bd79ac82d93271d7752d63fc64
β”‚  β”œβ”€ createdAtEpoch: 554
β”‚  β”œβ”€ createdAtBlock: 0x2d8f762a19658256cd2d9412bc0091dca449f03cfd3fb31aea2794de0243698a
β”‚  β”œβ”€ closedAtEpoch
β”‚  β”‚  β”œβ”€ id: 555
β”‚  β”‚  └─ startBlock: 0x13459e090c49bdb217e9609b160685b9a1db155181bdfb3db262a33122c3f39f (#15135298)
β”‚  └─ closedAtBlock: 0xf64c1a25f755d744f744fdef917eb5760235e6436f7f4f6d35d1142a598bb6e4 (#15136959)
└─ POI
   β”œβ”€ submitted: 0x646f45f2e0d2b4a838dcbf8c5030a0505c3b7b2bb734bc6d16d708f4d91fc07d
   β”œβ”€ match: Not-Found
   β”œβ”€ previousEpochPOI: Not-Found
   └─ lastEpochPOI: Not-Found

About the Procedure

The Arbitration Charter regulates the arbitration process. You can find it in Radicle project ID rad:git:hnrkrhnth6afcc6mnmtokbp4h9575fgrhzbay

For communications, please use this forum. Additionally, please monitor Graph Protocol’s Discord #arbitration (https://discord.gg/9gg7XvfggW ) for any arbitrator reaching out for more information.

Hi @86b ,

I’ve received the dispute in regards to the Europa subgraph (ID QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6) and am in the process of reviewing the request. Unfortunately, I no longer have the subgraph deployment schema for this subgraph, as I removed the schema from the database after its failure:

indexer@indexer-shell-0 [~] $ history | grep graphman | grep QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6 | grep -v grep
 436 graphman info QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6
 437 ./graphman info QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6
 439 ./graphman info QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6
indexer@indexer-shell-0 [~] $ 

I understand this was a mistake on my part. However, I will collect the eth_call_cache from around the time period requested. The call cache will be up until 14 July 2022.

According to my shell’s history file, the POI was generated for block get_manual_poi.sh QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6 14762100, one block prior to the reported failure.

I did attempt to close this allocation with a 0x0 POI, but was unable to due to an error message in the contract which I did not understand. (See linked transactions below.) I now understand that the error message indicated that I was to wait more than some number of epochs.

Our indexer is using Erigon Alpha v2022.06.06, graph-node version is v0.26.0, indexer-agent 0.19.3, and indexer-service 0.19.3 with the auto-agora patch.

Some background: this subgraph came up as a result of our testing the allocation optimiser tool which is one on the core developers’ task list. We’re working through a number of different use cases and patterns to try and reflect the wide variety of practices amongst the indexers.

Regards,
Kai

This post was flagged by the community and is temporarily hidden.

Thanks for the reply Kai!

Just to clarify here: You ended up closing out the allocation using 14762100 as your reference block for the manual POI calculation? Also, would you mind sharing your get_manual_poi.sh to better help understand how it’s being calculated on your end?

Thanks!

Sure thing, I’ve off-chained Europa to review for this RFP, and it failed right on the start block. I’m not certain how our nodes made it all the way to 14762100. I suspect that our nodes somehow skipped over the current failing block of 14762089, possibly because of an indexer node crash. I recall us having a number of problems with the index nodes crashing because of some subgraphs failing non-deterministically at the time. I’ll upload the schema as it currently sits on our DB now.

As far as the POI calculator goes, what it really does is queries the administrative endpoint for a POI then reports back with that. Sadly, I’m not familiar enough yet with the graph node code base to know how that GQL query is answered under the hood. See the code below:

#!/bin/sh
# Find all subgraph health:
#query {
#  indexingStatuses {
#    subgraph
#    synced
#    health
#  }
#}
INDEXER_ADDRESS='0xd75C4Dbcb215A6cf9097cfbCc70aAB2596B96a9C'
SUBGRAPH_DEPLOYMENT=$1
BLOCK_NUMBER=$2

if [ -z "$INDEXER_ADDRESS" ]; then
	echo 'Environment variable INDEXER_ADDRESS must be set to a valid indexer address'
	exit 1
fi

if [ -z "$SUBGRAPH_DEPLOYMENT" ] || [ -z "$BLOCK_NUMBER" ]; then
	echo "Usage: $0 [subgraph deployment IPFS hash] [last good block number]"
	exit 1
fi

BLOCK_HEX=$(echo "obase=16; $BLOCK_NUMBER" | bc | xargs printf "0x%s\n")


BLOCK_HASH=$(curl \
	-s -X POST -H 'Content-Type: application/json' \
	--data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["'$BLOCK_HEX'", false],"id":1}' \
	$INDEXER_EVM_RPC_URL | \
	jq -r '.result.hash')
if [ -z "$BLOCK_HASH" ]; then
		echo "Block $BLOCK_NUMBER not found"
		exit 2
fi

gql_query='query { proofOfIndexing(subgraph: \"'$SUBGRAPH_DEPLOYMENT'\", blockHash: \"'$BLOCK_HASH'\", blockNumber: '$BLOCK_NUMBER', indexer: \"'$INDEXER_ADDRESS'\") }'
json="{\"query\":\"$gql_query\"}"

POI=$(curl \
	-s -L -X POST -H 'Content-Type: application/json' \
	-d "$json" http://query-node:8030/graphql | \
	jq -r '.data.proofOfIndexing')
echo $POI
1 Like

Just confirmed, our index node is skipping blocks… somehow. It seems that a restart triggered it to jump up to 14762092. See below:

For the record, all non-pasteable documents will be posted to this public folder.

Our pod config is in the tar file called rfp-gdr14-podcfg.tgz.

I saw this happening to one of my subgraphs too. If you search in discord you’ll see me posting about it. I’m on mobile right now, so I don’t know which one it is, but it’s a problem for sure.

Thanks for the heads up @indexer_payne ! It seemed similar to Bug 3236 reported by @alexis some time back. To better understand the behaviour, I simulated the same conditions by restarting the index node offchaining Europa, but that restart does not seem to be the jump-to-next-block trigger point. I’ve seen this happen on a couple of other subgraphs in the past, however they behaved similar to 3236, not this. Any hypotheses are welcome.

This post was flagged by the community and is temporarily hidden.