The arbitrators are contacting indexer address 0xd75c4dbcb215a6cf9097cfbcc70aab2596b96a9c(semiotic-indexer) about disputes filed in the protocol.
To make the investigation as brief as possible, please provide the following information and any other relevant records about the open disputes:
Version of graph-node used.
Graph-node configuration in full.
Type and version of Ethereum node.
Table of PoIs generated by the affected subgraphs.
A dump of the call cache data from the database for the affected subgraphs.
Entity values as of the divergent block once identified.
This is not an all-inclusive list. Request for additional information may be made if considered necessary.
How to Get Relevant Data
You can use the following queries to extract information from the indexer software to be used in the evaluation.
# Get subgraphs schema
SELECT name FROM public.deployment_schemas where subgraph = 'QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6';
# Move to the subgraphs schema
SET SEARCH_PATH TO <RESULT_FROM_ABOVE>;
# Dump PoI TABLE - (from outside of psql console but on server)
pg_dump --dbname="<YOUR_DB_NAME" --host=<DB_HOST> --port=5432 --username=<USERNAME> --table='<SUBGRAPH_SCHEMA>."poi2$"' --file='<FILE_TO_SAVE_TO>'
# Dump call cache (on some setups this table may be in a different schema, check it with `select * from public.chains`)
pg_dump --dbname="<YOUR_DB_NAME" --host=<DB_HOST> --port=5432 --username=<USERNAME> --table='public.eth_call_cache"' --file='<FILE_TO_SAVE_TO>'
Once a divergent block is identified:
# loop through all entity tables and get changes for that block
# for each entity table in subgraph deployment schema:
select * from <entity_table> where lower(<DIVERGENT_BLOCK>);
Purpose of the Requirement
This requirement is related to the following disputes:
The Arbitration Charter regulates the arbitration process. You can find it in Radicle project ID rad:git:hnrkrhnth6afcc6mnmtokbp4h9575fgrhzbay
For communications, please use this forum. Additionally, please monitor Graph Protocol’s Discord #arbitration (https://discord.gg/9gg7XvfggW ) for any arbitrator reaching out for more information.
I’ve received the dispute in regards to the Europa subgraph (ID QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6) and am in the process of reviewing the request. Unfortunately, I no longer have the subgraph deployment schema for this subgraph, as I removed the schema from the database after its failure:
indexer@indexer-shell-0 [~] $ history | grep graphman | grep QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6 | grep -v grep
436 graphman info QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6
437 ./graphman info QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6
439 ./graphman info QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6
indexer@indexer-shell-0 [~] $
I understand this was a mistake on my part. However, I will collect the eth_call_cache from around the time period requested. The call cache will be up until 14 July 2022.
According to my shell’s history file, the POI was generated for block get_manual_poi.sh QmZDfRpakj4BTNaRUDiCAknzE5rbXQcQm2hRYzX8kNSpx6 14762100, one block prior to the reported failure.
I did attempt to close this allocation with a 0x0 POI, but was unable to due to an error message in the contract which I did not understand. (See linked transactions below.) I now understand that the error message indicated that I was to wait more than some number of epochs.
Our indexer is using Erigon Alpha v2022.06.06, graph-node version is v0.26.0, indexer-agent 0.19.3, and indexer-service 0.19.3 with the auto-agora patch.
Some background: this subgraph came up as a result of our testing the allocation optimiser tool which is one on the core developers’ task list. We’re working through a number of different use cases and patterns to try and reflect the wide variety of practices amongst the indexers.
Just to clarify here: You ended up closing out the allocation using 14762100 as your reference block for the manual POI calculation? Also, would you mind sharing your get_manual_poi.sh to better help understand how it’s being calculated on your end?
Sure thing, I’ve off-chained Europa to review for this RFP, and it failed right on the start block. I’m not certain how our nodes made it all the way to 14762100. I suspect that our nodes somehow skipped over the current failing block of 14762089, possibly because of an indexer node crash. I recall us having a number of problems with the index nodes crashing because of some subgraphs failing non-deterministically at the time. I’ll upload the schema as it currently sits on our DB now.
As far as the POI calculator goes, what it really does is queries the administrative endpoint for a POI then reports back with that. Sadly, I’m not familiar enough yet with the graph node code base to know how that GQL query is answered under the hood. See the code below:
#!/bin/sh
# Find all subgraph health:
#query {
# indexingStatuses {
# subgraph
# synced
# health
# }
#}
INDEXER_ADDRESS='0xd75C4Dbcb215A6cf9097cfbCc70aAB2596B96a9C'
SUBGRAPH_DEPLOYMENT=$1
BLOCK_NUMBER=$2
if [ -z "$INDEXER_ADDRESS" ]; then
echo 'Environment variable INDEXER_ADDRESS must be set to a valid indexer address'
exit 1
fi
if [ -z "$SUBGRAPH_DEPLOYMENT" ] || [ -z "$BLOCK_NUMBER" ]; then
echo "Usage: $0 [subgraph deployment IPFS hash] [last good block number]"
exit 1
fi
BLOCK_HEX=$(echo "obase=16; $BLOCK_NUMBER" | bc | xargs printf "0x%s\n")
BLOCK_HASH=$(curl \
-s -X POST -H 'Content-Type: application/json' \
--data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["'$BLOCK_HEX'", false],"id":1}' \
$INDEXER_EVM_RPC_URL | \
jq -r '.result.hash')
if [ -z "$BLOCK_HASH" ]; then
echo "Block $BLOCK_NUMBER not found"
exit 2
fi
gql_query='query { proofOfIndexing(subgraph: \"'$SUBGRAPH_DEPLOYMENT'\", blockHash: \"'$BLOCK_HASH'\", blockNumber: '$BLOCK_NUMBER', indexer: \"'$INDEXER_ADDRESS'\") }'
json="{\"query\":\"$gql_query\"}"
POI=$(curl \
-s -L -X POST -H 'Content-Type: application/json' \
-d "$json" http://query-node:8030/graphql | \
jq -r '.data.proofOfIndexing')
echo $POI
I saw this happening to one of my subgraphs too. If you search in discord you’ll see me posting about it. I’m on mobile right now, so I don’t know which one it is, but it’s a problem for sure.
Thanks for the heads up @indexer_payne ! It seemed similar to Bug 3236 reported by @alexis some time back. To better understand the behaviour, I simulated the same conditions by restarting the index node offchaining Europa, but that restart does not seem to be the jump-to-next-block trigger point. I’ve seen this happen on a couple of other subgraphs in the past, however they behaved similar to 3236, not this. Any hypotheses are welcome.
This dispute was made for the same reasons as GDR-13 by the same Fisherman where the reasons for submitting the dispute are the same. Like GDR-13, this indexer also wasn’t using a pre-release version of graph-node which allows Europa to be fully indexed, but it appears after feedback with Kai from Semiotic that they were operating in good faith and going off a POI script they use to close out failed subgraphs. However, there also seems to be some confusion on Semiotic’s behalf (like Protofire’s in GDR-13) regarding which block hash and number to use when calculating a failed subgraph’s POI.
Based on the findings, it appears that both parties were operating in good faith, but clarity will need to be made with regards to which block to use in POI calculations so that we can establish rock-solid guidelines for failed subgraph POIs.
Due to this, the Arbiters settled this dispute as a draw:
