Technical information
1. Version of graph-node used:
graphprotocol/graph-node:v0.22.0
2. Graph-node configuration in full:
index_node_1:
image: graphprotocol/graph-node:v0.22.0
command: graph-node --metrics-port=8040
restart: always
environment:
BLOCK_INGESTOR: 'indexer1'
ETHEREUM_RPC: 'mainnet:'
GRAPH_ETH_CALL_BY_NUMBER: 'true'
GRAPH_KILL_IF_UNRESPONSIVE: 'true'
GRAPH_NODE_ID: 'index_node_1'
IPFS: 'https://ipfs.network.thegraph.com'
NODE_ROLE: 'index-node'
POSTGRES_URL:''
GRAPH_ALLOW_NON_DETERMINISTIC_FULLTEXT_SEARCH: "true"
3. Type and version of Ethereum node:
openethreum 3.2.3 Parity (we used QuickNode provider that time)
4. Table of PoIs generated by the affected subgraphs:
5. A dump of the call cache data from the database for the affected subgraphs:
6. Entity values as of the divergent block once identified:
graph=# select * from eth_call_cache where contract_address = decode('D17A9bc90c582249e211a4f4b16721e7f65156c8','hex') and block_number = 11273929;
graph=# \x on;
Expanded display is on.
graph=# select * from eth_call_cache where contract_address = decode('D17A9bc90c582249e211a4f4b16721e7f65156c8','hex') and block_number = 11273929;
-[ RECORD 1 ]----+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id | \x35ad8633ee1a4fab086e471df15b570b6c43a8b09a24da7178793654e00a4b6a
return_value | \x000000000000000000000000000000000000000000002a5a058fc295ed000000000000000000000000000000000000000000000000000000000000000000006400000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000002a5a058fc295ed000000000000000000000000000000000000000000000000084595161401484a000000
contract_address | \xd17a9bc90c582249e211a4f4b16721e7f65156c8
block_number | 11273929
Select * for problem block:
We made this query:
graph=# \dt sgd7.*
List of relations
Schema | Name | Type | Owner
--------±------------------------------------±------±------
sgd7 | audius_network | table | root
sgd7 | claim_event | table | root
sgd7 | claim_processed_event | table | root
sgd7 | claim_round | table | root
sgd7 | decrease_stake_event | table | root
sgd7 | delegate | table | root
sgd7 | deregistered_service_provider_event | table | root
sgd7 | guardian_transaction_executed_event | table | root
sgd7 | increase_delegated_stake_event | table | root
sgd7 | increased_stake_event | table | root
sgd7 | poi2$ | table | root
sgd7 | proposal | table | root
sgd7 | proposal_outcome_evaluated_event | table | root
sgd7 | proposal_submitted_event | table | root
sgd7 | proposal_transaction_executed_event | table | root
sgd7 | proposal_vetoed_event | table | root
sgd7 | proposal_vote_submitted_event | table | root
sgd7 | proposal_vote_updated_event | table | root
sgd7 | registered_service_provider_event | table | root
sgd7 | remove_delegator_event | table | root
sgd7 | service_node | table | root
sgd7 | service_type | table | root
sgd7 | service_type_version | table | root
sgd7 | slash_event | table | root
sgd7 | undelegate_stake_event | table | root
sgd7 | update_deployer_cut_event | table | root
sgd7 | user | table | root
sgd7 | vote | table | root
for i in audius_network claim_event claim_processed_event claim_round decrease_stake_event delegate deregistered_service_provider_event guardian_transaction_executed_event increase_delegated_stake_event increased_stake_event proposal proposal_outcome_evaluated_event proposal_submitted_event proposal_transaction_executed_event proposal_vetoed_event proposal_vote_submitted_event proposal_vote_updated_event registered_service_provider_event remove_delegator_event service_node service_type service_type_version slash_event undelegate_stake_event update_deployer_cut_event user vote; do psql -c “select * from sgd7.$i where lower(block_range) = 11273929;” graph >> all_tables_events; done;
And got results:
id | audius_token_address | claims_manager_address | delegate_manager_address | governance_address | registry | service_provider_factory_address | service_type_manager_address | staking_address | registry_address | total_supply | total_audio_minted | total_audio_burned | total_tokens_staked | total_tokens_claimable | total_tokens_locked | total_tokens_delegated | max_delegators | min_delegation_amount | undelegate_lockup_duration | remove_delegator_lockup_duration | remove_delegator_eval_duration | decrease_stake_lockup_duration | update_deployer_cut_lockup_duration | funding_round_block_diff | funding_amount | recurring_community_funding_amount | community_pool_address | voting_quorum_percent | voting_period | execution_delay | max_in_progress_proposals | guardian_address | request_count | total_staked | vid | block_range
----±---------------------±-----------------------±-------------------------±-------------------±---------±---------------------------------±-----------------------------±----------------±-----------------±-------------±-------------------±-------------------±--------------------±-----------------------±--------------------±-----------------------±---------------±----------------------±---------------------------±---------------------------------±-------------------------------±-------------------------------±------------------------------------±-------------------------±---------------±-----------------------------------±-----------------------±----------------------±--------------±----------------±--------------------------±-----------------±--------------±-------------±----±------------
(0 rows)
id | claimer | rewards | new_total | block_number | vid | block_range
----±--------±--------±----------±-------------±----±------------
(0 rows)
id | rewards | claimer | old_total | new_total | block_number | vid | block_range
----±--------±--------±----------±----------±-------------±----±------------
(0 rows)
id | fund_amount | block_number | vid | block_range
----±------------±-------------±----±------------
(0 rows)
id | status | owner | expiry_block | created_block_number | ended_block_number | decrease_amount | new_stake_amount | vid | block_range
----±-------±------±-------------±---------------------±-------------------±----------------±-----------------±----±------------
(0 rows)
id | claimable_amount | amount | from_user | to_user | vid | block_range
----±-----------------±-------±----------±--------±----±------------
(0 rows)
id | type | sp_id | node | owner | endpoint | unstake_amount | block_number | vid | block_range
-----------------±-------------±------±----------------±-------------------------------------------±-------------------------------±-------------------------±-------------±----±------------
creator-node::4 | creator-node | 4 | creator-node::4 | 0xccd0855b67f89ed9058e3bf2926d5380182bbbfc | https://content-node.audius.co | 200000000000000000000000 | 11273929 | 4 | [11273929,)
(1 row)
id | target_contract_address | call_value | function_signature | call_data | return_data | block_number | vid | block_range
----±------------------------±-----------±-------------------±----------±------------±-------------±----±------------
(0 rows)
id | delegator | service_provider | increase_amount | block_number | vid | block_range
----±----------±-----------------±----------------±-------------±----±------------
(0 rows)
id | owner | new_stake_amount | increase_amount | block_number | vid | block_range
----±------±-----------------±----------------±-------------±----±------------
(0 rows)
id | name | description | proposer | submission_block_number | target_contract_registry_key | target_contract_address | call_value | function_signature | call_data | outcome | vote_magnitude_yes | vote_magnitude_no | num_votes | vid | block_range
----±-----±------------±---------±------------------------±-----------------------------±------------------------±-----------±-------------------±----------±--------±-------------------±------------------±----------±----±------------
(0 rows)
id | proposal | outcome | vote_magnitude_yes | vote_magnitude_no | num_votes | block_number | vid | block_range
----±---------±--------±-------------------±------------------±----------±-------------±----±------------
(0 rows)
id | proposal | proposer | name | description | vid | block_range
----±---------±---------±-----±------------±----±------------
(0 rows)
id | proposal | success | return_data | block_number | vid | block_range
----±---------±--------±------------±-------------±----±------------
(0 rows)
id | proposal | block_number | vid | block_range
----±---------±-------------±----±------------
(0 rows)
id | proposal | voter | vote | current_vote | voter_stake | block_number | vid | block_range
----±---------±------±-----±-------------±------------±-------------±----±------------
(0 rows)
id | proposal | voter | vote | voter_stake | current_vote | previous_vote | block_number | vid | block_range
----±---------±------±-----±------------±-------------±--------------±-------------±----±------------
(0 rows)
id | type | sp_id | node | owner | endpoint | stake_amount | block_number | vid | block_range
----±-----±------±-----±------±---------±-------------±-------------±----±------------
(0 rows)
id | status | owner | expiry_block | created_block_number | ended_block_number | delegator | vid | block_range
----±-------±------±-------------±---------------------±-------------------±----------±----±------------
(0 rows)
id | sp_id | owner | type | endpoint | delegate_owner_wallet | created_at | is_registered | vid | block_range
-----------------±------±-------------------------------------------±-------------±-------------------------------±-------------------------------------------±-----------±--------------±----±------------
creator-node::4 | 4 | 0xccd0855b67f89ed9058e3bf2926d5380182bbbfc | creator-node | https://content-node.audius.co | \xc892c75fa17e8b641a4843d0aa620792857d217a | 1604105379 | f | 15 | [11273929,)
(1 row)
id | is_valid | min_stake | max_stake | vid | block_range
----±---------±----------±----------±----±------------
(0 rows)
id | service_type | service_version | block_number | vid | block_range
----±-------------±----------------±-------------±----±------------
(0 rows)
id | target | amount | new_total | block_number | vid | block_range
----±-------±-------±----------±-------------±----±------------
(0 rows)
id | status | owner | expiry_block | created_block_number | ended_block_number | service_provider | amount | vid | block_range
----±-------±------±-------------±---------------------±-------------------±-----------------±-------±----±------------
(0 rows)
id | status | owner | expiry_block | created_block_number | ended_block_number | updated_cut | vid | block_range
----±-------±------±-------------±---------------------±-------------------±------------±----±------------
(0 rows)
id | balance | total_claimable_amount | claimable_stake_amount | claimable_delegation_received_amount | claimable_delegation_sent_amount | stake_amount | delegation_received_amount | delegation_sent_amount | has_stake_or_delegation | valid_bounds | deployer_cut | min_account_stake | max_account_stake | pending_decrease_stake | pending_remove_delegator | pending_update_deployer_cut | pending_undelegate_stake | created_at | vid | block_range
--------------------------------------------±-------------------------±-------------------------±-------------------------±-------------------------------------±---------------------------------±-------------------------±---------------------------±-----------------------±------------------------±-------------±-------------±-------------------------±---------------------------±-----------------------±-------------------------±----------------------------±-------------------------±-----------±------±--------------------
0xccd0855b67f89ed9058e3bf2926d5380182bbbfc | 200000000000000000000000 | 200000000000000000000000 | 200000000000000000000000 | 0 | 0 | 200000000000000000000000 | 0 | 0 | t | t | 100 | 200000000000000000000000 | 10000000000000000000000000 | | | | | 1604017727 | 40665 | [11273929,11273945)
(1 row)
id | proposal | vote | magnitude | voter | created_block_number | updated_block_number | vid | block_range
----±---------±-----±----------±------±---------------------±---------------------±----±------------
(0 rows)
Summary of the situation
We found that we have 7 disputes regarding Indexing QmTj6fHgHjuKKm43YL3Sm2hMvMci4AkFzx22Mdo9W3dyn8 (Audius).
As you may know, we are one of few Indexers who close allocation every day, that’s why we got 7 disputes, 1 by each epoch with this new subgraph.
Our investigation showed us that it’s our backup ETH data provider is at fault.
Before all these disputes we had a problem with our Archive node with tracing and activated backup QuickNode node subscription while we resync our Archive node.
There were no apparent problems with Indexing to this subgraph and we thought that everything was working fine. Unfortunately, we didn’t compare our POI with other Indexers, nor we had dispute alerting yet. So, we tried to solve a problem with another subgraph (Livepeer) where were errors, we wrote about it in Discord. We tried almost everything for Livepeer on our side and when we found these disputes about Audius we decided to switch QuickNode as the last option to another provider - Alchemy because sync for our Node with tracing was still in progress. That helped, errors for Livepeer have gone, and POI’s for Audius also fixed.
Now we prepared a script for comparing POI’s: GitHub - p2p-org/graphprotocol-poi-checker: Python script to check POI across multi indexers for defined subgraph
And check POI’s before everyday closing, everything still looks good, which confirms that problem was related to QuickNode.