Thanks for providing extensive data around the syncing of the QmTj6fHgHjuKKm43YL3Sm2hMvMci4AkFzx22Mdo9W3dyn8
subgraph @KonstantinRM! Your continued cooperation has been helpful in the arbitration investigation.
I’ve compared the proof of indexing (PoI), entity state, and Ethereum call cache data you shared with corresponding data from several reference indexers to identify the root cause of the PoI divergence. I’ll briefly summarize my findings here and link to data and analysis for anyone who wants to get into more nitty gritty details.
What happened?
-
PoI divergence began at block number
11273929
. -
Upon analyzing the events emitted in that block and the subgraph mappings for
QmTj6fHgHjuKKm43YL3Sm2hMvMci4AkFzx22Mdo9W3dyn8
I identified thehandleDeregisteredServiceProvider
mapping as the triggered mapping and theserviceProviderContract.getServiceProviderDetails(event.params._owner)
call as the only data fetched by that mapping (making it a likely cause of Entity divergence). -
I compared Ethereum call cache data and entity changes from the divergent block (#
11273929
) to confirm my hunch that the divergence was caused by a discrepancy inserviceProviderContract.getServiceProviderDetails(event.params._owner)
results. Both the entity state and Ethereum contract call cache values differed between the disputed indexer and reference indexers in such a way to suggest the Ethereum call differences were the primary driver of the entity differences. -
I believe the cause of divergence is the following difference in Ethereum contract call results:
Function: getServiceProviderDetails(_owner: address) Disputed indexer's results: deployerStake: 2.0E+23 deployerCut: 100 validBounds: True numberOfEndpoints: 1 minAccountStake: 2.0E+23 maxAccountStake: 1.0E+25 Reference indexers' results: deployerStake: 2.0E+23 deployerCut: 100 validBounds: True numberOfEndpoints: 0 minAccountStake: 0 maxAccountStake: 0
- I have not been able to reproduce the disputed indexer’s results of the Ethereum contract call in question.
Ongoing investigation
- Why did the Ethereum call results differ between the disputed indexer and the reference indexers? I haven’t been able to reproduce the disputed indexers contract call results to confirm the reason for the difference.
- While setting
GRAPH_ETH_CALL_BY_NUMBER
totrue
does introduce potential non-determinism in contract call results, the effect
is contained to blocks susceptible to a reorg (within a reasonable range of the chain head block). In this case the contract
call was made as of an old block, so this configuration value likely did not have any effect in this situation. - One theory is that the Ethereum node executed the contract as of a different block number (not #
11273929
) . To test this hypothesis I looped over a huge range of block numbers making the call for each in an unsuccessful attempt to reproduce the output results. - Could the Ethereum node have had issues that lead to an invalid contract storage state? I have tried the call in question against the Ethereum provider, but was unable to reproduce the results.
- Could the graph node have sent different Ethereum contract call requests? Call cache IDs are deterministically generated using
contract_address
,encoded_call_data,
andblock_hash
; since they match between the references and disputed we can reasonably assume that the Ethereum requests sent were the same. Logging from the affected graph node is not available to confirm the exact contract call request body.
- While setting