It’s been brought to our attention that the EBO subgraph is producing incorrect block data for Boba network (chain id eip155:288). This is preventing indexers from closing allocations and collecting indexing rewards.
In light of this, the Arbitration Council suggests indexers to close allocations by manually forcing the POI to a valid POI corresponding to a previous epoch. If that were not available, then close the allocations using POI 0x00…01. Until further notice, any disputes related to these allocations will automatically be drawn.
Update: The EBO issue has been solved, it’s now reporting the correct block number for Boba chain. The previous special ruling for Boba network POIs will stop being in effect starting next epoch, 1082.
@tmigone could you share anything about what as the cause of the issue and what will be done to improve infrastructure going forward so as to avoid the issue in the future?
Hey Matthew, for sure. So this was a combination of a few different factors:
A few days before the issue, the RPC for Boba network that was being used had to be rotated.
The new RPC was accidentally set to Boba BNB network instead of Boba. This was obviously a mistake as those are two completely different blockchains.
The EBO has some built-in checks to ensure it does not post bad data to the subgraph, these are being continuously added as new failure modes are experienced. A recent one was making sure that an endpoint configured for chain X was in fact returning data for that specific chain, this is done by checking the chain id the endpoint reports. This validation was not tripped because the EBO instance was not running the latest version that includes this logic.
As for avoiding this problem in the future, we are making sure that the EBO is running latest version. This specific problem would have been averted if that was the case. There are also other ongoing initiatives to address EBO issues in general:
EBO is currently using dRPC which offers some degree of robustness against provider going down. We are looking into allowing the EBO to accept a pool of RPCs and implement some form of consensus or cross checking between reported results. This is likely not being prioritized until EOY.