The qos oracle has been down for a month and nobody could tell

[AxiomaticAardvark]

posting this because the current stall is the smaller half of the story, and the bigger half is something none of us could see from the outside until we went looking.

everything below is verifiable from the chain and from public ipfs. no insider access, no special permissions. anyone can reproduce it.

what is actually broken

there are two separate failures and they got confused with each other.

the one everyone noticed: the publisher stops posting. it did that on 2026-07-29 for about 38 hours, and it is doing it again right now, 37 hours and counting since the last post at 2026-08-04 00:00 UTC. that is bad, but at least it is visible if you know to look at the DataEdge on gnosis.

the one nobody noticed: since 2026-07-01 the oracle’s subgraph has rejected every single message the publisher sends. it is synced to chain tip, it reports hasIndexingErrors: false, and it discards every post with:

valid: false
errorMessage: "0x8cbbe43f97f80efa6ba0a95f3d544e03f84db0ce is not a valid submitter."

that address is the publisher. the same one that has been posting all along.

so the newest OracleMessage in the subgraph is from 2026-08-04, and the newest MessageDataPoint it actually produced is from 2026-07-01 03:20. over a month where the posts kept arriving and none of them became data.

if you have been querying that subgraph at any point in the last month, you have been getting july 1st numbers. it does not error, it does not warn, it just answers with old data. a stale subgraph answers exactly like a fresh one.

the data was never missing

this is the part that changes what we should ask for.

the CIDs are all on gnosis in the DataEdge calldata, permanently. the payloads are pinned and fetchable from public ipfs right now. i pulled one from inside the gap, 2026-07-04, and got a 1.86MB json array with 2741 per-allocation records in it:

• indexer_wallet
• subgraph_deployment_ipfs_hash
• query_count
• proportion_indexer_200_responses
• avg_indexer_latency_ms
• avg_indexer_blocks_behind
• stdev_indexer_latency_ms
• total_query_fees

every number the subgraph would have materialised was sitting there the whole time. it just never got decoded.

so the problem is not access to data. the problem is that there is exactly one decoder, one team runs it, and when it silently stops accepting messages the entire network’s view of indexer quality stops with it and nothing says so.

what i am not asking for

i am not asking e&n to publish the raw gateway logs. those contain api keys, user ids and per-customer traffic. that is genuinely their business and it genuinely should not be public. it is also unnecessary, because the aggregated payloads are already public.

what would actually fix it

four things, none of them expensive, none of them giving up anything commercially sensitive.

1. publish and bless the decoder. there is a public reference subgraph, juanmardefago/gateway-qos-oracle-example-subgraph, but it is named “example” and i cannot tell whether it matches what is deployed at Dtr9rETvwokot4BSXaD5tECanXfqfJKcvHuaaEgPDD2D. confirm which source produces the live one and keep it current. then anyone can deploy their own and the network stops depending on a single deployment run by a single team.

2. make the submitter set auditable. whatever governs “is this a valid submitter” should be inspectable, ideally on chain. if it had been, this would have been a ten minute diagnosis on july 1st instead of a month of silence. right now a change to that list is indistinguishable from the oracle simply going quiet.

3. document the payload format. the schema of what lands on ipfs. i reverse engineered it from bytes this week and it was not hard, but nobody should have to, and a documented format means third parties can decode independently without guessing.

4. treat gateway_id as the multi-publisher field it already is. every entity in that schema carries gateway_id. the format was clearly designed for more than one gateway to publish. if that is supported, say so and document how, and qos stops being a single-operator service by construction.

this is not hypothetical, it already works

lodestar decodes the same chain and the same ipfs files independently and publishes what it finds at Lodestar | The Graph Protocol Analytics : the canonical numbers mirrored, plus our own probe measurements labelled separately so nobody confuses the two.

we also hold 2479 payloads captured straight from the CIDs, 2.6GB, for the window their subgraph threw away.

i am not saying this to advertise. i am saying it because it means the ask above is small. a second implementation already exists and produces the same numbers, which is the evidence that the decoder does not need to be a single point of failure.

what the community can do meanwhile

watch three things, not one, because watching one is how this hid for a month:

1. is the publisher posting. read the DataEdge on gnosis directly, not through the subgraph.

2. is the decoder accepting. check OracleMessage.valid on the newest message. if it is false, no new data is being produced no matter how healthy everything else looks.

3. how old is the data you are actually reading. the newest dayNumber, compared against now. this is the one that catches everything, and it is the one nobody had.

happy to share the queries for all three, they are a few lines each.

the short version

the qos oracle’s data has been public and fetchable this entire time. what failed is the only sanctioned way to read it, and it failed silently for over a month. making the decoder something the community can run, and making the submitter list auditable, costs almost nothing and means this cannot happen again quietly.

thanks to Yash (e&n) for chasing this internally, genuinely appreciated.

[tmigone]

Thanks Petko for the report! I think I have identified what the indexing problem is, just deployed a new subgraph version to studio which is syncing (unfortunately its not fast to sync). I’ll keep this thread updated about it.

[AxiomaticAardvark]

update, and a correction to my own proposal.

@juanmardefago got in touch and filled in a lot of history i did not have. posting it because it changes what i think should happen, and because most of it is not written down anywhere public.

on v1. it would work again if the subgraph were redeployed with a new whitelisted submitter. but that means a resync, and a resync means dropping old data, because fetching a year of payloads back from ipfs is too slow to be practical. so the fix is real but it is not free, and it costs history.

on v2. it is feature complete. it was never pushed to main because it needed testing against a production gateway, and because the design has each gateway operator running it themselves - and maintaining a clickhouse db with a year of qos data is not something e&n wanted to take on. so it stalled on operational appetite, not on code.

worth saying plainly: v2 is not an oracle. it is a graphql endpoint that a gateway operator hosts and serves their own data from. that is a better shape than what we have now and i did not know it already existed.

on my own proposal, which i now think is wrong in part. i specced publishing qos summaries as events on arbitrum, with a self hosted indexer over them, so that no single decoder could take the whole thing down. juan has already been down that road: this data does not scale well onchain, ipfs/arweave were too slow and unreliable especially for resyncs, and - the part i had not thought about properly - verifiability was never actually a requirement here. the data is inherently trusted. whatever the gateway reports is what exists. there is no second source to check it against, so cryptographic verification of it does not buy anyone anything. so i am dropping the onchain publication layer from what i proposed. it was solving a problem that does not exist, and the person who tried it first already found that out.

what i still think is worth building, and what it is not:

we are going ahead with direct probing of indexer endpoints, paid with tap receipts so we can choose which indexer answers rather than having a gateway choose for us. that gives latency, success rate, chainhead lag, and one thing no gateway telemetry can produce: correctness. we canonicalise responses with JCS, hash and cluster them, so an indexer that is fast and returns 200s and serves the wrong data shows up as wrong. v1 and v2 both count status codes, which cannot see that by construction.

what it does not give, and i want to be clear because i was sloppy about this earlier in the
thread: it does not give demand. query_count, served share, “why am i not getting traffic”. those are facts about which indexers a gateway chose to route to, and you cannot derive them without having served the queries. no amount of probing produces them. that half stays with whoever runs a gateway with real users.

juan also raised two fair criticisms i want to acknowledge rather than argue with. what we measure is not what the gateway measures, so our numbers cannot tell you whether your qos is affecting how the gateway routes to you. and probing indexer endpoints at volume could get probers blocked, which is a real risk i had not considered and will think about before scaling cadence up.

a concrete offer. v2 needs testing against a production gateway with a redpanda/kafka instance. lodestar runs its own gateway, a fork of edgeandnode/gateway, and our compose stack already ships redpanda, and the gateway is already writing to gateway_queries and gateway_attestations. so we can hook v2 up and actually test it. happy to run the clickhouse side too - we are not e&n and we do not mind maintaining it. that would unblock something that has been sitting finished for two years, and it would prove the multi publisher model works with someone other than e&n operating it.

what i would still ask for, unchanged from the original post and cheap either way: confirm which source produces the deployed subgraph, make the submitter set inspectable, and document the payload format. the monitoring point stands too - watch publisher liveness, decoder acceptance, and data age as three separate things, because watching only one is how a month went by.