Fantom-Chain outage on The Graph’s hosted service

The Graph’s hosted service recently experienced an outage on the Fantom network. We first identified the issue on September 14th and we immediately started to understand and triage the problem. Our engineering team was able to re-establish Fantom hosted service connectivity on September 16th.

The Graph is an essential infrastructure layer for the web3 stack and we are consistently working to increase redundancy of the technology to ensure things like this do not happen. For that exact reason, we had added a few new RPCs on The Graph to Fantom’s network; unfortunately, one of the RPCs could not handle the high throughput requests by The Graph’s hosted service, which is a core part of what caused this issue.

We are working on a stress-testing framework that - going forward - will thoroughly test RPCs before they are put into production. This should help mitigate any issues that might occur due to RPC failures on the hosted service in the future.

We sincerely apologize to all the teams that were affected and we truly appreciate the continued support of our community while our amazing engineers work to resolve issues like this in real-time. Thanks so much for your patience as we continue to help build a better future for the world. For now, back to work!