GraphOps Update November 2023

chris
Fri, 03 Nov 2023 15:50:43 +0000
Core Team Updates

GraphOps is excited to share our progress this month! We spent a wonderful week together with other Core Developers and aligned across many initiatives, set Q4 goals and progressed key decisions and lines of work.

We recorded three new Subgraph Radio workshops this month, covering operating the Radio as an Indexer (video 1, video 2), and using Graphcast CLI to signal Subgraph Upgrades to Indexers (video 3) as a Subgraph Developer. We also improved Radio shutdown signal handling and we released version 0.1.10 of Subgraph Radio. For Graphcast SDK, we have released version 0.5.2, with a number of performance improvements and bug fixes. We have also updated the ping pong example, and made Radio peer data available via GraphQL.

For Launchpad, we ran another session of Launchpad Office Hours and released new step-by-step guides for provisioning and upgrading a Kubernetes cluster using kubeadm. Additionally, OpenEBS Rawfile compatibility with newer Kubernetes versions was fixed and dependencies across many components have been updated.

For the Indexer stack, graphops/indexer-service-rs has been migrated to graphprotocol/indexer-rs. Indexer service component standardisation, testing and payment flow integration are progressing well.

Lastly, within Network Operations, Subgraph Network Substream Subgraph and SubgraphVersion entities have reached feature parity with the Graph Network Subgraph. We have also begun working on a new File Data Service under The World of Data Services.

Happy November everyone

Looking Back (what was delivered)

Subgraph Radio (Documentation, Docker Image, Radio Crate, Radio Source)

3 workshops recorded covering Subgraph Radio, aimed at Indexers and Subgraph Developers

Subgraph Radio Workshop for Indexers

Upgrade Presync demo for Indexers

Upgrade Presync demo for Subgraph Developers

Further improved shutdown signalling within Radios, every spawned task now shuts down gracefully

Added network peer logs

GraphQL endpoint now exposes peerData and localPeerData

Made a start on developing a self-hosted frontend within Subgraph Radio, with all required assets bundled in the Radio binary/docker image

Versions 0.1.8, 0.1.9 and 0.1.10 of Subgraph Radio have been released

Introduced fresh styles and new server frontend functionality

Implemented default addition of Waku discovery network enr to bootstrapping nodes

Now have access to divergence summary notifications at set intervals.

Subgraph Radio Development Activity (click for more details)
Graphcast SDK (Documentation, SDK Crate, SDK Source)

Default waku protocol switched from filter to relay on all Radios

Research on how to revise our Waku node configuration to improve connectivity, stability and scalability

Collaborated with the Waku team and validated our discv5 and peer management strategy, and message flow when using filter protocol

Versions 0.5.0, 0.5.1, and 0.5.2 of the SDK have been released

Refactored decode function and updated example

Introduced new GraphQL type ‘peerData’ and refactored peer helpers

Various performance improvements and bug fixes

Graphcast SDK Development Activity (click for more details)
Launchpad (Documentation, Starter, Charts, Namespaces)

Fixed OpenEBS compatiblity with newer Kubernetes versions

Updated Graph Node dashboards

Added new documentation with step-by-step guides to provision and upgrade a Kubernetes cluster

Ran another Launchpad Office Hours session (recording) with the Indexer community, demoing how to create a Kubernetes cluster with kubeadm

Launchpad Charts: Enhanced handling for Arbitrum versioning, including in the CI, and added default values in the Generic App

Added support for topologyAwareRouting in the Indexer service/agent

Updated dependencies across various components including Subgraphradio, Proxyd, and OpenEBS

Launchpad Namespaces: Adjustments made to CI configuration, including improved regex handling, and allowed primary shard connections from any host

Various components, such as Gnosis, Celo, Polygon, Monitoring, Ethereum, Arbitrum, Graph, Avalanche, Ingress and sealed-secrets dependencies have been updated in the CI

Launchpad Development Activity (click for more details)
Indexer Components (Indexer Components, Indexer Service Rewrite, Allocation Optimizer)

Collaborative efforts with E&N and Semiotic on standardising indexer-service and integrating payment flow

indexer-service-rs has been tested in local network environments (by E&N)

Research and early discussions on File Data Service, specified general project goals and design approaches
No PR activity reported.

Indexer Components Development Activity (click for more details)
Network Operations (Graph Network Subgraph, Graph Network Substreams)

Fixed issues with the File Data Sources refactors for all related subgraphs. Subgraphs are ready to be deployed as soon as UI changes are ready.

Subscriptions subgraph has been updated to allow for trusted partners through the gateway registry.

Graph Network Substream Subgraph and SubgraphVersion entities feature parity is reached

Introduced feature parity for subgraph and subgraph version entities in Graph Network Substreams.

Network Operations Development Activity (click for more details)
Ecosystem Relations

Recordings and agendas for Indexer Office Hours are available. Follow updates on Twitter @TheGraphIOH.

Looking Ahead (upcoming priorities)

Subgraph Radio

Prepare for 1.0.0 stable release

Stricter validation and better error messages relating to configuration variables

Improve periodic report log/summary + add state persistence for that

Move waku discovery ENR to SDK and use it if discv5 is enabled

Switch to using DB instead of a JSON file for state persistence

Launchpad

Adding Firehose support is our top priority

Indexer Components

Finish standardisation and payments for indexer-service

Setup a library for indexer-service and move current indexer-service to subgraph-indexer-service

Network Operations

Develop a proof of concept for File Data Service

Ecosystem Relations

Please join the community for Indexer Office Hours on Tuesday every week at 6pm UTC in the Graph Protocol Discord

Please join the community for Launchpad Office Hours on Wednesday every second week at 5pm UTC in the Graph Protocol Discord