GraphOps Update November 2023

Research & Development Core Team Updates
1

:astronaut: Executive Summary

GraphOps is excited to share our progress this month! We spent a wonderful week together with other Core Developers and aligned across many initiatives, set Q4 goals and progressed key decisions and lines of work.

We recorded three new Subgraph Radio workshops this month, covering operating the Radio as an Indexer (video 1, video 2), and using Graphcast CLI to signal Subgraph Upgrades to Indexers (video 3) as a Subgraph Developer. We also improved Radio shutdown signal handling and we released version 0.1.10 of Subgraph Radio. For Graphcast SDK, we have released version 0.5.2, with a number of performance improvements and bug fixes. We have also updated the ping pong example, and made Radio peer data available via GraphQL.

For Launchpad, we ran another session of Launchpad Office Hours and released new step-by-step guides for provisioning and upgrading a Kubernetes cluster using kubeadm. Additionally, OpenEBS Rawfile compatibility with newer Kubernetes versions was fixed and dependencies across many components have been updated.

For the Indexer stack, graphops/indexer-service-rs has been migrated to graphprotocol/indexer-rs. Indexer service component standardisation, testing and payment flow integration are progressing well.

Lastly, within Network Operations, Subgraph Network Substream Subgraph and SubgraphVersion entities have reached feature parity with the Graph Network Subgraph. We have also begun working on a new File Data Service under The World of Data Services.

Happy November everyone :sparkles:

:tada: Looking Back (what was delivered)

Subgraph Radio (Documentation, Docker Image, Radio Crate, Radio Source)

  • 3 workshops recorded covering Subgraph Radio, aimed at Indexers and Subgraph Developers
  • Further improved shutdown signalling within Radios, every spawned task now shuts down gracefully
  • Added network peer logs
  • GraphQL endpoint now exposes peerData and localPeerData
  • Made a start on developing a self-hosted frontend within Subgraph Radio, with all required assets bundled in the Radio binary/docker image
  • Versions 0.1.8, 0.1.9 and 0.1.10 of Subgraph Radio have been released
  • Introduced fresh styles and new server frontend functionality
  • Implemented default addition of Waku discovery network enr to bootstrapping nodes
  • Now have access to divergence summary notifications at set intervals.
Subgraph Radio Development Activity
  • Subgraph Radio pull requests merged:
    • PR #109, PR #96: Updated a few config names and shutdown handlers for better control flow struct
    • PR #107, PR #87: Updated test packages waku dep and grafana.json
    • PR #97, PR #108: Released versions 0.1.8 and 0.1.9
    • PR #104, PR #103: Introduced new styles and server frontend functionality
    • PR #98: Updated changelog
    • PR #88: Waku discovery network enr now added to bootstrapping nodes by default
    • PR #85: Divergence summary notification now available at a given interval

Graphcast SDK (Documentation, SDK Crate, SDK Source)

  • Default waku protocol switched from filter to relay on all Radios
  • Research on how to revise our Waku node configuration to improve connectivity, stability and scalability
  • Collaborated with the Waku team and validated our discv5 and peer management strategy, and message flow when using filter protocol
  • Versions 0.5.0, 0.5.1, and 0.5.2 of the SDK have been released
  • Refactored decode function and updated example
  • Introduced new GraphQL type ‘peerData’ and refactored peer helpers
  • Various performance improvements and bug fixes
Graphcast SDK Development Activity
  • Graphcast SDK pull requests merged:
    • PR #293, PR #302, PR #308: Released versions 0.5.0, 0.5.1, and 0.5.2
    • PR #303: Refactored decode function and updated example
    • PR #297, PR #307: Added content topic check at message handle, Removed filter subscription from content topic update
    • PR #296: Fixed empty set_waku_event_callback at shutdown and refactored mpsc sender
    • PR #299: Introduced peerData GraphQL type and refactored peer helpers

Launchpad (Documentation, Starter, Charts, Namespaces)

  • Fixed OpenEBS compatiblity with newer Kubernetes versions
  • Updated Graph Node dashboards
  • Added new documentation with step-by-step guides to provision and upgrade a Kubernetes cluster
  • Ran another Launchpad Office Hours session (recording) with the Indexer community, demoing how to create a Kubernetes cluster with kubeadm
  • Launchpad Charts: Enhanced handling for Arbitrum versioning, including in the CI, and added default values in the Generic App
  • Added support for topologyAwareRouting in the Indexer service/agent
  • Updated dependencies across various components including Subgraphradio, Proxyd, and OpenEBS
  • Launchpad Namespaces: Adjustments made to CI configuration, including improved regex handling, and allowed primary shard connections from any host
  • Various components, such as Gnosis, Celo, Polygon, Monitoring, Ethereum, Arbitrum, Graph, Avalanche, Ingress and sealed-secrets dependencies have been updated in the CI
Launchpad Development Activity
  • Launchpad Charts pull requests merged:
    • PR #169, PR #182, PR #180: Improved handling of Arbitrum versioning, within the CI, and added default values in the Generic App.
    • PR #173: Added support for topologyAwareRouting for the indexer service/agent.
    • PR #171, PR #172: Updated the Subgraphradio and Proxyd dependencies.
    • PR #175: Updated OpenEBS images to address csistorage API issues.
    • PR #176: New sections added to the Proxyd config interface.
    • PR #177: Removed P2P from Arbitrum charts.
    • PR #170: Updated Graph-Node dashboard.
  • Launchpad Namespaces pull requests merged:

Indexer Components (Indexer Components, Indexer Service Rewrite, Allocation Optimizer)

  • Collaborative efforts with E&N and Semiotic on standardising indexer-service and integrating payment flow
  • indexer-service-rs has been tested in local network environments (by E&N)
  • Research and early discussions on File Data Service, specified general project goals and design approaches
    No PR activity reported.
Indexer Components Development Activity

Network Operations (Graph Network Subgraph, Graph Network Substreams)

  • Fixed issues with the File Data Sources refactors for all related subgraphs. Subgraphs are ready to be deployed as soon as UI changes are ready.
  • Subscriptions subgraph has been updated to allow for trusted partners through the gateway registry.
  • Graph Network Substream Subgraph and SubgraphVersion entities feature parity is reached
  • Introduced feature parity for subgraph and subgraph version entities in Graph Network Substreams.
Network Operations Development Activity

Ecosystem Relations

:rocket: Looking Ahead (upcoming priorities)

  • Subgraph Radio
    • Prepare for 1.0.0 stable release
      • Stricter validation and better error messages relating to configuration variables
      • Improve periodic report log/summary + add state persistence for that
      • Move waku discovery ENR to SDK and use it if discv5 is enabled
      • Switch to using DB instead of a JSON file for state persistence
  • Launchpad
    • Adding Firehose support is our top priority
  • Indexer Components
    • Finish standardisation and payments for indexer-service
    • Setup a library for indexer-service and move current indexer-service to subgraph-indexer-service
  • Network Operations
    • Develop a proof of concept for File Data Service
  • Ecosystem Relations
    • Please join the community for Indexer Office Hours on Tuesday every week at 6pm UTC in the Graph Protocol Discord
    • Please join the community for Launchpad Office Hours on Wednesday every second week at 5pm UTC in the Graph Protocol Discord
8 Likes