GraphOps Update September 2023

Research & Development Core Team Updates
1

:astronaut: Executive Summary

GraphOps is excited to share the latest updates in our development efforts. We have released version 0.1.5 of Subgraph Radio, which adds persistence, rate limiting and API support for upgrade intent messages. We also released 0.0.2 of Graphcast CLI, improving the UX of sending upgrade intent messages for Subgraph Developers. For Graphcast SDK, we have introduced helpers for fetching peer and message counts, fixed support for Polygon subgraph deployments, and released version 0.4.2. For Launchpad, we made a variety of releases to Launchpad Charts and Launchpad Namespaces, tracking dependency updates. We also recorded a Launchpad Quick Start workshop and ran sessions of Launchpad Office Hours with the community. Our collaboration with Semiotic on the Indexer Service rewrite is progressing, with a partial implementation of the paid query flow, as well as many other improvements. We’ve also extended entity field coverage for our Graph Network Substreams implementation. Finally, Indexer Office Hours continues to be a hotbed of Indexer discussion and activity, and we are grateful to Indexers for showing up each week and giving their views. Thank you!

Happy September everyone :sparkles:

:tada: Looking Back (what was delivered)

Subgraph Radio (Documentation, Docker Image, Radio Crate, Radio Source)

  • Added peer and messages received counts as Prometheus metrics to Subgraph Radio
  • Added persistence and GraphQL API endpoint for UpgradeIntent messages to Subgraph Radio
  • Added rate limits for subgraph upgrade presyncing, with the defaults allowing a given subgraph deployment to be upgraded once per day
  • Added new indexing-status command to Graphcast CLI, which allows a Subgraph Developer to easily query the sync progress of their new deployment after sending an UpgradeIntent message
  • Improved Grafana dashboard user experience by allowing user to configure datasource when importing
  • One-shot CLI has been renamed to Graphcast CLI for clarity and consistency
  • Versions 0.1.4 and 0.1.5 of Subgraph Radio have been released
  • Released version 0.0.2 of Graphcast CLI
Subgraph Radio Development Activity
  • Subgraph Radio pull requests merged:
    • PR #47: Removed unnecessary use of new and build for upgradeIntent
    • PR #49: Introduced peer count and received message metrics
    • PR #50: Improved Grafana dashboard user experience by removing active overrides and updating the dashboard design
    • PR #51: Moved message processing to use timeouts for better efficiency
    • PR #52: Implemented persistence and cleanup of upgrade intent messages, introduced ratelimiting, and improved the API
    • PR #54, PR #57: Released versions 0.1.4 and 0.1.5
    • PR #55: Added local attestation without matching with remote
  • Graphcast CLI pull requests merged:
    • PR #2: Updated Labels workflow
    • PR #5: Rescoped CLI for sending UpgradeIntentMessage
    • PR #6: Used the latest SDK with query fix
    • PR #9: Renamed one-shot-cli to graphcast-cli
    • PR #10: Added upgrade-presync subcommand
    • PR #11: Added status reqwest timeout and documentation updates
    • PR #12: Released version 0.0.2

Graphcast SDK (Documentation, SDK Crate, SDK Source)

  • Introduced helpers for fetching peer and message counts
  • Fixed support for Polygon subgraph deployments
  • Released version 0.4.2 of the SDK
  • Various code improvements and bug fixes
Graphcast SDK Development Activity
  • Graphcast SDK pull requests merged:
    • PR #270: Replaced Polygon with Matic
    • PR #271: Resolved for latest deployment hash instead of a vec
    • PR #272: Moved message channel and re-exported Waku message type
    • PR #273: Introduced helper functions for peer node data
    • PR #274: Fixed graph account owned current version hash query
    • PR #276: Fixed discv5 toggle
    • PR #278: Released version 0.4.2

Launchpad (Documentation, Starter, Charts, Namespaces)

  • Published a workshop video running through the Launchpad Quickstart
  • We ran two Launchpad Office Hours sessions with the Indexer community, discussing updates to Launchpad, as well as Indexer requirements for the stack, and how to get started with Kubernetes
  • We added more documentation, covering different ways of using Launchpad (with starter, with helmfile, without helmfile) as well as our policy for Supported Namespaces
  • Proxyd deployments are now configured to track node consensus and intelligently route requests by default
  • Fixed Grafana dashboards that are provisioned in the monitoring namespace
  • Various releases of Charts and Namespaces to track latest dependency versions
  • Updated Launchpad Starter linux dependency script
Launchpad Development Activity
  • Launchpad Starter pull requests merged:
    • PR #32: Added the option to bypass CRD issues with the --skip-diff-on-install flag
    • PR #33: Modified the apply command to only skip on install with the --skip-on-install flag
    • PR #34: Updated launchpad Linux dependencies task, added apt update, and fixed consistency issues and k9s version compatibility
  • Launchpad Charts pull requests merged:
    • PR #109: Updated graph dependencies, datasource, and package versions
    • PR #111: Removed overwriting of default paths for k0s and updated README for the chart
    • PR #112: Updated chart version for new tag
    • PR #113: Updated proxyd to version 4 and made it consensus-aware
    • PR #114: Added generic-app chart
    • PR #117: Updated nimbus dependencies
    • PR #118: Fixed issues with release triggers and set releases as drafts
    • PR #119: Released nimbus 0.4.1
    • PR #120: Fixed missing jobs key in workflow
    • PR #121: Released graph-network-indexer 0.2.1
    • PR #122: Released graph-node 0.2.1
  • Launchpad Namespaces pull requests merged:
    • PR #71, PR #73: Updated graph dependencies
    • PR #76: Updated monitoring dependencies
    • PR #77: Added base layer label to sealed-secrets
    • PR #78: Updated ethereum dependencies
    • PR #79: Updated openebs-rawfile-localpv chart
    • PR #80: Adjusted proxyd service endpoint values
    • PR #82: Updated helm dependencies for datasource package
    • PR #84: Added missing arguments to erigon
    • PR #86: Updated securityContext and default rules

Indexer Components (Indexer Service Rewrite)

  • Introduced network query with GraphQL and variables support
  • Improved efficiency by moving graph-node instance to main
  • Various code organization and performance improvements
Indexer Components Development Activity
  • Indexer Service RS pull requests merged:
    • PR #24: Implemented partial paid query flow
    • PR #27: Introduced network query with GraphQL and variables support
    • PR #28: Moved graph-node instance to main to improve efficiency
    • PR #30: Fixed trailing / in subgraphs_base_url, resolved issue with graph-node response JSON escaping
    • PR #31: Cleaned modules and moved generic type to improve code organization
    • PR #32: Introduced server logging middleware
    • PR #33: Added slow and network rate limiter features
    • PR #34: Moved SubgraphDeploymentID tests to types.rs
    • PR #35: Refactored clap deserialize indexer_address into Address and introduced custom deserialize for SubgraphDeploymentID
    • PR #36: Improved Rust cache, used official cargo-llvm-cov install action in CI
    • PR #37: Split network subgraphs query to NetworkSubgraph for better code structure
    • PR #38: Added attestation signers to allocation monitor
    • PR #39: Recorded query metrics
    • PR #42: Refactored AttestationSigners, splitting it off

Network Operations (Graph Network Subgraph, Graph Network Substreams)

  • Added new field coverage to Allocation and Epoch entities, including indexing rewards, query fees, creator and activeForIndexer fields
Network Operations Development Activity
  • Graph Network Substreams pull requests merged:
    • PR #55, PR #56, PR #57: Added various features to the allocation and epoch entities, including indexing rewards, query fees, creator and activeForIndexer fields, and version updates.

Ecosystem Relations

:rocket: Looking Ahead (upcoming priorities)

  • Graphcast
    • Release comprehensive workshop video covering running Subgraph Radio as an Indexer
    • Improve documentation for Subgraph Developers using Upgrade Presync functionality
    • Handle system signals better for more graceful shutdown
    • Continue work on graphcast-web, a library for interacting with Graphcast Network from the web browser
    • Debug observed failures in Listener Radio
  • Launchpad
    • Continue to track, test and release dependency updates
    • Improve our documentation, particularly for provisioning a Kubernetes cluster
  • Indexer Components
    • Continue with Indexer Service rewrite and Scalar TAP integration
  • Network Operations
    • Implement Indexer Management and Cost Model Servers
  • Ecosystem Relations
    • Please join the community for Indexer Office Hours on Tuesday every week at 6pm UTC in the Graph Protocol Discord
    • Please join the community for Launchpad Office Hours on Wednesday every second week at 5pm UTC in the Graph Protocol Discord
7 Likes