The Graph Core Dev Call #27

:information_source: The 27th Core Dev call is happening next Thursday 2024-01-25T16:00:00Z.

Join us!

:point_right: Launch Meeting - Zoom

This is a recurring call where team contributors gather to discuss major updates from different working groups and brainstorm around active R&D tracks.

:calendar: Please subscribe to The Graph Foundation’s Ecosystem Calendar, so you don’t miss any of the upcoming calls!


:notebook: Tentative agenda:

  • Subgraph Radio updates [Petko, GraphOps] (10 mins)
  • Launchpad updates [Carlos, GraphOps] (10 mins)
  • Horizon brief presentation + Q&A [Pablo, Edge & Node] (20 mins)
  • TBD

:person_raising_hand: Got something to present and discuss?
Feel free to propose any new topic by replying to this thread.

:tv: The recording will be uploaded here.


:rocket: The Graph Core R&D Workspace

As The Graph is being built in public and in a decentralized way, we’ve put together a new publicly accessible The Graph Core R&D Workspace . You can use this page to follow along with each working group’s progress on significant workstreams and access meeting notes and recordings.

:crystal_ball: Monthly Core Dev updates

Core development teams are now posting monthly updates in this forum section Core Team Updates - The Graph.
Soon, you’ll also find regular updates from all Working Groups and Task Forces on all major components of The Graph stack. Stay tuned!


See you Thursday! :astronaut:

3 Likes

Is there any way to get access to the recording faster than it is currently uploaded to the youtube channel? I tried to access the Core R&D workspace as well but many of the linked vidoes are unlisted so I cannot view them.

2 Likes

Hi @Athsrueas,

We usually upload pretty quickly, but indeed this one is a bit delayed. It’ll be online soon. We’re done with editing. I’ll tag you here once it is.

As for the working group sessions you see in the Workspace, most of those playlists are now private, as requested by a good amount of core developers. We do want to work in public as much as possible, so I’ll think of an alternative that is suitable for both parties. Would you prefer the full conference recording of each working group session (around ~2-3 a week on average, each lasting 45 minutes), or some audio transcripts / summarized notes?

3 Likes

Thank you for the reply, if the team consensus is that the videos should be private I definitely respect that. I’m a graph advocate and I create content like what you are describing (summaries or articles pertaining to team updates sources from these videos) so while it would be more convenient to have a transcript I think actually it would reduce the quality of what I produce.

athsrueas.site

If the teams consent to giving me access I would like to be able to provide the community with updates on what the teams are working on.

2 Likes

Summary

Subgraph Radio Updates (3:16)

  • Introduction to Release 1.0.0
  • Key Features and Enhancements
  • Integration and Benefits
  • Community Engagement and Resources

Launchpad Report Sept. 2023 – Jan. 2024

  • Overview and Features
  • Toolkits and Preconfigured Namespaces
  • Development, Engagement, and Future Plans
  • Kubernetes Migration Experience

Graph Horizon Overview (26:39)

  • Concept and Goals
  • Economic and Security Enhancements
  • Protocol, Automation, and Decentralization
  • Fee Structure and Community Involvement
  • User Experience and Operational Efficiency
  • Community Engagement and Feedback

Subgraph Radio Updates (3:16)

Petko, of GraphOps, shared information on Subgraph Radio Release 1.0.0 which brings many new changes and updates. The presentation was previously shared at The Graph’s Community Talk and Indexer Office Hours; the notes detailing this release can be found below:

Subgraph Radio, initially introduced alongside the initial GRC and the concept of Graphcast, has achieved a significant milestone with release 1.0.0. This release brings notable improvements, including database persistence, which enhances scalability and data efficiency by replacing the older JSON file system. The update has also streamlined configuration validation, crucial for verifying Indexer addresses against on-chain data. The Graphcast network has been scaled up through a switch to a more efficient relay protocol, ensuring more stable and reliable message propagation.

Further developments include the integration of Subgraph Radio into the Squids Docker compose stack and Launchpad, facilitating easier access for beginners and Kubernetes users. For network participants, Subgraph Radio offers multiple benefits. It enables efficient data exchange among Indexers, featuring POI crosschecking to identify discrepancies and a Grafana dashboard for aggregated network intelligence. A standout feature is the subgraphs upgrade intent feature, which will allow subgraph developers to inform Indexers about new versions in advance, thus minimizing downtime.

Additionally, the platform provides configurable notifications for divergences and a GraphQL API for detailed analysis. Educational resources such as documentation and a YouTube playlist with workshops are made available, catering especially to subgraph developers. Community engagement is encouraged through the Graphcast channel on The Graph Discord, inviting discussions and feedback.

In summary, Subgraph Radio’s development marks a significant step forward in data exchange and network intelligence within The Graph ecosystem, enhancing the tools and features available to Indexers and other network participants.


Launchpad Report Sept. 2023 – Jan. 2024 (12:00)

Launchpad serves as a stack repository and tool stack, designed to streamline workflows through infrastructure layers. It operates as a starter repository and client-side tool stack on a host Kubernetes platform, focusing on a declarative approach for managing infrastructure and deployments. Launchpad facilitates repository cloning and customization, enabling the running of a scalable Indexer on The Graph’s decentralized network.

The platform utilizes Kubernetes software packages, known as charts, and namespaces for bundling related charts, providing core functionalities for deployment and interaction within Kubernetes. This includes monitoring, secrets management, storage, and ingress. Launchpad’s toolkit comprises Taskfile, Helmfile, Helm, and Kubectl, and it offers preconfigured namespaces for the Graph Indexer stack and major blockchain dependencies.

The starter repository can be found at GitHub - graphops/launchpad-starter: Starter repo for the Launchpad Kubernetes Toolkit, which users clone into their own private repository, such as /YOU/your-infra. This private clone becomes the user’s personal infrastructure as code.

The tool utilizes Helm charts from GitHub - graphops/launchpad-charts: Helm Charts for deploying Web3 applications into Kubernetes for application packaging and deployment. Helmfile, featured in the same GitHub repository, serves as a declarative Helm Release Orchestrator, and Taskfile is used to template common tasks and operations easily.

Preconfigured namespaces are provided, like those for monitoring, storage, Ethereum, and the graph-indexer, which can be found at GitHub - graphops/launchpad-namespaces: Preconfigured Kubernetes Namespaces using Helmfile. These namespaces come with stable and canary release channels to keep track of updates and include preconfigured Helm Releases. Users can customize the configuration for any release within a namespace, and some namespaces come with feature toggles or different flavors.

Launchpad’s development is continuous, introducing features like chart overrides, a new scaling interface, generic app charts, improved labels, and flavor-dependent defaults. The process involves a high rate of merged PRs, primarily via a dependency bot, and automated canary release pipelines. The platform also expands The Graph stack with support for additional chains, as evidenced by over 250 merged PRs authored by the Bot.

Community engagement is a priority, with Launchpad hosting office hours, improving documentation, and interacting on Discord, YouTube, and Twitter. Biweekly Discord sessions, YouTube recordings, and Twitter interactions, complemented by comprehensive documentation at GraphOps Docs, reflect this focus.

Future developments aim to integrate Firehose and enhance infrastructure support. Plans include improving documentation, expanding chain support (including Polygon, Celo, Optimism), and upgrading monitoring, storage namespaces, and the observability stack. Additionally, Launchpad aims to incorporate semantic search to enhance observability and support the Firehose project.


Question:

How has it been converting Indexers to Kubernetes?

Answer:

The experience has been positive with quite a few partners showing interest in migrating to Kubernetes. It’s an ongoing process, with many logging into it and seeing the benefits even if it’s not the same for everyone. More are interested in migrating rather than having already completed the migration.

Question:

Is there work involved in migrating to Kubernetes?

Answer:

Yes, there is work involved in migrating to Kubernetes. However, Kubernetes offers a better scaling solution in terms of the number of engineers needed to scale the infrastructure, especially after reaching a certain scale, compared to many alternative solutions.


Get support for Launchpad:

Graph Discord: #kubernetes-launchpad channel
Email GraphOps at: hello@graphops.xyz


Graph Horizon Overview (26:39)

Graph Horizon is a proposed update to The Graph, focusing on making it more modular, flexible, robust, decentralized, and resistant to censorship. It aims to move away from a monolithic structure, allowing a broader range of data services without needing governance approval, thus spurring innovation and diversifying services.

Economic and Security Model Changes

Graph Horizon plans significant changes to enhance economic security and clarity. It suggests making delegations slashable and introducing indexing fees to complement existing rewards, promoting permissionless participation and strengthening censorship resistance. The approach includes permissionless or decentralized arbitration, letting each data service customize its arbitration and verification processes, reducing governance involvement. A key part of the strategy is a modular staking contract where providers stake GRT to ensure service integrity, with verifiers having the power to slash stakes based on set rules.

Protocol Structure and Governance

Graph Horizon promotes permissionless composability, allowing new data services with unique rules, independent of core staking contract approval. The trust in the core staking contract is crucial, with discussions on whether to use a fully immutable contract or a time-locked upgrade process. This flexibility allows each data service to create its arbitration mechanism, possibly leading to a marketplace of arbitrators valued for their reputation and fairness.

Automation and Verification

A primary goal of Horizon is to increase automated, on-chain verifiability for data services, reducing reliance on human arbitration and improving economic security.

Decentralization and Censorship Resistance

Graph Horizon is committed to enhancing censorship resistance and decentralization, notably aiming to remove the need for oracles in indexing rewards.

Indexing and Fee Structure

Graph Horizon introduces GIP-58, adding indexing fees as a method for indexing subgraphs alongside existing rewards, without immediate changes to the latter.

Community Involvement and Protocol Evolution

Graph Horizon aims to wisely use GRT issuance, encouraging discussions on its best use without altering indexing rewards. It addresses concerns about consumer choice leading to centralization by suggesting automated selection and encouraging developers to spread indexing fees across multiple indexers, improving service quality and decentralization. It seeks more efficient interactions by removing the need for curation fees or protocol taxes for services not based on issuance and indexing rewards, potentially reducing costs and aiding The Graph’s growth. Community discussions are focused on evolving the protocol towards Graph Horizon, comparing the incremental improvements of the Brownfields approach with the new, parallel protocol structure of the Greenfield approach.

Operational Efficiency and User Experience

Graph Horizon aims to simplify the process for developers to index their subgraphs, ensuring more predictable pricing and a better overall experience. The urgency of these discussions is emphasized, given the limited time to address these significant changes.

Comments

The follow-up discussion centered on the evolving relationship between gateways and Indexers in the context of Graph Horizon and the quantity of gateways in relation to Indexers.

The discussion concluded with a reminder to engage in the community forum, emphasizing the importance of community input, especially in the design phase of what is expected to be a multi-month project, whether it adopts a Brownfield or Greenfield approach. The community was encouraged to share thoughts, participate in upcoming meetings, and contribute to the decision-making process.

Check out the forum post here!


Stay Tuned!

Join us next month for Core Devs Call #28!

Keep up to date by joining discussions in the forum, following The Graph on X/Twitter or joining the Discord server!

2 Likes

I appreciate this feedback. We did upload the video 3 days ago: https://youtu.be/FEjk43G24Oc

As you can see from above, Ethan always provides a summary, and we also do tweet threads with the highlights of each call. Feel free to reach out if there’s anything you think we could do differently. :saluting_face:

3 Likes

I have no complaints, I understand there can be delays with uploading the videos. The forum posts are always incredibly thorough and a great place to extend the conversation. I am simply unable to tune into the calls due to my schedule so I am always looking forward to the release of the video.

1 Like