This is a recurring call where team contributors gather to discuss major updates from different working groups and brainstorm around active R&D tracks.
Please subscribe to The Graph Foundation’s Ecosystem Calendar, so you don’t miss any of the upcoming calls!
Tentative agenda:
Scalar TAP (Timeline Aggregation Protocol) [Semiotic Labs]
Got something interesting to present and discuss? Feel free to propose any new topic by replying to this thread.
This call will be shorter than usual due to the holiday season.
The Graph Core R&D Workspace
As The Graph is being built in public and in a decentralized way, we’ve put together a new publicly accessible The Graph Core R&D Workspace . You can use this page to follow along with each working group’s progress on significant workstreams and access meeting notes and recordings.
Monthly Core Dev updates
Core development teams are now posting monthly updates in this forum section Core Team Updates - The Graph.
Soon you’ll also find regular updates from all Working Groups and Task Forces on all major components of The Graph stack. Stay tuned!
In Core Devs Call #23, several updates were discussed, find detailed notes below:
Community Updates (00:00)
Scalar Tap Integration Plan (05:26)
Permissionless Payment GIP (15:16)
Launchpad V2 (19:50)
Graphcast Updates (40:20)
Graph Node Updates (53:00)
As always, the community was encouraged to ask questions and participate actively in discussions.
Community Updates (00:00)
Core Devs Call #23 convened core contributors to present their recent work, upcoming features, and discuss protocol changes with community members. The Graph Ecosystem calendar is available at the bottom of these notes, which lists future calls and events. Monthly updates from the Core Developers can be found in the forum, with new information expected to be released in the upcoming days. A notable point of reference is The Graph Q2 2023 Quarterly Participant Update, which delivers a comprehensive insight into recent developments and key performance indicators. The recording of this update is available online, accompanied by a summary thread on Twitter/X.
For a recap on Scalar Tap from the last Core Developers Call reference:
Integration Plan
The Escrow subgraph should be completed by mid-August along with finalizing the integration of Gateway with the TAP library, a collaborative effort between Edge & Node and Semiotic AI. By August’s conclusion, the Gateway Receipt Aggregator is slated for completion, a task undertaken by both Edge & Node and Semiotic AI. Come mid-September, the smart contract audit is expected to be finalized. The latter part of September will see the completion of the portion of Indexer Service to Rust and its integration with Scalar Tap library/manager, a project headed by GraphOps and Semiotic AI.
Prior to the audit’s conclusion, smart contracts will be deployed to the “staging Testnet” by the end of August. This will precede the deployment of the integrated Scalar TAP solution on the “production Testnet” towards September’s close. October is designated for testing, encompassing benchmarking, gathering user and developer feedback, and multiple iterations. The deployment target is set for November.
For Indexers
Indexers are advised to stay updated for specifics regarding the upgrade around October. The standard window for updating the Indexer stack will be available to them. During the ongoing transition, the version of software reported by Indexers (through the /version path of indexer-service) will be a determining factor for the gateway in assessing the Indexer’s capability to support TAP. Receipts will be dispatched by the gateway to the Indexer based on this version check, ensuring a singular association between a query, a receipt, and one of the Scalar protocol versions. The Indexer stack will maintain backwards compatibility, allowing Indexers to simultaneously redeem both the old Scalar and new TAP Scalar receipts. Eventually, once the Gateway transitions to TAP Scalar receipts, the preceding method will be phased out. The tool graph-protocol-qts will be used to facilitate traffic generation for testing on testnet.
This proposal removes the limitation of specific governance-approved entities to pay query fees to Indexers in the network. Eliminating this limitation offers several advantages:
Censorship Resistance: A primary objective for Edge & Node is to enhance The Graph’s resistance to censorship. Making the system permissionless is a proven method to achieve this, as censorship resistance is inherently a trait of permissionless systems. The proposed change aims to eliminate the necessity of permissions to make payments into The Graph Protocol.
Facilitation of Novel Experiences: There’s an emergence of innovative experiences driven by The Graph, which stand to benefit from the removal of this restriction. A notable example is Playground Analytics, which leverages The Graph for data analytics, delivering value to its users. Though they could potentially offer their service via another gateway, doing so would introduce additional steps, escalating costs, and potentially compromising service quality. They would rather engage directly with Indexers for optimal user experience and cost-efficiency.
New Data Service Fees: Anticipation is building around the demand for Indexer-to-Indexer applications, like Firehose, for Indexing subgraphs. In this scenario, Indexers might not need a separate gateway and would prefer sending fees directly to fellow Indexers for a simpler feature implementation. This approach aligns with the vision of The Graph as a comprehensive platform for data services, evolving beyond just GraphQL and subgraphs.
A point of intersection between this discussion and the earlier mention of Scalar TAP is that if a governance-free Scalar TAP was released and added to the approved list of asset holders, it would inadvertently activate this proposal. This is because anyone could then pay Scalar TAP, and in turn, Scalar TAP could remit payment to the protocol. Rather than unintentionally allowing this, the aim is to inform all stakeholders and have the proposal deliberately accepted or rejected by the community. Should the proposal face rejection, there would be an inevitable need to impose additional governance layers on Scalar Tap. Stakeholders are encouraged to review the GIP on the forum and provide feedback, with questions being directed to the forum due to time constraints.
Launchpad V2 (19:50)
Launchpad serves as a foundational software stack designed for local machines, providing a systematic approach through any Indexing infrastructure. It’s particularly tailored to streamline the deployment and management of a scalable Indexer on The Graph’s decentralized network using Kubernetes.
Big shout out to Carlos Jorge whose relentless efforts have significantly contributed to this project!
Looking back, Launchpad V1, introduced last October, was envisioned as an integrated solution for the Indexing infrastructure. Its capabilities spanned from host orchestration to multi-chain operations, all while adhering to robust infrastructure principles. However, certain aspects of V1, such as the limited flexibility with Launchpad charts and the forced usage of the k0s Kubernetes distribution, didn’t resonate well with some Indexers. Moreover, there were concerns about the project’s maturity and the cumbersome workflow of the Launchpad-core module.
Addressing these concerns, Launchpad V2 was developed. This newer version offers greater flexibility, allowing the use of Launchpad charts outside the full stack and in any Kubernetes context. Moreover, V2 has done away with the k0s provisioning layer. This shift gives Indexers the freedom to select their preferred Kubernetes distribution. Another notable change is the replacement of launchpad-core with launchpad-namespaces, introducing a more streamlined system that offers features and flavors. Furthermore, V2 supports various release channels, showcasing a more modular design that empowers users to customize configurations more efficiently.
Launchpad is ideal for users planning on multi-host operations, those keen on Kubernetes for orchestrating their workloads, individuals willing to immerse themselves in Kubernetes, and teams with ambitions to scale their Indexing operations. As for the immediate future, there are plans to enhance the Launchpad documentation. Furthermore, to foster a tighter community and address concerns, Launchpad office hours will be revived, commencing on 16th August at 5 pm UTC, and continuing weekly thereafter. Users are also encouraged to join the dialogue on the Launchpad Discord channel.
Graphcast is an optional decentralized peer-to-peer communication layer within The Graph’s ecosystem. It enables real-time information exchange between Indexers and other network participants at minimal cost. However, message authenticity is verified through a signature by protocol participants.
Essentially, Graphcast operates as a gossip network designed to sidestep high on-chain transaction costs. It achieves this by enabling off-chain communication. The system comprises two main components: The Graphcast SDK, written in Rust and available on cratesio, and the Radios. The SDK integrates with The Graph stack, interacting with the graph-node and, when necessary, the Indexer management server. Radios dictate specific message formats and the logic behind each message.
The Subgraph Radio has integrated the POI (Proof of Indexing) Radio, evolving to accommodate a range of subgraph data sources. This integration means that multiple features now reside within a single Radio. Specifically, the Subgraph Radio houses the POI cross-checking feature, alerting Indexers to discrepancies in POIs. For deployment, a Docker image for Subgraph Radio is available on GitHub. The Radio provides Prometheus metrics and has a Grafana dashboard JSON.
There’s also work on the subgraph upgrade pre-syncing feature, allowing subgraph developers to announce new version releases (with the possibility of this being integrated into Subgraph Studio as well). The pre-syncing feature is not production ready yet but will be soon. Looking forward, Graphcast Web is in development and aims to bring Graphcast to browsers, providing an interface for Subgraph developers to announce upgrades and enhance data visualization.
Recent updates to graph-node have brought significant changes. Last month marked the full availability of Ethereum mainnet Substream powered subgraphs on the network, thanks to collaborative efforts from various teams. One primary focus has been the support for Substreams, especially given the challenges posed by database write bottlenecks. To address this, a batched writes improvement was introduced in the recent release, allowing database interactions to be batched up. This not only reduces time for store interactions and database connections but has led to performance improvements, especially for subgraphs with frequent database interactions.
In terms of future developments, the team is working on integrating Substreams for different graph-node supported protocols, including NEAR and Cosmos. Efforts are also directed towards supporting protocols without a native graph indexing integration, creating a more generic sink. Other upcoming features include passing Substream params from subgraph manifests and increased resilience and robustness for graph-node, allowing more flexibility for Indexers. An additional feature to look forward to is derived field loadings in mappings, allowing developers to avoid storing massive arrays on entities. V.0.32.0 of the graph-node is on the horizon with more functionalities, like periodic lock handlers and new file data sources after IPFS.
A question was asked regarding need for any adjustments within subgraph mappings to enable batch insert, with the answer being no. This process is entirely managed by graph-node and should be seamlessly integrated. It’s activated by default and any variations might only manifest in the logs during Indexing.
Stay Tuned!
Join us next month for Core Devs Call #24!
Keep up to date by joining discussions in the forum, following The Graph on Twitter/X or joining the Discord server.
The 23rd Core Dev call is happening this Thursday 2023-08-10T15:00:00Z.
Please subscribe to The Graph Foundation’s Ecosystem Calendar, so you don’t miss any of the upcoming calls!
Tentative agenda:
The recording will be uploaded here.
The Graph Core R&D Workspace
Monthly Core Dev updates
