Hello everyone, this is Christophe (stopher on Discord) from Protean Labs and I am very excited to share the project we’ve been working on as part of The Graph grant program: Subgrounds!
TL;DR: Subgrounds is a schema-based declarative Python library for querying, transforming and visualizing blockchain data indexed by The Graph. It aims to enable users to define their analyses with respect to the entities that are made available through subgraph schemas instead of the raw data that was indexed by the network.
Problem
Tremendous progress is being made to increase indexing performance and enhance the user experience of protocols using subgraphs as their data layer. However, there is still some work to be done with respect to the tooling surrounding the actual querying and usage of the indexed data, and we hope that Subgrounds can be a good step in that direction.
Having worked with subgraph data for some time ourselves, we have identified three pain points which Subgrounds addresses in a novel way:
-
Data wrangling: Everything between and including querying the data from The Graph to using the data in a productive way that is not context dependent (e.g.: formatting the data into dataframes, pagination for large queries, filling out missing values, etc.) requires re-implementing the same functionality which could otherwise be abstracted and provided in a library.
-
Crafting type safe queries: GraphQL queries are often represented as simple strings before being sent to The Graph for execution. This makes bugs caused by typos or outdated queries hard to catch before actually sending the invalid query and receiving an error response.
-
Data transformations: Performing data transformations requires the user to fetch data and is done on a case-by-case basis which makes common transformations hard to port across different subgraphs.
Solution
Our solution is Subgrounds: a declarative Python framework that combines flexible subgraph querying primitives with Dash components and Plotly objects to enable anyone to spin up a dashboard displaying subgraph data in a few lines of code (amongst other things).
Subgrounds’ design was informed by a very simple but powerful principle: empowering domain expertise without the need of technical expertise. In other words, our target demographic consists of domain experts (e.g.: DAO researchers, financial analysts, NFT connoisseurs) who wish to explore and visualize the trove of data indexed on The Graph but who are not proficient programmers or otherwise familiar with the nitty-gritty of data science.
Inspired by Bubbletea (from fellow grantee @huangkuan) as well as popular Python libraries like SQLAlchemy and sgqlc, Subgrounds makes use of a variety of technologies and approaches to facilitate the visualization of subgraph data:
-
Data visualization: Subgrounds is tightly integrated with Dash, the popular analytics-oriented web framework developed by Plotly allowing for plug-and-play functionality. Since Subgrounds components are extensions of Dash components, they can easily be added to any Dash app.
-
Type-safe query creation: Subgrounds performs automatic GraphQL schema introspection to generate objects that allow developers to query data from The Graph in pure Python (no raw GraphQL stored in strings needed!). Pagination is also automatic for larger queries.
-
Data transformations: Subgrounds supports the definition of synthetic fields - subgraph entity fields that are the result of some transformation applied to other fields. Synthetic fields are analogous to SQL views and can be accessed at querying time just like regular fields, streamlining the process of data transformation. Synthetic fields can also be composed of other synthetic fields, which makes them useful building blocks when computing complex transformations.
Conclusion
Although still in its infancy, Subgrounds aims to become the standard framework to build dashboards on top of The Graph and to serve as the “gateway drug” to The Graph for traditional Python devs. As part of our roadmap, we will be working with other developers to integrate features of other Python libraries built for The Graph (e.g.: Bubbletea) as well as supporting other plotting backends (e.g.: Altair). Now is also the time discuss the design of Subgrounds, share ideas and propose new features. Subgrounds is a co-creative effort and everyone’s input is welcome!
Checout the project’s repo here: GitHub - Protean-Labs/subgrounds: A framework for integrating The Graph data with dash components