The Graph + Singer = Blockchain data pipeline!

I’ve been doing a lot of work with blockchain data recently and I wanted a way to incorporate The Graph into a web2 data pipeline…so I built it.

A bit of background: Singer is an open-source spec for writing data extraction scripts (taps) and data loading scripts (targets). The idea is that any tap is compatible with any target, so you don’t need to roll your own script if one already exists for your source / destination!

I wrote a tap for The Graph so now you can load entities from subgraphs into Postgres, Snowflake, MongoDB, CSV, etc. or any other destination with a Singer target.

I wrote a blog post about it and put the tap on GitHub.

Feel free to try it out and let me know if you have any feedback!

4 Likes

This seems very useful. Im trying to have SQL beginners use blockchain data and have used Dune and Flipside. Was planning to use graph/subgraphs. Curious why you wanted to port a subgraph into another DB? why not just write SQL on Dune/flipside?

1 Like

@svishal2001 I originally built this for a client (a UniV3 allocation protocol) to support their user-facing analytics.

I wrote this adapter so that I could integrate data from The Graph into a traditional web2 data engineering workflow (dbt for transforms, Postgres for data warehouse, etc.).

I haven’t checked in a while but at the time, Dune didn’t offer API access and both Flipside had limited data sets. Whereas since anyone can write and deploy a subgraph, The Graph had more flexibility than Flipside / Dune.

Also, there was already a community around The Graph and lots of popular projects had written subgraphs, so I thought it made sense to leverage that pre-existing work.

makes sense. thanks for the update. Dune still doesnt have API access, though flipside just launched a free API tier.