StreamingFast September 2023 Update

:woman_astronaut: Summary

Hello everybody!

:tada: Looking back

  • Implemented better tooling to compare data from RPC (fireeth tools compare-blocks-rpc). Investigated issues with Polygon data and a fix is on its way.
  • ERC-20 Substreams built, using storage changes to great effect: 97% of transfers with previous and new balances! See blog post.
  • We rolled out substreams v1.1.12 with improvements on progress reporting and metrics. See screenshots and juicy details in our blog post.
  • Wrote new tool to compare blocks from RPC and Firehose.
  • V1 for the RPC Poller Firehose
  • (SF infra) Rolled out Sepolia testnet
  • Collected lots of feedback on pricing models for Firehose and Substreams in the network.
  • Good progress on the postgres Subservice (read: deployable unit!). Fast insertion built-in now (CSV → COPY FROM).
  • Solana support for Address Lookup Table, as a reusable module for Solana users.

:rocket: Looking ahead

  • Work on Extended Tracer merging (geth and Arbitrum).
  • Postgres sink driver abstraction for Clickhouse, and make it a Subservice.
  • Continued work on larger items:
    • Research for new languages supported by Substreams
    • Design of the portal/registry experience.
    • Draft first iteration of the RPC Poller Extractor for Firehose, on chains where instrumentation is hard
    • Design discussions around IPFS/Arweave/Filecoin/URL resolver support in Substreams.