StreamingFast March 2024 Update

Heey, lots brewing here. Check it out:

:tada: Looking back

:rocket: Looking ahead

In the next few weeks, we’re going to be diving deep into Substreams optimizations, implementing blockIndex/blockFiltering at the Substreams-level, alongside cost optimizations when reading cached files.

  1. This feature already exists within Firehose, and is what caused 90% reduction in costs with sparse Subgraphs when we first integrated into graph-node. One will be able to write a freeform Substreams module and have the whole chain indexed once, and then query those indexes to jump only to blocks with content you’re interested in. For example, we’ll have public modules that index all the ERC-20 from/to addresses, another module with contracts being called. That will be a chain agnostic way to sift through chains at extreme speeds. See this issue: Create a substreams module that filters out (reduce) the event logs based on the same language as the blockindex · Issue #415 · streamingfast/substreams · GitHub and its dependencies.
  2. This will avoid reading full blocks when cached files exist that would avoid the need to read the full block. The current engine is clocked on reading a stream of full merged-blocks, make it difficult to avoid reading them when everything has already been processed. We’re going to flip over the execution of tier2 jobs, to only dynamically load what is necessary, and what is missing (from the output module up the parent tree). See issue: Make tier2 jobs able to run only on cached mapper and stores outputs (without blocks when not needed) · Issue #418 · streamingfast/substreams · GitHub

These two major features will turn the Substreams engine into something much more powerful. Allowing even wallets, accounting firms, tax software, to be able to use this as a high speed backend, without the need to index all the things to live databases, with the incurred costs of RAM, CPU, disks, etc… We’re getting closer to a BigQuery engine that is historical, but also real-time and fork-aware, with its beautiful cursor-based navigation.

This will be awesome.


oh yeah and we were at ETH Denver, had a blast, and there are many folks interested by what we’re all doing. Chains and data consumers. Stay tuned!


and we should be releasing Sei integration soon… and the patch to bring the Firehose directly into geth is nearly merged, and there are 4-5 new protocols in the pipe for integration.