StreamingFast May 2023 Update

Research & Development Core Team Updates
1

:woman_astronaut: Exec summary

The whole team is heads down improving the Substreams stack, by taking on some of the worst-cases of Subgraphs. Multiple features were developed along the way:

  • Performance issues, and improving debuggability of such issues
  • Improved undo and reorg management reliability (with a second iteration of the RPC protocol). This requires an update on graph-node which is PR’d.
  • Massive improvements to the substreams gui, which now covers all the things substreams run did, and much more.
  • High-speed injection of Subgraph data from Substreams output (using COPY FROM per entities, and a posteriori parallel index creation).

Most of our efforts this week were to push the Uniswap-v3 Substreams to its limits, refactoring, designing new patterns, improving the whole stack. Dogfooding basically.

Our goal for next month is to demonstrate how powerful Substreams-based Subgraph can be, with a decent user experience. We also want to demonstrate how diverse sinks can solve different sets of problems: for instance, how a key-value store could massively improve the page-load time of info.uniswap.org … especially for those time series.

4 Likes
2

Can you outline some of the “massive” improvements to the GUI?

1 Like
3

Sorry for the late response here.

Some of the massive improvements from the GUI are:

  • Being able to search within the decoded bytestream, isolating each module.
  • Cycling through modules with three methods: prev/next (u and i), searching by fuzzy prefix with m, and navigating the graph of input dependencies (with M).
  • Search using text match, as well as jq style querying, for anything protobuf decodes to JSON.
  • Jump to next search matches with O and P
  • Navigate through blocks normally with o and p.
  • Show/hide the logs with l
  • Hit r to restart the stream, while keeping the context of your search and module/block position untouched. Perfect for iterating on some code, and restarting the stream.
  • Cycle through the bytes array decoding, with F, so they can show up as base64, hex or strings. And then, you can use the search feature to search through that format.
  • Massively sped up its performance
  • Better display of the progress in parallel operations.
  • Improved display in the Request tab too.
  • The gui also has some good support for reloading from a replay.log file (which is written each time a stream is started).

So quite a bit of new and shining things, which should make the gui the best thing to use, even before run.

I’ll make sure this ends up in docs somewhere.

2 Likes