Summary
Throughout February, significant efforts were focused on charting a course towards operationalizing Substreams and/or Substreams:SQL technology to address real-world analytics challenges. Numerous trials, experiments, and benchmarks were conducted using PostgreSQL, ClickHouse, and Substreams-Sink to tackle crucial scalability obstacles and facilitate data integration across various protocol data sources.
Substreams:SQL technology offers a great feature set, serving as an all-in-one solution for managing lite scale analytics scenarios characterized by small volume and/or isolated data. However, when it comes to common enterprise use cases, both PostgreSQL and ClickHouse encounter unresolvable shortcomings that render them impractical for enterprise production use cases.
Substreams-Sink presents a promising solution by abstracting Substreams data into a localized containment, enabling further integration into virtually any modern data warehousing solution, unlocking limitless analytics potential. However, several critical milestones must be achieved to render Substreams-Sink viable for enterprise-level use cases. Firstly, addressing the absence of key features such as data type support is essential to ensure proper export of Substreams data. Secondly, the development of necessary glue work around Substreams-Sink is needed to facilitate required automation for scale up.
Looking back (what was delivered)
- Various trial, experiment and benchmark attempts across Substreams:SQL and Substreams-Sink technology stack.
- SQL-based protocol transformations - additional coverage
- Completed all signal migration from Ethereum to Arbitrum
Looking ahead (upcoming priorities)
- Complete remaining exploration required to evaluate feasibility of Substreams-Sink, and identify feature gaps that need to be addressed.