Semiotic Labs January 2024 Update

Research & Development Core Team Updates
1

:woman_astronaut: Summary

Last month, the primary focus was passing all TAP Scalar’s local tests, completing Verifiable Firehose, and planning for the SQL data service and studio. Key accomplishments included:

  • Deploying and operationalizing the TAP and Escrow smart contracts on Sepolia and their respective subgraphs.
  • Additionally, a tool based on Verifiable Firehose was implemented for validating transaction data, significantly speeding up the Firehose syncing process. Efforts were also made to create a demo for Ethereum client teams and implement a Substreams Sink for ERA1 file format.
  • In the realm of SQL data service and studio, planning was based on E&N’s SQL plan, aiming to initially expose Indexers’ existing Postgres database, with a ClickHouse-based service to follow. This approach was chosen in response to developer feedback and to address data bootstrapping challenges.

:tada: Looking back (what was delivered)

TAP Scalar/payments

Our primary goal last month was to have all local tests pass. He is what we accomplished:

  • TAP smart contract deployed and operational on Sepolia (E&N / Tomas)
  • Escrow smart contract deployed and operational on Sepolia (E&N / Tomas)
  • Escrow subgraph deployed - on testnet, local (Semiotic / Carlos)
  • Prepare Docker files / CI build for TAP Aggregator (Semiotic / Alexis)
  • Session “How to run the Graph integration tests - with Theo at E&N” [recording]
  • Refactoring TAP (Semiotic / Gustavo)
    • Simplified the requirements/adapters API

Verifiable Firehose

  • We implemented a tool for validating all transactions and receipts from flat files against a set of header accumulators here
    • We validated the entire archive of pre-merge data in about 12 hours
      • Streaming flat files from Google Cloud Bucket
      • Lightweight verification < 10% CPU utilization and < 6 GB memory utilization on GCP e2-standard machine
    • This enables Indexers to trustlessly download flat files from peers and bootstrap their Firehose synching processes
      • From months to hours for Firehose synch
      • CAVEAT: Only trustless with regards to Receipts and Transactions, i.e., we don’t currently provide a method for validating trace data stored in flat files
  • We made progress on a demo for Ethereum client teams (Nethermind in particular), which showcases how to obtain historical data in a familiar format for EIP-4444 (ERA1 files)
    • We implemented a Substream that buffers one Epoch (8192 blocks) of receipts and transactions and appends the appropriate header, accumulator, here

We started implementing a Substreams Sink for storing data in ERA1 file format.

SQL data service and studio

  • Did planning for how the SQL data service & studio would be built
    • We used E&N’s SQL plan as a guide
    • Will start by exposing Indexers’ existing Postgres database (populated when indexing subgraphs)
      • ClickHouse will come later as a separate substreams-based data service
    • Semiotic is prepared to make all changes: graph-node, gateway, GUI/studio
    • Make as few changes as possible to launch the MVP
  • Why?
    • Started to hear a common theme in interviews that developers want to access existing data from The Graph
    • Also, by exposing the existing data, we solve the data bootstrapping problem that we would have with the ClickHouse approach

:rocket: Looking ahead (upcoming priorities)

TAP Scalar/payments

Milestone 0: Running the end-to-end tests on local-network

  • Running test Gateway with TAP Receipt Aggregator and Escrow Manager (E&N / Theo)
    • TAP Aggregator (E&N / Theo)
    • Gateway Escrow Manager (E&N / Theo)
    • Deploy TAP-compliant Gateway (E&N / Theo + Abhi)
  • Running test TAP-compliant Indexer (Semiotic / Carlos + Gustavo, support by Alexis)
  • Running the elementary end-to-end test on local network (E&N + Semiotic / Theo + Alexis?) - finish by middle of January
    • Indexer receives queries and receipts
    • Indexer aggregates receipts properly (RAVs)
    • Make sure that the Indexer collects payments for RAVs
  • Running more sophisticated end-to-end tests on local-network
    • Test failure modes

Milestone 1: Running the end-to-end tests on Testnet with help from Indexer DAO

  • Escrow subgraph deployed - on Testnet, on hosted service (Semiotic / Carlos)
    • Preparation for tests run by Indexer DAO on Testnet

Verifiable Firehose

  • Clean up flat file validation code (tests and documentation), polish repos, and share with The Graph community
  • Finish the EIP-4444 demo, sync up with Nethermind, and determine the next steps for the EIP-4444 solution
  • Begin work on the next phase for verifiability: building on VFH to build an efficiently verifiable blockchain data processing framework (zk-coprocessor), with the ultimate goal of efficient on-chain verification

SQL

  • Get feedback on the SQL data service and studio plan
  • Work toward MVP with E&N
6 Likes