Semiotic Labs February 2024 Update

Research & Development Core Team Updates
1

:woman_astronaut: Summary

Since late December, the TAP (Timeline Aggregation Protocol) initiative has shifted towards running integration tests using the local network suite. Over the last month, significant progress was made on Milestone 0, which involves running end-to-end tests on a local network. This included merging branches for integration and fixes, merging Indexer PRs, and successfully running the local network on different configurations.

Efforts to run the local network with TAP involved merging Hope’s branch for GraphOps integration, enabling a TAP-enabled Gateway, and implementing a TAP-compliant Indexer. Basic end-to-end tests succeeded in Indexer operations like receiving queries and aggregating receipts, with ongoing work to collect payments via RAV redeem.

Additionally, there were developments outside TAP testing, such as implementing the ERA1 files (our EIP-4444 solution) and preparing the Subgraph:SQL MVP for testing.

TAP

Last month

From the end of December => New direction: run integration tests using the local-network suite [TAP sync updates]

Milestone 0: Running the end-to-end tests on local-network

  1. Fixing local-network
  • Merge Jannis vs Theo branches => integration & fixes
  • Indexer PRs merged! (E&N / Theo and Ford!)
  • Local network running on two configurations:
    • Running locally on Mac M2 / ARM64 → docker/colima
    • Running on GCP/Linux x86, docker (still unstable!)
  1. Running local-network with TAP
  • Merging Hope’s branch
  • Running TAP-enabled Gateway (with TAP Aggregator & Gateway Escrow Manager)
  • Running TAP-compliant Indexer
  1. Running the basic end-to-end test on local-network
  • Indexer receives queries and receipts
  • Indexer aggregates receipts properly (RAVs)
  • Indexer collects payments via RAV redeem
    • To collect payments: close the allocation manually with Indexer CLI

Notable TAP PRs

Indexer-rs:

  • (tap-agent) Store failed receipts and RAVs for logging #108
  • (service) Rejection of senders that are in denylist #109
  • Fix support for sender signer accounts (think like indexer operator accounts) #110
  • Various smaller fixes #111 #112 #114 #116 #117

Indexer (indexer-common):

  • Update @graphprotocol/indexer-common to enable optional address-book. #818
  • Bump the dependency of cost-model to latest inside indexer-common. #829

TAP core:

TAP subgraph:

  • More thorough unit testing #15
  • Improved subgraph schema #16

TAP next month

  • Finish Milestone 0: Running the end-to-end tests on the local network
  • Escrow subgraph deployed - on Testnet, on hosted service
  • Merge “collect RAVs” branch (Indexer-agent)
  • “TAP works” → green light for Infra DAO!
  • Milestone 1:
    • Running the end-to-end tests on Testnet (Infra DAO, previously known as Indexer DAO;))
    • Ask Indexers / Infra DAO to do their “regular stuff.”
  • Running systematic end-to-end tests on the local network

Verifiable Firehose

Last month

  • We implemented the ERA1 files Substream sink, here

    • We tested the sink by consuming data from Streaming Fast’s Firehose endpoint using the ERA file Substreams package we wrote last month
    • We can exactly reproduce ERA1 files as expected from their spec (e.g. hash digests match expected values) for all pre-merge history

  • We have made update to the flat-file-decoder and header-accumulator, mostly cleaning up code and documenting

    • This code enables Indexers to download flat files from peers and trustlessly validate the correctness of blockchain data against Ethereum’s canonical history, e.g., enables Firehose “warp-sync.”

  • We also submitted a talk to EthDenver discussing zk-coprocessors and how The Graph is uniquely positioned to provide solutions in this space

Verifiable Firehose next month

  • Share flat file verification tools with Indexers and get feedback
  • Catch up with Nethermind and demo ERA1 file sink and plan out next steps for EIP-4444
  • Present at EthDenver :crossed_fingers:

SQL data service

Past Month

  • Graph-node implementation of MVP complete!
  • Login and Logging enabled on the website.
  • Basic Studio
  • Major changes to the plan:
    • Stage 1 - Testing:
      • VSCode Studio connected to graph-node. We’ll give access to a few people for testing. We eat the cost of SQL queries.
    • Stage 2 - Private Beta:
      • Deploy a SQL gateway. Restricted list of Indexers (whitelist in gateway) and Consumers (whitelist on the website). Flat price per query. Enable a website for API key management
    • Stage 3 - Subgraph:SQL MVP:
      • Remove access restrictions—potentially more complex pricing.

SQL data service next month

  • VSCode studio - this is the main blocker for Stage 1 now!
  • API key management
  • Python package?
4 Likes