# Semiotic Labs January 2024 Update

**URL:** <https://forum.thegraph.com/t/semiotic-labs-january-2024-update/5152>\
**Category:** Core Team Updates\
**Tags:** semiotic\
**Created:** [January 14, 2024, 2:22am UTC](https://forum.thegraph.com/t/semiotic-labs-january-2024-update/5152 "2024-01-14T02:22:21Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sam](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.thegraph.com/sam/32/757_2.png) [@Sam](https://forum.thegraph.com/u/Sam)\
**Post date:** [January 14, 2024, 2:22am UTC](https://forum.thegraph.com/t/semiotic-labs-january-2024-update/5152/1 "2024-01-14T02:22:21Z")

</div>

## 👩‍🚀 Summary

Last month, the primary focus was passing all TAP Scalar’s local tests, completing Verifiable Firehose, and planning for the SQL data service and studio. Key accomplishments included:

- Deploying and operationalizing the TAP and Escrow smart contracts on Sepolia and their respective subgraphs.
- Additionally, a tool based on Verifiable Firehose was implemented for validating transaction data, significantly speeding up the Firehose syncing process. Efforts were also made to create a demo for Ethereum client teams and implement a Substreams Sink for ERA1 file format.
- In the realm of SQL data service and studio, planning was based on E&N’s SQL plan, aiming to initially expose Indexers’ existing Postgres database, with a ClickHouse-based service to follow. This approach was chosen in response to developer feedback and to address data bootstrapping challenges.

## 🎉 Looking back (what was delivered)

### TAP Scalar/payments

Our primary goal last month was to have all local tests pass. He is what we accomplished:

- TAP smart contract deployed and operational on Sepolia (E&N / Tomas)
- Escrow smart contract deployed and operational on Sepolia (E&N / Tomas)
- Escrow subgraph deployed - on testnet, local (Semiotic / Carlos)
- Prepare Docker files / CI build for TAP Aggregator (Semiotic / Alexis)
- Session “How to run the Graph integration tests - with Theo at E&N” [[recording](https://drive.google.com/file/d/11CyXn0u_OnwmFzNv1yvU56k5kKav_aRn/view?usp=sharing)]
- Refactoring TAP (Semiotic / Gustavo)
  - Simplified the requirements/adapters API

### Verifiable Firehose

- We implemented a tool for validating all transactions and receipts from flat files against a set of header accumulators [here](https://github.com/semiotic-ai/header_accumulator)
  - We validated the entire archive of pre-merge data in about 12 hours
    - Streaming flat files from Google Cloud Bucket
    - Lightweight verification \< 10% CPU utilization and \< 6 GB memory utilization on GCP e2-standard machine

  - This enables Indexers to trustlessly download flat files from peers and bootstrap their Firehose synching processes
    - From months to hours for Firehose synch
    - CAVEAT: Only trustless with regards to Receipts and Transactions, i.e., we don’t currently provide a method for validating trace data stored in flat files

- We made progress on a demo for Ethereum client teams (Nethermind in particular), which showcases how to obtain historical data in a familiar format for EIP-4444 (ERA1 files)
  - We implemented a Substream that buffers one Epoch (8192 blocks) of receipts and transactions and appends the appropriate header, accumulator, [here](https://github.com/semiotic-ai/era_file_substream)

We started implementing a Substreams Sink for storing data in ERA1 file format.

### SQL data service and studio

- Did planning for how the SQL data service & studio would be built
  - We used E&N’s SQL plan as a guide
  - Will start by exposing Indexers’ existing Postgres database (populated when indexing subgraphs)
    - ClickHouse will come later as a separate substreams-based data service

  - Semiotic is prepared to make all changes: graph-node, gateway, GUI/studio
  - Make as few changes as possible to launch the MVP

- Why?
  - Started to hear a common theme in interviews that developers want to access _existing_ data from The Graph
  - Also, by exposing the existing data, we solve the data bootstrapping problem that we would have with the ClickHouse approach

## 🚀 Looking ahead (upcoming priorities)

### TAP Scalar/payments

#### Milestone 0: Running the end-to-end tests on local-network

- Running test Gateway with TAP Receipt Aggregator and Escrow Manager (E&N / Theo)
  - TAP Aggregator (E&N / Theo)
  - Gateway Escrow Manager (E&N / Theo)
  - Deploy TAP-compliant Gateway (E&N / Theo + Abhi)

- Running test TAP-compliant Indexer (Semiotic / Carlos + Gustavo, support by Alexis)
- Running the elementary end-to-end test on local network (E&N + Semiotic / Theo + Alexis?) - finish by middle of January
  - Indexer receives queries and receipts
  - Indexer aggregates receipts properly (RAVs)
  - Make sure that the Indexer collects payments for RAVs

- Running more sophisticated end-to-end tests on local-network
  - Test failure modes

#### Milestone 1: Running the end-to-end tests on Testnet with help from Indexer DAO

- Escrow subgraph deployed - on Testnet, on hosted service (Semiotic / Carlos)
  - Preparation for tests run by Indexer DAO on Testnet

### Verifiable Firehose

- Clean up flat file validation code (tests and documentation), polish repos, and share with The Graph community
- Finish the EIP-4444 demo, sync up with Nethermind, and determine the next steps for the EIP-4444 solution
- Begin work on the next phase for verifiability: building on VFH to build an efficiently verifiable blockchain data processing framework (zk-coprocessor), with the ultimate goal of efficient on-chain verification

### SQL

- Get feedback on the SQL data service and studio plan
- Work toward MVP with E&N