Testnet Docker Guide by StakeSquid

Community & Grants Resources
1

Hello future Graphstronauts :rocket:

As the testnet rapidly approaches, me and my partner @czarly have decided to make public the docker setup that we used during the Mission-Control-Testnet, and adapted it for the new testnet release.

Setting up the software infrastructure is extremely easy, and it runs all on the same machine, so you don’t need to worry about Kubernetes Voodoo or any other kind of black magic.

The advantage of using Docker, as opposed to systemd bare-metal setups, is that Docker is easy to manipulate around and scale up if needed. We personally ran the whole testnet infrastructure on the same machine, including a TurboGeth Archive Node (not included in this docker build). Another good thing about Docker is that the data is stored in named volumes on the docker host and can be exported / copied over to a bigger machine once more performance is needed.

The installer comes with everything that you need to get started with: graph-node, indexer, prometheus and grafana, all in a simple, easy to install package :slight_smile:

Note that you need access to an Ethereum Archive Node that supports EIP-1898 . The setup for the archive node is not included in this docker setup.

The minimum configuration should to be the CPX51 VPS at Hetzner. Feel free to sign up using our referral link – you can save 20€ and we get 10€ bonus for setting up more testnet nodes to support the network growth. :slight_smile:

You can either continue to read the rest of the guide here, or in the GitHub Repository.
We’ll try to keep both the forum post, and the GitHub Readme up to date throughout future changes.

Link to the GitHub Repository: GitHub - StakeSquid/graphprotocol-testnet-docker: Graph Protocol Testnet Docker Guide by StakeSquid. Easy to use, one-stop gateway into the Web3 universe via The Graph.
Link to the Video Guide: StakeSquid - The Graph Testnet Docker Guide - YouTube

TL;DR instructions:

  • git clone
  • create a start file
  • edit env vars as explained in the guide
  • bash start

If you have a solid Mainnet Archive Node and a solid Rinkeby Full node, and you don’t have any external port conflicts, it should work right out of the box.

We are planning to offer free access to multiple TurboGETH Mainnet Archive nodes and GETH Rinkeby Full nodes for the purpose of the Testnet only. More details will be released when they’re ready. :slight_smile:

Ethereum Archive Node Specs

Minimum Specs Recommended Specs Maxed out Specs
CPUs 16 vcore 32 vcore 64 vcore
RAM 32 GB 64 GB 128 GB
Storage 1.5 TB SATA SSD 7 TB NVME 7 TB NVME RAID 10

Note: The 1.5 TB requirement for storage is the absolute minimum, it needs to be at least SATA SSD as it doesn’t work with spinning disks. Also, only TurboGETH has that little space required. OE, Parity and GETH all take 7 TB at the very minimum, and expanding pretty fast.

Archive node options

Self-hosted Trace API Stable EIP-1898 Min Disk Size
OpenEthereum 3.0.x yes :heavy_check_mark: no :warning: yes :heavy_check_mark: 7 TB
OpenEthereum 3.1 yes :heavy_check_mark: no :warning: no :skull_and_crossbones: 7 TB
Parity 2.5.13 yes :heavy_check_mark: yes :heavy_check_mark: no :skull_and_crossbones: 7 TB
GETH no :warning: yes :heavy_check_mark: yes :heavy_check_mark: 7 TB
TurboGETH no :warning: no :warning: yes :heavy_check_mark: 1.5 TB
Service Providers (WIP)
Infura
Alchemy
ChainStack
Quiknode
Ankr

Graph Protocol Infrastructure Specs

Minimum Specs Recommended Specs Maxed out Specs
CPU 16 vcore 64 vcore 128 vcore
RAM 32 GB 128 GB 256/512 GB
Storage 300 GB SATA SSD 2 TB NVME 4 TB NVME RAID 10

The specs/requirements listed here come from our own experience during the testnet.

Your mileage may vary, so take this with a grain of salt and be ready to upgrade. :slight_smile:

  • The minimum specs will definitely get you running, but not for long, assuming you want to serve data for more than a few heavy-weight subgraphs in the future.
  • The recommended specs are a good setup for those that want to dip more than their feet in the indexing waters. Can serve a decent number of subgraphs, but it’s limited by the CPU if too many requests flow through.
  • The maxed out specs rule of thumb is basically more is better. More CPUs, more RAM, faster disks. There is never enough. IT…NEEDS…MORE!!!11

Closing note, regarding the specs mentioned above: ideally, they need to scale up proportional with your stake in the protocol.

Prerequisites

  • Docker Engine >= 1.10.0
  • Docker Compose >= 1.6.0
  • git

On a fresh Ubuntu server login via ssh and execute the following commands:

apt update -y
apt install docker.io docker-compose httpie git

Install from scratch

Run the following commands to clone the repository and set everything up:

git clone git@github.com:StakeSquid/graphprotocol-testnet-docker.git 
cd graphprotocol-testnet-docker 
git submodule init 
git submodule update 
git config user.email "you@example.com" 
git config user.name "Example User" 
git branch --set-upstream-to=origin/master

Install or Update the Agora and Qlog modules

To update those repos to the latest version just do the following command occasionally.

git submodule update

To use qlog or agora execute the runqlog or runagora scripts in the root of the repository.

./runagora --help
./runqlog --help

This will use the compiled qlog tool and extract queries since yesterday or 5 hours ago and store them to the query-logs folder.

./extract_queries_since yesterday 
./extract_queries_since "5 hours ago"

To make journald logs persistent across restarts you need to create a folder for the logs to store in like this:

mkdir -p /var/log/journal

That’s all.

Get a domain

To enable SSL on your host you should get a domain.

You can use any domain and any regsitrar that allowes you to edit DNS records to point subdomains to your IP address.

For a free option go to myFreenom and find a free domain name. Create a account and complete the registration.

In the last step choose “use dns” and enter the IP address of your server. You can choose up to 12 months for free.

Under “Service > My Domains > Manage Domain > Manage Freenom DNS” you can add more subdomains later.

Create 3 subdomains, named as follows:

index.domain.whatever
query.domain.whatever
dashboard.domain.whatever

Create a mnemonic

You need a wallet with a seed phrase that is registered as your operator wallet. This wallet will be the one that makes transactions on behalf of your main wallet (which holds and stakes the GRT). The operator wallet has limited functionality, and it’s recommended to be used for security reasons.

You need a 12-word, or 15-word mnemonic phrase in order for it to work.

To make yourself a mnemonic eth wallet you can go to this website and just press generate. You get a seed phrase in the input field labeled BIP39 Mnemonic. Scroll down a bit and find the select field labeled Coin. Select ETH as network in the dropdown and you find your address, public key and private key in the first row of the table if you scroll down the page in the section with the heading “Derived Addresses”. You can import the wallet using the private key into Metamask.

Run

In the root of the repo, create a file called start.sh and insert the following lines in it:

EMAIL=my@email \
INDEX_HOST=index.domain.whatever \
QUERY_HOST=query.domain.whatever \
GRAFANA_HOST=dashboard.domain.whatever \
ADMIN_USER=admin \
ADMIN_PASSWORD=change_me \
DB_HOST=postgres \
AGENT_DB_HOST=postgres2 \
DB_USER=user \
DB_PASS=db-password \
AGENT_DB_NAME=indexer-agent \
GRAPH_NODE_DB_NAME=graph-node \
MAINNET_RPC="mainnet:<ETH_RPC_URL>" \
RINKEBY_RPC="<infura url>" \
OPERATOR_SEED_PHRASE="<12 or 15 word private key seed phrase>" \
STAKING_WALLET_ADDRESS=<0x...> \
GEO_COORDINATES="<longitude latitude>" \
docker-compose up -d --remove-orphans --build $@

To start the software, just do bash start.sh

EMAIL is only used as contact to create SSL certificates. Usually it doesn’t receive any emails but is required by the certificate issuer.

QUERY_HOST, INDEX_HOST and GRAFANA_HOST should point to the subdomains created earlier.

ADMIN_USER and ADMIN_PASSWORD will be used by Grafana, Prometheus and AlertManager.

DB_HOST - do not modify this unless you’re running native (non-dockerized) PostgreSQL.

AGENT_DB_HOST - do not modify this unless you’re running native (non-dockerized) PostgreSQL.

DB_USER and DB_PASS will be used for initializing the PostgreSQL Databases (both index/query DB and indexer agent/service DB).

AGENT_DB_NAME is the name of the database used by the Indexer agent/service nodes.

GRAPH_NODE_DB_NAME is the name of the database used by the Index/Query nodes.

MAINNET_RPC should be your Mainnet Ethereum Archive node endpoint. This will be used by the index/query node ingestors.

RINKEBY_RPC should be your Rinkeby RPC - can be a fast/full node, can also be an infura account.

OPERATOR_SEED_PHRASE should belong to the operator wallet mnemonic phrase.

STAKING_WALLET_ADDRESS needs to be the address that you staked your GRT with.

To find out the GEO_COORDINATES you can search for an ip location website and check your server exact coordinates.

In case something goes wrong try to add --force-recreate at the end of the command, eg.: bash start.sh --force-recreate.

Containers:

  • Graph Node (query node) https://query.mydomain.tk
  • Graph Node (index node) https://index.mydomain.tk
  • Indexer Agent
  • Indexer Service
  • Indexer CLI
  • Postgres Database for the index/query nodes
  • Postgres Database for the agent/service nodes
  • Prometheus (metrics database) http://<host-ip>:9090
  • Prometheus-Pushgateway (push acceptor for ephemeral and batch jobs) http://<host-ip>:9091
  • AlertManager (alerts management) http://<host-ip>:9093
  • Grafana (visualize metrics) http://<host-ip>:3000
  • NodeExporter (host metrics collector)
  • cAdvisor (containers metrics collector)
  • Caddy (reverse proxy and basic auth provider for prometheus and alertmanager)

Updates and Upgrades

The general procedure is the following:

cd <project-folder>
git pull

This will update the scripts from the repository.

To upgrade the containers:

bash start --force-recreate <container-name>

To update Agora or Qlog repos to the latest version just do the following command occasionally:

git submodule update

To use qlog or agora execute the runqlog or runagora scripts in the root of the repository.

./runagora --help
./runqlog --help

Managing Allocations

  • To control the allocations, we will use the indexer-cli
  • To check our allocations, we will use the indexer-cli, other community-made tools, and later on, the Graph Explorer

How to check your allocations

Assuming you already have the graph-cli and indexer-cli installed, in the root of the directory, type:

./shell cli
graph indexer rules get all

You will be greeted by this table. By default, it’s set to these values below:

1422×117 4.92 KB

What do the table columns mean?

  • deployment - can be either global, or an IPFS hash of a subgraph of your choice
  • allocationAmount - refers to the GRT allocation that you want to set, either globally or for a specific subgraph, depending on your preference
  • parallelAllocations - influence how many state channels the gateways open with you for a deployment, and this in turn affects the max query request rate you can receive
  • minSignal - conditional decision basis ruled by the minimum Subgraph Signal
  • maxSignal - conditional decision basis ruled by the maximum Subgraph Signal
  • minStake - conditional decision basis ruled by the minimum Subgraph Stake
  • minAverageQueryFees - conditional decision basis ruled by the minimum average of query fees
  • decisionBasis - dictates the behavior of your rules

General caveats

  • Your total stake of a specific subgraph, or globally, will be calculated as follows:

allocationAmount x parallelAllocations = totalStake

Example:

allocationAmount 100 x parallelAllocations 5 = 500 GRT allocated

  • decisionBasis can be of three types: always , never and rules

decisionBasis always overrides the conditional decision basis rules that you might have set (minStake, minSignal, etc) and will ensure that your allocation is always active

decisionBasis never same as above, only that it will ensure that your allocation is always inactive

decisionBasis rules will give you the option of using the conditional decision basis

  • global rules will have, by default, an allocationAmount of 0.01 GRT and parallelAllocations set to 2

This means that by default, every time you set an allocationAmount of a specific subgraph, it will inherit parallelAllocations 2 rule from global.

To see the global rules merged into the rest of your allocations table, you can use the following command:

graph indexer rules get all --merged

How to set allocations?

Examples:

1.

graph indexer rules set {IPFS_HASH} allocationAmount 1000 decisionBasis always

Assuming the {IPFS_HASH} exists on-chain, this will set an allocationAmount of 1000 GRT and will ensure that the subgraph will always be allocated, through decisionBasis always

2.

graph indexer rules set global allocationAmount 100 parallelAllocations 10 decisionBasis always

This command will enable you to automatically allocate 100 GRT x 10 parallelAllocations to all the subgraphs that exist on-chain, with 10 parallel allocations each

3.

graph indexer rules set {IPFS_HASH} allocationAmount 1337 parallelAllocations 5 minSignal 100 maxSignal 200 decisionBasis rules

This command will enable you to use the decision basis conditional rules of minSignal and maxSignal. The subgraph will only get allocated by the Agent IF the network participants have a minimum of 100 signal strength and a maximum of 200 signal strength.

Generally speaking, you’ll be good to just use either the first command or the second one, as they’re not complicated to understand. Just be aware of the parallelAllocations number.

How to verify your allocations?

We can use the following command(s) with the indexer-cli:

This will only display the subgraph-specific allocation rules

graph indexer rules get {IPFS_HASH}

This will display the full rules table

graph indexer rules get all

This will display the full rules table with the global values merged

graph indexer rules get all --merged

What happens after you set your allocations?

The indexer-agent will now start to allocate the amount of GRT that you specified for each subgraph that it finds to be present on-chain.

Depending on how many subgraphs you allocated towards, it will take time for this action to finish.

Keep in mind that the indexer-agent once given the instructions to allocate, it will throw everything in a queue of transactions that you will not be able to close. For example, if you set global always then immediately after, you decide to set global rules or never it will do a full set of transactions for global always then go around and deallocate from them with your second transactions. This means that you will likely be facing a lot of delay between the input time and until the actions have finalized on-chain.

A workaround for this is to restart the indexer-agent app/container, as this will reset his internal queue managing system and start with the most fresh data that it has.

Another workaround is to either delete your rules with graph indexer rules delete {IPFS}

Managing cost models

What are cost models?

Cost models are indexer tools that they can use in order to set a price for the data that they serve.

You cannot earn GRT for the queries that you serve without a cost model.

The cost models are denominated in decimal GRT.

Cost models can have two parts:

  • The model — should contain the queries that you want to price
  • The variables — should contain the variables that the queries use

You can either have a static, simple cost model, or you can dive into complicated cost models based on your database access times for different queries that you serve across different subgraphs, etc.

The decision here is totally up to you :slightly_smiling_face:

How does a model look like?

The easiest cost model you can set, can look something like this:

default => price;

or 

query {...} => price;

Example — you’re serving every query at 0.01 GRT / query

default => 0.01;

How do the variables look like?

{                              
  "VALUE-1": "10.0006390502074853",
  "VALUE-2": "5",                 
  "VALUE-3": "3",                
  "VALUE-4": "1"               
}

Debugging and troubleshooting

First off, I strongly recommend having either graph-pino or pino-pretty installed for this, along with Grafana or something else to collect the data scraped by Prometheus and be easily accessible at a glance.

You will need NPM installed for this.

sudo apt install npm -y

To install graph-pino, you can simply do:

npm install -g --registry <https://registry.npmjs.org/> @graphprotocol/graph-pino

To install pino-pretty, you can simply do:

npm install -g pino-pretty

And you’re done. These two will greatly help you read through the indexer-agent and indexer-service logs to easily find errors and warning messages that might occur.

To use them, you can use the following examples:

Graph-pino with Docker

docker logs indexer-agent --tail 10 -f | graph-pino

Graph-pino with Journald

journalctl -fu indexer-agent -n 10 -f | graph-pino

Pino-pretty with Docker

docker logs indexer-agent --tail 10 -f | pino-pretty -c -t

Pino-pretty with Journald

journalctl -fu indexer-agent -n 10 -f | pino-pretty -c -t

Troubleshooting scenarios

  • Indexer-agent and Indexer-service containers are loop crashing

    This is an indication that your index-node cannot connect to either the PostgreSQL DB or your Ethereum Archive-node

    I mentioned this in discord, but worth mentioning it here as well:

    If, at any given time while you’re starting up your dockerized/k8s’ed infrastructure, your INDEX-NODE can’t connect to the ETH-ARCHIVE, your agent and service nodes will crash in a loop saying that they can’t reach the index-node, but in reality, it’s your index-node that can’t reach the archive-node. There is no mention in the agent/service logs saying that they crash because of that. Only a connection refused error, usually port 8030. And there is no mention in the index-node either that it can’t connect to the archive node, only that it stalls when connecting, with no timeout. I spent 6 hours trying to fix this, I thought it was an UFW issue blocking my docker containers, when it reality it was what I explained above.

  • You have allocated, but can’t see the subgraph indexing in Grafana?

    Check the indexer-agent logs immediately, the agent is probably still about to send the allocation transaction, but who knows ¯\_(ツ)_/¯

  • You have allocated, you can see the subgraph indexing, but its blocks behind number doesnt go down

    Check your archive-node logs, and if everything looks good, check your index-node logs after

  • Docker - Nginx load balanced indexer-service nodes are not getting queries or only part of them are

    When you compose-up, some of your containers, even if you set the correct order of container dependencies, will crash at start-up waiting for others to start first. By design, Nginx caches the internal Docker IPs at start-up. In order to mitigate that problem, add a script in the Nginx start sequence that executes the following command: nginx -s reload.

    Alternatively, you can just manually do docker exec -it nginx-loadbalancer nginx -s reload right from the host machine.

  • Other indexer-agent errors documentation from the logs can be found here:
    indexer/errors.md at main ¡ graphprotocol/indexer ¡ GitHub

  • You’re getting Could not find matching rule with non-null allocation

    This means that one of your rules has allocationAmount null — usually I’ve seen this being the global rule missing a value.

    To get rid of it, set graph indexer rules set global allocationAmount 0.01

14 Likes
2

This is a very understandable guide. Thank you so much. I give you the high five.

moon
moon1200×630 183 KB

2 Likes
3

That is sooo cooool :open_mouth: Do you have a high resolution version anywhere?
I want to set it as my wallpaper :laughing:

4

I can redo it in high resolution and send it tomorrow. :wink:

1 Like
5

That would be fantastic! :raised_hands:
Btw, I sent you a message in Discord.