We can run The Graph Indexer in a different ways. One of them can be a Public Cloud Provider like Amazon Web Services.
Description
- We run The Graph Indexer in AWS
- We use Kubernetes based on AWS EKS to manage all applications
- We run 4 Index node with ~ 5 subgraphs/node
- We run 3 Indexer services with autoscaling enabled to handle traffic spikes
- We run 2 Query nodes with autoscaling enabled to handle traffic spikes
- We indexing for about 30 subgraphs simultaneously
- We use Amazon Aurora PostgreSQL to store graph DB and its size is ~ 2 TB
- We use 2 Erigon nodes with EBS volumes and behind ALB for high availability and traffic balancing
- We use Fluent Bit to send all the logs from Kubernetes for OpenSearch
- We use Amazon OpenSearch to send alarms on errors from the application logs
- We use Amazon CloudWatch to send alarms for Amazon resources
AWS Services and costs
A list of Amazon Web Services and their costs to run Indexer in AWS Cloud can be the following
| # | AWS Service | Purpose | Usage | Cost, USD/m |
|---|---|---|---|---|
| 1 | Amazon Route 53 | DNS zone | 1 Zone, 1 mln/m standard queries | 0.90 |
| 2 | Amazon Virtual Private Cloud) | Networking | 1 NAT Gw, Data 1 TB/m, AZ 15 TB/m, Out 1 TB/m | 478.29 |
| 3 | Amazon Application Load Balancer | Kubernetes Ingress, Erigon LB | 2 ALB, 200 c/s, 200 r/s | 126.29 |
| 4 | Amazon Certificate Manager | ALB SSL certificate | 1 Cert | 0.00 |
| 5 | Amazon Elastic Kubernetes Service | Kubernetes Cluster | 1 EKS Cluster | 73.00 |
| 6 | Amazon Aurora | PostgreSQL DB | 1 db.r6g.2xlarge, Single-AZ, 2 TB, 20 mln/h IOPS | 3,882.54 |
| 7 | Amazon OpenSearch Service | OpenSearch | 3 m5.large.search, 200 GiB GP2 | 391.98 |
| 8 | Amazon Web Application Firewall | Web Application Firewall | 1 ACL, 3 rules, 10 mln requests | 14.00 |
| 9 | Amazon EC2 | Kubernetes Cluster nodes | 3 r6g.xlarge, 30 GiB GP3 | 3 x 149.57 |
| Erigon node | 2 c6i.4xlarge, 30 GiB GP3 | 2 x 498.80 | ||
| 10 | Amazon Elastic Block Store | Erigon node disk | 2 EBS 3000 GiB/ GP3/16000 IOPS/ 1000 MiB/s | 2 x 340.00 |
| 11 | Amazon Secrets Manager | EC2 instances secrets | 1 secret | 0.40 |
| 12 | Amazon Cognito | OpenSearch authentication | 100 MAUs | 0.00 |
| 13 | Amazon CloudWatch | Metrics, alarms | 100 Metrics, 20 Standard alarms, 10 GB ingested | 37.05 |
| 14 | Amazon IAM | Roles, policies | Roles, policies, identity providers | 0.00 |
| TOTAL | 7130.76 |
Note: All costs calculated for Ohio region and for On-Demand Instances
You can access estimate calculated in AWS Pricing Calculator via following link - The Graph Indexer - AWS Cloud (Public share links expire after 3 years).
Costs changes
Resulting cost may change mainly because of the following
- Network in/out data processed in VPC
- Network traffic processed by ALB
- PostgreSQL DB size and IOPS
- PostgreSQL DB snapshots
- Erigon nodes EBS volumes size snapshots
Costs optimizations
We can consider the following to optimize costs
- Use Amazon Aurora Serverless v2 with tunned autoscaling to optimize DB resources usage.
- Consider to use Reserved instances.
- Consider to use Erigon LB inside Kubernetes Cluster if you don’t need access to nodes from external resources.
Known issues
- Erigon nodes initial sync on EBS is very slow and may take up to 20 days