Testing Best Practices

In the absence of true Unit Testing I need to develop a more thorough testing framework for my subgraphs.

The process seems error prone and confusing to say the least and I’m curious to know what other people are doing or thoughts on my current plan. @Brendan @amxx

  1. Deploy Local Node
  2. Deploy Smart contracts to local Ganache
  3. Use template engine to create subgraph.yml that matches deployed local contracts
    3a. Programmatically deploy new subgraph
  4. Run tests that connect to deployed contracts
  5. Tests call smart contracts, AND make GraphQL queries, and then run expect on parsed GraphQL query results.

The part that makes this such a mess is the 2-3. Because normally you would want your tests to be unit tests, so we would want to break down the setup each time. But in this case, I’m not sure what is best.

Do I somehow script a Graph Docker container to run in each “beforeAll” so that I can deploy fresh contracts?

Do I run the graph Docker container once, and then in each before each, I generate a new template yml for the graph, and then programmatically deploy that to the local node, and then test it?

And seems like I’d spend a heck of a lot more time debugging running subprocesses from Node, than I would be writing tests directly.

All the possibilities seem brittle, ugly and error-prone.

3 Likes

To make the setup procedure of what you’re describing easier, we wrote graph test and released it some time ago. See the releases notes for graph-cli 0.18.0 here: Release v0.18.0 · graphprotocol/graph-cli · GitHub.

We’ve been using it for graph-node integration tests (details here) and it’s been very helpful.

But

  1. there’s still too much boilerplate code to write in the tests (a library could help with patching subgraphs, deploying them, waiting for certain blocks to be processed etc.)

  2. it’s not super fast.

I know that @fubhy has some ideas on how to improve this to provide a better testing experience. They’ve put a lot of effort into their subgraph development environment at Enzyme.

1 Like

Thanks for this @jannis, this is a great starting point for many of the points above. I’d also be interested to hear from @fubhy on what they are doing. :slight_smile:

Thanks!

1 Like

What we are doing is not far away from what you guys are describing.

Technology involved: Docker, Jest, Hardhat (forked version until this is resolved: feat: set relative block timestamp based on forked block by fubhy · Pull Request #979 · nomiclabs/hardhat · GitHub), GitHub Actions

About Hardhat: We run a local Hardhat development stack (mainnet fork) inside of Jest. We are not using Hardhat as a “task runner” because other tools are much more capable (e.g. the Jest test runner) and solve things like parallelization (better cpu core utilization, etc.) very nicely. So consider our Jest + Hardhat stack a custom breed where we only use the Hardhat EVM by directly importing lower-level functions from their npm package: We don’t use the HRE (Hardhat Runtime Environment) or any of the other globals that are exposed by their package. I’ve brought this up with the Hardhat maintainers in the past but haven’t heard back from them yet. Since that approach is a massive deviation from what they advertise their tool as (a task runner) I don’t think they’d reconsider and instead turn Hardhat into a library first and foremost, but who knows. We also use Jest instead of Mocha and have built our own Jest Matcher & Assymetric Matcher toolbelt (custom assertions for transactions, ethereum primitives, function input/output, etc.). Nonetheless we are very happy with Hardhat. Especially in comparison to Ganache: Stack traces, forking with caching, good performance, nice abstraction around ethereumjs-vm, etc.

About Jest: For some reason, the Ethereum community is quite invested into Mocha. We’ve had much better experience with Jest. Considering that Jest is also the #1 test runner in the rest of the JavaScript community I’m surprised that Ethereum is still locked into Mocha so much. Jest shines especially because of it’s parallelized test runner and the ability to control large parts of the test environment & initialization of the test context. Once you’ve mastered it, the interactive CLI in “watch”-mode is also quite a nice productivity boost (interactively narrow down on the tests suites you want to “watch”, re-run them with ease, prioritize previously failed tests, etc.).

Our integration tests for the subgraph work pretty much the same as laid out by you here:

  1. Boot the docker environment (including a custom Hardhat docker image, postgres, ipfs, graph-node). For testing, we don’t mount any volumes.
  2. Run the test suite

The different now is, that our test suite manages all the rest: Contract deployment, deployment of the subgraph, interacting with the protocol, asserting state (onchain + graphql queries).

For that, we’ve written a few hacky utility functions that

a) deploy our subgraph while replacing handlebars template placeholders with the addresses (and other dynamic values like start block, etc.) from the deployment output
b) wait for the subgraph to be synced

The tests then continue by interacting with the protocol. Custom Jest Matchers then poll the subgraph until it has been synced up to the block of the last relevant transaction receipt and then run assertions against that block by querying the store (using graphql) at that exact block number.

… You see, we’ve experienced all the same pain that you are describing. And yes, it certainly means that there is a lot to do in terms of (subgraph) developer tooling: Unit testing, integration testing, etc.

Integration testing is possible already, but it’s painful.

As Jannis said: We need custom test utils (I’d be happy to eventually open source ours although I am not sure how helpful those would be for the majority of the community as most people seem to be using Mocha) and we need ways to manipulate the graph-node from within the test suite so that it plays well with features like EVM snapshotting (also revert the subgraph to the block # of the snapshot), etc.

2 Likes

That’s really eye-opening to hear how you are doing it. Thanks for sharing!

I’m a Mocha person myself, but that’s mostly because it’s what everyone else is using in the Ecosystem. I’d be down to try Jest. Are you using Waffle at all in your suite?

No we currently don’t use Waffle although they recently added a couple of Jest matchers too! We have our own set of custom matchers. I am happy to share all of that though.

1 Like

Would be happy to see anything you had! :slight_smile:

Hey folks, thanks @fubhy and @jannis I’ve got a system of scripts unit testing working for my subgraph.

It’s not very elegant, and a whole bunch of boilerplate, and very slow, but it works!

@dennison Would you be up for sharing it? I’m curious what the result looks like.

Sure!

Give me a few days or a week to make something I would want to share. Right now it’s inside our repo but I could rebuild it with the stock “gravatar.sol” example that comes with the Graph Repo.

It’s an inconvenient solution but it is a solution until true Unit testing comes around. :slight_smile: Give me a few days and will share!

1 Like

Ian from Uniswap here. Also would love to check this out. We seriously need some better testing in our subgraphs.

2 Likes

Hey folks, quick update

So I updated my machine and the graph node and my test setup no longer works. Will investigate whats changed and report back!

Done!

I built it around Protofire’s Compound Governor Alpha Subgraph. This isn’t the subgraph we use at Tally but it’s darn good and public and I thought it would be great to show how to unit test around a known and popular subgraph.

So, this isn’t a perfect solution, but it’s been GREAT for me so far. I can actually UNIT TEST! Lordy, lordy, how deep its been.

Thank @jannis for the pointer to your repo- that is what definitely set me in the right direction. @ianlapham I think you will find this helpful.

Please let me know if something doesn’t work. I’m not the best at readme’s so I can always be a bit more descriptive I’m sure. I can’t wait till real unit testing is available!

3 Likes

Amazing thanks @dennison - will give this a look because we also use a fork of this specific subgraph for tracking Uniswap governance.

2 Likes

My pleasure, would be interested to hear your feedback, and if you improve it some way would love to see!

I’ve updated the repo, some people reported problems with npm install. It should be fresh now.

1 Like