Hello! I wanted to provide an update on this, Limechain have been working on this as Brendan mentioned. There were some iterations in terms of approach (many thanks to @fubhy and @Leo for providing guidance & feedback), and we are now at the point where it would be great to get some wider feedback on the framework and developer experience. Please do respond this thread, or DM me here or on Discord if you would be interested!
6 Likes
Fantastic! We’d love to give it a try. I’ll dm you
2 Likes
Further update - please see this Loom video for a demo of the progress so far on Subtest, by Petko from Limechain. The repository is available at Subtest - this is not currently ready for usage, but includes documentation on the overall approach and the user stories from the initial scoping document. It would be great to get feedback on the approach and implementation to inform the ongoing work!
4 Likes
This is really cool! I have a couple of questions:
- How difficult would it be to generalize the approach here to other projects using AssemblyScript (i.e. Near), in order to elicit a broader set of open-source contributions?
- How does the approach here differ, if at all, with prior work? (i.e. as-pect which is what NEAR links to in their docs as a testing framework).
3 Likes
Petko from Limechain here 
Thanks Brandon! We appreciate all the feedback we can get on this one. I’ll try to answer your questions given the way I see things:
- This framework is very specific to solving testing issues in The Graph protocol, in the sense that it’s mimicking the graph-node repository. I genuinely think
as-pectdoes an amazing job at encapsulated, general-purpose unit testing. - Again,
as-pectis great for unit testing. Subtest targets scenarios specific to The Graph, and more precisely - event handler functions and the store. It spins up a mock store and makes assertions on it. That’s it. It doesn’t add a whole lot of new testing capabilities in general to AS. That would be its own project.
Thanks again to you and everyone else for the feedback 
4 Likes
Hi Petko! I watched the Loom video and really like what I saw. The interface is exactly as I imagined in terms of setting up the store, running handlers, and asserting the results. Great job!
I have a couple of questions:
- Why Rust? It’s an interesting choice given that everything is written in AssemblyScript. I’m very curious to hear the rationale behind that decision.
- Will you have different “reporters”, like as-pect reporters? It’ll make this tool much easier to plug into others later. If it adheres to
as-pectperhaps we could even plug it into existing tools.
2 Likes
Hey Brendan, thanks a ton for the feedback, it really means a lot! 
Also - great questions! Here goes:
-
Just to reiterate, the
Subtestframework is split in two parts - the npm module for state methods and helper functions, which is what the subgraph developers will be importing and using in their test files. It’s written in AssemblyScript, because that way we can easily pass data around between the subgraph and the testing library, and also because subgraph devs are already used to AS. The second part is the Rust wrapper, which is used to inject all of the other host functions - type conversions, logging, etc. It’s also what will be used to run the tests and to log their results. Initially we tried to have that bit written in TypeScript, because it would be faster to develop, but we hit a snag, because that would’ve meant we’d need to rewrite all of the type conversions and the other host functions to TS (They are implemented in Rust in the graph-node repo). We figured we could try to port them from Rust to AS, but still that wasn’t any easier (it was actually worse in terms of effort). So we settled on just using them directly as an import in a Rust project. The reason was mostly practical. Along with that, there is a bit of a personal preference involved, since I am an avid Rust fan. And also Rust is a more mature language with a wider ecosystem of tools. Furthermore, we discovered that when importing straight from graph-node, we get a lot of other things for free - methods for calling functions from the WASM module, wrapper types for all sorts of WASM-related data, etc. Got a bit derailed here, but I hope that answers your question. 
-
To be honest, I haven’t looked at reporters in
as-pectmuch, but they seem very useful and a great addition to the framework in a post-MVP stage (here’s the MVP board for reference).
3 Likes
Interesting! I’m 100% on-board with code reuse. Perhaps down the road the community will create a TypeScript toolchain, but clearly it’s not worth the effort right now. Thank you for the explanation!
One more question: if a test fails does the test runner return a non-zero exit code? That would be a must have for the MVP! Our CI system would appreciate it
1 Like
No worries! A TypeScript version of the wrapper would indeed be very interesting, and I’d be up for the challenge, downside is it’s going to take a lot longer to build. But if it’s what the community wants it’s what the community wants.
Again, awesome question! Currently it’s not doing that but it’s a breeze to plug it in, as you pointed out, it’ll be very valuable feedback.
3 Likes
Hey guys! Great work so far on the framework. Took it for a test spin (even though I know its not ready yet) and ran into a snag. It is common in mappings to check and see if an entity exists by calling Entity.load() and checking if null. When trying to do this with the mock store, the load() function throws an error rather than returning null if the entity doesn’t exist.
I looked at the issues and project board and didn’t see this mentioned explicitly so i thought I would bring it up. Otherwise this is a fantastic development and way more smooth than our old testing implementation
1 Like
Thank you! That’s actually a pretty good catch and we will look into it ASAP. This is precisely why we appreciate feedback from subgraph devs!
Looks like that part is working great now!
But it looks like you changed the way that calling a subtest works. Before you passed in a path to the wasm file but now you just pass the datasouce name. This is easier but it doesn’t seem to work for datasouces that come from templates. When I try to call subtest TEMPLATE_NAME
2 Likes
I get the following error:
It should be the same as the 'name' field in the subgraph.yaml file, corresponding to the datasource you want to test.: Os { code: 21, kind: Other, message: "Is a directory" }'
1 Like
FIx is in 
thanks a ton again for the feedback
2 Likes
Since we’re at a stage where it would be awesome to settle on a name for the framework, in a truly decentralised fashion, I’m kindly asking everyone to cast their vote here, thanks!
UPDATE:
Just to quickly follow up on this:
Version 0.1.0 (MVP) of Matchstick is now live! 
You can check out the installation guide and docs here.
If you have any questions, feature requests, general feedback or simply want to reach out - feel free to do so on the newly created #matchstick-early-testers channel on The Graph Discord.
We’re now moving forward with MVP improvements for Matchstick & also starting work on a simple subgraph integration testing tool (more info on that can be found on this forum thread).
3 Likes
Hi @dennison
Have you used limechain for your contracts for testing subgraph ? we also have a pretty big subgraph and would love to see how you handle different/complicated scenarios.