Sometime in the last day or two, the Governance subgraph for PoolTogether stopped being indexed.
This subgraph is used by our governance app so that users can view and vote on proposals. It is a critical app that is used infrequently.
My guess is that this subgraph stopped being indexed because it didn’t receive enough traffic. When a subgraph stops receiving traffic, it dies. No indexer wants to index a subgraph with zero traffic.
Is my assumption correct? How can we fix this quickly? How can we fix this long-term?
We’re currently re-deploying the graph to the hosted service to at least get it back up and running.
We would pay an indexer directly for simply hosting this graph. Perhaps that what we need.
The issue with the governance subgraph not being picked up by indexers actually has more to do with the curation signal. The PoolTogether v3_1_0 subgraph has around 167.5k GRT in curation signal which is the reason it currently has 26 indexers on it as of writing this. The governance one only has 2.5k curation signal, which leads to lower indexing rewards and therefore less indexers. Query fee volume also plays a role, but right now number of indexers is mostly driven by curation signal (indexing rewards). Usually a subgraph with over 1k GRT in signal will be picked up by at least one indexer, but you would see more indexers if the subgraph had more curation signal on it. Roughly speaking, I believe if the curation signal was closer to ~20k GRT you should reliably get at least 4 indexers based on some of the data I’ve seen on this topic. Hope this helps!
Hmm… so unpopular subgraphs need at least $15k staked on them.
I’m sure we have enough GRT for this subgraph. However, we have a lot of Subgraphs. We won’t be able to cover them all.
The economics make me want to mash all of our subgraphs down into a single, monolithic subgraph so that we can combine signals and have strength in numbers. But it’s sacrificing engineering for the sake of curation. Hmm. Tricky.
Signal is not the only metric indexers keep an eye on but it’s probably the main one. And it could well be that the signal thresholds indexers have currently set is too high for smaller projects/subgraph/curators.
On that note: curators could help boost the signal if they understand the importance of the PoolTogether governance subgraph. @Oliver or @Slimchance should know where to best promote the subgraph to curators.
This makes it significantly easier for indexers to say “ok, this is the PoolTogether governance subgraph and I want to index that and any upgrades to it going forward”. Rather than having to pick out the right deployment IDs all the time.
Every indexer can define their own rules on when to index subgraph deployments published on the network: see Indexer | Graph Docs. Signal is probably the most used criterion.
Hey Brendan, I’m a curator in the network and have published a subgraph as well for the Rocket Pool protocol.
There’s a few issues that were at play here most of which comes down to the low signal on Pool Together Governance.
Indexers often times look to see if a subgraph has self signal (since this typically does not change mid their allocation). Subgraphs with less than 4,000 (based on current trends) will generally have 1-3 indexers. Once you are closer to the 10,000 range you’ll start seeing 3-6 indexers.
Additionally the indexer that allocated to your subgraph largely overallocated to it, which is giving themselves a suboptimal return. This also means that you wont have additional indexers signal after them since subsequent indexers would only make it more disproportional.
Since your normal PoolTogether v3_1_0 subgraph already has a high volume of query traffic, it isn’t quite as necessary to have your self signal here since market curators are incentivized to signal on this by the 10% query fees. On the Pool Together Governance the query fees have been low and should be expected to have less curator attention. This is where you would want to focus more of your self signal.
Agree with the responses above, but will add some color on the economics:
There are a couple of ongoing workstreams that could improve this:
@adamfuller and Graph Node core devs working to allow Data Sources to be more modular/concurrent within a subgraph, as opposed to all modifying the same global state, which would make it slighlty less onerous from an engineering perspective to combine subgraphs. Once this is in place, we could even make it easier for Indexers to intelligently “diff” subgraphs, so that when only one Data Source is updated, they don’t necessarily have to resync the others from scratch. Of course, there are other reasons you may still want to break a subgraphs logic into multiple small subgraphs, the most salient of which is probably if that subgraph make sense as a standalone building block to be composed/queries by other developers… this brings me to my next couple points.
There is some work ongoing identifying the best way to allow a GNS bonding curve to split signal across multiple core protocol bonding curves. This doesn’t address your core problem per se, though it does make it a lot easier from a UX perspective to manage a portfolio of signal across multiple subgraphs. Of course, doing this in a way that doesn’t blow up gas costs is the hard part, and this might be something that we need to save for L2.
This is a problem we’ve observed elsewhere as well, for example @chris anecdotally described this as the issue that the Synthetix subgraph fell behind the chainhead several months ago, even though there were other Indexers online that could have allocated on it, had it been profitable to do so. A few things to note here:
Moving to L2 should make this problem a lot less salient, as gas costs of opening/closing allocations are the largest cost associated with allocating on a new subgraph.
We’re exploring mechanisms which incentivize Indexers to “leave more space” for other Indexers on a given subgraph, however, such a mechanism is also likely to increase total allocation management gas costs in the network, because it would increase the average # of simultaneous allocations per subgraph. We hope to have more to share on this soon.
Hope this additional color is helpful, and thanks for raising the flag on this!
Thanks everyone for your thoughts and info! It’s been illuminating. Still, I have a wish list: one day I’d love to have a simple way to spend GRT to have a subgraph hosted.
Currently, it’s much easier, than it was 2-3 months ago. You couldn’t be frontrunned by BOT when you publish your subgraph, you can publish and signal on it at the same time. Also with the current GRT price 4k GRT it’s almost nothing to have a fault-tolerant data infrastructure for your needs. Especially with the understanding that it’s something like a deposit, not payment.
And I agree with other topic participants, if a developer is quite active and works with network participants, it’s really easy to solve any issue.
We, as p2p.org, for example, are ready to Index some important subgraphs even for free, if their founders really can’t find 4k GRT, because they are early-stage startups or non-profit organizations and so on. But they should ask about it because we can’t read minds (yet :D)
In addition to what @KonstantinRM said, there are proposals incoming that should enable this. The first step is to remove principle risk at the core protocol level bonding curves for subgraph developers. Once that is in place, it’s possible to get creative extra-protocol, for example zero-risk loans to subgraph developers to use as signal, for a nominal rate that feels a lot like a monthly SaaS payment. Stay tuned!
Hey @Brendan, just looping back on this. The first part of what is needed to support the above subscription experience has been published as a proposal here: GIP: Principal-Protected Bonding Curves
What I’m calling “signal renting” is to follow as a separate proposal. Feedback is much appreciated!