Pre-Proposal: Decentralizing Network Stake via Parameter Revision(s)

Governance & GIPs
31

The fact that the network isn’t fully functional yet makes it more difficult for smaller indexers to stand out. We can hope that the situation will change once we get there, but as a smallish indexer I’m getting concerned that the position of the large indexers is becoming entrenched, and as such these discussions are necessary.

We have to keep in mind that putting a hard cap or putting the delegation ratio on a curve would only incentivize indexers to create a multitude of smaller indexers. From a decentralization point of view, this isn’t desirable if these indexers are all managed by the same team, on the same infra. Some indexers are already doing this, and they probably have good technical or financial reasons to do so, but we should be careful not to incentivize it further. Sybil resistance is hard.

I’m afraid we’ll have to resort to “soft” measures, like educating network participants and maybe reviewing the way we display indexer performance data so delegators can make “better” choices for the sake of the network. Sorting by allocation or stake may not be optimal in this regard, even though is it probably the only metric that is impossible to game.

4 Likes
32

Good point. This needs to be considered as part of modelling a change; the mechanics for large delegators already over the cap and how they might behave should such a hard cap come into place.

I also think it’s important that ideas proposed don’t throw the balance of decentralization in the exact opposite direction. We need large indexers to be viable businesses just as much as we need smaller indexers to be viable businesses.

5 Likes
33

It’s worth mentioning that the Indexer selection algorithm heavily relies on the economic security that a particular Indexer provides, so by default bigger indexers (bigger stakes) get more queries.

As mentioned 3 top indexers can accommodate 50% of the total stake which means, that they could get tons of queries, and if they serve them well, small indexers won’t be able to get even a fraction of the network queries.

Also if we imagine that the top 3 Indexers serve 50% of the network’s queries and if even one of them fails, traffic will flow to smaller indexers which infrastructure can be congested because they weren’t ready for such a spike.

Introducing some limits to the total stake that any single indexer can have will let us have a more even distribution of queries with a more predictable amount of queries without enormous spikes.
It will be much better for the decentralization of the network if big whales will scale horizontally and not vertically.

7 Likes
34

The main issue isn’t that big indexers have an advantage, that’s a given, if you have more at stake it’s expected to have an edge over the competition. Big indexers will have enough headroom to be able to withstand costs with no income (like the current situation where basically all indexers can’t really pay the bills with the rewards because we can’t extract them without stopping for a month, currently being addressed in another proposal).

The main issue is that a single entity could potentially hold 100% of the current delegation pool, and have room to spare. That’s how unbalanced GRT bags currently are for indexers.

The top 2 biggest indexers (by own stake) can currently accept almost 5B GRT in delegations (50% of the total supply). Considering that the total supply will most likely not be 100% staked, since a part of it will be circulating in markets, used to pay queries, etc, this means that with the current delegation ration parameter, you could end up with a 2 node network.

At that point the benefit of having the decentralized network fades, and you would be much better of just having The Graph as a SaaS while avoiding having on-chain expenses.
As we know, scaling single nodes to be able to handle the traffic isn’t easy, that’s why having tons of nodes gives you the scalability needed to be able to process the volumes that the network is supposed to handle.

Now, the original idea discussed in the “Office hours” call isn’t to make low-stake indexers the best solution, the idea is to manage the extremes which currently can potentially have a big impact in network centralization (and thus, because of the current numbers mentioned above, making a network completely pointless).

When I say manage I mean put soft roadblocks so that being able to control 50%+ of the network isn’t just as easy as spin a node, put a ton of money, set fees that nobody else can compete with and drive them out of the game.

Just to exemplify this, these are some stats for the top 3 indexers by own stake:
image

#1 is giving away 81.7% of their own rewards to delegators, and has a 2.24B delegation cap.
With their own 140M stake, only taking 18% of their own rewards, it still yields them 8.7K GRT per day in rewards right now.

image
image886×89 12 KB

This means they could potentially do this indefinitely to drive out competition, gas costs associated for a single node aren’t that high to avoid this.

Now, only 7 indexers have 10M+ GRT as their own stake (5 in the 10-35M range, 2 in the 100M+ range), everybody else has ~4M or less (which accounts for 95%+ of the indexers in the network).

Now you are in a situation where predatory pricing can become an issue, which in the non-crypto world is handled by laws for a reason.

One way to avoid having a single entity potentially handling 100% of the delegation pool would be to simply reduce the delegation ration (currently 16x) so that they would be able to accept less delegation, and thus, a part of it would most likely flow to other indexers to avoid overdelegation of any node, and thus make it more decentralized. The problem with only reducing the delegation ratio is that it affects everybody equally (which is actually a good thing, from some perspective), but would also mean that low-end indexers might not make enough profit to sustain the infrastructure and maintenance costs.

To prevent that, you would have 2 choices:

  • Raise the bar for the minimum indexer stake (which, depending on how much you need to raise it, might end up directly prohibiting 90-95% of current indexers from even entering the market)
  • Set a soft cap on indexing, so that it starts to hit diminishing returns after a certain point.

Having 90-95% of the current network completely prohibited from even entering the market doesn’t sound like a great idea, since you still end up with some degree of centralization (albeit not as bad as having a single entity, or two, run the whole network).

Having a soft cap seems to be the best idea, since you would still be getting more and more profit as you stake more and more money, but you would probably hit a point where it stops making sense to spin up more nodes (because of the associated gas costs), and you would rather delegate a part of your already gigantic stake to other indexers.

You would still be making more money than everyone else in the network, and will still have the biggest share of the market, without any risks to the network health.

Having said all that, we still need to create a model where this all makes sense and isn’t too taxing on big indexers, since we still want to have reasons to invest, but I don’t think it’s in the best interest for anyone that a single entity holds the entire network stake, and thus makes the whole network essentially pointless.

9 Likes
35

Gary, thank you for writing this detailed and well thought out discussion starter. I share your concerns. The risks of the network becoming centralized in the hands of a handful of large Indexers can’t be denied. If control is accumulated by a few, these large Indexers could meet agreements (for instance about reward cuts and query fee cuts) among themselves to the detriment of the entire network and all its participants.

In theory, all market participants benefit from a strong, decentralized network. It would be only in the interest of the “Giga-Indexers” to drive smaller indexers (i.e., their competitors) out of the market. Everyone else in the network benefits greatly from a decentralized ecosystem in which there is a multitude of Indexers to choose from that engage in a healthy competition with each other.

To me as a Delegator, centralization would mean that I would become dependent on the reward structure the few Giga-Indexers dictate. Isn’t this exactly why we all are here, united in our vision to see a decentralized Web3 ecosystem flourish? As you already said Gary, what we need to find is the right balance in which the network can thrive.

So thanks for raising these concerns and starting a debate on this topic.

7 Likes
36

My continued thoughts, which are in no particular order.

  1. The single biggest issue to me right now is that we had this network set up for months with the 3% inflation rewards with no query performance. Using the restaurant-type analogy that’s been used above, everybody is serving the exact same burger (3% inflation rewards) right now. Indexers can’t change up the menu, they can’t offer a higher quality burger, they can’t offer fries and a shake with the burger. So how do indexers separate themselves and attract delegators? Price, mainly. A customer might well go out of their way to seek out a random small restaurant that’s giving away a free burger for a week (i.e., giving out a portion of their own rewards) or that’s charging less for their burgers (i.e., an indexer who takes a smaller cut of delegator rewards), but they’re not going to go out of their way to eat at a small restaurant that’s charging more for this burger than the In-n-Out across the street. (Ryabina and some others have also, I’m sure, attracted delegators by putting out tools like graphscan.io.)

  2. If we had query performance from the very beginning, delegator choices might have been very different than they have been. At the same time, it is also because we don’t have query performance that it might be too early to make sweeping changes to the system. Like derkhersh, I plan to look at query performance when they’re available and change my delegations accordingly. Maybe we need to consider a timeframe to pause the system and not allow new indexers/delegators until subgraphs are migrated and query performance come into play.

  3. The network is currently a free market system. The debate is somewhat whether that type of system is good or not or whether we need to change the protocol to change the system into something that’s less-than-free-market. Some people think it’s not working (too much centralization), and so there’s a proposal to change the system to drive a certain outcome (forced decentralization). Do consider that a ton of thought probably went into this network a really long time before this all got implemented, and we’re only two months into mainnet and talking about changing it? That doesn’t seem wise, and this discussion on the forum can drag out for weeks or even a month and that probably wouldn’t be a fraction of what The Graph’s team thought about when they first implemented the network.

  4. The more I think about it, if you were to put a cap on either (a) indexer stake or (b) delegations, it must be done going forward and not retroactively. Otherwise, you need to consider what ramifications the network and The Graph might face if they were to implement a change everybody has relied upon so far that ends up forcing the top indexers out of the network, forces hundreds of millions, if not more than a billion, of GRT in delegations to undelegate/not generate rewards for a month and cost another 0.5% burn to restake, and other similar consequences. These aren’t valueless tokens–you are playing with other peoples’ real money if you make a change like this, and unhappy people who feel they’ve lost a lot of money tend to do things like file lawsuits.

  5. Relatedly, as I believe has come through in the discussion, myopically focusing on the single issue of decentralization without addressing all the potential consequences of any proposed “solution” can create all sorts of other problems down the road that’s even worse than some vague issue of potential centralization down the line based on what’s currently happening.

  6. There has been a lot of “small indexers vs. big indexer” discussion and a lot of “indexer vs. delegator” discussion. Look, we’ve had a free-market system for months. If an indexer wanted delegators, they had to attract them. Many did! Some had Christmas/New Year “specials” where they gave out rewards with low to no fee cuts for a week or two. Some simply offered low cuts to begin with to attract delegators. Since then, some smaller indexers had been overdelegated, others weren’t overdelegated but still have quite a lot, and even more have generated plenty of delegators. But not every indexer felt the need to compete for delegators. Looking at graphscan, we’ve got close to 40 indexers who have 100% fee cuts and aren’t interested in delegators at all. We’ve also got plenty who are offering somewhere like 3 to 5% APY, and another group offering like 6 to 10% APY, when there’s a critical mass around 11% APY. The way the market works, an indexer just can’t offer offer some less-than-competitive APY and attract delegators due to some vague promise of query fees at some unknown time in the future. That was their choice. But again, it’s also the problem with having no query performance at this stage.

  7. What the above tells me is that, even if the system has flaws, even if delegators are choosing based on their own self-interest and not making the choices to benefit the network (whatever that might mean), indexers could well have taken things into their own hands to incentivize delegators to make certain choices that improve “decentralization.” I do note that this doesn’t solve the perceived issue where the top indexers have capacity to hold more delegations, but quite honestly I don’t see that as an issue at this point in time (too early).

  8. I also see it where some indexers seem to treat delegators as some sort of APY-chasing freeloaders while the indexers do all the work. If that’s how an indexer comes across, then consider maybe that’s one reason that indexer doesn’t get delegations. Indexers put in the time and money to run a business, but delegations also generate rewards by being in the network, and the indexer gets to take a cut out of delegated rewards. So given how the network runs, indexers need to really view this system as delegators being customers paying indexers a portion of their rewards in exchange for having the indexer run the infrastructure so that the delegators can earn the other 90% in rewards. Everybody wins, including the indexer, including the delegator, and including the network.

4 Likes
38

And yes, if you look at all the indexers who have set their cut to 100%, it is clear that the rewards on offer are currently enough to sustain operation, especially now that actively withdrawing rewards seems like it is happening asap.

Yes, the key word being currently.

The discussion/pre-proposal is more geared towards long term health of the network. As some have noticed, the more delegations come in, and the more money is on the line, the harder it is to implement changes if needed.

I’m really glad a lot of people are voicing their opinions, the good thing about opening up the discussion is that you get to see a lot of different perspectives, and can weight in different possible issues that might arise. This helps with modelling a bunch of scenarios to understand how this might play out, and understand whether change is needed or not.

Hey, we might even find out that change is not needed at all, everything is fine, and we still come out stronger than ever, helping anyone that might’ve had any doubts to clear their minds and push forward harder than ever :wink:

40

The problem with centralization is much bigger than most people seem to understand.
I want to go over a few key points that I mentioned in Discord and in voice chats, and hopefully make it more clear to everyone reading this pre-proposal. After that, I’ll go over the rest of the discussion here and try to address some of the topics that were discussed, in the order they were written. So brace yourselves, it’s gonna be one hell of a wall of text incoming.

Problem numero uno - Delegators herd mentality in going with "big number"

This is a huge issue imo and has been risen a few times over the course of the last few weeks, but none seem to understand the impact it has on the network.
I got a few examples coming.

First example, P2P:

  • Stake Owned: 35M
  • Delegated Stake: 528M
  • Allocates Stake: 564M
  • Percentage of allocated stake vs entire network’s stake: 23%
    _
  • Total number of Delegators in the network: 5268
  • P2P’s number of delegators: 1987
  • This represents 37% of ALL the people that delegated in the network.

Further issues:

  • They’re not active in the community
  • Never participated in Forum discussions or in Discord discussions
  • The only time they were active in the Discord was when they got called out they nuked their rewards
  • Their devOp in Discord has only ~50 messages all of them being questions of how the protocol works
  • They didn’t upgrade their infrastructure software for almost a full month, leading to some of their rewards being wiped out.
  • They tried to sweep it under the rug with a very flawed and inaccurate RCA
  • I tried to challenge their claims, never got any response back
  • To mitigate the above, they set their cuts at 0% for two weeks, causing the number of delegators they had to grow from 1400 to almost 2000

If we’re going to have people that have no clue what they’re doing and don’t care about neither the community nor the protocol, leading the network, then I’m afraid to think what’s going to happen in the future, when thousands of applications, developers and companies will depend on their services.

Second example, our own nodes at StakeSquid:

We currently have a total number of 5 indexer nodes running on Mainnet.
We didn’t publicly advertise them as being ours, we barely mentioned their existence in chats, and never bragged about how much rewards they generated, yadda yadda.

image
image1385×583 222 KB

The problem is that, inspite the fact we didn’t mention them much, the two in the list, sorted by Indexer Rewards and marked in yellow, have 98 and 89 delegators respectively. You might ask why is that a problem?

image

Let me compare it to the Indexer Node address that I personally have listed and advertised everywhere in the community:

image

Now you might ask what’s the difference between them?
Lower OWNED STAKE. That’s all :slight_smile:
The effective cut % is literally identical, at ~10% for all 5 of our nodes.

Conclusion, delegators have the herd mentality of going into “big number” and don’t care about too much else.

Problem numero dos - Insane operational costs will drive the market to the big indexers.

premise

Right now, we only have one subgraph on-chain, and makes everything cheap compared to what’s coming next. The 3% network inflation is distributed first proportional to subgraph signal, and then to allocated stake on individual subgraphs. The hosted service has ~8000+ subgraphs right now.

This means that if you want to have the full number of rewards coming your way (like we have now with only one subgraph) you will HAVE TO allocate towards ALL the subgraphs that have been signalled.

/premise

What does that imply? The allocation costs scale up linearly with every subgraph that has signal on it.
This, in and of itself has other major implications and I’ll go through them all.

First of all, let’s talk about today’s costs of allocations.

  • Each allocation costs ~$100 to open, and another ~$100 to close.
  • The default number of allocations on each subgraph, is 2.
  • That means for each subgraph, you’ll have to pay ~$400 for a full 28 day cycle (assuming you leave the allocations open for their max allo lifetime).
  • THIS IS ONLY FOR ONE SINGLE SUBGRAPH!!!

Each subgraph that is migrated on-chain will most likely have some GRT signalled on it.
In order to grab the whole network inflation, you will have to allocate to ALL of them

  • Example:
    • Currently there’s only one Subgraph, so the signal doesn’t matter.
    • Suddenly, 99 more subgraph are migrated and signal is rebalanced
    • The 3% inflation is split proportional to the subgraph signal
    • Network has 100 total signal, with 100 subgraphs and 1 signal each (all proportional, for this example)
    • This means that the indexers will have to allocate to ALL of the subgraphs that are on-chain if they want to keep their pre-existing baseline APY that they had before the migration happened

What does this mean? It means they’ll have to pay $40,000 USD in gas costs per month only to be able to keep their APY as it was pre-migration. Now, looking at the total number of subgraphs that the Hosted Service has, it’s easy to see that 100 number of migrated subgraphs being hit in the next few months.

Let's now run some numbers, shall we?

The network currently has 2.4b GRT allocated, with another 2b GRT sitting in LockToken Contracts, ready to be staked/delegated throughout the next few months, by various individuals.

image

For the sake of simplicity, let’s assume the network will stabilize at around ~50% of the total supply staked.

This will bring the network baseline APY to 6%, the math for this is simple, 4th grade math, inversely proportional values:
image

Again, in order to achieve that 6% APY you need to allocate to ALL the subgraphs that have signal.

Keeping in mind that $400 per subgraph operational cost (~40K USD total for 100 subgraphs x 2 parallelAllocations) that I mentioned above, here are a few calculations based on the protocol numbers, for different scenarios.

These calculations are based on the premise that the said indexer takes FULL advantage of the subgraph signal and allocates proportional to their signal, thus gathering the full 6% baseline network APY
(ie. If there are 100 subgraphs with proportional signal, and he only allocates to 25, he will only have 25% of the 6% baseline APY, meaning 1.5% APY)

The goal of this simulation is to determine the breakeven point for an indexer for different number of subgraphs present on-chain and having signal.

============================================================

SCENARIO 1
Indexer only has the 100,000 GRT minimum Stake and NO delegation

image

In this scenario, allocating to ONE subgraph cuts his revenue by half :slight_smile:
At TWO subgraphs present on-chain, he is below his breakeven point (if you include the infrastructure costs)
If more than two subgraphs have on-chain signal, this indexer goes bust the moment they’re added.

============================================================

SCENARIO 2
Indexer has 100K own stake, and full delegation capacity of 1.6M, and his effective cut% is ~10%

image

This indexer can only allocate towards 5 subgraphs before he goes below his breakeven point.
Any more than that and he goes out of business.

=============================================================

SCENARIO 3
Indexer has 5M own stake and NO delegation

image

If there are 100 subgraphs on-chain, this indexer can fully allocate towards ALL of them, and eventually turning about $4.5k in profit, excluding the infrastructure costs.

=============================================================

SCENARIO 4
Big whale indexer profit compared to other indexers

image

These are P2P’s numbers. At the network parameters mentioned above, they can turn over 600K USD in profit every month even if they allocate towards 200 subgraphs, lol.

=============================================================

Now why is this worrying?
When we get to 100 subgraphs on-chain, it’s simply game over for anyone that doesn’t have at least 5M GRT own stake. :slight_smile:
Why? First, due to breakeven costs being GIGANTIC.
Second, due to inability of maintaining the same APY a whale-indexer does, without going bankrupt.

Remember, if you can’t allocate to ALL the subgraphs that have signal, you’re diluting your baseline APY.

Thus, all the small indexers will go out of business as soon as a significant number of subgraphs are added on-chain, and will prefer to delegate towards whale-indexers instead, further driving the centralization of the network, to the point where it turns into the “Hosted Service 2.0”, just not under E&N custody, but under 10 mega-whales custody.

Is that the end goal for the protocol? I honestly hope it isn’t.

For comparison, this simulation is for all the nodes we run at StakeSquid:
5 nodes – Total 13M GRT own stake, 184M delegated stake, ~10% effective cut

image

Despite the fact that we only have ~2 times less allocated stake than P2P, the operational costs cripple our final turnover by a significant margin. Hence why the OP suggested lowering the delegation capacity and hard capping the delegation. It makes the protocol sybil resistant, as it’s not sustainable to run more than one node.

(Final note: ALL the calculations above are based on the current $1.8 GRT/USD price and an approximate cost of allocation. These numbers will vary, but the above is kind of the best case scenario, to be completely honest)

Problem numero tres - The delegator market is seeking high APY

We’ve seen the dynamics of the delegator market since the network launched, and pretty much everyone want to have as high of an APY as possible.

This problem is in direct relation to the second problem mentioned above.

If an indexer can’t allocate to all the subgraphs that have signal, the said indexer will only be generating a small part of the baseline APY.
His delegators will leave for the mega-whales that can easily sustain allocating to everything, further accelerating the centralization of the network.

Problem numero quatro - Query fees and query traffic distribution

The gateways are distributing traffic based on multiple factors, one of them being the economic security of an indexer.
This economic security comes from the indexer’s own stake, not allocated stake.
Right now, the top 5 indexers own 58.5% of the entire network stake.

Why does this matter? Well, it means that the majority of the traffic will be theirs, if it was to be based only on one single decision basis by the gateway. In reality, it probably won’t be this high though. But this is assuming the network stays at this rate of indexers, and no other mega-whales are joining, which I highly doubt it won’t happen. :slight_smile:

Second problem are the query fees.
The team estimates a $0.00001 cost per query.
At 12b queries per month (the current hosted service numbers) that will be equivalent to ~120,000 USD monthly, paid in query fees. At the current GRT/USD prices, that is a little over 66k GRT. Spread between 170 indexers. With the mega-whales having the economic security advantage over the rest of the network.

For query fees to be equal to the inflationary rewards, ~800k GRT per day, we would need 145 billion queries per month. This probably won’t happen any time soon, unless Solana or some other insanely fast blockchain is added to the protocol Mainnet.

(note: numbers above are based on the current price of $1.8 GRT/USD)

Possible solutions for the above mentioned problems

  1. Move the protocol economics to L2, pronto
    • This eliminates the gas price concerns and allows the network to function without small indexers getting BTFO the moment more subgraphs are migrated
  2. All of the OP mentions

Closing note: I know that you can capitalize on subgraphs that don’t have enough stake-on-signal in order to close the gap between rewards, but that’s not going to last for too long, just how an indexer that offers his own rewards to delegators won’t last either. :slight_smile:

17 Likes
41

I get this–though I don’t think it’s fair to suggest that delegators are bad actors in doing what they would do in every network. Wouldn’t one solution from the outset have just been to remove some of this information from the network UI? We obviously need some information about cuts and avoiding overdelegation and stuff like that, but otherwise let community tools like Ryabina’s give more of this information if a delegator wants to seek it out.

It sounds like many of these costs are built in regardless of whether “big indexers” exist though. So let’s run this hypothetical. Let’s say you just kick p2p out of the network entirely because you and other indexers just hate them that much. Okay, their portion of the 3% inflation rewards get spread across the remaining indexers. Maybe those 500 million delegations will get spread across indexers too, if those delegators stay in and don’t just sour on this network entirely and get out of it. You’re still going to have those allocation costs per subgraph. What’re the numbers then?

Sure, delegators want APY, but I don’t see why this is a “problem” – this is a fact of all crypto with staking/delegating mechanisms, it should have been well thought out already. Again, it’s somewhat unfair to blame delegators for doing what they always do in a free market–seek out high (and stable) APYs for their delegations.

By indexer’s own stake, do you mean the 140m framework has and 35m p2p has? If so, that sounds like a good reason why the network should perhaps have capped an indexer’s maximum stake at the beginning, so that we do not see 140m stakes like figment’s and framework’s in the network. At this point, though, it seems to me that any such cap must be implemented only going forward. I would, again, caution against any retroactive change that might get rid of several billion tokens from the network or practically force reallocation/redelegation of hundreds of millions of tokens.

2 Likes
44

I agree. I think we can all agree that Delegators are not malicious or unintelligent by nature and any sort of dehumanizing wording to that end needs to be put to rest. In fact, I’d like to call a moratorium on that from all parties now.

At the same time, we need to acknowledge that the direction delegations tend to flow towards is APY. This is rational behaviour however there are clear follow-on effects from this behaviour. The problem we face is similar to the issue of enticing users to use dApps rather than apps. Users care about what makes their life easier, not about how decentralized the underlying architecture is. This is a tough nut to crack.

Agreed in principle and modelling and exercising caution are a must. My understanding is that there was an expectation that more of the larger wallets would delegate.

47

Thank you for this. As a Delegator it is nice to see this actually be communicated by a Council member. I follow this forum even if I don’t post often, and I have seen several things said of Delegators that were not terribly respectful. I appreciate the call for a moratorium.

1 Like
48

Also want to support this moratorium, as it was this kind of language that caused me to come in so hot and unproductive yesterday. I think there are likely many ways to encourage different delegator behavior that are best served by assuming delegators will act rationally, and that the opposite assumption is likely incorrect and will prove unproductive.

49

One last thing to note here, and then I’ll once again step away for a bit so everyone else has room to speak.

On the subject of Framework labs and their 140m stake that was just added:

At the time of their stake, there were literally only three indexer pools that had room to accept a 140m delegation. One of those pools has it’s cut set at 100% , one at 97% and the other at 80%.

If you were looking to bring a substantial stake into the network, would you prefer to split it up 50 different way to 50 different indexers, would you delegate to one of the three indexers with enough available space for you on faith that they would adjust their cut down to an equitable level, or would you simply stake it yourself and start your own index?

It isn’t just the delegators who are acting rationally. The large indexers are as well. I doubt that all 140m GRT would have entered the network if Framework had been forced to accept 50 different trust-based relationships.

1 Like
50

This is a fantaastic thread.
@indexer_payne just to point out some modifications to your assumptions:

  • Clearly the $400/month per subgraph just to open and close allocations is untenable long term. This points to a clear need to move this logic to a Layer 2. I don’t see why this couldn’t be reduced to under $1 on L2. Anyone want to create that forum thread? :slightly_smiling_face:
  • I’d need to work through the math but I’m not sure that the assumption that you would need to signal on all the subgraphs to get the 6% from issuance is correct. Assume that there are only 2 Indexers that are each indexing half of the subgraphs, the network is still issuing the same 3% so I believe each Indexer would continue to get the full 6% effective rate (on 50% participation).

Hopefully these two points make the Indexer picture not as bad as presented.

7 Likes
51

It’s volatile based on both the ETH price and the TX gas price. My calculation was based on ~200 gwei which became the norm these days. The absolute minimum I’ve paid was ~$150-200 for a full cycle (so half of what I mentioned in the thread). But that’s because I’m camping for low gas prices with alarms set on 3 different tools to email me.
This is bad, nonetheless.

When you’re allocating towards subgraphs you need to take into consideration the ratio of Signal-to-Stake on that subgraph in order to be optimal. So in a perfect world, the stake that you have to allocate would have to be proportional to the Subgraph Signal.

BUT, there are inefficiencies, and this is something that I mentioned in the very last note of the thread, I called it Stake-to-Signal ratio, and it’s a market inefficiency that will eventually be covered by a healthy market. Just like we have the Indexers offering high APY by giving their own rewards away, they eventually get delegations and things get balanced out.

Plus, there’s the other side of the story. What if you allocate towards a set of subgraphs while it was efficient for you to do it, and then someone immediately comes in with a huge allocation and throws you into the inefficient area, you’d have to deallocate, and balance your stake again. Doing it less than every 28 days (max allo lifetime) will be very costly in terms of gas, and we go back to the 1st point you mentioned, that we need to move to L2 asap.

Allowing indexers to move freely inside the protocol without having to have worries about paying tens of thousands of dollars for allocations will make the protocol economy very robust imo, and leads towards a more healthy market. :slight_smile:

2 Likes
52

One last thought that I think we may have all completely missed while we were entrenching ourselves into our positions that occurs to me now.

Yes- we have two indexers who could, between the two of them, account for 4b+GRT in delegations.

While I do not know what percentage of the tokens currently locked into vesting schedules which are eligible to be staked (is that a published figure anywhere?) and this could affect how valid this next bit is a bit, hear me out.

If both of those indexers were to fill up their pools by, lets say, June it would likely mean that nearly all of the VC tokens that were released throughout May went into delegation! This would be an extraordinary outcome! In other words, it isn’t likely at all, but if it did come to pass it would absolutely not be “bad.” In other, other words, small indexers have not collectively staked enough GRT to accommodate a full load of delegations, and so we must be very careful with any attempts to restrict new indexes from popping up, even if they are a bit large for our comfort.

The early backers of the project, from what I can tell, are early backers of MANY projects, and likely do not have the time nor desire to “get to know the indexers” and develop relationships with 10 or more of them so as to spread out their delegations equitably. What these backers WILL have time for is splitting off their initial bulk delegations to the best performing small indexers, once there are market-based revenue streams to judge indexer quality by. If a small indexer is generating good results on a smallish delegated pool, the big wallets will feel confident that not only can this indexer maintain that APY with additional allocation to play with, but will likely exceed their past performance with the additional scalability of a sizable new delegation.

Again, in this sense, I view big box indexers to be a sort of clearing house for getting GRT into the network. As large wallets come out of lockup, they will likely prefer to start with one of two delegations to well capitalized entities and then branch out from there at their leisure.

As a small indexer your ability to generate income will always come from your technical/strategic decisions, as well as whatever you can do to set yourself apart from one another to retail investors. I admit that I feel for you guys right now, as there’s not a whole lot you can do about the query fees, but there’s still things that can be done to attract a delegator or two!

However, as Yaniv points out, perhaps there are infrastructure initiatives that could be pursued now, while things are pretty slow for you, that could dramatically reduce your costs down the road. If you can knock some of those out before query fees come into play, you could be sitting very pretty to immediately pursue your own niche strategies… ground that the large, more conventionally minded indexers will likely cede to you. Further, I can comfortably predict that simple infrastructure upgrades to lower cost and increase scalability will meet little to no pushback and are likely to be fast tracked where resources are available to do so. Changes to incentive structures are much thornier issues and require much more careful (and occasionally contentious) consideration.

53

Excellent thread!

I want to offer additional views on the situation. One of them is that I think it will be possible to capture close to the full 3% inflation rewards with clever selection of a single subgraph. This equilibrium emerges as most small indexers will have to use that strategy as lowest cost option doing business to stay in the game. This way indexers can just allocate on every reallocation to the highest APY subgraph what in theory can pay even above average. This also brings down server costs. Indexers can also take factors like cheaper archive nodes for non-trace subgraphs into consideration. This gives traces subgraphs a slightly higher APY. It doesn’t get much closer to a probably functioning prediction market. It anyway can’t be the goal of decentralization to have every indexer providing all subgraphs.

I’m sharing the OP thoughts and think all of the above stated is correct. I just put out additional scenarios that come to my mind.

The other strategy would be to setup a dao around a community indexer who gives most of his revenue to stakeholders after operational costs and ecosystem grants. If we are priced out individually it’s important to have alternatives for testnet participants who want to delegate to someone they trust to be consistent and in for the long game. Having a entity like a dao in place would involve electing a representative to participate in collaborations with the graph foundation.

I think a combination of the above scenarios can be a logical next step until cost savings emerge. How much combined economic weight have the testnet participants currently?

1 Like
54

Just wanted to make sure this was accurate. Here’s what I found.

If you exclude indexers who have self-staked more than 10m GRT then you will find that there is room for about 3.5b GRT in delegation space available from the “small indexers.” But that isn’t really true, because there are currently 35 small indexers with an indexer reward cut of 100%, so these pools aren’t really delagable. This group of indexers (of which only 3 have gone so far as to give themselves a trade name- a bad signal of future intention to be an active participant in the community, to my eye) have staked a little over 60m GRT, which eliminates roughly 1b in delegation space from the “small indexers.” So small indexers can provide space for only about 2.5b in delegations.

Two thoughts on this- I believe I read someplace that the economic model suggest that a balance of roughly 50% of circulating GRT on the market and 50% in delegation at any given time would be considered ideal (someone please correct me if I’m wrong on this, I could be misremembering something from another project and can’t seem to find any published figure on this). If that’s the case, 2.5b in delegation space is only barely enough to get us through the spring token unlock- and when you factor that there are also delegable tokens on locked vesting schedules- it actually falls short of that target. Even if it didn’t, or I’ve got the wrong target in mind, it would still seem unwise to run the network without enough delegation space to accommodate something approaching the full supply of GRT (even though, obviously, some amount of supply must remain on the open market in order for the network to function). So what is clear is that “small indexers” have quite a bit of economic weight in this system as a whole, but perhaps not indivdually. That’s why the following quote might be your answer to this problem, for any testnet members who are not terribly attached to notions of rugged indivdualism.

I might suggest something similar- some sort of “alliance of independent indexers” that works to set standards and indexer best practices. You could issue indexer certifications, and self-govern group strategy, vote amongst yourselves about which sorts of infrastructure projects to back, work towards solutions that can reduce indexing costs, etc. I might suggest a loose DAO of DAOs concept- I may have just missed it, but I haven’t seen a ton of mention of indexers giving any governance roles to their delegators, even informally- but I should think that would be something potential delegatorss would really latch onto.

The second thought is that there is kind of a big issue if more than 10% of indexer stake is held by indexers who are not accountable to delegator interests. Before the two large 140m GRT stakes were put down recently, this group of indexers accounted for nearly 20% of total indexing stake! If we are concerned about network participants who may not be acting in a manner conducive to the positive growth of the network, we might look there as well.

55

@Derkhersh

Also want to support this moratorium, as it was this kind of language that caused me to come in so hot and unproductive yesterday.

There’s no sign of such language prior to your posts.

Please remember that this is an open discussion for everyone, not a proposal. There’s no need for defensive posturing, and doing so only causes people to ignore your input, even if it’s valid arguments that are being presented in such a manner.

I’ve noticed most people have mostly ignored your ‘other’ posts and i appreciate your tone has changed somewhat, but you already account for around 40% of the posts in this thread. Please consider that.

@cryptovestor
I think it’s clear that Delegators are acting in completely valid ways. Any language that infers that certain tactics are ‘dumb’ or not valid somehow, is not welcome in this thread.

Observations of behaviour is fine, judgement is not. Let’s all stick to discussing the potential impacts of that behaviour, on the network and its participants at large.

@indexer_payne
Great post, i hope that people will add to it and focus on the content of your data/arguments more thoroughly.

Allowing indexers to move freely inside the protocol without having to have worries about paying tens of thousands of dollars for allocations will make the protocol economy very robust imo, and leads towards a more healthy market.

This reminds me, the idea of funds being “in protocol” as it applies to Delegators could be a solution here, if anything like caps are introduced.

I’ve noticed a few concerns about what will happen to those Delegated to any Indexers who are already over [hypothetical cap].

Regardless of the topic at hand, i feel that being able to change Indexers without a thawing period is something that will make delegation behaviour much more dynamic, and alleviate many concerns of Delegators and the punishment they current face if they find themselves on an Indexer who behaves/performs poorly.

Something to keep in mind, as any changes made should:

  1. Consider the impacts faced by all participants
  2. Try to avoid punishing them, simply for participating ahead of such changes.

@czarly

I want to offer additional views on the situation. One of them is that I think it will be possible to capture close to the full 3% inflation rewards with clever selection of a single subgraph.

I can’t see this being the case once we move beyond more than a handful of subgraphs.

And part of the problem (as pointed out by @indexer_payne ) is that even a handful of subgraphs already has a huge impact on the opex of smaller stakes, to the point where they’re excluded purely by overheads.

People seem to overlook that certain parameters, as discussed here, make other parameters such as ‘100k minimum stake’, completely impractical. I’d say that even telling people to be an Indexer with that amount of stake, after viewing the data within this very thread, would basically be in bad faith. Yes it’s technically possible, but it’s unlikely to prove feasible, and that may (very) soon apply to stakes much larger than 100k.

Finally - I’m not sure on the 2nd part of your post. I don’t think any Indexers who turn to delegation should be treated any differently from the rest.

I’d like to see people to focus on the content of arguments more, going forward. :+1:t2:

There’s already some great posts in here. Please respond to the arguments of those more directly, so that we can see the challenges to/for each point evolve in a productive and progressive manner.

I feel much of the thought and effort that went in to many posts has so far been largely dismissed or overlooked and i would rather not see quality posters discouraged from contributing further.

4 Likes
56

Original TON (Telegram Open Network) has a variable called “max-factor” the main purpose of which can be taken from this quote:

Here <max-factor> = 2.7 is the maximum ratio allowed between your stake and the minimal validator stake in the elected validator group. In this way you can be sure that your stake will be no more than 2.7 times the smallest stake, so the workload of your validator is at most 2.7 times the lowest one. If your stake is too large compared to the stakes of other validators, then it will be clipped to this value (2.7 times the smallest stake), and the remainder will be returned to you (i.e., to the controlling smart contract of your validator) immediately after elections.

And it worked very well in practice, whales couldn’t just throw money in, they had to scale horizontally.

I understand that The Graph isn’t blockchain, but some dynamic adjustments per epoch could be introduced to avoid enormous spread between participants.

The other thing is to introduce some kind of parameter that will let big whales reduce their economic security for the Indexer selection algorithm, which will give them an option to lower the number of queries if they understand that their infra can’t handle all of them based on their natural economic security.

1 Like