GIP-0002: Rewards destination for indexer's indexing rewards

We absolutelly agree with prev post espcially having the exciting news that new subgraphs are coming on mainnet which would lead to significant operational costs increase.

3 Likes

Thought I’d just knock this timeline together to give a bit of perspective on how this will impact indexers.

This is what I would consider the optimistic scenario for GIP4 progressing to making rewards available to indexers - that is an audit available within a timely manner and no delays relating to testing the implementation (bear in mind that GIP4 is a more complex solution than GIP2 so expect additional overhead here).

We are looking at approximately at least 3 more months of delay for indexers to start to cover (or recover) expenditures relating to opex and tax obligations. During this period indexers are expected to not scale back their operations but in fact expand on them both from a hardware perspective and ETH as well due to the migration of subgraphs to mainnet, incurring costs higher than seen previously throughout the 9 months of mission control+mainnet.

@Brandon and @yaniv , can I please implore you to make a decision on whether GIP2 can act as an interim solution as soon as you find out when audits can take place (I believe this may ultimately act as the deciding factor?). Brandon, I believe you mentioned you will find this out towards the end of this week.

13 Likes

Completely agree with your thoughts. :+1:

1 Like

I am very disappointed with the timing on gip-02. I wish it were accepted as quickly as the network launch back in December. I understand its governance process and blah blah, but seriously, it’s already tested and voted. What do we waiting for? The “reasons” Brandon mentioned above won’t have a significant impact, and @Pete-LunaNova well described why it so.
But what’s really important is that right now a lot of small indexers who don’t have delegators or have a few (I checked it’s about 90-100 indexers with <1m delegated) are considering shutting down their nodes. They have a lot of rewards accumulated during this time which can be withdrawn entirely by unstaking all. Is it good for the network to lose them? Sorry if I seem rude. I’m not big into political correctness. :heart: The Graph.

6 Likes

I personally know several indexers who will unstake, what the point to wait 28 days to withdraw future rewards then? if they can take all… I would definitely do the same if I didn’t have delegates

2 Likes

Totally agree with the previous indexers. What was the point of discussing and voting on GIP-0002 for two months all together if the council effectively ignored it? If GIP-0002 is not implemented any time soon, we won’t have any other choice other than to stop and unbond our indexer for a month in order to extract the rewards.

2 Likes

What a cumbersome outcome for an already ironed solution with almost unanimously voting. I’m very surprised if not to say the least.

In the light of the last news about subgraphs migration to the mainnet, Indexers need to expand their infra, but instead of that, they will be willing to optimize their costs, and because of gas prices, it will be hard to join the party with new allocations for upcoming subgraphs.

3 Likes

Just wondering, what was the point to ask for opininion (voting process) and got >99% support to go ahead with tested and vetted solution, if at the same time minority < 1% (by fact a few users whose involvment and contribution in community is uncertain) could completely deminish and eliminate effort of whole community and active members. (this is obvious and absolutelly normal that it is impossible to meet and satisfy 100% of community, that’s why voting procces is in place which aim is to help to make a desicion which majority is asking for)

  1. This fact is at least not fair and disrespectfull to 99% of communty and as we can see it even pushes indexers to stop their nodes.
  2. It brought up a worry that voting procces is not matured enough where voice of majority of community still means “nothing”

We kindly ask you to go ahead and implement GIP-0002 as soon as possible - technicaly it could take just a few hours.

Once GIP-0002 is implemented we are open and happy to discuss any other proposals and changes which would improve network stability and community health.

5 Likes

Thank you for taking the time to post your suggestion.

The challenge here is how we get all stakeholders (delegators, developers, indexers, curators etc) to pull together in the best interests of the protocol. Those blocking this issue from moving forward need to recognise the severity of the problem and respond in an appropriate timescale. Without decisive action we run the risk that this situation continues to the point that Indexers are forced to withdraw, damaging the whole ecosystem in terms of both reputation and efficacy.

The GIP-0002 solution was presented by The Graph team (now Edge and Node), discussed at length, smart contract modifications implemented, audited, tested and finally a public vote for all stakeholders led to a near-unanimous decision to move forward with it. Those who now wish to potentially block GIP-0002 in favour of another solution should have been clear about this much earlier and we could have had a vote on multiple options. To attempt to switch to another proposal now, that delays an already untenable situation, puts the whole network at risk.

The real issue, is not that there are not other potentially viable solutions, it is that we are out of time to explore and then implement them!

Indexers need access to liquid rewards now, which was the original design intent – this problem is an error that needs to be corrected, not next week, not next month, but now. Indexers are keen to play their part in the ecosystem in partnership with delegators and other stakeholders, for this they have incurred significant operational expenses since the start of testnet 8 months ago, have put in substantial unpaid effort and now have accumulated serious tax liabilities. Preparation for the migration of more Subgraphs will require yet more expense and effort from these same parties.

8 Likes

You’re right. Why did The Graph Council not voice the issues they were concerned about when we discussed the GIP-0002 in the past? Why do they reject after voting and come up with another option that takes longer to complete? It was a vicious cycle and we spent 3 months on this problem but ended up waiting for at least another 2 to 3 months for another solution.

5 Likes

As a delegator, I have been following this discussion closely over the last few months. I feel the indexer community has described in sufficient enough details the logical need to obtain access to rewards and I think this is broadly understood at this point.

GIP2 is not without flaws and the councils decision shows they recognize that and take it serious. However, the decision to delay introduces indexer challenges that are not in the best interest of any network participant. If indexers start shutting down, then we all lose in the pursuit of our mutual goal to grow The Graph ecosystem.

I have full faith in this community and I believe most everyone here wants to do what’s right for the project, whether it’s the council, indexers, delegators, curators. Let’s not have our differing viewpoints lead to decisions that may result in serious damages to one network group.

I applaud that we are working on a new GIP that addresses concerns raised in GIP2. I have full trust that the new GIP will receive broader support across the aisles and will later be approved as a more equitable solution. With that trust in mind, I also support implementing GIP2 asap as an interim solution, so that indexers can address both their existential concerns as well as subgraph migration readiness over the coming weeks.

5 Likes

With the Graph Mainnet Migration Path announcement made here and the reaction to the proposal of GIP0004 to replace GIP0002 I feel the need to instill some urgency, to ensure that everyone fully understands the gravity of the time-to-revenue Indexers face and the impact of waiting for GIP0004. I am writing this now because waiting for the next Council meeting is too late. With hindsight and knowledge of the Mainnet Migration Plan, this is the strength of opinion I would have delivered to the Council meeting.

Indexers have made valid points about the burden of their financial situation and how waiting on GIP0004 worsens the it for many. These issues skew aggressively against smaller indexers. Please take the time to read their thoughts in this thread and understand the weight they carry. My focus here is on exploring realistic timeframes and getting this sorely needed fix in place as soon as possible.

The migration announcement sets out plans to start migration tests in April during which Indexers will see no revenue despite the requirement to scale up operations both opex (labor, hosting) and capex (compute). The migration announcement sets out plans to move into production apps after that. With GIP0004, Indexers will continue see no realized revenue until at least June/July at best so there is an expectation that they continue to scale and provide services into live migrations with no revenue. This is not acceptable in any business environment.

When taking into consideration governance process, technical review of GIP0004, auditing being an unknown timeline but critical path to a GIP0004 upgrade, we are looking at late June/July before rewards start to hit Indexer wallets as an optimistic case. Thank you to @Pete-LunaNova and @dancube for setting this out so clearly. Given the current bull market and how it creates extreme difficulty in sourcing auditors, especially auditors that already have knowledge of the codebase, we could be waiting any amount of time for a GIP0004 audit - Indexer’s cannot tolerate such uncertainty anymore.

We need our Indexers to move into testing and migration with their minds focussed on delivering an excellent service, not struggling to stay afloat or crushed under tax obligations because by the time they can access any of their revenue, a whole year has passed since Mission Control started.

The migration announcement crystallizes my personal opinion - we need to resolve the Indexer rewards issue now with GIP0002. I implore everyone that in responding to this, we focus on how we come together to fix the shortfall promptly. Lets move into Mainnet Migration with confidence that all stakeholders are financially positioned to play their part. If we have to resolve some GIP0002 risks and we have assessed those risks appropriately, lets do so retroactively and as promptly as we can.

Thank you to everyone that has contributed and continues to engage in the discussion.

21 Likes

I’ve had a lot of conversations today with many stakeholders on this topic, as well as with @cryptovestor (in context of our recent Council meeting vs ‘the now’) and i appreciate being able to witness a lot of people making their voices heard in here and elsewhere. It’s no surprise to myself that Indexers, after a long period of patience and understanding, are being vocal about their grievances with all of this. It is the exact sentiment i expected.

I just wanted to lend my support to the above and say that the extra context provided by time, thought, discussion and the newly-revealed Subgraph Migration timeline has made it even clearer to me just how critical a solution is to this matter. In my opinion, one that adds further runway in addition to a thawing period is simply no longer a feasible consideration as our immediate next step. Not unless we’re OK with seeing many Indexers leave the set, once faced with the additional unpaid work/infra/costs that shall come along with this next step for the network.

The tax considerations/timelines presented above by some may not apply to every Indexer, due to jurisdiction, but i’ve seen this topic deliberated over for many weeks/months at this point. And i’ve noticed that many of those who have been vocal on this particular matter, were some of those who were motivated enough to join the new (unpaid) testnet, lending their time and energy, and even testing the GIP 0002 implementation as part of that. People willing to do unpaid testnet work are the types we should aim to retain, and their efforts stretch far prior to mainnet or the ‘new’ testnet…

Of course, the current ‘tax’ deadlines facing some may not be reason enough to move forward with the GIP. But it’s something that affects all Indexers to a various degree. And this is still only one part of the many other factors that contribute to the urgency of this situation, such as:

  • It never being intended that Indexers were unable to extract their rewards.
  • The fact that Indexers have given their time/energy/costs far prior to the initiation of mainnet (~8 months so far…)
  • The snapshot ‘signal’ that drew a 99.72% positive vote, which is not only a matter of consensus, but also reflects on how engaged Indexers are as a group.
  • Further to that - Ignoring such a clear result (when a vote was expected to follow) and the opinions of some of our most engaged stakeholders may set a bad precedence and lack of faith in our governance processes in the future.
  • The risk of centralizing the network through inaction on this matter, as Indexers drop out.

… and many other points - all come together in convincing me that we should move forward with this GIP as soon as possible, and then address any perceived imbalances or improvements (preferably, very) shortly after.

There’s a lot of people here with nothing to show for the past 8 months other than gains on paper, or a dashboard. Not even something to cover their time and expenses thus far.

If we want those people to stick around, we need to start allowing them to see something tangible for their efforts. Not working for ‘free’ while also indebting themselves further, just to stay involved in their role. And aside from that, for the network to thrive, it needs them to be able to perform their role well, with whatever infrastructure required being covered as part of that equation. The network can not be propped up by goodwill or expected to scale through what is essentially still ‘donated’ infrastructure.

And we can’t expect the network to not centralize, while also observing that the realistic, continued lack of realized compensation, is now on a course that could see us hitting the 1 year mark.


Side note: I would appreciate further clarity (from an E&N developer) on how exactly the unthawing mechanism in GIP 0004 works. I’ve spoken with a number of people on this and seen/heard some concerns over this implementation. It’s not possible for any Council member to deliberate over this, whether it’s implemented now or later, without further information.

14 Likes

Good opportunity to provide some deeper insight into the voting results of GIP2. Beside the token support of 99.72%, GIP2 also received a broad wallet support of about 90%. Moreover, the engagement of the voting also extended to the Delegator community that casted 37% (135) of all votes with 82% of them supporting GIP2. Below chart provides more details.

While some valid concerns were raised during the GIP2 discussions which the community seems committed to address, I think the voting data demonstrates that the Delegator community at large also supported this GIP for all the reasons mentioned in this thread.

Screen Shot 2021-03-25 at 3.38.45 PM

11 Likes

Some update about the current status of the PR related to GIP 0004.

The way the implementation works is by using the exact mechanism as removing stake. It will do a weighted average of time and amount of tokens to update the new unlock block for rewards in the pool. I find that this is suitable for unstaking where indexers have more control over when to initiate. However, in the case of rewards, both closeAllocation and claim will trigger a recalculation. This could make it hard to estimate when the withdrawal is unlocked or affect indexers’ allocation strategy as now the two processes will be coupled with the withdrawal process.

We are evaluating some options:

  • Indexers initiate unlocking of rewards in the pool like when they unstake, to have the update of unlocking date detached from depositing rewards.
  • Devise a more fluid mechanism.

For reference: Withdraw indexing rewards to beneficiary address with thawing period by abarmat ¡ Pull Request #456 ¡ graphprotocol/contracts ¡ GitHub

4 Likes

Thanks for that update Ariel.

This sounds to me like the most tangible solution that will allow indexers to rely on a date at which they will receive their rewards upon triggering a withdrawal, a useful property to have when it comes to forecasting ahead when it comes to expenses. I can’t imagine the implications on tax calculation for the current weighted average method or the ‘more fluid mechanism’.

The fact that we are still exploring options like this tells me that GIP0004 is a long way away indeed, and that Jim’s timeline of June/July does sound like a best case scenario.

2 Likes

Hey everyone, thanks for all your additional input. I appreciate everyone’s contributions to this discussion and commitment to finding the best solution.

I want to reiterate that we take the urgency of this issue very seriously, especially in light of the recent announcements around subgraph migration. As I noted in my previous update:

I would not personally support delaying on GIP-0002 if it meant Indexers waiting substantially longer to claim their indexing rewards

I do feel like we have an obligation to explore all viable proposals in working towards the best solution (and these extra few days have given us that opportunity), but we shouldn’t let great get in the way of good for a problem needs a short term solution.

Here are some updates that were promised in my previous post:

  • Graph Council Meeting notes have been published and can be found here
  • GIP-0004 has been published to my main branch on Radicle (Ariel has already shared the Github PR).
  • The earliest we can get GIP-0004 reviewed by an external auditor is the first two weeks of May, which we feel is too long to wait on a fix for this issue.

The last option being explored is whether The Graph Council itself could sign off on GIP-0004, given that the changes are limited in scope, and that we have three external (to E&N) Solidity experts on The Council. This is a benefit of having technical domain experts on The Council.

We’re meeting tomorrow morning to walk through the code changes line by line and discuss whether we would be comfortable with this approach. After that, several Graph Council members have expressed willingness to do an async vote on GIP-0002/GIP-0004 in the next few days, to accelerate the schedule for an upgrade. We will have another update to post on this by EOD tomorrow.


Lastly, I’d like to clarify a few (what I consider to be) misunderstandings in some of the feedback above:

A lot of concern seems due to unfounded worries that somehow GRT price may be negatively affected by Indexers having access to their rewards.

This is not at all where the concern lies. Thawing periods in our protocol, and mechanisms for adding “slowness” to cryptoeconomic protocols in general (whether they involve voting, staking, etc.) serve many purposes, and it was never the goal of GIP-0002 to remove that. It was simply a side effect of trying to address another unrelated problem with a minimum viable solution.

I understand its governance process and blah blah

I realize decentralized governance, and the time it has taken to get the tooling and processes to support it set up, may seem like an unnecessary burden, but I believe strongly that there is value in doing things right from day one. Our commitment to this ethos is part of what has brought so many people into The Graph community and is a requirement of launching a decentralized network.

what was the point to ask for opininion (voting process) and got >99% support to go ahead with tested and vetted solution, if at the same time minority < 1% (by fact a few users whose involvment and contribution in community is uncertain) could completely deminish and eliminate effort of whole community and active members.

During the early days of The Graph’s decentralized governance, the role of SnapShot community polls is to be a valuable information signal to The Graph Council and Foundation, along with other inputs like discussion in the forums and at Protocol Townhalls. It is not intended to be a direct governance mechanism, though I do believe that it could lay the groundwork for such a mechanism in the future.

One problem with adopting the results of any vote “as is” is that the community was never given the chance to vote between GIP-0002 and GIP-0004. They simply voted between GIP-0002 and an undesirable status quo, and I would wager that any proposal that improved upon the status quo (including GIP-0004) would have also likely received a “yes” vote. Voting mechanisms like this, in general, are extremely vulnerable to the ordering in which proposals are presented.

Currently, it’s the mandate of The Council to take into account both the quantitative feedback offered by polls, as well as the substance and content of specific feedback being offered. This mandate, more than anything, is why we felt it was appropriate to take a few extra days to consider all options and not out of some irrational desire to diminish the voice of anyone in the community.

All the processes and tooling that we’ve set up, from a transparent GIP process, to a Graph Council with representatives from all stakeholder groups, to SnapShot community polling, to Protocol Townhalls, has been with the intention of elevating the voice of as many folks in the community as possible. I know getting these self-same processes set up is part of what has taken so much time, but I promise this will get smoother and we’ll continue to iterate until we push the boundaries of what an effective decentralized governance process can look like. :sparkles:

13 Likes

Hi Brandon,

Thanks for the update. I appreciate the effort your team are putting in to this.

I am concerned that you still consider GIP-004 a viable alternative to GIP-0002 in the short-term. Even if GIP-0004 was somehow ready to go tomorrow, it still takes 28 days to get liquid rewards due to the thaw period. For small Indexers, having run operations and incurred tax liabilities with no income for the last 8 months, this is simply too long to wait. One Indexer unstaked and closed up shop yesterday and another one earlier today. I have just seen another enquire about how to do this; the trickle may soon become a torrent without a quick way to get these people paid for their efforts. Given the number of direct competitors to The Graph coming online, it is vital we retain and nurture our ecosystem talent to survive in the long-term.

In my initial response I did make the point that “A lot of concern seems due to unfounded worries that somehow GRT price may be negatively affected by Indexers having access to their rewards.”, but this was not specifically directed at you. Although removing thawing for rewards was never the goal of GIP-0002, the discussion around it has highlighted how a lengthy thaw period can enable large, wealthy players to gain advantage over smaller indexers, therefore some of the GIP-0002 side effects may actually be serendipitous in helping us move towards a better model.

At the right time, I’d appreciate a response to the other points I raised so that we could get into the details. I’m keen to understand if there’s anything I’ve missed. I’d also like to explore whether a significantly reduced thawing period for rewards (say 1-3 days) could be a good compromise in the longer term. This could still disincentivize the sort of behaviour you wish to avoid but not hinder smaller Indexers, less able to deal with very lumpy revenue flows than larger players. This shorter thaw duration may also deter large Indexers from offering back-channel immediate rewards to their delegators, a very real risk with a 28-day period.

5 Likes

Thanks for laying this all out. I don’t disagree with anything you’ve said, but I don’t think the urgency of more than 50% of indexers potentially pulling out of the network entirely can be overstated.

Yes, you may be right that GIP-0004 would win in a vote over GIP-0002, however this is not a common scenario I’ve seen in protocol governance. An option was put together, it was voted on, and the results were overwhelmingly in favour of the proposal as-is. If GIP-0004 were already being contemplated at the time, then GIP-0002 never should have been put up for a vote.

While I respect that The Graph Council gets the final say on this matter, I would strongly urge the Council to move forward with GIP-0002 and then have a separate vote on GIP-0004 with clear intention that it replaces GIP-0002’s implementation if approved.

Is that a bit cumbersome? Yes. However, it more or less solves everyone’s concerns in a relatively short amount of time while providing respect to the Indexers who are in an incredibly risky financial position, and understandably so, while also putting to bed the issues of potentially timing withdrawals and notions of fairness when compared with Delegators (of which I am one among the 6,000+).

I strongly feel that not implementing GIP-0002 now threatens the viability of The Graph at one of its most critical junctures.

8 Likes

What do you mean by “sign off”? Are you suggesting a go ahead with implementation in place of a professional audit?