GIP-0028: Subsidized Query Settlement

[ari]

Abstract

This proposal extends the indexer reward subsidy to take into account both indexing subgraphs as well as query fee collection to ensure we close the loop of the lifecycle of an allocation.

Motivation

The network provides incentives to bootstrap the operation of indexers for new subgraphs which may not have pre-existing demand to attract indexers. The way it works is that each subgraph in the network is allotted a portion of the total network token issuance, based on the proportional amount of total curation signal that subgraph has. That amount, in turn, is divided between all the Indexers staked on that subgraph proportional to their amount of allocated stake.

Today, indexers collect indexing rewards when they close the allocation to a subgraph. Consequently, the subsidy only covers half of the allocation lifecycle, and indexers might choose never to collect query fees due to changing network conditions or any other strategy. By placing all rewards distribution at the end of the allocation lifecycle, we align the incentives to ensure that the network subsidy includes all the actions that indexers need to perform to end up with a lively network.

Specification

Allocation Lifecycle

The lifecycle of an Allocation starts by setting aside tokens from the global indexer stake into a subgraph and finishes when claiming the query fees from the rebate pool.

Current:

1. Create Allocation

2. Close Allocation → distribute indexing rewards

3. Collect query fees through payment vouchers

4. Claim Allocation Rebate → distribute query rewards

Proposed:

1. Create Allocation

2. Close Allocation

3. Collect query fees through payment vouchers

4. Claim Allocation Rebate → distribute both indexing and query rewards

Store Indexing Rewards

The main change is that indexing rewards are no longer distributed when indexers call closeAllocation() but it will be delayed to a later stage in the Allocation lifecycle. Rewards will still accrue until closeAllocation is called, but we will set aside those rewards when the indexer presents a non-zero POI.

To support this, we will add an indexingRewards state variable in the Allocation struct and initialize it to zero when indexers create the allocation in allocateFrom().

Then, after at least one epoch has passed, the indexer will close the allocation, the protocol will use the RewardsManager.takeRewards() function to mint the accrued tokens to the Staking contract and get the amount of tokens created. We will store those tokens in the Allocation.indexingRewards state variable instead of sending them directly to the indexers & delegators.

Distributing Indexing Rewards

The lifecycle of an Allocation ends when the claim() function is called. Claiming is a public function that anyone can call passing a valid AllocationID.

Any token distribution, both for query (from rebate pool) and indexing rewards will happen when claiming the Allocation.

Changes:

• After claiming from the rebate pool, calculate the totalRewards by checking if query rewards received are greater than zero, if that is the case we will let the indexer get their indexing rewards. If query rewards are zero the indexer will forfeit indexing rewards. The formula is totalRewards = queryRewards + indexingRewards

• Collect delegation rewards for both stored indexing rewards and queries coming out from the rebate pool. The delegation rewards are then sent into the delegation pool for the indexer that created the allocation.

• Reset the indexingRewards state variable in the Allocation struct.

• Send the indexer part of rewards to the indexer (both indexing + query rewards). The formula is indexerRewards = totalRewards - delegationRewards.

Additional interface changes

The function to claim an allocation currently has an extra parameter to decide whether to restake the rewards or not. However, this is not necessary and in fact can be troublesome as anyone can call the public claim() function.

We propose to use the internal _sendRewards() function along with the existing Rewards Destination feature to decide whether to restake or not.

# old interface

function claim(address _allocationID, bool _restake) external override;

# new interface

function claim(address _allocationID) external override;

Gas Considerations

• We are using 20,000 more gas to store the new indexingRewards variable in the Allocation struct during closeAllocation(), some of that is recovered when the variable is cleaned up during claim().

• Doing all token transfers during claim() instead of having it split into two different calls is more efficient, saving around 10,000 gas.

Special Considerations

• Indexers that haven’t collected any query fees are not elegible for indexing rewards. Those rewards stored during close allocation will be burned after claiming.

• Claiming rebates requires waiting a seven-day period to ensure that all funds are collected properly from payment channnels, this has the side effect of delaying indexing rewards to when all the conditions are met.

Backwards Compatibility

If we remove the unused _stake from the claim() function that would require any agent using the contracts to update.

Once the upgrade to the new mechanism happens, all of the Allocation structs for created allocations will have the indexingRewards variable set to zero, which is convenient as we don’t need to initialize them. Whenever indexers close any of those allocations the protocol will calculate the accrued rewards and fill that variable for use during claim().

Validation

Audits

The implementation is not yet audited.

Testnet

The implementation has not yet been deployed to Testnet.

Copyright Waiver

Copyright and related rights waived via CC0.

[cryptovestor]

Thanks for your work on this Ariel!

It is unfortunate that the solution to the lack of liveliness on the query rebate side of the network is to back-load the indexerRewards in the transaction lifecycle. This means Indexers lose some abillity to run a lean operation in terms of gas costs - an ability which might be critical for some Indexers in the current market conditions (relatively high gas costs, relatively low query rebates).

I absolutely see the importance of introducing this change, but wanted to highlight how it might impact Indexers behaviours/business today. Keen to hear from Indexer running lean operations.

[Fattox]

Thanks Ariel, it’s always nice to see a new idea of yours brought to the table.

And I like this idea on the surface, but have some concerns about how this could affect indexer dynamics a little.

Maybe i’m misunderstanding some details, could you elaborate on a few points:

1. Gas Costs

From what i can tell, this would have a net cost increase on closing allocations of <5%?
If so, not a huge concern imo.

2. Dependance on Traffic

*(Regarding the requirement for queries to claim rewards)*

One of the best ways for an Indexer (especially those already against the odds due to size) to run a feasible operation is to stay on top of the ever-growing list of subgraphs on the network, and allocating accordingly and early.

When i say “accordingly”, i mean in the context of remaining profitable, by paying attention to metrics such as the Signal to stake ratio.

One concern i have here is that these Indexers could actually be punished in some cases by acting quickly or dynamically to changes in signal or new subgraphs.

For instance - what if the subgraph gets signal, but no query usage beyond that point? Or so little traffic, that the gateway never sends any of the ‘handfuls’ of queries through to them (gateway preference or sheer bad luck). The latter partly concerns the behaviour of the gateway selection i guess, but that’s another matter.

One thing we saw in the past was Indexers allocating without first syncing a subgraph and failing to be in sync before being closed by maxallocation/0x0 actions. For that, we have a solution (offchain sync), and your punishment for hoping to benefit before being synced is greeted with “tough luck”. That’s fine.

But now in addition to this - and the need to pay attention to signal - do you think smaller indexers may also feel they could be stuck waiting to observe queries/fees coming in on a subgraph, before allocating to said subgraph? It seems they would either need to confirm usage somehow, or risk allocating to something that becomes a zombie sub with merely some signal and no eventual adoption.

And what if a subgraph breaks before receiving fees? This is no fault of the indexer, and they could be following legit signal, yet a fault in the subgraph code may see their time and fees wasted merely for being quick to act and having faith to allocate and support that subgraph. At least at the moment, there’s some recouperation in place for their effort via rewards.

TLDR - I think some of the dynamicism, attentiveness and goodwill we see from some Indexers could see them negatively impacted, should a subgraph be deployed and never used (or used very little).

3. Centralization Risks?

As @cryptovestor said, i too feel that this could impact smaller operations more. Some points being listed above, and others come to mind. But i would summarize the more glaring points as something like:

• Gateway and thus traffic preference - Sending queries to larger indexers first, is how i understand it works for now? - if so, this aspect already exists regardless of this proposal. But some situations could make it more lethal in some cases (zero payouts due to low/zero query load).
• Smaller stakeholders unable to risk supporting ‘unknown’ subgraphs until usage/adoption is proven, due to the walled-garden effect presented by ‘risk vs fees’ in a potential zero-payout (zero query) scenario. This excludes them by default from many opportunities for profit in the network.
• Less dynamic Indexer behaviour overall due to the above, means less support on new subgraphs, which is a downside for the health of the network. Less redundancy, potentially slower bootstrapping and/or overall query support, etc.

I feel these points would lead to situations that only impact smaller indexers, or by far the most, and just contribute to a potentially compounding effect on centralization.

I would also be wary in general of changes that may further confine the borders in which the vast majority of Indexers are able to operate within, since we have thus far made no strides beyond discussion towards stake decentralization.

4. Curation as Signal, or?

As mentioned above, should Indexers wait until others take the chance on a subgraph and confirm usage, before allocating to it? If so, what becomes the purpose of Signal? Or is confirmation of query load the task of Curators now? And how is adoption (not only 'intent of usage' via Signal) to be proven, before the initial Indexer adoption, in any case?

==========

I guess most of the above concerns would be alleviated if the ‘must have at least 1 query’ part was left out.

Some thoughts on that also:

• If it’s worth serving queries, and traffic exists, the traffic will be served on a subgraph that attracts enough live allocations, via the intended purpose of Signal.
• If it’s worth it - and/or made part of the closing process (in this proposal) - to claim query fees, they will be claimed.

… Right?

Some Indexers already serve for free indirectly by not bothering to claim, because they deem it not worth the gas overhead (or can’t claim, broken sub, etc). These are still paid for by the end-user, of course.

And i think it also demonstrates that query fees, at least for now, are not the only reason that Indexers are supporting this traffic. And we can’t claim that this behaviour is only due to rewards, because those come in independantly of your service being open for traffic.

So which issue are we trying to solve most here? Claiming fees, or supporting traffic? It feels like both, the latter being a side-effect. As stated said in the OP "end up with a lively network". But it feels like the solutions to each of these things could be at odds with one another, or at least present further cause-and-effect considerations.

I’m kind of at odds, because i think trying to make query support a required action is not inherently a bad thing to impose. Just that certain situations with this model could have the biggest negative consequences on those with the smallest leeway.

If i’m missing some points that put aside these concerns, i’d appreciate some feedback. I’m sure i may have overlooked some points, but hopefully there’s some weight to most of what i have said.

[grassets-tech]

Hi Ariel and All,

First of all great to see new suggestions to improve protocol experience, specifically for indexers. Especially query related, as this has been under radar for a while. Hopefully that means more queries to come
Overall the idea is great and objective and we support the effort to develop mechanisms that encourage all indexers to not only index but to serve queries for a much as possible.
By all means, Query Traffic is a cornerstone of The Graph protocol and indexers who does not serve queries are not genuine as should be discouraged to be part of the protocol.
Although we truly believe the idea is great, we have major concerns regarding the current proposal:

• Curation
  The signal curated to the subgraph is perfectly designed mechanism to bootstrap the go-to-market for a new subgraph/protocol entering The Graph Query Marketplace. A trustless and effective mechanism to get indexers onboard providing decentralized service.
  As @Fattox also highlighted, the role of the curation and signal may lose to its fundamentals.
  Unless the signal represents the amount of GRT pre-paid for traffic (just an example), it will be losing its current role for indexers as the reliable indicator that the subgraph is worth indexing. As the consequence for subgraph developers it could mean slower adoption, back to traditional go-to-market channels or even choosing a centralized service.

• Decentralization
  This is something so important and fundamental not only to The Graph but to the whole Web3. There is no need to emphasise the importance of the decentralization, but just and observation:
  Recently enough, some of the large indexers announced that they will subsidise query service by allocating to all the subgraph on the network, which is great and generous. But smaller indexers may not effort it and even more, as large indexers are subsidising query fees, what is left for the smaller ones ?

• Operational Costs
  Especially small indexers, and may be even some medium indexers may experience increased cost pressure. As @cryptovestor
  perfectly said – we may lose ability to run lean operation. And unfortunately it’s not <5% of gas cost increase as @Fattox mentioned, it’s also increased frequency of reallocation and opportunity cost implications by allocating to the subgraph which may have a strong signal but may not deliver enough queries to everyone who allocated or not at all for a period of time.

For example. Highlighted subgraph is UMA, it used to be fairly busy in terms of query… And back to the previous points, we may see raise to the bottom for the query fees to get some query traffic.

Once again, we understand the topic is important and the proposal should facilitate the migration, but have concerns that we believe worth considering.

Sincerely,
grassets-tech team

[aderks]

I agree, at first the concept is welcomed, but under the surface lies a bunch of concerns. Whole heartedly agree with Jim, Gary and grassets concerns they brought forward.

As an indexer that tries to run a lean operation, we’re always on the look out for the best subgraphs to allocate to. This usually means allocating to 5-10 smaller subgraphs at a time, that may serve queries. Our indexer is one among the many that are currently serving queries for “free”. Why are we serving queries for free? See below for our redemption list for 23 allocations:

0.00006, 0.02161, 0.00004, 0.00011, 66.14445, 0.04076, 69.26446, 0.00298, 65.28008, 0.000352365863720152, 15.073066037363190532, 0.014364071698426673, 0.00002, 0.016179047612100701, 8.527449827474131129, 0.003824522284304093, 7.499420292823071733, 8.50111700408674684, 9.277489038993155282, 0.004920007696095281, 5.939274236987069928, 0.669361404041231612, 2.051954569233608946

As you can see these are mostly low amounts and from a business standpoint it doesn’t make sense to claim these individually. Now that there is the addition of batching claims, I feel having the choice to redeem queries fees should be left up to the indexers discretion.

It seems too many issues are trying to be addressed within this new addition of closing/claiming, essentially holding indexer rewards hostage and diminishing a indexers flexibility. As Gary mentioned, I think we need to take a step back, re-evaluate and ask ourselves what are we trying to solve here?

• Completing the indexer rewards lifecycle?

  • If this is the case, only costing an additional <5% GAS on close and saving GAS in the long run, seems like a reasonable addition to the closeAllocation function. Holding indexer rewards hostage and implementing this additional risk doesn’t seem to support the basis of this issue, in my opinion.

• Supporting traffic? Optics that queries are in fact being served?

  • I believe traffic support is there, it’s just not being met with a measurable metric. There is a measurable metric, but that only comes with redeeming query fees. Maybe we need additional count metrics that show queriesServed per allocation and totalQueriesServed?

• Penalizing indexers that do not serve queries?

  • If this is the root issue, then it seems there could be alternate options to deny indexers their rewards that are not open to serving queries. What about some mechanism in place to check/test if an indexer is open and capable of serving queries before allowing an allocation to a subgraph? If they fail this capability test, they can’t allocate until the issue is fixed. Even enforcing a global default cost model, prior to allocating could help in this scenario. It seems counter productive to allow indexers to allocate, earn rewards, and then endure the risk of losing it all based on the potential of serving 1 query.

Alongside the concerns brought up in the previous posts, what would happen with allocation disputes? This new function is gameable in the fact that indexers could just send a query to themselves to ensure they receive their rewards. Can this be disputed? If not, then what’s stopping me from sending 1 query to ensure my indexer/delegators receive their rewards?

Curious to hear more opinions on this matter.

[Mike-LunaNova]

Thank you for this proposal. We echo the sentiment of previous posters that it is good to see new suggestions which may improve the network.

The main outcome of this proposal is that it will force indexers to claim their query fees if they want the indexing rewards. This implies that it may be some time before we see sufficient query fees on the network to motivate indexers to claim query rewards without further incentive. This is disappointing, however we, like many here, are here for the long-term and realize “Rome was not built in a day”.

Shifting indexer attention to the query side of the network would indeed be a good thing, as this is it’s raison d’etre! However, as others have articulated, this proposal risks further handicapping small players and accelerating centralization.

As aderks has noted, one undesirable outcome could be that we end up with smaller indexers sending fake queries to try and realize Indexing rewards on subgraphs they’ve allocated to. This is similar to indexers self-signalling on subgraphs to try and boost their rewards. Whilst neither of these behaviours are expressly forbidden in the protocol, they do strike us as inappropriate, and we would prefer not to have an environment where indexers are motivated to behave this way.

As it stands we would probably not vote for this proposal. However removing the requirement on Indexers to have at least served 1 query could make it more workable.

Alternatively, refining the requirement so that Indexers only lose rewards if they were specifically requested to serve queries on that subgraph but served none, would be an interesting approach. This is unlikely to be straightforward to implement but would at least prevent smaller Indexers being punished if the reason they served no queries was because of insufficient query traffic or insufficient stake to get selected.

[ellipfra]

This proposal has several downsides already well explained above, I’ll only point out that by increasing the minimum cost cost of allocations (requiring redeem and claim for low query volume subgraphs), we will incentivize indexers to only pick allocations with large signal, and to maintain a smaller number of allocations. Increasing this cost goes in the wrong direction, we want indexers to open 100x more allocations.

I perfectly understand that we want to make sure that indexers do their #1 job, which is to serve queries. In this bootstrap phase, indexers have really little economic incentives to perform well on this job. We currently have to rely on the good faith of the indexers, and their commitment to make the decentralized network a success. But we forget that this is not the only part of an indexer job that relies on their good faith: for instance, maintaining honest reward and query cut has been identified at the beginning of mainnet. The community seems to be doing a good job at identifying bad actors in this regard, it built dashboards and educated delegators to avoid bad players. We need to do the same for queries: build dashboard and rank indexers on their query performance: not just the claimed GRT amount, but also user impacting metrics like error rate, chain head, and latency. Peer-pressure and delegations will then have its effect.

One of the reasons we might be considering this change is to close the GRT loop which will allow, among other things, to pay the curators. To resolve this problem in this bootstrap phase, an alternative solution could be to run a temporary and subsidized service to call redeem and claim on behalf of the indexers. Some engineering is required to collect the vouchers, but it seems to me like a good band-aid to a short-term bootstraping problem with little risks.

[DataNexus]

I appreciate everyone who has shared ideas so far on this topic. While I mostly share a similar sentiment I would like to get a little bit more information.

Do we have any way to see out of the 142 currently allocated indexers which have actually served 0 queries over the past 28/56 epochs? There is something to be said if a large percentage are ‘rent seeking’. The proposed mechanism would enforce either good behavior or phase them out of the network (the later would increase existing indexers share of rewards).

[chris]

I recognise the importance of achieving the objectives that this GIP sets out. While increasing the minimum cost basis for indexers is undesirable, it’s vital to ensure the complete allocation lifecycle plays out and query fee volume information settles on-chain so that both observers, and the protocol itself, have an accurate view of query activity.

One issue that hasn’t been highlighted yet is that the Gateway currently facilitates query fee payments using a trusted key. As far as I understand, the AllocationExchange contract authorises this key to issue query payment vouchers to the Indexer, which can then be redeemed on-chain for GRT query fees.

If this GIP were implemented with the requirement of non-zero query fees, it would grant that key the ability to govern the disbursement of indexing rewards to Indexers. This change substantially increases the trust assumptions attached to that key, giving it the power to censor indexing rewards.

[ari]

@chris thanks for bringing that up. This is an accurate observation, the gateway today have some discretion via both the indexer selection algorithm and the signing of vouchers to influence who receives queries. As more entities run a gateway that factor should dissipate. It’s important that the selection criteria to be clear, auditable and public for all indexers to review.

[ari]

@ellipfra this proposal intended to be a way to use an in-protocol mechanism to get the query fees signal back to the network and ensure it self-sustains with no external intervention.

I agree that a different idea using extra-protocol incentives could be to provide a service where indexers post the vouchers to get redeemed. It could even have some logic to only process the ones that collect less than the voucher amount. The transaction cost subsidy would require some entity, possibly the Foundation, to fund the service.

[Brandon]

Thanks for writing this up @ari!..I’m strongly in favor of the main thrust of this proposal, but echo the sentiment that others have expressed that we should seek to remove the requirement for positive query fees and reliance on the Gateway.

To add some color to the discussion above, this proposal isn’t so much about query fees not being enough of an incentive, but rather making the fixed costs of serving queries as close to zero as possible. This makes it economically rational to serve any positive amount of queries, and thus makes rational self-interested Indexers (all you altruists aside ) more responsive to query fees, which we should expect to result in higher quality of service for the network.

In terms of the effects on smaller Indexers, it’s true that aggregate costs to Indexers would go up if Indexers didn’t change their behavior in response to the protocol change, but I find it more likely, as @ellipfra noted above that we would see fewer Indexers per subgraph. This prediction is described by \text{Eq }(20) from Prysm Group’s PoI Spoofing Analysis which relates fixed costs to # of unique Indexers as follows:

\text{# of unique Indexers} \propto\sqrt{\frac{\text{indexing rewards}}{\text{query fixed costs}+\text{indexing fixed costs}}}

So while it’s true that this should increase the fixed costs per Indexer allocation, I wouldn’t necessarily expect the overall costs to Indexers to increase as Indexers adjust their number of allocations accordingly.

As @ellipfra noted, fewer Indexers on a subgraph is undesirable, all other things being equal. However, in the context of this proposal, I believe that it is better to have fewer Indexers on a subgraph that are all fully responsive to the query fee incentives, rather than more Indexers per subgraph that are largely indifferent to the query fee incentive, given the large step up in fixed costs to collect fees for even a single query.

[ari]

Thank you all for the feedback about the proposal.

From the comments, the main concern is that an honest indexer doing everything right to serve queries, under some circumstances, will not receive any query fee, whether because of the indexer selection algorithm not picking it or the subgraph is still in the early stage.

I’m sharing an update to the original proposal with some changes to allow an indexer to self-sign a zero query voucher if they didn’t receive any query from the gateway.

Allow indexers to self-sign an empty voucher.

Indexers need to be able to collect a voucher to unlock indexing rewards. To break any relationship with the gateway, they could self-sign an empty voucher when they cannot get one with query fees.

It requires:

• Deploy an updated AllocationExchange (let’s call it VoucherCollector ) that only accepts zero-fee vouchers.
• The VoucherCollect will use the interface for redeeming as the AllocationExchange.
• The VoucherCollector needs to be different than the one holding the funds. The former is only for self-signed empty vouchers.
• Allow the VoucherCollect as an asset holder in the Staking contract so it can call collect().
• Configure the VoucherCollector with a shared private key that any indexer can use to sign their empty vouchers.

Allow the Staking contract to collect zero query fees.

• Update the Staking contract to accept calls to collect with query = 0. This function currently reverts under that condition.
• Whenever collect() is called on an allocation, flag it to be eligible to release indexing rewards.

Release rewards only if the indexer called collect at least once.

• During a call to the rebates claim, test if the allocation had the collect function called, then release the indexing rewards if true. The number of query fees collected does not matter, as zero will be an accepted value.

Additional thoughts

The VoucherCollect matches the same interface of the AllocationExchange to make it easier to reason and use by clients.

Also, redeeming from VoucherCollect needs to consume roughly the same gas as calling the AllocationExchange to avoid defaulting to the former when query fees are too low.

Finally, another way to design the VoucherCollector is to just create a contract that supports both types of vouchers (self-signed and gateway signed) to avoid adding more moving parts.

Open for comments!

[chris]

This addresses my primary concern (indexing rewards censorship) and I’m now fully supportive of this GIP. Thanks Ariel!

[cryptovestor]

I am still on leave mostly, so would like to lean on some of the Indexers that provided excellent feedback on the original proposal, so we can try and get this proposal moving quickly. Gary is out at the moment, would love to hear from @grassets-tech @aderks @Mike-LunaNova @ellipfra @DataNexus (even if it’s just a one line approval based on previous input, lets get this one moving.

[DataNexus]

I’m in favor of combining the claim() with the closeAllocation() funciton. As long as indexers are able to send themselves an empty voucher then I don’t have any issues with the minimum of 1 query for collecting rewards & QF.

This has an obvious benefit to curation, but on the indexer side I think it will help weed out some of the ‘minimalist rent seekers’. This proposal better defines ‘minimal work to be compensated’.

[Mike-LunaNova]

Thanks for the tag @cryptovestor. Apologies for the delay in replying, I too have been away and just got back.

Ariel, thanks for your hard work on this further modification. It addresses a concern we raised and makes the proposal significantly more viable.

Is the plan for the indexer-agent to automatically sign zero query vouchers where required or will Indexers have to do this manually to successfully close allocations on subgraphs that they received no queries for?

[grassets-tech]

Sorry for the late reply @cryptovestor and thanks for tagging.
Many thanks @ari for listening the feedback and working out the solution.
We fully support the GIP with the proposed mitigation via self sign of vouchers when otherwise not available.

[Oliver]

Seems like a good time to resurface this proposal in light of the broad consensus, especially from the Indexer community. Is there any update on the progress or expected timeline to advance this in the governance process to a Council vote @ari? Thanks!