Subgraph to monitor Indexer KPIs

Community & Grants Grants & RFPs
1

Hello,

I find the ability to select a proper indexer to delegate to pretty difficult for Delegators as they are many metrics to consider, including past performances and actions. In the Discord channel many Delegators are frustrated for having chosen the “wrong” indexer.

I feel that more than everything, there is a lack of transparency on an Indexer’s behavior for Delegators to make an informed choice, and what better way to solve this issue than a Subgraph? The “Graph Network” subgraph does contain some details, but they are not so easy to extract or manipulate.

I would like to apply for a grant to develop a specialized subgraph that monitors various Indexer KPIs, ideally building a type of overall “Indexer Trust Score” that would include various metrics:

1. Historical change of fee cut and frequency:

This would monitor all changes in Query Fee Cut / Reward Fee cut. Small and punctual changes might be fine, but significant and/or frequent change (eg: moving from 10% to 100%) should be a red flag.

2. Unallocated Stake:

Indexers keeping an important percentage of unallocated delegated stake for two long should be a red flag.

3. Allocation closing frequency:

Indexers can chose to close their allocations whenever they want, but Indexers closing it as a higher frequency (some are doing it daily) are making an effort (financially with gas cost at least) to bring some certainty to Delegators on their rewards. This should give some bonus points.

4. Number and quality of subgraphs indexed:

Indexers indexing many subgraphs are more reliable from a Delegator perspective (A Delegator is less exposed to drops in query fees on a subgraph) , and they should follow the curation signals to maximize indexing rewards. A significant deviation from it (eg: indexing subgraphs with no usage), should be a concern for Delegators.

5. Financial reward

APY is of course an important metric for Delegators (past and current performance)

I might have more ideas of metrics and KPIs that could be added along the way. Do you see any other important metrics? Would you support such proposal?

Cheers

2 Likes
2

Hi Matheos,

I believe a well thought out and detailed proposal such as yours always deserves feedback so I’m trying to give you my honest opinion on your idea as no one has responded so far, unfortunately.

I like your idea. In my opinion, it could be important to document the KPI’s you have mentioned. One difficulty I see with your proposal is that delegators may find it difficult to navigate a subgraph and extract information from it. I’m not sure about your technical implementation and whether or not you have plans to somehow visualize the data from the subgraphs. But this could be an important step towards encouraging delegators to use what you have built.

You’ve also asked for other metrics to monitor. If you think about the recent P2P incident, it would be interesting to monitor similar situations that have happened.

However, some of the things you describe can also be found on some of the websites that are built around the subject. Historical change of fee cut for instance can be found on Grafana.

1 Like
3

Ultimately, Eva at The Graph Foundation will have to assess whether or not your proposal is a great contribution to the community and should receive a grant. I’d say, have a go and submit your proposal.

4

HI @SpaceTraveler,

Thanks for the feedback, appreciated!

Indeed I agree and though about it as well in the meanwhile: as the Delegator role is not very technical, a user interface would be needed to visualize the data. I will add that. A clear documentation of the KPIs and methodology would be indeed quite important.

I really like all the tools made by the community and in particular the Graphana one. They are a good source of inspiration, I am just thinking as most of these tools are made by Indexers, there is a little conflict of interest that worries me.

To summarize, I will add this:

  • Add a web user interface on top of the subgraph
  • Add clear documentation on the KPIs and methodology
  • Check the list of KPIs used in other community tools and analyze past incidents to see which KPIs are missing.

Cheers

1 Like
5

Hey @matheos this sounds like a really great idea. Over time the community may converge on a measure for Indexer reputation but I think it’s very positive to have multiple implementations and discussions around how to calculate this and agree a subgraph is the perfect tool to calculate reputation.

Seems like a clear value add for the community. Thanks for sharing your proposal for input! Would love to hear from others what other factors they would want to include in Indexer reputation and how they would weight the different factors.

2 Likes
6

Hi @yaniv ,

Thanks for the feedback!

I think we can start with just raw KPIs as above, and later introduce a weight for these KPIs to compute an overall indexer reputation score.

I believe there are different profiles of delegators and they would have their own weights, but yeah it would be great to have some community inputs on what weight they would give on each KPI!

7

This is super cool thanks for sharing!

8

No problem! It’s a pretty useful site.

9

Hi there,

For anyone following this thread, I implemented a first version of the proposed subgraph here with some of the metrics => Indexer KPI Subgraph

You can see for example which Indexers are changing their parameters and how often (the maximum I saw is 16 changes over the last 30 days).

Metrics implemented:

  • Indexer Background:
    • createdAtTimestamp: UNIX Timestamp when the Indexer went live
    • vesting/managedAmount: Amount of GRT received as part of the official testnet program
    • vesting/beneficiary: Ethereum address of the beneficiary of the vesting contract
  • Indexer Parameters
    • ownStake: GRT Tokens staked in the protocol by the Indexer itself
    • delegatedStake: GRT Tokens delegated to the Indexer
    • allocatedStake: GRT Tokens allocated by the Indexer in the protocol
    • allocationRatio: Ratio of allocated tokens versus allocation capacity
    • maximumDelegation: Maximum GRT Tokens that can be delegated to this Indexer before overdelegation
    • delegationRatio: GRT Tokens delegated to this indexer versus maximum delegation
    • isOverDelegated: Flag to indicate if this Indexer is overdelegated
  • Delegation Parameters
    • indexingRewardCutRatio: Indexing Reward Cut Ratio (between 0 and 1)
    • queryFeeCutRatio: Query Fee Cut Ratio (between 0 and 1)
    • parameterUpdates: List of all parameters update
  • Metrics of last 30 days (rolling period - experimental)
    • lastMonthParametersUpdateCount: Number of updates on the parameters over the last 30 days
    • lastMonthDelegatorRewardRate: Reward per GRT token delegated over the last 30 days

Any feedback is appreciated!
Cheers

3 Likes
10

Nice work! I look forward to seeing some dashboards spawn from this! @stakemachine :upside_down_face:

1 Like
11

Definitely will take a look :nerd_face:

12

Speaking as a delegator, it would be very nice to have an info dashboard that had the official approval of the foundation. Right now there are a ton of useful tools that have been built by the community, but it can be pretty tough for the average person to keep track of them all. The proliferation of dashboards may also lead to a proliferation of phishing attacks- especially since it is commonplace for dashboards to have a “connect wallet” function. I believe I have already come across one such malicious “rewards calculator” that’s made the rounds. It seems likely that it is simply an IP address phish (no wallet connect feature, thank goodness) made to evaluate juicy targets for hack attempts. These scam attempts will only get more sophisticated as the value GRT rises and becomes more attractive to steal.

While I don’t think anyone wants to have fewer tools, there are definitely downsides to such a decentralized information regime. If there was one officially official information source, the other sources would just fill in the gaps, or provide a more granular level of detail.

This is on topic, ffs