Request for information about disputes #GDR-8

The arbitrators are contacting indexer address 0x8e297141f9dc76f4d71741da1a1e47294cf0fde2 (0xfury) about disputes filed in the protocol.

To make the investigation as brief as possible, please provide the following information and any other relevant records about the open disputes:

  • Version of graph-node used.
  • Graph-node configuration in full.
  • Type and version of Ethereum node.
  • Table of PoIs generated by the affected subgraphs.
  • A dump of the call cache data from the database for the affected subgraphs.
  • Entity values as of the divergent block once identified.

This is not an all-inclusive list. Request for additional information may be made if considered necessary.

How to Get Relevant Data

You can use the following queries to extract information from the indexer software to be used in the evaluation.

# Get subgraphs schema
SELECT name FROM public.deployment_schemas where subgraph = 'QmTj6fHgHjuKKm43YL3Sm2hMvMci4AkFzx22Mdo9W3dyn8';

# Move to the subgraphs schema
SET SEARCH_PATH TO <RESULT_FROM_ABOVE>;

# Dump PoI TABLE - (from outside of psql console but on server)
pg_dump --dbname="<YOUR_DB_NAME" --host=<DB_HOST> --port=5432 --username=<USERNAME>  --table='<SUBGRAPH_SCHEMA>."poi2$"' --file='<FILE_TO_SAVE_TO>'

# Dump call cache (on some setups this table may be in a different schema, check it with `select * from public.chains`)
pg_dump --dbname="<YOUR_DB_NAME" --host=<DB_HOST> --port=5432 --username=<USERNAME>  --table='public.eth_call_cache"' --file='<FILE_TO_SAVE_TO>'

Once a divergent block is identified:

# loop through all entity tables and get changes for that block
# for each entity table in subgraph deployment schema:
select * from <entity_table> where lower(<DIVERGENT_BLOCK>);

Purpose of the Requirement

This requirement is related to the following disputes:

├─ 0x9c02325bbec827fdeaa3d3b988183667b4d3a48c0cd51ebd4b91d3f8e162be28
│  ├─ Type: Indexing
│  ├─ Indexer: 0x8e297141f9dc76f4d71741da1a1e47294cf0fde2
│  ├─ Fisherman: 0xec022aa5960c97764237f3dde3ed4065dd7ecf2f
│  ├─ SubgraphDeployment
│  │  └─ id: 0x014e8a3184d5fad198123419a7b54d5f7c9f8a981116462591fbb1a922c39811 (QmNRkaVUwUQAwPwWgdQHYvw53A5gh3CP3giWnWQZdA2BTE)
│  ├─ Allocation
│  │  ├─ id: 0x4437a474ccaebdaecf6f3f6073efdf81cfe4b390
│  │  ├─ createdAtEpoch: 203
│  │  ├─ createdAtBlock: 0xa39ee498b3a6ed72d7b9c62bc95c3eb6e0ee913d5cbe3f82be0b6f095b19e7dc
│  │  ├─ closedAtEpoch: 230
│  │  └─ closedAtBlock: 0x5149b5bd19ab5e63582e6d71bbaef6fc10f1b6bb7532fd1e0b8d3e1cdfd4e28f (#12980970)
│  └─ POI
│     ├─ submitted: 0x45e611f051ae4cba9d9570b07d3228a4e06d2fccab03a80f57cfe176fe0a20ab

About the Procedure

The Arbitration Charter regulates the arbitration process. You can find it in Radicle project ID rad:git:hnrkrhnth6afcc6mnmtokbp4h9575fgrhzbay

For communications, please use this forum. Additionally, please monitor Graph Protocol’s Discord #arbitration (https://discord.gg/9gg7XvfggW ) for any arbitrator reaching out for more information.

Just as an opening note - I have been travelling this week (and the next) and informed one of the Arb Council members (Ariel) that i would be unable to respond to this dispute in a timely manner, due to this. I also asked if the above request for data was set in stone, due to some concerns i had, and was informed that (no) this thread is merely created as a catalyst for discussion.

So i will start the discussion, by asking - what is the basis for this dispute?

  • My POI was cross-checked against 2 other Indexers, and appears to be valid.
  • The POI in question also matches the POI submitted for my previous allocation to the same subgraph (as the subgraph had errored in a prior epoch).
  • A dispute was not raised against the previous allocation.

The points above lead me to believe that the POI itself is unlikely to be the reason this dispute was raised. I imagine that instead, most likely, this dispute was filed due a clash of perceptions over the Arbitration Charter. Due to witnessing these perceptions play out in public discourse, i had already anticipated (and prepared for) a dispute being submitted against myself.

Maybe the Fisherman would like to clarify the motivations behind this dispute, so i’m not left to make assumptions and present a misleading defence or redundant information to the Arb Council?

1 Like

Alright, as the fisherman of this dispute, let me make it more clear.

This dispute belongs to a group of disputes:

https://forum.thegraph.com/t/request-for-information-about-disputes-gdr-5/
https://forum.thegraph.com/t/request-for-information-about-disputes-gdr-6/

In which the point is that you have opened the allocation after the subgraph has been broken.
It means that you simply could not generate a valid POI by default and even “Last epoch before a bug” doesn’t work in this case.

According to the community consensus, this is not allowed, otherwise, all indexers would index Omen.
Even guys who agree with you, based on likes, didn’t allocate on this subgraph despite on much higher profit than average.
As you said, you knew about:

  1. This subgraph is broken.
  2. Other community members have strong opinions about this.
    You intended to get more profit despite these points and didn’t say publicly, why do you think that it’s ok.

You can’t refer to the different versions of the Arbitration charter as Inflex, because the main point is simple: “All indexers should work honestly”. Arbitration charter just helps to understand what is totally prohibited and what is not so good to do. Did you make your job honest, or did you just rush for a profit with some confidence that it will not be a reason for slashing? I can’t say for arbitrators, but from my point of view, you should be slashed.

The graph team has seen all these discussions in Discord and on the Forum, Brandon and other team members were tagged several times and didn’t say No or Yes. I suggest they leave this question to the community or the arbitrators, avoiding direct answers. This uncertainty was the reason for all of these 3 disputes, we could avoid this situation, with an official position, but we didn’t get it before. Let’s get it here.

If this case is not the reason for slashing, please provide us examples of where POI could be this kind of reason? Because for now, it seems, that we don’t have any possible cases to be slashed for different POIs. Why do we have this type of dispute then? Moreover, any indexer can open an allocation and commit any random POI if it doesn’t represent anything.

As an indexer, you have two money streams: One of them for “indexing”, which you should “Proof” to get Indexer Rewards, and another one for serving queries. The subgraph is broken and you are not “indexing” it, you indexed it more than a month ago, how logically you can provide Proof of Indexing and get Indexer Rewards?

Please, let’s make a final decision here. Both conclusions will be fine. If it’s legal, just clarify this and allow everyone to get these rewards.

2 Likes

Also you spoke about “The previous allocation” (I guess: 0xd647beb09318f085c4297af94f82202e9b3776e7). This allocation was opened before the issue with this subgraph. And the community was agreed that cases like this shouldn’t be slashed. As you may know, for example, @indexer_payne was in the same situation and he closed the first one with the last valid POI, which was opened before the issue, and the second one with 0x0 as many other indexers.

Our second allocation was due to the agent ignoring the rules and not reconciling, so basically a mistake :sweat:

But you still closed it manualy with 0x0 because it was the right way, isn’t it?

While i’m not the subject of the dispute, nor the instigator, as fellow indexer and stakeholder in the protocol, i do see it important to react regardless and point out some inaccuracies.

Let me point out that this is a misrepresentation. There is no consensus on this topic, definitely not in the sense of “this shouldn’t be allowed”, from following closely i’d say it’s leaning the opposite.

Or maybe, it’s about following the signal, that has INCREASED, despite being “dead”. Following the network signal is what indexers are supposed to do. And as such, yes, honest.

Ford can be seen, in his personal opinion, quite clear. E.g. Discord

It’s not random AT ALL. Seems you’re making it grotesk in an attempt to prove something that’s not there.

Now this i agree with, the unclarity isn’t helpful to the community.

3 Likes

Looks like consensus from my point of view.

1 Like

Not sure how you derive “consensus” from that ? And your screenshot is scrolled FAR down. In the last week 10 new allocations were created, of which 1 was closed next epoch with a 0 POI, the rest remains open. What does that say about where any consensus might be at ?

Since it’s not my indexer being disputed, i’ll leave it to Fattox to argument his case, but the cherrypicking / misrepresenting from the fisherman is baffling.

1 Like

Other indexers haven’t allocated in Opyn because of old version of the Charter. Basically, missed profits due to miscommunication.
I think slashing person ought to be competent and be aware of the latest version of the charter. Thus, you shall be punished for your dispute and lose 10 grand.

2 Likes

I agree that there is no consensus on this being bad or malicious or whatever you think it is.

Based on the charter and signal on the subgraph and the fact that queries are being served i do not think this is a slashable offense at all. Many indexers are waiting for a formal resolution before they dare to allocate, but some of us already agree that this cannot be slashable offense give just the above two reasons alone, hence why we are allocating to the subgraph.

There is NO community consensus on this whatsoever, not the way the fisherman presented it. Lets stick to the facts not data manipulation.

2 Likes

I think he meant, that majority was agreed that this subgraph and allocation to it it’s some kind of grey zone. You changed your mind yesterday, after Inflex’s words (and I like his points), before that you didn’t allocate, right? I don’t want to quote your previous messages about this topic. The same thing with us and other Indexers.
We are not sure is it ok, or not, to allocate on this subgraph. That’s why we avoid allocating to Opyn, it’s too risky. At least now, before an official decision.
I’m really glad that I’m not an Arbitrator because it seems a really complicated case. I agree with both sides, and yeah, I think it wasn’t consensus about “We all agree that it’s bad” rather “Oh, it seems unclear, let’s avoid it to be safe” at least from our side.

1 Like

I already stated in the other topic (i think) that i was heavily against this before. I changed my mind after a convo on discord with payne a while ago, because its pretty clear that signal is there and queries exist so its not like this isnt doing what it supposed to (apart from not syncing due to broken subgraph).

I decided to allocate yesterday yes, because i expect this to be resolved soon so i can risk a few days before the decision is made. Although based on the charter this seems quite clear honestly.

1 Like

Hi @fisherman . Thanks for your prompt reply.

This will be a long post, and not entirely directed at you, but at the Indexer community too. As i feel i need to defend certain people and processes more than i need to justify my actions.

The waters are quite prone to becoming ‘muddied’ when money (rewards/revenue) is involved, but a large part of my motivations here were to make an example of certain things. Mostly the lack of attentiveness or critical thought/input given to this subject by some who (nevertheless) commented on it.

From my perspective, the higher-rate upsides of this allocation were only possible due to a widespread misperception over this clause. This is something i was surprised at, given that from my perspective, everything was quite clear. And i know we have plenty of highly competent, engaged people in the Indexer community.

Disputes & Precendence

There is quite a sensitive balance of power when it comes to disputes. The role of an Indexer involves attempting to cover every consideration that a Fisherman has, in order to avoid getting disputed, and then some. But the risk presented to the Fisherman is rather low compared to that presented by the stake of most Indexers.

For this reason, i feel that Fishermen should be held to the absolute highest standard. Not only because of the risk:reward often being slanted in their favour, but because any invalid disputes present ‘noise’, and demand a lot of time and effort for the Indexer and the Arbitration Council. I feel without having high expectations of our Fisherman, we could attract opportunists.

I’ll echo something i said in public not long ago. Again, in anticipation of a lot of the disputes we’re seeing crop up today:
1

Since we’re here now, i will try and make this reply thorough. Not only to defend myself, but to make this very point, and why i feel the majority were the ones at fault, in this instance.

By the end, i hope it may be clear that there was actually no need for me to defend my actions, and that this dispute is invalid.

Clause & Effect

Excuse the pun, but that's why we're here, it seems. A disagreement over the clause which allowed Indexers to take rewards. Oft cited, incorrectly, over the past 6 weeks or so.

For any viewers who are unaware. Here is the original clause:

If the Indexer is unable to produce a valid PoI for whatever reason, then they must close the allocation with a so-called “zero PoI,” which is a PoI comprising all zeros. For example, if a subgraph has a bug that prevents indexing up until the current epoch, then a zero PoI should be submitted and indexing rewards must not be collected for that subgraph. An exception to this rule is if the allocation being closed was opened before the subgraph bug occurred. In this case, the Indexer may submit the last valid PoI they produced for the subgraph and collect indexing rewards for the allocation.

So, it was quite clear in the past. If a subgraph has an error, you should not collect rewards for that. Whether this clause made sense is another matter. But this is where the need for discussion and input comes in to play - something which is often requested but rarely achieved at times, in matter of governance.

The Discussion

A common complaint around this subject has always been the need for more transparency. People saying they have been left with uncertainty over this particular rule. From my own perspective, i can't understand this, as the discussion which led up to this clause being changed was all public. As was the commit itself.

I feel it’s important that i cover this timeline in full, not only to paint a picture of my own understanding and rationality, but also because i often saw the Arb Council (or others) come under attack if they demonstrated their confusion when accused of “a lack of tranparency”.

Everything seemed entirely clear to myself and a few others, but a kind of ‘hive perception’ seemed to appear, and still be persisting even now, judging by the state of Opyn and some of the replies to these disputes).

From my own perspective, here’s how it went, in public nontheless:

  1. In the Arbitration Charter thread on this very forum, concern over the clause in question was raised by @That3Percent .

  1. After no response to the above, he later added:
    image

To me, his arguments made perfect sense, and was something i had considered lightly in the past. But having never ran afoul of this clause, i never took the time to put too much thought in to it. But there is a certain irony to the idea of dismissing historical data, when everything we index is historical. A subgraph error does not invalidate the prior data.

And more importantly - The idea that we should somehow step infront of Curators and the open market, by determining value in another manner like this, is not congruent with the importance we claim to place on the role and responsibilities of our Curators. It could be argued that in this case, the Curator community was not necessarily responsible for the majority of the signal, but this does not mean we should begin making exceptions which sidestep the function of this role.

  1. Some clarification to these ideas was requested by @KonstantinRM , to which Zac replied:

image

While Zac is not part of the Arb Council, his thoughts on this matter were at least clear to me, at this point. And made perfect sense.

  1. The topic was raised in Discord at some point, to which @Ford responded with his own thoughts.

image

  1. Followed by Zac adding some colour, to the balance of risk, as he saw it. As he states, he doesn’t speak for the Arb Council, but since they have been quite open about their interest in malicious actions, his opinion was that you could balance your risk against the perception of various courses of action, depending on how much faith you had in the Arbitrators:

image

Again, no input by this point from the Arb Council, but discussion is how we move things forward. And these (imo, strong) arguments were the only thing leading the way on this topic. To very little feedback, and no resistance.

But it’s important to remember that this Charter was always intended to be created via input of the entire community. E&N being just one part of that community, and save for a few other people, they were pretty much the only ones active in this particular discussion.

  1. After some time, @ari added his own thoughts on this.

image

  1. And presumably this was then further considered, because after this point, the clause was updated…

image

For clarity, here is what the original clause was replaced with:

If a subgraph encounters an error that prevents it from being synced up to a the first block of the epoch, then an Indexer should submit the last valid PoI for the block before the error occured. Indexers may continue to collect indexing rewards on a broken subgraph until the market indicates it is no longer useful to index that subgraph and removes all signal from the subgraph.

Hiding in Plain Sight

This is where a lot of the 'problems' seemed to begin. And as you see from the above, i'm not sure we can say in good faith that this was due to bad communication on the Arb Council's behalf.

The main issue seemed to be is that people were not checking Radicle for the current state of the Charter. But instead referring to an old HackMD link which only existed because people (for various reasons) were not referring to Radicle directly.

When the post by Brandon refers to the commit on Radicle, i’m not sure where else people felt they should be looking to as the source of truth. But somewhere along the line, people began taking HackMD as the source of truth, despite it being outdated at that point.

So should consensus be based upon this widespread misunderstanding, or the actual (current) state of the Charter, simply because most people overlooked it and kept sharing a version that had since been made invalid?

I think the answer is obvious, but even if you’d disagree (a point the Fisherman seems to be trying to now cling to), i’d like to see it argued that i was not behaving rationally, or that acting upon this updated clause was in any way malicious. Attentiveness to dynamic variables within governance is no different to an Indexer reacting to changes in signal.

Risk vs Reward

In all honesty, i did not notice this updated clause myself for the first day. It was only brought to my attention when @cryptovestor made the following post on the forum:

image

It was at this point my interest was piqued.

So i began to look in to the new clause. And once it became clear what it meant, i was confused - why have these gaps not yet been filled after a day? Am i missing something? Usually the Indexer community would act quite fast and such big opportunities for rational profit-maximization would not last so long.

After reading through these discussions again, I concluded that Jim’s point of “how these dynamics become irrelevant in the future” had essentially already happened - by them removing the former requirement, in which an allocation on a broken subgraph was required to be made before the current epoch in which a subgraph had an error, to be eligible for rewards.

And this seems to be the point at which many people had a mental block, even if they were aware of the updated clause. When real money is at risk, people second-guess themselves and wait for validation of the group. Plenty of psychological research backs up this phenomena, but even being aware of it, it’s a hard thing to get past at times when you have very real, substantial skin in the game.

So, i began to calculate my own risk. How long would it take to break-even versus my current reward rate, plus a potential slash, if i have indeed missed something and open myself up to that risk?

The initial answer was, about 9 days (at the current state of things). Though i also considered that i could 0x0 myself if things were not looking good at any point before then.

For the upsides, especially for my Delegators, who would not be at risk of slashing, it seemed a perfectly good idea even with some slight uncertainty remaining at the back of my mind. While this sheet was not perfect, it was a good enough way to re-calculate my risk on a daily basis.

The bigger issues was reinforcing my own confidence in my perceptions of the new clause, and i have @cryptovestor to thank for that, as we discussed it many times since the majority perception remained against my own, and on occasion left me feeling a little doubtful in the face of those misperceptions.

And this opportunity (gap) was not one that i expected to still be ongoing right up until the present moment. I expected that a number of things could happen:

  • Opyn could remove the signal for the time being
  • Opyn could migrate signal to a new working subgraph
  • Or (most likely) other Indexers would come and dilute the allocation pool of the subgraph.

Not that the latter would be a bad thing, as it would have validated my actions and removed the 1% of doubt that was playing on my mind.

I’m quite sure at this point, we will see many more allocations on Opyn, but as i said, this already went on far longer than i expected. Hopefully this explanation pushes things forward on matters of diligence, if nothing else.

Final Notes

I think i have demonstrated completely rational and justified action on my part.

More importantly, i hope i have fought the corner of some of those on the Arb Council, or even others (EG @Ford @cryptovestor ) who often took the brunt of frustration in public conversations, from people who held views of a perceived lack of transparancy. Which i felt, and feel i’ve demonstated, was not justified.

To be quite honest, this dispute aside, i hope that @cryptovestor received apologies from certain people who acted quite hostile to him over this matter. In this matter and others, he’s proven time and again to be a guiding light to the Indexer community and more than lives up to his role. But his guidance on this matter was ignored, unappreciated or outright dismissed.

Outcomes

I will close by saying that i hope to see the losing party punished, whoever that may be. Not only because i would like to see some 'absolute' outcomes happen in arbitration, and thus the diligence raised by all parties... but also because i edited this post on my phone. :dizzy_face:
5 Likes

Referring to the same charter, without exact citations, there are lines that say draws should always be preferred for the Arb Council.

@ari
it seems this dispute is stuck. I think it should be closed with a draw.
As well as Request for information about disputes #GDR-5 - #8 by ariel
Thank you.

@ari @Oliver @jannis @Brandon @Adam

Hello guys, as far as I remember there is 2 weeks limit for making a decision. This dispute was opened 5 months ago and there is still no decision and I can’t get my GRT tokens back.
Can you help me with that and close it and GDR-7 with “draw”?

We resolved the Dispute as a draw adapting the original interpretation of the charter with further discussion by the community and core contributors on the forums. As discussed on multiple occasions, we consider that there is value in serving old data. However, the Arbitrators suggest that the clause “9. Valid Proofs of Indexing for a given epoch.” should be reviewed, updated, and eventually put to the vote for ratification. In addition to that, a modification in the graph-node is needed so that the failed block produces a deterministic POI to make it standard across multiple indexing runs.