Hey guys.
First of all, I don’t want to dispute anyone or slash anyone. I want to understand the community’s thoughts on this, and what this Indexer thinks about this way of “working”.
So, it’s an Open Discussion, I didn’t create any disputes on it.
All of us understand revenue flows for Indexers, how they are designed and why.
Signals\Curation market is quite young yet and tokens signaled to subgraphs are not so huge now. So it’s really easy to exploit this for getting more rewards.
This open discussion about @suntzu, because examples are really colored from my point of view. I believe that some other Indexers also could use it, but maybe not so massive.
Why I open this discussion instead of DM: because if it’s wrong behavior I believe it will be better to prevent other Indexers from doing the same, that’s why I think it should be public.
TLDR: he creates his own, useless subgraphs, signals on them, and allocates to them. I saw it several times, but now it transformed into an even worse case. Abounded subgraphs with 0 signals get signals, after that allocation, but if someone also jumps in with allocation, Suntzu jumps out with signals and allocation to another abounded subgraph. So, from my point of view even the first part is quite gray, but the last part it’s totally black. Also, pretty important from my point of view, that this Indexer doesn’t index (or really rare) real subgraphs.
Long version with examples:
Suntzu has at least 3 addresses (but I believe he has more):
It’s a first part, quite gray because some work is definitely done, with 0 benefits to the network, but it’s directly not prohibited.
But yesterday I spotted this one:
Someone signaled to abounded subgraph (that’s basically ok)
And I think you know what I meant (Yes, we also allocated when spotted it, because with friends be somewhere much funny, isn’t it?). Suntzu didn’t jump out yet, because profitability is still 1.5x times higher than average.
I understand, that all of these, officially not prohibited. But from my point of view, it’s totally bad behavior, that provides 0 benefits to the Network and just exploited reward system.
What do you think guys? @suntzu your thoughts are also really interesting.
Sharing my thoughts here as I’ve been working to prepare disputes for a few of these allocations. (Ourz-mainnet, Ba1tD, Test Contract, Bot Bait v2). Since we’re openly discussing them I’ll hold off since we can likely resolve it.
I can confirm that 0x534a9ac3d3401f80800375680bded6a84137d296 belongs to SunTzu.
This address has multiple bogus subgraphs with active signal on it. These should be deprecated.
Another Indexer made me aware of this specific case a while ago and I was still trying to organise my thoughts on the topic.
There are many behaviours that can be conducted within the limits of the network but might fall under the banner of “bad behavior.” My personal opinion is that any behaviour that is net negative for the network and its inhabitants is eligible for a dispute, so that precedence is set in stone. It could be argued that this subgraph spamming and signal boosting activity is stealing earnings from Indexers that don’t choose to behave like this. Although maybe not on the same level of malicious behaviour, we do have precedent being set on previous disputes that include Indexer’s behaviour.
@suntzu, from my personal experience interacting with them since Mission Control, is not a bad actor. However Indexers are going to push the limits of the protocol and when the incentives aren’t yet balanced between staking and query fees, we’re going to continue to see Indexers push those limits. If they push them repeatedly, we need to act and draw the line if appropriate.
I think it is honorable that @KonstantinRM brings this up as an open discussion and I’d like to read more opinions. However the dispute process is there for a reason and I believe it should be allowed to perform its role, should someone choose to raise a dispute. And I think @suntzu long standing reputation since Mission Control should play into the decision of any dispute. I think the outcome could really go either way as it sits on a sort of “behaviour boundary” and this sort of ambiguity is exactly what I think the Arbitration process can help with.
First time posting - I allocate for Figment, so I was the actor in step 3.
I do agree that this action is exploiting those who behave as intended and that dispute process can help set precedence.
TBH, I have considered this strategy, after all, there was no clear rules that said you can’t. My understanding is that a person can do everything by themselves - develop subgraph (whether it be useful or not), signal it, index it, allocate to it, delegate to whoever allocate to it, etc. They can be any of the combinations of those roles, and that’s okay.
The part that wasn’t okay in my opinion was valid signals on useless subgraphs (and jumping ship quietly but that’s a natural consequence of this whole process). The purpose of signals becomes corrupted without indexers’ awareness. I do think useless subgraphs with signals should be allowed to exist, because I imagine we won’t want to implement extra measures that say if a subgraph is useful or not, or if a signal is valid. Experienced indexers should be able to look at the signal and whoever was involved to decide allocation, but then this is still asymmetric information. Aside from the signals, trust is required from indexers to indexers (at least in my allocation processes), and it was an issue in this case.
The good thing is that I don’t think this problem will persist when/if query fee becomes the dominant factor of allocation process… the power of data could potentially replace my need to trust
Man’s exploiting the rewards system, that lacks proper market dynamics. Don’t think any dispute shall be opened, and if it will, I will strongly advocate for a “draw” resolution.
If you observe someone squeezing higher profits for himself/his delegators - write a ten-line python script and dilute his allocations with the immediate following. If you are too lazy to do so / or your risk management is too prohibitive (gas costs, reputation losses, etc) - these are your problems.
The position “he takes more, meaning we get less” is similar to crying for mommy when you get hit. In a decentralized system and open market economy, you are supposed to hit back, instead of calling for regulation. Every actor is equal and only smart-contract is king.
I agree that the rewards system currently lacks proper market dynamics and we would like to keep a decentralized system with open market economy.
At this stage it might be beneficial for us to try to take on the perspective of system designers, who have the choice to make the system better, so that the environment itself encourages givers and matchers as well as discourage takers.
Imho there should be something we can do together, not specifically regulations, and not ‘everyone for themselves’…
From gathering up everyone’s thoughts, it seems like there is a feeling there are no rules surrounding the actions so how could there be a dispute. Clearly it seems the intention was there to benefit financially by exploiting a loophole in the system as shown by examples provided. Just a discussion of the behavior would more than likely not deter such behavior IMO.
The opportunity was there due to lack of internal controls or inadequate policies to cover malicious behavior. This activity or something similar would most likely repeat if not addressed. The bottom line was a wrong occurred and was discovered. There wasn’t a mutual benefit which is typical for work being completed (Entity benefits and individual is compensated). In this case, only the individual benefits with no benefit to The Graph. Why would we condone this type of action? I would think the action breaches the ethical behavior of the charter.
To further illustrate the indexer rewards that we paid for indexing garbage, this is the list of subgraphs published by Suntzu, signaled by his curator address, then allocated to starting since July:
Self Published
Indexer Rewards
Delegator Rewards
IPFS hash
Bot Bait v2
20,964.50
29,752.33
QmRavjdwiaU7mFWT7Uum28Lf6y6cm397z6CdZPpLcFj9iR
BOT BAIT
3,178.17
-
QmdjRnY9HrZ9B6tmiNZnbuiHkVeYLmdQjR3J3vRvqEBTpn
migrate 1
19,709.43
28,544.66
QmS7VGsn5s8UTMrebMRVNub2qCBYK19Qvg4dGNdTqsHX4k
Nirn
12,174.19
17,885.54
QmU3NicVe14LzQ3hNhyLQ1JCZu8uRyRjKejj1iD7oAKge1
Test Contract
33,626.28
46,526.75
QmRGtdrgcz5aZUpzM3nr9yhHa8QuQv2MMbb8xr2D3mbuoh
Total
89,652.57
122,709.28
There are others that he has signaled & allocated to which were published by others. Below I’m including the ones that are obviously garbage:
Published by Others
botbait
2,216.50
-
QmYyjXjcvjihvqsziicVJAAtES9RpvBhhGjYtCSookU6xd
Test Deployment1
1,500.11
698.17
QmUJwidYPoTfbyrx1YUgp3VzjCaSkWCrptKch2sJypEA4o
Bot bait-v2
1,660.30
-
QmVW6RHBa6kTYkUPhTXrQt2HCgbjZfdWLJwWFZrsAqNdhN
Ba1tD
19,452.65
27,738.31
QmXmxDHRt2oPWTto7Nrv1cJBKza4PMPzLdupFUpkBk55p1
Dummy Subgraph 1
16,363.01
24,039.49
QmNukFUkc6DspWQx8ZzRSvbpsBWaiPirQdbYPq6Qc4B4Wi
Total
41,192.57
52,475.97
In total, he has grossed 130,845.14 GRT for himself and 175,185.25 GRT for his delegators.
Edit: for additional reference, since the launch of the mainnet, we have paid the entirety of the curation sector 16,793.34 GRT (and an airdrop which I do not have the figures for). With more GRT used in curation, it becomes more resilient to wasteful actions like the above. Currently, however, there is greater incentive for indexers to manipulate their rewards then there is for curators to fulfill their role.
The substance of the Arbitration charter is intended to ensure that the Arbitrator is fulfilling their role of supporting a healthy and functioning network where Indexers perform their work correctly while minimizing the risk to honest Indexers of being economically penalized while interacting with the protocol in good faith.
The indexing provided here was being performed correctly as far as I am aware. However, this was performed in bad faith that served as a net negative to indexers and the network. Signal was moved to and from self-published useless subgraphs with the sole purpose of high proportion allocations from self signal and removing all signal when they were no longer the sole indexer.
This was self-admittedly done by the indexer due to the current market economics deemed not enough rewards for themselves and its delegators.
I do believe this discussion was very much needed so thank you @KonstantinRM
This does bring a lot to the table and possible edge cases that fall into black, white and grey.
While larger indexers may have the power to drown these allocations, are we saying that’s now part of the space? We have an old guard of the larger indexers that have enough resources and funds to waste drowning signal proportions they believe to be bad behavior that could have been an allocation to a valid subgraph? We could all do this to compete in reality, but at what cost?
Providing a less robust infrastructure to our customers and indexing ourselves to remain competitive.
I believe if this were not to end in dispute on this occasion it would send a strong signal :pun intended: to every other indexer not allocate to valid subgraphs if they don’t like the numbers. Just make your own and extract rewards from the protocol. The descendants of our decisions now are the heirs to our future.
I applaud @suntzu for being forthcoming and would like to hear more from him on this to shake out our growing pains together.
Regarding the trash subgraphs that I have allocated (some I created and some not mine), I created it a few months ago for the purpose of burning the fees of the former Frontrunning bot.
I simply understand this is open market and I am trying to compete for higher APY than whales to attract more delegators. This will make the network more decentralized on the plus side, but the approach I’ve used is clearly not pretty.
Many other indexers are allocating trash subgraphs ( exp : nirn , badger-harvests , InitialAcquisition ,ChrisTestingV3 …).
Indexer: I allocate to subgraphs and provide a valid POI when close allocate. Anyone can immediately allocate after me.
Curator: I have the right to signal to any subgraphs and I lost network fees for that.
Ethics: I agree it’s not right to unsignal and switch to other subgraphs.
Permissions: I think I have the right to do that with curator and indexer permissions.
This was not ethically correct for the network and I fixed it by allocating other available subgraphs of the network.
Sorry for doing bad things to the network. Actions like this will never happen again.
I have a few thoughts I would like to share that have been percolating in my brain for a couple days.
First, I do agree that this case is pretty straightforward with the evidence that has been presented, and I do think that it should be discouraged/disallowed, but I am not sure how effective such a rule would be for future offenses. I say that because while in this case it is easy to see that these are useless subgraphs and that the curation signals were done in a malicious manner, I doubt it would ever be this straightforward for future disputes.
In the case of useless subgraphs, we can see they are useless when they’re named things like “Bot Bait” and “To Be Removed,” but what is stopping an indexer from using a legitimate subgraph with very little indexing overhead, traffic, and signal to the same effect? Or do others not see that as being the same level of “useless” to the network?
And in the case of malicious curation signals, it might be difficult to differentiate between one that is legitimate and one that is malicious.
For instance, recently I allocated to a small and lightweight subgraph that quickly had a significant portion of signal removed. Instead of reallocating to a different subgraph, I decided to signal around the same number to make up for the difference. Originally, I thought I would probably remove the signal after this round of allocations, but I might also change my mind and allocate to the same subgraph again. Now let’s say I did change my mind, if it got to the day I was planning on refreshing my allocations, and the signal proportion was thrown off too much by another allocation, I would be thinking about just closing my allocation and removing the signal. Would that be considered malicious?
Personally, I think that line of thinking and those actions are legitimate, but I can also see how an indexer would potentially be able to model their actions off of a legitimate line of thinking (like mine) while still effectively accomplishing the same thing as in this case of malicious behavior. I would love to hear what other indexers think of my line of thinking/actions from the paragraph above and whether you all agree that they are legitimate.
Another thought I had was that this strategy will only become more difficult to execute as the market matures. As total signaled GRT grows, it will become more and more difficult to signal enough to meaningfully increase rewards, especially when it starts becoming important to calculate the opportunity loss for the unallocated tokens you are signaling.
So, assuming that most other indexers agree that my recent actions and line of thinking aren’t malicious, I would probably lean toward disallowing this obvious case of useless subgraph indexing and malicious signaling, but I would be wary of more vague rules that would make the (what I consider) legitimate actions a potentially slashable offense, because I think indexers should be able to look at all of the options available within the protocol when trying to achieve a certain return for themselves and their delegators, like I did, without the fear of getting slashed for it.
I’m still thinking this all through though, so I’d love to hear what others think as well.
I agree it will be increasingly difficult to differentiate legitimate ones from ones that are malicious. We will likely see less ‘bot bait’ names. But it seems like this research is something that the fisherman role can still detect. For example, tying a curation address to an indexer is always a bit tricky. I can think if a few points that I could inspect within the subgraph to determine whether or not there is malicious activity.
I think the issue that is raised is signaling to subgraphs which provide no potential value to the network and doing so in order to capitalize on indexing rewards.
Indexers should be able to add signal. This can be an important way that The Graph is able to support subgraphs with low query traffic but serve important data (Pool Together Governance is a great example of this). Also indexers often times help Dapps build subgraphs - they should be allowed to signal and index this to ensure the data is ready for the new consumer.
I was was interested in seeing how others perceived this situation, before throwing my own hat in the ring. I was also hoping we’d get to discuss this topic this at greater length on the latest IOH, but we ran short on time.
This ‘action’ was partly enabled via a relatively low amount of activity (or moreso ‘weight’) in the Curation market. The gains to be had by working the balance of these markets to your advantage, potentially in any role of the protocol, are not something we can account for without intervention. For now, i guess.
So since we’re “not there yet”, i guess these discussions need to happen and community-led solutions are required in lieu of anything protocol-driven.
My 2c
As we know, Indexers should generally ‘follow signal’. But in this case, the Indexer was following his own signal. This is not inherently bad nor malicious, as an Indexer can also act as a Curator if he wishes.
But in this case, the launching of his own subs (in some of the cases listed) in a manner that had seemingly no purpose other than personal advantage, adds a different tone to the situation. There also seemed to be no obvious motives leant towards something more ‘whitehat’. Such as pushing a boundary to set a governance precedence in a positive manner, since this behaviour was conducted over a pretty long-term and only ceased once it was highlighted by this thread.
There are some similarities between this case and a prior dispute:
Someone could have allocated on top of SunTzu, and in some cases did, and then the signal was removed by SunTzu. Minato would have likely also acted in the same manner, had someone allocated on top of him and removed the advantages he sought.
The only difference that i see, is that SunTzu was not using the (incorrect) community perception of “no delegating to a broken sub” to his advantage. Though, since that perception was already cleared up by the above dispuite, how could he?
Therefore if we ignore that detail, it’s very close to being the same situation. *But maybe someone disagrees with my take on that comparison?
Ergo - If you read the conclusions made by the Arbitration Council in the other dispute, i would imagine they would lean in a very similar direction as they did there:
As for the larger pattern of behavior, it is confirmed that there was a coordinated pattern of behavior from the disputed indexer to create, deploy, curate on, index, and collect indexing rewards on these subgraphs. Based on the data available, the investigators could not find strong reason to believe this action was in any way beneficial to the goals of the network. As such the behavior can be characterized as being purely driven by a goal of exploiting the failed subgraphs exception
… the ending there, could be substituted by what was essentially the overall end-goal in both of these examples: Profit
With a reactive Indexer set, my first instinct was that these opportunities would have mostly been short-lived.
Extraction
In this situation, his advantage acts as a kind of ‘tax’ on the Indexer community via rewards absorption. The size of this tax and advantage depending partly on the size of the Indexer, other Indexers reacting to signal, etc etc.
And In my opinion, the ‘size’ of the extraction taken off the plate of other Indexers should play in to how severe these actions should be looked upon. At least to some degree.
It’d be a lot of effort to go back and check the historical network stats, total staked, etc. And try to calculate a more ‘fair’ average picture of things. But since many of the most important metrics didn’t change too drastically, i decided to just base a rough calculation off the current network stats, using data from GraphScan:
With an Indexer of his size, taking a more average share of income - as in, not underperforming, or overperforming - i would calculate that in 5 months, he would have most likely made somewhere in the realm of just over 600k GRT
*Note that in the past 5 months, he gained around 2.1mil in delegations/stake total, so if anything, the average income would have tilted **lower** if anything, by anywhere up to ~15%. Source: - StakeMachine
Now, if we go back and check his actual total earnings from allocations closed in the past 5 months (via GraphScan/CSV export), his total revenue was actually around: 830k GRT
*I invite anyone to calculate this for themselves and correct any errors i made in my napkin math, of course.
Beyond the example above, it’s hard to say exactly how much versus ‘average’ he would have made, if he had simply acted in an engaged manner and taken other opportunities on under-allocated subs.
"Bad" Actions
This situation aside, i also wanted to briefly mention another point.
If we’re to agree on boundaries for behaviour based on how things look ‘ethically’ and what is and isn’t OK on the network, i think this shouldn’t be the only instance that lands under the spotlight. And i have to wonder - is this example only noticed or highlighted so much because it concerns Indexers more directly as a group?
And i don’t say that in some self-accusation of us having self-serving optics. I say it that to point out that, as a group, on average, Indexers are likely the most engaged, and possibly the most vocal or proactive participant in the network.
Therefore, things that affect us more directly, maybe catch our eye more, get the most engagement, etc. Because the actions that directly impact our role, are what we’re most aware of, and naturally what we’re most concerned about.
But if we’re going to set boundaries and choose to police things other than ‘bad POIs’, then we should also cast the net over things that maybe affect us less, and other participants more. The most obvious example being - Sandwich Attacks
*For anyone unaware - this is when an Indexer changes cuts to their own benefit, just before closing allocations, and then revert to more attractive cuts afterwards. Baiting in delegations, which will never receive the APY advertised by the Indexer.
These instances have happened a lot, often by repeat offenders, to the ire of many Delegators. If an Indexer runs for the max allocation period, they could get away with this for a minimum of 2 months. As Indexers, we’re not locked in to these bad positions via the moves of ‘bad actors’ for 28 days, but instead, 1 epoch (~1 day).
During these periods, bad actors may take some, most, or even all of the rewards from their Delegators. Some Delegators may be unaware of this for even longer. But of course, at some point, part of this responsibility is theirs to own.
We’ve fostered a habit of saying “be careful who you delegate to”, which does have merit in many ways. But if we’re going to consider penalizing Indexers for actions that impact our own rewards, and policing what’s acceptable - as a community - shouldn’t we also care about bad actors that take from other participants?
Every bad action affects the perception of Indexers as a group to a degree. And maybe even the project itself, especially the mechanisms tied to participation within protocol. We should be concerned about any clear-cut bad action no matter which group/role it hinders from honest participation.
And i direct all of the above to any participant, not only Indexers. If you have concerns, air them (as @KonstantinRM did here). Or, if you’re more confident, dispute them.
Yes, this takes 10k GRT to initiate. And this is only my opinion, but despite the Arbitration system moving much slower than i think we’d all like, i personally have faith in the members there to not act unreasonably (regarding that deposit) if presented with a good-faith dispute that highlights a legitimate or unprecedented concern.
Final Note
I have seen SunTzu as a great participant ever since testnet, and i was a little disappointed to see this from him. I was leaning towards appealing others to consider his past actions heavily, if any dispute or action was going to be taken against him by someone.
But after seeing the amount that seems to have been extracted and the length of time over which these actions were taken, a line for goodwill being considered by most people, may have been crossed at some point.
*Edit - Fixed a calculation error i made by not excluding by date range (too used to Etherscan, woops!)
I think this case just highlights the need for more subgraphs to migrate to mainnet.
In a fair market these actions would be very costly and would have little to no implications on inderxer rewards.
Until the migration happens the curation market will continue to be broken, it just looks like now it has started to affect the other network participants and not only the curators.
I remember being 4 years old walking into our living room and seeing a 5 Deutsche Mark coin on the table. I knew my parents used money to buy things, so I was aware it had value. I took it and went back my room, happy that I had some of it too now. Later that day, my dad asked if I had seen the coin and I said ‘sure, I have it’. He then sat me down and explained to me the concept of stealing. My happiness over having the coin was quickly replaced by feeling bad and remorseful. It became an important lesson to me.
We just celebrated our first year birthday at The Graph and we have only just begun the walk of a long journey. The Graph is a complex protocol and we will encounter more instances like these in the future. The situations won’t nearly be as black and white like my experience as a 4 year old, but rather different shades of grey. Our combined wealth of wisdom will be required to keep The Graph ecosystem and protocol maturing into the foundational web3 infrastructure pillar we aspire for. Continuous improvements to strengthen the security and integrity of the protocol are essential to close loopholes and exploitative patterns that are purely self-centered and which don’t serve the purpose of the network in any way.
I want to join others in thanking @KonstantinRM for not only bringing this case into the public Forum, but also doing so in a way that has allowed a constructive dialogue to pursue. You have set a blueprint for how these situations can be introduced to the community productively and it’s incredibly important for our journey at The Graph. Thank you!
