Request for information about disputes #GDR-3

Governance & GIPs Arbitration
1

The arbitrators are contacting indexer address 0x76e84025978a0c16fc7ba483718b667388f2973c (minatofund) about disputes filed in the protocol.

To make the investigation as brief as possible, please provide the following information and any other relevant records about the open disputes:

  • Version of graph-node used.
  • Graph-node configuration in full.
  • Type and version of Ethereum node.
  • Table of PoIs generated by the affected subgraphs.
  • A dump of the call cache data from the database for the affected subgraphs.
  • Entity values as of the divergent block once identified.

This is not an all-inclusive list. Request for additional information may be made if considered necessary.

How to Get Relevant Data

You can use the following queries to extract information from the indexer software to be used in the evaluation.

# Get subgraphs schema
SELECT name FROM public.deployment_schemas where subgraph = 'QmTj6fHgHjuKKm43YL3Sm2hMvMci4AkFzx22Mdo9W3dyn8';

# Move to the subgraphs schema
SET SEARCH_PATH TO <RESULT_FROM_ABOVE>;

# Dump PoI TABLE - (from outside of psql console but on server)
pg_dump --dbname="<YOUR_DB_NAME" --host=<DB_HOST> --port=5432 --username=<USERNAME>  --table='<SUBGRAPH_SCHEMA>."poi2$"' --file='<FILE_TO_SAVE_TO>'

# Dump call cache (on some setups this table may be in a different schema, check it with `select * from public.chains`)
pg_dump --dbname="<YOUR_DB_NAME" --host=<DB_HOST> --port=5432 --username=<USERNAME>  --table='public.eth_call_cache"' --file='<FILE_TO_SAVE_TO>'

Once a divergent block is identified:

# loop through all entity tables and get changes for that block
# for each entity table in subgraph deployment schema:
select * from <entity_table> where lower(<DIVERGENT_BLOCK>);

Purpose of the Requirement

This requirement is related to the following disputes:

├─ 0x68982a39c7cd4ccc1f22e2bd00de0de20b06cd3e19cb3a65d931d7af410e05e1
│  ├─ Type: Indexing
│  ├─ Status: Undecided (0.13 days ago)
│  ├─ Indexer: 0x76e84025978a0c16fc7ba483718b667388f2973c
│  ├─ Fisherman: 0xe3671ff90401c21e2bb55b2dfab4506cede113e5
│  ├─ SubgraphDeployment
│  │  └─ id: 0x69c184d914985b6681499479840877426f9c1c3e369ba618d8e32a1ea65ff63b (QmVTUfdp5sJR4uNLq8jM1zR6TLyepejz9gd38YpNT9We5Q)
│  ├─ Allocation
│  │  ├─ id: 0x347032a87c92a79f3eacbf2f4262dbb7eba5165d
│  │  ├─ createdAtEpoch: 215
│  │  ├─ createdAtBlock: 0x0654ccd498a1224b6bcacdd34689f4b8902aa70804975bd479ec7bd390b39be7
│  │  ├─ closedAtEpoch: 216
│  │  └─ closedAtBlock: 0x972dabc5337c9540d9e31212866832a07ec3c61956584d9a2986ce81b34984aa (#12883331)
│  └─ POI
│     └─ submitted: 0x367e3cb4ab168ef8231ab1228a51cc6399bca1c17545515573a91b8ddd9cf532
├─ 0xa6e48245e348d1533e31cba5bac4da3125104c927c53b11fa99ab95a8ba15a27
│  ├─ Type: Indexing
│  ├─ Status: Undecided (0.13 days ago)
│  ├─ Indexer: 0x76e84025978a0c16fc7ba483718b667388f2973c
│  ├─ Fisherman: 0xe3671ff90401c21e2bb55b2dfab4506cede113e5
│  ├─ SubgraphDeployment
│  │  └─ id: 0x949d775a90ef63b46d95fc90cf326dea9cceef9b3c4fcb04f9e16581dc7058ee (QmYLnK6HwxTn3qPe2tURFUQYcxa4VVxLLvdbt6hVpBXmzd)
│  ├─ Allocation
│  │  ├─ id: 0x5037caea050b4dda604ef5efe908715e5cb71103
│  │  ├─ createdAtEpoch: 211
│  │  ├─ createdAtBlock: 0xc0d8ef23fefd3590ba30c0ffcc748c952637d7348ccc9d06de5031c43381ec67
│  │  ├─ closedAtEpoch: 212
│  │  └─ closedAtBlock: 0xe504360b3f50a5b7a62e92ed93e0afb97cf778c2a40303e578dfd46349e2a5a9 (#12855756)
│  └─ POI
│     └─ submitted: 0x82d3c31866310dceefbad6dd5e4c0721f32b7c16bd93ea16f89be91c870f4ff0
├─ 0x21309e1849923a4e6997280130907daa7cf3e087438b0d23fc01e578e0461dc2
│  ├─ Type: Indexing
│  ├─ Status: Undecided (0.13 days ago)
│  ├─ Indexer: 0x76e84025978a0c16fc7ba483718b667388f2973c
│  ├─ Fisherman: 0xe3671ff90401c21e2bb55b2dfab4506cede113e5
│  ├─ SubgraphDeployment
│  │  └─ id: 0xb71738e2cfee4a013449cd09dd89fc1eeb2ead8941e2a8a006d1db605d761d26 (QmafMrc51kqWZdBF9u3XzerN2gYAb8wUPU7dHiHuxWewjK)
│  ├─ Allocation
│  │  ├─ id: 0x73f3810a241d4966f18d0427d5244afec9f2ecf1
│  │  ├─ createdAtEpoch: 215
│  │  ├─ createdAtBlock: 0x0db22dbadd706ad2135bf5e4829852fd3f98402c6ab74508bf1bf23ccf5d68f9
│  │  ├─ closedAtEpoch: 216
│  │  └─ closedAtBlock: 0x1f691464f3df6c97da0b4f0a259674cb9aa7ec55b9f4d944a101f5b9a15ff2c0 (#12883325)
│  └─ POI
│     └─ submitted: 0x02911d0e46f521bc791524d6aa9d858db2d82688a7ffdecde3caac0ff8f53a38
├─ 0xa2f6ba08ec3ad3e04c4f3ec2ed4344ebb32ba867923ec14b14e46311d954ff59
│  ├─ Type: Indexing
│  ├─ Status: Undecided (0.13 days ago)
│  ├─ Indexer: 0x76e84025978a0c16fc7ba483718b667388f2973c
│  ├─ Fisherman: 0xe3671ff90401c21e2bb55b2dfab4506cede113e5
│  ├─ SubgraphDeployment
│  │  └─ id: 0x6265679b3f40b30663c7e6f27828c5b1df274b0f422aeede33ac2008100a4df9 (QmUxkM4kkYDyVEcUcBGWrvhj5Y6f2uvUTkPTmPjtm76A6k)
│  ├─ Allocation
│  │  ├─ id: 0xd9036a08e37d0810451a2da60056c1cc935069e7
│  │  ├─ createdAtEpoch: 214
│  │  ├─ createdAtBlock: 0xab6a0e71041e365deb0be08dc88e7a0f762adb73189d8d7c5ffd7247b07e55e9
│  │  ├─ closedAtEpoch: 215
│  │  └─ closedAtBlock: 0x7347437d0f83310d5e6e3348f0b7474ab3237b248ad13d4b65b47dc1a0b2d27e (#12876424)
│  └─ POI
│     └─ submitted: 0xf9cb11c4cec2e710d80c8e1770a75b2a502c795c9b75b6836853738f3d26b6a2
├─ 0x274b39de33d7cf9f4e9feaf3331f56078b0546884aecae156a1f292d42412af8
│  ├─ Type: Indexing
│  ├─ Status: Undecided (0.13 days ago)
│  ├─ Indexer: 0x76e84025978a0c16fc7ba483718b667388f2973c
│  ├─ Fisherman: 0xe3671ff90401c21e2bb55b2dfab4506cede113e5
│  ├─ SubgraphDeployment
│  │  └─ id: 0xc35cbc58760eae95e5accbd7f866dfb1f89060482b4feb525b795259cfc220aa (QmbVGE6Ma2AqzVzKQip6S81dAiSuHeQLny47jzFTLa3DAq)
│  ├─ Allocation
│  │  ├─ id: 0x30740c34017407cc939e559b2bbe96b711e3d088
│  │  ├─ createdAtEpoch: 212
│  │  ├─ createdAtBlock: 0x1f2d50cac27c46ea67ebe48016aacc56529cea28fd95f20db163930103e0193e
│  │  ├─ closedAtEpoch: 213
│  │  └─ closedAtBlock: 0xdd855884efe31cab39686d1cbe165f1a5f89acbf1f17e4fad8a69d32bac64b94 (#12862431)
│  └─ POI
│     └─ submitted: 0xc2d56a308c3560a511a0ca06ab4bf8d8f3c9891687c21980507e3a97c9ace7a6
└─ 0xf6dd49f420750111fe0ec51b4e5c4f175ce2a65a864d6f714e7c2ef66829031a
   ├─ Type: Indexing
   ├─ Status: Undecided (0.13 days ago)
   ├─ Indexer: 0x76e84025978a0c16fc7ba483718b667388f2973c
   ├─ Fisherman: 0xe3671ff90401c21e2bb55b2dfab4506cede113e5
   ├─ SubgraphDeployment
   │  └─ id: 0x1c153b5e9ba74823153c451300a053beb9dcdef20549b22901ef1d502e21d8eb (QmQEGuJtXiwWni2bsmDDPPor9U1nfQ8s2kbQ6XYCwNjojG)
   ├─ Allocation
   │  ├─ id: 0x0b089eeed8cf24c41f9e74cc750fc654eb5f0fc4
   │  ├─ createdAtEpoch: 213
   │  ├─ createdAtBlock: 0xc7f621c45794507aec687a72900de26a669d019aa1ba58839a677e4c5b90f937
   │  ├─ closedAtEpoch: 214
   │  └─ closedAtBlock: 0x929b7b23e791a2cc7c86ce2010fd99ede6cfe32c429c8e991849cf288168ebab (#12869026)
   └─ POI
      └─ submitted: 0xb751485346603a1d165b822499aeb1bd2a5122542c5f40ae3720d666be18126c

About the Procedure

The Arbitration Charter regulates the arbitration process. You can find it in Radicle project ID rad:git:hnrkrhnth6afcc6mnmtokbp4h9575fgrhzbay

For communications, please use this forum. Additionally, please monitor Graph Protocol’s Discord #arbitration (https://discord.gg/9gg7XvfggW ) for any arbitrator reaching out for more information.

4 Likes
2

Bumping this request to indexer 0x76e84025978a0c16fc7ba483718b667388f2973c to present relevant data to help with the dispute investigation.

3

Hi there, we confirmed all these allocations faced subgraph failure after opening. Thus, we closed them with the last valid POI. We think our action fulfills the exception case.

An exception to this rule is if the allocation being closed was opened before the subgraph bug occurred. In this case, the Indexer may submit the last valid PoI they produced for the subgraph and collect indexing rewards for the allocation.

The poi_table_dump and entities_dump data are attached to the Google Drive link. Please feel free to check it. Also, we will try to provide more details with the analysis tool if it is needed.

├─ 0x68982a39c7cd4ccc1f22e2bd00de0de20b06cd3e19cb3a65d931d7af410e05e1
│  ├─ Type: Indexing
│  ├─ Status: Undecided (0.13 days ago)
│  ├─ Indexer: 0x76e84025978a0c16fc7ba483718b667388f2973c
│  ├─ Fisherman: 0xe3671ff90401c21e2bb55b2dfab4506cede113e5
│  ├─ SubgraphDeployment
│  │  └─ id: 0x69c184d914985b6681499479840877426f9c1c3e369ba618d8e32a1ea65ff63b (QmVTUfdp5sJR4uNLq8jM1zR6TLyepejz9gd38YpNT9We5Q)
│  ├─ Allocation
│  │  ├─ id: 0x347032a87c92a79f3eacbf2f4262dbb7eba5165d
│  │  ├─ createdAtEpoch: 215
│  │  ├─ createdAtBlock: 0x0654ccd498a1224b6bcacdd34689f4b8902aa70804975bd479ec7bd390b39be7
│  │  ├─ closedAtEpoch: 216
│  │  └─ closedAtBlock: 0x972dabc5337c9540d9e31212866832a07ec3c61956584d9a2986ce81b34984aa (#12883331)
│  └─ POI
│     └─ submitted: 0x367e3cb4ab168ef8231ab1228a51cc6399bca1c17545515573a91b8ddd9cf532

Allocation Tx
https://etherscan.io/tx/0x2f48b4592a11a8a7d518b669451371a62f6aa3b7d04c4920f6107c2d3d1de2b2

Allocation Block
12876853

Last Block Before the Subgraph Bug 
12876876

├─ 0xa6e48245e348d1533e31cba5bac4da3125104c927c53b11fa99ab95a8ba15a27
│  ├─ Type: Indexing
│  ├─ Status: Undecided (0.13 days ago)
│  ├─ Indexer: 0x76e84025978a0c16fc7ba483718b667388f2973c
│  ├─ Fisherman: 0xe3671ff90401c21e2bb55b2dfab4506cede113e5
│  ├─ SubgraphDeployment
│  │  └─ id: 0x949d775a90ef63b46d95fc90cf326dea9cceef9b3c4fcb04f9e16581dc7058ee (QmYLnK6HwxTn3qPe2tURFUQYcxa4VVxLLvdbt6hVpBXmzd)
│  ├─ Allocation
│  │  ├─ id: 0x5037caea050b4dda604ef5efe908715e5cb71103
│  │  ├─ createdAtEpoch: 211
│  │  ├─ createdAtBlock: 0xc0d8ef23fefd3590ba30c0ffcc748c952637d7348ccc9d06de5031c43381ec67
│  │  ├─ closedAtEpoch: 212
│  │  └─ closedAtBlock: 0xe504360b3f50a5b7a62e92ed93e0afb97cf778c2a40303e578dfd46349e2a5a9 (#12855756)
│  └─ POI
│     └─ submitted: 0x82d3c31866310dceefbad6dd5e4c0721f32b7c16bd93ea16f89be91c870f4ff0

Allocation Tx
https://etherscan.io/tx/0x31e7f1dd7f002ebd29eeb4b6d8aeb342b42800438a6dd3958e25731a8582f709

Allocation Block
12849963 

Last Block Before the Subgraph Bug 
12855676

├─ 0x21309e1849923a4e6997280130907daa7cf3e087438b0d23fc01e578e0461dc2
│  ├─ Type: Indexing
│  ├─ Status: Undecided (0.13 days ago)
│  ├─ Indexer: 0x76e84025978a0c16fc7ba483718b667388f2973c
│  ├─ Fisherman: 0xe3671ff90401c21e2bb55b2dfab4506cede113e5
│  ├─ SubgraphDeployment
│  │  └─ id: 0xb71738e2cfee4a013449cd09dd89fc1eeb2ead8941e2a8a006d1db605d761d26 (QmafMrc51kqWZdBF9u3XzerN2gYAb8wUPU7dHiHuxWewjK)
│  ├─ Allocation
│  │  ├─ id: 0x73f3810a241d4966f18d0427d5244afec9f2ecf1
│  │  ├─ createdAtEpoch: 215
│  │  ├─ createdAtBlock: 0x0db22dbadd706ad2135bf5e4829852fd3f98402c6ab74508bf1bf23ccf5d68f9
│  │  ├─ closedAtEpoch: 216
│  │  └─ closedAtBlock: 0x1f691464f3df6c97da0b4f0a259674cb9aa7ec55b9f4d944a101f5b9a15ff2c0 (#12883325)
│  └─ POI
│     └─ submitted: 0x02911d0e46f521bc791524d6aa9d858db2d82688a7ffdecde3caac0ff8f53a38

Allocation Tx
https://etherscan.io/tx/0x752b5d73d618ea075163e52045b64f95456e7f4f3c676df50221dd05a82fd2d1

Allocation Block
12876866

Last Block Before the Subgraph Bug 
12876870

├─ 0xa2f6ba08ec3ad3e04c4f3ec2ed4344ebb32ba867923ec14b14e46311d954ff59
│  ├─ Type: Indexing
│  ├─ Status: Undecided (0.13 days ago)
│  ├─ Indexer: 0x76e84025978a0c16fc7ba483718b667388f2973c
│  ├─ Fisherman: 0xe3671ff90401c21e2bb55b2dfab4506cede113e5
│  ├─ SubgraphDeployment
│  │  └─ id: 0x6265679b3f40b30663c7e6f27828c5b1df274b0f422aeede33ac2008100a4df9 (QmUxkM4kkYDyVEcUcBGWrvhj5Y6f2uvUTkPTmPjtm76A6k)
│  ├─ Allocation
│  │  ├─ id: 0xd9036a08e37d0810451a2da60056c1cc935069e7
│  │  ├─ createdAtEpoch: 214
│  │  ├─ createdAtBlock: 0xab6a0e71041e365deb0be08dc88e7a0f762adb73189d8d7c5ffd7247b07e55e9
│  │  ├─ closedAtEpoch: 215
│  │  └─ closedAtBlock: 0x7347437d0f83310d5e6e3348f0b7474ab3237b248ad13d4b65b47dc1a0b2d27e (#12876424)
│  └─ POI
│     └─ submitted: 0xf9cb11c4cec2e710d80c8e1770a75b2a502c795c9b75b6836853738f3d26b6a2

Allocation Tx
https://etherscan.io/tx/0xc34806d25ffde7ee12880bfd9c933e7f0b61e29ec6f11c8e9519a3a1fd5cfbd1

Allocation Block
12869137

Last Block Before the Subgraph Bug 
12869143

├─ 0x274b39de33d7cf9f4e9feaf3331f56078b0546884aecae156a1f292d42412af8
│  ├─ Type: Indexing
│  ├─ Status: Undecided (0.13 days ago)
│  ├─ Indexer: 0x76e84025978a0c16fc7ba483718b667388f2973c
│  ├─ Fisherman: 0xe3671ff90401c21e2bb55b2dfab4506cede113e5
│  ├─ SubgraphDeployment
│  │  └─ id: 0xc35cbc58760eae95e5accbd7f866dfb1f89060482b4feb525b795259cfc220aa (QmbVGE6Ma2AqzVzKQip6S81dAiSuHeQLny47jzFTLa3DAq)
│  ├─ Allocation
│  │  ├─ id: 0x30740c34017407cc939e559b2bbe96b711e3d088
│  │  ├─ createdAtEpoch: 212
│  │  ├─ createdAtBlock: 0x1f2d50cac27c46ea67ebe48016aacc56529cea28fd95f20db163930103e0193e
│  │  ├─ closedAtEpoch: 213
│  │  └─ closedAtBlock: 0xdd855884efe31cab39686d1cbe165f1a5f89acbf1f17e4fad8a69d32bac64b94 (#12862431)
│  └─ POI
│     └─ submitted: 0xc2d56a308c3560a511a0ca06ab4bf8d8f3c9891687c21980507e3a97c9ace7a6

Allocation Tx
https://etherscan.io/tx/0x478e0a034abb95595344a93ae1a23a615eecb73a973f987000fe2714312bab6c

Allocation Block
12856913

Last Block Before the Subgraph Bug 
12860423

└─ 0xf6dd49f420750111fe0ec51b4e5c4f175ce2a65a864d6f714e7c2ef66829031a
   ├─ Type: Indexing
   ├─ Status: Undecided (0.13 days ago)
   ├─ Indexer: 0x76e84025978a0c16fc7ba483718b667388f2973c
   ├─ Fisherman: 0xe3671ff90401c21e2bb55b2dfab4506cede113e5
   ├─ SubgraphDeployment
   │  └─ id: 0x1c153b5e9ba74823153c451300a053beb9dcdef20549b22901ef1d502e21d8eb (QmQEGuJtXiwWni2bsmDDPPor9U1nfQ8s2kbQ6XYCwNjojG)
   ├─ Allocation
   │  ├─ id: 0x0b089eeed8cf24c41f9e74cc750fc654eb5f0fc4
   │  ├─ createdAtEpoch: 213
   │  ├─ createdAtBlock: 0xc7f621c45794507aec687a72900de26a669d019aa1ba58839a677e4c5b90f937
   │  ├─ closedAtEpoch: 214
   │  └─ closedAtBlock: 0x929b7b23e791a2cc7c86ce2010fd99ede6cfe32c429c8e991849cf288168ebab (#12869026)
   └─ POI
      └─ submitted: 0xb751485346603a1d165b822499aeb1bd2a5122542c5f40ae3720d666be18126c

Allocation Tx
https://etherscan.io/tx/0xe7a0ac51010e0f8ef2d78f511825d66881a1be5221a5d8339008c0eab68f1506

Allocation Block
12862500

Last Block Before the Subgraph Bug 
12862506
4

Hi all,

As the fisherman in these disputes, I felt I should share evidence on why I have disputed the MinatoFund indexer.

Reasons for Disputes

  • Incorrect POIs
  • Malicious and unethical intentions

Part 1 - Incorrect POIs

POI comparisons per Allocation ID:

0x5037caea050b4dda604ef5efe908715e5cb71103

  • POI actual: 0xb429cb40d26ffa8aec1f0ec31eb6bd5d97ccf0e6062559a4570a521c97a5caac
  • Provided: 0x82d3c31866310dceefbad6dd5e4c0721f32b7c16bd93ea16f89be91c870f4ff0

0x30740c34017407cc939e559b2bbe96b711e3d088

  • POI actual: 0x5bdf4c32ef9498197f087e4c570a98eefa7a83de65818445434e311259939be2
  • Provided: 0xc2d56a308c3560a511a0ca06ab4bf8d8f3c9891687c21980507e3a97c9ace7a6

0x0b089eeed8cf24c41f9e74cc750fc654eb5f0fc4

  • POI actual: 0xa2a9fae6226689dbb9755a9cbeed0b8c274d68f48e4ad16c5866f0bd29048cdf
  • Provided: 0xb751485346603a1d165b822499aeb1bd2a5122542c5f40ae3720d666be18126c

0xd9036a08e37d0810451a2da60056c1cc935069e7

  • POI actual: 0xe11d7fe5fc7b7d0701976c793336732c83bbb2ac9c4b5dfd5e8c2b071f478972
  • Provided: 0xf9cb11c4cec2e710d80c8e1770a75b2a502c795c9b75b6836853738f3d26b6a2

0x73f3810a241d4966f18d0427d5244afec9f2ecf1

  • POI actual: 0xd39937dea78486c35c1127cee227487d72ff6bfdadf93dd3c7c926241bef932f
  • Provided: 0x02911d0e46f521bc791524d6aa9d858db2d82688a7ffdecde3caac0ff8f53a38

0x347032a87c92a79f3eacbf2f4262dbb7eba5165d

  • POI actual: 0xeeac889fb37dac0b0cc1ed25a157d60f3ecbca4856bf9d844720b7eab13c97c0
  • Provided: 0x367e3cb4ab168ef8231ab1228a51cc6399bca1c17545515573a91b8ddd9cf532

0xac8773569201ffa1674072b877ce2ce627b5ed83

  • POI actual: 0x00a4abb1b26ef2ce6ea7e1a9273de044698ed03aa3b82969f0e07a95be427b0a
  • Provided: 0x726c200d523918e2dfee10b3d2bf8178c9886960fb994b543acbb8968620136f

0x15380014b3313e172b363c707b8bb961110ecbb1

  • POI actual: 0x3b66fde5a41e0a4fc8d6b4241c40c01d0b8cbc7f40fb05c816f9cf67e5642f4b
  • Provided: 0x8323030fba1dd60a4eba0af9bc9aafb385035a9d529348f7049a1ef4150ced42

The reasons for the POI discrepancies are due to the MinatoFund indexer either using the incorrect epoch block and block hashes to calculate the POI, or due to other various unknowns.

Part 2 - Malicious/Unethical Intentions

After digging deeper into the above discrepancies, I uncovered what appears to be a scheme of Creating, Publishing, Curating, Indexing, and Intentional Deterrence by the party in question, which ultimately led to filing these disputes.

It appears the MinatoFund indexer is involved in a scheme where they will create a bad subgraph, deploy it, curate to it, and allocate to it just before it fails. Based on their post above and on Discord, it is clear that the Indexer is exploiting the following which can be found in the HackMD document outlining the Arbitration Charter https://hackmd.io/MMHefHW5TxOOLtqqutP-VA?both#9-Valid-Proofs-of-Indexing-for-a-given-epoch:

An exception to this rule is if the allocation being closed was opened before the subgraph bug occurred. In this case, the Indexer may submit the last valid PoI they produced for the subgraph and collect indexing rewards for the allocation.

I believe the intentional creation of a soon-to-be-failed subgraph was acting maliciously by creating a barrier to entry for other indexers in order to seal a larger share of the pool for the MinatoFund indexer. As time went on and the barrier was proving to be successful, the curation on said subgraph also increased dramatically from the 2000s to the 18000+ range. One could also say the malicious party was also attempting to bait other indexers into closing without a 0x0 if they unknowingly found themselves on one of these failed subgraphs after the failed block passed.

The following table shows the specific subgraphs in question (Dummy Subgraphs 2 and 3) and how soon each subgraph fails from creation, allocated, etc in blocks:

IPFS Subgraph Failed at Block Subgraph Created at Block Indexer Allocated at Block Subgraph Created to Failed (number of blocks) Subgraph Created to Allocated (number of blocks) Indexer Allocated to Failed (number of blocks)
QmYLnK6HwxTn3qPe2tURFUQYcxa4VVxLLvdbt6hVpBXmzd 12855677 12849955 12849963 5722 8 5714
QmbVGE6Ma2AqzVzKQip6S81dAiSuHeQLny47jzFTLa3DAq 12860424 12856901 12856913 3523 12 3511
QmQEGuJtXiwWni2bsmDDPPor9U1nfQ8s2kbQ6XYCwNjojG 12862507 12862491 12862500 16 9 7
QmUxkM4kkYDyVEcUcBGWrvhj5Y6f2uvUTkPTmPjtm76A6k 12869144 12869077 12869137 67 60 7
QmafMrc51kqWZdBF9u3XzerN2gYAb8wUPU7dHiHuxWewjK 12876871 12876790 12876866 81 76 5
QmVTUfdp5sJR4uNLq8jM1zR6TLyepejz9gd38YpNT9We5Q 12876877 12876836 12876853 41 17 24
QmUqXdxB5f9f6EuDPYcSEASCwUCQTBMU1LbsVntuQjamkc 12883374 12883340 12883358 34 18 16
QmZUrJSbDpVSGQemAnq2dHq4LyEXX7YQDvEm6vh2weoEej 12946174 12946155 12946165 19 10 9

The table shows how the scheme was honed as time passed, especially given how quickly the last 6 subgraphs failed after launch and allocation.

I believe the entire party involved in the scheme to be that associated with the MinatoFund indexer or they themselves due to the above data and due to simple block explorer analysis. Here are the addresses involved:

Subgraph Publisher: 0x210c43953bad387654fb4c997ccc1609a6b1125e
MinatoFund Indexer Rewards Address: 0xf3d2BaE54313a3991b086D93576d953D0baE7241
Subgraph Curator (Funded by MinatoFund indexer rewards address): 0x520F7445286cC4140Be54e9FBfE8Aa80a2B3F60E

The curation pool on these subgraphs as they are deployed is funded by the MinatoFund indexer’s reward address.


Given the information provided along with providing invalid POIs, I believe this indexer to be acting dishonestly within the protocol and should be judged accordingly.
8 Likes
5

Heh, I think @juanmardefago can also take a look at each individual subgraph and figure out exactly how those subs failed the way they failed. I know he created a subgraph for me that was rigged to fail at a specific block.

Gonna be fun to watch this one develop. Grabbing some popcorn in the meantime :popcorn:.

Oh yeah, forgot to add, hope we get that variable slashing percentage implemented cus there’s a lot of GRT to be deleted from here, if deemed necessary :grin:

2 Likes
6

Thanks for this work and details. This reads plain simply like an attack on the network and -if confirmed- should be slashed for each incident.

2 Likes
7

agreed, if all of the report details are correct, this has been an awful behaviour and deserves to be slashed for each of those separately… great work @86b !

2 Likes
8

Okay, we would like to give some clarifications and explain why this thing happened.

Part 1

Actually, we are also not sure about why the POIs are incorrect. After faced the subgraph failure, we did the exactly the same steps shown below to retrieve POIs and close allocations manually.

Before generating the POIs, the node is synchronized to the last valid block. That part can be confirmed by comparing the dumped poi_table and entities with the reference.

Part 2

In our case, it seems like community members are more concerned with the motives and ethical standards of this behavior, rather than whether it complies with the Charter. We fully respect the opinions of the community, and hope that everyone can understand our experience and thoughts before drawing conclusions.

First, we will not deny that some of our actions in this case are not decent and inappropriate, but please do not assume that we are malicious at the beginning. We have been actively participating in the protocol all the time trying to interact with new feature and contributing to the community. After the new Graph explorer and subgraph studio launched, we immediately migrate the HOPR subgraph we made to enrich the ecosystem. When we tried to signal it, we realized there was a curator bot taking advantage of it. That was super annoying, and somehow discouraging legit curators to signal subgraphs. We deployed Dummy Subgraphs at that time to consume the bot’s token, and named Dummy to prevent new curators accidentally signal on them and lost just like other subgraph developers who created bot bait subgraphs.

Then, the evil parts came in, we did think that the exception rule can be used to create an exclusive environment just like described below.

We definitely didn’t think about baiting other indexers to index on the failed subgraph, or letting them do something wrong. Also, before seeing the replies this case, we thought our behavior was kind of in the gray zone, although it’s not decent, still under the exception case. Thus, we did not consider it as an malicious intention, and we had no plan to hide the relations between the publisher, indexer, and curator addresses. Just in case someone can DM us if they think that’s not cool.

Overall

We think the major issues are

  1. How to determine whether the indexer is abusing the exceptions or not in a clear definition? And which degree are we located.

This is still a very blur area for all of us. Apparently, there are some indexers indexing on failed subgraphs, and prepare to close allocation at the very last moment with the last valid PoI. How bad we are comparing with them, and how to judge the degree of abusing in a quantitative way, and give an appropriate slash.

  1. How malicious are we?

We did bad thing, that’s not deniable, and we apologize for that. We are absolutely willing to take punishes as warning. It also can prevent other indexers to do thing like that, so that is 100% necessary.

The question followed up is, are we purely malicious in this case? We think this case is still defining the boundary of the exception rule. Slashing for each allocation is really harsh for us. We can promise that we won’t do things in gray zone anymore, and sharing the potential risks we find with the community in time. Please give us a chance as the active participant.

Appendix

Failed Subgraph Trick Disclosure

Maybe some people have interests in how we control the subgraph failure, so we will reveal it, and the community can figure out the best way to prevent subgraph developrs do things like that.

The trick is quite straight forward, in the subgraph, we track token transfer count from a specific address to another, once we send couple of transactions, mapping.ts gonna throw an Error and fail the subgraph.

9

To your HOPR comment, the HOPR guys were complaining in the previous week’s office hours that someone published their subgraph under an account that wasn’t theirs lol :sweat_smile: What you did there didn’t help anyone, just created more confusion. :slightly_smiling_face:

10

Oh, really… Actually, we had a talk with one of their team members, he thought our work is nice. Also, we are the creator, major DApp user and in charge of the subgraph maintenance, we don’t think anything wrong with publishing the subgraph under our account. At least, curators can know that is a community driven work, so they can give signals based on that truth. And curators should deicide if the action is confusing or not, right? We think this way is quite clear in the decentralized protocol, which is not built for only official team to develop & publish subgraph.

Anyway, we really appreciate your information and thoughts. We gonna keep in touch and double check with the HOPR team. If they would like to do further works or just feel offended, we will transfer the subgraph ownership to them.

Yeah, just like we said, it is still an early stage for the whole protocol. Everyone is figuring the standards of the process like migrating subgraphs, identifying if a project is legit and etc. We did something wrong, we should take the corresponding consequences. We have different opinion with other community members, we discuss here.

11

The arbitration council has completed the investigation of this set of disputes. Thank you for the detailed description of your findings, processes and motivations @86b and @minato; your transparency has helped make this a smooth process.

The investigation sought to confirm the claims of both parties and determine whether the actions of the disputed indexer could be considered a contribution to the network within the spirit of the arbitration charter.

The fisherman for this dispute mentions two reasons for submitting:

  • PoIs submitted on-chain do not match what the fisherman produced as a reference.
  • The fisherman has classified observed behavior on the disputed allocations as “malicious/unethical” and generally not aligning with the spirit of the proposed arbitration charter’s specifications.

The investigation will test the two hypothesis proposed by the fisherman about the disputed indexers behavior.

PoIs

The fisherman has compared the disputed allocation PoIs against references generated from the fisherman’s own graph node and found that they do not match. As part of the investigation the disputed deployments were indexed on a local machine to have an additional reference.

Once the deployments finished syncing (reached their failure points) reference PoIs were generated for the disputed allocations using the start block of the epoch prior to the epoch where they failed. Initially the references did not match the disputed indexers submitted PoIs; however they did match the fisherman’s reference PoIs. The discrepancy could be due to a difference in the block number used to generate the PoIs, so more reference PoIs were generated by looping through the range of blocks each allocation was active and generating a reference PoI as of each block (using the graph-disputes CLI tool). I found a match on the reference PoIs generated for the failed block number - 1. I consider this a reasonable interpretation of the Arbitration charter in its current state (still in flight) which states that the last valid PoI should be used.

Analyzing the PoI tables shared by the disputed indexer, they’ve been found to be identical to the corresponding tables from the disputed indexer. This comes as no surprise after the above investigation, but is further evidence that the indexer properly synced the deployments in question.

In conclusion, the PoIs submitted with the disputed allocations have been confirmed by the investigation as being valid and correct. The initial investigation by the fisherman that indicated “invalid” PoIs used a different (equally valid) scheme for generating PoIs. The fisherman used the start block of epoch prior to failure while the disputed indexer used failed block - 1 The fact that both PoI generation schemes produced different but valid results is of course a good indicator that further clarification in the charter is necessary; the arbitration has already been updated to remove the ambiguous statements around failed subgraphs and graph node work is in progress to support PoIs on the failed block of a subgraph so the network can come to consensus on failures.

Malicious behavior

The fisherman has classified observed behavior on the disputed allocations as “malicious/unethical” and generally not aligning with the spirit of the proposed arbitration charter’s specifications. The disputed indexer repeated a coordinated sequence of actions consisting of the following steps:

  1. Create and deploy a new subgraph to the network that is expected to fail at a certain block (within an epoch or two).
  2. Purchase curation shares on the subgraph.
  3. Allocate towards the subgraph deployment with their indexer and sync it until it fails.
  4. Close allocation using a PoI from the block before the failure.

Steps 1-3 created an exclusive environment for the indexer to collect rewards on the new subgraph deployments without competition from other indexers and in step 4 they were able to use the specific failed subgraphs exception in the arbitration charter to close the allocations of the now failed subgraphs with a valid PoI. The steps were repeated and optimized over time by the disputed indexer.

The pattern of behavior outlined above created a profitable situation for the indexer; meanwhile, did it contribute in any meaningful way to the network? The disputed indexer stated that the scheme was initially crafted to entice bots into curating on “zombie” deployments and thus wasting their resources. As investigators we are unable to find any other way to confirm this intention. Given the economics of the situation, with a real intention to grieve the frontrunning bots it would be reasonable to expect the indexer to attempt coordination in a public forum to alert curators and indexers of their behavior. The data that the indexer defined in the subgraph, curated to, and indexed indexes ERC-721 contracts, but was designed for the purpose of failing at specific times and did not provide any valuable data indexing to the network. The subgraphs were not queried on the network and all 6 of them are based on the same template with altered failure modes.

In conclusion, the disputed indexer showed a pattern of behavior that was intended purely to capture rewards without providing the intended value of an indexer on the network. Indeed, the disputed indexer themselves has been transparent in their intentions stating that “we did think that the exception rule can be used to create an exclusive environment just like described below.”

Conclusion

After investigating the disputed allocations with the goal of confirming the two working hypothesis we’ve concluded that the PoIs submitted when closing the allocations were valid attestations of syncing that subgraphs data. As for the larger pattern of behavior, it is confirmed that there was a coordinated pattern of behavior from the disputed indexer to create, deploy, curate on, index, and collect indexing rewards on these subgraphs. Based on the data available, the investigators could not find strong reason to believe this action was in any way beneficial to the goals of the network. As such the behavior can be characterized as being purely driven by a goal of exploiting the failed subgraphs exception (no longer in the arbitration charter but once considered canon.) The recommendation is that the disputed indexer be slashed under the clear expectation that has been set in public forums that arbitrators will be looking closely at the intent of disputed allocations behavior for each case.

Investigation Data and Results

Number Deployment (IPFS) PoI Table Divergence Dispute Allocation Indexer Rewards (wei GRT) Indexer rewards (GRT) Indexer Rewards (% of total stake) Failed at BlockNumber Deployment (bytes32) PoI PoI_fisherman PoI_arbRef (prevEpoch) PoI_arbRef (failed-1) PoI_rainbowMatches (arbRef) CreatedAtEpoch ClosedAtEpoch CreatedAtBlock ClosedAtBlockNumber ClosedAtBlockHash
2 QmYLnK6HwxTn3qPe2tURFUQYcxa4VVxLLvdbt6hVpBXmzd none 0xa6e48245e348d1533e31cba5bac4da3125104c927c53b11fa99ab95a8ba15a27 0x5037caea050b4dda604ef5efe908715e5cb71103 1.004198911884378e+21 1004.198911884378 0.085388698525 12855677 0x949d775a90ef63b46d95fc90cf326dea9cceef9b3c4fcb04f9e16581dc7058ee 0x82d3c31866310dceefbad6dd5e4c0721f32b7c16bd93ea16f89be91c870f4ff0 0xb429cb40d26ffa8aec1f0ec31eb6bd5d97ccf0e6062559a4570a521c97a5caac 0xb429cb40d26ffa8aec1f0ec31eb6bd5d97ccf0e6062559a4570a521c97a5caac 0x82d3c31866310dceefbad6dd5e4c0721f32b7c16bd93ea16f89be91c870f4ff0 none 211 212 0xc0d8ef23fefd3590ba30c0ffcc748c952637d7348ccc9d06de5031c43381ec67 12855756 0xe504360b3f50a5b7a62e92ed93e0afb97cf778c2a40303e578dfd46349e2a5a9
5 QmbVGE6Ma2AqzVzKQip6S81dAiSuHeQLny47jzFTLa3DAq none 0x274b39de33d7cf9f4e9feaf3331f56078b0546884aecae156a1f292d42412af8 0x30740c34017407cc939e559b2bbe96b711e3d088 3.4239980482113584e+21 3423.998048211359 0.29114823132 12860424 0xc35cbc58760eae95e5accbd7f866dfb1f89060482b4feb525b795259cfc220aa 0xc2d56a308c3560a511a0ca06ab4bf8d8f3c9891687c21980507e3a97c9ace7a6 0x5bdf4c32ef9498197f087e4c570a98eefa7a83de65818445434e311259939be2 0x5bdf4c32ef9498197f087e4c570a98eefa7a83de65818445434e311259939be2 0xc2d56a308c3560a511a0ca06ab4bf8d8f3c9891687c21980507e3a97c9ace7a6 none 212 213 0x1f2d50cac27c46ea67ebe48016aacc56529cea28fd95f20db163930103e0193e 12862431 0xdd855884efe31cab39686d1cbe165f1a5f89acbf1f17e4fad8a69d32bac64b94
6 QmQEGuJtXiwWni2bsmDDPPor9U1nfQ8s2kbQ6XYCwNjojG none 0xf6dd49f420750111fe0ec51b4e5c4f175ce2a65a864d6f714e7c2ef66829031a 0x0b089eeed8cf24c41f9e74cc750fc654eb5f0fc4 4.001022239045631e+21 4001.022239045631 0.340213555022 12862507 0x1c153b5e9ba74823153c451300a053beb9dcdef20549b22901ef1d502e21d8eb 0xb751485346603a1d165b822499aeb1bd2a5122542c5f40ae3720d666be18126c 0xa2a9fae6226689dbb9755a9cbeed0b8c274d68f48e4ad16c5866f0bd29048cdf 0xa2a9fae6226689dbb9755a9cbeed0b8c274d68f48e4ad16c5866f0bd29048cdf 0xb751485346603a1d165b822499aeb1bd2a5122542c5f40ae3720d666be18126c none 213 214 0xc7f621c45794507aec687a72900de26a669d019aa1ba58839a677e4c5b90f937 12869026 0x929b7b23e791a2cc7c86ce2010fd99ede6cfe32c429c8e991849cf288168ebab
4 QmUxkM4kkYDyVEcUcBGWrvhj5Y6f2uvUTkPTmPjtm76A6k none 0xa2f6ba08ec3ad3e04c4f3ec2ed4344ebb32ba867923ec14b14e46311d954ff59 0xd9036a08e37d0810451a2da60056c1cc935069e7 2.346331016311035e+21 2346.331016311035 0.199512416734 12869144 0x6265679b3f40b30663c7e6f27828c5b1df274b0f422aeede33ac2008100a4df9 0xf9cb11c4cec2e710d80c8e1770a75b2a502c795c9b75b6836853738f3d26b6a2 0xe11d7fe5fc7b7d0701976c793336732c83bbb2ac9c4b5dfd5e8c2b071f478972 0xe11d7fe5fc7b7d0701976c793336732c83bbb2ac9c4b5dfd5e8c2b071f478972 0xf9cb11c4cec2e710d80c8e1770a75b2a502c795c9b75b6836853738f3d26b6a2 none 214 215 0xab6a0e71041e365deb0be08dc88e7a0f762adb73189d8d7c5ffd7247b07e55e9 12876424 0x7347437d0f83310d5e6e3348f0b7474ab3237b248ad13d4b65b47dc1a0b2d27e
1 QmVTUfdp5sJR4uNLq8jM1zR6TLyepejz9gd38YpNT9We5Q none 0x68982a39c7cd4ccc1f22e2bd00de0de20b06cd3e19cb3a65d931d7af410e05e1 0x347032a87c92a79f3eacbf2f4262dbb7eba5165d 974522571664905000000 974.522571664905 0.08286527011 12876877 0x69c184d914985b6681499479840877426f9c1c3e369ba618d8e32a1ea65ff63b 0x367e3cb4ab168ef8231ab1228a51cc6399bca1c17545515573a91b8ddd9cf532 0xeeac889fb37dac0b0cc1ed25a157d60f3ecbca4856bf9d844720b7eab13c97c0 0xeeac889fb37dac0b0cc1ed25a157d60f3ecbca4856bf9d844720b7eab13c97c0 0xcd7a76c99ad622a1f3bece7b495389546180e5444a1e6b5a272846304756ffd7 none 215 216 0x0654ccd498a1224b6bcacdd34689f4b8902aa70804975bd479ec7bd390b39be7 12883331 0x972dabc5337c9540d9e31212866832a07ec3c61956584d9a2986ce81b34984aa
3 QmafMrc51kqWZdBF9u3XzerN2gYAb8wUPU7dHiHuxWewjK none 0x21309e1849923a4e6997280130907daa7cf3e087438b0d23fc01e578e0461dc2 0x73f3810a241d4966f18d0427d5244afec9f2ecf1 495387889841351100000 495.387889841351 0.042123653669 12876871 0xb71738e2cfee4a013449cd09dd89fc1eeb2ead8941e2a8a006d1db605d761d26 0x02911d0e46f521bc791524d6aa9d858db2d82688a7ffdecde3caac0ff8f53a38 0xd39937dea78486c35c1127cee227487d72ff6bfdadf93dd3c7c926241bef932f 0xd39937dea78486c35c1127cee227487d72ff6bfdadf93dd3c7c926241bef932f 0x02911d0e46f521bc791524d6aa9d858db2d82688a7ffdecde3caac0ff8f53a38 none 215 216 0x0db22dbadd706ad2135bf5e4829852fd3f98402c6ab74508bf1bf23ccf5d68f9 12883325 0x1f691464f3df6c97da0b4f0a259674cb9aa7ec55b9f4d944a101f5b9a15ff2c0
13 Likes
12

This has certainly been an interesting thread. Some thoughts…

The Arbitration Charter has not yet been ratified by the Council insofar as I know, which gives the Arbitrator impunity to act outside of the limitations on the Charter.

If we go ahead and slash in this case, it would set an unusual precedent - that an Indexer can be slashed for a correct PoI (assuming that the Arbitrator agrees with the assessment that the PoI is indeed correct, which I think is reasonable for the Charter pre-clarification around errors).

Real-world courts (at least in the US) aren’t as swayed by minor technicalities in the face of broader principles contrary to what movies might make you believe. It works well, and we should adopt the same position.

In this case, the creation of the subgraphs to catch bots and the collection of exclusive rewards are separate things. The claimed intent to catch bots is a distraction because collecting rewards would not be necessary to catch bots.

Everything about this case stinks. No matter how it is ruled some lessons should be incorporated into the Charter to prevent similar situations. Some suggestions:

  • There is no valid PoI for any subgraph created as a part of an economic attack
    • 0x0 must be used instead
    • But, liability gets hard to determine here if an innocent Indexer was brought into participation in the attack an automated fashion.
  • Intentional abuse of loopholes in the Arbitration Charter is grounds for slashing

Taking the broader view, I think a lot of the problem here comes from the existence of Indexer Rewards and specifically the proportion of them in relation to query fees. Query fees can’t be “stolen” so none of these attacks make any sense in a world dominated by query fees. Hopefully we’re there economically by the time we move to remove the Arbitrator as a role since an automated tool would not have the nuance to differentiate between what is malicious or not.

9 Likes
13

The arbitrators accepted the disputes following the conclusions described by @Ford in his post about the investigation.

This is the on-chain transaction that resolved the disputes:

6 Likes