IPFS Sync of a directory with a lot of files?

Questions & Feedback
1

I’m trying to do an IPFS sync of a directory with a lot of small metadata files (Over 10,000 files, but each one is under 1 kbyte) to TheGraph’s IPFS node (to ensure ipfs.cat doesn’t timeout when my subgraph is processing). I need it to be synced as a wrapped directory so that the files retain their filenames. However, I keep getting a 413 Entity too large error when I try to sync it to TheGraph’s IPFS node (https://api.thegraph.com/ipfs/api/v0) (I’m syncing via ipfsClient.addAll).

Is there a way to fix this? I’m fairly it’s due to the large number of files (or perhaps total size of the files), as I’ve tried syncing with a directory with around 50 files and that worked fine.

2

Hey @globetro! Have you tried already without syncing to the IPFS node - did you get the ipfs.cat timeout? If the directory is readily available via peering, it may be able to sync successfully? Though 10,000 files is quite a lot

3

cc @Leo who is working on File Data Sources, we’ll need to handle this directory use case