Piecestore: file does not exist error

Node Operators FAQ
, , ,
1

Recently I’ve seen an uptick of the following error in my logs:

2024-08-12T04:10:20-04:00 WARN collector unable to delete piece {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "ZBKXBN6T3WI3Z3XDKS5UQFNEICFQ4S66XM7WKVVKZOQXIWCNIO5Q", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).Stat:124\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:340\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).DeleteWithStorageFormat:320\n\tstorj.io/storj/storagenode/pieces.(*Store).DeleteSkipV0:359\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:112\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:68\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/collector.(*Service).Run:64\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}

I’ve had ~86k instances of this log in the last 24 hours on a large node (filled 50TB one). What would be the steps to troubleshoot this?

2

These pieces with TTL were moved by the garbage collector to the trash before the expiration date, because the customer has deleted them before the expiration.
So, kind of “normal”

3

The satellite is slc, so the customer is storj. Are you really deleting all those TTL files? If so, why?

4

Saltlake is not simply satellite used by Storj to store their own data, rather it is the satellite used by Storj for testing and for capacity reservation (filling up nodes with random data, paid as normal data).

Data uploaded for capacity reservation has a TTL, Time To Live, therefore it expires after a certain time has passed. This is done to A. Make sure capacity is freed up after a while (unless new TTL data is uploaded), and B. Make cleanup of data faster by avoiding garbage collection (ignoring the current issues where some TTL data isnt collected when expired).

So the reason that Storj is “deleting” the TTL data, is because the data is meant to be deleted.

1 Like
5

where can I make a suggestion that this particular error message be muted or moved to a lower category like INFO?

1 Like
6

Storage Node feature requests - voting