ERROR collector unable to delete piece

If you are working from the command line you can execute the command below replacing file.txt with the filename of the file you wish to create.

echo.> file.txt

You can also easily do this using windows explorer.
Open the folder → Right click in folder → Hover over new → select Text Document. This should create a new blank file waiting to be named. Just name it to match the missing file.

Let me know if this doesn’t work for you and I can explore other potential issues.

but in which folder should I put the created file?

exempl

|2021-07-20T21:21:36.926-0300|ERROR|collector|unable to delete piece|{Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: FRJGXEF52MAXHCIHTTDSBJSEZ6VPMDQQW6L7ZXP2JYKASD37PUSQ, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n[tstorj.io/storj/storage/filestore.(*blobStore](http://tstorj.io/storj/storage/filestore.(*blobStore)).Stat:103\n[tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache](http://tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache)).pieceSizes:239\n[tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache](http://tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache)).Delete:220\n[tstorj.io/storj/storagenode/pieces.(*Store](http://tstorj.io/storj/storagenode/pieces.(*Store)).Delete:299\n[tstorj.io/storj/storagenode/collector.(*Service](http://tstorj.io/storj/storagenode/collector.(*Service)).Collect:97\n[tstorj.io/storj/storagenode/collector.(*Service](http://tstorj.io/storj/storagenode/collector.(*Service)).Run.func1:57\n[tstorj.io/common/sync2.(*Cycle](http://tstorj.io/common/sync2.(*Cycle)).Run:92\n[tstorj.io/storj/storagenode/collector.(*Service](http://tstorj.io/storj/storagenode/collector.(*Service)).Run:53\n[tstorj.io/storj/private/lifecycle.(*Group](http://tstorj.io/storj/private/lifecycle.(*Group)).Run.func2.1:87\n\truntime/pprof.Do:40\n[tstorj.io/storj/private/lifecycle.(*Group](http://tstorj.io/storj/private/lifecycle.(*Group)).Run.func2:86\n[tgolang.org/x/sync/errgroup.(*Group](http://tgolang.org/x/sync/errgroup.(*Group)).Go.func1:57}|

in which folder do i see this piece?

Piece ID: FRJGXEF52MAXHCIHTTDSBJSEZ6VPMDQQW6L7ZXP2JYKASD37PUSQ

I am looking into what the path to the blob file is currently. I’ll get back to you as soon as possible.

1 Like

Each satellite has their own folder in the blobs folder in your storage location. For the satellite this piece is on that’s v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa. The subfolders you see inside there represent the first 2 characters of the piece id. So in your case the folder should be fr. The filename inside that folder is the rest of the piece id with a .sj1 extension. So jgxef52maxhcihttdsbjsez6vpmdqqw6l7zxp2jykasd37pusq.sj1. Path and filenames are in lower case, which is a little annoying since they are logged in upper case. On windows systems this may not matter, but linux/unix systems are case sensitive.

Entire path should be:
[StorjDataLocation]\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\fr\jgxef52maxhcihttdsbjsez6vpmdqqw6l7zxp2jykasd37pusq.sj1

Please replace the [StorjDataLocation] part with where your storj data is stored.

Last note: If you use explorer to create the file as @Code_Breaker mentioned. Make sure you have displaying of known file extensions switched on. Otherwise you may unknowingly be naming your file jgxef52maxhcihttdsbjsez6vpmdqqw6l7zxp2jykasd37pusq.sj1.txt.

It may be easier to just run this in CMD:

echo.> [StorjDataLocation]\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\fr\jgxef52maxhcihttdsbjsez6vpmdqqw6l7zxp2jykasd37pusq.sj1

Again, please replace the [StorjDataLocation] part with where your storj data is stored.

From the logs I see that you’ve got multiple pieces that have this issue though. So you will need to do this for all of them. You can use https://convertcase.net/ to convert the upper case piece id’s to lower case.

5 Likes

In the logs you posted it seems to have all been for the same satellites. If you have the same issue on other satellites, here’s a reference for satellites and corresponding ID’s and blobs folders.

Satellite Name: satellite.stefan-benten.de
Satellite ID: 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW
Satellite ID hex: 004AE89E970E703DF42BA4AB1416A3B30B7E1D8E14AA0E558F7EE26800000000
Blobs Folder: abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa

Satellite Name: ap1.storj.io (formerly: asia-east-1.tardigrade.io)
Satellite ID: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6
Satellite ID hex: 84A74C2CD43C5BA76535E1F42F5DF7C287ED68D33522782F4AFABFDB40000000
Blobs Folder: qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa

Satellite Name: us1.storj.io (formerly: us-central-1.tardigrade.io)
Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S
Satellite ID hex: A28B4F04E10BAE85D67F4C6CB82BF8D4C0F0F47A8EA72627524DEB6EC0000000
Blobs Folder: ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa

Satellite Name: eu1.storj.io (formerly: europe-west-1.tardigrade.io)
Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs
Satellite ID hex: AF2C42003EFC826AB4361F73F9D890942146FE0EBE806786F8E7190800000000
Blobs Folder: v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa

Satellite Name: saltlake.tardigrade.io
Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE
Satellite ID hex: 7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000
Blobs Folder: pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa

Satellite Name: europe-north-1.tardigrade.io
Satellite ID: 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB
Satellite ID hex: F474535A19DB00DB4F8071A1BE6C2551F4DED6A6E38F0818C68C68D000000000
blobs folder: 6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa

Satellite Name: us2.storj.io (formerly: us2.tardigrade.io)
Satellite ID: 12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo
Satellite ID hex: 04489F5245DED48D2A8AC8FB5F5CD1C6A638F7C6E75EFD800EF2D72000000000
blobs folder: arej6usf33ki2kukzd5v6xgry2tdr56g45pp3aao6llsaaaaaaaa
5 Likes

thank you very much…resolved. Code_Breaker, BrightSilence. :+1:

1 Like

it was just this new error = could not send delete piece to trash

What to do in this case?

2021-07-22T09:18:38.418-0300 ERROR piecedeleter could not send delete piece to trash {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Piece ID”: “IXZSEWGXS7QNUD6YJBQ672VQZYWE7462NYEVZBYRJZUNH7HHX57Q”, “error”: “pieces error: v0pieceinfodb: sql: no rows in result set”, “errorVerbose”: “pieces error: v0pieceinfodb: sql: no rows in result set\n\tstorj.io/storj/storagenode/storagenodedb.(*v0PieceInfoDB).Get:131\n\tstorj.io/storj/storagenode/pieces.(*Store).MigrateV0ToV1:404\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:348\n\tstorj.io/storj/storagenode/pieces.(*Deleter).deleteOrTrash:185\n\tstorj.io/storj/storagenode/pieces.(*Deleter).work:135\n\tstorj.io/storj/storagenode/pieces.(*Deleter).Run.func1:72\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

These should be a one off. It only tries to do regular deletes once. So there isn’t really a need to fix this.

But these errors do suggest you may have had file loss. Do you have other errors in the logs as well? Most importantly in lines with GET_AUDIT or GET_REPAIR in them. But it’s worth posting any type of error you encounter.

@Alexey
Maybe the solution can be made a topic in a FAQ category? Something like “Which data folder belongs to which satellite?” or something like that?

1 Like

I see another error exclusively on the EU1 satellite:

ERROR   collector       unable to delete piece{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "FIJ57I2RNE4SKNKA7BTKBUSGWCGVOXXN53OMF4WDR57S5AXAAPBQ", "error": "pieces error: filestore error: file does not exist"

Please, try to search for this piece in your logs to see a history - when it was removed (or maybe lost)

I only see log entries with the request to delete. Earlier logs have been truncated already.
What puzzles me is that this error is on the EU satellite only.

Unlikely. We need to have a history for this piece to make such a claim.
Until than - the node has lost this piece for some reason. We need to figure out this reason. The good news - your node is lucky because the customer has deleted a piece before it was audited.

But this is what it is. I have checked on a second node and see the fully same issue: Piece not found for deletion, no history for the piece and on the 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs satellite only:

2021-09-12T01:54:31.291Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-09-12T02:54:30.900Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-09-12T03:54:33.001Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T04:54:32.094Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T05:54:35.624Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T06:54:30.527Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T07:54:34.709Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T08:54:31.471Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T09:54:30.268Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T10:54:30.448Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T11:54:31.718Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T12:54:42.283Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T13:54:31.886Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T14:54:31.008Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T15:54:32.503Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T16:54:30.062Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T17:54:30.332Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T18:54:33.861Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T19:54:42.558Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T20:54:41.174Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T21:54:30.031Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T22:54:32.415Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T23:54:33.619Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T00:54:34.540Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T01:54:33.878Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T02:54:39.011Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T03:54:48.723Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T04:54:32.028Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T05:54:30.803Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T06:54:30.089Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T07:54:29.646Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T08:54:30.608Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T09:54:30.241Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T10:54:31.271Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T11:54:32.768Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T12:54:29.694Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T13:54:32.187Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T14:54:31.265Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T15:54:32.339Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T16:54:29.653Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T17:54:34.853Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T18:54:32.025Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T19:54:30.056Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T20:54:37.118Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T21:54:34.489Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T22:54:30.083Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T23:54:30.158Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T00:54:30.190Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T01:54:29.761Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T02:54:30.885Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T03:54:43.525Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T04:54:34.061Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T05:54:29.932Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T06:54:31.411Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T07:54:31.937Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T08:54:30.520Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T09:54:31.471Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T10:54:30.872Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T11:54:39.847Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T12:54:31.585Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T13:54:36.579Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T14:54:39.164Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T15:54:31.407Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T16:54:30.231Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T17:54:33.449Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T18:54:32.542Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T19:54:30.766Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T20:54:29.895Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T21:54:32.854Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T22:54:31.948Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T23:54:29.997Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T00:54:30.704Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T01:54:30.469Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T02:54:30.412Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T03:54:35.896Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T04:54:31.287Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T05:54:30.399Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T05:57:29.528Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T06:57:33.369Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T07:57:27.653Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T08:57:26.486Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T09:57:33.226Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T10:57:27.636Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T11:57:26.901Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T12:57:34.250Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T13:57:26.959Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T14:57:26.798Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T15:57:27.008Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T16:57:31.347Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T17:57:28.230Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T18:57:27.078Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T19:57:27.055Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T20:57:28.856Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T21:57:26.677Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T22:57:30.757Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T23:57:28.039Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T00:57:26.525Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T01:57:26.954Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T02:57:26.843Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T03:57:26.552Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T04:57:31.311Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T05:57:27.380Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T06:57:26.844Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }

The log shows entries as far back as they are available.

I created an issue for this case: [storagenode] The garbage collector is trying to delete a piece over and over again · Issue #4192 · storj/storj · GitHub
I’m still not convinced that the piece was not lost by the node.
But so far the statistics are in favor of the presence of the bug somewhere.

Could you please find records where an uplink was actually tried to upload a piece (or it has been uploaded) to your node?

Yes, but I need a script doing that. I can’t go through every line of the log manually.
Maybe someone has such a script. I think it should work that way that it finds every instance of “unable to delete piece” gets the piece id from that line and then search for the corresponding upload or at least something other than “unable to delete piece”.

You can use the script from the issue

Unfortunately logs don’t go back long enough to see when the pieces were uploaded or what else has happened before deletion attempts.
One node has several pieces with delete issues (7) others have less (1).

I seem to be having a similar issue, however I do have my logs.

All seem to follow a pattern of a delete expired followed by several unable to delete piece errors.

I’ve knocked up a quick script to look at what is causing ‘unable to delete piece’ errors in today’s log, then to look back and report all mentions of those pieces until the first ‘unable to delete piece’ errors start.
This node is a raspberry pi docker storage node which has had USB FS issues, so I may have possibly lost data on hangups, there was an extensive fsck repair after I replaced the USB adapter with a more reliable one, however storage nodes shouldn’t really care if a piece cannot be deleted. If it’s gone there should be no work to do and no error - maybe a warning and then regard as success.
I’ve had 30 ‘unable to delete piece’ errors today already since the logs were rotated 10 hours ago.

== LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ ==
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:20:34.618Z	INFO	piecestore	upload started	{"Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 344865003008}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:20:53.870Z	INFO	piecestore	uploaded	{"Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Size": 2319360}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:24:18.513Z	INFO	piecestore	download started	{"Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:24:28.203Z	INFO	piecestore	downloaded	{"Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:23:25.824Z	INFO	collector	delete expired	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ"}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:39:23.102Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

== P4UDM2RZOJKNXSTQ5QV35WTZQHDQ766QN3MQJROUTIG353VU5LFA ==
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:10:09.486Z	INFO	piecestore	upload started	{"Piece ID": "P4UDM2RZOJKNXSTQ5QV35WTZQHDQ766QN3MQJROUTIG353VU5LFA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 344889131008}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:10:09.531Z	INFO	piecestore	uploaded	{"Piece ID": "P4UDM2RZOJKNXSTQ5QV35WTZQHDQ766QN3MQJROUTIG353VU5LFA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Size": 512}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:23:25.454Z	INFO	collector	delete expired	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "P4UDM2RZOJKNXSTQ5QV35WTZQHDQ766QN3MQJROUTIG353VU5LFA"}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:39:22.985Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "P4UDM2RZOJKNXSTQ5QV35WTZQHDQ766QN3MQJROUTIG353VU5LFA", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

== ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A ==
/mnt/log/storj//node.log-20210904.gz:2021-09-03T13:58:06.489Z	INFO	piecestore	upload started	{"Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 344906312192}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T13:58:06.557Z	INFO	piecestore	uploaded	{"Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Size": 23552}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:03:50.449Z	INFO	piecestore	download started	{"Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:03:51.956Z	INFO	piecestore	downloaded	{"Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:23:25.250Z	INFO	collector	delete expired	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A"}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:39:22.670Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2 Likes