You should be run it when the node is stopped actually.
--delete one more timeBut now it’s too late. Will hope that your node would not lose audit score to much
Correct. That’s exactly what I do. Rsync once while running to reduce downtime and get the bulk of the transfer completed. Then stop the node and run rsync again. And then I do it once more to ensure I get 0 files transferred.
My rsync command is as folllows:
rsync -avHAXxP --delete --stats --human-readable src/ dest/
I’ve used this procedure to move nodes around dozens of times with good results with respect to data integrity. I’ve also recommended this procedure to others. Not sure what the difference was this time. I’ll keep an eye on the audit score and see if it recovers. Thanks!
@Alexey, so I’ve been getting endless log entries for delete failures for these two pieces since the last time I posted. Looks like the garbage collector is tenacious and just doesn’t want to give up.
Today I thought I’d try using touch to create the missing files and perhaps satisfy the delete request. First I confirmed that the files were indeed not there:
$ cd /path/to/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
$ ls co/i2or26selndvxujtrgzxutzar6ibijekkqjzr7m4j6x2amzsza.sj1
ls: cannot access 'co/i2or26selndvxujtrgzxutzar6ibijekkqjzr7m4j6x2amzsza.sj1': No such file or directory
$ ls ac/iahwntg2gc2sdsl6vnqhpiijbmlrkccwkopikn5hynphkm35hq.sj1
ls: cannot access 'ac/iahwntg2gc2sdsl6vnqhpiijbmlrkccwkopikn5hynphkm35hq.sj1': No such file or directory
Then I created the files:
$ touch co/i2or26selndvxujtrgzxutzar6ibijekkqjzr7m4j6x2amzsza.sj1
$ touch ac/iahwntg2gc2sdsl6vnqhpiijbmlrkccwkopikn5hynphkm35hq.sj1
Sure enough, the next time the garbage collector came by, there was no longer an error, but it resulted in a delete expired log entry. Since then, there have been no further logs entries for these pieces, which indicates to me that the issue is resolved. The files I created have also been successfully deleted.
In this case, is delete expired as good as delete successful or is there an issue here?
Relevant log entries are as follows:
2021-03-12 09:31:52 storj03: {"L":"INFO","T":"2021-03-12T16:31:52.568Z","N":"collector","M":"delete expired","Satellite ID":"12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs","Piece ID":"COI2OR26SELNDVXUJTRGZXUTZAR6IBIJEKKQJZR7M4J6X2AMZSZA"}
2021-03-12 09:31:52 storj03: {"L":"INFO","T":"2021-03-12T16:31:52.503Z","N":"collector","M":"delete expired","Satellite ID":"12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs","Piece ID":"ACIAHWNTG2GC2SDSL6VNQHPIIJBMLRKCCWKOPIKN5HYNPHKM35HQ"}
2021-03-12 08:31:52 storj03: {"L":"ERROR","T":"2021-03-12T15:31:52.702Z","N":"collector","M":"unable to delete piece","Satellite ID":"12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs","Piece ID":"COI2OR26SELNDVXUJTRGZXUTZAR6IBIJEKKQJZR7M4J6X2AMZSZA","error":"pieces error: filestore error: file does not exist","errorVerbose":"pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:99\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:298\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:81\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:80\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-03-12 08:31:52 storj03: {"L":"ERROR","T":"2021-03-12T15:31:52.544Z","N":"collector","M":"unable to delete piece","Satellite ID":"12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs","Piece ID":"ACIAHWNTG2GC2SDSL6VNQHPIIJBMLRKCCWKOPIKN5HYNPHKM35HQ","error":"pieces error: filestore error: file does not exist","errorVerbose":"pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:99\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:298\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:81\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:80\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
Thanks.
Thank you for your feedback and method to solve an issue!
I mention that behavior for the team, maybe there’s a bug somewhere.
Delete expired is just your node deleting a piece that had an expiry date after that date has been reached. It’s just a different kind of delete.
Hi, how do you understand where the file was actually located, is there a mess in the logs? There are no direct paths! How can you tell from this where the file was?
2021-11-08T09:12:24.694Z ERROR collector unable to delete piece " “Piece ID”: “5G25O3QBB52IMIK66QS2XU3DQSTK52VPJWT47RTYRX6ORA5P5G3Q"pieces error: filestore error: file does not exist”, “errorVerbose”: “pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
Here is a list of the current satellites, their IDs and their folder names inside storage/blobs/…
Name: ap1.tardigrade.io
ID: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6
Blobs: qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa
Name: europe-north-1.tardigrade.io
ID: 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB
Blobs: 6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa
Name: eu1.tardigrade.io
ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs
Blobs: v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
Name: saltlake.tardigrade.io
ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE
Blobs: pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa
Name: us1.tardigrade.io
ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S
Blobs: ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa
Name: us2.tardigrade.io
ID: 12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo
Blobs: arej6usf33ki2kukzd5v6xgry2tdr56g45pp3aao6llsaaaaaaaa
The first two characters/numbers of a piecestore gives you the subfolder inside the satellites folder.
So your piecestore “5G25O3QBB52IMIK66QS2XU3DQSTK52VPJWT47RTYRX6ORA5P5G3Q”
on satellite ??? should be (after turning into lowercases):
<node-path>/storage/blobs/<satellite-folder>/5g/25o3qbb52imik66qs2xu3dqstk52vpjwt47rtyrx6ora5p5g3q.sj1
Do we have a fix for this in the meantime.
I just stumbled across the same error message again in my logs:
2022-02-08T09:47:22.027Z ERROR collector unable to delete piece {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "CS2UM6J5QHDWQT34H4H5ZL35FEEMHUZINSTDH26OLDZOCBDHECAA", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
I have this several times in the logs and it seems to be exlusively happen for the 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs satellite.
I use a PowerShell script to create the files which are then deleted on next hourly check.
Is this still happening on new pieces? Do you have the log history for that piece? I wonder if this is still expiration events followed by a delete attempt or something else.
Maybe this is the case and these are old(er) pieces. I don’t see an upload in the logs available.
But still there are many tries to delete those pieces:
"SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs",
"CS2UM6J5QHDWQT34H4H5ZL35FEEMHUZINSTDH26OLDZOCBDHECAA": {
"2022-01-31T09:41:44.354Z": "unable to delete piece",
"2022-01-31T10:41:54.056Z": "unable to delete piece",
"2022-01-31T11:41:52.533Z": "unable to delete piece",
"2022-01-31T12:41:53.313Z": "unable to delete piece",
"2022-01-31T13:41:52.962Z": "unable to delete piece",
"2022-01-31T14:41:54.998Z": "unable to delete piece",
"2022-01-31T15:43:49.476Z": "unable to delete piece",
"2022-01-31T16:43:04.748Z": "unable to delete piece",
"2022-01-31T17:44:25.327Z": "unable to delete piece",
"2022-01-31T18:41:51.393Z": "unable to delete piece",
"2022-01-31T19:41:51.972Z": "unable to delete piece",
"2022-01-31T20:41:57.918Z": "unable to delete piece",
"2022-01-31T21:41:49.669Z": "unable to delete piece",
"2022-01-31T22:42:57.248Z": "unable to delete piece",
"2022-01-31T23:41:50.698Z": "unable to delete piece",
"2022-02-01T00:41:50.840Z": "unable to delete piece",
"2022-02-01T01:41:49.440Z": "unable to delete piece",
"2022-02-01T02:41:49.487Z": "unable to delete piece",
"2022-02-01T03:41:50.003Z": "unable to delete piece",
"2022-02-01T04:41:54.783Z": "unable to delete piece",
"2022-02-01T05:42:30.732Z": "unable to delete piece",
"2022-02-01T06:41:44.367Z": "unable to delete piece",
"2022-02-01T07:44:23.124Z": "unable to delete piece",
"2022-02-01T08:41:44.414Z": "unable to delete piece",
"2022-02-01T09:42:04.175Z": "unable to delete piece",
"2022-02-01T10:42:14.248Z": "unable to delete piece",
"2022-02-01T11:41:48.801Z": "unable to delete piece",
"2022-02-01T12:41:48.712Z": "unable to delete piece",
"2022-02-01T13:41:48.989Z": "unable to delete piece",
"2022-02-01T14:41:44.463Z": "unable to delete piece",
"2022-02-01T15:41:58.785Z": "unable to delete piece",
"2022-02-01T16:41:47.078Z": "unable to delete piece",
"2022-02-01T17:41:44.808Z": "unable to delete piece",
"2022-02-01T18:41:44.376Z": "unable to delete piece",
"2022-02-01T19:41:44.519Z": "unable to delete piece",
"2022-02-01T20:41:54.092Z": "unable to delete piece",
"2022-02-01T21:41:47.830Z": "unable to delete piece",
"2022-02-01T22:41:48.557Z": "unable to delete piece",
"2022-02-01T23:41:45.073Z": "unable to delete piece",
"2022-02-02T00:41:46.515Z": "unable to delete piece",
"2022-02-02T01:41:46.742Z": "unable to delete piece",
"2022-02-02T02:41:49.001Z": "unable to delete piece",
"2022-02-02T03:41:47.703Z": "unable to delete piece",
"2022-02-02T04:41:47.607Z": "unable to delete piece",
"2022-02-02T05:41:44.402Z": "unable to delete piece",
"2022-02-02T06:41:44.523Z": "unable to delete piece",
"2022-02-02T07:41:49.087Z": "unable to delete piece",
"2022-02-02T08:41:49.217Z": "unable to delete piece",
"2022-02-02T09:41:51.332Z": "unable to delete piece",
"2022-02-02T10:44:10.729Z": "unable to delete piece",
"2022-02-02T11:42:33.730Z": "unable to delete piece",
"2022-02-02T12:43:03.966Z": "unable to delete piece",
"2022-02-02T13:42:27.791Z": "unable to delete piece",
"2022-02-02T14:43:39.545Z": "unable to delete piece",
"2022-02-02T15:42:00.066Z": "unable to delete piece",
"2022-02-02T16:41:38.892Z": "unable to delete piece",
"2022-02-02T17:41:41.589Z": "unable to delete piece",
"2022-02-02T18:41:30.223Z": "unable to delete piece",
"2022-02-02T19:41:32.661Z": "unable to delete piece",
"2022-02-02T20:41:31.979Z": "unable to delete piece",
"2022-02-02T21:41:30.915Z": "unable to delete piece",
"2022-02-02T22:41:37.037Z": "unable to delete piece",
"2022-02-02T23:41:28.822Z": "unable to delete piece",
"2022-02-03T00:41:32.566Z": "unable to delete piece",
"2022-02-03T01:41:29.164Z": "unable to delete piece",
"2022-02-03T02:41:27.526Z": "unable to delete piece",
"2022-02-03T03:41:43.698Z": "unable to delete piece",
"2022-02-03T04:41:49.403Z": "unable to delete piece",
"2022-02-03T05:43:00.238Z": "unable to delete piece",
"2022-02-03T06:41:24.375Z": "unable to delete piece",
"2022-02-03T07:41:24.616Z": "unable to delete piece",
"2022-02-03T08:41:26.091Z": "unable to delete piece",
"2022-02-03T09:41:26.511Z": "unable to delete piece",
"2022-02-03T10:41:27.007Z": "unable to delete piece",
"2022-02-03T11:41:24.462Z": "unable to delete piece",
"2022-02-03T12:41:27.801Z": "unable to delete piece",
"2022-02-03T13:41:27.861Z": "unable to delete piece",
"2022-02-03T14:41:27.389Z": "unable to delete piece",
"2022-02-03T15:41:28.293Z": "unable to delete piece",
"2022-02-03T16:41:24.464Z": "unable to delete piece",
"2022-02-03T17:41:24.364Z": "unable to delete piece",
"2022-02-03T18:41:26.641Z": "unable to delete piece",
"2022-02-03T19:41:24.655Z": "unable to delete piece",
"2022-02-03T20:41:25.048Z": "unable to delete piece",
"2022-02-03T21:41:26.631Z": "unable to delete piece",
"2022-02-03T22:41:24.510Z": "unable to delete piece",
"2022-02-03T23:41:27.429Z": "unable to delete piece",
"2022-02-04T00:41:28.016Z": "unable to delete piece",
"2022-02-04T01:41:24.389Z": "unable to delete piece",
"2022-02-04T02:41:24.462Z": "unable to delete piece",
"2022-02-04T03:41:24.651Z": "unable to delete piece",
"2022-02-04T04:41:26.767Z": "unable to delete piece",
"2022-02-04T05:41:24.624Z": "unable to delete piece",
"2022-02-04T06:41:27.214Z": "unable to delete piece",
"2022-02-04T07:41:24.389Z": "unable to delete piece",
"2022-02-04T08:41:25.413Z": "unable to delete piece",
"2022-02-04T09:41:24.395Z": "unable to delete piece",
"2022-02-04T10:41:27.090Z": "unable to delete piece",
"2022-02-04T11:41:24.431Z": "unable to delete piece",
"2022-02-04T12:41:27.146Z": "unable to delete piece",
"2022-02-04T13:41:26.453Z": "unable to delete piece",
"2022-02-04T14:41:26.456Z": "unable to delete piece",
"2022-02-04T15:41:24.386Z": "unable to delete piece",
"2022-02-04T16:41:24.463Z": "unable to delete piece",
"2022-02-04T17:41:27.072Z": "unable to delete piece",
"2022-02-04T18:41:24.440Z": "unable to delete piece",
"2022-02-04T19:41:24.397Z": "unable to delete piece",
"2022-02-04T20:41:24.426Z": "unable to delete piece",
"2022-02-04T21:41:24.731Z": "unable to delete piece",
"2022-02-04T22:41:24.468Z": "unable to delete piece",
"2022-02-04T23:41:24.429Z": "unable to delete piece",
"2022-02-05T00:41:24.613Z": "unable to delete piece",
"2022-02-05T01:41:27.346Z": "unable to delete piece",
"2022-02-05T02:41:27.165Z": "unable to delete piece",
"2022-02-05T03:41:24.415Z": "unable to delete piece",
"2022-02-05T04:41:27.491Z": "unable to delete piece",
"2022-02-05T05:41:24.403Z": "unable to delete piece",
"2022-02-05T06:41:25.070Z": "unable to delete piece",
"2022-02-05T07:41:27.371Z": "unable to delete piece",
"2022-02-05T08:41:27.471Z": "unable to delete piece",
"2022-02-05T09:41:26.341Z": "unable to delete piece",
"2022-02-05T10:41:27.185Z": "unable to delete piece",
"2022-02-05T11:41:26.141Z": "unable to delete piece",
"2022-02-05T12:41:26.925Z": "unable to delete piece",
"2022-02-05T13:41:26.924Z": "unable to delete piece",
"2022-02-05T14:41:26.749Z": "unable to delete piece",
"2022-02-05T15:41:25.087Z": "unable to delete piece",
"2022-02-05T16:41:24.449Z": "unable to delete piece",
"2022-02-05T17:41:24.972Z": "unable to delete piece",
"2022-02-05T18:41:26.728Z": "unable to delete piece",
"2022-02-05T19:41:25.049Z": "unable to delete piece",
"2022-02-05T20:41:24.455Z": "unable to delete piece",
"2022-02-05T21:41:24.442Z": "unable to delete piece",
"2022-02-05T22:41:24.415Z": "unable to delete piece",
"2022-02-05T23:41:25.026Z": "unable to delete piece",
"2022-02-06T00:41:27.724Z": "unable to delete piece",
"2022-02-06T01:41:25.728Z": "unable to delete piece",
"2022-02-06T02:41:27.394Z": "unable to delete piece",
"2022-02-06T03:41:24.983Z": "unable to delete piece",
"2022-02-06T04:41:26.179Z": "unable to delete piece",
"2022-02-06T05:41:25.034Z": "unable to delete piece",
"2022-02-06T06:41:24.891Z": "unable to delete piece",
"2022-02-06T07:41:25.654Z": "unable to delete piece",
"2022-02-06T08:41:26.984Z": "unable to delete piece",
"2022-02-06T09:41:26.918Z": "unable to delete piece",
"2022-02-06T10:41:27.978Z": "unable to delete piece",
"2022-02-06T11:41:27.760Z": "unable to delete piece",
"2022-02-06T12:41:24.462Z": "unable to delete piece",
"2022-02-06T13:41:27.857Z": "unable to delete piece",
"2022-02-06T14:41:27.344Z": "unable to delete piece",
"2022-02-06T15:41:26.854Z": "unable to delete piece",
"2022-02-06T16:41:24.405Z": "unable to delete piece",
"2022-02-06T17:41:24.529Z": "unable to delete piece",
"2022-02-06T18:41:25.054Z": "unable to delete piece",
"2022-02-06T19:41:26.256Z": "unable to delete piece",
"2022-02-06T20:41:24.601Z": "unable to delete piece",
"2022-02-06T21:41:24.395Z": "unable to delete piece",
"2022-02-06T22:41:24.487Z": "unable to delete piece",
"2022-02-06T23:41:24.398Z": "unable to delete piece",
"2022-02-07T00:41:27.993Z": "unable to delete piece",
"2022-02-07T01:41:26.986Z": "unable to delete piece",
"2022-02-07T02:41:24.440Z": "unable to delete piece",
"2022-02-07T03:41:26.550Z": "unable to delete piece",
"2022-02-07T04:41:26.190Z": "unable to delete piece",
"2022-02-07T05:41:24.503Z": "unable to delete piece",
"2022-02-07T06:41:24.468Z": "unable to delete piece",
"2022-02-07T07:41:29.011Z": "unable to delete piece",
"2022-02-07T08:41:28.992Z": "unable to delete piece",
"2022-02-07T09:41:34.616Z": "unable to delete piece",
"2022-02-07T10:42:25.776Z": "unable to delete piece",
"2022-02-07T11:41:33.046Z": "unable to delete piece",
"2022-02-07T12:49:15.394Z": "unable to delete piece",
"2022-02-07T13:41:54.053Z": "unable to delete piece",
"2022-02-07T14:41:52.540Z": "unable to delete piece",
"2022-02-07T15:41:55.984Z": "unable to delete piece",
"2022-02-07T16:41:32.067Z": "unable to delete piece",
"2022-02-07T17:41:36.515Z": "unable to delete piece",
"2022-02-07T18:41:32.372Z": "unable to delete piece",
"2022-02-07T19:41:33.036Z": "unable to delete piece",
"2022-02-07T20:41:31.889Z": "unable to delete piece",
"2022-02-07T21:41:32.796Z": "unable to delete piece",
"2022-02-07T22:41:34.926Z": "unable to delete piece",
"2022-02-07T23:41:35.881Z": "unable to delete piece",
"2022-02-07T23:47:15.154Z": "unable to delete piece",
"2022-02-08T00:47:14.935Z": "unable to delete piece",
"2022-02-08T01:47:17.798Z": "unable to delete piece",
"2022-02-08T02:47:17.083Z": "unable to delete piece",
"2022-02-08T03:47:23.702Z": "unable to delete piece",
"2022-02-08T04:47:13.361Z": "unable to delete piece",
"2022-02-08T05:47:15.022Z": "unable to delete piece",
"2022-02-08T06:48:22.070Z": "unable to delete piece",
"2022-02-08T07:47:14.444Z": "unable to delete piece",
"2022-02-08T08:47:28.825Z": "unable to delete piece",
"2022-02-08T09:47:22.027Z": "unable to delete piece",
"2022-02-08T10:47:15.326Z": "unable to delete piece",
"2022-02-08T11:47:12.023Z": "unable to delete piece"
}
This seems to be the issue on Github here: https://github.com/storj/storj/issues/4192