ERROR collector unable to delete piece

But this is what it is. I have checked on a second node and see the fully same issue: Piece not found for deletion, no history for the piece and on the 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs satellite only:

2021-09-12T01:54:31.291Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-09-12T02:54:30.900Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-09-12T03:54:33.001Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T04:54:32.094Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T05:54:35.624Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T06:54:30.527Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T07:54:34.709Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T08:54:31.471Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T09:54:30.268Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T10:54:30.448Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T11:54:31.718Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T12:54:42.283Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T13:54:31.886Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T14:54:31.008Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T15:54:32.503Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T16:54:30.062Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T17:54:30.332Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T18:54:33.861Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T19:54:42.558Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T20:54:41.174Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T21:54:30.031Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T22:54:32.415Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T23:54:33.619Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T00:54:34.540Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T01:54:33.878Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T02:54:39.011Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T03:54:48.723Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T04:54:32.028Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T05:54:30.803Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T06:54:30.089Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T07:54:29.646Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T08:54:30.608Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T09:54:30.241Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T10:54:31.271Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T11:54:32.768Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T12:54:29.694Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T13:54:32.187Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T14:54:31.265Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T15:54:32.339Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T16:54:29.653Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T17:54:34.853Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T18:54:32.025Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T19:54:30.056Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T20:54:37.118Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T21:54:34.489Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T22:54:30.083Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T23:54:30.158Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T00:54:30.190Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T01:54:29.761Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T02:54:30.885Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T03:54:43.525Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T04:54:34.061Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T05:54:29.932Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T06:54:31.411Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T07:54:31.937Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T08:54:30.520Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T09:54:31.471Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T10:54:30.872Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T11:54:39.847Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T12:54:31.585Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T13:54:36.579Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T14:54:39.164Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T15:54:31.407Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T16:54:30.231Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T17:54:33.449Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T18:54:32.542Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T19:54:30.766Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T20:54:29.895Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T21:54:32.854Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T22:54:31.948Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T23:54:29.997Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T00:54:30.704Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T01:54:30.469Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T02:54:30.412Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T03:54:35.896Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T04:54:31.287Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T05:54:30.399Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T05:57:29.528Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T06:57:33.369Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T07:57:27.653Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T08:57:26.486Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T09:57:33.226Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T10:57:27.636Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T11:57:26.901Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T12:57:34.250Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T13:57:26.959Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T14:57:26.798Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T15:57:27.008Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T16:57:31.347Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T17:57:28.230Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T18:57:27.078Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T19:57:27.055Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T20:57:28.856Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T21:57:26.677Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T22:57:30.757Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T23:57:28.039Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T00:57:26.525Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T01:57:26.954Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T02:57:26.843Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T03:57:26.552Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T04:57:31.311Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T05:57:27.380Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T06:57:26.844Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }

The log shows entries as far back as they are available.

I created an issue for this case: [storagenode] The garbage collector is trying to delete a piece over and over again · Issue #4192 · storj/storj · GitHub
I’m still not convinced that the piece was not lost by the node.
But so far the statistics are in favor of the presence of the bug somewhere.

Could you please find records where an uplink was actually tried to upload a piece (or it has been uploaded) to your node?

Yes, but I need a script doing that. I can’t go through every line of the log manually.
Maybe someone has such a script. I think it should work that way that it finds every instance of “unable to delete piece” gets the piece id from that line and then search for the corresponding upload or at least something other than “unable to delete piece”.

You can use the script from the issue

Unfortunately logs don’t go back long enough to see when the pieces were uploaded or what else has happened before deletion attempts.
One node has several pieces with delete issues (7) others have less (1).

I seem to be having a similar issue, however I do have my logs.

All seem to follow a pattern of a delete expired followed by several unable to delete piece errors.

I’ve knocked up a quick script to look at what is causing ‘unable to delete piece’ errors in today’s log, then to look back and report all mentions of those pieces until the first ‘unable to delete piece’ errors start.
This node is a raspberry pi docker storage node which has had USB FS issues, so I may have possibly lost data on hangups, there was an extensive fsck repair after I replaced the USB adapter with a more reliable one, however storage nodes shouldn’t really care if a piece cannot be deleted. If it’s gone there should be no work to do and no error - maybe a warning and then regard as success.
I’ve had 30 ‘unable to delete piece’ errors today already since the logs were rotated 10 hours ago.

== LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ ==
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:20:34.618Z	INFO	piecestore	upload started	{"Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 344865003008}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:20:53.870Z	INFO	piecestore	uploaded	{"Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Size": 2319360}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:24:18.513Z	INFO	piecestore	download started	{"Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:24:28.203Z	INFO	piecestore	downloaded	{"Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:23:25.824Z	INFO	collector	delete expired	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ"}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:39:23.102Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

== P4UDM2RZOJKNXSTQ5QV35WTZQHDQ766QN3MQJROUTIG353VU5LFA ==
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:10:09.486Z	INFO	piecestore	upload started	{"Piece ID": "P4UDM2RZOJKNXSTQ5QV35WTZQHDQ766QN3MQJROUTIG353VU5LFA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 344889131008}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:10:09.531Z	INFO	piecestore	uploaded	{"Piece ID": "P4UDM2RZOJKNXSTQ5QV35WTZQHDQ766QN3MQJROUTIG353VU5LFA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Size": 512}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:23:25.454Z	INFO	collector	delete expired	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "P4UDM2RZOJKNXSTQ5QV35WTZQHDQ766QN3MQJROUTIG353VU5LFA"}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:39:22.985Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "P4UDM2RZOJKNXSTQ5QV35WTZQHDQ766QN3MQJROUTIG353VU5LFA", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

== ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A ==
/mnt/log/storj//node.log-20210904.gz:2021-09-03T13:58:06.489Z	INFO	piecestore	upload started	{"Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 344906312192}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T13:58:06.557Z	INFO	piecestore	uploaded	{"Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Size": 23552}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:03:50.449Z	INFO	piecestore	download started	{"Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:03:51.956Z	INFO	piecestore	downloaded	{"Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:23:25.250Z	INFO	collector	delete expired	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A"}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:39:22.670Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2 Likes

Can you provide that script?

2 Likes

script is very highly specific to my setup but here you go:

$ cat look-errors.sh 
#!/bin/sh
LOGDIR=/mnt/log/storj
if [ "$1" ]; then
	LOG=$1
else
	LOG=$LOGDIR/node.log
fi

echo "looking in $LOG..."

FRAGS="`grep 'ERROR' $LOG|grep 'unable to '|sed -e 's/.*"Piece ID": "//' -e 's/".*//'|sort -u`"

for FRAG in $FRAGS; do
	echo "== $FRAG =="
	zgrep "$FRAG" $LOGDIR//node.log-*.gz | awk '1;/unable to/{exit}'
	echo ""
done
5 Likes

Thanks. I’ll see if I can make it work for me setup.

Storj keeps stopping and is getting hung up on being able to read/write/delete some files

That would normally understandable but there’s not a clear problem as the iSCSI slice has plenty of space with 1.5TB free and the permissions are open read/write/all.

Is there a scrub command or something to run to verify the integrity of the whole StorjDB and flag bad files for manual deletion/overwrite/repair?

Log pull:

2021-10-09T10:03:45.138-0400	ERROR	piecestore:cache	error getting current used space: 	{"error": "CreateFile S:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\nf/2delxkcnwfrwhkjcnhu4qeiixuds5eemxx3bs3pkj3kij4suxq.sj1: The file or directory is corrupted and unreadable.; CreateFile S:\\Storj\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\xi/eejste6pi6qgzjq5sewdxqiizx2pa4e3sabndypulqqhnfjesq.sj1: The file or directory is corrupted and unreadable.", "errorVerbose": "group:\n--- CreateFile S:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\nf/2delxkcnwfrwhkjcnhu4qeiixuds5eemxx3bs3pkj3kij4suxq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- CreateFile S:\\Storj\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\xi/eejste6pi6qgzjq5sewdxqiizx2pa4e3sabndypulqqhnfjesq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-10-09T10:03:45.138-0400	ERROR	services	unexpected shutdown of a runner	{"name": "piecestore:cache", "error": "CreateFile S:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\nf/2delxkcnwfrwhkjcnhu4qeiixuds5eemxx3bs3pkj3kij4suxq.sj1: The file or directory is corrupted and unreadable.; CreateFile S:\\Storj\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\xi/eejste6pi6qgzjq5sewdxqiizx2pa4e3sabndypulqqhnfjesq.sj1: The file or directory is corrupted and unreadable.", "errorVerbose": "group:\n--- CreateFile S:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\nf/2delxkcnwfrwhkjcnhu4qeiixuds5eemxx3bs3pkj3kij4suxq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- CreateFile S:\\Storj\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\xi/eejste6pi6qgzjq5sewdxqiizx2pa4e3sabndypulqqhnfjesq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-10-09T10:03:45.139-0400	INFO	piecestore	upload canceled	{"Piece ID": "PBBFK3B6T4Y4EDF2IIHIH4SDR25TP4U3IGU36TBPHX3MRQY5B5EQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-10-09T10:03:45.139-0400	INFO	piecestore	downloaded	{"Piece ID": "USZBEQH2SOQXNTXTO64CLZ5UHCDPCBBOLQUYBSZVAPYYXTFIDSYA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-10-09T10:03:45.139-0400	INFO	piecestore	downloaded	{"Piece ID": "6UMXYHBID4UUOXON5ZASCUOVKGATEWXNC7RMJXWD6P5G3RJ6J7IQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-10-09T10:03:45.139-0400	INFO	piecestore	downloaded	{"Piece ID": "USZBEQH2SOQXNTXTO64CLZ5UHCDPCBBOLQUYBSZVAPYYXTFIDSYA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-10-09T10:03:45.140-0400	INFO	piecestore	upload canceled	{"Piece ID": "D624YVXN7CRGMSMGLAP23ZAH4M5V4AXJR2F2XVOQICNZFZB7TXDA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 2105344}
2021-10-09T10:03:45.152-0400	FATAL	Unrecoverable error	{"error": "CreateFile S:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\nf/2delxkcnwfrwhkjcnhu4qeiixuds5eemxx3bs3pkj3kij4suxq.sj1: The file or directory is corrupted and unreadable.; CreateFile S:\\Storj\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\xi/eejste6pi6qgzjq5sewdxqiizx2pa4e3sabndypulqqhnfjesq.sj1: The file or directory is corrupted and unreadable.", "errorVerbose": "group:\n--- CreateFile S:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\nf/2delxkcnwfrwhkjcnhu4qeiixuds5eemxx3bs3pkj3kij4suxq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- CreateFile S:\\Storj\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\xi/eejste6pi6qgzjq5sewdxqiizx2pa4e3sabndypulqqhnfjesq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-10-09T11:18:23.531-0400	INFO	Configuration loaded	{"Location": "C:\\Program Files\\Storj\\Storage Node\\config.yaml"}
2021-10-09T11:18:23.543-0400	INFO	Operator email	{"Address": "chattphotos@gmail.com"}
2021-10-09T11:18:23.543-0400	INFO	Operator wallet	{"Address": "0x5ad4a2f04F50b6Dc48F7B79c1bc22A6927575E4a"}
2021-10-09T11:18:24.158-0400	INFO	Telemetry enabled	{"instance ID": "161sXNjyPE65jjC5uNXTy1jvwqprwTDzXKHc4UVdBExeUUyGiW"}
2021-10-09T11:18:24.191-0400	INFO	db.migration	Database Version	{"version": 53}
2021-10-09T11:18:24.371-0400	INFO	preflight:localtime	start checking local system clock with trusted satellites' system clock.
2021-10-09T11:18:24.924-0400	INFO	preflight:localtime	local system clock is in sync with trusted satellites' system clock.
2021-10-09T11:18:24.925-0400	INFO	Node 161sXNjyPE65jjC5uNXTy1jvwqprwTDzXKHc4UVdBExeUUyGiW started
2021-10-09T11:18:24.926-0400	INFO	Public server started on [::]:28967
2021-10-09T11:18:24.926-0400	INFO	Private server started on 127.0.0.1:7778
2021-10-09T11:18:24.926-0400	INFO	trust	Scheduling next refresh	{"after": "8h29m21.496028277s"}
2021-10-09T11:18:24.926-0400	INFO	bandwidth	Performing bandwidth usage rollups
2021-10-09T11:18:25.026-0400	INFO	piecestore	upload started	{"Piece ID": "5LM4EFHYKQXFVSOY67W2DMCKMSH6YTP75JOKIRALXQOYZ2NLB7MA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.027-0400	INFO	piecestore	upload started	{"Piece ID": "UY4JLNSBE7MKWCPWN3W5VAAV7USXHM6XEBJV3UEOUYMF2TK3XJ3Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.027-0400	INFO	piecestore	upload started	{"Piece ID": "QSRRXL5SLHCEWDP7PTZ5AXYCYIBYGPGBDCNZF2DYKHZUNGD2GLMQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.027-0400	INFO	piecestore	upload started	{"Piece ID": "XMZHB4IBKXGKYBXYYQ27NZSNED2LDH2TA4ZFRMKKBBVDVHSPKXFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.026-0400	INFO	piecestore	upload started	{"Piece ID": "BKAQ2ABXEPVR5TCXYQ4BCMDRIOFSSL365ECJ5I4HWGI5AM22SO7Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.028-0400	INFO	piecestore	upload started	{"Piece ID": "OGOG622REJQD6MF75NXF66G7P63ML6E3A34IGDAM254AQIS6LK3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.029-0400	INFO	piecestore	upload started	{"Piece ID": "QZIQM2FKQA7O552UZ7V6SAMN4GOUZ63KQJBLF7OZUNWG4HXQZNJA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.036-0400	INFO	piecestore	upload started	{"Piece ID": "UXQFAOLOKBZXQGU6F7AQ6Q2E4WM2ZIU3GYQFFKB3JLHAZUIMI4HA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.067-0400	INFO	piecestore	download started	{"Piece ID": "YF6BDCQG35MMPCZZY7T4VCKPLSAEHCFUBYNZTSD3SY7TRUVMCDRQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-10-09T11:18:25.074-0400	INFO	piecestore	upload started	{"Piece ID": "T2SHUVOE3UPXK7NFW5JVJI35GHA4METEXNJ7JSIDTGATUQSMS45A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.075-0400	INFO	piecestore	download started	{"Piece ID": "RV2QLBVUY7D6IHRRLJDAQBAILLJLK6BHSHPN6IV3QF2UA464VA6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR"}
2021-10-09T11:18:25.095-0400	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "G2IOC3TEFGJVWXPOLMNP3ENJSNHVCGUS2XBB72PUFT5IWLKR7Y7A", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-10-09T11:18:25.102-0400	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "YHURXVS34GKO65CGGJ6E4ROUMXSCVKQSMJK3E2XLRJ4WU4S3KIFQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

Usually, if a database is malformed, logs should tell you that.

Here are some info on how to fix malformed databases:

Since this got merged, lots of information to filter through, what’s the answer @Alexey ?

I mark critical response because I can’t babysit Storj all day restarting the service

and

The last one is marked as a solution

While that was the fix for the specific unable to delete errors, it seems your node is having underlying problems that need fixing. There are lots of The file or directory is corrupted and unreadable. errors in your logs indicating you have file system corruption. Please run chkdsk on the drive in question to find and fix file system issues.

1 Like