ERROR collector unable to delete piece

[jammerdan]

@Alexey

Unlikely

But this is what it is. I have checked on a second node and see the fully same issue: Piece not found for deletion, no history for the piece and on the 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs satellite only:

2021-09-12T01:54:31.291Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-09-12T02:54:30.900Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-09-12T03:54:33.001Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T04:54:32.094Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T05:54:35.624Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T06:54:30.527Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T07:54:34.709Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T08:54:31.471Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T09:54:30.268Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T10:54:30.448Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T11:54:31.718Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T12:54:42.283Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T13:54:31.886Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T14:54:31.008Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T15:54:32.503Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T16:54:30.062Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T17:54:30.332Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T18:54:33.861Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T19:54:42.558Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T20:54:41.174Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T21:54:30.031Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T22:54:32.415Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-12T23:54:33.619Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T00:54:34.540Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T01:54:33.878Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T02:54:39.011Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T03:54:48.723Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T04:54:32.028Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T05:54:30.803Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T06:54:30.089Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T07:54:29.646Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T08:54:30.608Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T09:54:30.241Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T10:54:31.271Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T11:54:32.768Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T12:54:29.694Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T13:54:32.187Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T14:54:31.265Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T15:54:32.339Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T16:54:29.653Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T17:54:34.853Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T18:54:32.025Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T19:54:30.056Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T20:54:37.118Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T21:54:34.489Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T22:54:30.083Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-13T23:54:30.158Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T00:54:30.190Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T01:54:29.761Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T02:54:30.885Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T03:54:43.525Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T04:54:34.061Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T05:54:29.932Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T06:54:31.411Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T07:54:31.937Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T08:54:30.520Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T09:54:31.471Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T10:54:30.872Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T11:54:39.847Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T12:54:31.585Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T13:54:36.579Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T14:54:39.164Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T15:54:31.407Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist "}
2021-09-14T16:54:30.231Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T17:54:33.449Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T18:54:32.542Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T19:54:30.766Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T20:54:29.895Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T21:54:32.854Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T22:54:31.948Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-14T23:54:29.997Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T00:54:30.704Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T01:54:30.469Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T02:54:30.412Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T03:54:35.896Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T04:54:31.287Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T05:54:30.399Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T05:57:29.528Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T06:57:33.369Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T07:57:27.653Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T08:57:26.486Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T09:57:33.226Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T10:57:27.636Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T11:57:26.901Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T12:57:34.250Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T13:57:26.959Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T14:57:26.798Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T15:57:27.008Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T16:57:31.347Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T17:57:28.230Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T18:57:27.078Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T19:57:27.055Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T20:57:28.856Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T21:57:26.677Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T22:57:30.757Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-15T23:57:28.039Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T00:57:26.525Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T01:57:26.954Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T02:57:26.843Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T03:57:26.552Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T04:57:31.311Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T05:57:27.380Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }
2021-09-16T06:57:26.844Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "7JZYN5GBS4W3I7MOPBA7CP5PIKWORKVKSPP2H63MJFIF25WOTUCQ" }

The log shows entries as far back as they are available.

[Alexey]

I created an issue for this case: [storagenode] The garbage collector is trying to delete a piece over and over again · Issue #4192 · storj/storj · GitHub
I’m still not convinced that the piece was not lost by the node.
But so far the statistics are in favor of the presence of the bug somewhere.

Could you please find records where an uplink was actually tried to upload a piece (or it has been uploaded) to your node?

[jammerdan]

@Alexey

Could you please find records where an uplink was actually tried to upload a piece (or it has been uploaded) to your node?

Yes, but I need a script doing that. I can’t go through every line of the log manually.
Maybe someone has such a script. I think it should work that way that it finds every instance of “unable to delete piece” gets the piece id from that line and then search for the corresponding upload or at least something other than “unable to delete piece”.

[Alexey]

You can use the script from the issue

[jammerdan]

Unfortunately logs don’t go back long enough to see when the pieces were uploaded or what else has happened before deletion attempts.
One node has several pieces with delete issues (7) others have less (1).

[waistcoat]

I seem to be having a similar issue, however I do have my logs.

All seem to follow a pattern of a delete expired followed by several unable to delete piece errors.

I’ve knocked up a quick script to look at what is causing ‘unable to delete piece’ errors in today’s log, then to look back and report all mentions of those pieces until the first ‘unable to delete piece’ errors start.
This node is a raspberry pi docker storage node which has had USB FS issues, so I may have possibly lost data on hangups, there was an extensive fsck repair after I replaced the USB adapter with a more reliable one, however storage nodes shouldn’t really care if a piece cannot be deleted. If it’s gone there should be no work to do and no error - maybe a warning and then regard as success.
I’ve had 30 ‘unable to delete piece’ errors today already since the logs were rotated 10 hours ago.

== LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ ==
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:20:34.618Z	INFO	piecestore	upload started	{"Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 344865003008}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:20:53.870Z	INFO	piecestore	uploaded	{"Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Size": 2319360}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:24:18.513Z	INFO	piecestore	download started	{"Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:24:28.203Z	INFO	piecestore	downloaded	{"Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:23:25.824Z	INFO	collector	delete expired	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ"}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:39:23.102Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "LQAJGDCIR25R74WQJRONHMEYYHUOYBN3PKP2E4CQ4FVMDUXPAYNQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

== P4UDM2RZOJKNXSTQ5QV35WTZQHDQ766QN3MQJROUTIG353VU5LFA ==
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:10:09.486Z	INFO	piecestore	upload started	{"Piece ID": "P4UDM2RZOJKNXSTQ5QV35WTZQHDQ766QN3MQJROUTIG353VU5LFA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 344889131008}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:10:09.531Z	INFO	piecestore	uploaded	{"Piece ID": "P4UDM2RZOJKNXSTQ5QV35WTZQHDQ766QN3MQJROUTIG353VU5LFA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Size": 512}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:23:25.454Z	INFO	collector	delete expired	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "P4UDM2RZOJKNXSTQ5QV35WTZQHDQ766QN3MQJROUTIG353VU5LFA"}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:39:22.985Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "P4UDM2RZOJKNXSTQ5QV35WTZQHDQ766QN3MQJROUTIG353VU5LFA", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

== ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A ==
/mnt/log/storj//node.log-20210904.gz:2021-09-03T13:58:06.489Z	INFO	piecestore	upload started	{"Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 344906312192}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T13:58:06.557Z	INFO	piecestore	uploaded	{"Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Size": 23552}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:03:50.449Z	INFO	piecestore	download started	{"Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
/mnt/log/storj//node.log-20210904.gz:2021-09-03T14:03:51.956Z	INFO	piecestore	downloaded	{"Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:23:25.250Z	INFO	collector	delete expired	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A"}
/mnt/log/storj//node.log-20210919.gz:2021-09-18T14:39:22.670Z	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "ZC5NYYKNWFLWZY34UMUKVUKIRLGJ3GOE25L7S2R7HEBQQV4GJD7A", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

[jammerdan]

@waistcoat

I’ve knocked up a quick script to look at what is causing ‘unable to delete piece’ errors in today’s log, then to look back and report all mentions of those pieces until the first ‘unable to delete piece’ errors start.

Can you provide that script?

[waistcoat]

script is very highly specific to my setup but here you go:

$ cat look-errors.sh 
#!/bin/sh
LOGDIR=/mnt/log/storj
if [ "$1" ]; then
	LOG=$1
else
	LOG=$LOGDIR/node.log
fi

echo "looking in $LOG..."

FRAGS="`grep 'ERROR' $LOG|grep 'unable to '|sed -e 's/.*"Piece ID": "//' -e 's/".*//'|sort -u`"

for FRAG in $FRAGS; do
	echo "== $FRAG =="
	zgrep "$FRAG" $LOGDIR//node.log-*.gz | awk '1;/unable to/{exit}'
	echo ""
done

[jammerdan]

Thanks. I’ll see if I can make it work for me setup.

[7tigers]

Storj keeps stopping and is getting hung up on being able to read/write/delete some files

That would normally understandable but there’s not a clear problem as the iSCSI slice has plenty of space with 1.5TB free and the permissions are open read/write/all.

Is there a scrub command or something to run to verify the integrity of the whole StorjDB and flag bad files for manual deletion/overwrite/repair?

Log pull:

2021-10-09T10:03:45.138-0400	ERROR	piecestore:cache	error getting current used space: 	{"error": "CreateFile S:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\nf/2delxkcnwfrwhkjcnhu4qeiixuds5eemxx3bs3pkj3kij4suxq.sj1: The file or directory is corrupted and unreadable.; CreateFile S:\\Storj\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\xi/eejste6pi6qgzjq5sewdxqiizx2pa4e3sabndypulqqhnfjesq.sj1: The file or directory is corrupted and unreadable.", "errorVerbose": "group:\n--- CreateFile S:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\nf/2delxkcnwfrwhkjcnhu4qeiixuds5eemxx3bs3pkj3kij4suxq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- CreateFile S:\\Storj\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\xi/eejste6pi6qgzjq5sewdxqiizx2pa4e3sabndypulqqhnfjesq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-10-09T10:03:45.138-0400	ERROR	services	unexpected shutdown of a runner	{"name": "piecestore:cache", "error": "CreateFile S:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\nf/2delxkcnwfrwhkjcnhu4qeiixuds5eemxx3bs3pkj3kij4suxq.sj1: The file or directory is corrupted and unreadable.; CreateFile S:\\Storj\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\xi/eejste6pi6qgzjq5sewdxqiizx2pa4e3sabndypulqqhnfjesq.sj1: The file or directory is corrupted and unreadable.", "errorVerbose": "group:\n--- CreateFile S:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\nf/2delxkcnwfrwhkjcnhu4qeiixuds5eemxx3bs3pkj3kij4suxq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- CreateFile S:\\Storj\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\xi/eejste6pi6qgzjq5sewdxqiizx2pa4e3sabndypulqqhnfjesq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-10-09T10:03:45.139-0400	INFO	piecestore	upload canceled	{"Piece ID": "PBBFK3B6T4Y4EDF2IIHIH4SDR25TP4U3IGU36TBPHX3MRQY5B5EQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-10-09T10:03:45.139-0400	INFO	piecestore	downloaded	{"Piece ID": "USZBEQH2SOQXNTXTO64CLZ5UHCDPCBBOLQUYBSZVAPYYXTFIDSYA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-10-09T10:03:45.139-0400	INFO	piecestore	downloaded	{"Piece ID": "6UMXYHBID4UUOXON5ZASCUOVKGATEWXNC7RMJXWD6P5G3RJ6J7IQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-10-09T10:03:45.139-0400	INFO	piecestore	downloaded	{"Piece ID": "USZBEQH2SOQXNTXTO64CLZ5UHCDPCBBOLQUYBSZVAPYYXTFIDSYA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-10-09T10:03:45.140-0400	INFO	piecestore	upload canceled	{"Piece ID": "D624YVXN7CRGMSMGLAP23ZAH4M5V4AXJR2F2XVOQICNZFZB7TXDA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 2105344}
2021-10-09T10:03:45.152-0400	FATAL	Unrecoverable error	{"error": "CreateFile S:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\nf/2delxkcnwfrwhkjcnhu4qeiixuds5eemxx3bs3pkj3kij4suxq.sj1: The file or directory is corrupted and unreadable.; CreateFile S:\\Storj\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\xi/eejste6pi6qgzjq5sewdxqiizx2pa4e3sabndypulqqhnfjesq.sj1: The file or directory is corrupted and unreadable.", "errorVerbose": "group:\n--- CreateFile S:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\nf/2delxkcnwfrwhkjcnhu4qeiixuds5eemxx3bs3pkj3kij4suxq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- CreateFile S:\\Storj\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\xi/eejste6pi6qgzjq5sewdxqiizx2pa4e3sabndypulqqhnfjesq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-10-09T11:18:23.531-0400	INFO	Configuration loaded	{"Location": "C:\\Program Files\\Storj\\Storage Node\\config.yaml"}
2021-10-09T11:18:23.543-0400	INFO	Operator email	{"Address": "chattphotos@gmail.com"}
2021-10-09T11:18:23.543-0400	INFO	Operator wallet	{"Address": "0x5ad4a2f04F50b6Dc48F7B79c1bc22A6927575E4a"}
2021-10-09T11:18:24.158-0400	INFO	Telemetry enabled	{"instance ID": "161sXNjyPE65jjC5uNXTy1jvwqprwTDzXKHc4UVdBExeUUyGiW"}
2021-10-09T11:18:24.191-0400	INFO	db.migration	Database Version	{"version": 53}
2021-10-09T11:18:24.371-0400	INFO	preflight:localtime	start checking local system clock with trusted satellites' system clock.
2021-10-09T11:18:24.924-0400	INFO	preflight:localtime	local system clock is in sync with trusted satellites' system clock.
2021-10-09T11:18:24.925-0400	INFO	Node 161sXNjyPE65jjC5uNXTy1jvwqprwTDzXKHc4UVdBExeUUyGiW started
2021-10-09T11:18:24.926-0400	INFO	Public server started on [::]:28967
2021-10-09T11:18:24.926-0400	INFO	Private server started on 127.0.0.1:7778
2021-10-09T11:18:24.926-0400	INFO	trust	Scheduling next refresh	{"after": "8h29m21.496028277s"}
2021-10-09T11:18:24.926-0400	INFO	bandwidth	Performing bandwidth usage rollups
2021-10-09T11:18:25.026-0400	INFO	piecestore	upload started	{"Piece ID": "5LM4EFHYKQXFVSOY67W2DMCKMSH6YTP75JOKIRALXQOYZ2NLB7MA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.027-0400	INFO	piecestore	upload started	{"Piece ID": "UY4JLNSBE7MKWCPWN3W5VAAV7USXHM6XEBJV3UEOUYMF2TK3XJ3Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.027-0400	INFO	piecestore	upload started	{"Piece ID": "QSRRXL5SLHCEWDP7PTZ5AXYCYIBYGPGBDCNZF2DYKHZUNGD2GLMQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.027-0400	INFO	piecestore	upload started	{"Piece ID": "XMZHB4IBKXGKYBXYYQ27NZSNED2LDH2TA4ZFRMKKBBVDVHSPKXFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.026-0400	INFO	piecestore	upload started	{"Piece ID": "BKAQ2ABXEPVR5TCXYQ4BCMDRIOFSSL365ECJ5I4HWGI5AM22SO7Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.028-0400	INFO	piecestore	upload started	{"Piece ID": "OGOG622REJQD6MF75NXF66G7P63ML6E3A34IGDAM254AQIS6LK3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.029-0400	INFO	piecestore	upload started	{"Piece ID": "QZIQM2FKQA7O552UZ7V6SAMN4GOUZ63KQJBLF7OZUNWG4HXQZNJA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.036-0400	INFO	piecestore	upload started	{"Piece ID": "UXQFAOLOKBZXQGU6F7AQ6Q2E4WM2ZIU3GYQFFKB3JLHAZUIMI4HA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.067-0400	INFO	piecestore	download started	{"Piece ID": "YF6BDCQG35MMPCZZY7T4VCKPLSAEHCFUBYNZTSD3SY7TRUVMCDRQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-10-09T11:18:25.074-0400	INFO	piecestore	upload started	{"Piece ID": "T2SHUVOE3UPXK7NFW5JVJI35GHA4METEXNJ7JSIDTGATUQSMS45A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1197958500992}
2021-10-09T11:18:25.075-0400	INFO	piecestore	download started	{"Piece ID": "RV2QLBVUY7D6IHRRLJDAQBAILLJLK6BHSHPN6IV3QF2UA464VA6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR"}
2021-10-09T11:18:25.095-0400	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "G2IOC3TEFGJVWXPOLMNP3ENJSNHVCGUS2XBB72PUFT5IWLKR7Y7A", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-10-09T11:18:25.102-0400	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "YHURXVS34GKO65CGGJ6E4ROUMXSCVKQSMJK3E2XLRJ4WU4S3KIFQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

[Pac]

Usually, if a database is malformed, logs should tell you that.

Here are some info on how to fix malformed databases:

More help about database disk image is malformed

You could try to fix it by following these instructions: If it doesn’t work, a node can still be recovered even if some databases were to be lost: here is how they can be recreated: In last resort if everything else fails, deleting all database files should work, but you would lose all statistics for current month, and possibly some of your earnings for current month also, but at least the node would be “saved”.

[7tigers]

Since this got merged, lots of information to filter through, what’s the answer @Alexey ?

I mark critical response because I can’t babysit Storj all day restarting the service

[Alexey]

ERROR collector unable to delete piece

Each satellite has their own folder in the blobs folder in your storage location. For the satellite this piece is on that’s v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa. The subfolders you see inside there represent the first 2 characters of the piece id. So in your case the folder should be fr. The filename inside that folder is the rest of the piece id with a .sj1 extension. So jgxef52maxhcihttdsbjsez6vpmdqqw6l7zxp2jykasd37pusq.sj1. Path and filenames are in lower case, which is a litt…

and

ERROR collector unable to delete piece

Edit: This list may now be outdated, latest list is maintained here: Original message for reference In the logs you posted it seems to have all been for the same satellites. If you have the same issue on other satellites, here’s a reference for satellites and corresponding ID’s and blobs folders. Satellite Name: satellite.stefan-benten.de Satellite ID: 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW Satellite ID hex: 004AE89E970E703DF42BA4AB1416A3B30B7E1D8E14AA0E558F7EE26800000000 Blobs …

The last one is marked as a solution

[BrightSilence]

While that was the fix for the specific unable to delete errors, it seems your node is having underlying problems that need fixing. There are lots of The file or directory is corrupted and unreadable. errors in your logs indicating you have file system corruption. Please run chkdsk on the drive in question to find and fix file system issues.

[geeksheikh]

I’m not understanding the solution. My service keeps dying after start up due to an error like this. Could you please help me figure out how to get it running again?

2022-04-09T16:06:23.611Z ERROR piecestore download failed {"Piece ID": "MVXCEBR342F4XCLEHGGX3CEKSYTIP4YSUCYGRWODLJ5GKL7VKPIQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "error": "trust: rpc: tcp connector failed: rpc: dial tcp 34.92.229.119:7777: operation was canceled; quic connector failed: quic: context canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: dial tcp 34.92.229.119:7777: operation was canceled; quic connector failed: quic: context canceled\n\tstorj.io/common/rpc.Dialer.dialEncryptedConn:238\n\tstorj.io/common/rpc.Dialer.DialNodeURL.func1:114\n\tstorj.io/common/rpc/rpcpool.(*Pool).get:105\n\tstorj.io/common/rpc/rpcpool.(*Pool).Get:128\n\tstorj.io/common/rpc.Dialer.dialPool:202\n\tstorj.io/common/rpc.Dialer.DialNodeURL:109\n\tstorj.io/storj/storagenode/trust.Dialer.func1:52\n\tstorj.io/storj/storagenode/trust.IdentityResolverFunc.ResolveIdentity:44\n\tstorj.io/storj/storagenode/trust.(*Pool).GetSignee:153\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:134\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:498\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}

2022-04-09 16:06:26,613 INFO waiting for processes, storagenode to die
2022-04-09T16:06:27.172Z ERROR piecestore:cache error getting current used space: {"error": "context canceled; context canceled; context canceled; context canceled; context canceled; context canceled", "errorVerbose": "group:\n--- context canceled\n--- context canceled\n--- context canceled\n--- context canceled\n--- context canceled\n--- context canceled"}

2022-04-09T16:06:27.954Z INFO piecestore downloaded {"Piece ID": "G53WXTF7DXVX2ENM7YGNQFTEB46KTSVMCPOJCYP7NCA23NMH3CVA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-04-09T16:06:28.104Z INFO piecestore downloaded {"Piece ID": "VQUNTYQ3LS5XHJ3MY7PZOLEP3WKKXKJLQ56IMYXPNKCP4X5MBXKQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-04-09T16:06:28.146Z INFO piecestore downloaded {"Piece ID": "UDNU6AWK3RUB7WMRTJ44VREPR6VQ7DNNU7SM62UV2XG3IX2TPOUA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-04-09T16:06:28.161Z ERROR piecestore error sending hash and order limit {"error": "context canceled"}

2022-04-09T16:06:28.161Z INFO piecestore download canceled {"Piece ID": "RWHGCJEU3R2MLXGKG3IVREHYHUIX7HBQBKQVJFARNYPEVSX2OPYQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR"}
2022-04-09T16:06:28.274Z ERROR collector unable to update piece info {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "BPFG3EB5LYM5MGGD64BOGU3X75MOVR54OBKR2ACILAYIX6YSLCZQ", "error": "pieceexpirationdb: context canceled", "errorVerbose": "pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).DeleteFailed:99\n\tstorj.io/storj/storagenode/pieces.(*Store).DeleteFailed:547\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:103\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

[Alexey]

Seems your disk cannot keep up. How is it connected? Is it a network drive? If so - they are not supported and not recommended. The only working storage network protocol is iSCSI.

I would also recommend to check your databases: https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed-

[geeksheikh]

Hmm, it would be strange if it were disk speed as it’s a Raid 5 across 7 disks. As you can see from the speed tests at various block sizes, the disks seem to be performing more than adequately.

The databases also look fine according to the scan any other thoughts on how to resolve this?

UPDATE - It seems like the deletes have expired now and the service isn’t dying anymore. That was a pretty strange bump…any ideas what might have happened?

[Alexey]

The RAID5 is working with the speed of the slowest disk in the array.
When I saying “the speed”, I meant not the speed of transfer, but latency and load.
If the disk is busy, all operations stacked to the queue to be processed. When the operation is waiting too long in the queue - it may become canceled.

[Toyoo]

@Alexey

The RAID5 is working with the speed of the slowest disk in the array.
When I saying “the speed”, I meant not the speed of transfer, but latency and load.

It’s even worse than that for small writes. Whenever a small write happens, the system needs to read the whole stripe, and only then write a modified one.

I have access to an old system with a RAID5 over 6 old drives that is only capable of ~40 random 4k writes per second (a single HDD usually does 250).

[bobbll]

I have the same problem with a node as on the screen, but I don’t know how to find the error. there is no error in the node log, where can I still look?