Node Suspended and Unsuspended within a day (No Serial Errors)

Node Operators troubleshooting
1

Node has been running for years with no issues, and I haven’t seen anything in the logs that would indicate this is a serial error like some of the other posts. Here are the recent logs

Some errors but only this type

2022-04-16T17:26:06.353Z ERROR collector unable to delete piece {Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: LTS5RYCXFPYAADUQJ4CPXSUDK5PCOKFAHVLKNPLTRHQHFCSO2SGQ, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2022-04-16T17:26:06.361Z ERROR collector unable to delete piece {Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: TXPRW2HLJQOMDGNI4Z6HWR2T3V34YRTF4KQ6SRLUNHETAXOBRXEQ, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2022-04-16T17:26:06.374Z ERROR collector unable to delete piece {Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: RFDTQWT7PCTE2G5TRSNY45XACVFBVL6IQJN3ZFH2OVLZ7QLHP4VQ, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2022-04-16T17:26:06.419Z ERROR collector unable to delete piece {Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: ABJJQDPDGVGIWWPYNR26KL7UUEJW3II5B3C52RDDPEAFD3MCWXFQ, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2022-04-16T17:26:06.429Z ERROR collector unable to delete piece {Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: 536CKDFYQDAH33DPU3EQD6IPXJHDUXHMYX4RDQJJOEKU53YZFOMA, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2022-04-16T17:26:06.444Z ERROR collector unable to delete piece {Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: ACQDK4PLUEFFQZRU3XPUX7YNTPLST7PCFKKORRNNPYSMEYXO5YNQ, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}

More recent logs

2022-04-21T15:52:26.612Z INFO piecestore uploaded {Piece ID: QZAD7PCDWNMO2NMVGU7MEHSEL3OG4FUYKD5GAZC7GZT7OE7J4BCA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 163840}
2022-04-21T15:52:26.719Z INFO piecestore upload started {Piece ID: EH4HC2GZOKC37FX2APQDE4PLCUROUPNTXCXQNQJ7RLV55R2KEGEQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 7829892427246}
2022-04-21T15:52:26.768Z INFO piecestore uploaded {Piece ID: EH4HC2GZOKC37FX2APQDE4PLCUROUPNTXCXQNQJ7RLV55R2KEGEQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 2304}
2022-04-21T15:52:26.929Z INFO piecestore uploaded {Piece ID: LEQIIDCWXYRVKRFKSDGGSBM2NOJ5ZNQ3VBR2ZTFWM5NDWS5EOI3Q, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 238080}
2022-04-21T15:52:27.277Z INFO piecestore upload started {Piece ID: BB3KAPMF365NKCCZ52ONHSFW3WZQOSCY3ZRFDB62IWWFBAT5VFZA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 7829892185838}
2022-04-21T15:52:27.382Z INFO piecestore uploaded {Piece ID: BB3KAPMF365NKCCZ52ONHSFW3WZQOSCY3ZRFDB62IWWFBAT5VFZA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 4352}
2022-04-21T15:52:27.448Z INFO piecestore upload started {Piece ID: 4Y46V5Y5DZSLFADST2NY7VWSINPKDCYTWVR5NNLCGAZGJI6KUH6Q, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 7829892180974}
2022-04-21T15:52:27.489Z INFO piecestore uploaded {Piece ID: 4Y46V5Y5DZSLFADST2NY7VWSINPKDCYTWVR5NNLCGAZGJI6KUH6Q, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 3328}
2022-04-21T15:52:27.924Z INFO piecestore upload started {Piece ID: VKDFI2HBJPWGCJQSDK32C7X5JHAEOS67YCF3XJWDZ3XIGFYUSOFQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 7829892177134}
2022-04-21T15:52:27.931Z INFO piecestore uploaded {Piece ID: VKDFI2HBJPWGCJQSDK32C7X5JHAEOS67YCF3XJWDZ3XIGFYUSOFQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 3328}
2022-04-21T15:52:28.867Z INFO piecedeleter delete piece sent to trash {Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Piece ID: VCJTGA6M7QD2NZRYW4YRC24IC72FZ2P5K5BUC54JRK5VZI3O7MEQ}
2022-04-21T15:52:30.136Z INFO piecestore upload started {Piece ID: QNWCZYM5DW4HM3EB4QKLGORJWOCP3NL6NVPMCUNNE6URFZH2FE4A, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 7829892173294}
2022-04-21T15:52:31.141Z INFO piecestore upload started {Piece ID: 7SBNCD4ERIY4J7NVF3DVUVCSBZYVAXHTQ4VAKTIIMUHRBKCT3M7Q, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 7829892173294}
2022-04-21T15:52:31.146Z INFO piecestore uploaded {Piece ID: 7SBNCD4ERIY4J7NVF3DVUVCSBZYVAXHTQ4VAKTIIMUHRBKCT3M7Q, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 3584}
2022-04-21T15:52:31.578Z INFO piecestore uploaded {Piece ID: QNWCZYM5DW4HM3EB4QKLGORJWOCP3NL6NVPMCUNNE6URFZH2FE4A, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 181504}
2022-04-21T15:52:32.236Z INFO piecestore download started {Piece ID: ZMBUHYGDKD6ZFR5IB2KYADBB54SVGIOW5PSPTSTG74ZELEMWVPDQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET}
2022-04-21T15:52:32.282Z INFO piecestore downloaded {Piece ID: ZMBUHYGDKD6ZFR5IB2KYADBB54SVGIOW5PSPTSTG74ZELEMWVPDQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET}
2022-04-21T15:52:32.342Z INFO piecestore upload started {Piece ID: 75F4ASCKMMPZKQIEKUGGKFNH7BOF56NLTDYEDBSBXJ2KWJUMJOPQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 7829891987182}
2022-04-21T15:52:32.408Z INFO piecestore uploaded {Piece ID: 75F4ASCKMMPZKQIEKUGGKFNH7BOF56NLTDYEDBSBXJ2KWJUMJOPQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 3584}
2022-04-21T15:52:32.895Z INFO piecestore download started {Piece ID: S5VA3YNCTCIGVLTQ3OHPSJNNCLGBEGL4VN7CZQEJHEQHYIIMMZYQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET}
2022-04-21T15:52:33.143Z INFO piecestore upload started {Piece ID: SKPQYVSO4XSNBBB5NDKTUUIBLQWN5GZ3MC6IAMQWZ5TSKIEWU6FQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 7829891983086}
2022-04-21T15:52:33.232Z INFO piecestore downloaded {Piece ID: S5VA3YNCTCIGVLTQ3OHPSJNNCLGBEGL4VN7CZQEJHEQHYIIMMZYQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET}
2022-04-21T15:52:33.551Z INFO piecestore upload started {Piece ID: 6Y7NH44KRIMC6NEXEI2XSMZVRTDUQR4RANX2UR2OCNYHNVZSJTKQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 7829891983086}
2022-04-21T15:52:33.558Z INFO piecestore uploaded {Piece ID: 6Y7NH44KRIMC6NEXEI2XSMZVRTDUQR4RANX2UR2OCNYHNVZSJTKQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 5888}
2022-04-21T15:52:33.588Z INFO piecestore uploaded {Piece ID: SKPQYVSO4XSNBBB5NDKTUUIBLQWN5GZ3MC6IAMQWZ5TSKIEWU6FQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 181504}
2022-04-21T15:52:34.085Z INFO piecestore download started {Piece ID: AIGYITULUF5PTW2CC2XNH6ZF2Q7IWGOLXAQERM62NEZ5HRR7QIZQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET_REPAIR}
2022-04-21T15:52:34.145Z INFO piecestore upload started {Piece ID: TQVWBTY7KO4F5ID5QJ4DEAPNOJPFIML6T3OZUEWOZWJ4NBANL7WA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 7829891794670}
2022-04-21T15:52:34.150Z INFO piecestore uploaded {Piece ID: TQVWBTY7KO4F5ID5QJ4DEAPNOJPFIML6T3OZUEWOZWJ4NBANL7WA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 768}
2022-04-21T15:52:34.273Z INFO piecestore downloaded {Piece ID: AIGYITULUF5PTW2CC2XNH6ZF2Q7IWGOLXAQERM62NEZ5HRR7QIZQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET_REPAIR}
2022-04-21T15:52:34.429Z INFO piecestore download started {Piece ID: I2FJ3X2NNWDXKGSI3XT6YOBXGQ2LXHQSXJ4XDSFGILJFGHWZWROA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET}
2022-04-21T15:52:34.787Z INFO piecestore downloaded {Piece ID: I2FJ3X2NNWDXKGSI3XT6YOBXGQ2LXHQSXJ4XDSFGILJFGHWZWROA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET}
2022-04-21T15:52:34.804Z INFO piecestore download started {Piece ID: 5YGFBYLAHI63U4ACQXGS3PMUKC27EYJLPG7CWPKQNSGITK2OSMKQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET}
2022-04-21T15:52:35.301Z INFO piecestore download started {Piece ID: URD4VEKQE255NPW6JEAUQ4OL5PRECUMMJYTE4G67NSAXBWA3XVBQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET_REPAIR}
2022-04-21T15:52:36.056Z INFO piecestore upload started {Piece ID: EGDKI5JSY5PT3D3LZ7OC6IYBU223QLN56C6RP6QN42CLXBRUDDDQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 7829891793390}
2022-04-21T15:52:36.196Z INFO piecestore uploaded {Piece ID: EGDKI5JSY5PT3D3LZ7OC6IYBU223QLN56C6RP6QN42CLXBRUDDDQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 9984}
2022-04-21T15:52:36.744Z INFO piecestore downloaded {Piece ID: 5YGFBYLAHI63U4ACQXGS3PMUKC27EYJLPG7CWPKQNSGITK2OSMKQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET}
2022-04-21T15:52:36.745Z INFO piecestore uploaded {Piece ID: NBSJ6QQHANKLNIKLDG4WBW5O3NJV5NSDKIH564ASB6AXLG56QAXQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Size: 894976}
2022-04-21T15:52:36.920Z INFO piecestore upload started {Piece ID: AZS6A24AMW527GNLQNUSZSGQS6NQBCGOLOKZEJPE42F36TMRCNMQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 7829890887406}
2022-04-21T15:52:36.966Z INFO piecestore upload started {Piece ID: XRT2U7HT4SMUCXDKKUMOBF2VCFFZ7RQ6R3DE7G4RHLHOR44CYZTQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 7829890887406}
2022-04-21T15:52:36.976Z INFO piecestore download started {Piece ID: RWMM3E3XBOSA7BLEXLB4BRB3SLUZBXHL4Z7YXBNQHLLX7TBE4CCA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET}
2022-04-21T15:52:36.994Z INFO piecestore uploaded {Piece ID: XRT2U7HT4SMUCXDKKUMOBF2VCFFZ7RQ6R3DE7G4RHLHOR44CYZTQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 2816}
2022-04-21T15:52:37.084Z INFO piecestore uploaded {Piece ID: AZS6A24AMW527GNLQNUSZSGQS6NQBCGOLOKZEJPE42F36TMRCNMQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 36864}
2022-04-21T15:52:37.791Z INFO piecestore upload started {Piece ID: EDYIPTLWFL6VSP6LKQJH2BVAHALDJLBK2PL3PS7MHPJOMANIC2MQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 7829890846702}
2022-04-21T15:52:38.304Z INFO piecestore downloaded {Piece ID: URD4VEKQE255NPW6JEAUQ4OL5PRECUMMJYTE4G67NSAXBWA3XVBQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET_REPAIR}
2022-04-21T15:52:38.781Z INFO piecestore download started {Piece ID: 3RTS2SNG4VJXA4EOY3PUDPQNDAIH336V2TFQIL4775VUVS5U66WQ, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: GET_AUDIT}
2022-04-21T15:52:38.828Z INFO piecestore upload started {Piece ID: U4MYKZXAL7V654D5C5GDYNN7OLSMK4X2FJGDFKVDHZMAXKVH4WWA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 7829890846702}
2022-04-21T15:52:38.834Z INFO piecestore uploaded {Piece ID: U4MYKZXAL7V654D5C5GDYNN7OLSMK4X2FJGDFKVDHZMAXKVH4WWA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 3840}
2022-04-21T15:52:38.942Z INFO piecestore downloaded {Piece ID: 3RTS2SNG4VJXA4EOY3PUDPQNDAIH336V2TFQIL4775VUVS5U66WQ, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: GET_AUDIT}
2022-04-21T15:52:39.537Z INFO piecestore upload started {Piece ID: 5TOT3LTAZZ2YFXDV4RMB2W6NAFJPEEWIZXWQ5LDVOMVVOJEYRW4Q, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Available Space: 7829890842350}

I haven’t made any changes that would cause this, and my suspension and uptime are near 100% on all other nodes. Is there anything I can do other than wait for the suspension score to go back up? My suspension on eu north is 14%

2

Looks like the same issue here

3

I don’t think so, I don’t have any of those serial already exists error. One strange thing is that the ap1 satellite went from 50% suspension to 100% in like an hour and now only one satellite is in suspension.

4

I also have these random moments where my nodes get instantly suspended and also instantly unsuspended. I have just accepted this as the new normal

5

Hmm very strange. Now the second satellite is unsuspended, went from 15% to 100% basically instantly. Sounds like there is some type of bug with the suspension calculation.

6

Nodes with a lot of data can get suspended and unsuspended in a matter of hours.
It seems to be the “normal behavior” because they do receive lots of audit checks.

Usually in my case, this happens when my disks cannot keep up (most are SMR disks… booo, I know): at that moment satellites don’t get replies to their numerous audit checks because disks are crawling, so nodes get suspended which gives more time to them for recovering, and they can start responding to audits again, and get reinstated rapidly…

Nodes’ scores get refreshed regularly, but not in real time. So I think your node didn’t actually jump from 15% to 100% instantly, it just happened to gradually recover from 15% to 100% from one satellite’s check-in to the next one (I believe they are several hours appart).

1 Like
7

Same here. Had it with looking at the dashboard of a node and saw the suspension message, 5 minutes later I refreshed and the message was gone. The sad thing is, that it is not useful this way.

8

It’s still worth to check for errors in your logs. Maybe you can improve something to do not become suspended due to timeouts or maybe temporary database/disk errors.

1 Like
9

@Alexey isnt it clear at this point that the issue isnt on the SNO end? there are several of us here that went from 100% to something much lower, then back to 100% I cant speak for the other operators, but I have several terrabytes of data stored on my node, and im not using any notable bandwidth between the node being suspended and it coming back online. This would indicate that my node did NOTHING to repair the data, and not enough time has gone by for that to greatly impact my score…

With respect, something is wrong with the satellite if this is happening to multiple people at the same time.

10

Another data point…
Why would i only have a crappy suspension score on AP1? I cannot control where within the container data gets stored, so clearly all data is intermixed on drives… why is AP1 saying that my node isnt operating correctly, when the other 5 Satellites say i am a pretty OK dude?

image
image833×284 21.7 KB

11

Same here. AP1 is showing me as suspended on one of my 5 nodes. Did receive 5 emails this morning for different suspensions on different nodes, but currently only suspended on one node. I see no errors in my logs specifically for this particular satellite either (121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6).

Ended up restarting the docker instance, and it jumped from 54% to 60%, and now I’m not suspended…Just waited another 15 minutes, now at 74%. What’s going on?

EDIT: Fun note - Restarting the docker and waiting a bit ended up bumping up my suspension score up…3 times. I’m not sitting at 100% :man_shrugging: You can see where I restarted my docker:

image
image773×487 17.7 KB

12

Because each satellite audits your node independently and they audits only data of their customers not the data of customers of another satellite.

Your logs should contains errors related to AP1 (use its SatelliteID 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 for search errors specifically related to this satellite). You need to search for GET_AUDIT or GET_REPAIR, both can affect the suspension score if there is any error during download a piece.
Since we increased a repair threshold, your node get more GET_REPAIR requests. And if it’s not able to process them in time - it will throw some errors, as result your suspension score will be affected.
When the piece is successfully downloaded, your suspension score is updated again. With high frequency it can jump down and up pretty fast, but updated on your node with a delay.