Now it hit me: 'Your node has been suspended'

Node Operators troubleshooting
64

Yeah, I totally see this. But I don’t have logs so far back. :frowning_face:
But I have just checked node2 again it looks like there are looks back to 19th of February but still:

docker logs storagenode2 2>&1 | grep -E 'GET_REPAIR' | grep 'ERROR' | wc -l
0
65

Yeah, that does look like this may indeed be a case where it isn’t all logged.

66

I run SAS drives. There are several experiments running on the same hardware. I generally leave filesystems alone… especially if I already started a project on the filesystem… unless I have a problem.

Apparently one can also set nouser_xattr in the mount options /etc/fstab for the partition.

I didn’t see that option in the first go round in the documentation. It’s not listed in the general xattr or the fstab man pages. It’s listed in the ext4 section of the mount man page.

The ext4 filesystem is an an advanced level of the ext3 filesystem which incorporates scalability and reliability enhancements for supporting large filesystem.

The options journal_dev, noload, data, commit, orlov, oldalloc, [no]user_xattr [no]acl, bsddf, minixdf, debug, errors, data_err, grpid, bsdgroups, nogrpid sysvgroups, resgid, resuid, sb, quota, noquota, grpquota, usrquota and [no]bh are backwardly compatible with ext3 or ext2.

So something like this in /etc/fstab should work for ext4.

<file system>   <mount point>   <type>         <options>       <dump>  <pass>
UUID=8c...     /opt/mount-point   ext4    nouser_xattr,noatime    0       0
1 Like
67

i had a long period where during start up bandwidth.db locked would happen, and it never caused any permanent harm, but it was usually only a short while, can’t give exact times… but less than an hour and more than a few minutes.

so yeah maybe its possible it could go bad, especially if bandwidth.db is locked for extended periods… but again i have had times where the storage was doing terrible for various reasons, and most likely had bandwidth.db locked in those cases to… for longer periods but not 100% sure

i’ve had a lot of weird issues and stalls, but the storagenode has taken it like a champ, so these days it takes a good deal to make me worried as it seems like the storagenode software is pretty lenient in such cases of intermittent issues.

not saying that is always the case…

1 Like
68

Today I’ve got 2nd node suspended in passed 3 days and there is no obvious solution to fix that?
I’ve checked the log file and there is some warnings and errors that I could not understand:

2022-02-20T13:43:53.410+0400 WARN console:service unable to get Satellite URL {Satellite ID: 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW, error: console: trust: satellite \118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\ is untrusted, errorVerbose: console: trust: satellite \118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\ is untrusted\n\tstorj.io/storj/storagenode/trust.(*Pool).getInfo:238\n\tstorj.io/storj/storagenode/trust.(*Pool).GetNodeURL:177\n\tstorj.io/storj/storagenode/console.(*Service).GetDashboardData:186\n\tstorj.io/storj/storagenode/console/consoleapi.(*StorageNode).StorageNode:45\n\tnet/http.HandlerFunc.ServeHTTP:2047\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2879\n\tnet/http.(*conn).serve:1930}
2022-02-20T14:42:43.980+0400 ERROR collector unable to delete piece {Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: T7CKEKFLZ7LMWM56FPYJT6EINMSLJ4II6EP6ICANUJX63IS5H77A, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2022-02-21T20:22:10.492+0400 ERROR collector unable to delete piece {Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: T7CKEKFLZ7LMWM56FPYJT6EINMSLJ4II6EP6ICANUJX63IS5H77A, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2022-02-21T20:24:55.119+0400 WARN console:service unable to get Satellite URL {Satellite ID: 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW, error: console: trust: satellite \118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\ is untrusted, errorVerbose: console: trust: satellite \118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\ is untrusted\n\tstorj.io/storj/storagenode/trust.(*Pool).getInfo:238\n\tstorj.io/storj/storagenode/trust.(*Pool).GetNodeURL:177\n\tstorj.io/storj/storagenode/console.(*Service).GetDashboardData:186\n\tstorj.io/storj/storagenode/console/consoleapi.(*StorageNode).StorageNode:45\n\tnet/http.HandlerFunc.ServeHTTP:2047\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2879\n\tnet/http.(*conn).serve:1930}
2022-02-21T20:25:02.697+0400 WARN console:service unable to get Satellite URL {Satellite ID: 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW, error: console: trust: satellite \118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\ is untrusted, errorVerbose: console: trust: satellite \118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\ is untrusted\n\tstorj.io/storj/storagenode/trust.(*Pool).getInfo:238\n\tstorj.io/storj/storagenode/trust.(*Pool).GetNodeURL:177\n\tstorj.io/storj/storagenode/console.(*Service).GetDashboardData:186\n\tstorj.io/storj/storagenode/console/consoleapi.(*StorageNode).StorageNode:45\n\tnet/http.HandlerFunc.ServeHTTP:2047\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2879\n\tnet/http.(*conn).serve:1930}
2022-02-21T20:33:03.173+0400 ERROR piecestore download failed {Piece ID: J75DJVN377BUCSKWGCUF6BPBD66C2P43M373GMFPZSOXZBNG22AQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET, error: drpc: remote closed the stream, errorVerbose: drpc: remote closed the stream\n\tstorj.io/drpc/drpcstream.(*Stream).HandlePacket:202\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:211}
2022-02-21T21:00:22.517+0400 ERROR piecestore download failed {Piece ID: CGVIMCROKYU7CX65T6P7C37KZDEBNSONYWTKHMZPBOXKHMV5ECNQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET, error: drpc: remote closed the stream, errorVerbose: drpc: remote closed the stream\n\tstorj.io/drpc/drpcstream.(*Stream).HandlePacket:202\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:211}
2022-02-21T21:22:03.605+0400 ERROR collector unable to delete piece {Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: T7CKEKFLZ7LMWM56FPYJT6EINMSLJ4II6EP6ICANUJX63IS5H77A, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2022-02-21T22:06:01.071+0400 ERROR piecestore download failed {Piece ID: 3GFH2VJBYGJI656NXMWXE4JGCDEEF4XSXHTQ2XSL2ZWUQOSOKRXQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET, error: drpc: remote closed the stream, errorVerbose: drpc: remote closed the stream\n\tstorj.io/drpc/drpcstream.(*Stream).HandlePacket:202\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:211}
2022-02-21T22:22:03.448+0400 ERROR collector unable to delete piece {Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: T7CKEKFLZ7LMWM56FPYJT6EINMSLJ4II6EP6ICANUJX63IS5H77A, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2022-02-21T22:40:01.985+0400 ERROR piecestore download failed {Piece ID: 674XPYOQQWYL7IIKK7YZAKKFGJA66X7YNBTB3O62NDGXMJL7ZBPQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET, error: drpc: remote closed the stream, errorVerbose: drpc: remote closed the stream\n\tstorj.io/drpc/drpcstream.(*Stream).HandlePacket:202\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:211}
2022-02-21T22:40:32.784+0400 ERROR piecestore download failed {Piece ID: 674XPYOQQWYL7IIKK7YZAKKFGJA66X7YNBTB3O62NDGXMJL7ZBPQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET, error: drpc: remote closed the stream, errorVerbose: drpc: remote closed the stream\n\tstorj.io/drpc/drpcstream.(*Stream).HandlePacket:202\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:211}
2022-02-21T22:48:25.410+0400 ERROR piecestore download failed {Piece ID: 674XPYOQQWYL7IIKK7YZAKKFGJA66X7YNBTB3O62NDGXMJL7ZBPQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET, error: drpc: remote closed the stream, errorVerbose: drpc: remote closed the stream\n\tstorj.io/drpc/drpcstream.(*Stream).HandlePacket:202\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:211}
2022-02-21T23:01:50.231+0400 ERROR piecestore:cache error getting current used space: {error: context canceled; context canceled; context canceled; context canceled; context canceled; context canceled, errorVerbose: group:\n— context canceled\n— context canceled\n— context canceled\n— context canceled\n— context canceled\n— context canceled}
2022-02-21T23:04:23.881+0400 ERROR collector unable to delete piece {Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: T7CKEKFLZ7LMWM56FPYJT6EINMSLJ4II6EP6ICANUJX63IS5H77A, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2022-02-21T23:05:48.391+0400 WARN console:service unable to get Satellite URL {Satellite ID: 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW, error: console: trust: satellite \118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\ is untrusted, errorVerbose: console: trust: satellite \118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\ is untrusted\n\tstorj.io/storj/storagenode/trust.(*Pool).getInfo:238\n\tstorj.io/storj/storagenode/trust.(*Pool).GetNodeURL:177\n\tstorj.io/storj/storagenode/console.(*Service).GetDashboardData:186\n\tstorj.io/storj/storagenode/console/consoleapi.(*StorageNode).StorageNode:45\n\tnet/http.HandlerFunc.ServeHTTP:2047\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2879\n\tnet/http.(*conn).serve:1930}
2022-02-21T23:05:51.949+0400 WARN console:service unable to get Satellite URL {Satellite ID: 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW, error: console: trust: satellite \118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\ is untrusted, errorVerbose: console: trust: satellite \118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\ is untrusted\n\tstorj.io/storj/storagenode/trust.(*Pool).getInfo:238\n\tstorj.io/storj/storagenode/trust.(*Pool).GetNodeURL:177\n\tstorj.io/storj/storagenode/console.(*Service).GetDashboardData:186\n\tstorj.io/storj/storagenode/console/consoleapi.(*StorageNode).StorageNode:45\n\tnet/http.HandlerFunc.ServeHTTP:2047\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2879\n\tnet/http.(*conn).serve:1930}
2022-02-21T23:12:16.584+0400 ERROR piecestore download failed {Piece ID: 674XPYOQQWYL7IIKK7YZAKKFGJA66X7YNBTB3O62NDGXMJL7ZBPQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET, error: drpc: remote closed the stream, errorVerbose: drpc: remote closed the stream\n\tstorj.io/drpc/drpcstream.(*Stream).HandlePacket:202\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:211}
2022-02-21T23:16:23.911+0400 WARN console:service unable to get Satellite URL {Satellite ID: 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW, error: console: trust: satellite \118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\ is untrusted, errorVerbose: console: trust: satellite \118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\ is untrusted\n\tstorj.io/storj/storagenode/trust.(*Pool).getInfo:238\n\tstorj.io/storj/storagenode/trust.(*Pool).GetNodeURL:177\n\tstorj.io/storj/storagenode/console.(*Service).GetDashboardData:186\n\tstorj.io/storj/storagenode/console/consoleapi.(*StorageNode).StorageNode:45\n\tnet/http.HandlerFunc.ServeHTTP:2047\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2879\n\tnet/http.(*conn).serve:1930}
2022-02-21T23:16:40.136+0400 ERROR piecestore download failed {Piece ID: 674XPYOQQWYL7IIKK7YZAKKFGJA66X7YNBTB3O62NDGXMJL7ZBPQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET, error: drpc: remote closed the stream, errorVerbose: drpc: remote closed the stream\n\tstorj.io/drpc/drpcstream.(*Stream).HandlePacket:202\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:211}

The log file since 23/11/2021 consists about 83k marks with errors and warnings;
Used capasity grows;
the trash is about 40GB of 4,5TB - dunno if it’s important.

69

I´ll add that, the suspension will allow for Node to eventually keep up, get out of suspension and return to normal operation.
At least that´s what happened to my Synology nodes.
Funny enough, my Windows nodes were completely ok…who would guess :stuck_out_tongue:

70

My Win nodes on Docker quite ok, but something strange with Win Gui.
In a 1st suspension the computer hangs till force reboot and after rebooting the disk with node not initialised. I have thought that the disk was dead and the node too.
I’ve switched the sata connectors and the disk comes in system again.
That’s luck but that unrecognized shit with the node is drilling the brain.
And the trash in node space keeps growing.

71

My both windows GUI nodes…all ok, and actually after checking the dashboard, they even took a bigger “beating” than both the Synology ones.

72

Only GET_AUDIT and GET_REPAIR, the other events doesn’t affect suspension score.
Errors related to “file not found” for collector please see in ERROR collector unable to delete piece

For others in this thread

The online score and related suspension is not the same as a drop in suspension score.
Drop in online score is related to downtime and will recover in the next 30 days online.
Drop in suspension score is a little bit better than drop in audit score, but can end with disqualification and this is much more dangerous even if it could recover faster than online score.

The suspension score can be affected only during GET_AUDIT or GET_REPAIR. Please, filter only these events.

Please, do not post about suspension for downtime in this thread - it’s unrelated to troubleshooting of undetected drop in suspension score.

Downtime doesn’t affect suspension score

2 Likes
73

Stats of today.
Node1:

Screenshot_2022-02-22_121853
Screenshot_2022-02-22_1218531129×423 73 KB 

docker logs storagenode1 2>&1 | grep -E 'GET_AUDIT' | grep 'failed' | wc -l
0
docker logs storagenode1 2>&1 | grep -E 'GET_REPAIR' | grep 'ERROR' | wc -l
30

The errors are 24x broken pipe and 6x use of closed network connection

Node2:

Screenshot_2022-02-22_122443
Screenshot_2022-02-22_1224431164×465 84.6 KB 

docker logs storagenode2 2>&1 | grep -E 'GET_AUDIT' | grep 'failed' | wc -l
0
docker logs storagenode2 2>&1 | grep -E 'GET_REPAIR' | grep 'ERROR' | wc -l
4

Errors are 4x broken pipe

Node3:

Screenshot_2022-02-22_122756
Screenshot_2022-02-22_1227561171×454 82.4 KB 

docker logs storagenode3 2>&1 | grep -E 'GET_AUDIT' | grep 'failed' | wc -l
0
docker logs storagenode3 2>&1 | grep -E 'GET_REPAIR' | grep 'ERROR' | wc -l
3

Errors were 2x broken pipe and 1x use of closed network connection

Again I think the error numbers do not justify a suspension.

74

Adding another data point:

saltlake2

Now:

saltlake

Log for last 134 hours:

docker logs storagenode 2>&1 | grep -E 'GET_REPAIR' | grep 'ERROR' | grep '1wFTA' | wc -l
0
docker logs storagenode 2>&1 | grep -E 'GET_AUDIT' | grep 'ERROR' | grep '1wFTA' | wc -l
0
75

@thepaul Here we have an example now for the impact of the repair job on the audit score.

I was concerned that the repair job might do this in 2 situations.

  1. Repair job doesn’t care about randomness and might disqualify you because it checks exactly the pieces you are missing. (not the reason here)
  2. Repair job has no maximum concurrency. A storage node can stack up multiple pending repair jobs and fail them all at once. This is impossible with the audit job because the audit job has a maximum concurrency and will wait before starting the next round.
3 Likes
76

I’m having what I believe is the same issue. One of my nodes is suspended, first in 2 satellites, today in 3 more but first 2 are ok.

The errors I am seeing are broken pipe as well.

2022-02-24T05:04:18.654Z        ERROR   piecestore      download failed {"Piece ID": "BJEH6KMXI5FB7WDMPZSRWYVGBGPXQUAINZXTSMBYPWT4OTNHD3VQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.2:28967->23.88.101.156:48516: write: broken pipe", "errorVerbose": "write tcp 172.17.0.2:28967->23.88.101.156:48516: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:570\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-02-24T05:04:18.662Z        ERROR   piecestore      download failed {"Piece ID": "IUA67MVUVJPGML3CVLUJTY4XAUZU2GKGTYTBDUV6KOXUOACK6IPQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.2:28967->116.203.197.184:49636: write: broken pipe", "errorVerbose": "write tcp 172.17.0.2:28967->116.203.197.184:49636: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:570\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-02-24T05:04:18.701Z        ERROR   piecestore      download failed {"Piece ID": "OWH6H35PXKXIMYY6CSITB3J32O3FENCPWA3CCMNWN2CVNFJCBNCA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.2:28967->116.203.76.110:49976: write: broken pipe", "errorVerbose": "write tcp 172.17.0.2:28967->116.203.76.110:49976: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:570\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-02-24T05:04:18.748Z        ERROR   piecestore      download failed {"Piece ID": "UJQZ55EXKP7FC4PCBVE3K7ZFTU4IARORROBYEJYT2RXYHD7AMJ7Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.2:28967->5.161.82.40:59590: write: broken pipe", "errorVerbose": "write tcp 172.17.0.2:28967->5.161.82.40:59590: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:570\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-02-24T05:04:18.757Z        ERROR   piecestore      download failed {"Piece ID": "OUSPYRQVBOK57KBQVIZQVTI6L2JM6NGZQ46LGKRUNXLVX347Q7IQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.2:28967->116.203.197.184:55720: write: broken pipe", "errorVerbose": "write tcp 172.17.0.2:28967->116.203.197.184:55720: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:570\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-02-24T05:04:18.770Z        ERROR   piecestore      download failed {"Piece ID": "NT55HCY32LJ5TO5RA4RN5CPZMFZ7URAORIQ5SMXTBXOX372O5RFQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.2:28967->5.161.80.131:51376: write: broken pipe", "errorVerbose": "write tcp 172.17.0.2:28967->5.161.80.131:51376: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:570\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-02-24T05:04:18.925Z        ERROR   piecestore      download failed {"Piece ID": "7C5JK6OQPJUF2R6X3ZZDZWZUDKXKQYABWQBLUUD2P7LVAEJWFKBA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.2:28967->195.201.119.188:35118: write: broken pipe", "errorVerbose": "write tcp 172.17.0.2:28967->195.201.119.188:35118: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:570\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-02-24T05:04:18.981Z        ERROR   piecestore      download failed {"Piece ID": "7NZXIYEASMNIHCCILTB7IGRVJDX4C2ZWHXJQ7CAAIG76CEXL3QDQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.2:28967->5.161.50.62:46428: write: broken pipe", "errorVerbose": "write tcp 172.17.0.2:28967->5.161.50.62:46428: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:570\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-02-24T05:04:18.994Z        ERROR   piecestore      download failed {"Piece ID": "BY6IAJXSUH3PCLSWJEAD644UPLTHXKPDD3TT7XPUNNGAMCQPTEZA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.2:28967->142.132.168.249:39634: write: broken pipe", "errorVerbose": "write tcp 172.17.0.2:28967->142.132.168.249:39634: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:570\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-02-24T05:04:19.032Z        ERROR   piecestore      download failed {"Piece ID": "7TAHREXTXDUFUF6PFVGPISTZGFNPW7XKLSHZ7VBGQL7XRDO3YF6A", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.2:28967->195.201.119.188:48136: write: broken pipe", "errorVerbose": "write tcp 172.17.0.2:28967->195.201.119.188:48136: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:570\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
77

So, kept the nodes running, hoping they don’t get disqualified. And this is what it is now:

Node1:

node1
node11144×427 73.1 KB

Node2:

node2
node21149×434 72.9 KB

Node3:

node3
node31177×436 75.5 KB

So obviously all of them have recovered back to a (good) workable state. Weird, weird, weird.

1 Like
78

Seeing the amount of nodes getting suspended is kinda showing some issues going on the network 412 according to the API stats. Saw it hit 430 now its 412 seems a bit alarming. In jan there was a max of 8 suspended nodes.

79

Yes, we saw a lot of comments about suspended nodes lately. Maybe there is some kind of issue.

80

Nodes again suspended… :confounded:
Wasn’t quick enough to take screenshots, but I logged into the dashboard, it said nodes suspendes on us1-satellite. I refresh 5 minutes later, everything back out of suspensioin.

This suspension system ist totally useless this way.

1 Like
81

Hi!
Got a suspension from saltlake about hour ago, then I’ve stopped a node, deleted trust-cache.json, started a node, trust-cache.json rolled up. It seems quite ok, the suspension mode had closed.
But it was appeared at a flat ground and there is no obvious problems with node for me.
there is only one WARN:

2022-04-29 14:28:49,656 WARN received SIGTERM indicating exit request

2022-04-29T14:29:47.307Z WARN console:service unable to get Satellite URL {"Process": "storagenode", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "error": "console: trust: satellite \"118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\" is untrusted", "errorVerbose": "console: trust: satellite \"118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\" is untrusted\n\tstorj.io/storj/storagenode/trust.(*Pool).getInfo:238\n\tstorj.io/storj/storagenode/trust.(*Pool).GetNodeURL:177\n\tstorj.io/storj/storagenode/console.(*Service).GetDashboardData:186\n\tstorj.io/storj/storagenode/console/consoleapi.(*StorageNode).StorageNode:45\n\tnet/http.HandlerFunc.ServeHTTP:2047\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2879\n\tnet/http.(*conn).serve:1930"}

Don’t know how to react in this case.
Please help.

82

GET_AUDIT log:

`2022-04-29T14:24:40.828Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "SMRLA4E2RJPGSNZ2KHPUZ4ZWOTEDKKE4JIA2CC34IXJQMWDEPWJQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_AUDIT` `"}`

`2022-04-29T14:24:41.249Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "SMRLA4E2RJPGSNZ2KHPUZ4ZWOTEDKKE4JIA2CC34IXJQMWDEPWJQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_AUDIT` `"}`

`2022-04-29T14:26:16.365Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "IYZXHX2VD32G45SANEDLFTHB4UKIZUXK5QUOSSKCHMZW54NZAEOA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_AUDIT` `"}`

`2022-04-29T14:26:16.717Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "IYZXHX2VD32G45SANEDLFTHB4UKIZUXK5QUOSSKCHMZW54NZAEOA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_AUDIT` `"}`

`2022-04-29T14:30:27.153Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "AX2PURPF4OZKPE4LHRDI5ZIVAFHFO2X7U4ODVKJE4UR7GFNHR2HQ", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Action": "` `GET_AUDIT` `"}`

`2022-04-29T14:30:27.371Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "AX2PURPF4OZKPE4LHRDI5ZIVAFHFO2X7U4ODVKJE4UR7GFNHR2HQ", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Action": "` `GET_AUDIT` `"}`

`2022-04-29T14:35:26.569Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "QS4YK5O4VTFZAZRE2JF3BF37JIM2NXHZN5SSBB22BFGHOXSNFPWQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_AUDIT` `"}`

`2022-04-29T14:35:27.153Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "QS4YK5O4VTFZAZRE2JF3BF37JIM2NXHZN5SSBB22BFGHOXSNFPWQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_AUDIT` `"}`
83

GET_REPAIR log:

`2022-04-29T14:26:08.084Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "RKBPEU7SHFIOMYRRANS5XVHJER5LQP7RH6DXCADXAUZ4BOG5EU3Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:08.652Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "CS52YEDHNVOVD2YOKFGQUJEXXHTZORASIBEQWQ3X5GWE5UFNRYRQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:10.670Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "LBCT6CMMX62H26E7P5FC3VRAFVCX7PMSUMREHGHZZ6NLROLDOOOA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:11.240Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "CS52YEDHNVOVD2YOKFGQUJEXXHTZORASIBEQWQ3X5GWE5UFNRYRQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:12.832Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "LBCT6CMMX62H26E7P5FC3VRAFVCX7PMSUMREHGHZZ6NLROLDOOOA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:15.715Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "KE6RH2ZZBZM5KW6IRMKEHKJ6O27FAQ57TGFCHCRD22TH5VRVFZ7A", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:16.304Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "KE6RH2ZZBZM5KW6IRMKEHKJ6O27FAQ57TGFCHCRD22TH5VRVFZ7A", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:19.080Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "UXBNYG4KGHBDXEQQSYCWQATMAABQYDHVO7PW5KGQHF35SNJE45FA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:21.095Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "UXBNYG4KGHBDXEQQSYCWQATMAABQYDHVO7PW5KGQHF35SNJE45FA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:24.346Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "4VUV6SQWIYGBPRKJOKISYQEJUJLKFVPGFFLFLRH3FI2LYVICIGLA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:25.989Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "4VUV6SQWIYGBPRKJOKISYQEJUJLKFVPGFFLFLRH3FI2LYVICIGLA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:27.048Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "NTUTSBG52PHPSNIFCSCLMQTL7TCSRZIVUUHEDBZYKRHV2NFBEUUA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:29.797Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "NTUTSBG52PHPSNIFCSCLMQTL7TCSRZIVUUHEDBZYKRHV2NFBEUUA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:37.513Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "L7EO2776HSDP7RALFFZLI27KCLD6NKXFS6EP4HNJKULIXKQQN7LQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:37.767Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "L7EO2776HSDP7RALFFZLI27KCLD6NKXFS6EP4HNJKULIXKQQN7LQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:39.279Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "5KGZT4OQVRKSMOH7VLWVDIESLFZFXJJVYHKXXRNEVUUI4DXOV4BQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:41.272Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "5KGZT4OQVRKSMOH7VLWVDIESLFZFXJJVYHKXXRNEVUUI4DXOV4BQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:44.888Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "R567T4QUKPWR7NYXMLIZ6OGPTOGENVGVD3QJWGTDH7ZOQJ34IWTQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:45.116Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "R567T4QUKPWR7NYXMLIZ6OGPTOGENVGVD3QJWGTDH7ZOQJ34IWTQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:48.552Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "4EJOBMOOSXM3K7IWYNRHIOJJYUHFEZS2SA7IZCV7UIPIUNMG6GRQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:49.061Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "4EJOBMOOSXM3K7IWYNRHIOJJYUHFEZS2SA7IZCV7UIPIUNMG6GRQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:51.588Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "3LYCUXODBGYCMWMFXRR357A737CGRPSZTHRGOYAKONNJ55XZW4JA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:51.817Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "3LYCUXODBGYCMWMFXRR357A737CGRPSZTHRGOYAKONNJ55XZW4JA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:52.289Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "5T2FASBZ32N5J7JM6UYHJ4O7B6EAWNKM24EZS3BA7BEQUO2TW3NQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:52.738Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "USWGPGBJIFWEE3Z7YPQLIUJQEC2PGSPC5IBK5YPDAT5JU72L3IYA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:52.945Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "USWGPGBJIFWEE3Z7YPQLIUJQEC2PGSPC5IBK5YPDAT5JU72L3IYA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:55.230Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "5T2FASBZ32N5J7JM6UYHJ4O7B6EAWNKM24EZS3BA7BEQUO2TW3NQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:57.527Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "6XMVIECKXGCWB3OYHYV4FGMYXJKSMUZ2U3PGU6XS6U5V5FGEGVTQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:26:57.915Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "6XMVIECKXGCWB3OYHYV4FGMYXJKSMUZ2U3PGU6XS6U5V5FGEGVTQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:02.597Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "USUT37IP7NDZFLWZH25I5Y6TDOBSRL6OOHU3KAMIGLCTD7TOULOQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:03.071Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "2PVNT5AJRIVC2BUWEJEEMXXEUI662UIX4AA7NUAWPPR3RATHSJAQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:03.367Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "USUT37IP7NDZFLWZH25I5Y6TDOBSRL6OOHU3KAMIGLCTD7TOULOQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:03.834Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "GB3FRG6VZM6MPQ7SAXK66GJSMON4NI6XEGBMNWX2E5OBWCSRDV5Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:04.095Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "2PVNT5AJRIVC2BUWEJEEMXXEUI662UIX4AA7NUAWPPR3RATHSJAQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:06.419Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "GB3FRG6VZM6MPQ7SAXK66GJSMON4NI6XEGBMNWX2E5OBWCSRDV5Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:06.946Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "MTCQWLQ6R7L2DZPYFDCY3W2VUT5U7KG5YTROQJGK5V2PUYP6DH3A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:07.029Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "VXQSFJN4XOMMUREWM7M72VUJYEJUQ35NZG4VCPZBRTKQMO62UTCA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:07.993Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "VXQSFJN4XOMMUREWM7M72VUJYEJUQ35NZG4VCPZBRTKQMO62UTCA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:09.807Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "MTCQWLQ6R7L2DZPYFDCY3W2VUT5U7KG5YTROQJGK5V2PUYP6DH3A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:15.001Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "FT5E74ZO2WWXALUIPORV7E4FM52YMFE4YIQXFATNQTGQH54RMTEQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:16.279Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "FT5E74ZO2WWXALUIPORV7E4FM52YMFE4YIQXFATNQTGQH54RMTEQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:18.393Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "QWNHE4ZA63ODIA25XUIH72N7OHHHMACKGNPLPRFPIXOM75GI5TBQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:19.327Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "LIXZY27QWG6XAF4DHZLTQFYWVEUDHM4JQDJQO6XT4BZHY6VAWD5A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:20.241Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "QWNHE4ZA63ODIA25XUIH72N7OHHHMACKGNPLPRFPIXOM75GI5TBQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:20.345Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "NBRTHOCJRHIG22S5NE4UHLYSUNJSLAXYX44VZNCLDFJMFRM6DFNQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:20.599Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "NBRTHOCJRHIG22S5NE4UHLYSUNJSLAXYX44VZNCLDFJMFRM6DFNQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:21.491Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "XB2ZUO4ASOZGVI2JMZE3O4QVFWSPDL73GTLMI6L5YAT7GQY3XGJQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:21.548Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "LIXZY27QWG6XAF4DHZLTQFYWVEUDHM4JQDJQO6XT4BZHY6VAWD5A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:21.661Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "XB2ZUO4ASOZGVI2JMZE3O4QVFWSPDL73GTLMI6L5YAT7GQY3XGJQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:24.399Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "CVB6ZPQLU4DWKT2TUYMUQ6Q5MR36F4BB4NGR6JFG5BHPSAVXSL4A", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:24.698Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "CVB6ZPQLU4DWKT2TUYMUQ6Q5MR36F4BB4NGR6JFG5BHPSAVXSL4A", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:25.256Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "GW52GKUJMHG5YLJCCKR2BKSANQQQLATJBPNYEQEJ3U2G2TO365GQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:27.971Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "GW52GKUJMHG5YLJCCKR2BKSANQQQLATJBPNYEQEJ3U2G2TO365GQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:29.772Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "BONWDBNRTIOP7Y4R4MKZFNMYTP6XNVRRASYNRDNVHZDHCTWPVTQA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:29.839Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "S7GULU4UOSXHXAFFS4L45THPW6L5B2X73JGZ2NAYM4DWPP4TLFWA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:30.027Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "BONWDBNRTIOP7Y4R4MKZFNMYTP6XNVRRASYNRDNVHZDHCTWPVTQA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:32.829Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "S7GULU4UOSXHXAFFS4L45THPW6L5B2X73JGZ2NAYM4DWPP4TLFWA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:33.253Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "TLKYX3VOAK47CVNU4BR36R2ELHGCBGTTWC4VD2OZ74XQKU77SO5A", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:33.673Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "TLKYX3VOAK47CVNU4BR36R2ELHGCBGTTWC4VD2OZ74XQKU77SO5A", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:34.758Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "OYQXENH37PMJVAZDVYQCY7QANZDSLM7IYCUYM5QFMXHYZTZSFSLA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:36.783Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "OYQXENH37PMJVAZDVYQCY7QANZDSLM7IYCUYM5QFMXHYZTZSFSLA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:38.677Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "6UI7DATG34ZFVHVTVNFUGCJBDH4RFI6GFNFEBSPWCF2IA6OADH2Q", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:39.420Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "6UI7DATG34ZFVHVTVNFUGCJBDH4RFI6GFNFEBSPWCF2IA6OADH2Q", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:41.892Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "ZE4LT37L4ICC5ADK3GD574VKB5ZQQISMWWU7OFSM6CRMTJH7F3SQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:45.783Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "W245P3RYUPMCNPAW46I2L73JFBWR6QLL6P4SSXVGFQIE2QCGP22A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:47.289Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "W245P3RYUPMCNPAW46I2L73JFBWR6QLL6P4SSXVGFQIE2QCGP22A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:55.939Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "7ODA4TJ6TN3UTGVOILFCWLUUXJHBCRFNANHPZW4NTUC4UHCVXPQQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:56.057Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "ZE4LT37L4ICC5ADK3GD574VKB5ZQQISMWWU7OFSM6CRMTJH7F3SQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:27:56.133Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "7ODA4TJ6TN3UTGVOILFCWLUUXJHBCRFNANHPZW4NTUC4UHCVXPQQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:28:07.733Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "BT3USMO5BAEPMBINNF4GRCFRMAUNVAMYAJRIK3G2L7CPT5PX3SBA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:28:07.999Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "BT3USMO5BAEPMBINNF4GRCFRMAUNVAMYAJRIK3G2L7CPT5PX3SBA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:28:13.678Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "LPORFKYGUFT2HKVZPMJXKWYE7AWJGG5LFIWJMKYF4IVDNFBMNBOA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:28:15.196Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "AFLRGN5OCRN3LXJM7U6DBOPCLF6A6SEJXRFBSEXGYN4LOV5QNTHA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:28:15.397Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "LPORFKYGUFT2HKVZPMJXKWYE7AWJGG5LFIWJMKYF4IVDNFBMNBOA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:28:16.511Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "AFLRGN5OCRN3LXJM7U6DBOPCLF6A6SEJXRFBSEXGYN4LOV5QNTHA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:28:26.467Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "2JZ6476W4J7G2G6VL3Y7FA5UXFO7PTAFF37XH4ZMH5AXNDQQWW2A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:28:27.875Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "2JZ6476W4J7G2G6VL3Y7FA5UXFO7PTAFF37XH4ZMH5AXNDQQWW2A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:28:37.678Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "UKAYOKFTT5VYN7LU6BMDU3TULJDFQR4UOV35EAAZYCEGNX5XYDXQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:28:39.294Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "UKAYOKFTT5VYN7LU6BMDU3TULJDFQR4UOV35EAAZYCEGNX5XYDXQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:28:45.400Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "OMGMAEPU24OPTFLTDHZW6YGEFU55D7VEX3ZAM5IUMFNK55Q2EZNA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:28:46.735Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "OMGMAEPU24OPTFLTDHZW6YGEFU55D7VEX3ZAM5IUMFNK55Q2EZNA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:28:48.434Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "WHYYH3TXQOKTFTMBIEJGVPOJ3PJ7DRBZIL3IZOSN3M5VQO3UAEMQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:28:49.709Z INFO piecestore download canceled {"Process": "storagenode", "Piece ID": "WHYYH3TXQOKTFTMBIEJGVPOJ3PJ7DRBZIL3IZOSN3M5VQO3UAEMQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:05.456Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "PL22GBFJOGMCOACBNERFJX3LTHY53SL7MINKIJRQZODFI5PTY2ZQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:05.592Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "6LDX2VNW32JCI3YOPYC7NN2JVQ3QPMVWCSGXE67ONK67YPPS4Q5A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:06.456Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "6LDX2VNW32JCI3YOPYC7NN2JVQ3QPMVWCSGXE67ONK67YPPS4Q5A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:07.742Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "PL22GBFJOGMCOACBNERFJX3LTHY53SL7MINKIJRQZODFI5PTY2ZQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:11.185Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "VOPG5UVLVQ5S6FH6F5VC6WB6ZM4ZVOLNQACGUVV34SMP6AVEIZVA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:12.491Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "VOPG5UVLVQ5S6FH6F5VC6WB6ZM4ZVOLNQACGUVV34SMP6AVEIZVA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:15.628Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "N34AH3HTLJYGMGCQQRDVJSTC5PQ3EDM2XQS3EO4VKS5QJY3JC73A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:16.858Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "N34AH3HTLJYGMGCQQRDVJSTC5PQ3EDM2XQS3EO4VKS5QJY3JC73A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:18.109Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "MOADIMKBHF4RRYH35X46I3XIKX4MVS44BA3LX3PWFMD5A7NR234A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:18.343Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "MOADIMKBHF4RRYH35X46I3XIKX4MVS44BA3LX3PWFMD5A7NR234A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:21.435Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "QVIEWVQJH6LI6RACPHC3EZWYM36R5VS5SUY4BGEYJ32HWZPK2G2Q", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:23.081Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "QVIEWVQJH6LI6RACPHC3EZWYM36R5VS5SUY4BGEYJ32HWZPK2G2Q", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:26.039Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "CIMDDBUU73AAIU2GH7IGHB4PWRXCJQ6HI2VC63MHYPT6CGJA7XLA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:26.196Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "CIMDDBUU73AAIU2GH7IGHB4PWRXCJQ6HI2VC63MHYPT6CGJA7XLA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:32.274Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "MIIFHKPUI6H4ZGLUBJGKYYJO4SHT4UMVO6LSS6TUCPAAYLZ32CLA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:32.482Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "MIIFHKPUI6H4ZGLUBJGKYYJO4SHT4UMVO6LSS6TUCPAAYLZ32CLA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:37.205Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "5YKFFV7NXIKPIT3V4MJBLPR7WNA3ZWSIVIRVZ5IKNLMMI74CF5XQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:37.803Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "5YKFFV7NXIKPIT3V4MJBLPR7WNA3ZWSIVIRVZ5IKNLMMI74CF5XQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:38.566Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "AVL4OHFRF35ANOHBKBC2UQHYNEGKTQ4PWIKFAWLF5EMK4W7S5AMQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:39.398Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "YIPGMFBCB7ANZBXSL2LRWVS2AI3KVAI2J4O4XSTUJQXDA6WROGOA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:39.895Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "AVL4OHFRF35ANOHBKBC2UQHYNEGKTQ4PWIKFAWLF5EMK4W7S5AMQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:40.102Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "YIPGMFBCB7ANZBXSL2LRWVS2AI3KVAI2J4O4XSTUJQXDA6WROGOA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:43.432Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "Y6CD32FWJIKBZWXH6C7IBAPASVXJ7PIERMYTQV4FAL2BOXITD5FQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:43.584Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "Y6CD32FWJIKBZWXH6C7IBAPASVXJ7PIERMYTQV4FAL2BOXITD5FQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:45.003Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "NLBU2XROE62EDGYJBDYB6CY3MFRQCP7G7N5IR2W3OLO5WX7A7KDQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:47.223Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "NLBU2XROE62EDGYJBDYB6CY3MFRQCP7G7N5IR2W3OLO5WX7A7KDQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:48.888Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "M6BI5XFW5ZBMVLCWQIS6MEQYRINHNSNWOWEXOOLVPDFZUBNGCP3A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:50.110Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "M6BI5XFW5ZBMVLCWQIS6MEQYRINHNSNWOWEXOOLVPDFZUBNGCP3A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:52.779Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "UBUUNDQXFD5Y3YMDFFGTF5W3LSOFGL5XKVOYIM46TH6BFRQZ4SEQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:52.909Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "UBUUNDQXFD5Y3YMDFFGTF5W3LSOFGL5XKVOYIM46TH6BFRQZ4SEQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:29:59.710Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "NMPWTNEJDUJO33EPHHSNTIX2UICLASVSFB5OEJDFDYYMRU5FJGGQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:30:00.806Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "NMPWTNEJDUJO33EPHHSNTIX2UICLASVSFB5OEJDFDYYMRU5FJGGQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:30:02.496Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "VWXEUHU7KF6UL2US4A3K3NXOZX3AWB6TGJU7WZXUK2BE6OYNIEWA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:30:02.667Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "VWXEUHU7KF6UL2US4A3K3NXOZX3AWB6TGJU7WZXUK2BE6OYNIEWA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:30:02.688Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "GCKIEZNQZJXFGF5ITHVPPGWFDTK24K4OECB5X4RTP4VD3M2ADZBA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:30:02.821Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "5THO54UJRVXZEMH7TW5GOCXGDETZVALQF6BNT5CSGAW5LDQUB3JQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:30:03.816Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "GCKIEZNQZJXFGF5ITHVPPGWFDTK24K4OECB5X4RTP4VD3M2ADZBA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:30:03.862Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "5THO54UJRVXZEMH7TW5GOCXGDETZVALQF6BNT5CSGAW5LDQUB3JQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:30:08.131Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "T33MZTO3LWD73YB6WBQRVRACSSY4MYWCXO33SIEPKUOQXNC4V6YQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:30:08.294Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "T33MZTO3LWD73YB6WBQRVRACSSY4MYWCXO33SIEPKUOQXNC4V6YQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:30:10.186Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "QOL3OLVRZ3Y5ZYTEAYK36OPTGYWV32DY4S3LYF653W7FFHMCEEXA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:30:11.415Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "QOL3OLVRZ3Y5ZYTEAYK36OPTGYWV32DY4S3LYF653W7FFHMCEEXA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`

`2022-04-29T14:30:11.981Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "COLE4NVBJD5AJCQNVX5BJSWHDMEB5KELWE65KSR57JXJX3SXZTCA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "` `GET_REPAIR` `"}`