Node periodically offline

Node Operators troubleshooting
,
1

Hello Storj community!

Maybe someone knows about this issue, so I can finally fix it.

Almost every morning my node is offline; I read the logs and the following loop appears. A docker restart storagenode fixes it.

Thanks in advance,
If more info is needed please let me know.

storj@cadiz-1:~$ docker logs --tail 30 storagenode
2024-05-09T06:56:38Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-09T06:56:38Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-09T06:56:38Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-09T06:56:38Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-09T06:56:38Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-09T07:11:38Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-09T07:11:38Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-09T07:11:38Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-09T07:11:38Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-09T07:11:38Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-09T07:26:38Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-09T07:26:38Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-09T07:26:38Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-09T07:26:38Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-09T07:26:38Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-09T07:41:38Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-09T07:41:38Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-09T07:41:38Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-09T07:41:38Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-09T07:41:38Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-09T07:56:38Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-09T07:56:38Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-09T07:56:38Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-09T07:56:39Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-09T07:56:39Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-09T08:11:38Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-09T08:11:38Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-09T08:11:38Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-09T08:11:38Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-09T08:11:38Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2

If you redirected logs to the file, you need to check the log of the node in that file instead of docker logs, because after redirection the docker logs will show logs only from the updater.

1 Like
3

Hello @Alexey,

It looks like it’s something with a failed filewalker run:

2024-05-10T06:19:16Z    ERROR   pieces  lazyfilewalker failed   {"Process": "storagenode", "error": "lazyfilewalker: signal: killed", "errorVerbose": "lazyfilewalker: signal: killed\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkSatellitePiecesToTrash:160\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:578\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:258\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-05-10T06:19:16Z    ERROR   filewalker      failed to get progress from database    {"Process": "storagenode", "error": "gc_filewalker_progress_db: context canceled", "errorVerbose": "gc_filewalker_progress_db: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*gcFilewalkerProgressDB).Get:47\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash:154\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:585\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:258\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-05-10T06:19:16Z    ERROR   filewalker      failed to reset progress in database

I’ll try adding this line to disable the lazy filewalker: pieces.enable-lazy-filewalker: false.

Thanks for the help

Full logs:

2024-05-10T06:19:16Z    INFO    lazyfilewalker.gc-filewalker    subprocess exited with status   {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "status": -1, "error": "signal: killed"}
2024-05-10T06:19:16Z    ERROR   pieces  lazyfilewalker failed   {"Process": "storagenode", "error": "lazyfilewalker: signal: killed", "errorVerbose": "lazyfilewalker: signal: killed\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkSatellitePiecesToTrash:160\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:578\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:258\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-05-10T06:19:16Z    ERROR   filewalker      failed to get progress from database    {"Process": "storagenode", "error": "gc_filewalker_progress_db: context canceled", "errorVerbose": "gc_filewalker_progress_db: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*gcFilewalkerProgressDB).Get:47\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash:154\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:585\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:258\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-05-10T06:19:16Z    ERROR   filewalker      failed to reset progress in database    {"Process": "storagenode", "error": "gc_filewalker_progress_db: context canceled", "errorVerbose": "gc_filewalker_progress_db: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*gcFilewalkerProgressDB).Reset:58\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash.func1:171\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash:244\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:585\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:258\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-05-10T06:19:16Z    ERROR   retain  retain pieces failed    {"Process": "storagenode", "cachePath": "config/retain", "error": "retain: filewalker: context canceled", "errorVerbose": "retain: filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash:178\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:585\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:258\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-05-10T06:23:11Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-10T06:23:16Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-10T06:23:16Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-10T06:23:29Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-10T06:23:29Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-10T06:38:10Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-10T06:38:10Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-10T06:38:10Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-10T06:38:10Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-10T06:38:10Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-10T06:53:10Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-10T06:53:11Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-10T06:53:11Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-10T06:53:11Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-10T06:53:11Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-10T07:08:10Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-10T07:08:10Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-10T07:08:10Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-10T07:08:10Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-10T07:08:10Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-10T07:23:10Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-10T07:23:10Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-10T07:23:10Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-10T07:23:10Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-10T07:23:10Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-10T07:38:10Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-10T07:38:10Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-10T07:38:10Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-10T07:38:10Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-10T07:38:10Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}```
4

== disk is too slow

yes, this should solve these errors, but will load the disk.

The only weird thing that you do not have any requests, is your node offline?

5

Node container is up but the node has no requests ongoing.

I’m also reciving a lot of these warnings before going offline, I didn’t noticed before:

2024-05-10T19:37:06Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DCOT2YM56TNWZJW275CSINHUFQVSIRQYCIPYUONAHGGDCWCV7VXQ", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:07Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DCBDCYXJSAJEVID7H7SZMYVKGLREXXSQMHDYR5YYGDVT5NAQ6C5A", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:08Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DCMFT5IXR25IEDT7FRHQFALFXUICKTT3EXGQFMTVBKRKZX74LFDA", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:09Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DC6IR4FBMP62UXF7CAJQ37KMRREGQBOT7BJUWYCGTPPWJPC2TBUQ", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:09Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DCFXJQVHEFAZRIX6QAFLETU6Z3TFAWOGXWIMTQO47LDH7MZL7FMA", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:11Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DCUU2SMMZ5VCUB54VPYZBC7E6WNWKUQFQT5RGWSZ4BVQX3FQGHZA", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:11Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DCAWOODAXOJTHKYWTBSO3TXJ24UHSFP7R3OCJ55WX4PMW45ITZUQ", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}

Full log:

2024-05-10T19:36:54Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DCIIFYEIONXQBL7QDALFVKM7RMAO7XEFQX6BEIB4XPTGHGCJEZIQ", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:36:57Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DCIGJCUTFN5IY55FXAJT2R7H7V2TEQ3KRZIJPNOISXDVUVK7FOKQ", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:36:58Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DC73JV4ZF5FTKTAZXAC4DRDTI4HWM5UIRKD3BZNHW6TRMYMU7F3Q", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:01Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DCGLXXGOQD4AG2DW6UFUWB35BZNPRANWODFFVKHLVHLBG3S4YDSQ", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:03Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DC2LBLBTAABLJPQYFQULXODLJKO3DY2Q4TJKDSVCPPMQXFUM4PCA", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:05Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DC7TGTBIMIUIUHYYLUOOKPUBDKTLKS63ZEO2LRKLCQVLDOWE6OCA", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:06Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DCOT2YM56TNWZJW275CSINHUFQVSIRQYCIPYUONAHGGDCWCV7VXQ", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:07Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DCBDCYXJSAJEVID7H7SZMYVKGLREXXSQMHDYR5YYGDVT5NAQ6C5A", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:08Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DCMFT5IXR25IEDT7FRHQFALFXUICKTT3EXGQFMTVBKRKZX74LFDA", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:09Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DC6IR4FBMP62UXF7CAJQ37KMRREGQBOT7BJUWYCGTPPWJPC2TBUQ", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:09Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DCFXJQVHEFAZRIX6QAFLETU6Z3TFAWOGXWIMTQO47LDH7MZL7FMA", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:11Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DCUU2SMMZ5VCUB54VPYZBC7E6WNWKUQFQT5RGWSZ4BVQX3FQGHZA", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:11Z    WARN    retain  failed to trash piece   {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DCAWOODAXOJTHKYWTBSO3TXJ24UHSFP7R3OCJ55WX4PMW45ITZUQ", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:408\n\tstorj.io/storj/storagenode/retain.(*Service).trash:418\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:377\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).processTrashPiece:111\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).writeLine:98\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*trashHandler).Write:77\n\tio.copyBuffer:432\n\tio.Copy:389\n\tos/exec.(*Cmd).writerDescriptor.func1:560\n\tos/exec.(*Cmd).Start.func2:717"}
2024-05-10T19:37:11Z    INFO    lazyfilewalker.gc-filewalker    subprocess exited with status   {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "status": -1, "error": "signal: killed"}
2024-05-10T19:37:11Z    ERROR   pieces  lazyfilewalker failed   {"Process": "storagenode", "error": "lazyfilewalker: signal: killed", "errorVerbose": "lazyfilewalker: signal: killed\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkSatellitePiecesToTrash:160\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:578\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:258\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-05-10T19:37:11Z    ERROR   filewalker      failed to get progress from database    {"Process": "storagenode", "error": "gc_filewalker_progress_db: context canceled", "errorVerbose": "gc_filewalker_progress_db: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*gcFilewalkerProgressDB).Get:47\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash:154\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:585\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:258\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-05-10T19:37:13Z    ERROR   filewalker      failed to reset progress in database    {"Process": "storagenode", "error": "gc_filewalker_progress_db: context canceled", "errorVerbose": "gc_filewalker_progress_db: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*gcFilewalkerProgressDB).Reset:58\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash.func1:171\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash:244\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:585\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:258\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-05-10T19:37:13Z    ERROR   retain  retain pieces failed    {"Process": "storagenode", "cachePath": "config/retain", "error": "retain: filewalker: context canceled", "errorVerbose": "retain: filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash:178\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:585\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:258\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-05-10T19:40:47Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-10T19:40:48Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-10T19:40:48Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-10T19:40:48Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-10T19:40:48Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-10T19:55:47Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-10T19:55:48Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-10T19:55:48Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-10T19:55:48Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-10T19:55:48Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-10T20:10:47Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-10T20:10:48Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-10T20:10:48Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-10T20:10:48Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-10T20:10:48Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-10T20:25:47Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-10T20:25:48Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-10T20:25:48Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-10T20:25:48Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-10T20:25:48Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-10T20:40:47Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-10T20:40:48Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-10T20:40:48Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-10T20:40:48Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-10T20:40:48Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
6

== slow disk

Could you please open your external address with it’s port in your mobile browser using the mobile internet (not WiFi)?

What’s version of storagenode?

I would recommend:

  1. Stop and remove the container
  2. Unmount the disk
  3. Check and fix errors with sudo fsck
    • Perhaps you would need to do it several times until it wouldn’t throw any error
  4. Mount the disk back with sudo mount -a
  5. check databases: https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed
  6. Run the node back
7

Version is v1.102.3

After applying pieces.enable-lazy-filewalker: false node is still shutting down during the nights, filewalker was not causing the issue after all:

2024-05-11T18:28:38Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"Process": "storagenode", "Piece ID": "SS3ZXMRZJHCMFX6J3DNIEOLPVVJVN2SIMBPC5CDEHCRBR7WSH7WQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "5.161.77.10:21858"}
2024-05-11T18:28:38Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"Process": "storagenode", "Piece ID": "W3TD7S7FZWBHH7AIYKCMDN5X4GTCWTWPUKHFT3XNR5SW57OQ7HWA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.101:34200"}
2024-05-11T18:28:38Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"Process": "storagenode", "Piece ID": "NXH237CCZFMSVPS345VRUZ5ONKUCNXF6BQYKYVT32N5YIDGEXALA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.205.225:37584"}
2024-05-11T18:29:40Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-11T18:29:41Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-11T18:29:41Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-11T18:29:41Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-11T18:29:41Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-11T18:44:40Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-11T18:44:41Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-11T18:44:41Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-11T18:44:41Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-11T18:44:41Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-11T19:00:05Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-11T19:00:07Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-11T19:00:07Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-11T19:00:07Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-11T19:00:07Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-11T19:14:40Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-11T19:14:42Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-11T19:14:42Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-11T19:14:42Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-11T19:14:42Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}

Could you please open your external address with it’s port in your mobile browser using the mobile internet (not WiFi)?

I will try next time it’s down, but I think it will not open, because I’m reciving the ‘offline node’ email notification.

I would recommend:

I’m triying the fsck and db check, I’ll be back with the results.

1 Like
8
  • Running fsck multiple times doesn’t return errors.
  • DBs are fine.
  • Not able to connect to the node through IP:PORT using my mobile network (container is up, but node is offline)

The thing is, the node is shutting down without showing logs. I keep restarting the node manually when it gets offline.

9

You could give this a try: First steps, if your node is offline

10

Then you need to compare your WAN IP on your router with IP on Open Port Check Tool - Test Port Forwarding on Your Router, if they not match, your ISP placed you behind their CGNAT.
In that case you may ask them to disable it and allow to have a public IP, it can be dynamic, but must be public (WAN IP will match IP from yougetsignal).