Zero Ingress for a node that always worked

Node Operators troubleshooting
,
1

Hi, I have tried to migrate my node onto a new HDD through dd.
This meant that it was offline for about 3 days.

The migration failed because the new HDD had corrupt data and so I never used the new HDD for the node.

When I restarted the node with the old HDD it went find for some time and after some time I stopped receiving ingress completely.

This has been happening for a few days now.
At the same time the trash is getting really big and the “disk” used keeps getting smaller.

Here the last few lines of logs:

2025-11-13T00:27:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false}, “interval”: “10m0s”}
2025-11-13T00:32:14Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2025-11-13T00:32:15Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.141.2”}
2025-11-13T00:32:15Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-11-13T00:32:15Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.141.2”}
2025-11-13T00:32:15Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2025-11-13T00:37:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false, “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false}, “interval”: “10m0s”}
2025-11-13T00:47:14Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2025-11-13T00:47:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.141.2”}
2025-11-13T00:47:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-11-13T00:47:15Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.141.2”}
2025-11-13T00:47:15Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2025-11-13T00:47:27Z INFO bandwidth Persisting bandwidth usage cache to db {“Process”: “storagenode”}
2025-11-13T00:47:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false}, “interval”: “10m0s”}
2025-11-13T00:47:51Z INFO orders finished {“Process”: “storagenode”, “satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “count”: 30}
2025-11-13T00:47:51Z INFO orders finished {“Process”: “storagenode”, “satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “count”: 302}
2025-11-13T00:47:52Z INFO orders finished {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “count”: 7511}
2025-11-13T00:47:56Z INFO orders finished {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “count”: 20356}
2025-11-13T00:57:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false, “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false}, “interval”: “10m0s”}
2025-11-13T01:02:14Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2025-11-13T01:02:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.141.2”}
2025-11-13T01:02:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-11-13T01:02:15Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.141.2”}
2025-11-13T01:02:15Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2025-11-13T01:07:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false}, “interval”: “10m0s”}
2025-11-13T01:09:07Z INFO trust Scheduling next refresh {“Process”: “storagenode”, “after”: “6h8m37.488945849s”}
2025-11-13T01:17:14Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2025-11-13T01:17:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.141.2”}
2025-11-13T01:17:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-11-13T01:17:15Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.141.2”}
2025-11-13T01:17:15Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2025-11-13T01:17:18Z INFO hashstore beginning compaction {“Process”: “storagenode”, “satellite”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “store”: “s1”, “stats”: {“NumLogs”:54,“LenLogs”:“38.1 GiB”,“NumLogsTTL”:17,“LenLogsTTL”:“1.6 GiB”,“SetPercent”:0.8026068769363741,“TrashPercent”:0.3011965642053018,“TTLPercent”:0.013412556138157618,“Compacting”:false,“Compactions”:0,“Today”:20405,“LastCompact”:0,“LogsRewritten”:0,“DataRewritten”:“0 B”,“DataReclaimed”:“0 B”,“DataReclaimable”:“7.5 GiB”,“Table”:{“NumSet”:82766,“LenSet”:“30.6 GiB”,“AvgSet”:396381.35620907135,“NumTrash”:27091,“LenTrash”:“11.5 GiB”,“AvgTrash”:454451.23354619613,“NumTTL”:1848,“LenTTL”:“522.8 MiB”,“AvgTTL”:296668.81385281385,“NumSlots”:262144,“TableSize”:“16.0 MiB”,“Load”:0.31572723388671875,“Created”:20404,“Kind”:0},“Compaction”:{“Elapsed”:0,“Remaining”:0,“TotalRecords”:0,“ProcessedRecords”:0}}}
2025-11-13T01:17:18Z INFO hashstore compact once started {“Process”: “storagenode”, “satellite”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “store”: “s1”, “today”: 20405}
2025-11-13T01:17:20Z INFO hashstore compaction computed details {“Process”: “storagenode”, “satellite”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “store”: “s1”, “nset”: 82468, “nexist”: 82766, “modifications”: true, “curr logSlots”: 18, “curr logSlots size”: “16.0 MiB”, “next logSlots”: 18, “next logSlots size”: “16.0 MiB”, “candidates”: [86], “rewrite”: [86], “duration”: “2.704992471s”}
2025-11-13T01:17:21Z INFO hashstore records rewritten {“Process”: “storagenode”, “satellite”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “store”: “s1”, “records”: 0, “bytes”: “0 B”, “duration”: “29.772403ms”}
2025-11-13T01:17:25Z INFO hashstore hashtbl rewritten {“Process”: “storagenode”, “satellite”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “store”: “s1”, “duration”: “4.700941115s”, “total records”: 82468, “total bytes”: “30.5 GiB”, “rewritten records”: 0, “rewritten bytes”: “0 B”, “trashed records”: 0, “trashed bytes”: “0 B”, “restored records”: 0, “restored bytes”: “0 B”, “expired records”: 298, “expired bytes”: “67.2 MiB”, “reclaimed logs”: 1, “reclaimed bytes”: “70.6 MiB”, “reclaim ratio”: “+Inf”}
2025-11-13T01:17:25Z INFO hashstore compact once finished {“Process”: “storagenode”, “satellite”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “store”: “s1”, “duration”: “7.652718782s”, “completed”: true}
2025-11-13T01:17:25Z INFO hashstore finished compaction {“Process”: “storagenode”, “satellite”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “store”: “s1”, “duration”: “7.653234941s”, “stats”: {“NumLogs”:53,“LenLogs”:“38.0 GiB”,“NumLogsTTL”:16,“LenLogsTTL”:“1.5 GiB”,“SetPercent”:0.8023346626664586,“TrashPercent”:0.30089137637921753,“TTLPercent”:0.012560598559642866,“Compacting”:false,“Compactions”:0,“Today”:20405,“LastCompact”:20405,“LogsRewritten”:1,“DataRewritten”:“0 B”,“DataReclaimed”:“70.6 MiB”,“DataReclaimable”:“7.5 GiB”,“Table”:{“NumSet”:82468,“LenSet”:“30.5 GiB”,“AvgSet”:396958.7288160256,“NumTrash”:26946,“LenTrash”:“11.4 GiB”,“AvgTrash”:455607.3260595265,“NumTTL”:1695,“LenTTL”:“488.7 MiB”,“AvgTTL”:302354.1238938053,“NumSlots”:262144,“TableSize”:“16.0 MiB”,“Load”:0.3145904541015625,“Created”:20405,“Kind”:0},“Compaction”:{“Elapsed”:0,“Remaining”:0,“TotalRecords”:0,“ProcessedRecords”:0}}}
2025-11-13T01:17:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false, “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false}, “interval”: “10m0s”}
2025-11-13T01:27:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false}, “interval”: “10m0s”}
2025-11-13T01:32:14Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2025-11-13T01:32:15Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.141.2”}
2025-11-13T01:32:15Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-11-13T01:32:15Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.141.2”}
2025-11-13T01:32:15Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2025-11-13T01:37:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false, “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false}, “interval”: “10m0s”}
2025-11-13T01:47:14Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2025-11-13T01:47:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.141.2”}
2025-11-13T01:47:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-11-13T01:47:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.141.2”}
2025-11-13T01:47:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2025-11-13T01:47:27Z INFO bandwidth Persisting bandwidth usage cache to db {“Process”: “storagenode”}
2025-11-13T01:47:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false}, “interval”: “10m0s”}
2025-11-13T01:47:36Z INFO orders finished {“Process”: “storagenode”, “satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “count”: 2}
2025-11-13T01:47:36Z INFO orders finished {“Process”: “storagenode”, “satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “count”: 359}
2025-11-13T01:47:37Z INFO orders finished {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “count”: 6289}
2025-11-13T01:47:41Z INFO orders finished {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “count”: 25114}
2025-11-13T01:57:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false, “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false}, “interval”: “10m0s”}
2025-11-13T02:02:14Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2025-11-13T02:02:15Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.141.2”}
2025-11-13T02:02:15Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-11-13T02:02:15Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.141.2”}
2025-11-13T02:02:15Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2025-11-13T02:07:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false, “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false}, “interval”: “10m0s”}
2025-11-13T02:17:14Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2025-11-13T02:17:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.141.2”}
2025-11-13T02:17:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-11-13T02:17:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.141.2”}
2025-11-13T02:17:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2025-11-13T02:17:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false, “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false}, “interval”: “10m0s”}
2025-11-13T02:27:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false}, “interval”: “10m0s”}
2025-11-13T02:32:14Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2025-11-13T02:32:15Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.141.2”}
2025-11-13T02:32:15Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-11-13T02:32:16Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.141.2”}
2025-11-13T02:32:16Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2025-11-13T02:37:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false}, “interval”: “10m0s”}
2025-11-13T02:47:14Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2025-11-13T02:47:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.141.2”}
2025-11-13T02:47:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-11-13T02:47:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.141.2”}
2025-11-13T02:47:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2025-11-13T02:47:27Z INFO bandwidth Persisting bandwidth usage cache to db {“Process”: “storagenode”}
2025-11-13T02:47:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false}, “interval”: “10m0s”}
2025-11-13T02:47:59Z INFO orders finished {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “count”: 2601}
2025-11-13T02:47:59Z INFO orders finished {“Process”: “storagenode”, “satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “count”: 16}
2025-11-13T02:47:59Z INFO orders finished {“Process”: “storagenode”, “satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “count”: 127}
2025-11-13T02:48:04Z INFO orders finished {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “count”: 24825}
2025-11-13T02:57:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false, “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false}, “interval”: “10m0s”}
2025-11-13T02:58:37Z INFO retain Retain job queued {“Process”: “storagenode”, “cachePath”: “config/retain”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2025-11-13T02:58:37Z INFO retain Prepared to run a Retain request. {“Process”: “storagenode”, “cachePath”: “config/retain”, “Created Before”: “2025-11-09T14:56:33Z”, “Filter Size”: 1088017, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2025-11-13T02:58:37Z INFO lazyfilewalker.gc-filewalker starting subprocess {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2025-11-13T02:58:37Z INFO lazyfilewalker.gc-filewalker subprocess started {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2025-11-13T02:58:39Z INFO lazyfilewalker.gc-filewalker.subprocess Database started {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Process”: “storagenode”}
2025-11-13T02:58:39Z INFO lazyfilewalker.gc-filewalker.subprocess gc-filewalker started {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Process”: “storagenode”, “createdBefore”: “2025-11-09T14:56:33Z”, “bloomFilterSize”: 1088017}
2025-11-13T03:02:14Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2025-11-13T03:02:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.141.2”}
2025-11-13T03:02:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-11-13T03:02:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.141.2”}
2025-11-13T03:02:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2025-11-13T03:07:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false}, “interval”: “10m0s”}
2025-11-13T03:17:14Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2025-11-13T03:17:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.141.2”}
2025-11-13T03:17:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-11-13T03:17:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.141.2”}
2025-11-13T03:17:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2025-11-13T03:17:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false}, “interval”: “10m0s”}
2025-11-13T03:27:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false}, “interval”: “10m0s”}
2025-11-13T03:32:14Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2025-11-13T03:32:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.141.2”}
2025-11-13T03:32:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-11-13T03:32:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.141.2”}
2025-11-13T03:32:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2025-11-13T03:37:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false, “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false}, “interval”: “10m0s”}
2025-11-13T03:47:14Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2025-11-13T03:47:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.141.2”}
2025-11-13T03:47:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-11-13T03:47:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.141.2”}
2025-11-13T03:47:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2025-11-13T03:47:27Z INFO bandwidth Persisting bandwidth usage cache to db {“Process”: “storagenode”}
2025-11-13T03:47:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false}, “interval”: “10m0s”}
2025-11-13T03:47:46Z INFO orders finished {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “count”: 1767}
2025-11-13T03:47:46Z INFO orders finished {“Process”: “storagenode”, “satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “count”: 70}
2025-11-13T03:47:47Z INFO orders finished {“Process”: “storagenode”, “satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “count”: 156}
2025-11-13T03:47:52Z INFO orders finished {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “count”: 23190}
2025-11-13T03:50:37Z INFO reputation:service node scores updated {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Total Audits”: 30178, “Successful Audits”: 27270, “Audit Score”: 1, “Online Score”: 0.8594203965129129, “Suspension Score”: 1, “Audit Score Delta”: 0, “Online Score Delta”: 0, “Suspension Score Delta”: 0}
2025-11-13T03:50:37Z INFO reputation:service node scores updated {“Process”: “storagenode”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Total Audits”: 77008, “Successful Audits”: 75520, “Audit Score”: 0.9995546539117, “Online Score”: 0.8586614813626308, “Suspension Score”: 1, “Audit Score Delta”: 0.00007574781323371749, “Online Score Delta”: 0, “Suspension Score Delta”: 0}
2025-11-13T03:50:38Z WARN reputation:service node scores worsened {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Total Audits”: 980475, “Successful Audits”: 955443, “Audit Score”: 1, “Online Score”: 0.9055078119241236, “Suspension Score”: 1, “Audit Score Delta”: 0, “Online Score Delta”: -0.00003532299939368322, “Suspension Score Delta”: 0}
2025-11-13T03:50:38Z INFO reputation:service node scores updated {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Total Audits”: 134744, “Successful Audits”: 129228, “Audit Score”: 1, “Online Score”: 0.8876684916474233, “Suspension Score”: 1, “Audit Score Delta”: 0, “Online Score Delta”: 0, “Suspension Score Delta”: 0}
2025-11-13T03:57:27Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false, “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false}, “interval”: “10m0s”}
2025-11-13T03:58:06Z INFO lazyfilewalker.gc-filewalker.subprocess gc-filewalker completed {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Pieces Skipped”: 0, “Process”: “storagenode”, “piecesCount”: 1658984, “Total Pieces To Trash”: 14275, “Trashed Pieces”: 14275}
2025-11-13T03:59:03Z INFO lazyfilewalker.gc-filewalker subprocess finished successfully {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2025-11-13T03:59:03Z INFO retain Moved pieces to trash during retain {“Process”: “storagenode”, “cachePath”: “config/retain”, “Deleted pieces”: 14275, “Failed to delete”: 0, “Pieces failed to read”: 0, “Pieces count”: 1658984, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Duration”: “1h0m25.72595004s”, “Retain Status”: “enabled”}
2025-11-13T04:02:14Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2025-11-13T04:02:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.141.2”}
2025-11-13T04:02:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-11-13T04:02:14Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.141.2”}
2025-11-13T04:02:14Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}

2

Open the dashboard and see your scores. If you drop below 60% online time, entry stops. This is normal. It will resume when your online time is above 60%. From the logs, it seems like the node is working.

But from the logs it seems that your online score is good

1 Like
3

for me it look like it try to update all the time to latest 1.141.2 version.

4

Its written you are up to date.

if your software is older than …
→ New version is being rolled out but hasn’t made it to this node yet

Just wait. You were offline so satellite wait for better online time… Something like that.

1 Like
5

then why it written so many times that it try do download new version?

1 Like
6

klick the link and you see.
Its normal with mine too. But it will just see if there is a new version.

You were offline for three Days!!!

2025-11-15T07:43:59Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2025-11-15T07:43:59Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.139.6"}
2025-11-15T07:43:59Z    INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode"}
2025-11-15T07:43:59Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.139.6"}
2025-11-15T07:43:59Z    INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode-updater"}
7

It doesn’t try to download a new version. It downloads the JSON file with versions :slight_smile:

1 Like
8

Yes I know. The score never got worse than 80%. from memory it was at least 85%

9

Do you have a free space?

10

No

Screenshot 2025-11-16 124820
Screenshot 2025-11-16 124820836×443 35.7 KB

11

I don’t get it. I had over 200Gb free space and then all of a sudden the trash kept piling up, going well above the total HDD space. The enormous trash is now sitting there since more than 1 week.

12

Does the physical drive have free space?

You could try:

Stop Node

Delete “used_space_per_prefix.db”

Start Node

Won’t hurt anything, may fix problem. That’s a lot of Trash.

1 Like
13

I have tried this. Now I am at the stage where there starts to be free space (or storj’s accounting terms). Physically there was always about 12% of space left.
Let’s see if anything changes now that storj sees some free space (although it still thinks that it’s overused).
For the moment there is still 0 ingress whatsoever.

14

What does it show now?

Did removing used_space db do anything?

15

image
image839×448 22.5 KB

Still 0 ingress.

16

Are you on hashstore? If so, check the folders in trash; they should be empty. Have you tried with the filewalker enabled? After I finished switching from piecestore to hashstore, I had to enable the filewalker once to let the node know there was nothing left.

1.9 Tb of trash is too much

17

Those terms don’t mean anything to me.
I am on version 1.141.2 currently.

18

What is the trash size on the disk?

du --si -d1 /mnt/storj/storagenode/storage

It could be big, if the node was offline too long, in that case you need to wait until it will expire and will be removed from the disk.
Accordingly the stat from the dashboard - you have an overusage and 5GB left, so the node will not accept any ingress, because the allocated already used fully with 43.5GB on top.

19

In the end ot was a matter of time. The trash took a lot time to be deleted but as soon as storj recognized that there was enough free space it restarted ingress.

2 Likes