Available disk space shows incorrect values after Docker restart (even on a small 19 GB node)

[dzeto]

Hi,

I’m struggling to fix the “used/available” statistics for my nodes in the dashboard. I noticed that every time I restart a node, the statistics start showing incorrect values.

I tested this on a clean node, as follows:

1. Node filled up to 19 GB
2. I stopped/started the Docker container
3. Statistics for used space are now wrong (it shows only 5 GB instead of 19 GB)
4. Running du --si on the node drive shows 19 GB of used space

What I tried:
I started with "storage2.piece-scan-on-startup: true", but the results are the same. It still shows only 5 GB of used space after one hour.

Any suggestions on what I should do are welcome.

Thanks!

[dzeto]

Below you can see debug logs for this node:

DEBUG LOGS

2025-09-05 18:44:40,391 INFO Set uid to user 0 succeeded
2025-09-05 18:44:40,393 INFO RPC interface ‘supervisor’ initialized
2025-09-05 18:44:40,393 INFO supervisord started with pid 1
2025-09-05 18:44:41,395 INFO spawned: ‘processes-exit-eventlistener’ with pid 58
2025-09-05 18:44:41,397 INFO spawned: ‘storagenode’ with pid 59
2025-09-05 18:44:41,398 INFO spawned: ‘storagenode-updater’ with pid 60
2025-09-05T18:44:41Z INFO Configuration loaded {“Process”: “storagenode-updater”, “Location”: “/app/config/config.yaml”}
2025-09-05T18:44:41Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “healthcheck.details”}
2025-09-05T18:44:41Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “healthcheck.enabled”}
2025-09-05T18:44:41Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage2.piece-scan-on-startup”}
2025-09-05T18:44:41Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.email”}
2025-09-05T18:44:41Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “console.address”}
2025-09-05T18:44:41Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “contact.external-address”}
2025-09-05T18:44:41Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-disk-space”}
2025-09-05T18:44:41Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet”}
2025-09-05T18:44:41Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet-features”}
2025-09-05T18:44:41Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.private-address”}
2025-09-05T18:44:41Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.address”}
2025-09-05T18:44:41Z INFO Anonymized tracing enabled {“Process”: “storagenode-updater”}
2025-09-05T18:44:41Z INFO Running on version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.135.5”}
2025-09-05T18:44:41Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2025-09-05T18:44:41Z INFO Configuration loaded {“Process”: “storagenode”, “Location”: “/app/config/config.yaml”}
2025-09-05T18:44:41Z INFO Anonymized tracing enabled {“Process”: “storagenode”}
2025-09-05T18:44:41Z DEBUG tracing collector started {“Process”: “storagenode”}
2025-09-05T18:44:41Z INFO Operator email {“Process”: “storagenode”, “Address”: “my-email”}
2025-09-05T18:44:41Z INFO Operator wallet {“Process”: “storagenode”, “Address”: “my-wallet”}
2025-09-05T18:44:41Z DEBUG Version info {“Process”: “storagenode”, “Version”: “1.135.5”, “Commit Hash”: “131aa8e0aaefe4c82bf583c562b040fdfceebb35”, “Build Timestamp”: “2025-08-12 17:15:50 +0000 UTC”, “Release Build”: true}
2025-09-05T18:44:41Z INFO server kernel support for server-side tcp fast open remains disabled. {“Process”: “storagenode”}
2025-09-05T18:44:41Z INFO server enable with: sysctl -w net.ipv4.tcp_fastopen=3 {“Process”: “storagenode”}
2025-09-05T18:44:41Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.135.5”}
2025-09-05T18:44:41Z INFO New version is being rolled out but hasn’t made it to this node yet{“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-09-05T18:44:41Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.135.5”}
2025-09-05T18:44:41Z INFO New version is being rolled out but hasn’t made it to this node yet{“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2025-09-05T18:44:42Z DEBUG version Allowed minimum version from control server. {“Process”: “storagenode”, “Minimum Version”: “1.135.5”}
2025-09-05T18:44:42Z DEBUG version Running on allowed version. {“Process”: “storagenode”, “Version”: “1.135.5”}
2025-09-05T18:44:42Z INFO Telemetry enabled {“Process”: “storagenode”, “instance ID”: “12SCav961HUFjbFzMcsv3zuDMPwuddA59HgVC86sWjeqX7GmoFg”}
2025-09-05T18:44:42Z INFO Event collection enabled {“Process”: “storagenode”, “instance ID”: “12SCav961HUFjbFzMcsv3zuDMPwuddA59HgVC86sWjeqX7GmoFg”}
2025-09-05T18:44:42Z INFO db.migration Database Version {“Process”: “storagenode”, “version”: 62}
2025-09-05T18:44:42Z DEBUG db Database version is up to date {“Process”: “storagenode”, “version”: 62}
2025-09-05T18:44:42Z DEBUG trust Fetched URLs from source; updating cache {“Process”: “storagenode”, “source”: “https://static.storj.io/dcs-satellites”, “count”: 4}
2025-09-05 18:44:42,996 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-05 18:44:42,996 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-05 18:44:42,996 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-05T18:44:42Z DEBUG trust Satellite is trusted {“Process”: “storagenode”, “id”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2025-09-05T18:44:42Z DEBUG trust Satellite is trusted {“Process”: “storagenode”, “id”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2025-09-05T18:44:42Z DEBUG trust Satellite is trusted {“Process”: “storagenode”, “id”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2025-09-05T18:44:42Z DEBUG trust Satellite is trusted {“Process”: “storagenode”, “id”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
2025-09-05T18:44:42Z INFO preflight:localtime start checking local system clock with trusted satellites’ system clock. {“Process”: “storagenode”}
2025-09-05T18:44:43Z INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock. {“Process”: “storagenode”}
2025-09-05T18:44:43Z DEBUG servers started {“Process”: “storagenode”, “items”: [“server”, “debug”]}
2025-09-05T18:44:43Z INFO Node 12SCav961HUFjbFzMcsv3zuDMPwuddA59HgVC86sWjeqX7GmoFg started {“Process”: “storagenode”}
2025-09-05T18:44:43Z INFO Public server started on [::]:57610 {“Process”: “storagenode”}
2025-09-05T18:44:43Z INFO Private server started on 127.0.0.1:7778 {“Process”: “storagenode”}
2025-09-05T18:44:43Z DEBUG services started {“Process”: “storagenode”, “items”: [“version:chore”, “trust”, “contact:chore”, “bandwidth”, “pieces:trash”, “hashstore”, “piecestore:cache”, “piecestore:monitor”, “retain”, “ordersfilestore”, “piecemigrate:chore”, “orders”, “reputation:chore”, “nodestats:cache”, “console:endpoint”, “gracefulexit:blobscleaner”, “gracefulexit:chore”, “forgetsatellite:chore”, “collector”]}
2025-09-05T18:44:43Z INFO bandwidth Persisting bandwidth usage cache to db {“Process”: “storagenode”}
2025-09-05T18:44:43Z INFO failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See UDP Buffer Sizes · quic-go/quic-go Wiki · GitHub for details. {“Process”: “storagenode”}
2025-09-05T18:44:43Z DEBUG pieces:trash starting to empty trash {“Process”: “storagenode”}
2025-09-05T18:44:43Z INFO trust Scheduling next refresh {“Process”: “storagenode”, “after”: “3h47m48.377674646s”}
2025-09-05T18:44:43Z INFO piecemigrate:chore all enqueued for migration; will sleep before next pooling {“Process”: “storagenode”, “active”: {“12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”: false, “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”: false, “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”: false, “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”: false}, “interval”: “10m0s”}
2025-09-05T18:44:43Z INFO collector expired pieces collection started {“Process”: “storagenode”}
2025-09-05T18:44:43Z DEBUG piecestore:monitor readability check done {“Process”: “storagenode”, “Duration”: “108.83µs”}
2025-09-05T18:44:43Z INFO pieces:trash emptying trash started {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2025-09-05T18:44:43Z DEBUG piecestore:monitor writability check done {“Process”: “storagenode”, “Duration”: “184.781µs”}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker starting subprocess {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2025-09-05T18:44:43Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2025-09-05T18:44:43Z INFO lazyfilewalker.used-space-filewalker starting subprocess {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2025-09-05T18:44:43Z DEBUG contact:chore Starting cycle {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
2025-09-05T18:44:43Z DEBUG contact:chore Starting cycle {“Process”: “storagenode”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2025-09-05T18:44:43Z DEBUG contact:chore Starting cycle {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2025-09-05T18:44:43Z DEBUG contact:chore Starting cycle {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess started {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2025-09-05T18:44:43Z INFO lazyfilewalker.used-space-filewalker subprocess started {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2025-09-05T18:44:43Z INFO collector expired pieces collection completed {“Process”: “storagenode”, “count”: 0}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker started {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Process”: “storagenode”, “dateBefore”: “2025-08-29T18:44:43Z”}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess Database started {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Process”: “storagenode”}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker completed {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Process”: “storagenode”, “bytesDeleted”: 0, “numKeysDeleted”: 0}
2025-09-05T18:44:43Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Process”: “storagenode”}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2025-09-05T18:44:43Z INFO pieces:trash emptying trash finished {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “elapsed”: “40.107385ms”}
2025-09-05T18:44:43Z INFO pieces:trash emptying trash started {“Process”: “storagenode”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker starting subprocess {“Process”: “storagenode”, “satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess started {“Process”: “storagenode”, “satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker started {“Process”: “storagenode”, “satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “dateBefore”: “2025-08-29T18:44:43Z”, “Process”: “storagenode”}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess Database started {“Process”: “storagenode”, “satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Process”: “storagenode”}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker completed {“Process”: “storagenode”, “satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Process”: “storagenode”, “bytesDeleted”: 0, “numKeysDeleted”: 0}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully {“Process”: “storagenode”, “satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2025-09-05T18:44:43Z INFO pieces:trash emptying trash finished {“Process”: “storagenode”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “elapsed”: “35.726813ms”}
2025-09-05T18:44:43Z INFO pieces:trash emptying trash started {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker starting subprocess {“Process”: “storagenode”, “satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess started {“Process”: “storagenode”, “satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker started {“Process”: “storagenode”, “satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Process”: “storagenode”, “dateBefore”: “2025-08-29T18:44:43Z”}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess Database started {“Process”: “storagenode”, “satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Process”: “storagenode”}
2025-09-05T18:44:43Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker completed {“Process”: “storagenode”, “satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Process”: “storagenode”, “bytesDeleted”: 0, “numKeysDeleted”: 0}
2025-09-05T18:44:44Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully {“Process”: “storagenode”, “satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
2025-09-05T18:44:44Z INFO pieces:trash emptying trash finished {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “elapsed”: “41.300087ms”}
2025-09-05T18:44:44Z INFO pieces:trash emptying trash started {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2025-09-05T18:44:44Z INFO lazyfilewalker.trash-cleanup-filewalker starting subprocess {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2025-09-05T18:44:44Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess started {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2025-09-05T18:44:44Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2025-09-05T18:44:44Z INFO pieces used-space-filewalker completed {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Lazy File Walker”: true, “Total Pieces Size”: 1113917480, “Total Pieces Content Size”: 1110982184, “Total Pieces Count”: 5733, “Duration”: “120.349493ms”}
2025-09-05T18:44:44Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2025-09-05T18:44:44Z INFO lazyfilewalker.used-space-filewalker starting subprocess {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2025-09-05T18:44:44Z INFO lazyfilewalker.used-space-filewalker subprocess started {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2025-09-05T18:44:44Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker started {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Process”: “storagenode”, “dateBefore”: “2025-08-29T18:44:44Z”}
2025-09-05T18:44:44Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess Database started {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Process”: “storagenode”}
2025-09-05T18:44:44Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker completed {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Process”: “storagenode”, “bytesDeleted”: 0, “numKeysDeleted”: 0}
2025-09-05T18:44:44Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Process”: “storagenode”}
2025-09-05T18:44:44Z DEBUG version Allowed minimum version from control server. {“Process”: “storagenode”, “Minimum Version”: “1.135.5”}
2025-09-05T18:44:44Z DEBUG version Running on allowed version. {“Process”: “storagenode”, “Version”: “1.135.5”}
2025-09-05T18:44:44Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2025-09-05T18:44:44Z INFO pieces:trash emptying trash finished {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “elapsed”: “36.809325ms”}
2025-09-05T18:44:44Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“Process”: “storagenode”, “satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2025-09-05T18:44:44Z INFO pieces used-space-filewalker completed {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Lazy File Walker”: true, “Total Pieces Size”: 2234131968, “Total Pieces Content Size”: 2228816384, “Total Pieces Count”: 10382, “Duration”: “58.334051ms”}
2025-09-05T18:44:44Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2025-09-05T18:44:44Z INFO lazyfilewalker.used-space-filewalker starting subprocess {“Process”: “storagenode”, “satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2025-09-05T18:44:44Z INFO lazyfilewalker.used-space-filewalker subprocess started {“Process”: “storagenode”, “satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2025-09-05T18:44:44Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {“Process”: “storagenode”, “satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Process”: “storagenode”}
2025-09-05T18:44:44Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“Process”: “storagenode”, “satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2025-09-05T18:44:44Z INFO pieces used-space-filewalker completed {“Process”: “storagenode”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Lazy File Walker”: true, “Total Pieces Size”: 1632803328, “Total Pieces Content Size”: 1629433344, “Total Pieces Count”: 6582, “Duration”: “69.670781ms”}
2025-09-05T18:44:44Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
2025-09-05T18:44:44Z INFO lazyfilewalker.used-space-filewalker starting subprocess {“Process”: “storagenode”, “satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
2025-09-05T18:44:44Z INFO lazyfilewalker.used-space-filewalker subprocess started {“Process”: “storagenode”, “satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
2025-09-05T18:44:44Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {“Process”: “storagenode”, “satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Process”: “storagenode”}
2025-09-05T18:44:44Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“Process”: “storagenode”, “satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
2025-09-05T18:44:44Z INFO pieces used-space-filewalker completed {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Lazy File Walker”: true, “Total Pieces Size”: 39454464, “Total Pieces Content Size”: 39359744, “Total Pieces Count”: 185, “Duration”: “40.871066ms”}
2025-09-05T18:44:44Z DEBUG contact:endpoint pinged {“Process”: “storagenode”, “by”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “srcAddr”: “35.207.125.74:58836”}
2025-09-05T18:44:44Z DEBUG contact:endpoint pinged {“Process”: “storagenode”, “by”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “srcAddr”: “35.207.125.74:46200”}
2025-09-05T18:44:44Z DEBUG contact:endpoint pinged {“Process”: “storagenode”, “by”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “srcAddr”: “35.212.53.193:29534”}
2025-09-05T18:44:45Z DEBUG contact:endpoint pinged {“Process”: “storagenode”, “by”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “srcAddr”: “35.212.53.193:14729”}
2025-09-05T18:44:45Z DEBUG contact:endpoint pinged {“Process”: “storagenode”, “by”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “srcAddr”: “35.215.103.1:22719”}
2025-09-05T18:44:45Z DEBUG contact:endpoint pinged {“Process”: “storagenode”, “by”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “srcAddr”: “35.215.103.1:13578”}
2025-09-05T18:44:46Z DEBUG contact:endpoint pinged {“Process”: “storagenode”, “by”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “srcAddr”: “34.2.148.210:8474”}
2025-09-05T18:44:46Z DEBUG contact:endpoint pinged {“Process”: “storagenode”, “by”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “srcAddr”: “34.2.148.210:59660”}
2025-09-05T18:44:47Z DEBUG piecestore upload started {“Process”: “storagenode”, “Piece ID”: “PLYJJINZQ3V24ZSWRGYN2M2IV6UVP3LWCUSQ7L2SHW2V5T2DCVUA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT_REPAIR”, “Remote Address”: “91.98.141.238:26593”, “Available Space”: 494979692760}
2025-09-05T18:44:47Z INFO piecestore uploaded {“Process”: “storagenode”, “Piece ID”: “PLYJJINZQ3V24ZSWRGYN2M2IV6UVP3LWCUSQ7L2SHW2V5T2DCVUA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT_REPAIR”, “Remote Address”: “91.98.141.238:26593”, “Size”: 11520}
2025-09-05T18:44:49Z DEBUG piecestore upload started {“Process”: “storagenode”, “Piece ID”: “OEPD5MCXU7VKDSEFD3WCTIYCXL3PWRKV3BZ6SRAKKLMKROWXTPZA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT_REPAIR”, “Remote Address”: “91.98.141.238:44940”, “Available Space”: 494979680728}

My start command looks like this:
docker run -d --restart unless-stopped --stop-timeout 300 --network=container:line-2
-e WALLET=“(hidden)”
-e EMAIL=“(hidden)”
-e ADDRESS=“(hidden)”
-e STORAGE=“500GB”
–user $(id -u):$(id -g)
–mount type=bind,source=“/root/STORJ/HDD-12/node-12/identity/storagenode/”,destination=/app/identity
–mount type=bind,source=“/root/STORJ/HDD-12/node-12/”,destination=/app/config
–name node-12 storjlabs/storagenode:latest --operator.wallet-features=zksync-era

Thank you!

[dzeto]

My current workaround is to remove this file, which ensures that the usage stats are displayed correctly on startup:

rm /root/STORJ/HDD-12/node-12/storage/used_space_per_prefix.db

Is there any better / more clean approach?
Thanks!

[Walter1]

Did you do the migration towards hashstore previously? Unfortuantely as of now there seems no cleaner approach. Just rerunning the lazy-filewalker also doesn’t solve it, but the prefix-db removal plus lazy-filewalker and therefore a new prefix-db creation.

[Alexey]

Hello @dzeto,
Welcome to the forum!

It may fix itself after a week, or you may brute speedup it like this:

High Trash usage - 27 TB - no more uploads

Only wait until all used-space-filewalkers will finish the scan since the last restart. Because right now you logs shows only several starts of the used-space-filewalkers but only a few completed. I have a suspicion that we might introduced a bug in used-space-filewalker, when we tried to implement a continuation of the used-space-filewalker. You can test that: Stop and remove the container (or stop the service, if you use a service based setup) Rename the used_space_per_prefix.db database …

[alpharabbit]

@Alexey

It may fix itself after a week

This is a 19 GB node, not TB.
Filewalker should need only minutes.

[Alexey]

I mean that this cache database prevents the filewalker runs very often, so it will not be modified until some time will pass. It’s not related to how long the filewalker may work
And this is why deleting of this database allows the filewalker to do its work immediately.

[arrogantrabbit]

THis is all fixing symptoms. The problm is with the setup: sqlite database corruptions happen when sqlite assumptions about storage are not satisfied – eitehr sync is not ensured, or locking does not work.

If this is on windows + docker – then all bets are off.

I strongly recommend running nodes directly in the host OS, this will avoid a lot of issues in the unnecessary bloat. Nodes don’t need nor benefit from containerization.

As a stop-gap measure you can keep databases inside the container, not on the external mount. This will ensure the locking consistency by staygin with the OS.

[Alexey]

Windows docker works well. At least I can assume by my nodes - I didn’t have problems since migrated to docker desktop with wsl2.
The only inconvenience, that it’s not a service, but a user application, so you need to login to make it work.
The method with a scheduled task unfortunately stopped to work from some version.
I also tried to use a Rancher Desktop, it worked well, including starting with a scheduled task, however, recently their virtual network become unstable (nodes started to go to offline randomly), so I returned back to the Docker Desktop and it worked stable since than.