2 nodes offline, Toolbox Windows

[eddy89]

There were problems with the DNS providers, which I was able to quickly resolve with a SUBSCRIPTION. After I saw misconfig on the nodes this morning, I noticed that the DNS is offline and only updates with a subscription. I restarted everything and 2 nodes don’t want to start, tried screenschotts.trobleschott and shows that everything is ok . what’s wrong?

Logs:

onfiguration loaded {“Location”: “C:\Program Files\Storj\Storage Node\config.yaml”}
2023-07-05T10:50:13.461+0200 INFO Invalid configuration file key {“Key”: “contact.external-address”}
2023-07-05T10:50:13.461+0200 INFO Invalid configuration file key {“Key”: “server.private-address”}
2023-07-05T10:50:13.461+0200 INFO Invalid configuration file key {“Key”: “operator.email”}
2023-07-05T10:50:13.461+0200 INFO Invalid configuration file key {“Key”: “storage.allocated-bandwidth”}
2023-07-05T10:50:13.461+0200 INFO Invalid configuration file key {“Key”: “server.address”}
2023-07-05T10:50:13.461+0200 INFO Invalid configuration file key {“Key”: “storage.allocated-disk-space”}
2023-07-05T10:50:13.461+0200 INFO Invalid configuration file key {“Key”: “operator.wallet”}
2023-07-05T10:50:13.461+0200 INFO Invalid configuration file key {“Key”: “healthcheck.details”}
2023-07-05T10:50:13.461+0200 INFO Invalid configuration file key {“Key”: “storage.path”}
2023-07-05T10:50:13.461+0200 INFO Invalid configuration file key {“Key”: “healthcheck.enabled”}
2023-07-05T10:50:13.461+0200 INFO Invalid configuration file key {“Key”: “operator.wallet-features”}
2023-07-05T10:50:13.461+0200 INFO Invalid configuration file value for key {“Key”: “log.caller”}
2023-07-05T10:50:13.462+0200 INFO Anonymized tracing enabled
2023-07-05T10:50:13.463+0200 INFO Running on version {“Service”: “storagenode-updater”, “Version”: “v1.80.10”}
2023-07-05T10:50:13.463+0200 INFO Downloading versions. {“Server Address”: “https://version.storj.io”}
2023-07-05T10:50:13.918+0200 INFO Current binary version {“Service”: “storagenode”, “Version”: “v1.80.10”}
2023-07-05T10:50:13.918+0200 INFO New version is being rolled out but hasn’t made it to this node yet {“Service”: “storagenode”}
2023-07-05T10:50:14.235+0200 INFO Current binary version {“Service”: “storagenode-updater”, “Version”: “v1.80.10”}
2023-07-05T10:50:14.235+0200 INFO New version is being rolled out but hasn’t made it to this node yet {“Service”: “storagenode-updater”}
2023-07-05T10:53:45.859+0200 INFO Got a signal from the OS: “terminated”
2023-07-05T10:55:28.899+0200 INFO Configuration loaded {“Location”: “C:\Program Files\Storj\Storage Node\config.yaml”}
2023-07-05T10:55:28.899+0200 INFO Invalid configuration file key {“Key”: “storage.allocated-bandwidth”}
2023-07-05T10:55:28.899+0200 INFO Invalid configuration file key {“Key”: “storage.path”}
2023-07-05T10:55:28.899+0200 INFO Invalid configuration file key {“Key”: “operator.wallet-features”}
2023-07-05T10:55:28.899+0200 INFO Invalid configuration file key {“Key”: “server.address”}
2023-07-05T10:55:28.899+0200 INFO Invalid configuration file key {“Key”: “contact.external-address”}
2023-07-05T10:55:28.899+0200 INFO Invalid configuration file key {“Key”: “server.private-address”}
2023-07-05T10:55:28.899+0200 INFO Invalid configuration file key {“Key”: “healthcheck.details”}
2023-07-05T10:55:28.899+0200 INFO Invalid configuration file key {“Key”: “storage.allocated-disk-space”}
2023-07-05T10:55:28.899+0200 INFO Invalid configuration file key {“Key”: “healthcheck.enabled”}
2023-07-05T10:55:28.899+0200 INFO Invalid configuration file key {“Key”: “operator.email”}
2023-07-05T10:55:28.899+0200 INFO Invalid configuration file key {“Key”: “operator.wallet”}
2023-07-05T10:55:28.899+0200 INFO Invalid configuration file value for key {“Key”: “log.output”}
2023-07-05T10:55:28.899+0200 INFO Invalid configuration file value for key {“Key”: “log.caller”}
2023-07-05T10:55:28.899+0200 INFO Invalid configuration file value for key {“Key”: “log.level”}
2023-07-05T10:55:28.900+0200 INFO Anonymized tracing enabled
2023-07-05T10:55:28.901+0200 INFO Running on version {“Service”: “storagenode-updater”, “Version”: “v1.80.10”}
2023-07-05T10:55:28.901+0200 INFO Downloading versions. {“Server Address”: “https://version.storj.io”}
2023-07-05T10:55:28.915+0200 ERROR Error retrieving version info. {“error”: “version checker client: Get "https://version.storj.io": dial tcp: lookup version.storj.io: no such host”, “errorVerbose”: “version checker client: Get "https://version.storj.io": dial tcp: lookup version.storj.io: no such host\n\tstorj.io/storj/private/version/checker.(*Client).All:68\n\tmain.loopFunc:25\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tmain.cmdRun:136\n\tstorj.io/private/process.cleanup.func1.4:399\n\tstorj.io/private/process.cleanup.func1:417\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:113\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:79\n\tstorj.io/private/process.ExecWithCustomConfig:74\n\tstorj.io/private/process.Exec:64\n\tmain.(*service).Execute.func1:56\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}
2023-07-05T11:10:28.906+0200 INFO Downloading versions. {“Server Address”: “https://version.storj.io”}
2023-07-05T11:10:29.377+0200 INFO Current binary version {“Service”: “storagenode”, “Version”: “v1.80.10”}
2023-07-05T11:10:29.377+0200 INFO New version is being rolled out but hasn’t made it to this node yet {“Service”: “storagenode”}
2023-07-05T11:10:29.753+0200 INFO Current binary version {“Service”: “storagenode-updater”, “Version”: “v1.80.10”}
2023-07-05T11:10:29.753+0200 INFO New version is being rolled out but hasn’t made it to this node yet {“Service”: “storagenode-updater”}
2023-07-05T11:25:28.903+0200 INFO Downloading versions. {“Server Address”: “https://version.storj.io”}
2023-07-05T11:25:29.382+0200 INFO Current binary version {“Service”: “storagenode”, “Version”: “v1.80.10”}
2023-07-05T11:25:29.382+0200 INFO New version is being rolled out but hasn’t made it to this node yet {“Service”: “storagenode”}
2023-07-05T11:25:29.438+0200 INFO Current binary version {“Service”: “storagenode-updater”, “Version”: “v1.80.10”}
2023-07-05T11:25:29.438+0200 INFO New version is being rolled out but hasn’t made it to this node yet {“Service”: “storagenode-updater”}
2023-07-05T11:40:28.913+0200 INFO Downloading versions. {“Server Address”: “https://version.storj.io”}
2023-07-05T11:40:29.373+0200 INFO Current binary version {“Service”: “storagenode”, “Version”: “v1.80.10”}
2023-07-05T11:40:29.373+0200 INFO New version is being rolled out but hasn’t made it to this node yet {“Service”: “storagenode”}
2023-07-05T11:40:29.427+0200 INFO Current binary version {“Service”: “storagenode-updater”, “Version”: “v1.80.10”}
2023-07-05T11:40:29.427+0200 INFO New version is being rolled out but hasn’t made it to this node yet {“Service”: “storagenode-updater”}
2023-07-05T11:48:12.753+0200 INFO Configuration loaded {“Location”: “C:\Program Files\Storj\Storage Node\config.yaml”}
2023-07-05T11:48:12.755+0200 INFO Invalid configuration file key {“Key”: “operator.email”}

[Vadim]

check logs from storage nodes that not work, there you will find errors why it not work

[eddy89]

read different errors like : “error”: “lazyfilewalker: context canceled” , “error”: “pieces error: filestore error: context canceled” , Unfortunately I can’t do anything with it

20 lines of the log:
2023-09-12T12:19:29+02:00 ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {“satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “error”: “context canceled”}
2023-09-12T12:19:29+02:00 ERROR pieces failed to lazywalk space used by satellite {“error”: “lazyfilewalker: context canceled”, “errorVerbose”: “lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:71\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:707\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2023-09-12T12:19:29+02:00 INFO lazyfilewalker.used-space-filewalker starting subprocess {“satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2023-09-12T12:19:29+02:00 ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {“satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “error”: “context canceled”}
2023-09-12T12:19:29+02:00 ERROR pieces failed to lazywalk space used by satellite {“error”: “lazyfilewalker: context canceled”, “errorVerbose”: “lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:71\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:707\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2023-09-12T12:19:29+02:00 INFO lazyfilewalker.used-space-filewalker starting subprocess {“satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2023-09-12T12:19:29+02:00 ERROR gracefulexit:blobscleaner couldn’t receive satellite’s GE status {“error”: “context canceled”}
2023-09-12T12:19:29+02:00 ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {“satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “error”: “context canceled”}
2023-09-12T12:19:29+02:00 ERROR pieces failed to lazywalk space used by satellite {“error”: “lazyfilewalker: context canceled”, “errorVerbose”: “lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:71\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:707\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2023-09-12T12:19:29+02:00 INFO lazyfilewalker.used-space-filewalker starting subprocess {“satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
2023-09-12T12:19:29+02:00 ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {“satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “error”: “context canceled”}
2023-09-12T12:19:29+02:00 ERROR pieces failed to lazywalk space used by satellite {“error”: “lazyfilewalker: context canceled”, “errorVerbose”: “lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:71\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:707\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
2023-09-12T12:19:29+02:00 INFO lazyfilewalker.used-space-filewalker starting subprocess {“satelliteID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”}
2023-09-12T12:19:29+02:00 ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {“satelliteID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “error”: “context canceled”}
2023-09-12T12:19:29+02:00 ERROR pieces failed to lazywalk space used by satellite {“error”: “lazyfilewalker: context canceled”, “errorVerbose”: “lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:71\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:707\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”}
2023-09-12T12:19:29+02:00 INFO lazyfilewalker.used-space-filewalker starting subprocess {“satelliteID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”}
2023-09-12T12:19:29+02:00 ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {“satelliteID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “error”: “context canceled”}
2023-09-12T12:19:29+02:00 ERROR pieces failed to lazywalk space used by satellite {“error”: “lazyfilewalker: context canceled”, “errorVerbose”: “lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:71\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:707\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”}
2023-09-12T12:19:29+02:00 ERROR piecestore:cache error getting current used space: {“error”: “filewalker: context canceled; filewalker: context canceled; filewalker: context canceled; filewalker: context canceled; filewalker: context canceled; filewalker: context canceled”, “errorVerbose”: “group:\n— filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n— filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n— filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n— filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n— filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n— filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}
2023-09-12T12:19:29+02:00 FATAL Unrecoverable error {“error”: “piecestore monitor: disk space requirement not met”, “errorVerbose”: “piecestore monitor: disk space requirement not met\n\tstorj.io/storj/storagenode/monitor.(*Service).Run:135\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}
2023-09-12T12:32:24+02:00 INFO Configuration loaded {“Location”: “C:\Program Files\Storj\Storage Node\config.yaml”}
2023-09-12T12:32:24+02:00 INFO Anonymized tracing enabled
2023-09-12T12:32:24+02:00 INFO Operator email {“Address”: “elmedin.metzingen3@outlook.de”}
2023-09-12T12:32:24+02:00 INFO Operator wallet {“Address”: “0x6Ed7D31d758d24dfFE6A467fd5700212C93DefA5”}
2023-09-12T12:32:37+02:00 INFO Telemetry enabled {“instance ID”: “13tBLtMkSxGwP2W7hdWHDLiMwG6nWtuHnA9rV7LjLn54j85XcE”}
2023-09-12T12:32:37+02:00 INFO Event collection enabled {“instance ID”: “13tBLtMkSxGwP2W7hdWHDLiMwG6nWtuHnA9rV7LjLn54j85XcE”}
2023-09-12T12:32:37+02:00 INFO db.migration Database Version {“version”: 54}
2023-09-12T12:32:37+02:00 INFO preflight:localtime start checking local system clock with trusted satellites’ system clock.
2023-09-12T12:32:38+02:00 INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.
2023-09-12T12:32:38+02:00 INFO bandwidth Performing bandwidth usage rollups
2023-09-12T12:32:38+02:00 INFO trust Scheduling next refresh {“after”: “5h45m23.477313539s”}
2023-09-12T12:32:38+02:00 INFO pieces:trash emptying trash started {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”}
2023-09-12T12:32:38+02:00 INFO Node 13tBLtMkSxGwP2W7hdWHDLiMwG6nWtuHnA9rV7LjLn54j85XcE started
2023-09-12T12:32:38+02:00 INFO Public server started on [::]:28967
2023-09-12T12:32:38+02:00 INFO Private server started on 127.0.0.1:7778
2023-09-12T12:32:38+02:00 WARN piecestore:monitor Disk space is less than requested. Allocated space is {“bytes”: 283266360320}
2023-09-12T12:32:38+02:00 ERROR piecestore:monitor Total disk space is less than required minimum {“bytes”: 500000000000}
2023-09-12T12:32:38+02:00 ERROR services unexpected shutdown of a runner {“name”: “piecestore:monitor”, “error”: “piecestore monitor: disk space requirement not met”, “errorVerbose”: “piecestore monitor: disk space requirement not met\n\tstorj.io/storj/storagenode/monitor.(*Service).Run:135\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}
2023-09-12T12:32:38+02:00 ERROR nodestats:cache Get pricing-model/join date failed {“error”: “context canceled”}
2023-09-12T12:32:38+02:00 INFO lazyfilewalker.used-space-filewalker starting subprocess {“satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2023-09-12T12:32:38+02:00 ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {“satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “error”: “context canceled”}
2023-09-12T12:32:38+02:00 ERROR pieces failed to lazywalk space used by satellite {“error”: “lazyfilewalker: context canceled”, “errorVerbose”: “lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:71\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:707\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2023-09-12T12:32:38+02:00 INFO lazyfilewalker.used-space-filewalker starting subprocess {“satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2023-09-12T12:32:38+02:00 ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {“satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “error”: “context canceled”}
2023-09-12T12:32:38+02:00 ERROR pieces failed to lazywalk space used by satellite {“error”: “lazyfilewalker: context canceled”, “errorVerbose”: “lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:71\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:707\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2023-09-12T12:32:38+02:00 INFO lazyfilewalker.used-space-filewalker starting subprocess {“satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2023-09-12T12:32:38+02:00 ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {“satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “error”: “context canceled”}
2023-09-12T12:32:38+02:00 ERROR pieces failed to lazywalk space used by satellite {“error”: “lazyfilewalker: context canceled”, “errorVerbose”: “lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:71\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:707\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2023-09-12T12:32:38+02:00 INFO lazyfilewalker.used-space-filewalker starting subprocess {“satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
2023-09-12T12:32:38+02:00 ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {“satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “error”: “context canceled”}
2023-09-12T12:32:38+02:00 ERROR pieces failed to lazywalk space used by satellite {“error”: “lazyfilewalker: context canceled”, “errorVerbose”: “lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:71\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:707\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
2023-09-12T12:32:38+02:00 INFO lazyfilewalker.used-space-filewalker starting subprocess {“satelliteID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”}
2023-09-12T12:32:38+02:00 ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {“satelliteID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “error”: “context canceled”}
2023-09-12T12:32:38+02:00 ERROR pieces failed to lazywalk space used by satellite {“error”: “lazyfilewalker: context canceled”, “errorVerbose”: “lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:71\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:707\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”}
2023-09-12T12:32:38+02:00 INFO lazyfilewalker.used-space-filewalker starting subprocess {“satelliteID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”}
2023-09-12T12:32:38+02:00 ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {“satelliteID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “error”: “context canceled”}
2023-09-12T12:32:38+02:00 ERROR pieces failed to lazywalk space used by satellite {“error”: “lazyfilewalker: context canceled”, “errorVerbose”: “lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:71\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:707\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”}
2023-09-12T12:32:38+02:00 ERROR piecestore:cache error getting current used space: {“error”: “filewalker: context canceled; filewalker: context canceled; filewalker: context canceled; filewalker: context canceled; filewalker: context canceled; filewalker: context canceled”, “errorVerbose”: “group:\n— filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n— filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n— filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n— filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n— filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n— filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}
2023-09-12T12:32:38+02:00 ERROR gracefulexit:chore error retrieving satellites. {“error”: “satellitesdb: context canceled”, “errorVerbose”: “satellitesdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*satellitesDB).ListGracefulExits.func1:152\n\tstorj.io/storj/storagenode/storagenodedb.(*satellitesDB).ListGracefulExits:164\n\tstorj.io/storj/storagenode/gracefulexit.(*Service).ListPendingExits:59\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).AddMissing:58\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).Run:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}
2023-09-12T12:32:38+02:00 ERROR pieces:trash emptying trash failed {“error”: “pieces error: filestore error: context canceled”, “errorVerbose”: “pieces error: filestore error: context canceled\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).EmptyTrash:161\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:316\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:406\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1.1:83\n\tstorj.io/common/sync2.(*Workplace).Start.func1:89”}
2023-09-12T12:32:38+02:00 FATAL Unrecoverable error {“error”: “piecestore monitor: disk space requirement not met”, “errorVerbose”: “piecestore monitor: disk space requirement not met\n\tstorj.io/storj/storagenode/monitor.(*Service).Run:135\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}

[Stob]

@eddy89

2023-09-12T12:32:38+02:00 FATAL Unrecoverable error {“error”: “piecestore monitor: disk space requirement not met”, “errorVerbose”: “piecestore monitor: disk space requirement not met

This FATAL error is the one shutting down the node. The minimum space for a node is 500GB and this node doesn’t think there is that much space available on the disk or less than 500GB has been assigned:

2023-09-12T12:32:38+02:00 WARN piecestore:monitor Disk space is less than requested. Allocated space is {“bytes”: 283266360320}
2023-09-12T12:32:38+02:00 ERROR piecestore:monitor Total disk space is less than required minimum {“bytes”: 500000000000}

If the node isn’t detecting space correctly then you can change the config.yaml to tell the node not to check the available space on start:

@Alexey

You can temporary disable this check by reducing value for storage2.monitor.minimum-disk-space option in the config.yaml file or specify it as a command line option after the image name in your docker run command, i.e. --storage2.monitor.minimum-disk-space=400GB
It should start and run a filewalker, it will fill up the database and then storagenode should recognize its used space.

[eddy89]

My nodes have only been in operation for 3 months and are supposedly full of loud Windows, but the dashboard shows me something completely different . I already wrote it (Node usage does not match)

[Vadim]

do you have some other files there that takes space?

[eddy89]

only storj node

[daki82]

i think we solved it, the problem is exfat with clustersize of 8MB

Node usage does not match

In Windows it shows me that the node is almost full, but in the dashboard it is only a fraction of it [screenshot.19] [screenshot.20] sorry for my bad english