Filewalker error

lovaszl · January 4, 2024, 8:19am

I would like a little help as to what the error is:
Node. runs in Synology docker on a separate ext4 disk.
I recently noticed that there is more than a 1TB difference between the actual disk usage and the one indicated by the dashboard.
Reading the topics here, Alexey drew attention to the lazyfilewalker entries in the log and it is indeed the error.
But you are referring to a satellite that I cannot identify.

Please help me how to solve this problem.
On condition that it is not related to the UDP communication “error” that has existed for a month, because QUIC does not work for me either. As I read, this is a synology feature.

Thanks

2023-12-08T22:19:43Z	ERROR	pieces	failed to lazywalk space used by satellite	{"process": "storagenode", "error": "lazyfilewalker: signal: killed", "errorVerbose": "lazyfilewalker: signal: killed\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:83\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:717\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2023-12-08T22:19:43Z	ERROR	lazyfilewalker.used-space-filewalker	failed to start subprocess	{"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "context canceled"}
2023-12-08T22:19:43Z	ERROR	pieces	failed to lazywalk space used by satellite	{"process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:71\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:717\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2023-12-08T22:19:43Z	ERROR	lazyfilewalker.used-space-filewalker	failed to start subprocess	{"process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "context canceled"}
2023-12-08T22:19:43Z	ERROR	pieces	failed to lazywalk space used by satellite	{"process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:71\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:717\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2023-12-08T22:19:43Z	ERROR	piecestore:cache	error getting current used space: 	{"process": "storagenode", "error": "filewalker: context canceled; filewalker: context canceled; filewalker: context canceled", "errorVerbose": "group:\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:726\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:726\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:726\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-12-08T22:19:59Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Error": "contact: failed to ping storage node using QUIC, your node indicated error code: 0, rpc: quic: timeout: no recent network activity"}

storj_volume2

Alexey · January 4, 2024, 8:25am

This is mean that the disk has a great latency and prevent the filewalker to finish its work.
You may try to disable the lazy filewalker to allow it to run with a normal priority

pieces.enable-lazy-filewalker: false

save the config and restart the node. You may also use it as an argument after the image name in your docker run command:

docker run -d ...
...
storjlabs/storagenode:latest \
--pieces.enable-lazy-filewalker="false"

lovaszl · January 4, 2024, 8:54am

Thanks, I inserted the value into the “config.yaml” file and restarted the container, now I’m waiting to see what happens.

lovaszl · January 4, 2024, 6:23pm

How long should it take me to see the result, because the disk is full according to the node.

Alexey · January 5, 2024, 2:33am

Unpredictable, depends on your hardware. On my biggest Windows docker node (7TB) the longest lazy gc-filewalker (12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S) took 6h 23m

2023-12-14T06:36:32Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker started   {"process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "process": "storagenode", "createdBefore": "2023-12-06T17:59:59Z", "bloomFilterSize": 2097155}
2023-12-14T12:59:47Z    INFO    lazyfilewalker.gc-filewalker    subprocess finished successfully        {"process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}

The used space lazy filewalker took 2 days

2023-12-16T10:36:14Z    INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started   {"process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "process": "storagenode"}
2023-12-18T09:33:05Z    INFO    lazyfilewalker.used-space-filewalker    subprocess finished successfully        {"process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}

usually the not lazy filewalker should take in 2-3 times less time.

lovaszl · January 7, 2024, 9:49am

Thank you for all your help!
I look at the logs, but there are no entries about filewalker, no space has been freed.
Offtopic question:
I start a second node in docker because the first one is full.
I have to register with a new email address for a new token and on a different port, but I can continue using the same ip address and payment address, am I right? Thanks.

snorkel · January 7, 2024, 10:36am

@lovaszl
Yes, you can use the same IP and according to ToS, you have to use the same wallet for all nodes. Storj pays the transfer fee to your wallet, so if you were using say 5 wallets, Storj would pay 5x fee. That’s not very healty. So, yes, use the same wallet for all your nodes, no matter were they are installed.
You can use the same email, but you must request a new token and create a new identity.
Take a look at my ports for 2 nodes on same machine. If you get stuck, using different ports for all maight help.
https://forum.storj.io/t/my-docker-run-commands-for-multinodes-on-synology-nas/22034/7?u=snorkel

Alexey · January 8, 2024, 2:08am

You should have entries about a filewalker (search for walk) and retain process (search for retain) in your logs, unless your node were re-created more often than once in two weeks.