Storj node error 1067. Lazyfilewalker cache error

[blanaru]

Hello! I reinstalled my old win10 with win server 2016, hoping that my node will not restart randomly and storj service will fail to start.
Now I got a strange issue: error 1067 and storj service is failing to start.
I reinstalled the app, changed as usual the zksynk option in config.yaml and all seem to work ok for 2 days. Now I got again the same problem.
I looked on previous posts and it seems that there are some error logs found below.
I use an external usb drive for this setup and it worked fine for 2 years without getting this error.

2024-03-28T22:15:37-07:00 INFO lazyfilewalker.used-space-filewalker starting subprocess {“satelliteID”:
“12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2024-03-28T22:15:37-07:00 ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {“satell
iteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “error”: “context canceled”}
2024-03-28T22:15:37-07:00 ERROR pieces failed to lazywalk space used by satellite {“error”: “lazyfilewalke
r: context canceled”, “errorVerbose”: “lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewal
ker.(*process).run:71\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatelli
te:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:717\n\tstorj.io/storj/storagenode/piec
es.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io
/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “Satellite ID”: “12L
9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2024-03-28T22:15:37-07:00 ERROR piecestore:cache error getting current used space: {“error”: “filew
alker: context canceled; filewalker: context canceled”, “errorVerbose”: "group:\n— filewalker: context canceled\n\tsto
rj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).W
alkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:726\n\ts
torj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\tr
untime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.f
unc1:75\n— filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\ts
torj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/piece
s.(*Store).SpaceUsedTotalAndBySatellite:726\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/stor
j/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func

[blanaru]

I think I found the issue: if I add the zksync line in config.yaml, the node gets 1067 error. When I remove it, the node works fine.
The string added is attached.

[Alexey]

Hello @blanaru,
Welcome back!

@blanaru

context canceled

This is mean a readable timeout - your disk is too slow to respond.
I guess you use a VM, so it’s kind of expected to have a slower respond because of virtualization on a foreign OS (you likely use Linux as a host, right?).

You need to speed up a disk.

1. Stop the node
2. Check and fix issues with the disk
3. Perform a defragmentation
4. Enable an automatic defragmentation, if you disabled it (it’s enabled by default)
5. Start the node

If you still see similar errors, then you may perform this adjustment too:

NTFS Disable 8dot3name

Fresh off an ReFS volume that went RAW, I’ve decided to be equally reckless and disable the 8 dot 3 short names for my replacement NTFS volume. I was shocked to not find a single mention of it in this forum, so I’ll report back once this bad decision ends up in another catastrophic fail, but the numbers looks promising for the Storj trillion tiny file configurations. [8484.image_thumb_0763777C] Disable across system: fsutil behavior set disable8dot3 Strip existing: fsutil 8dot3name strip /f…

It should help a lot.

If you could still have this error, I would like to suggest either convert it to a docker node running directly on your host, or disable a lazy filewalker:

Huge drop of Average disk space used

How is your disk connected? Is it SMR? It’s so slow (“context canceled” errors for every attempt), that even lazy wilewalker cannot finish its work. Please try to disable lazy filewalker, it will turn on the normal (with a normal priority): # run garbage collection and used-space calculation filewalkers # as a separate subprocess with lower IO priority (default true) pieces.enable-lazy-filewalker: false save the config and restart the node. If you use docker, you may provide this flag after …

@blanaru

I think I found the issue: if I add the zksync line in config.yaml, the node gets 1067 error. When I remove it, the node works fine.
The string added is attached.

Perhaps you have a duplicate key. Search for the same option across the config.

[blanaru]

Thank you Alexey for the fast reply!
I checked the config.yaml and there is no other lines like this one…
I am thinking that perhaps I have added the command wrong …

[Alexey]

No, it looks correct. Maybe only add a new line after the option and save the config, then restart the node.

[blanaru]

Ok, so I added a new line and save it. After node restart I still got that error.

[Alexey]

Please try to stop the service and run it like this (PowerShell):

& "$env:ProgramFiles\Storj\Storage Node\storagenode.exe" run --config-dir "$env:ProgramFiles\Storj\Storage Node\\" --log.output=stderr

You should see, why is it not happy.

[blanaru]

It says why is not happy:
Error: While parsing config: yaml: unmarshal errors:
line 342: mapping key “operator.wallet-features” already defined at line 134

now looking at line 134, here is what is written:

[blanaru]

I checked another node to see whats on that line, and I found out that it is the same line…

[Mitsos]

@blanaru

unmarshal errors:
line 342: mapping key “operator.wallet-features” already defined at line 134

Your node says that I saw operator.wallet-features on line 342, but I already saw it on line 134. I don’t know which of the two to use.

[daki82]

It turns out, that the new version does not handle double entries in the yaml , so:

search the line to edit, do only add a new line if nothing is to find.

[blanaru]

you are totally right abd it makes sense. After few trial and error, I found the correct command that now is working perfect:

[heunland]

If you are trying to opt in to zksync-era, you should specify the following instead. Also please note the quotes that need to be included:

operator.wallet-features: ['zksync-era']

See also ZkSync Payments - Storj Docs
Please note that zksync Lite will no longer be available soon, so makes more sense to opt in to zksync-era now.

Here is how you can check if you properly opted in the to your preferred payout option ZkSync Payments - Storj Docs

[blanaru]

Even greater! Thank you for sharing this!
I have opted in to zksync era:

[Alexey]

You are correct, the quotes are required, if you use square brackets.
It also works without square brackets, if there is only one value. However, if use several values (i.e. ["zksync-era", "zksync"]), then square brackets are required.