ERROR lazyfilewalker: signal: killed

Node Operators troubleshooting
, , ,
1

Does not seem to be an operational issue, really. What do you think? Nodes are running normal with this coming up from time to time.

2023-08-16T15:58:48Z    ERROR   pieces  failed to lazywalk space used by satellite      {"process": "storagenode", "error": "lazyfilewalker: signal: killed", "errorVerbose": "lazyfilewalker: signal: killed
\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:83
\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105
\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:707
\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57
\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87
\truntime/pprof.Do:44
\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86
\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}

Seems to happen, when a new version is installed. Currently on that node: v1.85.1

2

What’s OS? What’s HDD? Is it SMR?

3 
Linux raspberrypi 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 
17:24:16 BST 2023 aarch64 GNU/Linux

… and CMR

4

How much free memory? Preferably when this issue is happen.
I suspect either overheated USB controller or OOM.
So may be it’s a good idea to search for OOM in your journals, i.e.

journalctl | grep -i oom | tail

It’s also advisable to check disk for errors.

5

Thx. I’ve reduced sd card usage with the following tips.

Does not provide a result.

Last time I’ve checked the disks, 100% was fine.

Typically my Raspi is out of CPU, with more or less IOWAIT, but RAM is fine:

IMG_2907
IMG_29071125×899 120 KB

It’s not looking that way always, but as you can see, CPU usage seems to be a limiting factor with 2 nodes serving.

I’ve limited RAM usage for both nodes:

 --memory=1200m \
6

im seeing this in logs for 1.85.1

1 Like
7

is it context canceled or killed?

8

In my case. But I think you’ve asked @ItsHass

1 Like
9

You are correct. I asked @ItsHass
I found killed error in logs of one of my nodes (Windows, Docker Desktop). It has enough free RAM (24GiB), so I do not believe it’s related to RAM.
But I did chkdsk on that time (read-only), so maybe related.

1 Like