After disabling lazy mode the filewalkers didn't print anything to the logs

[pangolin]

@Alexey

then it’s not finished yet or not started at all.

I wonder what message from the none lazy filewalker we should see in the log when it starts? I had the lazy one disabled and there was no filewalker message after 5 days. So I switched back to lazy filewalker.

Is the none lazy filewalker even functional in latest releases?

[digitalfrank]

@pangolin

Is the none lazy filewalker even functional in latest releases?

I have disable the lazy filewalker and i haven’t no message in the log. I have notify that the filewalker runnnig when i see the % of use of disks, if 0-5% it’s not runnng, if 95-100% it’ running.

I have for 50% of nodes 1.99.3 of version and other 50% 1.97.3

[Alexey]

what’s your version?
It should print out some messages, at least the used-space-filewalker, if you didn’t disable it.

[pangolin]

version 1.97.3 windows

[Alexey]

Did you disable the used-space-filewalker on start (it’s enabled by default)?

[pangolin]

I guess you mean this option?

storage2.piece-scan-on-startup: true

[Alexey]

Yes. It’s enabled by default, so you do not need to add it explicitly, unless you disabled it, in which case you need either comment out it or set to true.

[digitalfrank]

i have all default and i don’t see record in the log for filewalker, any filewalker. I view on dashboard that the total amount of TB and TBm are correct but i don’t view nothing in the log. The nodes are or 1.99.3 and 1.97.3.

[daki82]

@pangolin

storage2.piece-scan-on-startup: true

I have this one too, but used and tbm differ alot. i will comment it out, enable the lazy and logging to error and run it a few days.

First these popped up:

2024-04-06T22:28:52+02:00	ERROR	piecestore	download failed	{"Piece ID": "XS4RKIQXIVN4KXDRELSJQKMGRTX362EDTV4VAN3OLCR6KLUDY6PQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 8960, "Remote Address": "79.127.201.209:45560", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:621\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-04-06T22:28:52+02:00	ERROR	piecestore	download failed	{"Piece ID": "YEFTZMHDCCI75XDKMJY6NR4BR43QYSH3UWVUP35GILLIS5J5KI3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 18688, "Remote Address": "79.127.205.225:35094", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:621\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-04-06T22:28:52+02:00	ERROR	piecestore	download failed	{"Piece ID": "QC2Q5DJ3UPHWY6PKPPFJB26EPV7DE6JIJO3QVMGYFB7M5KZILH7Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 11008, "Remote Address": "79.127.201.209:45588", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:621\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-04-06T22:28:52+02:00	ERROR	piecestore	download failed	{"Piece ID": "DIXHJFK7XXINGFC6Q5DXJPFYM62MSPH2DSKQCHVWLQAWAX3CWJ7A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 72448, "Size": 290048, "Remote Address": "79.127.223.130:43364", "error": "trust: rpc: tcp connector failed: rpc: context canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: context canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}

ver 1.99.3

[Andrew]

1.95.1, 1.97.3, 1.99.3 - docker. No logs from filewalkers too. Only if lazy-filewalker enabled. So it’s been broken for a while. HDD utilization 100% so used-space-filewalker definitely working but logs are missing.

[Alexey]

@daki82

dial tcp: lookup us1.storj.io: operation was canceled

Either a network error (your current DNS is unable to resolve the hostname, or the DNS traffic is blocked) or a consequence of the termination command somewhere earlier, it could be a FATAL error.

[daki82]

@Alexey

consequence of the termination command somewhere earlier

so its normal for an restart (i did one for config change).
filewalk finished.
only 176 normal upload download errors, no change to the dashboard.

[Alexey]

I guess so. The termination command could be sent before they finished their work, so they likely would complain.

[jammerdan]

I don’t see any log entries for filewalker either when lazy is turned off.
The load suggests that it is running but there is not a single line of anything of that in the logs.

@Alexey
Any specific wording I could grep for to see at least something in the logs? There are no results for filewalker, used, space, walker etc.

Version is 1.99 and log level is info.

[Alexey]

Perhaps this version has a bug regarding logging. I didn’t get a confirmation on this regard though.
Could you please try to increase a logging level?
If you have a debug port enabled, you can do it on the fly:

@littleskunk

For Everyone
All of our processes contain a debug endpoint. It is now possible to change the log level without having to restart the process. This works with storage nodes as well as with our S3 gateway or any other binary you might think of. Details on how to reach the debug endpoint are here: Guide to debug my storage node, uplink, s3 gateway, satellite