I wonder what message from the none lazy filewalker we should see in the log when it starts? I had the lazy one disabled and there was no filewalker message after 5 days. So I switched back to lazy filewalker.
Is the none lazy filewalker even functional in latest releases?
I have disable the lazy filewalker and i haven’t no message in the log. I have notify that the filewalker runnnig when i see the % of use of disks, if 0-5% it’s not runnng, if 95-100% it’ running.
I have for 50% of nodes 1.99.3 of version and other 50% 1.97.3
what’s your version?
It should print out some messages, at least the used-space-filewalker, if you didn’t disable it.
version 1.97.3 windows
Did you disable the used-space-filewalker on start (it’s enabled by default)?
I guess you mean this option?
storage2.piece-scan-on-startup: true
Yes. It’s enabled by default, so you do not need to add it explicitly, unless you disabled it, in which case you need either comment out it or set to true.
i have all default and i don’t see record in the log for filewalker, any filewalker. I view on dashboard that the total amount of TB and TBm are correct but i don’t view nothing in the log. The nodes are or 1.99.3 and 1.97.3.
I have this one too, but used and tbm differ alot. i will comment it out, enable the lazy and logging to error and run it a few days.
First these popped up:
2024-04-06T22:28:52+02:00 ERROR piecestore download failed {"Piece ID": "XS4RKIQXIVN4KXDRELSJQKMGRTX362EDTV4VAN3OLCR6KLUDY6PQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 8960, "Remote Address": "79.127.201.209:45560", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:621\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-04-06T22:28:52+02:00 ERROR piecestore download failed {"Piece ID": "YEFTZMHDCCI75XDKMJY6NR4BR43QYSH3UWVUP35GILLIS5J5KI3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 18688, "Remote Address": "79.127.205.225:35094", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:621\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-04-06T22:28:52+02:00 ERROR piecestore download failed {"Piece ID": "QC2Q5DJ3UPHWY6PKPPFJB26EPV7DE6JIJO3QVMGYFB7M5KZILH7Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 11008, "Remote Address": "79.127.201.209:45588", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:621\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-04-06T22:28:52+02:00 ERROR piecestore download failed {"Piece ID": "DIXHJFK7XXINGFC6Q5DXJPFYM62MSPH2DSKQCHVWLQAWAX3CWJ7A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 72448, "Size": 290048, "Remote Address": "79.127.223.130:43364", "error": "trust: rpc: tcp connector failed: rpc: context canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: context canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
ver 1.99.3
1.95.1, 1.97.3, 1.99.3 - docker. No logs from filewalkers too. Only if lazy-filewalker enabled. So it’s been broken for a while. HDD utilization 100% so used-space-filewalker definitely working but logs are missing.
Either a network error (your current DNS is unable to resolve the hostname, or the DNS traffic is blocked) or a consequence of the termination command somewhere earlier, it could be a FATAL error.
so its normal for an restart (i did one for config change).
filewalk finished.
only 176 normal upload download errors, no change to the dashboard.
I guess so. The termination command could be sent before they finished their work, so they likely would complain.
I don’t see any log entries for filewalker either when lazy is turned off.
The load suggests that it is running but there is not a single line of anything of that in the logs.
@Alexey
Any specific wording I could grep for to see at least something in the logs? There are no results for filewalker, used, space, walker etc.
Version is 1.99 and log level is info.
Perhaps this version has a bug regarding logging. I didn’t get a confirmation on this regard though.
Could you please try to increase a logging level?
If you have a debug port enabled, you can do it on the fly:
There are at least 2 other threads about this. Does someone investigate it?
On linux, after disabling lazy file walker, the piece scan on startup dosen’t get any entries in the log, not for start, not for finish.
Maybe it’s on Windows too.
Also, the Garbage collector is missing the log entries in the same situation.
With lazy off, you will get only these logs about walkers:
-pieces:trash
-collector
-retain.
Node ver. 1.102.
I shared with the team. But I guess until you wouldn’t create an issue, it unlikely would be taken earlier.
if it’s not in the config.yaml, I simply add it, right? @Alexey