I have a log rotation, as stated above the config is:
docker run -d --restart unless-stopped --stop-timeout 300 \
(...) \
--log-opt max-size=20m --log-opt max-file=3 \
--name "storagenode" storjlabs/storagenode:latest \
--storage2.monitor.minimum-disk-space=100MB \
--storage2.piece-scan-on-startup=true \
--pieces.enable-lazy-filewalker=false
So I enable scan on startup, but disable the lazy filewalker (in order to fall back to the non-lazy file walker).
But, this is the output of docker logs:
root@STORJ3:~# docker logs storagenode 2>/dev/null | head -n 55
downloading storagenode-updater
downloading storagenode
2023-11-10 09:51:11,647 INFO Set uid to user 0 succeeded
2023-11-10 09:51:11,661 INFO RPC interface 'supervisor' initialized
2023-11-10 09:51:11,662 INFO supervisord started with pid 1
2023-11-10 09:51:12,664 INFO spawned: 'processes-exit-eventlistener' with pid 48
2023-11-10 09:51:12,669 INFO spawned: 'storagenode' with pid 49
2023-11-10 09:51:12,672 INFO spawned: 'storagenode-updater' with pid 50
2023-11-10T09:51:12Z INFO Configuration loaded {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2023-11-10T09:51:12Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2023-11-10T09:51:12Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "console.address"}
2023-11-10T09:51:12Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2023-11-10T09:51:12Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "server.address"}
2023-11-10T09:51:12Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "operator.wallet"}
2023-11-10T09:51:12Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "contact.external-address"}
2023-11-10T09:51:12Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "storage2.database-dir"}
2023-11-10T09:51:12Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2023-11-10T09:51:12Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "server.private-address"}
2023-11-10T09:51:12Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2023-11-10T09:51:12Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "operator.email"}
2023-11-10T09:51:12Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "healthcheck.details"}
2023-11-10T09:51:12Z INFO Invalid configuration file value for key {"Process": "storagenode-updater", "Key": "log.development"}
2023-11-10T09:51:12Z INFO Invalid configuration file value for key {"Process": "storagenode-updater", "Key": "log.level"}
2023-11-10T09:51:12Z INFO Invalid configuration file value for key {"Process": "storagenode-updater", "Key": "log.output"}
2023-11-10T09:51:12Z INFO Invalid configuration file value for key {"Process": "storagenode-updater", "Key": "log.caller"}
2023-11-10T09:51:12Z INFO Anonymized tracing enabled {"Process": "storagenode-updater"}
2023-11-10T09:51:12Z INFO Running on version {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.90.2"}
2023-11-10T09:51:12Z INFO Downloading versions. {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-11-10T09:51:13Z INFO Configuration loaded {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-11-10T09:51:13Z INFO Anonymized tracing enabled {"process": "storagenode"}
2023-11-10T09:51:13Z INFO Operator email {"process": "storagenode", "Address": "jwvdveer@gmail.com"}
2023-11-10T09:51:13Z INFO Operator wallet {"process": "storagenode", "Address": "0x4A93997a2a8aCd9F95A8362c130D4Dbee8D2Dfe3"}
2023-11-10T09:51:13Z INFO server existing kernel support for server-side tcp fast open detected {"process": "storagenode"}
2023-11-10T09:51:13Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.90.2"}
2023-11-10T09:51:13Z INFO New version is being rolled out but hasn't made it to this node yet {"Process": "storagenode-updater", "Service": "storagenode"}
2023-11-10T09:51:13Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.90.2"}
2023-11-10T09:51:13Z INFO New version is being rolled out but hasn't made it to this node yet {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2023-11-10T09:51:13Z INFO Telemetry enabled {"process": "storagenode", "instance ID": "127HsBPUCHDawobg1LMDaLNDWDtcHYjov4MYBYBrm88ePm5Tegn"}
2023-11-10T09:51:13Z INFO Event collection enabled {"process": "storagenode", "instance ID": "127HsBPUCHDawobg1LMDaLNDWDtcHYjov4MYBYBrm88ePm5Tegn"}
2023-11-10 09:51:13,957 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-11-10 09:51:13,957 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-11-10 09:51:13,958 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-11-10T09:51:14Z INFO db.migration Database Version {"process": "storagenode", "version": 54}
2023-11-10T09:51:14Z INFO preflight:localtime start checking local system clock with trusted satellites' system clock. {"process": "storagenode"}
2023-11-10T09:51:15Z INFO preflight:localtime local system clock is in sync with trusted satellites' system clock. {"process": "storagenode"}
2023-11-10T09:51:15Z INFO Node 127HsBPUCHDawobg1LMDaLNDWDtcHYjov4MYBYBrm88ePm5Tegn started {"process": "storagenode"}
2023-11-10T09:51:15Z INFO Public server started on [::]:28967 {"process": "storagenode"}
2023-11-10T09:51:15Z INFO bandwidth Performing bandwidth usage rollups {"process": "storagenode"}
2023-11-10T09:51:15Z INFO Private server started on 127.0.0.1:7778 {"process": "storagenode"}
2023-11-10T09:51:15Z INFO trust Scheduling next refresh {"process": "storagenode", "after": "2h31m41.322289225s"}
2023-11-10T09:51:15Z INFO pieces:trash emptying trash started {"process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2023-11-10T09:51:15Z WARN piecestore:monitor Disk space is less than requested. Allocated space is {"process": "storagenode", "bytes": 2173001064320}
root@STORJ3:~# docker logs storagenode 2>&1 | grep -i walk
root@STORJ3:~#
root@STORJ3:~# ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.6 165616 10752 ? Ss 09:45 0:00 /sbin/init
root 116 0.0 0.8 31328 13952 ? S<s 09:45 0:00 /lib/systemd/systemd-journald
systemd+ 148 0.0 0.5 16120 8320 ? Ss 09:45 0:00 /lib/systemd/systemd-networkd
systemd+ 152 0.0 0.8 25532 13556 ? Ss 09:45 0:00 /lib/systemd/systemd-resolved
root 154 0.0 0.1 9492 2816 ? Ss 09:45 0:00 /usr/sbin/cron -f -P
message+ 155 0.0 0.3 8556 4736 ? Ss 09:45 0:00 @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syroot 158 0.0 1.0 34300 16128 ? Ss 09:45 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
syslog 161 0.0 0.3 222400 5248 ? Ssl 09:45 0:00 /usr/sbin/rsyslogd -n -iNONE
root 171 0.0 0.4 14904 7040 ? Ss 09:45 0:00 /lib/systemd/systemd-logind
root 242 0.0 2.1 1356508 33528 ? Ssl 09:45 0:12 /usr/bin/containerd
root 245 0.0 0.1 8396 2432 pts/0 Ss+ 09:45 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 vt220
root 247 0.0 0.1 8396 2304 pts/1 Ss+ 09:45 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud pts/1 115200,38400,9600 vt220
root 249 0.0 0.1 8396 2176 pts/2 Ss+ 09:45 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud pts/2 115200,38400,9600 vt220
root 252 0.0 0.1 8396 2176 pts/3 Ss+ 09:45 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud pts/3 115200,38400,9600 vt220
root 258 0.0 0.1 8396 2432 pts/4 Ss+ 09:45 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud pts/4 115200,38400,9600 vt220
root 260 0.0 0.5 15424 9216 ? Ss 09:45 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root 291 0.0 4.8 1536856 76540 ? Ssl 09:45 0:08 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 380 0.0 0.2 10204 3328 pts/5 Ss 09:45 0:00 /bin/bash
root 618 0.0 0.6 719596 10816 ? Sl 09:51 0:04 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 23b70e4a2c4bac8ed278e9ba4212fef848root 639 0.0 0.1 24724 2604 ? Ss 09:51 0:13 /usr/bin/python2 /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
root 693 0.0 0.0 3736 384 ? S 09:51 0:00 /bin/bash /bin/stop-supervisor
root 694 6.2 1.6 1256496 26360 ? Sl 09:51 21:58 /app/storagenode run --config-dir config --identity-dir identity --metrics.app-suffix=-root 695 0.0 0.2 1242760 3956 ? Sl 09:51 0:17 /app/storagenode-updater run --binary-location /app/storagenode --config-dir config --iroot 1541 0.0 0.2 12896 3328 pts/5 R+ 16:45 0:00 ps -aux
root@STORJ3:~# free -m
total used free shared buff/cache available
Mem: 1536 193 1136 0 206 1342
Swap: 0 0 0
root@STORJ3:~# uptime
17:11:50 up 7:26, 0 users, load average: 11.53, 13.34, 13.76
So there’s a full log spanning 7+ hours, not containing anything about a file walker. Nor a process listed looking like it or significant memory use.
So it seems that disabling the lazy filewalker, also disables the back-up (non-lazy) filewalker. But using the lazy filewalker filles up the disk to the rim, killing the node.
