Storj docker container fills drive with logs

[1arrcy1]

The above screeshots explain what the issue is, Docker does not have a default data limit when writing log files. The result is the drive filling up causing potential catastophic issues (corrupted storj), when Ram gets filled and there is no swap left.

My setup is having docker installed in lxc in proxmox
Proxmox>LXC>docker-Direct mount bind with host

I want to setup lograte or something similar to limit the max size of the log file. Problem is i don’t want to shutdown/nuke the container because i don’t want to risk starting over. I would gladly receive advice how i can mitigate this issue.

Best regards,
Storj operator

[nerdatwork]

All my nodes keep restarting since last night: "/var/log/supervisor/supervisord.log does not exist"

Yes, the simplest way is to configure logrotate to manage your storagenode logs. It should already be running on your system. I recommend placing a logrotate config specifically for your node(s) in the /etc/logrotate.d folder. For example, I have a file /etc/logrotate.d/storagenode (it can have any name). The file contents are: /path/to/logs/storj*/storagenode*.log { su username username weekly maxsize 200M rotate 52 dateext dateformat -%Y%m%d …

[1arrcy1]

i tried it, but it doesn’t work

/var/lib/docker/containers/a9b6f57e749032ab09e1192cce357fa0bbb29412c0150b241d5b15b8a1522657/a9b6f57e749032ab09e1192cce357fa0bbb29412c0150b241d5b15b8a1522657-json.log {
weekly
maxsize 200M
rotate 52
dateext
dateformat -%Y%m%d
compress
delaycompress
missingok
notifempty
copytruncate
}

[1arrcy1]

i tried sh -c “truncate -s 0 /var/lib/docker/containers/a9b6f57e749032ab09e1192cce357fa0bbb29412c0150b241d5b15b8a1522657/a9b6f57e749032ab09e1192cce357fa0bbb29412c0150b241d5b15b8a1522657-json.log”

Which solved the issue because its now empty, but it continues to keep writing logs, with absurd sizes. Also im not longer able to read the logs in portainer

[974ffa23d764cf444641]

Add to /etc/docker/daemon.json

  "log-driver": "local",
  "log-opts": {
    "max-file": "3",
    "max-size": "500m"
  }

*docker service restart needed

this settings will be your defaults for all containers

[1arrcy1]

i ended up doing this
nano /etc/docker/daemon.json
{
“log-driver”: “json-file”,
“log-opts”: {“max-size”: “100m”, “max-file”: “3”}
}
docker stop -t 300 storagenode
reboot now
docker start storagenode
let’s wait and see
unfortunately it did go to 0s up time :

welp that sucks

[agente]

I’m curious to know how long it takes to traffic back again…

[donald.m.motsinger]

@1arrcy1

Problem is i don’t want to shutdown/nuke the container because i don’t want to risk starting over.

Removing the container doesn’t remove the data.

[1arrcy1]

It still is not fixed Data log over 200 MB currently

[Alexey]

Perhaps you have other containers?

docker ps -a

Also you may try to specify log options directly in your docker run command:

--log-opt max-size=100m \
--log-opt max-file=3 \

they should be provided before the image name (not after). And the max usage for these options should be 300MiB

[JoshieGarza]

why dont u simply change the log level in the config.yaml file and restart the node?

the minimum log level to log

log.level: error

[Alexey]

seems they already did so. Now they are wonder, why is it so?
(you shouldn’t reduce the log level below the info).