Seeking help with Debian root drive being full

[rml52]

@donald.m.motsinger

Then follow this to prune dangling images

I did the prune command a couple days ago, while it did free up some space, that file size tool ncdu still shows the current docker has 12gigs of logs.

I just want someone from Storj to tell me it is safe to do something and maybe provide safe steps on how to do it because this node if 5 years old, has a lot of used data and I don’t want to loose it and have to start all over.

Based on nuggets of comments from these last couple days, I speculate that I will have to stop the node, change the config.yaml to put the logs somewhere else, MAYBE delete the xxxx.jason-log file in the screenshots??? (that is the part I want someone to tell me is ok to do), then restart the node, and setup logrotate to auto delete the new log location

[rml52]

@Knowledge Would you be able to answer or know a storjling who can?

[rml52]

Seems to me that this may be a bigger issue for Storj Labs to look into. Older nodes not having logs rotated out trapped in the docker.

[lyoth]

this is not a Storj Lab issue…
There are docker run commands you can use to log rotate it.

[Knowledge]

Well it sounds like a Docker issue. You can configure Docker to rotate and purge logs based on size limits. See here…

Docker container logs taking all my disk space

You can also remove the container and recreate it which will also purge the logs.

[donald.m.motsinger]

@rml52

I did the prune command a couple days ago, while it did free up some space, that file size tool ncdu still shows the current docker has 12gigs of logs.

It’s difficult to help you if you don’t answer all questions.

@donald.m.motsinger

Can you look with “tail” what the last lines of the log file are?

[rml52]

@Knowledge

You can also remove the container and recreate it which will also purge the logs.

I have done this and it doesn’t go down that I why I want to make sure I don’t break anything

[rml52]

@donald.m.motsinger

Can you look with “tail” what the last lines of the log file are?

last log shows 2024-04-24T20:26:13

[donald.m.motsinger]

@rml52

last log shows 2024-04-24T20:26:13

Ok, that means, it’s the right logfile and currently used. If you don’t need the logs, just empty that file (don’t delete it!) with

echo > /path/to/file

[rml52]

I ran the command. all seems healthy. node didn’t go offline. and df -h shows 28% down from 97%

[rml52]

Spoke too soon, just got the email that my node is down. Troubleshooting now

[rml52]

Rebooted VM. Back online. Drive space still at 28%, Dashboard says up and OK.

[jammerdan]

@rml52

and it doesn’t go down

Then it sounds that something is wrong with your docker installation. If you remove the container the log should be deleted with it.

[kocoten1992]

Hmm, I’ve check my own container, it show 7GB of logs as well (and I’ve watchtower so it got replacement when new node version came out?).

[Alexey]

    command:
      - --log.output=config/storagenode.log

And your logs will be written to the file storagenode.log on your data location.
Then you likely will need to setup a logrotate on your host:

Best method to redirect logs to independant storage in Linux

Hi phillipe, This is easy to do with logrotate since you can use wildcards to define your paths and files. I placed a logrotate config file in /etc/logrotate.d/ called storagenode . It is a plain text file. My logrotate config file is as follows. Don’t include comments in your config file (there is an uncommented version below). /mnt/logs-usb/storj*/storagenode*.log { #the wildcard (*) characters tell logrotate to look in any subfolder that #begins with storj, and rotate any log that starts wi…

How to create daily storagenode logs

are you running in a NAS or Windows or Linux? In Ubuntu I use the “logrotate” command in a cron job to create/compress log files. First you have to make sure you have added in the “Log Location” and “Log level” fields in the “config.yaml” Then look in the “/etc/logrotate.d/” folder, you’ll see a list of files, those are the configs used to rotate logfiles ( usually found in “/var/log/”). I just put a “storj” config file in the logrotate.d/ and it’s done automatically To rotate my logs i use …

[Alexey]

Perhaps you may empty them like suggested by

Seeking help with Debian root drive being full

Ok, that means, it’s the right logfile and currently used. If you don’t need the logs, just empty that file (don’t delete it!) with echo > /path/to/file

However, it’s possible that you have other running containers, which may log to there. So I would suggest to stop and remove all containers, include stopped, you may see them with the command

docker ps -a

Then check a usage, it should be reduced.
After that run the needed containers back.

[JWvdV]

Perhaps this might help:

docker run -d --restart unless-stopped --stop-timeout 300 \
       (...) \
        --log-opt max-size=20m --log-opt max-file=3 \
        --name "storagenode" storjlabs/storagenode:latest \
        (...)

[Alexey]

It’s suggested there:

@jammerdan

--log-driver local \
	--log-opt mode=non-blocking \
	--log-opt max-size=100m \
	--log-opt max-file=4 \
	--log-opt compress=false \

but I suppose was ignored…