Can I get some help troubleshooting why my Raspberry Pi setup keeps locking up?
It’ll run fine for a while, maybe 12 hours or so, and then it goes down, can’t ssh in, can’t VNC in, storage node is down, and I have to pull the power supply and reboot it.
Where should I start? Where are logs located? Thx
I would start with your powersupply, If your powering a USB hard drive your taking alot more power from the RPI itself, You may need to upgrade to a better one.
Good idea, but I’m connected to a NAS over Ethernet. So I don’t think it would be a power issue.
And before anyone else says it: yes I did setup a static mount via the instructions.
If your running a NAS I wouldnt run a node though RPI over a network to run a node though.
The RPi 3, 2, and 1 have shared USB bandwidth with the Ethernet interface and USB ports.
High network load combined with high external USB HDD usage will most likely overload the SoC USB port. It’s very likely that most RPi3 nodes are going to struggle to remain online once all the network caps get removed.
yeah I realize it might not be an ideal setup. I wanted to give it a try anyways. My NAS can’t run Docker, which was what I was going to do at first.
That said, there shouldn’t be a bandwidth issue, so I’m not sure what the real limitation would be.
Do you monitor your RPI with netdata and prometheus? Maybe it runs out of ram and starts swapping. that would make it unresponsive.
Take a look a docker stats to see what kinda usage is happening on your node.
Thanks, I’ve got netdata installed now so I’ll keep my eye on that.
OK the first thing I’m noticing now is that the swap is definitely climbing over time. It’s gone from low 40% to low 50% in just the time since I last wrote.
It’s a very small swap it would seem, but that’s good (from what I understand) for a Pi because it’ll kill the SD card otherwise.
Total swap:
is it swap or zram? If it is zram I wouldn’t worry about it unless you run out of RAM.
What’s your RAM usage?
The PI should not have any swap on disk because it will kill the SD and make it unresponsive anyway. Not very useful on a Pi
Good question… not sure. It’s just what netdata is calling swap at the overview at the top. Here’s the full overview:
I mean, I definitely don’t have a swap partition like you normally would.
aaaaaand now netdata is down
storage node still up for now though
ugh… yeah there is definitely something wrong.
Show us a screenshot of “htop” sorted by memory usage.
Also the output of
cat /proc/swaps
Hello @cy2k,
Welcome to the forum!
I would assume that you use a OS with GUI. It’s not recommended to run your node on the raspberry PI with a GUI. You will waste at least 200MB of RAM to handle GUI. The raspberry PI doesn’t have a much RAM to waste it to GUI. It’s much better to run the OS only with CLI.
Also, I would recommend to limit the RAM usage for the storagenode container with a --memory=800m option. In case of OS with GUI this limit should be much lower, something like --memory=600m
*200MB
Limiting the memory of a container will prevent the RPI from becoming unresponsive but as far as I remember, it kills the container and restarts it. That’s not very healthy to the node either.
And certainly doesn’t solve the problem. The RAM shouldn’t rise until 100% if everything is working normally.