Post upgrade to 26.2, my SN, which is getting some awesome traffic btw, is crashing pretty frequently. Running on Pi 3 with mem limit set to 600. Anyone about that could maybe assist in diagnosing the issue? Takes quite a while to bring things back up.
Would you be able to show us some logs from your storage node to help figure out what is happening?
Of course! Just tell me which logs you’d like and I’ll pastebin them next time the server comes back online.
docker logs --tail 100 storagenode
also, we were having some issues with watchtower eating resources by spawning multiple instances of itself, so let’s just make sure that’s not the case with
Hi Cameron - here you go: https://paste.thedoodleproject.net/?29cd563adee4be53#4uKNYo1KzcDeaqacKobEHzkv8oezaKNNqXoooS9JZaRF
I’m not a watchtower user so that shouldn’t apply.
After looking through that - I think those logs just show that docker itself is hung after a number of restarts. I’ve got a tail on docker logs now and will also post whatever is on screen next time it dies on me.
@doodlemania2 Can we have a large chunk of the logs?
Please execute the following command:
docker logs --tail 10000 storagenode >& log.txt
and share the log.txt file.
Thanks for your patience. It looks like you’re getting a lot of uploads! It might be that you want to set your concurrency limit flag to some value, but it would be also helpful for us to get some profiling information from the
localhost:your-debug-port/debug/pprof/heap endpoint around the time that your node is crashing
Can you share instructions on how to do those two things (concurrency and grabbing debug info) or some links to those?? My crashes appear to be either docker or raspberry crashes so there may need to be something to peak there.
Found concurrency and set to 7 per the docs.
Excellent! Be sure to keep us updated if that helps at all. You should be able to access the debug endpoint by checking your config file for your
debug.addr and visiting
/debug/pprof/heap at that address in your web browser