I had no choice and had to take my nodes offline, I had some hardware issues I had to iron out.
I’ve tried to bring them all back online and i’ve been having a lot of issues with things hanging up, the dashboard websites not loading, docker becoming unresponsive, etc.
Before I get too much more in the weeds trying to get these things working, should I just give it up? Is being offline for that long unrecoverable?
So what appears to be my main issue is that, via docker logs, the nodes appear to be functioning.
However, the dashboards don’t load, and docker commands against the nodes don’t ever complete, things like that. And eventually I need to restart the containers.
Where should I start with troubleshooting what seems to be more a Docker issue, than a node issue…?
Can you give the results of docker ps -a via command line. What version of docker for Mac are you using? I remember a while back it was recommended to use version 2.1.0.5, although this may not be related. The node was working fine before the downtime?
docker ps -a results
(I know, I’m not running Watchtower right now. I’ll put it back soon once everything’s working):
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9cb1f00a9f62 storjlabs/storagenode:latest "/entrypoint" 4 hours ago Up 3 hours 0.0.0.0:14002->14002/tcp, 0.0.0.0:28969->28967/tcp storagenode2
326d3d2e3ad5 storjlabs/storagenode:latest "/entrypoint" 4 hours ago Up 3 hours 0.0.0.0:14001->14002/tcp, 0.0.0.0:28968->28967/tcp storagenode1
b5cb4fdd1898 storjlabs/storagenode:latest "/entrypoint" 5 hours ago Up 3 hours 0.0.0.0:28967->28967/tcp, 0.0.0.0:14003->14002/tcp storagenode3
Interesting. When I launch the CLI dashboard I get this in the logs:
2021-02-10T21:10:39.123Z INFO Configuration loaded {"Location": "/app/config/config.yaml"}
2021-02-10T21:10:39.168Z INFO Identity loaded. {"Node ID": "removed"}
Have you made sure you are running the latest storagenode version? You can also modify the config.yaml file and change the log level to debug. There might be more info available. You will need to restart the node for this to take effect.
ok turned on debug logs, restarted. Definitely seeing some errors, but again, it does still seem to be working somewhat. Except that the dashboards don’t load.
Unfortunately I am running low on ideas. You could try checking your databases for errors (if you haven’t done so already). Although I would be surprised if all of your nodes have that same issue. Seems more likely a docker or configuration issue. You could try rolling back your docker desktop version to an earlier one.
Well there was nothing mentioned in the official install docs about which version of Docker to use, so at first, yes I just installed the latest. Which was 3.x something… Rolled it back to the last 2.x version, and then after searching the forum some more, I found that thread. I’ve since installed 2.1.0.5 and everything has been smooth sailing for ~24 hours.
I was migrating my nodes from a linux box to a Mac, and on the linux box everything “just worked” (haha) so I assumed on the Mac I could just install Docker and run with it. Bad assumption!
Might be a good idea to mention Docker version numbers in the setup documentation, @Alexey is that something you could look at?