Node dashboard not working

My web dashboard does not seem to work. It shows the page below, but never moves on.

If I stop the node and restart it becomes accessible again for a short time, then it hangs again.

Troubleshooting steps I already tried:

  • I checked the Database and it does not show any error
  • I checked the log and found two issues:

ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused

I understand this should be due to a satellite being down.

"error": "system clock is out of sync: clock off by 119.994759 minutes"

I checked my system’s clock and it looks all right.
Fri Oct 2 08:24:47 UTC 2020

I am on CET (i.e. my client shows the time above as 10:24:47 UTC+2). Can this be the issue?

Thanks for helping

Use utility like nettime to sync time

(for windows) https://www.timesynctool.com/
(for linux) you can search the forum or google

Thanks for helping.
The clock is already synced automatically. In fact the error shows that the clock is off by 2h exactly (the time zone difference between client and server).

I’m not even sure this is the issue because it does not explain why the dashboard works right after I restart the node.

Roberto

I would start by fixing the clock issue. Perhaps the time zone for your machine is not set properly?

Done. Same thing happens.
The dashboard works after reboot, then stops working 5 minutes later.

Is this a linux node? Check the iowait while this is happening?

iostat -y 5

You are looking for the %iowait. -y 5 means it will refresh every 5s until you end the command

It is Ubuntu.

%iowait fluctuates between 1 and 3%
image

Okay. Does the CLI dashboard work when the web dashboard is not working? And the results of docker ps -a when the web dash isn’t working?

CLI dashboard used to work (even if it took a while to fully load. 1 min or so).
Now, it does not seem to load any more. This is all it shows, then waits forever until ctrl-c.

ps -a (while both web and CLI dashboards are waiting endlessly)

PID TTY          TIME CMD

10826 pts/0 00:00:00 bash
10827 pts/0 00:00:00 docker
10853 pts/0 00:00:00 storagenode
11431 pts/1 00:00:00 ps

Nothing strange on docker:

My next steps would be to stop the node and check the disk for errors. Then I would check the databases for errors as well (step 5). Make sure you have the minimum version of sqlite installed, or the check may not work properly.

1 Like

I’m facing the same issue. This is the second time with this version (1.14.7). The first time I solved it by stopping the node which btw, it took more than one hour with the command “docker stop -t 300 storagenode”

Everything on the node seems to be working fine. Docker ok, the listenning ports ok as well:

But when i issue a telnet to port 14002 of the container, i get refused:
imagen

Something weird with this version, and moreover, is my node working fine? The port 28967 is listenning…
Is there some way to check my node, but not querying the node itself? I mean, checking the satellite directly or something else.

Thanks

The CLI dashboard and API

The issue should be fixed in the rolling release:

I’m not facing any issue in the loading screen. On the other hand, the issue is that the connection is being refused by container despite the port is shown as listening in netstat table.

Btw @Alexey, any coments concerning my question about checking the node stats but without having to query the node directly?

Thanks.

Have you tried this ?

mmm, I think that you haven’t read the post…

this is to check the dashboard of the node, which I say that is not working.

As you can see in my previous post, when I issued a telnet to the docker and port 14002, i got refused.

Apart of asking what can be the cause of this previous behavior, I was asking if there are other ways to see the node stats (by querying to the satellite or something else), because the port 28967, is still working fine, so right now, I’m not able to see if my node is working or not.

Why are you trying to use telnet to connect to docker though 14002 this is a webserver port not telnet

Telnet to a given port is used for troubleshooting purposes, to see if port is listening properly.

Only can be used with tcp services, but in this case we are talking about tcp 14002.

But your also using the IP that docker uses not though the local IP, Wouldnt you try to connect to the local IP not the docker IP?

Is the same. The host is natting the request to the container real ip. However, since i’m doing this from the host itself, I have direct visibility to the container, so I can do the telnet directly to its ip.

Just check it in your configuration.

But if your checking locally on the same system there shouldnt be an issue seeing if the port is open, less there is a config or a issue on that system.