Node Offline, no error in logs. Help!

Node Operators troubleshooting
1

Hi there SNO’s,

My node is offline after changing the Payment address in the configuration.
I am running a Linux CentOS 7 machine with multiple nodes, and one of them did not come online after the change.

It is not the firewall or port forwarding, as I exchanged the ports of two nodes, and still only the same node has issues (same node under a different port)

In the logs there is no error but the dashboards shows OFFLINE.

The steps I took to changes the Payment address:

  • stop node: docker stop -t 300 storagenode1
  • remove the node: docker rm storagenode1
  • create a new node with the configuration:
docker run -d --restart unless-stopped -p 28969:28967 \
    -p 127.0.0.1:14003:14002 \
    -e WALLET="myaddress" \
    -e EMAIL="my@email.com" \
    -e ADDRESS="my.personal.address.com:28969" \
    -e BANDWIDTH="1PB" \
    -e STORAGE="10TB" \
    --mount type=bind,source="/home/user/.local/share/storj/identity/storj_identity1",destination=/app/identity \
    --mount type=bind,source="/media/user/storj_folder1",destination=/app/config \
    --name storj_kappa storjlabs/storagenode:beta

Node info

Storage Node Dashboard ( Node Version: v0.29.3 )

======================

ID           12idSMzaHLhRQmUB7kk5GeRBAof6qsQpYCa2nYR7DCk9up5Q67V
Last Contact OFFLINE
Uptime       14m4s

                   Available       Used     Egress      Ingress
     Bandwidth        1.0 PB     1.8 TB     1.6 TB     253.0 GB (since Jan 1)
          Disk        7.7 TB     2.3 TB
Internal 127.0.0.1:7778

I have been a couple hours since it is offline, and I am afraid it might DQ soon, which would be too bad as it is carrying 2.2TB.

Anyone willing to support to debug this?

2

Can you show command above this with the -p option?

3

Note sure why it didn’t show before, but it does now:

docker run -d --restart unless-stopped -p 28969:28967 \
4

storjlabs/storagenode:beta “/entrypoint” 19 minutes ago Up 19 minutes 127.0.0.1:14003->14002/tcp, 0.0.0.0:28969->28967/tcp storagenode

Is this what it still shows ?

It should show this new name instead of storagenode

5

Shoot sorry, forgot to make things consistent. That is the docker from storj_kappa.

78fcdfb4a5a1        storjlabs/storagenode:beta   "/entrypoint"            37 minutes ago      Up 36 minutes             127.0.0.1:14003->14002/tcp, 0.0.0.0:28969->28967/tcp   storj_kappa
6

Can you confirm you have 6 files in this folder?

7

Confirmed:

-rw-r--r--. 1 lsage lsage  546 Jul 17  2019 ca.1563361970.cert
-rw-r--r--. 1 lsage lsage 1076 Jul 17  2019 ca.cert
-rw-------. 1 lsage lsage  241 Jul 17  2019 ca.key
-rw-r--r--. 1 lsage lsage 1088 Jul 17  2019 identity.1563361970.cert
-rw-r--r--. 1 lsage lsage 1618 Jul 17  2019 identity.cert
-rw-------. 1 lsage lsage  241 Jul 17  2019 identity.key
8

As there was an ISP network outage of 3.5 hours this month, I wouldn’t be surprised if my node has been disqualified by now :(.

9

Disqualification based on uptime is not in force currently.

Please show last 20 lines of your log.

docker logs --tail 20 storj_kappa

10

These are all the entries in the log file:

$ sudo docker logs storj_kappa
2020-01-19T13:46:56.982Z	INFO	Configuration loaded from: /app/config/config.yaml
2020-01-19T13:46:57.006Z	INFO	Operator email: EMAIL_REDACTED
2020-01-19T13:46:57.007Z	INFO	operator wallet: ADDRESS_REDACTED
2020-01-19T13:46:57.465Z	INFO	version	running on version v0.29.3
2020-01-19T13:46:57.474Z	INFO	db.migration	Database Version	{"version": 28}
2020-01-19T13:46:57.937Z	INFO	Node 12idSMzaHLhRQmUB7kk5GeRBAof6qsQpYCa2nYR7DCk9up5Q67V started
2020-01-19T13:46:57.937Z	INFO	Public server started on [::]:28968
2020-01-19T13:46:57.937Z	INFO	Private server started on 127.0.0.1:7778
2020-01-19T13:46:57.937Z	INFO	bandwidth	Performing bandwidth usage rollups
2020-01-19T13:46:57.937Z	INFO	trust	Scheduling next refresh	{"after": "4h44m48.181862607s"}
2020-01-19T13:46:58.004Z	INFO	piecestore:monitor	Remaining Bandwidth	{"bytes": 998185035401728}
2020-01-19T13:46:58.042Z	INFO	version	running on version v0.29.3
2020-01-19T14:01:58.368Z	INFO	version	running on version v0.29.3
2020-01-19T14:16:58.365Z	INFO	version	running on version v0.29.3
2020-01-19T14:31:58.369Z	INFO	version	running on version v0.29.3
11

You set your port to 28969 though :frowning:

13

Please, compare the WAN IP on the main page of your router with IP there: https://www.yougetsignal.com/tools/open-ports/
The IP should match.
Please, check your external address:

nslookup my.personal.address.com 8.8.8.8

The IP should be the same as your public and WAN IPs.

Also, please, make sure that the local IP of your PC is the same as used in your port forwarding rule. It could change during reboot of PC or your router.

14

Hi Alexey,

Thank you for your message! I have checked this rigorously, local and remote ip’s are the same. I confirmed that it has nothing to do with the port forwarding nor firewall, but exchanging the port of two of the nodes on the same machine.

But now it seems that even though I specify port 28969 in my config (just confirmed this), it is still looking for port 28968 as shown in the logs…

Any Ideas? @Alexey @nerdatwork?

Thanks for your support!

Apparently I am on my reply limit for my user on this forum… So I will have to update this post:

Stopped and removed the storj_kappa node, and created a new node: storj_kappa3, with the config above (port 28969!). But it still says listening on 28968…

$ sudo docker logs storj_kappa3
2020-01-19T14:52:32.642Z	INFO	Configuration loaded from: /app/config/config.yaml
2020-01-19T14:52:32.665Z	INFO	Operator email: REDACTED
2020-01-19T14:52:32.665Z	INFO	operator wallet: REDACTED
2020-01-19T14:52:33.148Z	INFO	version	running on version v0.29.3
2020-01-19T14:52:33.158Z	INFO	db.migration	Database Version	{"version": 28}
2020-01-19T14:52:33.631Z	INFO	trust	Scheduling next refresh	{"after": "6h15m26.223642696s"}
2020-01-19T14:52:33.631Z	INFO	bandwidth	Performing bandwidth usage rollups
2020-01-19T14:52:33.631Z	INFO	Node 12idSMzaHLhRQmUB7kk5GeRBAof6qsQpYCa2nYR7DCk9up5Q67V started
2020-01-19T14:52:33.631Z	INFO	Public server started on [::]:28968
2020-01-19T14:52:33.631Z	INFO	Private server started on 127.0.0.1:7778
2020-01-19T14:52:33.693Z	INFO	piecestore:monitor	Remaining Bandwidth	{"bytes": 998185035401728}
2020-01-19T14:52:33.739Z	INFO	version	running on version v0.29.3

Removing the config.yaml and restarting the node did the trick! Thank you @Alexey !

1 Like
15

Please, remove the config.yaml from the storage location and restart the node

16

had the exact same problem today on my raspberry pi 3…I have checked IPs, renamed container, removed config.yaml and only after a few restarts and some downtime it finally went back online…maybe something happened on Storj today? who knows, glat it’s back online!

1 Like
17

Offline can be only if your identity is not signed, or your network does not configured properly: doesn’t updated DDNS, changed public IP (and you did not use the DDNS), your local IP is changed, you forwarded a wrong port, your ISP placed you behind their NAT.

1 Like