Node offline - stuck on "performing bandwidth usage rollups"

When I started the node originally and it stayed online and transferred data ingress/egress, but now it stays offline. I’ve been troubleshooting my Raspberry Pi4 node for several days now. I’ve searched through the forums and have tried to get it running from the different solutions previously posted.

Here is my docker file:
docker run -d --restart unless-stopped --stop-timeout 300
-p 28967:28967
-p 127.0.0.1:14002:14002
-e WALLET=“0xca4450b33XXXXXeBb6eE78A7c8099”
-e EMAIL="rawzyXXX@gmail.com"
-e ADDRESS=“rawzy.ddns.net:28967
-e STORAGE=“1.7TB”
–log-opt max-size=50m
–log-opt max-file=10
–mount type=bind,source=/home/pi/.local/share/storj/identity/storagenode,destination=/app/identity
–mount type=bind,source=/mnt/storagenode,destination=/app/config
–name storagenode storjlabs/storagenode:beta

docker exec storagenode ls /app/identity: (above the 1500 and 1000 size)
ca.1589429666.cert
ca.cert
ca.key
identity.1589429666.cert
identity.cert
identity.key

df -h:
Filesystem Size Used Avail Use% Mounted on
/dev/root 27G 2.1G 24G 9% /
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 2.0G 8.9M 1.9G 1% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/mmcblk0p6 253M 53M 200M 21% /boot
/dev/sda1 1.8T 266M 1.7T 1% /mnt/storagenode
tmpfs 391M 0 391M 0% /run/user/1000

ls -lisa on /mnt/storagenode:
2 4 drwxr-xr-x 4 pi pi 4096 May 19 18:21 storagenode

docker ps -a: (note that I have stopped, removed, and re-created the container multiple times in my troubleshooting)
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4f7e2f17e9a1 storjlabs/watchtower “/watchtower storage…” 5 hours ago Up 5 hours watchtower
3b8b008fa3de storjlabs/storagenode:beta “/entrypoint” 5 hours ago Up 5 hours 127.0.0.1:14002->14002/tcp, 0.0.0.0:28967->28967/tcp storagenode

This is the entire log. On the previous re-creations it would constantly get stuck on the last entry.
docker logs storagenode:
2020-05-19T22:21:22.228Z INFO Configuration loaded {“Location”: “/app/config/config.yaml”}
2020-05-19T22:21:22.229Z INFO tracing disabled
2020-05-19T22:21:22.232Z INFO Operator email {“Address”: “XXXX@gmail.com”}
2020-05-19T22:21:22.232Z INFO Operator wallet {“Address”: “0xca4450b330C883XXXXc8099”}
2020-05-19T22:21:22.739Z INFO db.migration Database Version {“version”: 36}
2020-05-19T22:21:23.260Z INFO preflight:localtime start checking local system clock with trusted satellites’ system clock.
2020-05-19T22:21:24.497Z INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.
2020-05-19T22:21:24.498Z INFO bandwidth Performing bandwidth usage rollups
2020-05-19T22:21:24.498Z INFO trust Scheduling next refresh {“after”: “7h19m37.971152627s”}
2020-05-19T22:21:24.498Z INFO Node 1VYAocFcxUNEzPgY2ixWJk7dzJ4KUQHJAraV4Ck2LT6drH9o4a started
2020-05-19T22:21:24.499Z INFO Public server started on [::]:28967
2020-05-19T22:21:24.499Z INFO Private server started on 127.0.0.1:7778
2020-05-19T23:21:24.498Z INFO bandwidth Performing bandwidth usage rollups
2020-05-20T00:21:24.498Z INFO bandwidth Performing bandwidth usage rollups
2020-05-20T01:21:24.498Z INFO bandwidth Performing bandwidth usage rollups
2020-05-20T02:21:24.498Z INFO bandwidth Performing bandwidth usage rollups

docker exec -it storagenode ping -c 10 google.com = success

This one is interesting to me. This directory location is NOT present, but I don’t know how to redirect it.
docker exec -it storagenode /app/storagenode setup
2020-05-20T03:25:12.077Z INFO Configuration loaded {“Location”: “/root/.local/share/storj/storagenode/config.yaml”}
2020-05-20T03:25:12.077Z INFO tracing disabled
Error: storagenode configuration already exists (/root/.local/share/storj/storagenode)

docker exec -it storagenode /app/storagenode version:
2020-05-20T03:27:11.455Z INFO Configuration loaded {“Location”: “/root/.local/share/storj/storagenode/config.yaml”}
2020-05-20T03:27:11.456Z INFO tracing disabled
Release build
Version: v1.3.3
Build timestamp: 23 Apr 20 17:37 UTC
Git commit: b736d43f7e52aad2854872055261181f542d9120

docker exec -it storagenode /app/storagenode config shows the above directory location, but again, I’m not sure if that is secluded to the container or meant to be on the Pi itself. I say this because the rest of the config does not populate my wallet/ddns/email or any other pertinent information.

At this point I believe that the node is likely disqualified due to downtime, however I would like to figure out what is wrong with the device so I know how to fix it/what to look for if I do have to re-create the node.

Also, my port forwarding is working and I have ensured that no firewalls are in place.

What do you guys think?

Id say the issue is with a ddns that needs to be updated.

What does your web dashboard show? Check to see if your node is suspended or disqualified.

@nerdatwork - Storage Node Dashboard ( Node Version: v1.3.3 )

======================

ID 1VYAocFcxUNEzPgY2ixWJk7dzJ4KUQHJAraV4Ck2LT6drH9o4a
Last Contact OFFLINE
Uptime 6h13m17s

               Available         Used     Egress      Ingress
 Bandwidth           N/A     245.0 MB     4.0 MB     240.9 MB (since May 1)
      Disk        1.7 TB     194.7 MB

Internal 127.0.0.1:7778
External rawzy.ddns.net:28967

@deathlessdd - I didn’t look at that, let me make sure the noip duc is running.

  • update * …I can’t believe I didn’t think to look at that. This is what the problem was. Thank you. Let me drag my face across the keyboard for a couple more hours so I don’t forget haha.
1 Like

On a positive note, I know a lot more about docker and the node itself now lol.

Its always the littlethings that screwup everything.

That’s CLI dashboard not web dashboard. Try visiting 127.0.0.1:14002 in Chrome or firefox

Also can you show screenshot of that docker command. I want to make sure you aren’t using curly quotes in it. You can blur out the personal stuff.

@nerdatwork I’m using headless pi and I don’t have a web browser on it. The docker command is in the initial post :slight_smile:

Thanks again :slight_smile:

You should update it so you can view your dashboard on your local network. It has pretty useful information now.

I saw it but screenshot will show if you are using curly quotes or not in your docker command.

Sorry, wasn’t trying to be a jerk about that. Here it is.

I just changed the docker config to be able to do it. Thanks for the pointer!

Just stopped/removed/and restarted it

1 Like

Have the same problem, and on my dachboard it says offline. My ddns is updated

2020-06-22T20:42:26.875Z INFO preflight:localtime start checking local system clock with trusted satellites’ system clock.
2020-06-22T20:42:27.830Z INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.
2020-06-22T20:42:27.830Z INFO trust Scheduling next refresh {“after”: “7h17m55.256048319s”}
2020-06-22T20:42:27.830Z INFO bandwidth Performing bandwidth usage rollups
2020-06-22T20:42:27.830Z INFO Node xxxxxxxxxxxxxxxxxxxxxxxx started
2020-06-22T20:42:27.830Z INFO Public server started on [::]:28967
2020-06-22T20:42:27.831Z INFO Private server started on xxxxx

Try this checklist:

Woooow… wooooow. I just needed to restart my router… Time to call my
internet provider to get a new one. Thanks for help

1 Like