Anything that can be done , to make Storj a little more robust?

Node Operators troubleshooting
1

So i had switched my cable modem to a bridging modem and then installed an EdgeRouter X, and basically got Storj back online.
Then i was having additional problems with the ER-X and ended up returning it.
I wanted to continue to have as router between my homelab and the cable modem, so i used my spare etherport on my homelab server, and ran VyOS in a vm (which by the way, is the same OS they run in the ER-X, i found out during trouble shooting)
I got it basically working but it killed my internedt usage quota in about a weekend. And Storj had come back online while losing bandwidth, so ihad to make a decision and go back the cable-modem routing again.
And can you tell what happend, ince i’m posting this story here ? Yep you guessed it, Storj has not come back online.
After i initially went back to the cable modem, i did lose the ip address it was all working with beforehand, and have since fixed that, and can ping my outside name i register it too.
I’ve also checked the port-forwarding, with the tool you reference, and that says its ok too.
Now i can’t even get the Storj.local:14002 page to come up, gaaahhhh.
After all this is 2023 can’t it be more resilient and not take so much searching to find the answer ?

Thanks in advance i know Alexy probably has the answer on the tip of his tounge, Alexy just port some of that knowledcge into a command of some sort, to try and repair, it from the inside, lol.

Thanks in advance.

2

i can assure you, that its not storj being unstable, but your configuration…

did your lan ip change, if the online part is configured correctly, that would seem to be a possible place to look.

its a matter of the basic routing being configured.

i would suggest you run through this part of the storj setup, and double check everything.

it could also be something as simple as your docker run command or storage node configuration depending on how you run the node.
especially if your lan ip’s have changed.

1 Like
3

Also check your nodes logs for any reason why it might not be starting up. But yeah, I think routing issues are the most likely cause.

4

Ugghhh, my local ip address is static
my public ip address has not changed, and i have it registered on a domain that will get dynamically update if it does

I cannot open the :14002 webpage.
nothing got changed on Storj but since the web page doesn’t open it has to be something in there

| SGC
March 8 |

  • | - |

i can assure you, that its not storj being unstable, but your configuration…

did your lan ip change, if the online part is configured correctly, that would seem to be a possible place to look.

its a matter of the basic routing being configured.

i would suggest you run through this part of the storj setup, and double check everything.

docs.storj.io

Step 3. Setup Port Forwarding - Storj Node Operator Docs

  • Storj Node Operator Docs

it could also be something as simple as your docker run command or storage node configuration depending on how you run the node.
especially if your lan ip’s have changed.

5

here’s what looks inapropriate in the logs

2023-03-08T18:57:59.164Z INFO Configuration loaded {“Process”: “storagenode-updater”, “Location”: “/app/config/config.yaml”}
2023-03-08T18:57:59.165Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “contact.external-address”}
2023-03-08T18:57:59.165Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet”}
2023-03-08T18:57:59.165Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-bandwidth”}
2023-03-08T18:57:59.166Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “healthcheck.details”}
2023-03-08T18:57:59.166Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.email”}
2023-03-08T18:57:59.166Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.private-address”}
2023-03-08T18:57:59.167Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet-features”}
2023-03-08T18:57:59.167Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “healthcheck.enabled”}
2023-03-08T18:57:59.167Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-disk-space”}
2023-03-08T18:57:59.168Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.address”}
2023-03-08T18:57:59.168Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “console.address”}
2023-03-08T18:57:59.168Z INFO Invalid configuration file value for ke{“Process”: “storagenode-updater”, “Key”: “log.level”}
2023-03-08T18:57:59.169Z INFO Anonymized tracing enabled {“Process”: “storagenode-updater”}
2023-03-08T18:57:59.173Z INFO Running on version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.72.5”}
2023-03-08T18:57:59.180Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2023-03-08T18:57:59.217Z INFO Configuration loaded {“Process”: “storagenode”, “Location”: “/app/config/config.yaml”}
2023-03-08T18:57:59.218Z INFO Anonymized tracing enabled {“Process”: “storagenode”}
2023-03-08T18:57:59.219Z INFO Operator email {“Process”: “storagenode”, “Address”: “mocallins…”}
2023-03-08T18:57:59.220Z INFO Operator wallet {“Process”: “storagenode”, “Address”: “0x38…edited…b5”}
2023-03-08 18:58:00,221 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-03-08 18:58:00,222 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-03-08 18:58:00,222 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

but i don’t ever remember messing with a config file.

| SGC
March 8 |

  • | - |

i can assure you, that its not storj being unstable, but your configuration…

did your lan ip change, if the online part is configured correctly, that would seem to be a possible place to look.

its a matter of the basic routing being configured.

i would suggest you run through this part of the storj setup, and double check everything.

docs.storj.io

Step 3. Setup Port Forwarding - Storj Node Operator Docs

  • Storj Node Operator Docs

it could also be something as simple as your docker run command or storage node configuration depending on how you run the node.
especially if your lan ip’s have changed.

7

Does that get generated ?

8

actually i think this is normal…

as you can see later in the log it does catch the configuration.
and apply wallet and email and such

been a long time since i tinker with the storagenodes, lots have changed…

if memory serves the errors are when it starts up before it finished downloading the most recent version allowed for your storagenode.

so those can be disregarded.

the local storagenode web access should always work, even if the storagenode doesn’t get web access… this indicates so other kind of problem, again routing / firewall related usually.
maybe antivirus… some antivirus software’s come with firewalls today.

the 14002 webdash board is configurable and uses the local ip, if that isn’t working then nothing else certainly will…

getting that to work might be a good place to start…
i am pretty sure the node will keep rebooting until it gets online, but pretty sure the dashboard would work regardless, even if the node is in a restart loop until it gets connected to the storj network.

so i’m fairly certain the dashboard should work, most of the time…
but again its been a while since i tinkered with this…

however that would be where i would start, because its the closest and simplest.
fix that and maybe you will figure out why the rest of the setup stopped working…

personally my money is still on your lan ip of your storagenode changed, because you replaced your router with a cable modem and thus got a new dhcp server.
or don’t have a dhcp server at all :smiley: i guess is also an option but then i suspect nothing would work.

9

Those logs happen because the storagenode-updater uses the same config file as the storagenode itself, but doesn’t support a lot of the options for storagenode. You can ignore those. What are the last log lines you see?

Also, your email address is visible in the logs. You should remove that as bots constantly spider the web for email addresses and add them to spam lists.

2 Likes
10

I stated earlier i have a static ip address on my local subnet.

| SGC
March 8 |

  • | - |

mocallins:

2023-03-08T18:57:59.164Z INFO Configuration loaded {“Process”: “storagenode-updater”, “Location”: “/app/config/config.yaml”}
2023-03-08T18:57:59.165Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “contact.external-address”}
2023-03-08T18:57:59.165Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet”}
2023-03-08T18:57:59.165Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-bandwidth”}
2023-03-08T18:57:59.166Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “healthcheck.details”}
2023-03-08T18:57:59.166Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.email”}
2023-03-08T18:57:59.166Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.private-address”}
2023-03-08T18:57:59.167Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet-features”}
2023-03-08T18:57:59.167Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “healthcheck.enabled”}
2023-03-08T18:57:59.167Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-disk-space”}
2023-03-08T18:57:59.168Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.address”}
2023-03-08T18:57:59.168Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “console.address”}
2023-03-08T18:57:59.168Z INFO Invalid configuration file value for ke{“Process”: “storagenode-updater”, “Key”: “log.level”}

actually i think this is normal…

as you can see later in the log it does catch the configuration.
and apply wallet and email and such

mocallins:

2023-03-08T18:57:59.220Z INFO Operator wallet {“Process”: “storagenode”, “Address”: “0x38…edited…b5”}

been a long time since i tinker with the storagenodes, lots have changed…

if memory serves the errors are when it starts up before it finished downloading the most recent version allowed for your storagenode.

so those can be disregarded.

mocallins:

I cannot open the :14002 webpage.

the local storagenode web access should always work, even if the storagenode doesn’t get web access… this indicates so other kind of problem, again routing / firewall related usually.
maybe antivirus… some antivirus software’s come with firewalls today.

the 14002 webdash board is configurable and uses the local ip, if that isn’t working then nothing else certainly will…

getting that to work might be a good place to start…
i am pretty sure the node will keep rebooting until it gets online, but pretty sure the dashboard would work regardless, even if the node is in a restart loop until it gets connected to the storj network.

so i’m fairly certain the dashboard should work, most of the time…
but again its been a while since i tinkered with this…

however that would be where i would start, because its the closest and simplest.
fix that and maybe you will figure out why the rest of the setup stopped working…

personally my money is still on your lan ip of your storagenode changed, because you replaced your router with a cable modem and thus got a new dhcp server.
or don’t have a dhcp server at all :smiley:i guess is also an option but then i suspect nothing would work.

11

Ugghhh, well i have no idea right now, becasue i think i’ve probably really screwed it up, trying to fix some permissions issues with revocation.db

Who should be the owner of all these files ?

12

huh well maybe i didn’t damage it to bad:

2023-03-09T13:32:57.963Z INFO Invalid configuration file value for ke{“Process”: “storagenode-updater”, “Key”: “log.encoding”}
2023-03-09T13:32:57.964Z INFO Anonymized tracing enabled {“Process”: “storagenode-updater”}
2023-03-09T13:32:57.969Z INFO Running on version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.73.4”}
2023-03-09T13:32:57.973Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2023-03-09T13:32:58.023Z INFO Configuration loaded {“Process”: “storagenode”, “Location”: “/app/config/config.yaml”}
2023-03-09T13:32:58.024Z INFO Anonymized tracing enabled {“Process”: “storagenode”}
2023-03-09T13:32:58.026Z INFO Operator email {“Process”: “storagenode”, “Address”: “mo…om”}
2023-03-09T13:32:58.026Z INFO Operator wallet {“Process”: “storagenode”, “Address”: “0x38…4b5”}
2023-03-09 13:32:59,028 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-03-09 13:32:59,028 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-03-09 13:32:59,028 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

and no dashboard.

13

you should see downloaded and uploaded successful in the logs… or atleast downloaded…

the dashboard is independently configured, so the node can function without it being accessible, again most likely a configuration issue… not sure i’ve ever seen any other issues that would bring down the dashboard without the node being down.

14

I replied showing the end of the log

15

you should get something like this if the node actually goes online…
it will just keep rebooting if it isn’t connected to the internet correctly

2023-03-09T19:05:03.852Z INFO piecestore uploaded {“Process”: “storagenode”, “Piece ID”: “ZQKQEQDWHIZJ6WCAXJBWL6Z6J3UACINNVLNNK3NNNTIME4CAHKEQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 181504, “Remote Address”: “63.250.57.147:57144”}
2023-03-09T19:05:03.929Z INFO piecestore upload started {“Process”: “storagenode”, “Piece ID”: “UQ7E7ZHA5UFLVYA6IS5PMVKTJS3CWEBOWPL6CATPTR4KPWNPEZKQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Space”: 1730716933202, “Remote Address”: “63.250.57.147:51792”}
2023-03-09T19:05:04.094Z INFO piecestore upload started {“Process”: “storagenode”, “Piece ID”: “AKPAYSFG4VG7XNDYV3NSJRNSB53LO535R34C6RH3YH6STJMXM4BA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Space”: 1730716933202, “Remote Address”: “63.250.57.147:41260”}
2023-03-09T19:05:04.203Z INFO piecestore uploaded {“Process”: “storagenode”, “Piece ID”: “AKPAYSFG4VG7XNDYV3NSJRNSB53LO535R34C6RH3YH6STJMXM4BA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 26112, “Remote Address”: “63.250.57.147:41260”}
2023-03-09T19:05:05.380Z INFO piecestore downloaded {“Process”: “storagenode”, “Piece ID”: “LPC5CLG25B6CFEYGJ3AOEMRPCSBNUDY3T6UA4OLYKSJ5ALI3CE2Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 2319360, “Remote Address”: “63.250.57.147:57138”}

16

The dashboard is available on localhost only by default. So, if node is running, you can open a dashboard on the same host with URL http://localhost:14002, unless you changed a listening port.
So, please show your full docker run command with all options (you can mask the private information) between two new lines with three backticks, like this:

```
docker run -d ....
```

P.S.
You do not need to copy the message you respond to :slight_smile: , the forum marks the origin in your answer and you can see on what post it is actually replying.
You may also select a needed part of the message when you replying and click the Quote button, see

17

Awesome Alexey finally chimed in, lol.
I’m lazy so i put the command in a file

[storjnode]:</home/mike/> cat storjrunm
docker run -d --restart unless-stopped --stop-timeout 300
-p 28967:28967/tcp
-p 28967:28967/udp
-p 14002:14002 \

-e WALLET="0x38.edit.4b5" \
-e EMAIL="mo.edit.m" \

-e ADDRESS=“collinsinfotech.com:28967”
-e STORAGE=“2.7TB”
–user id -u:id -g
–mount type=bind,source=“/home/mike/.local/share/storj/identity/storagenode”,destination=/app/identity
–mount type=bind,source=“/data”,destination=/app/config
–name storagenode storjlabs/storagenode:latest
[storjnode]:</home/mike/>

1 Like
18

I copied the output as you asked, and saw nothing like this.

| SGC
March 9 |

  • | - |

you should get something like this if the node actually goes online…
it will just keep rebooting if it isn’t connected to the internet correctly

2023-03-09T19:05:03.852Z INFO piecestore uploaded {“Process”: “storagenode”, “Piece ID”: “ZQKQEQDWHIZJ6WCAXJBWL6Z6J3UACINNVLNNK3NNNTIME4CAHKEQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 181504, “Remote Address”: “63.250.57.147:57144”}
2023-03-09T19:05:03.929Z INFO piecestore upload started {“Process”: “storagenode”, “Piece ID”: “UQ7E7ZHA5UFLVYA6IS5PMVKTJS3CWEBOWPL6CATPTR4KPWNPEZKQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Space”: 1730716933202, “Remote Address”: “63.250.57.147:51792”}
2023-03-09T19:05:04.094Z INFO piecestore upload started {“Process”: “storagenode”, “Piece ID”: “AKPAYSFG4VG7XNDYV3NSJRNSB53LO535R34C6RH3YH6STJMXM4BA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Space”: 1730716933202, “Remote Address”: “63.250.57.147:41260”}
2023-03-09T19:05:04.203Z INFO piecestore uploaded {“Process”: “storagenode”, “Piece ID”: “AKPAYSFG4VG7XNDYV3NSJRNSB53LO535R34C6RH3YH6STJMXM4BA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 26112, “Remote Address”: “63.250.57.147:41260”}
2023-03-09T19:05:05.380Z INFO piecestore downloaded {“Process”: “storagenode”, “Piece ID”: “LPC5CLG25B6CFEYGJ3AOEMRPCSBNUDY3T6UA4OLYKSJ5ALI3CE2Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 2319360, “Remote Address”: “63.250.57.147:57138”}

19

then your node isn’t connecting to the storj network, however it does seem to start up correctly, so its most likely a routing issue, you should check all the steps and configurations related to that.

20

I’m pretty sure my network is fine, all my other devices including my main pc have no problems connecting or pinging my outside address.
Even the Storj node can ping it via the name, smh.

21

if you have multiple ip’s on the storagenode host, it can be a good idea to
define those in the docker run command, sometimes the node can decide to run on different IP’s than one expects.

-p 192.168.0.10:28967:28967/tcp
-p 192.168.0.10:28967:28967/udp
-p 192.168.0.10:14002:14002 \

would look something like that