ERROR dashboard.sh v1.77.2

[itbrigadier]

Hi all.
After updating to v1.77.2, the dashboard stopped working in the console.
Error text

$ sudo docker exec -it storagenode /app/dashboard.sh
2023-04-20T07:56:53.081Z        INFO    Configuration loaded    {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2023-04-20T07:56:53.081Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "contact.external-address"}
2023-04-20T07:56:53.081Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "identity.cert-path"}
2023-04-20T07:56:53.081Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "operator.wallet"}
2023-04-20T07:56:53.081Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "operator.email"}
2023-04-20T07:56:53.081Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "server.private-address"}
2023-04-20T07:56:53.081Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "server.address"}
2023-04-20T07:56:53.081Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "storage.allocated-disk-space"}
2023-04-20T07:56:53.081Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "identity.key-path"}
2023-04-20T07:56:53.081Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "storage.allocated-bandwidth"}
2023-04-20T07:56:53.081Z        INFO    Anonymized tracing enabled      {"Process": "storagenode"}
2023-04-20T07:56:53.082Z        FATAL   Failed to load identity.        {"Process": "storagenode", "error": "file or directory not found: open : no such file or directory", "errorVerbose": "file or directory not found: open : no such file or directory\n\tstorj.io/common/identity.Config.Load:326\n\tmain.cmdDashboard:80\n\tmain.newDashboardCmd.func1:49\n\tstorj.io/private/process.cleanup.func1.4:399\n\tstorj.io/private/process.cleanup.func1:417\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:113\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:79\n\tmain.main:26\n\truntime.main:250"}

Web console works

There are no errors in the logs.
OS Ubuntu 18.04 x64
This error on all updated nodes, restarting the container does not solve the problem.

[Alexey]

I notified the team. My nodes are not updated yet.
Could you please try

docker exec -it storagenode ./dashboard.sh --identity-dir identity

could you please also give a result of this command:

docker exec -it storagenode grep identity config/config.yaml

[itbrigadier]

sudo docker exec -it storagenode ./dashboard.sh --identity-dir identity
2023-04-20T08:46:13.561Z        INFO    Configuration loaded    {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2023-04-20T08:46:13.562Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "storage.allocated-disk-space"}
2023-04-20T08:46:13.562Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "identity.key-path"}
2023-04-20T08:46:13.562Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "server.address"}
2023-04-20T08:46:13.562Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "identity.cert-path"}
2023-04-20T08:46:13.562Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "server.private-address"}
2023-04-20T08:46:13.562Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "storage.allocated-bandwidth"}
2023-04-20T08:46:13.562Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "operator.wallet"}
2023-04-20T08:46:13.562Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "operator.email"}
2023-04-20T08:46:13.562Z        INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "contact.external-address"}
2023-04-20T08:46:13.562Z        INFO    Anonymized tracing enabled      {"Process": "storagenode"}
2023-04-20T08:46:13.563Z        FATAL   Failed to load identity.        {"Process": "storagenode", "error": "file or directory not found: open : no such file or directory", "errorVerbose": "file or directory not found: open : no such file or directory\n\tstorj.io/common/identity.Config.Load:326\n\tmain.cmdDashboard:80\n\tmain.newDashboardCmd.func1:49\n\tstorj.io/private/process.cleanup.func1.4:399\n\tstorj.io/private/process.cleanup.func1:417\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:113\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:79\n\tmain.main:26\n\truntime.main:250"}

sudo docker exec -it storagenode grep identity config/config.yaml
# path to the certificate chain for this identity
identity.cert-path: identity/identity.cert
# path to the private key for this identity
identity.key-path: identity/identity.key
# identity version(s) the server will be allowed to talk to

[elek]

Thanks the report @itbrigadier.

Seems to be broken by cmd/storagenode: refactor main.go · storj/storj@3cf8963 · GitHub

I created a patch to fix it: https://review.dev.storj.io/c/storj/storj/+/10249

Unfortunately, it will be included only in the next release…

[BrightSilence]

Would it be possible to stop the rollout of the current version to prevent this from breaking on more nodes until the fixed version is available? It’s kind of messing with how I monitor my nodes.

[Vadim]

there is also error on GUI version that i reported here.
Storagenode 1.77.2 wont stop - Node Operators - Storj Community Forum (official)

[elek]

Would it be possible to stop the rollout of the current version to prevent this from breaking on more nodes until the fixed version is available?

Sure thing. I asked to stop the rollout process.

[PocketSam]

Hello friends.
I’ve got a weird problem. After reinstalling an OS I’ve configured a node but discovered I can’t take a look at the node dashboard.

$ docker exec -t storagenode /app/dashboard.sh
2023-04-21T14:51:28.923Z INFO Configuration loaded {“Process”: “storagenode”, “Location”: “/app/config/config.yaml”}
2023-04-21T14:51:28.923Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “storage.allocated-bandwidth”}
2023-04-21T14:51:28.923Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “identity.key-path”}
2023-04-21T14:51:28.923Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “contact.external-address”}
2023-04-21T14:51:28.923Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “operator.email”}
2023-04-21T14:51:28.923Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “operator.wallet”}
2023-04-21T14:51:28.923Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “identity.cert-path”}
2023-04-21T14:51:28.923Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “server.address”}
2023-04-21T14:51:28.923Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “storage.allocated-disk-space”}
2023-04-21T14:51:28.923Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “server.private-address”}
2023-04-21T14:51:28.923Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “operator.wallet-features”}
2023-04-21T14:51:28.923Z INFO Anonymized tracing enabled {“Process”: “storagenode”}
2023-04-21T14:51:28.925Z FATAL Failed to load identity. {“Process”: “storagenode”, “error”: “file or directory not found: open : no such file or directory”, “errorVerbose”: “file or directory not found: open : no such file or directory\n\tstorj.io/common/identity.Config.Load:326\n\tmain.cmdDashboard:80\n\tmain.newDashboardCmd.func1:49\n\tstorj.io/private/process.cleanup.func1.4:399\n\tstorj.io/private/process.cleanup.func1:417\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:113\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:79\n\tmain.main:26\n\truntime.main:250”}

That’s weird as it looks like node works fine, but I think this is a serious problem anyway.
To run a node I use this command:
$ docker run -d --restart unless-stopped --privileged --stop-timeout 600 -p 28967:28967 -p 28967:28967/udp -p 14002:14002 -e WALLET=“0x00000000000000000000000000” -e EMAIL=“email@domain.tld” -e ADDRESS=“domain.address.site:28967” -e STORAGE=“6TB” --mount type=bind,source=/storj/identity/,destination=/app/identity --mount type=bind,source=/storj/config/,destination=/app/config --name storagenode storjlabs/storagenode:latest

All files required to run a node were stored in one folder. So the command to run a node would be the same.
ls -lAh /storj/identity/
total 56K
-rw-r–r-- 1 sam sam 558 Nov 19 2021 ca.1637314033.cert
-rw-r–r-- 1 sam sam 1.1K Nov 19 2021 ca.cert
-rw------- 1 sam sam 241 Nov 19 2021 ca.key
-rw-r–r-- 1 sam sam 1.1K Nov 19 2021 identity.1637314033.cert
-rw-r–r-- 1 sam sam 1.6K Nov 19 2021 identity.cert
-rw------- 1 sam sam 241 Nov 19 2021 identity.key
-rw------- 1 sam sam 32K Nov 19 2021 revocations.db
drwxr–r-- 2 sam sam 240 Nov 19 2021 storagenode

Permissions also seem fine for me.

Any suggestions are welcome.
Thanks!

[PocketSam]

From inside of the container it also can see files
sam@storj:/storj$ docker exec -it storagenode bash
root@8f11226bd001:/app# ls identity/
ca.1637314033.cert ca.cert ca.key identity.1637314033.cert identity.cert identity.key revocations.db storagenode
root@8f11226bd001:/app#

[Roberto]

ERROR dashboard.sh v1.77.2

Hi all. After updating to v1.77.2, the dashboard stopped working in the console. Error text $ sudo docker exec -it storagenode /app/dashboard.sh 2023-04-20T07:56:53.081Z INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/config.yaml"} 2023-04-20T07:56:53.081Z INFO Invalid configuration file key {"Process": "storagenode", "Key": "contact.external-address"} 2023-04-20T07:56:53.081Z INFO Invalid configuration file key {"Process": "sto…

[Stob]

Did this rollout get stopped?

With reports that the Windows service is not stopping correctly this could cause issues for the rollout of any future versions with the fixes.

Storagenode 1.77.2 wont stop

I try to stop service, win gui service writing stopping that all, and work even dashboard is access able [stoping] .

[Alexey]

I think so, but wait for Monday.

[cubesmi]

Only one of my 4 nodes updated to v1.77.2 on the same host & watchtower instance, so I guess the rollout was stopped.

According to logs the v1.77.2 node seems to be working fine apart from the dashboard defect. Can someone confirm that the defect only affects the console dashboard?

[donald.m.motsinger]

--identity-dir /app/identity --config-dir /app/config

doesn’t solve the dashboard problem.

[brizio71]

Hello, I have the same issue:

docker exec -it storagenode /app/dashboard.sh

2023-04-27T06:39:20.020Z INFO Configuration loaded {“Process”: “storagenode”, “Location”: “/app/config/config.yaml”}
2023-04-27T06:39:20.024Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “contact.external-address”}
2023-04-27T06:39:20.026Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “healthcheck.details”}
2023-04-27T06:39:20.026Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “console.address”}
2023-04-27T06:39:20.026Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “version.server-address”}
2023-04-27T06:39:20.026Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “storage.allocated-bandwidth”}
2023-04-27T06:39:20.026Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “server.private-address”}
2023-04-27T06:39:20.027Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “identity.key-path”}
2023-04-27T06:39:20.027Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “server.address”}
2023-04-27T06:39:20.027Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “identity.cert-path”}
2023-04-27T06:39:20.027Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “operator.wallet”}
2023-04-27T06:39:20.027Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “storage.allocated-disk-space”}
2023-04-27T06:39:20.027Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “healthcheck.enabled”}
2023-04-27T06:39:20.027Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “operator.wallet-features”}
2023-04-27T06:39:20.027Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “operator.email”}
2023-04-27T06:39:20.028Z INFO Anonymized tracing enabled {“Process”: “storagenode”}
2023-04-27T06:39:20.031Z FATAL Failed to load identity. {“Process”: “storagenode”, “error”: “file or directory not found: open : no such file or directory”, “errorVerbose”: “file or directory not found: open : no such file or directory\n\tstorj.io/common/identity.Config.Load:326\n\tmain.cmdDashboard:80\n\tmain.newDashboardCmd.func1:49\n\tstorj.io/private/process.cleanup.func1.4:399\n\tstorj.io/private/process.cleanup.func1:417\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:113\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:79\n\tmain.main:26\n\truntime.main:250”}

what is the procedure to install the patch ?

Thanks

[Alexey]

Wait for the automatic update. There is no other way.

[brizio71]

ok but I will have some issue on uptime or any other actvities on node ?

Does the new version already release ? what is the version number ?

[donald.m.motsinger]

@brizio71

ok but I will have some issue on uptime or any other actvities on node ?

The node runs fine. This bug only affects the cli dashboard.

[neicedai]

root@Ubuntu-2204-jammy-amd64-base /usr/local/bin # ./storagenode dashboard --address 127.0.0.1:47778
2023-04-27T09:49:20.133Z INFO Configuration loaded {“Process”: “storagenode”, “Location”: “/root/.local/share/storj/storagenode/config.yaml”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “storage.allocated-bandwidth”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “contact.external-address”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “operator.email”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “identity.key-path”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “server.private-address”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “operator.wallet”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “console.address”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “operator.wallet-features”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “identity.cert-path”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “storage.allocated-disk-space”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “server.address”}
2023-04-27T09:49:20.133Z INFO Anonymized tracing enabled {“Process”: “storagenode”}
2023-04-27T09:49:20.133Z FATAL Failed to load identity. {“Process”: “storagenode”, “error”: “file or directory not found: open : no such file or directory”, “errorVerbose”: “file or directory not found: open : no such file or directory\n\tstorj.io/common/identity.Config.Load:326\n\tmain.cmdDashboard:80\n\tmain.newDashboardCmd.func1:49\n\tstorj.io/private/process.cleanup.func1.4:399\n\tstorj.io/private/process.cleanup.func1:417\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:113\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:79\n\tmain.main:26\n\truntime.main:250”}
Need to wait new release?

[Alexey]

Hello @neicedai,
Welcome to the forum!

ERROR dashboard.sh v1.77.2

Wait for the automatic update. There is no other way.