ERROR dashboard.sh v1.77.2

PocketSam · April 21, 2023, 3:07pm

From inside of the container it also can see files
sam@storj:/storj$ docker exec -it storagenode bash
root@8f11226bd001:/app# ls identity/
ca.1637314033.cert ca.cert ca.key identity.1637314033.cert identity.cert identity.key revocations.db storagenode
root@8f11226bd001:/app#

Roberto · April 21, 2023, 5:49pm

Stob · April 22, 2023, 11:39am

Did this rollout get stopped?

With reports that the Windows service is not stopping correctly this could cause issues for the rollout of any future versions with the fixes.

Alexey · April 22, 2023, 12:01pm

I think so, but wait for Monday.

cubesmi · April 22, 2023, 2:28pm

Only one of my 4 nodes updated to v1.77.2 on the same host & watchtower instance, so I guess the rollout was stopped.

According to logs the v1.77.2 node seems to be working fine apart from the dashboard defect. Can someone confirm that the defect only affects the console dashboard?

donald.m.motsinger · April 23, 2023, 5:31pm

--identity-dir /app/identity --config-dir /app/config

doesn’t solve the dashboard problem.

brizio71 · April 27, 2023, 6:42am

Hello, I have the same issue:

docker exec -it storagenode /app/dashboard.sh

2023-04-27T06:39:20.020Z INFO Configuration loaded {“Process”: “storagenode”, “Location”: “/app/config/config.yaml”}
2023-04-27T06:39:20.024Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “contact.external-address”}
2023-04-27T06:39:20.026Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “healthcheck.details”}
2023-04-27T06:39:20.026Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “console.address”}
2023-04-27T06:39:20.026Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “version.server-address”}
2023-04-27T06:39:20.026Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “storage.allocated-bandwidth”}
2023-04-27T06:39:20.026Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “server.private-address”}
2023-04-27T06:39:20.027Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “identity.key-path”}
2023-04-27T06:39:20.027Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “server.address”}
2023-04-27T06:39:20.027Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “identity.cert-path”}
2023-04-27T06:39:20.027Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “operator.wallet”}
2023-04-27T06:39:20.027Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “storage.allocated-disk-space”}
2023-04-27T06:39:20.027Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “healthcheck.enabled”}
2023-04-27T06:39:20.027Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “operator.wallet-features”}
2023-04-27T06:39:20.027Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “operator.email”}
2023-04-27T06:39:20.028Z INFO Anonymized tracing enabled {“Process”: “storagenode”}
2023-04-27T06:39:20.031Z FATAL Failed to load identity. {“Process”: “storagenode”, “error”: “file or directory not found: open : no such file or directory”, “errorVerbose”: “file or directory not found: open : no such file or directory\n\tstorj.io/common/identity.Config.Load:326\n\tmain.cmdDashboard:80\n\tmain.newDashboardCmd.func1:49\n\tstorj.io/private/process.cleanup.func1.4:399\n\tstorj.io/private/process.cleanup.func1:417\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:113\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:79\n\tmain.main:26\n\truntime.main:250”}

what is the procedure to install the patch ?

Thanks

Alexey · April 27, 2023, 7:30am

Wait for the automatic update. There is no other way.

brizio71 · April 27, 2023, 7:53am

ok but I will have some issue on uptime or any other actvities on node ?

Does the new version already release ? what is the version number ?

donald.m.motsinger · April 27, 2023, 9:24am

The node runs fine. This bug only affects the cli dashboard.

neicedai · April 27, 2023, 9:54am

root@Ubuntu-2204-jammy-amd64-base /usr/local/bin # ./storagenode dashboard --address 127.0.0.1:47778
2023-04-27T09:49:20.133Z INFO Configuration loaded {“Process”: “storagenode”, “Location”: “/root/.local/share/storj/storagenode/config.yaml”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “storage.allocated-bandwidth”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “contact.external-address”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “operator.email”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “identity.key-path”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “server.private-address”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “operator.wallet”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “console.address”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “operator.wallet-features”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “identity.cert-path”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “storage.allocated-disk-space”}
2023-04-27T09:49:20.133Z INFO Invalid configuration file key {“Process”: “storagenode”, “Key”: “server.address”}
2023-04-27T09:49:20.133Z INFO Anonymized tracing enabled {“Process”: “storagenode”}
2023-04-27T09:49:20.133Z FATAL Failed to load identity. {“Process”: “storagenode”, “error”: “file or directory not found: open : no such file or directory”, “errorVerbose”: “file or directory not found: open : no such file or directory\n\tstorj.io/common/identity.Config.Load:326\n\tmain.cmdDashboard:80\n\tmain.newDashboardCmd.func1:49\n\tstorj.io/private/process.cleanup.func1.4:399\n\tstorj.io/private/process.cleanup.func1:417\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:113\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:79\n\tmain.main:26\n\truntime.main:250”}
Need to wait new release?

Alexey · April 28, 2023, 7:33am

Hello @neicedai,
Welcome to the forum!

Miglecz · April 28, 2023, 7:08am

Hello,

There is a defect related to v1.77.2.
It is not present in previous v1.76.2.
My configuration is RaspberryPI4 with latest everything.

Steps to reproduce:

command line execution: docker exec -it storagenode /app/dashboard.sh

Expected result: displayed “Storage Node Dashboard” refreshing in a few seconds with “Status ONLINE”

Actual result:

2023-04-28T07:00:22.251Z        FATAL   Failed to load identity.        {"Process": "storagenode", "error": "file or directory not found: open : no such file or directory", "errorVerbose": "file or directory not found: open : no such file or directory\n\tstorj.io/common/identity.Config.Load:326\n\tmain.cmdDashboard:80\n\tmain.newDashboardCmd.func1:49\n\tstorj.io/private/process.cleanup.func1.4:399\n\tstorj.io/private/process.cleanup.func1:417\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:113\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:79\n\tmain.main:26\n\truntime.main:250"}

Web dashboard:
While the web dashboard describes “STATUS Online” as expected, without problems.

Stob · April 28, 2023, 7:18am

This has already been reported and there is a new version to fix. However that version (1.77.3) also had a problem on Windows nodes which needs another fix and has delayed the new version (1.77.6 / 1.78.x) rollout.

nerdatwork · May 3, 2023, 10:27am

Hiii Awesome Alexey,

I hope your day is going great so far. Here I am with an odd issue.

This is what the log shows.

2023-05-03T10:11:52.597Z INFO piecestore download started               { "Piece ID": "POM2YLYDA6BWVXI2GNUTFD25VFBLOIT3TSF5BH2DKL56VHZOZ2ZQ", "Satellite ID": EU1 "Action": "GET", "Offset": 390912, "Size": 2304, "Remote Address": "172.17.0.1:42464"}
2023-05-03T10:11:52.693Z INFO piecestore downloaded                     { "Piece ID": "POM2YLYDA6BWVXI2GNUTFD25VFBLOIT3TSF5BH2DKL56VHZOZ2ZQ", "Satellite ID": EU1 "Action": "GET", "Offset": 390912, "Size": 2304, "Remote Address": "172.17.0.1:42464"}
2023-05-03T10:11:55.681Z INFO Configuration loaded      { "Location": "/app/config/config.yaml"}
2023-05-03T10:11:55.682Z INFO Invalid configuration file key    { "Key": "storage2.orders.sender-dial-timeout"}
2023-05-03T10:11:55.682Z INFO Invalid configuration file key    { "Key": "server.address"}
2023-05-03T10:11:55.682Z INFO Invalid configuration file key    { "Key": "storage.allocated-bandwidth"}
2023-05-03T10:11:55.682Z INFO Invalid configuration file key    { "Key": "identity.cert-path"}
2023-05-03T10:11:55.683Z INFO Invalid configuration file key    { "Key": "kademlia.external-address"}
2023-05-03T10:11:55.683Z INFO Invalid configuration file key    { "Key": "storage2.trust.exclusions"}
2023-05-03T10:11:55.683Z INFO Invalid configuration file key    { "Key": "kademlia.operator.email"}
2023-05-03T10:11:55.683Z INFO Invalid configuration file key    { "Key": "storage.allocated-disk-space"}
2023-05-03T10:11:55.683Z INFO Invalid configuration file key    { "Key": "server.private-address"}
2023-05-03T10:11:55.683Z INFO Invalid configuration file key    { "Key": "identity.key-path"}
2023-05-03T10:11:55.683Z INFO Invalid configuration file key    { "Key": "kademlia.operator.wallet"}
2023-05-03T10:11:55.683Z INFO Anonymized tracing enabled        { "storagenode"}
2023-05-03T10:11:55.684Z FATAL Failed to load identity.         { "error": "file or directory not found: open : no such file or directory", "errorVerbose": "file or directory not found: open : no such file or directory\n\tstorj.io/common/identity.Config.Load:326\n\tmain.cmdDashboard:80\n\tmain.newDashboardCmd.func1:49\n\tstorj.io/private/process.cleanup.func1.4:399\n\tstorj.io/private/process.cleanup.func1:417\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:113\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:79\n\tmain.main:26\n\truntime.main:250"}
2023-05-03T10:11:55.772Z INFO piecestore download started               { "Piece ID": "DPIYUEH2CHJMXUJSPSW3CJYLSV7S4ELHGDJ4AMFE5ZLJ26IO63MQ", "Satellite ID": US1 "Action": "GET", "Offset": 0, "Size": 7936, "Remote Address": "172.17.0.1:42466"}
2023-05-03T10:11:55.945Z INFO piecestore download canceled              { "Piece ID": "DPIYUEH2CHJMXUJSPSW3CJYLSV7S4ELHGDJ4AMFE5ZLJ26IO63MQ", "Satellite ID": US1 "Action": "GET", "Offset": 0, "Size": 0, "Remote Address": "172.17.0.1:42466"}
2023-05-03T10:11:56.852Z INFO piecestore download started               { "Piece ID": "6P6NSK77CPSI7YIIG2U2AT7II5YX3GPGNLW2TLXT4SPF4QZBV5HA", "Satellite ID": AP1 "Action": "GET_REPAIR", "Offset": 0, "Size": 512, "Remote Address": "172.17.0.1:42470"}
2023-05-03T10:11:57.025Z INFO piecestore downloaded                     { "Piece ID": "6P6NSK77CPSI7YIIG2U2AT7II5YX3GPGNLW2TLXT4SPF4QZBV5HA", "Satellite ID": AP1 "Action": "GET_REPAIR", "Offset": 0, "Size": 512, "Remote Address": "172.17.0.1:42470"}

Docker run command

docker run -d --restart unless-stopped --stop-timeout 300 -p 14002:14002 -p 28967:28967/tcp -p 28967:28967/udp -p 127.0.0.1:5999:5999 -e WALLET=“XX” -e EMAIL=“XX” -e ADDRESS=“XX:28967” -e STORAGE=“7.9TB” --mount type=bind,source=“C:\Users\XXX\AppData\Roaming\Storj\Identity\storagenode”,destination=/app/identity --mount type=bind,source=“F:\StorjV3”,destination=/app/config --name storagenode storjlabs/storagenode:latest

Here are the contents of identity folder even though the error says file not found.

Its just the CLI dashboard that won’t load but web dashboard has no issues.

Node works without any issues as seen in log. My powershell script for dashboard also works without any issue.

Alexey · May 4, 2023, 2:23am

Hello @nerdatwork,
as you may see it’s a known issue.

nerdatwork · May 4, 2023, 2:29am

Thank you. This thread never showed up in my feed

Alexey · May 4, 2023, 3:39am

There is also another one:

elek · May 8, 2023, 2:10pm

Update:

This dashboard problem should be fixed from v1.77.3 / v1.78.x → already rolled out to the same 30% percentage of the storagenodes where the v1.77.2 rollout started (but stopped). If you use the updater, the dashboard should work for you…
The windows service restart problem is under review, should be part of v1.78.3 (or v1.79, but I would prefer to release it ASAP as a point release)

Alexey · May 11, 2023, 3:52am

5 posts were split to a new topic: My node has been displaying offline