Multi-Node Dashboard

One does eventually operate multiple nodes. What do you guys use to monitor them all at once?

Linux and docker running?

I have created a Grafana dashboard that sends me email alerts if one of my storage nodes doesn’t operate within my expectation.

1 Like

Yes. Ubuntu 20.04 and docker.

Does it work well? Was considering something like that and looking for good configs.

2 Likes

If you’re interested, you can help me extending my bash script, coz I have just one node in operation. I can provide some options and optimizations based on your needs and you can verify, if it works well. Nothing to install separately, just make sure, the required libraries are installed.

Check this - MND Dashboard - Payout Flow

you can install it by following this and taking the most recent binary from github
doesn’t quite do all the fancy stuff that grafana can do tho, but it is very light weight and i find it pretty reasonable.

Know I may be late to the party, but I can’t seem to get the api key necessary for MND. I’m on Windows, so don’t know if you have any experience, but any help would be dearly appreciated!

This is the closest I’ve gotten (command is running on the storj node itself, not multi node dashboard server, is that correct?)…:

C:\Program Files\Storj\Storage Node>storagenode.exe info --config-dir “C:\Program Files\Storj\Storage Node” --identity-dir “C:\Users\administrator.SVENDSEN\AppData\Roaming\Storj\Identity\storagenode”
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x131c5d4]

goroutine 1 [running]:
github[.]com/spacemonkeygo/monkit/v3.newSpan.func1(0xc00028f180)
/go/pkg/mod/github[.]com/spacemonkeygo/monkit/v3@v3.0.17/ctx.go:147 +0x30e
panic({0x1b4ef60, 0x1a69260})
/usr/local/go/src/runtime/panic.go:1047 +0x266
go.uber[.]org/zap.(*Logger).clone(…)
/go/pkg/mod/go.uber.org/zap@v1.16.0/logger.go:253
go.uber[.]org/zap.(*Logger).With(0x0, {0xc00018a0c0, 0x1ec2900, 0x1})
/go/pkg/mod/go.uber.org/zap@v1.16.0/logger.go:164 +0x54
main.main.func1(0x1c88bca)
/go/src/storj.io/storj/cmd/storagenode/main.go:476 +0xe5
storj.io/private/process.cleanup.func1(0x1a76320, {0xc00020dc40, 0x0, 0x4})
/go/pkg/mod/storj.io/private@v0.0.0-20220512110458-f0ade82ff3db/process/exec_conf.go:276 +0x882
github[.]com/spf13/cobra.(*Command).execute(0x1a76320, {0xc00020dc00, 0x4, 0x4})
/go/pkg/mod/github[.]com/spf13/cobra@v1.1.3/command.go:852 +0x60e
github[.]com/spf13/cobra.(*Command).ExecuteC(0x1a756a0)
/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x3ad
github[.]com/spf13/cobra.(*Command).Execute(…)
/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897
storj[.]io/private/process.ExecWithCustomConfigAndLogger(0x1a756a0, 0xc0, 0x101aefb80, 0xc00004a000)
/go/pkg/mod/storj.io/private@v0.0.0-20220512110458-f0ade82ff3db/process/exec_conf.go:93 +0x189
main.main()
/go/src/storj.io/storj/cmd/storagenode/main.go:479 +0x45

C:\Program Files\Storj\Storage Node>

Hello @KasperSvendsenGit ,
Welcome to the forum!

You may try to modify your command a little bit:

storagenode.exe info --config-dir "C:\Program Files\Storj\Storage Node\\" --identity-dir "C:\Users\administrator.SVENDSEN\AppData\Roaming\Storj\Identity\storagenode"

More like even this would be enough:

storagenode.exe info --config-dir "C:\Program Files\Storj\Storage Node\\"

Hi Alexey,

Thanks for your response, turns out it was because I was running the command as a non-admin user, and that was causing some permission issue in C:\Program Files …

I am however still struggling with adding a node to the dashboard. The web GUI’s “add-first-node” page doesn’t react when pasting in the information… The add node button doesn’t do anything…

Attempting to install via the json output from my node gives me the following error:

.\multinode.exe add .\storj-srv-01.json
2022-06-18T12:27:01.808+0200    INFO    Configuration loaded    {"Location": "C:\\Users\\administrator.SVENDSEN\\AppData\\Roaming\\Storj\\Multinode\\config.yaml"}
Error: illegal base64 data at input byte 4

I also noticed in the “multinode.exe add --help”, that I can enter the values of each property which seems to get me the closest. The service is started and according to the node’s own dashboard, it’s also online. TCP and UDP is forwarded to the local node’s IP:

.\multinode.exe add --api-secret "[Hidden for privacy]" --name "Storj-SRV-01" --node-id "[Hidden for privacy]" --public-address "[Hidden for privacy]:28967"
2022-06-18T12:41:31.597+0200    INFO    Configuration loaded    {"Location": "C:\\Users\\administrator.SVENDSEN\\AppData\\Roaming\\Storj\\Multinode\\config.yaml"}
Error: node is not reachable: rpc: tcp connector failed: rpc: dial tcp [Hidden for privacy]:28967: i/o timeout

When you adding a new node to the multinode dashboard you should use your external address and port.

This seems appear when you use an info command to generate an API key for some reason.
You can try several times (each time it generates a new IP) and pass it to the multinode dashboard. Or you can add manually via UI or with add command too.
You may also try to generate an API key with the issue-apikey command (it requires to specify an --identity-dir location).