[Tech Preview] Storage Node Multinode Dashboard

littleskunk · February 12, 2021, 6:54pm

This is a very early test version of the multinode dashboard. Yes, it is difficult to install and not all of you will have the tools required to compile this. Yes, the multinode dashboard has a lot of bugs and is essentially useless at the moment. If you still want to try it here is what you have to do.

First, compile it including the web interface. For the moment we have to use the latest release branch for that. At some point, we will publish binaries on github so that you can download und setup the multinode dashboard without having to compile it.

git clone https://github.com/storj/storj -b release-v1.34 $HOME/multinode
cd $HOME/multinode
go install ./cmd/multinode
go install ./cmd/identity
cd web/multinode
npm install
npm run build

After that let’s set up the multinode dashboard. Yes, we are aware that this step should be 2 lines and not 4. We are working on that already.

identity create multinode --difficulty 1
multinode setup --console.static-dir "$HOME/multinode/web/multinode"
multinode create-schema
multinode run

Now you should be able to see the multinode dashboard running on port 15002 by default. Open it in your webbrowser. In order to add a storage node to the multinode dashboard you need a storage node api key.

Please only add storage nodes that are running at least v1.22.2. If you add an older storage node the multinode dashboard will break and stop displaying even the nodes you had added successfully earlier. You would be forced to manually delete the entry from the SQLite database.

storagenode issue-apikey --log.output stdout --config-dir ... --identity-dir ...

After adding your first storage node you will notice that the dashboard is showing only a very limited amount of information and in many fields even display wrong values. For the moment I would just ignore these details. This early test is more about getting you prepared for the next round.

Good luck everyone.

hoarder · February 12, 2021, 7:24pm

Is there a possibility of dashboard actually breaking something on nodes or it’s just a display tool?

littleskunk · February 12, 2021, 7:25pm

It should be display only. If you manage to crash your storage node we would have to fix that asap.

kevink · February 12, 2021, 7:26pm

Is it possible to post some pictures for those who don’t want to go through the installation process? (or can’t)

hoarder · February 12, 2021, 7:28pm

Alright, will test it on nodes I don’t value

hoarder · February 12, 2021, 8:36pm

It worked well up until the point of me adding the node. Docker is still on 1.21.2. Too bad.

Can it use internal dashboard instead of going over internet?

littleskunk · February 12, 2021, 8:58pm

Yes it should. I would expect to be able to add localhost:13010 (storj-sim port mapping)

You can open the sqlite3 db. .schema will help you.

hoarder · February 12, 2021, 9:10pm

Yes, the unfortunate issue is all my nodes run in docker so I don’t have a node to connect to yet. Maybe one day I’ll move them to native linux.

itbrigadier · February 13, 2021, 8:59am

Hello.
Ubuntu 18.04

    root@ubuntu:~/multinode# go install ./cmd/multinode
cmd/multinode/main.go:11:2: cannot find package "github.com/spf13/cobra" in any of:
        /usr/src/github.com/spf13/cobra (from $GOROOT)
        /root/go/src/github.com/spf13/cobra (from $GOPATH)
cmd/multinode/main.go:12:2: cannot find package "github.com/zeebo/errs" in any of:
        /usr/src/github.com/zeebo/errs (from $GOROOT)
        /root/go/src/github.com/zeebo/errs (from $GOPATH)
cmd/multinode/main.go:13:2: cannot find package "go.uber.org/zap" in any of:
        /usr/src/go.uber.org/zap (from $GOROOT)
        /root/go/src/go.uber.org/zap (from $GOPATH)
cmd/multinode/main.go:15:2: cannot find package "storj.io/common/fpath" in any of:
        /usr/src/storj.io/common/fpath (from $GOROOT)
        /root/go/src/storj.io/common/fpath (from $GOPATH)
cmd/multinode/main.go:16:2: cannot find package "storj.io/private/cfgstruct" in any of:
        /usr/src/storj.io/private/cfgstruct (from $GOROOT)
        /root/go/src/storj.io/private/cfgstruct (from $GOPATH)
cmd/multinode/main.go:17:2: cannot find package "storj.io/private/process" in any of:
        /usr/src/storj.io/private/process (from $GOROOT)
        /root/go/src/storj.io/private/process (from $GOPATH)
cmd/multinode/main.go:18:2: cannot find package "storj.io/storj/multinode" in any of:
        /usr/src/storj.io/storj/multinode (from $GOROOT)
        /root/go/src/storj.io/storj/multinode (from $GOPATH)
cmd/multinode/main.go:19:2: cannot find package "storj.io/storj/multinode/multinodedb" in any of:
        /usr/src/storj.io/storj/multinode/multinodedb (from $GOROOT)
        /root/go/src/storj.io/storj/multinode/multinodedb (from $GOPATH)

littleskunk · February 13, 2021, 5:29pm

Your golang installation is incorrect.

itbrigadier · February 13, 2021, 10:54pm

Thanks, it worked.
We are waiting for the docker update on 1.22.2
screenshot

madest · February 14, 2021, 8:32am

I apologize for the stupid question, but where can I find the api key for the node?

littleskunk · February 14, 2021, 11:01am

How about the search function? Inlcuding my post now there are 4 hits for “api” in this threads. I am sure you can find the other 2 hits right at the beginning

madest · February 14, 2021, 4:56pm

yes thanks for the hint. set everything up, looked, closed

fmoledina · February 18, 2021, 7:22pm

I’m testing this out with a Docker node and I’m unable to add a node. I get the following error:

2021-02-18T12:15:44.326-0700    ERROR   console:endpoint        controllers/nodes.go:190        list node infos internal error  {"error": "nodes service error: rpc: tls: first record does not look like a TLS handshake", "errorVerbose": "nodes service error: rpc: tls: first record does not look like a TLS handshake\n\tstorj.io/common/rpc.TCPConnector.DialContext:91\n\tstorj.io/common/rpc.Dialer.dialEncryptedConn:180\n\tstorj.io/common/rpc.Dialer.DialNodeURL.func1:101\n\tstorj.io/common/rpc/rpcpool.(*Pool).Get:87\n\tstorj.io/common/rpc.Dialer.dialPool:146\n\tstorj.io/common/rpc.Dialer.DialNodeURL:100\n\tstorj.io/storj/multinode/nodes.(*Service).ListInfos.func1:90\n\tstorj.io/storj/multinode/nodes.(*Service).ListInfos:149\n\tstorj.io/storj/multinode/console/controllers.(*Nodes).ListInfos:188\n\tnet/http.HandlerFunc.ServeHTTP:2042\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2843\n\tnet/http.(*conn).serve:1925"}
storj.io/storj/multinode/console/controllers.(*Nodes).ListInfos
        /root/multinode/multinode/console/controllers/nodes.go:190
net/http.HandlerFunc.ServeHTTP
        /usr/lib/go-1.15/src/net/http/server.go:2042
github.com/gorilla/mux.(*Router).ServeHTTP
        /root/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210
net/http.serverHandler.ServeHTTP
        /usr/lib/go-1.15/src/net/http/server.go:2843
net/http.(*conn).serve
        /usr/lib/go-1.15/src/net/http/server.go:1925

The console endpoint for the storagenode is http-only and is available via http://<IP>:<PORT> on a different machine on the LAN.

Any insight on how to resolve this? Thanks.

littleskunk · February 23, 2021, 11:34am

You need to add the private port. So the port that you are sending to the satellite is also the port you have to enter in the multinode dashboard.

littleskunk · February 23, 2021, 11:38am

I have updated my multinode dashboard to the latest version and I also added all of my storage nodes. I can now see the free space and the traffic of all of my nodes. I was watching especially my free space. That is working quite good. I hope soon the payout column will also get corrected.

I am now running my multinode dashboard as a service similar to the storage node service.

# This is a SystemD unit file for the Storage Node
# To configure:
# - Update the user and group that the service will run as (User & Group below)
# - Ensure that the Storage Node binary is in /usr/local/bin and is named storagenode (or edit the ExecStart line
#   below to reflect the name and location of your binary
# - Ensure that you've run setup and have edited the configuration appropriately prior to starting the
#   service with this script
# To use:
# - Place this file in /etc/systemd/system/ or wherever your SystemD unit files are stored
# - Run systemctl daemon-reload
# - To start run systemctl start storagenode

[Unit]
Description  = Storage Node service
After        = network-online.target
Wants        = network-online.target

[Service]
User         = storagenode
Group        = storagenode
ExecStart    = /home/storagenode/go/bin/multinode run
Restart      = always
Type         = simple
NotifyAccess = main

[Install]
WantedBy     = multi-user.target

And I am writing everything into a logfile. On my multinode config file:

# can be stdout, stderr, or a filename
log.output: /home/storagenode/multinode1.log

fmoledina · February 23, 2021, 3:34pm

<facepalm>. I was trying to use the API port instead. I’ll replace with the node private port and try again. Thanks!

littleskunk · February 23, 2021, 3:46pm

Thank you very much for your feedback. I will address that and maybe the team has an idea to make that a bit more clear which port to pick.

inoc · February 25, 2021, 3:54pm

Hello
Tried to set it up on my Ubuntu 20.04 and I must admit I am not master of npm etc but pls read below errors encountered upon trying and setting up (first block of instructions)
how do I “install peer dependencies myself” as WARN messages tell me to do?
tahank you in advance!

nicolas@nicolas-VirtualBox:~/multinode/web/multinode$ npm install
npm WARN @fimbul/bifrost@0.21.0 requires a peer of tslint@^5.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN @vue/cli-plugin-typescript@4.5.9 requires a peer of @vue/compiler-sfc@^3.0.0-beta.14 but none is installed. You must install peer dependencies yourself.
npm WARN @vue/cli-service@4.5.9 requires a peer of @vue/compiler-sfc@^3.0.0-beta.14 but none is installed. You must install peer dependencies yourself.
npm WARN sass-loader@8.0.0 requires a peer of node-sass@^4.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN sass-loader@8.0.0 requires a peer of fibers@>= 3.1.0 but none is installed. You must install peer dependencies yourself.
npm WARN tslint-consistent-codestyle@1.16.0 requires a peer of tslint@^5.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN multinode@0.0.1 No description
npm WARN multinode@0.0.1 No repository field.
npm WARN multinode@0.0.1 No license field.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fork-ts-checker-webpack-plugin-v5@npm:fork-ts-checker-webpack-plugin@5.2.1 (node_modules/fork-ts-checker-webpack-plugin-v5):
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: Invalid dependency type requested: alias
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: vue-loader-v16@npm:vue-loader@16.1.2 (node_modules/vue-loader-v16):
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: Invalid dependency type requested: alias
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.1.3 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {“os”:“darwin”,“arch”:“any”} (current: {“os”:“linux”,“arch”:“x64”})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.13 (node_modules/watchpack-chokidar2/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {“os”:“darwin”,“arch”:“any”} (current: {“os”:“linux”,“arch”:“x64”})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.13 (node_modules/webpack-dev-server/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {“os”:“darwin”,“arch”:“any”} (current: {“os”:“linux”,“arch”:“x64”})
audited 1580 packages in 87.467s
found 0 vulnerabilities
nicolas@nicolas-VirtualBox:~/multinode/web/multinode$ npm run build
multinode@0.0.1 build /home/nicolas/multinode/web/multinode
vue-cli-service build
ERROR Error loading /home/nicolas/multinode/web/multinode/vue.config.js:
ERROR SyntaxError: Unexpected reserved word
/home/nicolas/multinode/web/multinode/node_modules/stylelint/node_modules/cosmiconfig/dist/Explorer.js:54
for await (const place of this.config.searchPlaces) {
^^^^^
SyntaxError: Unexpected reserved word
at createScript (vm.js:80:10)
at Object.runInThisContext (vm.js:139:10)
at Module._compile (module.js:617:28)
at Object.Module._extensions…js (module.js:664:10)
at Module.load (module.js:566:32)
at tryModuleLoad (module.js:506:12)
at Function.Module._load (module.js:498:3)
at Module.require (module.js:597:17)
at require (internal/module.js:11:18)
at Object. (/home/nicolas/multinode/web/multinode/node_modules/stylelint/node_modules/cosmiconfig/dist/index.js:12:17)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! multinode@0.0.1 build: vue-cli-service build
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the multinode@0.0.1 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /home/nicolas/.npm/_logs/2021-02-25T13_18_30_335Z-debug.log
nicolas@nicolas-VirtualBox:~/multinode/web/multinode$