Multinode seems to stop where it should continue

jammerdan · March 5, 2023, 2:51am

After updating Multinode I see these errors on the same unchanged sql database:

2023-03-05T02:59:47.571+0100    ERROR   console:endpoint        list node trusted satellites internal error       {"error": "nodes: rpc: tcp connector failed: rpc: tls peer certificate verification: tlsopts: peer ID did not match requested ID", "errorVerbose": "nodes: rpc: tcp connector failed: rpc: tls peer certificate verification: tlsopts: peer ID did not match requested ID\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2023-03-05T03:00:04.476+0100    ERROR   console:endpoint        list node trusted satellites internal error       {"error": "nodes: rpc: tcp connector failed: rpc: tls peer certificate verification: tlsopts: peer ID did not match requested ID", "errorVerbose": "nodes: rpc: tcp connector failed: rpc: tls peer certificate verification: tlsopts: peer ID did not match requested ID\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2023-03-05T03:00:08.754+0100    ERROR   console:endpoint        list node trusted satellites internal error       {"error": "nodes: rpc: tcp connector failed: rpc: tls peer certificate verification: tlsopts: peer ID did not match requested ID", "errorVerbose": "nodes: rpc: tcp connector failed: rpc: tls peer certificate verification: tlsopts: peer ID did not match requested ID\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2023-03-05T03:04:41.447+0100    ERROR   console:endpoint        get bandwidth monthly error       {"error": "bandwidth: context canceled", "errorVerbose": "bandwidth: context canceled\n\tstorj.io/storj/multinode/bandwidth.(*Service).getMonthly:250\n\tstorj.io/storj/multinode/bandwidth.(*Service).Monthly:57\n\tstorj.io/storj/multinode/console/controllers.(*Bandwidth).Monthly:45\n\tnet/http.HandlerFunc.ServeHTTP:2084\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2916\n\tnet/http.(*conn).serve:1966"}
2023-03-05T03:05:19.216+0100    ERROR   console:endpoint        total usage internal error        {"error": "storage web api controller: storage: rpc: tcp connector failed: rpc: tls peer certificate verification: tlsopts: peer ID did not match requested ID", "errorVerbose": "storage web api controller: storage: rpc: tcp connector failed: rpc: tls peer certificate verification: tlsopts: peer ID did not match requested ID\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2023-03-05T03:05:44.402+0100    ERROR   console:endpoint        could not get total disk space    {"error": "storage: rpc: tcp connector failed: rpc: tls peer certificate verification: tlsopts: peer ID did not match requested ID", "errorVerbose": "storage: rpc: tcp connector failed: rpc: tls peer certificate verification: tlsopts: peer ID did not match requested ID\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}

I haven’t seen such errors with the previous version on the same sql database. Furthermore this seems to make Multinode to stop proceeding. The My-node overview displays fine, while Payouts and Bandwidth & Disk all overviews remain empty resp. show only zero value.
In any way I don’t think Multinode should stop when there are errors with single nodes. Instead it should indicate an error and proceed with the next node and creating the totals with the data that have been sucessfully gathered instead of nothing.

Alexey · March 5, 2023, 5:00am

perhaps some misconfiguration of the nodes - either their identity is mixed, or ports.
But I agree, it should not stop working if some nodes are misconfigured.
However, you likely need either fix the root cause nodes or remove them from the multinode dashboard.

You may also create a bug on our GitHub.

jammerdan · March 5, 2023, 10:35am

I still don’t have a Github account.

I just reverted back to the old version which seems not to have that issue.

Alexey · March 5, 2023, 2:07pm

Storjlings’ requests have a lower priority… But I can create it for you.
Could you please describe (in details) how to reproduce it?

jammerdan · March 6, 2023, 3:00am

I don’t know exactly yet what the reason for this behavior is. Even the older version has some issues displaying information on the payout page. This might be related or something totally different. Also everything might depend on the current load of a node. But again: The Multinode should basically never stop processing or displaying data if there is some node related error. It should simply proceed with the next node.

So for the current situation to reproduce the error message, you need to add a node to the Multinode and then stop and disable it. Then you spin up another/new node with the IP and port from the first node. So the Multinode thinks it connects to the first node but in reality connects to the second node.
This should create the error that I see that peer ID does not match requested ID.
The reason is that I have disabled and replaced some older nodes which are still in the Multinode.

Now for some reason the older version of Multinode does ignore this error and proceeds like it should. (As said sometimes there are issues displaying total bandwidth and total estimated earnings which indicates that there are issues reaching individual nodes) but normally it shows all information.

The new version instead does not show anything on the payout page and only zero values on the bandwith page and displays the posted errors in the Multinode terminal. The old version does not show these errors.

Roberto · March 6, 2023, 6:30am

I haven’t had this curve for months. On individual nodes it has resumed working since 1.73

In addition to the sum that is always wrong

jammerdan · March 6, 2023, 4:06pm

It’s not good if you cannot see how your nodes are doing.

Roberto · March 6, 2023, 4:12pm

Only the curve doesn’t work and the total, and only on the multi-node, isn’t nice though

Alexey · March 12, 2023, 5:19am

github.com/storj/storj

[multinode dashboard] Since v1.71.2 the disk usage graph shows 0

opened 05:38AM - 09 Feb 23 UTC

AlexeyALeonov

Bug

**Description** The disk usage graph shows 0 instead of real usage. **Screenshots** ![image](https://user-images.githubusercontent.com/26858949/217725346-eae7ecc5-936e-4096-a6b1-3b44d73246e8.png) **Steps to reproduce the issue:** 1. Update multinode dashboard to v1.71.2 2. Add several (or at least one node v1.71.2) 3. Switch to **Bandwidth & Disk** and check the ***Average Disk Space Used This Month*** graph **Describe the results you expected:** The graph shows 0B across the month, the ***Total Disk Space*** pie chart shows used space. **Describe the results you received:** The ***Average Disk Space Used This Month*** graph numbers should be close to the ***Total Disk Space*** pie chart. **Possible Fix** Rollback the multinode dashboard to v1.68.2 it will show the graph, however the wrong too, see https://github.com/storj/storj/issues/5519 **Your environment** - Operating system and version: Windows 10 Pro 10.0.19045 N/A Build 19045 - Additional environment details (Raspberry PI, Docker, VMWare, etc.): binary and docker

github.com/storj/storj

[multinode dashboard] Shows free disk space instead of free space in the allocation

opened 06:51AM - 07 Oct 22 UTC

AlexeyALeonov

Bug Needs Estimation

**Description** When you open a bandwidth and disk usage of the individual node, you will see a free space on the disk instead of free space in the allocation. See https://forum.storj.io/t/multi-node-dashboard-shows-wrong-free-space/19868  **Screenshots** The single node dashboard: ![image](https://user-images.githubusercontent.com/26858949/194484430-0b86b1cf-db3f-4b24-ab16-b0f32fdb761e.png) The node selected in the multinode dashboard: ![image](https://user-images.githubusercontent.com/26858949/194484501-d7d943ab-04d1-4f93-bd12-f200be5b94f9.png) **Steps to reproduce the issue:** 1. Install multinode dashboard 2. add node to the multinode dashboard 3. Open **Bandwidth & disk** menu 4. Select the node in the **Nodes** dropdown 5. You will see a bandwidth and disk usage, but the free space is showed for the disk, not allocation 6. Open a single node dashboard (the integrated one, on localhost:14002 by default), you will see a free space in the allocation, not on the disk. **Describe the results you expected:** When we open bandwidth and disk usage for the node in the multinode dashboard, we should see a free space in the allocation, not on the disk **Describe the results you received:** When we open bandwidth and disk usage for the node in the multinode dashboard, we can see a free space on the disk instead of free space in the allocation **Your environment** - Operating system and version: any - Additional environment details (Raspberry PI, Docker, VMWare, etc.): ``` multinode version Version: v1.64.1 Build timestamp: 19 Sep 22 23:01 +0400 Git commit: 01dfa305452c264b839358e95ff1d9e2ccbedcec ```

jammerdan · March 23, 2023, 4:27am

I switched to new version to see if there is any progress…
You really gotta love multinode:

I mean how are you supposed to keep track of multiple nodes if this thing does not display anything useful.

Alexey · March 23, 2023, 4:49am

for me it at least shows bandwidth

Alexey · March 23, 2023, 5:05am

By the way, you may remove broken nodes and add a correct ones (the API key doesn’t match the address and port for them).

I created issue for you: [multinode dashboard] Bandwidth usage, disk usage and payout become zero if one of the nodes is misconfigured · Issue #5701 · storj/storj · GitHub

jammerdan · March 23, 2023, 5:36am

For me it does not. At least the payout came back, but this part is still not showing:
balance

I have done that in the meantime. All nodes in ‘my nodes’ list are displaying as being online and showing their data in that list.

asturking · May 22, 2023, 4:52pm

It seems that this topic is a bit sleepy and there isn’t much information available about it.

Currently, I have the panel with 2 nodes, and I’m having trouble adding 3 additional nodes that I have. There are no error messages displayed, and there is nothing unusual. I obtain the data in the same way for the working nodes as for the ones that are not working, and I can’t figure out the reason for the failure.

I see that there is a pull request created regarding this issue (web/multinode: Show appropriate error while adding new node by Pranav2612000 · Pull Request #4293 · storj/storj · GitHub).

Also, I have another question: Is there a way to import node data in a bulk manner?
Although I believe the data entered is correct, I think it would be a good way to verify if the data is entered correctly.

jammerdan · May 22, 2023, 5:04pm

I’d suggest to check if the API-Keys are correct. I had a similar problem:

Recreate them and try both ways maybe.

Yes, you can put them into a json file and import that:

asturking · May 22, 2023, 5:15pm

You know the format?

I see this info at the main.go file but I dont know how should be the code

# add nodes from json file containing array of nodes data
$ multinode add nodes.json
# add node from json file containing a single node object
$ multinode add node.json
# read nodes data from stdin
$ cat nodes.json | multinode add -
`,
	}

	runCfg   Config
	setupCfg Config
	addCfg   struct {
		NodeID        string `help:"ID of the storage node" default:""`
		Name          string `help:"Name of the storage node" default:""`
		APISecret     string `help:"API Secret of the storage node" default:""`
		PublicAddress string `help:"Public IP Address of the storage node" default:""`

		Config
	}
	confDir     string
	identityDir string
)

jammerdan · May 22, 2023, 5:25pm

Like this:

[
    {
        "publicAddress":  "ip1:port1",
        "id":  "xxx",
        "apiSecret":  "yyy",
        "name":  "node1"
    },
    {
        "publicAddress":  "ip2:port2",
        "id":  "xxx",
        "apiSecret":  "yyy",
        "name":  "node2"
    }
]