Multinode Dashboard: One node does not display statistics

sembeth · January 13, 2023, 5:34pm

Hello,
I use the Multinode Dashboard and I appreciate the work that went into it, but I hope it gets more attention at some point.

Currently I have a problem with one storagenode that does not display statistics:

The normal dashboard (port 14002) of the same storagenode displays all statistics properly:

The Storagenode runs in Docker on Debian 11, which is up to date.

I tried:

Restarting the server.
Stopping, removing, updating the Docker image, running the storagenode.
Using the public and internal IP with the correct port 28967.
Use the multinode dashboard in Docker.
Use the latest version of multinode.exe and remove all files under

…AppData\Roaming\Storj\Multinode

and used a fresh “multinode.exe setup”

Use

docker exec -it storj1 /app/storagenode issue-apikey --config-dir config --identity-dir identity --log.output stdout

to generate a new api-key.

I have set the debug level of the multinode dashboard to trace to find hints, unfortunately there are almost no log entries.

But all to no success.

My other storagenodes all show stats on the multinode dashboard with no problem. Just this one doesn’t.

I have run out of ideas.

I would really appreciate any advice on what I could try.

Thank you very much for your time.

digitalfrank · January 13, 2023, 9:14pm

Hi, I have another two problem with new multinode.exe 1.70.2 on windows 10 pro. One I don’t see the average disk used this month. I restart the exe but nothing. I downgraded to 1.69.2 and i see the graph. other problem is the total space of nodes that is not correct. The space used is real but free is not real, it’s minor.

sembeth · January 13, 2023, 11:39pm

A little update on my problem. If I set up a completely new multinode dashboard and then add just the one storagenode that is causing problems, I don’t see any stats under “My Nodes”.

But when I click on Bandwidth & Disk, I see everything as expected

which matches the normal dashboard on port 14002

So the good thing is that the multinode dashboard can retrieve data from the storagenode that is having problems.

But I still have no idea why my other nodes show stats on “My Nodes” but the one does not.

Alexey · January 14, 2023, 8:41am

Please try to use the multinode dashboard binary with the same version as your oldest node.

Alexey · January 14, 2023, 9:03am

github.com/storj/storj

[multinode dashboard] Shows free disk space instead of free space in the allocation

opened 06:51AM - 07 Oct 22 UTC

closed 12:36PM - 01 Aug 23 UTC

AlexeyALeonov

Bug SNO

**Description** When you open a bandwidth and disk usage of the individual node, you will see a free space on the disk instead of free space in the allocation. See https://forum.storj.io/t/multi-node-dashboard-shows-wrong-free-space/19868  **Screenshots** The single node dashboard: ![image](https://user-images.githubusercontent.com/26858949/194484430-0b86b1cf-db3f-4b24-ab16-b0f32fdb761e.png) The node selected in the multinode dashboard: ![image](https://user-images.githubusercontent.com/26858949/194484501-d7d943ab-04d1-4f93-bd12-f200be5b94f9.png) **Steps to reproduce the issue:** 1. Install multinode dashboard 2. add node to the multinode dashboard 3. Open **Bandwidth & disk** menu 4. Select the node in the **Nodes** dropdown 5. You will see a bandwidth and disk usage, but the free space is showed for the disk, not allocation 6. Open a single node dashboard (the integrated one, on localhost:14002 by default), you will see a free space in the allocation, not on the disk. **Describe the results you expected:** When we open bandwidth and disk usage for the node in the multinode dashboard, we should see a free space in the allocation, not on the disk **Describe the results you received:** When we open bandwidth and disk usage for the node in the multinode dashboard, we can see a free space on the disk instead of free space in the allocation **Your environment** - Operating system and version: any - Additional environment details (Raspberry PI, Docker, VMWare, etc.): ``` multinode version Version: v1.64.1 Build timestamp: 19 Sep 22 23:01 +0400 Git commit: 01dfa305452c264b839358e95ff1d9e2ccbedcec ```

sembeth · January 14, 2023, 9:20pm

Hello and thank you for your reply. I appreciate it.

I tried the multinode dashboard in versions 1.69.2, 1.68.2, 1.67.3 and deleted the master.db and config.yaml each time under …AppData\Roaming\Storj\Multinode.

Unfortunately, still no statistics under “My Nodes”.

The problem is that my other nodes with the same version work fine. Only this one does not.

I suspect something is wrong with this one node, but I have no idea what. It would be really helpful if the multinode dashboard had a lot more logs of what is happening to see what is not working.

sembeth · January 14, 2023, 10:50pm

Little update:
I added some log entries in

multinode/nodes/service.go/ListInfos

for the storagenode that is causing problems, I get:

ERROR nodes:service nodes/service.go:193 ListInfos earned error {“error”: “payouts: sql: Scan error on column index 0, name "comp_at_rest": converting NULL to int64 is unsupported”}

Is it possible that one of the SQLite databases of my Storagenode is broken?

When I don’t add earned.Total, I get statistics for everything else.

Alexey · January 15, 2023, 1:59am

yes, it’s possible. You may check your databases for this node:

digitalfrank · January 15, 2023, 5:24am

Thanks for reply, but the other problem? You have a reply? I don’t see the average as you see in the photo. I downgrade to multinode.exe of version 1.69.2 and i see the graph.
Thanks.

Alexey · January 15, 2023, 8:24am

I do not have a reply from the team, they do not work on weekends. But I guess that the latest version of the multinode dashboard is not compatible with the old storagenode’s version.

sembeth · January 15, 2023, 12:49pm

Well, it seems that it was indeed a corrupted database. I stopped the node, deleted all .db files and restarted the node. Now it works fine and shows the statistics in the multinode dashboard. Of course, the statistics are wrong at the moment, but that’s fine.

I guess that not displaying the statistics was a feature to indicate that something is wrong with this node.

Thank you very much for your valuable time, @Alexey.

One last note: The orders.db was quite large at 622M. I’m not sure if this database is supposed to be that big.

vladro · January 15, 2023, 3:49pm

should be 0-32 kb usually

vladro · January 15, 2023, 3:51pm

hmmm- checked around 50+ of my nodes- all of them are 0-32kb, one is 163mb- is it normal?

Alexey · January 16, 2023, 4:16am

This database is not used anymore to store orders, the orders now stored on the disk as files in the orders subfolder in the config location by default.