2 nodes on win server/Hyper-v/linux setup issue

Hi guys, 8 days ago I enabled my second node. Both of them are on the same windows server unit with hyper-v as 2 separated VM Linux systems.

Both dashboards were showing online. Unfortunately as soon I enable the second node. My first node stops receiving/sending any data. I was thinking it is normal as is full. Today I rebooted node 1 looks like it is getting back some data unfortunately my dashboard was freeze for all 8 days. After reboot, I see “europe-north-1.tardugrade.i …” 63% Online and I got 3 dollars more on the current month earning.

Now I can not connect to the dashboard for node 2 also it is showing port 14003 as not reachable. Can you advise if my ports or the command structure are ok?

The second node via that 8 days get 35GB. All satellites were 100%.
I do not want to test, I can be disqualified on my node 1. That one is from the beginning of the v3 and it is full of data.

Please any advice?




Node_2


Do you use the same identity for both nodes (don’t do it!)?

2 Likes

No is not the same. Different email, different auth token, different identity generated.

I’m afraid an audit score below 60% usually means the node got already disqualified on corresponding satellites :persevere:

Whenever scores start dropping below 95/90%, it’s urgent to investigate and fix issues.

Like @zuik I would have thought such a behavior could be caused by 2 nodes using the same identity which is a fast road to disqualification for both nodes. It’s a real shame satellites can’t detect 2 nodes are using the same identity and warn the SNO accordingly :confused:


Anyways, I’m not sure that’s what’s wrong in your case as it seems both of your nodes are pointing to different identity folders : “/home/[user]/.local/share/storj/identity/storagenode” & “/local/share/Identity/storagenode”.
So unless they contain the same identity files, or if one folder is a symlink to the other, or if the 2 identities got mixed up… that’s probably not the issue.
This said, because they seem like default folder locations, it could have been easy to mix up destinations between nodes I guess. A good practice is to store identities within storage disks, so they are where the data they belong to is.

Did you check your nodes’ logs? There’re probably plenty of errors popping up in them that could shed some light on what went wrong, to better understand what to fix.

Node 1 on that 3 satellites is disqualified now over the year from that point they added that suspension/audit graphs. Then also they related that to SMR HDD.
Now we have graphs, unfortunately, they froze for 8 days

The dashboard was for that 8 days showing 99% online and the same amount of money for this month. Just after reboot on day 8. It updated to 60% +3$.

I was setting up nodes2 on Linux all was done on that VM. I do not copy any data between two instances of Linux. I will check the log.

Log:
-----------------------------------------------------------Logs Node2---------------------------
@storj_node_2:~$ screen
JEW3X2BRHTAC6W35L6CYS5YAQLFKJRUMOWZJCNV46MSM6B4WA"}
2021-06-27T16:33:37.601Z INFO piecedeleter delete piece sent to trash {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Piece ID”: “RTNGVVGK5MT36LGACLQ5PV2A2YMVRNQHKS6LYMIMUHRS3QZH2MGA”}
2021-06-27T16:34:13.009Z INFO piecestore upload started {“Piece ID”: “4PSY4S3PD4KDJCPS6ASITI6TZZJP5MGFEFY2ZDBTLY36SOZ2ZIBA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 6964070917632}
2021-06-27T16:34:13.385Z INFO piecestore uploaded {“Piece ID”: “4PSY4S3PD4KDJCPS6ASITI6TZZJP5MGFEFY2ZDBTLY36SOZ2ZIBA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 181504}
2021-06-27T16:34:18.547Z INFO piecestore upload started {“Piece ID”: “YJSYESJ6QBYEB4EABUXL2YDU4I3RV3HZTDVKFVNFPMULLJQBANPA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 6964070735616}
2021-06-27T16:34:18.892Z INFO piecestore uploaded {“Piece ID”: “YJSYESJ6QBYEB4EABUXL2YDU4I3RV3HZTDVKFVNFPMULLJQBANPA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 181504}
2021-06-27T16:34:18.894Z INFO piecestore upload started {“Piece ID”: “B3Z7TSAXC7YG74VFL54WNQWSHN6MYTKIXHUXGXC24B73ZNRQ2APQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Space”: 6964070553600}
2021-06-27T16:34:23.616Z INFO piecestore uploaded {“Piece ID”: “B3Z7TSAXC7YG74VFL54WNQWSHN6MYTKIXHUXGXC24B73ZNRQ2APQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 362752}
2021-06-27T16:34:57.650Z INFO piecedeleter delete piece sent to trash {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Piece ID”: “CWFHRHPACKUCCOKJSSVTRV4ERTZRK566XTEM6Z6ZJL4Y6EDQWCKA”}
2021-06-27T16:37:05.722Z INFO piecedeleter delete piece sent to trash {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Piece ID”: “MNWTF5JU5RQ46F7S3MUYG7ABJBYPTWGI67KQRGEWFBPQNGJ72CVA”}
2021-06-27T16:38:06.326Z INFO piecestore upload started {“Piece ID”: “KZYL454PLC3W7CEKR6ET7BZLRGIJDMQHVMVP7AZQWSBUD2F2R25Q”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT_REPAIR”, “Available Space”: 6964070190336}
2021-06-27T16:38:16.408Z INFO piecestore uploaded {“Piece ID”: “KZYL454PLC3W7CEKR6ET7BZLRGIJDMQHVMVP7AZQWSBUD2F2R25Q”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT_REPAIR”, “Size”: 1620224}
2021-06-27T16:38:55.576Z INFO piecedeleter delete piece sent to trash {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Piece ID”: “EPDMSB77W3Q7EZLQWBWCZ7L6CTUBI2MUA7IEMFPYKYR324DBWNWA”}
2021-06-27T16:38:55.577Z INFO piecedeleter delete piece sent to trash {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Piece ID”: “Y7TDVQX3A3XW5D7C2OCK3GJ6BOLWHONINANCIMJEYWXQRF2FSDQQ”}
2021-06-27T16:38:55.578Z INFO piecedeleter delete piece sent to trash {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Piece ID”: “IU3ORNF2CY2WSVF5K5A3WEK3PJT4UKR5BC3WGI4XLDENAA7M6GGA”}
2021-06-27T16:39:14.157Z INFO piecestore upload started {“Piece ID”: “DQ4RATSDNX7SMSOLYCSFXANVKMYLF7OECW7OU5TNIUCDGRDREITA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Space”: 6964068569600}
2021-06-27T16:39:14.588Z INFO piecestore uploaded {“Piece ID”: “DQ4RATSDNX7SMSOLYCSFXANVKMYLF7OECW7OU5TNIUCDGRDREITA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 60416}
2021-06-27T16:39:23.302Z INFO piecedeleter delete piece sent to trash {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Piece ID”: “DQ4RATSDNX7SMSOLYCSFXANVKMYLF7OECW7OU5TNIUCDGRDREITA”}
2021-06-27T16:40:23.410Z INFO piecedeleter delete piece sent to trash {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Piece ID”: “L6X7P2R6MVZMTVXXPAL5GTHDZ7RHE6TUQZPKMZTKBFPN34FK6L4A”}
2021-06-27T16:40:23.411Z INFO piecedeleter delete piece sent to trash {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Piece ID”: “NRJ77KUYPFWZCWD7GRITL5TM3DGCNWMVFYI7DWHYZPR7BDHEMEGA”}
2021-06-27T16:40:47.973Z INFO piecestore upload started {“Piece ID”: “WF2OXZFVFPDW7HEN2ODY3SG4SQKWD5N5NTBJMFHWUFKRXTNPZ3VA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 6964068508672}
2021-06-27T16:40:48.360Z INFO piecestore uploaded {“Piece ID”: “WF2OXZFVFPDW7HEN2ODY3SG4SQKWD5N5NTBJMFHWUFKRXTNPZ3VA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 181504}
2021-06-27T16:40:54.116Z INFO piecestore upload started {“Piece ID”: “EZOVLHPGUHVTKEYBN2ESXEKFJJRYCH4DF5CFCKNBSNXR3ANAKCGA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 6964068326656}
2021-06-27T16:40:54.612Z INFO piecestore uploaded {“Piece ID”: “EZOVLHPGUHVTKEYBN2ESXEKFJJRYCH4DF5CFCKNBSNXR3ANAKCGA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 181504}
2021-06-27T16:42:10.842Z INFO piecestore upload started {“Piece ID”: “WCUS3CY3U5YRFFCUKB64M2WTXHFU63HG7C4EDEHUAYZYDDX4FOAQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 6964068144640}
2021-06-27T16:42:11.228Z INFO piecestore uploaded {“Piece ID”: “WCUS3CY3U5YRFFCUKB64M2WTXHFU63HG7C4EDEHUAYZYDDX4FOAQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 181504}
@storj_node_2:~#

Based on your docker run command aren’t you forwarding traffic from node2 to the same port node1 runs on? the 28968:28967 part?

that was the only way I get node2 online.

Node1
docker run -d --restart unless-stopped -p 28967:28967 -p 14002:14002
Node 1 working ok with that comand.

Node2
docker run -d --restart unless-stopped --stop-timeout 300
-p 28968:28968/tcp
-p 28968:28968/udp
-p 127.0.0.1:14003:14003
Is that correct, if I have the second node on the same Lan, connected to the same router?

This wouldn’t be right. Ports on the right side of colons are internal ports: they should never change. The command you showed earlier is the right one for node2: with ports configured like follows:

[...]
 -p 28968:28967/tcp \
 -p 28968:28967/udp \
 -p 127.0.0.1:14003:14002 \
[...]

All nodes can be on the same LAN. No problem with that. In fact, they can even be on the same machine behind the same local IP. As long as port redirections are setup correctly on the ISP router, nodes can basically be anywhere on your local network.

The log excerpt you showed seems okay. Could you search for lines that contain “error” or “warning” in them?

1 Like

Please, do not forward dashboard’s port - it has no protection and anyone in the world can see your private information, use the secured remote access instead: How to remote access the web dashboard - Node Operator

Regarding remained parts all looks good. They are behind the same public IP, so they will share an ingress and can receive as a one node in total. So, stopping ingress looks right.

By the way - you doesn’t need to have several Linux VMs to run the storagenode, you can run them both on the same VM, but with different disks and identities.

Hi Alexey, I will try that “secured remote access”.

When I pull CLI dashboard egress/ingress changed, Node 2 it is getting some data. 1400X is just for the web interface and audits/suspensions, month earrings. Any advice to get that working?

I also try with (local IP):1400X and I got “This site can’t be reached” Issue is somewhere on that command line structure or that VM.

@Pac node 2 gets just that small amount of logs and there are no “errors” or “warning” Compare to Node1.

replace

to
-p 14003:14002 \
And your node will be available from the local network and since you have a port forwarding rule for that port - everyone in the world.

1 Like

Hi Alexy, I will left as is for now. That makes sense when I use the command line structure from the old node 1. When was created a long time ago there was missing that part. Probably was added later for security. Thank you, guys! That can be closed now. Now I need to think about security. Thank you, Alexy!

1 Like