Wrong Online Score?

So I run this new node as a systemd service… When I go to the dashboard, I see this.
image
Then again, when I pick the sat from options on top I see this…


So there is traffic. How come the sat decided my node was offline all the time and set online score for it = 0, while there is traffic between the sat and the node every day since it got started!? :rofl:
Other nodes I operate have a similar graph from this sat, but online score is 100%.

you would see that if the satellite is blocked and not the customers…
that would be my guess

1 Like

What? What? What? :smiley:

customers upload data to your node…

the satellite tracks uptime, if the satellite is blocked by a firewall your node will look like its offline to that particular satellite.

2 Likes

No. It is not blocked in any way. Other nodes on the same machine do not have this issue and the node obviously gets traffic from the sat.

do you have any audit traffic on that satellite ? that would be the only way to verify that it is not blocked

Yes. And audit score is 100%. :slight_smile:
Again - it is not blocked and traffic is clearly visible on the graph! Isn’t that just weird? :crazy_face:

nice, just sanity check for me, i was wondering about this one:


How much audit data do you have ?

On this node from that sat in question - 256B of audit data - only from yesterday - 27th of Jan, 2022. The node was started on the 12th of Jan, 2022. No other sats show such symptoms. No other nodes on the same machine show issues.

I have no idea then but I have 20 char

have you considered using a professional firewall solution, or a normal router rather than that custom iptables hack job…

you keep having connection issues… seems very possible that you might not understand iptables well enough to use it for such a complex setup…

i can’t say that is the truth… because i duno enough about iptables to even remotely understand your iptables firewall setup… but now i mentioned it again…

ill stop mentioning it from now on… but your custom setup does create a big barrier for who is able to actually help you diagnose the issues.

2 Likes

I have a router and a switch, I am the professional firewall.

Sure. Many have, but this is not a connection issue.

No way. Been doing it for years and is not complex at all.

To the question. How are online checks marked in the log? Because the node gets traffic from the sat, but sat thinks it’s off. Again - no other node has this issue and again - all nodes I operate have similar pattern of traffic with this sat.

You probably only had 1 audit that somehow failed for that satellite

Online scores are calculated using audit successes and failures to reach a node during audit attempt. So a successful audit increases the score, and if your node is not reachable when a satellite tries to audit, the score goes down. Since this check happens satellite side, you won’t see anything in the logs if your node was not reachable during the audit attempt.

1 Like

seems very professional.
maybe you want to use nftables these days…
you know… something that is actually designed to be able to work as a modern firewall.

1 Like

I’m agree with @SGC , you need to simplify your firewalling solution. If the satellite cannot reach your node - the online score will drop. The audit score is almost unrelated to this check - you can have a only one successful audit (before you applied your setup) and your audit score will be 100%, the online score just simple check - is node responding on audit requests or not.
Your online score shows that it’s not responding on audit requests from this particular satellite, so you need to examine your firewall rules and make sure that you do allow 28967 TCP/UDP to your node from any source (do not specify the source IP or source IP range!), normally you doesn’t need to have an outbound rules, but if you have - you need to add another one and allow any traffic from your node (and any port, i.e. you can set only local source IP, but not the source port!) to any destination (any IP and any port, TCP/UDP).

2 Likes

It is actually quite simple - block everything, allow only what is used.

This is what I am trying to figure out because no other node on the same machine has this issue. Maybe too fast to judge - timed out because of many requests. Seems weird. The connection is 1Gbps and uncapped by any shaper.

That is definitely allowed.
image
image

Sure. The following rules are set.

iptables -t filter -P OUTPUT ACCEPT
ip6tables -t filter -P OUTPUT ACCEPT

I would like to repeat this key part again - no other node on the same machine has this issue*. Some even started getting the 1.47.3 update already.

And looks like audits passed.

root@server:~$ journalctl -u nodeX | grep GET_AUDIT

Jan 27 02:14:30 server storagenode[2767]: 2022-01-27T02:14:30.332Z INFO piecestore download started {“Piece ID”: “N4IGAQJSQB2HD27S7ANB6MXFIH6X4VZU4INH4IVHQ3UBWDAI62VA”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “GET_AUDIT”}
Jan 27 02:14:30 server storagenode[2767]: 2022-01-27T02:14:30.396Z INFO piecestore downloaded {“Piece ID”: “N4IGAQJSQB2HD27S7ANB6MXFIH6X4VZU4INH4IVHQ3UBWDAI62VA”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “GET_AUDIT”}
Jan 27 06:08:14 server storagenode[2767]: 2022-01-27T06:08:14.588Z INFO piecestore download started {“Piece ID”: “LZ3BHYH7NTNKBRRXD5QH6BHY53ZJCFCHUMBLLTAKV2T5BGUXDQVQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
Jan 27 06:08:14 server storagenode[2767]: 2022-01-27T06:08:14.725Z INFO piecestore downloaded {“Piece ID”: “LZ3BHYH7NTNKBRRXD5QH6BHY53ZJCFCHUMBLLTAKV2T5BGUXDQVQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
Jan 27 17:10:30 server storagenode[2767]: 2022-01-27T17:10:30.909Z INFO piecestore download started {“Piece ID”: “WDVVIW5F6SGSCW3WO37GWN5OBD2V75PB3X3W5AKQWPX4MBBOHAPA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
Jan 27 17:10:31 server storagenode[2767]: 2022-01-27T17:10:31.046Z INFO piecestore downloaded {“Piece ID”: “WDVVIW5F6SGSCW3WO37GWN5OBD2V75PB3X3W5AKQWPX4MBBOHAPA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
Jan 27 18:09:39 server storagenode[2767]: 2022-01-27T18:09:39.473Z INFO piecestore download started {“Piece ID”: “G6ZQVOGMD5ENJUVREPWTP2SDA2VL5F6XRN7B4VRZVWLNI33OTQAQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_AUDIT”}
Jan 27 18:09:39 server storagenode[2767]: 2022-01-27T18:09:39.760Z INFO piecestore downloaded {“Piece ID”: “G6ZQVOGMD5ENJUVREPWTP2SDA2VL5F6XRN7B4VRZVWLNI33OTQAQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_AUDIT”}
Jan 27 22:30:23 server storagenode[2767]: 2022-01-27T22:30:23.999Z INFO piecestore download started {“Piece ID”: “HZ2YZDJH46NFXNMM3F2KV2TJN4WM4MRDPZWJ4XVW46772MOJ5ETQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
Jan 27 22:30:24 server storagenode[2767]: 2022-01-27T22:30:24.249Z INFO piecestore downloaded {“Piece ID”: “HZ2YZDJH46NFXNMM3F2KV2TJN4WM4MRDPZWJ4XVW46772MOJ5ETQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
Jan 28 01:21:33 server storagenode[2767]: 2022-01-28T01:21:33.248Z INFO piecestore download started {“Piece ID”: “TM7OROLQN5LTXV7RGN7YSYCLUXOOJRYB2DVTQCP6OQ43XXHBMLHA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET_AUDIT”}
Jan 28 01:21:33 server storagenode[2767]: 2022-01-28T01:21:33.502Z INFO piecestore downloaded {“Piece ID”: “TM7OROLQN5LTXV7RGN7YSYCLUXOOJRYB2DVTQCP6OQ43XXHBMLHA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET_AUDIT”}
Jan 28 05:36:24 server storagenode[2767]: 2022-01-28T05:36:24.518Z INFO piecestore download started {“Piece ID”: “W7SGAS32LJ24IMGF2HMMMNZYX3RECZJ4G7P2BIFHQNCVJLPEBULQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
Jan 28 05:36:24 server storagenode[2767]: 2022-01-28T05:36:24.669Z INFO piecestore downloaded {“Piece ID”: “W7SGAS32LJ24IMGF2HMMMNZYX3RECZJ4G7P2BIFHQNCVJLPEBULQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
Jan 28 21:53:20 server storagenode[2767]: 2022-01-28T21:53:20.948Z INFO piecestore download started {“Piece ID”: “HMRDR6F6PTXDSTFFXFGUOM2E4JS7QEJ3AFAWFIWRKLAPKU7KAR7A”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_AUDIT”}
Jan 28 21:53:21 server storagenode[2767]: 2022-01-28T21:53:21.015Z INFO piecestore downloaded {“Piece ID”: “HMRDR6F6PTXDSTFFXFGUOM2E4JS7QEJ3AFAWFIWRKLAPKU7KAR7A”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_AUDIT”}
Jan 28 23:03:00 server storagenode[2767]: 2022-01-28T23:03:00.516Z INFO piecestore download started {“Piece ID”: “LXIBWKFUJTI3GLKMUWV4AMUHCX5UM65P5EATOORGRVWVIQERP2ZQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_AUDIT”}
Jan 28 23:03:00 server storagenode[2767]: 2022-01-28T23:03:00.578Z INFO piecestore downloaded {“Piece ID”: “LXIBWKFUJTI3GLKMUWV4AMUHCX5UM65P5EATOORGRVWVIQERP2ZQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_AUDIT”}

So, what I believe is this is NOT:

  • connection speed related (too many requests may lead to timeouts)
  • firewall related (obviously not - ports are open, audits are success, communication is possible)
  • OS related (no limits set, on the contrary - quite increased and resources are high on this machine)
  • hardware failure related (like not being able to cope with the requests it gets)
  • CloudFlare DNS problem, if they have some interruptions on their mega hyper TB speed network - this is what is being used as DNS because domain name is utilized instead of IP because of the dual setup with IPv6. Ha-ha! Ha-ha-ha! Maybe this is what it could be, but it is not proxied. Just using the free DNS for A and AAAA records.

What I believe it is:

  • bug - sat ap1 thinks nodex offline all the time, while it keeps on passing all audits and keeps on getting traffic from it. :crazy_face:

Could any storjling check the logs on the sat? I can provide the node ID, if needed, no matter the piece IDs are visible in the audit log output above and the node could be easily identified. Then we would know what the sat thinks of the node.

It just sounds like you don’t want to be helped, you’re stubborn and don’t want to admit that the problem is probably on your end. Instead you suggest some weird bug on a satellite that’s serving thousands of other nodes with no problem. And not only on this issue, I find you consistently condescending every time someone questions your setup.
Most people are here to help but please understand that your setup is quite unique so it’s understandable than most people would expect the issue there and not in the satellite software.
However it’s weird that you get audits, hopefully that means the satellite considers the node online.

Might have to take some of that back

I can assume that you have a separate rules for each node. Worth to check the rule for this particular node and compare with working ones.

This is an application, there is no magic. If something working not as expected, check what’s different.

Your port maybe openeded for the port checker, but closed for this particular satellite (you may be blocking somewhere in your setup only some IPs based on the range or the rule for this node has an error or typo).

Perhaps this satellite cannot reach your node only periodically.
You can check date time intervals, when your node were offline:

1 Like