Wrong Online Score?

svet0slav · January 28, 2022, 7:58am

So I run this new node as a systemd service… When I go to the dashboard, I see this.

Then again, when I pick the sat from options on top I see this…

So there is traffic. How come the sat decided my node was offline all the time and set online score for it = 0, while there is traffic between the sat and the node every day since it got started!?

Other nodes I operate have a similar graph from this sat, but online score is 100%.

SGC · January 28, 2022, 10:10am

you would see that if the satellite is blocked and not the customers…
that would be my guess

svet0slav · January 28, 2022, 10:27am

What? What? What?

SGC · January 28, 2022, 10:30am

customers upload data to your node…

the satellite tracks uptime, if the satellite is blocked by a firewall your node will look like its offline to that particular satellite.

svet0slav · January 28, 2022, 10:31am

No. It is not blocked in any way. Other nodes on the same machine do not have this issue and the node obviously gets traffic from the sat.

TheMightyGreek · January 28, 2022, 12:19pm

do you have any audit traffic on that satellite ? that would be the only way to verify that it is not blocked

svet0slav · January 28, 2022, 12:22pm

Yes. And audit score is 100%.
Again - it is not blocked and traffic is clearly visible on the graph! Isn’t that just weird?

TheMightyGreek · January 28, 2022, 12:34pm

nice, just sanity check for me, i was wondering about this one:

How much audit data do you have ?

svet0slav · January 28, 2022, 12:46pm

On this node from that sat in question - 256B of audit data - only from yesterday - 27th of Jan, 2022. The node was started on the 12th of Jan, 2022. No other sats show such symptoms. No other nodes on the same machine show issues.

TheMightyGreek · January 28, 2022, 1:12pm

I have no idea then but I have 20 char

SGC · January 28, 2022, 2:14pm

have you considered using a professional firewall solution, or a normal router rather than that custom iptables hack job…

you keep having connection issues… seems very possible that you might not understand iptables well enough to use it for such a complex setup…

i can’t say that is the truth… because i duno enough about iptables to even remotely understand your iptables firewall setup… but now i mentioned it again…

ill stop mentioning it from now on… but your custom setup does create a big barrier for who is able to actually help you diagnose the issues.

svet0slav · January 28, 2022, 4:04pm

I have a router and a switch, I am the professional firewall.

Sure. Many have, but this is not a connection issue.

No way. Been doing it for years and is not complex at all.

To the question. How are online checks marked in the log? Because the node gets traffic from the sat, but sat thinks it’s off. Again - no other node has this issue and again - all nodes I operate have similar pattern of traffic with this sat.

pietjebell · January 28, 2022, 4:22pm

You probably only had 1 audit that somehow failed for that satellite

baker · January 28, 2022, 4:42pm

Online scores are calculated using audit successes and failures to reach a node during audit attempt. So a successful audit increases the score, and if your node is not reachable when a satellite tries to audit, the score goes down. Since this check happens satellite side, you won’t see anything in the logs if your node was not reachable during the audit attempt.

github.com

storj/storj/blob/bd36a41a9ebe4d855ad654daf3856728e2796d98/docs/blueprints/storage-node-downtime-tracking-with-audits.md

# Storage Node Downtime Tracking With Audits

## Abstract

This document describes a means of tracking storage node downtime with audits and using this information to suspend and disqualify.

## Background

The previous implementation of uptime reputation consisted of a ratio of online audits to offline audits. We encountered a problem where some nodes' reputations would quickly become destroyed over a relatively short period of downtime due to the frequency of auditing any particular node being directly correlated with the number of pieces it holds. To solve this problem we need a system that takes into account not only how many offline audits occur, but _when_ they occur as well.

## Design

The solution proposed here is to use a series of sliding windows to indicate a general timeframe in which offline audits occur. Each window keeps two separate tallies indicating how many online audits and total audits a particular node received within its timeframe. Once a window is complete, it is scored by calculating the percentage of total audits for which it was online. We can average these scores over a trailing period of time, called the _tracking period_, to determine an overall "online score" to be used for suspension and disqualification. By granting each individual window the same weight in the calculation of the overall average, the effect of any particularly unlucky period can be minimized while still allowing us to take the failures into account over a longer period.

Storage node downtime can have a range of causes. For those storage node operators who may have fallen victim to a temporary issue, we want to give them a chance to diagnose and fix it before disqualifying them for good. For this reason, we are introducing suspension as a component of disqualification.

Once a node's online score has fallen below an _offline threshold_, it is _suspended_ and enters a period of review. A suspended node will not receive any new pieces, but can continue to receive download and audit requests for the pieces it currently holds. However, its pieces are considered to be unhealthy. We repair a segment if it contains too many unhealthy pieces, at which point we may transfer the repaired pieces from a suspended node to a more reliable node. If at any point during the review period we find that a node's score has risen above the offline threshold, it is unsuspended, or _reinstated_, but it remains _under review_. This prevents nodes from alternating between suspension and reinstatement without consequence.

The review period consists of one _grace period_ and one _tracking period_. The _grace period_ is given to fix whatever issue is causing the downtime. After the grace period has expired, any offline audits will fall within the scope of the tracking period, and thus will be used in the node's final evaluation. If at the end of the review period, the node is still suspended, it is disqualified. Otherwise, the node is no longer _under review_.

This file has been truncated. show original

SGC · January 28, 2022, 5:59pm

seems very professional.
maybe you want to use nftables these days…
you know… something that is actually designed to be able to work as a modern firewall.

Alexey · January 29, 2022, 8:09am

I’m agree with @SGC , you need to simplify your firewalling solution. If the satellite cannot reach your node - the online score will drop. The audit score is almost unrelated to this check - you can have a only one successful audit (before you applied your setup) and your audit score will be 100%, the online score just simple check - is node responding on audit requests or not.
Your online score shows that it’s not responding on audit requests from this particular satellite, so you need to examine your firewall rules and make sure that you do allow 28967 TCP/UDP to your node from any source (do not specify the source IP or source IP range!), normally you doesn’t need to have an outbound rules, but if you have - you need to add another one and allow any traffic from your node (and any port, i.e. you can set only local source IP, but not the source port!) to any destination (any IP and any port, TCP/UDP).

svet0slav · January 29, 2022, 10:01am

It is actually quite simple - block everything, allow only what is used.

This is what I am trying to figure out because no other node on the same machine has this issue. Maybe too fast to judge - timed out because of many requests. Seems weird. The connection is 1Gbps and uncapped by any shaper.

That is definitely allowed.

Sure. The following rules are set.

iptables -t filter -P OUTPUT ACCEPT
ip6tables -t filter -P OUTPUT ACCEPT

I would like to repeat this key part again - no other node on the same machine has this issue*. Some even started getting the 1.47.3 update already.

And looks like audits passed.

root@server:~$ journalctl -u nodeX | grep GET_AUDIT

Jan 27 02:14:30 server storagenode[2767]: 2022-01-27T02:14:30.332Z INFO piecestore download started {“Piece ID”: “N4IGAQJSQB2HD27S7ANB6MXFIH6X4VZU4INH4IVHQ3UBWDAI62VA”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “GET_AUDIT”}
Jan 27 02:14:30 server storagenode[2767]: 2022-01-27T02:14:30.396Z INFO piecestore downloaded {“Piece ID”: “N4IGAQJSQB2HD27S7ANB6MXFIH6X4VZU4INH4IVHQ3UBWDAI62VA”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “GET_AUDIT”}
Jan 27 06:08:14 server storagenode[2767]: 2022-01-27T06:08:14.588Z INFO piecestore download started {“Piece ID”: “LZ3BHYH7NTNKBRRXD5QH6BHY53ZJCFCHUMBLLTAKV2T5BGUXDQVQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
Jan 27 06:08:14 server storagenode[2767]: 2022-01-27T06:08:14.725Z INFO piecestore downloaded {“Piece ID”: “LZ3BHYH7NTNKBRRXD5QH6BHY53ZJCFCHUMBLLTAKV2T5BGUXDQVQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
Jan 27 17:10:30 server storagenode[2767]: 2022-01-27T17:10:30.909Z INFO piecestore download started {“Piece ID”: “WDVVIW5F6SGSCW3WO37GWN5OBD2V75PB3X3W5AKQWPX4MBBOHAPA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
Jan 27 17:10:31 server storagenode[2767]: 2022-01-27T17:10:31.046Z INFO piecestore downloaded {“Piece ID”: “WDVVIW5F6SGSCW3WO37GWN5OBD2V75PB3X3W5AKQWPX4MBBOHAPA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
Jan 27 18:09:39 server storagenode[2767]: 2022-01-27T18:09:39.473Z INFO piecestore download started {“Piece ID”: “G6ZQVOGMD5ENJUVREPWTP2SDA2VL5F6XRN7B4VRZVWLNI33OTQAQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_AUDIT”}
Jan 27 18:09:39 server storagenode[2767]: 2022-01-27T18:09:39.760Z INFO piecestore downloaded {“Piece ID”: “G6ZQVOGMD5ENJUVREPWTP2SDA2VL5F6XRN7B4VRZVWLNI33OTQAQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_AUDIT”}
Jan 27 22:30:23 server storagenode[2767]: 2022-01-27T22:30:23.999Z INFO piecestore download started {“Piece ID”: “HZ2YZDJH46NFXNMM3F2KV2TJN4WM4MRDPZWJ4XVW46772MOJ5ETQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
Jan 27 22:30:24 server storagenode[2767]: 2022-01-27T22:30:24.249Z INFO piecestore downloaded {“Piece ID”: “HZ2YZDJH46NFXNMM3F2KV2TJN4WM4MRDPZWJ4XVW46772MOJ5ETQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
Jan 28 01:21:33 server storagenode[2767]: 2022-01-28T01:21:33.248Z INFO piecestore download started {“Piece ID”: “TM7OROLQN5LTXV7RGN7YSYCLUXOOJRYB2DVTQCP6OQ43XXHBMLHA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET_AUDIT”}
Jan 28 01:21:33 server storagenode[2767]: 2022-01-28T01:21:33.502Z INFO piecestore downloaded {“Piece ID”: “TM7OROLQN5LTXV7RGN7YSYCLUXOOJRYB2DVTQCP6OQ43XXHBMLHA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET_AUDIT”}
Jan 28 05:36:24 server storagenode[2767]: 2022-01-28T05:36:24.518Z INFO piecestore download started {“Piece ID”: “W7SGAS32LJ24IMGF2HMMMNZYX3RECZJ4G7P2BIFHQNCVJLPEBULQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
Jan 28 05:36:24 server storagenode[2767]: 2022-01-28T05:36:24.669Z INFO piecestore downloaded {“Piece ID”: “W7SGAS32LJ24IMGF2HMMMNZYX3RECZJ4G7P2BIFHQNCVJLPEBULQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
Jan 28 21:53:20 server storagenode[2767]: 2022-01-28T21:53:20.948Z INFO piecestore download started {“Piece ID”: “HMRDR6F6PTXDSTFFXFGUOM2E4JS7QEJ3AFAWFIWRKLAPKU7KAR7A”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_AUDIT”}
Jan 28 21:53:21 server storagenode[2767]: 2022-01-28T21:53:21.015Z INFO piecestore downloaded {“Piece ID”: “HMRDR6F6PTXDSTFFXFGUOM2E4JS7QEJ3AFAWFIWRKLAPKU7KAR7A”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_AUDIT”}
Jan 28 23:03:00 server storagenode[2767]: 2022-01-28T23:03:00.516Z INFO piecestore download started {“Piece ID”: “LXIBWKFUJTI3GLKMUWV4AMUHCX5UM65P5EATOORGRVWVIQERP2ZQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_AUDIT”}
Jan 28 23:03:00 server storagenode[2767]: 2022-01-28T23:03:00.578Z INFO piecestore downloaded {“Piece ID”: “LXIBWKFUJTI3GLKMUWV4AMUHCX5UM65P5EATOORGRVWVIQERP2ZQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_AUDIT”}

So, what I believe is this is NOT:

connection speed related (too many requests may lead to timeouts)
firewall related (obviously not - ports are open, audits are success, communication is possible)
OS related (no limits set, on the contrary - quite increased and resources are high on this machine)
hardware failure related (like not being able to cope with the requests it gets)
CloudFlare DNS problem, if they have some interruptions on their mega hyper TB speed network - this is what is being used as DNS because domain name is utilized instead of IP because of the dual setup with IPv6. Ha-ha! Ha-ha-ha! Maybe this is what it could be, but it is not proxied. Just using the free DNS for A and AAAA records.

What I believe it is:

bug - sat ap1 thinks nodex offline all the time, while it keeps on passing all audits and keeps on getting traffic from it.

Could any storjling check the logs on the sat? I can provide the node ID, if needed, no matter the piece IDs are visible in the audit log output above and the node could be easily identified. Then we would know what the sat thinks of the node.

TheMightyGreek · January 29, 2022, 1:20pm

It just sounds like you don’t want to be helped, you’re stubborn and don’t want to admit that the problem is probably on your end. Instead you suggest some weird bug on a satellite that’s serving thousands of other nodes with no problem. And not only on this issue, I find you consistently condescending every time someone questions your setup.
Most people are here to help but please understand that your setup is quite unique so it’s understandable than most people would expect the issue there and not in the satellite software.
However it’s weird that you get audits, hopefully that means the satellite considers the node online.

TheMightyGreek · January 29, 2022, 1:25pm

Might have to take some of that back

Alexey · January 29, 2022, 3:15pm

I can assume that you have a separate rules for each node. Worth to check the rule for this particular node and compare with working ones.

This is an application, there is no magic. If something working not as expected, check what’s different.

Your port maybe openeded for the port checker, but closed for this particular satellite (you may be blocking somewhere in your setup only some IPs based on the range or the rule for this node has an error or typo).

Perhaps this satellite cannot reach your node only periodically.
You can check date time intervals, when your node were offline: