Sudden drop in Egress and Audit

My new node (45 days) was running smoothly. Suddenly a couple of days ago, the Egress started dropping and now the audit score is dropping as well.

I tried updating the node - no improvement
I tried restarting the node - nothing either

Ingress is good but Egress is just poor. I don’t know how to fix this. Audit score is what worrying me. Audit score is at 99.5 now.

PS. I have zero downtime till now.

Need help and suggestions.

While the egress does appear to have slumped, it looks to me more likely to be a change of customer behaviour rather than an issue with your node. One of my own nodes of a similar age (135GB stored) has seen 12GB of egress this month so far, close enough to your egress. That peak might simply have been a customer downloading a big file. Also, without knowing the actual size of the node its hard to get a ballpark figure of how much egress you should have seen. There are lots of other factors that affect egress: Node size, Node location, Customer location, Node performance, Customer behaviour, etc.

For the audit score, just to make sure, are you seeing the 99.5% in the “Online” section or the “Audit” section of the sattelites?

If it is a 99.5% online score, it is fine. While you say you have had 0 downtime up to now, it is possible that during a reboot (I can see the node uptime is 43h) an audit was missed. As the node is young, a single missed audit can have a moderate impact on the online score. A 99.5% online score is perfectly fine (100% would be better of course, has to be above 60% to avoid suspension).

If the 99.5% is the “Audit” score, then that can be an actual issue. Going below 98% can result in a DQ. To check why these audits have failed, please search the logs. Any failed audits should appear with why it failed (e.g. did not find a file, did not respond in time, etc). If this is the case and the node is failing data audits, it is very important to figure out what is going wrong and fix it before its too late.

2 Likes

99.5 Audit isnt an issue, it may be a blip with your ISP. If it continues to go down, let us know. Otherwise it should go back up after a while.

Thanks for the reply. My online score is 100. Audit is 99.5. Yes, I restarted the node to see if it fixes the issue with the audit. I ll go through the logs and see any uncertainty.

Sure. Will go through the logs and monitor if it further dips. Thanks

Found something in the log. Log file is about 1.8GB, don’t know if it’s normal. But I found out a lot of race lost or node shutdown errors while uploading.

One more thing node restart on its own frequently now.

While races lost are normal and fine, having that many also indicate an underlying issue/bottleneck.

Could you please provide more information about your current setup?

In addition, search the logs for other errors that may relate to the audits. If you see an “ERROR” on upload/download and the log shows “unexpected EOF” that is also (as far as I know) fine and common. We are looking for any other logs that may indicate why your node is restarting, or why the audits are failing.

I ll see more logs. My set up is

I have 3 nodes running in this pc. This node is the oldest (45 days). And the other 2 nodes are 30 days old. Those 2 nodes doesn’t have any issues. They both have very low Ingress and Egress but they are consistent and audit is 100.

PC spec

Ryzen 3900X
64GB RAM
OS installed in 2TB nvme
3X 16TB HDD

There network filters nodes by subnet so having 3 nodes wont gain you more data than having a single node. If you have multiple drives it is typically best to wait until one is close to full before starting another node.

That said, if these three nodes are working against the same drive you may be having issues with the I/O keeping up with all the activity the nodes generate.

2 Likes

Three nodes have separate HDDs. But I do note some kind of I/O errors for the same HDD in the event viewer. I have restarted the entire PC now and all the nodes are back to their best. Ingress is healthy and so does Egress. I expect Audit to be back to 100 soon. Lets wait and see.

2 Likes

If you have not already, it might be a good idea to run a S.M.A.R.T. test to get an overview of the health of your drive. If the drive is failing, you could always migrate to a new drive before that happens so you dont lose the progress you already have.

2 Likes

With all respect, falled audit score is an issue, no matter how small. This is mean, that your node is failed an audit, either a missed piece or corrupted. This will not heal itself, the satellite may audit this piece over and over again, until it wouldn’t be deleted from your node by the customer.
So, this is an issue.
@Mjdaran
You must figure out, what’s going on by analyzing your logs:

You need to check your logs for failed GET_AUDIT and GET_REPAIR, failed uploads/downloads doesn’t matter for your node’s reputation.

@Mjdaran it’s also the break of Supplier Terms & Conditions
You must not run more than a one node on the single drive.

Fair enough. In one thread we are told that in some cases a failed audit over a length of time due to caching failure is fine. In another it is not fine to fail any audit. We need to be consistent in our messaging. If the fsync is disabled and end users are occasionally going to fail an audit when they reset unexpectedly, do we still think any failed audit is a problem and needs investigating?

It’s an honest question because with fsync disabled now we are going to get plenty of SNO’s who are likely to see a 99.5% audit score.

I’m sure that any failed audit is a problem. fsync didn’t add much, if the hardware is unstable.

1 Like