Constant decline of the online score on the eu1.storj.io satellite

alarmed with this tread i checked my nodes,

some have EU stats loosing up to 94-90%, and some saltlake too, and US and AP in norm (98%) i will monitor situation, but my blind guess basted on some old observations with uptimerobot is this: online audit gets rejected when traffic hits like last weeks. i will check later and compare. and post results.

(nodes are in EU countries)

1 Like

But traffic no longer hits like last week and stats continue to drop. This means it is still happening.

Considering online score is measured as an average over the last 30 days and assuming my online score was 100% for the days 18 days before 26 September and considering my online score now is 99.58%, I deduce that for the last 12 days the average online score would be 99.20% in my case. If things continue as they are now, I will probably see my online score converge to this value. Not a big problem, but strange this affects EU1 only nonetheless.

what region are you from? EU or US?

I am in EU (Madrid, Spain).

i know some spain IPs have problem, so i refused from Spain nodes

this is starting to be a problem…

Node in Europe…

Thank you! I shared with the team.
And I think I was wrong, mtr from your node to the satellite could be not helpful, because auditors are separate processes and they not necessarily run near the satellite itself :frowning:
Anyway, I would hope that this information would help to confirm or fix the issue.

auditor instance has been moved recently between cloud providers.

There is a small change – but still possible – that the network of the new provider is not as reliable as the old one (GCP), and as we think it is.

To further debug this issue, I moved back EU1 auditor to GCP (and planning to do the same with AP1). Let’s wait a few days and check if it helps…

7 Likes

Isn’t EU1 the only sat that has been switched to hashstore? At least my nodes changed 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs.migrate themselves to {"PassiveMigrate":false,"WriteToNew":true,"ReadNewFirst":true,"TTLToNew":true}

It’s likely irrelevant to the current issue, because we also have had a similar drop in online score on AP1: Online status degrade on ap1

Please check the audits history with the scripts above - is there an improvement?

In my case EU1 score is now stable. It has not dropped since you made that change.

1 Like

The number of answered audits should increase.

No, it isn’t.

Rollout has been started on SLC,AP1,EU1. But we stopped increasing the rollout percentage, to rollout first the Windows memory improvement.

1 Like

will it be update directly from 137 to 139 or it will be 137, 138, 139

Not yet decided. But we consider skipping 138.

It is probably a bit early to be certain, but the change Elek made 2 days ago is looking promising - no drops since then.

Will report back again in a couple more days

1 Like

There is a confirmation of the online score growing from the linked thread:

Yeah, seems to be going up now :slight_smile:

1 Like

Not really going up yet - I expect we’ll have to wait a couple more weeks for that, but certainly less decline

I can confirm as well.