Suspension score dropped with no errors, then reverted

Node Operators troubleshooting
1

What would be the reason for suspension score dropping with no log entries shown with “grep GET_AUDIT | grep failed”?

Online score on satellite I’m least online on at 99.3%, all audits at 100%. Suspension scores on satellites below 100% at 94.26%, 99.99%, 99.47%.

2

Well now they’re all back to 100%, what the heck…

3

Try to search for ERROR, they should catch it.
And also, if you re-created the container, the previous logs are lost and now you have only fresh logs without errors.

4

I did also search with “grep GET_AUDIT | grep error", no results. This is on logs that go back months, last error was in 10th month…

5

It’s case sensitive, you need to search for ERROR or use -i option to ignore case.

6

Right, right, that was my bad on the case!

“grep GET_AUDIT | grep ERROR” return the same results.

7

Something similar happend to me: Now it hit me: 'Your node has been suspended' - #4 by jammerdan

It is very possible that your node was so busy or whatever that it could not even log anything.

8

It is possible I was doing something on the computer that bogged down the resources but I fail to see how it would be possible to not generate the log entry. If my understanding is right you would need to have an issue with returning audit data - drop the audit request packets on the network side; or receive and fail to respond, caused by bogged down disk, which would generate a log entry; or the node to crash which would be obvious. Looks like something in-between happened.

9

I had this happen on 2 separate windows nodes (on different machines) yesterday. Both machines are solely used for storj and when i check the logs there was no errors at all. Checked nodes this morning both nodes recovered from around 54% back to 100%.

10

Lol. Same here yesterday. 2 nodes suspended due to heavy disk usage. But I did not bother to find a cause as the suspended score recovered almost immediately when the disk relaxed again.

11

Yeah, looks like a peak yesterday, though I was shuffling some data around in the background as well. But there should be a log entry, I’m curious to know what went wrong, unless satellite got overwhelmed or something.

12

One of my nodes have these errors:

13

So the GET_REPAIR failures are dropping the suspension score?

I don’t seem to have any though. I tried looking for entries with ERROR && GET_REPAIR, excluding (“use of closed” && “broken pipe”) and didn’t get any such GET_REPAIR errors in the last 4 weeks. Didn’t get any GET_AUDIT errors altogether so I don’t know what should be wrong.

Also I didn’t find any of Alexey’s errors.

14

And also audit score, if failed with known error.

15

Hello, is there any case that the whole network faces problems? I don’t think my connection went off any time and i even got suspended on one satellite with this percentages:

Suspension

59.19 %

Audit

100 %

Online

99.94 %

I also have 0 GET_AUDIT errors and just one GET_REPAIR …
What is happening?

16

Unlikely. But there is can be a bug somewhere.
Please, try to search for ERROR event. If you use a docker version and if you did not redirect logs, then likely your logs are gone after re-creation of the container and there is no traces anymore.

17

No i checked even previous logs and as i said there are not any more logs with errors than those i said above. I run the command in all logs that were used while i was offline and i know that because the logs before don’t have any errors.

From a statistical point of view though, watching all those guys getting suspended lately yeah its pretty obvious that there are some bugs going on

18

Yup, definitely more likely there’s some system bugs affecting less than 0.1% of all nodes rather than those 0.1% of nodes have some kind of internet or node issue :+1:

19

This is weird. You should have some errors anyway. The suspension score should not be affected silently.
It maybe database locks or something other, but must be.

20

That’s what I thought, there must be some log entries, but based on what I searched I could not find any either. Although mine did not drop nearly as drastically.