Suspension score dropped with no errors, then reverted

Node Operators troubleshooting
1

What would be the reason for suspension score dropping with no log entries shown with “grep GET_AUDIT | grep failed”?

Online score on satellite I’m least online on at 99.3%, all audits at 100%. Suspension scores on satellites below 100% at 94.26%, 99.99%, 99.47%.

2

Well now they’re all back to 100%, what the heck…

1 Like
3

Try to search for ERROR, they should catch it.
And also, if you re-created the container, the previous logs are lost and now you have only fresh logs without errors.

4

I did also search with “grep GET_AUDIT | grep error", no results. This is on logs that go back months, last error was in 10th month…

5

It’s case sensitive, you need to search for ERROR or use -i option to ignore case.

6

Right, right, that was my bad on the case!

“grep GET_AUDIT | grep ERROR” return the same results.

7

Something similar happend to me: Now it hit me: 'Your node has been suspended' - #4 by jammerdan

It is very possible that your node was so busy or whatever that it could not even log anything.

8

It is possible I was doing something on the computer that bogged down the resources but I fail to see how it would be possible to not generate the log entry. If my understanding is right you would need to have an issue with returning audit data - drop the audit request packets on the network side; or receive and fail to respond, caused by bogged down disk, which would generate a log entry; or the node to crash which would be obvious. Looks like something in-between happened.

9

I had this happen on 2 separate windows nodes (on different machines) yesterday. Both machines are solely used for storj and when i check the logs there was no errors at all. Checked nodes this morning both nodes recovered from around 54% back to 100%.

1 Like
10

Lol. Same here yesterday. 2 nodes suspended due to heavy disk usage. But I did not bother to find a cause as the suspended score recovered almost immediately when the disk relaxed again.

2 Likes
11

Yeah, looks like a peak yesterday, though I was shuffling some data around in the background as well. But there should be a log entry, I’m curious to know what went wrong, unless satellite got overwhelmed or something.

12

One of my nodes have these errors:

13

So the GET_REPAIR failures are dropping the suspension score?

I don’t seem to have any though. I tried looking for entries with ERROR && GET_REPAIR, excluding (“use of closed” && “broken pipe”) and didn’t get any such GET_REPAIR errors in the last 4 weeks. Didn’t get any GET_AUDIT errors altogether so I don’t know what should be wrong.

Also I didn’t find any of Alexey’s errors.

14

And also audit score, if failed with known error.

1 Like
15

Hello, is there any case that the whole network faces problems? I don’t think my connection went off any time and i even got suspended on one satellite with this percentages:

Suspension

59.19 %

Audit

100 %

Online

99.94 %

I also have 0 GET_AUDIT errors and just one GET_REPAIR …
What is happening?

16

Unlikely. But there is can be a bug somewhere.
Please, try to search for ERROR event. If you use a docker version and if you did not redirect logs, then likely your logs are gone after re-creation of the container and there is no traces anymore.

17

No i checked even previous logs and as i said there are not any more logs with errors than those i said above. I run the command in all logs that were used while i was offline and i know that because the logs before don’t have any errors.

From a statistical point of view though, watching all those guys getting suspended lately yeah its pretty obvious that there are some bugs going on

2 Likes
18

Yup, definitely more likely there’s some system bugs affecting less than 0.1% of all nodes rather than those 0.1% of nodes have some kind of internet or node issue :+1:

19

This is weird. You should have some errors anyway. The suspension score should not be affected silently.
It maybe database locks or something other, but must be.

20

That’s what I thought, there must be some log entries, but based on what I searched I could not find any either. Although mine did not drop nearly as drastically.