Just noticed in my emails from overnight that Asia also triggered a suspension email. Timing is the same as others here.
No errors during that time in the logs. Audits are 100.0%. No indication of an issue on the dashboard.
I do see ERRORs for database locked sprinkled around but the times don’t line up at all. Considering they just implemented these suspensions yesterday and given all of these seemingly false positives reported here, I think Storj has some work to do on the triggers.
From a feedback perspective, the email itself should include a support link that tells SNOs how to identify the symptoms and common resolutions for different scenarios. Also a notification when it is off suspension is needed.
Node ID: 12UzxJS8iFiYuL7AqpSE7tmjkp8B4aCNopjNZ5yrQP3TBymgCzc
Your Storage Node on the asia-east-1 Satellite was suspended. You were suspended on 2020-04-23 at 06:06 UTC.
Looks like I have one too. 1UGvhMLgbvhDgauKayZDogrvect5Ekv3C4zxXMEUdipSvFJtHy, eu-west-1
162 “database is locked” errors, 130 of them are “failed to add order”. 0 failed audits in node logs, audit score in sno dashboard is, however, under 100% for all satellites.
Seagate SMR drive - ST8000AS0002.
No other suspensions so far even though I have more nodes on same drives, one of the nodes is now under extremely heavy io load.
Hey everyone, We are sorry for all of the confusion with these suspension mode emails. Your nodes are not going to be DQ’ed because of this. We prematurely setup automatic emails for suspension mode, these emails have now been paused so you should no longer be receiving them and we will enable them again once we are able to fix the known issues.
The issue is in the storage node logs we are missing some error logs that tell you if you failed an audit or not. Suspension mode is currently triggering too rapidly so nodes are going in and out of suspension mode within minutes. Don’t worry your node will NOT be DQ’ed because of this. We also found an issue with Storage Node database locking up which we are investigating.
Thank you for all of your feedback on this topic. this feedback has been tremendously helpful for us to track down and resolve the last issues with this feature!
Is it possible to have those kinds of logs in a separate one? Having a single log with all the traffic (valid or not) and the system errors are not really usable if you are aiming long uptimes on big node.
Currently I have to stop and restart the service every 2 week because the log goes bigger than 2GB witch I’m not comfortable with (and it’s a waste of space).
Having a storagenode_trafic.log and a storagenode_system.log would be much more logical and easy to investigate.
I also received an email regarding node suspension, grepped the log for related errors, none found
also took a look at the dahsboard api and it says audit totalCount: 178, audit successCount: 178, score is 1
as you already mentioned this could be related to some error not getting logged, but im not sure it wouldnt get written to the db used for the audit errors, were such errors should show up, or are they not shown/visible in the success count/total count or score variables?
in case you want to go through the logs, the node id is 1yq49oGxkNRLHUimexB7euadsqH1mhc6CkXJjZnYXe3p7ebUex and the suspended sattelite is saltlake
Thank you @brandon for the update.
I missed this in the blueprint phase, but I think notifications to the SNO is a big area for improvement and this will only increase the importance.
I didn’t see my thoughts on notifications on the ideas portal, so I submitted one there. It would be great for the SNO to be able to configure notifications(and which notifications) to the service they prefer. https://ideas.storj.io/ideas/V3-I-191
As long as “storagenode.exe” is running the “storagenode.log” files is locked and you can’t delete it (it said it is open by a process). You have to stop the service to delete or rename the file yourself.
If at the end of the day “storagenode.exe” stopped writing to “storagenode.log”, renamed it “storagenode-20200432.log” and start wrinting a new “storagenode.log” it would release the previous day logs and you could delete them without stopping the service.
Actually there is another solution, when you open a file in windows API you have 2 modes exclusive or not. If it’s exclusive only your process can mess with it, it’s safer but for logs not always the best option. If you open it in write mode, other people can write on it and /or delete it…
I would still keep logs if you can afford some space on your HDD. They are very critical in finding bugs and saving lot of troubleshoot time. I agree its annoying to stop and rename log file but its worth the trouble. You can transfer that file elsewhere but having it has more pros than cons.