# My node has been auto-suspended **URL:** \ **Category:** troubleshooting\ **Tags:** suspension\ **Created:** [May 20, 2020, 9:56am UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978 "2020-05-20T09:56:56Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![niccotnt](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/niccotnt/32/1124_2.png) [@niccotnt](https://forum.storj.io/u/niccotnt)\ **Post date:** [May 20, 2020, 9:56am UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/1 "2020-05-20T09:56:56Z")
This night my node has been suspended by 3 satellites with no reason at all. Node is fully functional, no power loss, connected to UPS, no failures at all. When i checked logs, many errors “database is locked” were present. I restarted the node, and everything went fine. Now it’s recovering fast, many PUT and GET are coming in without errors. I hope I will be unsuspended very soon. This is clearly a design flaw, sqlite is not suitable for multi-threaded applications out of the box, it must be configured properly. This is serious bacuse nodes cannot be left unattended, I’m forced to schedule a stop/start in the crontab every day. I think storj team cannot talk about node reputation as long as the node can get auto-suspended by design flaws. ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [May 20, 2020, 10:24am UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/2 "2020-05-20T10:24:56Z")
It’s a known issue that’s being worked on. In the mean time you can try to vacuum and defrag the databases. This has helped some users, though it won’t eliminate the issue entirely. Hopefully an update will be ready soon to get rid of some of these issues. May I ask, is it the used\_serial.db file for you as well? Or orders.db? Those are the two we’ve seen have issues so far. ---
**Author:** ![niccotnt](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/niccotnt/32/1124_2.png) [@niccotnt](https://forum.storj.io/u/niccotnt)\ **Post date:** [May 20, 2020, 10:37am UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/3 "2020-05-20T10:37:26Z")
The locked file was used\_serial.db. Can you confirm these issues won’t affect reputation, until they are fixed? Thanks ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [May 20, 2020, 10:59am UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/4 "2020-05-20T10:59:21Z")
> [@niccotnt](#): > > I think storj team cannot talk about node reputation as long as the node can get auto-suspended by design flaws. Well that means you don’t expect a response from our side. Ok thats fine for me 😃 ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [May 20, 2020, 11:17am UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/5 "2020-05-20T11:17:00Z")
Currently it can lead to suspension, but not disqualification, that has been paused for now while the issues are being resolved. So it won’t effect anything permanent. Reputation recovers quickly, so I wouldn’t worry about that too much. Storjlabs is actually pretty responsive with this stuff, everything I’m telling you I heard from them. > [@littleskunk](#): > > Well that means you don’t expect a response from our side. Ok thats fine for me 😃 I doubt that was @niccotnt’s intention. 😉 Feel free to add anything I haven’t mentioned or correct anything I’ve been wrong about. ---
**Author:** ![niccotnt](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/niccotnt/32/1124_2.png) [@niccotnt](https://forum.storj.io/u/niccotnt)\ **Post date:** [May 20, 2020, 11:24am UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/6 "2020-05-20T11:24:44Z")
Sorry english is not my native language, I mean “I think storj team cannot trust the node reputation algorithm, until these issues are fixed.” ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [May 20, 2020, 11:44am UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/7 "2020-05-20T11:44:44Z")
I understand that suspension mode is not working great. We are working on a solution. There is a different thread that outline some of the solutions. In this thread I want to explain why we need suspension mode in the meantime even if it is not great. Disabling it would make it worse. Your node is returning an error on audits. The error might not be correct but we can’t simply ignore it. If we would ignore the audit error that would mean we risk file durability. A repair download or a customer download will hit the same error message. Thanks to suspension mode we are aware of that error rate and can trigger repair before we are danger of losing any files. Many storage node operators also managed to decrease the error rate by calling vacuum. From that perspective suspension mode is working as intended and we need to keep it enabled. Disqualification after 7 days in suspension is still disabled. ---
**Author:** ![niccotnt](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/niccotnt/32/1124_2.png) [@niccotnt](https://forum.storj.io/u/niccotnt)\ **Post date:** [May 20, 2020, 12:01pm UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/8 "2020-05-20T12:01:41Z")
Yes I agree with you, I’m confident you keep disqualification for suspension disabled until the “database locked” issue is resolved. Of course the main goal for operators is to leave the nodes unattended, setting just some basic monitoring, when all the hardware/configuration requirements are met. I would like to be notified only if some real problem occurs (i.e. disk failures, inodes corruption) and I’d like to run weekly maintenance. Checking node logs every few hours can be frustrating, expecially if I have a daytime job. ---
**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\ **Post date:** [May 20, 2020, 12:23pm UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/9 "2020-05-20T12:23:10Z")
> [@niccotnt](#): > > I would like to be notified only if some real problem occurs Use [uptimerobot.com](http://uptimerobot.com) to get notified when your node is offline. You would get email notification and then you can investigate the issue. ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [May 20, 2020, 12:51pm UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/10 "2020-05-20T12:51:12Z")
> [@niccotnt](#): > > I would like to be notified only if some real problem occurs It technically is a real problem, it’s just not one that you can fix. Would you rather have your node go into suspension mode quietly without being notified? ---
**Author:** ![niccotnt](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/niccotnt/32/1124_2.png) [@niccotnt](https://forum.storj.io/u/niccotnt)\ **Post date:** [May 20, 2020, 2:38pm UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/11 "2020-05-20T14:38:56Z")
I want notifications for suspension mode, but I do not want suspension mode gets triggered for internal issues. ---
**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\ **Post date:** [May 20, 2020, 3:16pm UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/12 "2020-05-20T15:16:53Z")
> [@niccotnt](#): > > I want notifications for suspension mode Did you use a real email address in your node setup ? Suspension emails are sent on that email. ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [May 20, 2020, 3:44pm UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/13 "2020-05-20T15:44:57Z")
If we could all predict which issues were going to pop up, that distinction could be made. But I don’t think Storjlabs has a crystal ball. Whether an issue is caused by a problem on the node system or might be related to a software shortcoming can only be found out if it pops up and is subsequently reported. When it did, they immediately took steps to prevent nodes from getting disqualified and started to work on a fix. It was already explained why suspension couldn’t just be disabled in the mean time. I understand that it’s a little annoying to be caught in this, but you can’t catch everything in internal testing. Or as Apple would say it: “We have determined that a small percentage of storagenodes may exhibit problems with database locks” Difference is, when Apple introduced a bad keyboard design they stuck with it for years across several models. Storjlabs is working on an actual fix right away. It just takes some time to build that, test it internally and push the release. ---
**Author:** ![Hacker](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/hacker/32/1129_2.png) [@Hacker](https://forum.storj.io/u/Hacker)\ **Post date:** [May 20, 2020, 8:30pm UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/14 "2020-05-20T20:30:04Z")
Same here: “usedserialsdb error: disk I/O error: The device is not ready.” @littleskunk > [@littleskunk](#): > > There is a different thread that outline some of the solutions. Can you please link this thread you are referring to? Thank you. ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [May 20, 2020, 10:54pm UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/15 "2020-05-20T22:54:24Z")
We all have to use the same search feature… But this time I did it for you. > [@"Action": "GET\_AUDIT", "error": "usedserialsdb error: database is locked",](https://forum.storj.io/t/action-get-audit-error-usedserialsdb-error-database-is-locked/6424/6): > > I’m running a docker version on Debian GNU/Linux. Here’s the procedure I use to vacuum and check the databases: Stop the node Vacuum the dbs integrity\_check the dbs Restart the node Here’s a bash check which should do that for you. Please change the database directory to reflect where yours are on your system. docker stop -t 300 storagenode && dbs=$(ls /opt/storj/storage/\*.db) c1="VACUUM;" c2="PRAGMA integrity\_check;" for i in $dbs do sqlite3 $i "$c1" done for i in $dbs do sqlite3 $i "… This script is for linux+docker. I’m not sure what your setup is. ---
**Author:** ![Hacker](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/hacker/32/1129_2.png) [@Hacker](https://forum.storj.io/u/Hacker)\ **Post date:** [May 21, 2020, 12:04am UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/16 "2020-05-21T00:04:20Z")
Thank you very much. I didn’t know what to search for and wouldn’t have known if I found the thread littleskunk has been referring to. I am on Windows 10 but I can adapt the workflow. I have already performed an integrity check about a week ago due to suspected database corruption (which I tried and hopefully fixed for pricing.db) but can perform it again and I can also include a vacuum beforehand. Thanks again! ---
**Author:** ![niccotnt](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/niccotnt/32/1124_2.png) [@niccotnt](https://forum.storj.io/u/niccotnt)\ **Post date:** [May 21, 2020, 9:10am UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/17 "2020-05-21T09:10:53Z")
My node has been un-suspended yesterday, but this night other email alerts came in… This is the situation now: ![image](https://storj-s3.bcdn.literatehosting.net/original/2X/4/4fa257c59a53d4b1fbcac7b746f5bafdaaef3566.png) I have no errors in logs, here are some screenshots: ![image](https://storj-s3.bcdn.literatehosting.net/original/2X/a/aab9b13ec04a54cba9222a729a3e4f96e0959711.png) ![image](https://storj-s3.bcdn.literatehosting.net/original/2X/9/9ea2756bf6858bac77d7d2f490a3871575063d06.png) (uptime = 2m because i restarted it) Any idea? can you provide a link explaining the “suspension” algorithm ? maybe i can understand what’s happening. ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [May 21, 2020, 9:53am UTC](https://forum.storj.io/t/my-node-has-been-auto-suspended/6978/18 "2020-05-21T09:53:50Z")
There’s a blueprint here that goes into how it works, but it won’t contain specifics about this database locked issue. > Keep in mind that disqualification after suspension isn’t active atm.