# Now it hit me: 'Your node has been suspended' **URL:** \ **Category:** troubleshooting\ **Tags:** suspension\ **Created:** [February 11, 2022, 9:37am UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239 "2022-02-11T09:37:16Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Bivvo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/bivvo/32/7825_2.png) [@Bivvo](https://forum.storj.io/u/Bivvo)\ **Post date:** [February 19, 2022, 10:31pm UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/23 "2022-02-19T22:31:05Z")
Only one or two failures? That can’t be. How huge are your logs? Can you share your run command and uptime? And how old are your nodes? All 3 running on the same machine? ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [February 19, 2022, 10:31pm UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/24 "2022-02-19T22:31:09Z")
And out of suspension for node1: ![node1_out](https://storj-s3.bcdn.literatehosting.net/original/2X/c/c668139ba85b6e70e53f13321982d7f1ebdb8de3.jpeg) Storj, seriously? ---
**Author:** ![anon27637763](https://storj.bcdn.literatehosting.com/letter_avatar/anon27637763/32/5_5575768a8748004e209b776fc1b2916d.png) [@anon27637763](https://forum.storj.io/u/anon27637763)\ **Post date:** [February 19, 2022, 10:48pm UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/25 "2022-02-19T22:48:05Z")
> [@jammerdan](#): > > out of suspension for node1: [https://forum.storj.io/t/node-suspended-randomly/17093/29](https://forum.storj.io/t/node-suspended-randomly/17093/29) @javcia says… > I started getting random suspensions on my nodes after enabling QUIP. Already disabled on one of the nodes and looks like no more suspensions. Do you have UDP/QUIC enabled? If so, try disabling it. If disabling QUIC stops suspensions, then we have a UDP/QUIC problem. * * * EDIT: Adding a data point… My node is more than 24 months old. Version 1.48.2 … I have not enabled UDP/QUIC. Therefore, my node displays “QUIC Misconfigured”. Recently, I had some ISP outage issue … fiber laser power problem, out of my control entirely… Here are my stats: ![Screenshot_2022-02-19_21-35-47](https://storj-s3.bcdn.literatehosting.net/original/2X/4/48c46763557421911fc22494e63a66ef05884431.jpeg) ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [February 19, 2022, 10:48pm UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/26 "2022-02-19T22:48:18Z")
Could you show what the errors on the GET\_REPAIR are? Though, they may not show all of the problem, as you must have failed more for the score to drop this much. But it may give a clue as to what might be going on. ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [February 19, 2022, 10:49pm UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/27 "2022-02-19T22:49:03Z")
I have rebooted and now my logs are gone. ☹ ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [February 19, 2022, 10:52pm UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/28 "2022-02-19T22:52:10Z")
Ahh that’s a shame. Since you’ve had returning issues now, it may be a good idea to redirect your logs to a file to prevent losing information. > **[How do I redirect my logs to a file?](https://docs.storj.io/node/resources/faq/redirect-logs/)** > > How do I redirect my logs to a file? # 1. To redirect the logs to a file, stop your Node: > CLI Install docker stop -t 300 storagenode GUI WIndows Install Elevated PowerShell: > Stop-Service storagenode Or click the “Stop” button in the Windows Services... ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [February 19, 2022, 10:54pm UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/29 "2022-02-19T22:54:18Z")
Yeah. I keep forgetting that. As Windows guy you always like to reboot without second thought because it can only get better… 😁 ---
**Author:** ![anon27637763](https://storj.bcdn.literatehosting.com/letter_avatar/anon27637763/32/5_5575768a8748004e209b776fc1b2916d.png) [@anon27637763](https://forum.storj.io/u/anon27637763)\ **Post date:** [February 19, 2022, 10:58pm UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/30 "2022-02-19T22:58:47Z")
Too tempting to insert a Linux v. Windows joke… > [@jammerdan](#): > > Windows guy you always like to reboot “OK → Next → Next → OK → Install virus → Next → OK → BSOD” ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [February 20, 2022, 2:37am UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/31 "2022-02-20T02:37:19Z")
You can replace “install virus” with “allow Windows to rollout a new Windows version” ---
**Author:** ![anon27637763](https://storj.bcdn.literatehosting.com/letter_avatar/anon27637763/32/5_5575768a8748004e209b776fc1b2916d.png) [@anon27637763](https://forum.storj.io/u/anon27637763)\ **Post date:** [February 20, 2022, 2:53am UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/32 "2022-02-20T02:53:34Z")
I love Geek humor! The last MS Windows version I actually ran on anything for more than 10 minutes was XP… back in 2009… My MS Windows knowledge is so out of date that I can’t actually use a Windows machine at all anymore. When asked to troubleshoot a Windows machine, I immediately try to find the terminal and type in lspci or lsblk … After trying `man cmd` I realize the machine is broken and needs Debian therapy. * * * Back to OP… I suspect UDP/QUIC is the problem. My guess is the audit checks are timing out due to packet loss … maybe related to hardware speed and/or Internet connection quality. I briefly tried paging through OP’s posts to figure out what kind of issues were encountered in the past… but it’s fairly difficult to put everything together if I haven’t been paying close attention. I might be completely wrong … it’s happened before, and will happen again… but I asked the question anyway. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [February 20, 2022, 2:56am UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/33 "2022-02-20T02:56:56Z")
The only question - why so small number of nodes? We have a lot of nodes enabled UDP, but it hits only a small percentage of them. ---
**Author:** ![anon27637763](https://storj.bcdn.literatehosting.com/letter_avatar/anon27637763/32/5_5575768a8748004e209b776fc1b2916d.png) [@anon27637763](https://forum.storj.io/u/anon27637763)\ **Post date:** [February 20, 2022, 3:02am UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/34 "2022-02-20T03:02:19Z")
> [@Alexey](#): > > nodes enabled UDP, but it hits only a small percentage UDP has too many variables to consider. The entire pathway to the node needs to pass UDP packets cleanly. That doesn’t always happen. Since there is no way to ensure QoS of the entire pathway, a percentage of SNOs will have a problem… leading to failed audits. It’s just a guess. Maybe QUIC audit fail, followed by a TCP sanity check – then hard fail… I don’t know. ---
**Author:** ![mr.Viking](https://storj.bcdn.literatehosting.com/letter_avatar/mr.viking/32/5_5575768a8748004e209b776fc1b2916d.png) [@mr.Viking](https://forum.storj.io/u/mr.Viking)\ **Post date:** [February 20, 2022, 9:15am UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/35 "2022-02-20T09:15:58Z")
My ISP periodically can drop down my internet (one time in month about 1-5 minutes of downtime). When its has been happened again and internet has been up I got message about suspended nodes. It is first time when I got this messages. May be this information can help to troubleshoot. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [February 20, 2022, 9:41am UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/36 "2022-02-20T09:41:31Z")
> [@mr.Viking](#): > > I got message about suspended nodes. Perhaps you are talking about the online score. The downtime doesn’t affect the suspension score, which can be affected only by problems with audits, it’s not related to downtime. The difference is huge - the online score can be 0%, but your node could be disqualified after downtime more than 30 days. But in case of failed audits your node can be disqualified if audit score would fall below 60% The suspension score is affected when your node online, answers on audit requests but return unexpected error instead of piece. If problem would not be fixed, sooner or later your node will be disqualified. The known errors like “file not found” or “disk i/o” affects the audit score immediately. So reducing of suspension score is much more dangerous than reducing online score. The reducing of audit score even worse - it leads to disqualification straight away. ---
**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\ **Post date:** [February 20, 2022, 10:08am UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/37 "2022-02-20T10:08:30Z")
> [@anon27637763](#): > > Maybe QUIC audit fail > ---
**Author:** ![anon27637763](https://storj.bcdn.literatehosting.com/letter_avatar/anon27637763/32/5_5575768a8748004e209b776fc1b2916d.png) [@anon27637763](https://forum.storj.io/u/anon27637763)\ **Post date:** [February 20, 2022, 11:38am UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/38 "2022-02-20T11:38:19Z")
It’s good that problems are being identified and fixed so quickly… But it’s unclear if your post is meant to mean that OP’s issue isn’t connected to QUIC because the node seems to have re-entered suspension less than 3 days ago… Or if you mean that OP’s issue may be QUIC related because audits through QUIC have been reported to be problematic. ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [February 20, 2022, 1:03pm UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/39 "2022-02-20T13:03:25Z")
So let’s check: Node1 has recovered more but on 2 satellites it went down again: ![Screenshot_2022-02-20_134542](https://storj-s3.bcdn.literatehosting.net/original/2X/8/8f57cd7b78068693397daa386b5cf5d25dcde769.jpeg) ```auto docker logs storagenode1 2>&1 | grep -E 'GET_AUDIT' | grep 'failed' | wc -l 0 docker logs storagenode1 2>&1 | grep -E 'GET_REPAIR' | grep 'failed' | wc -l 7 ``` Errors were: 6x `write: broken pipe` and 1x `use of closed network connection` Node2 seems to be slowly recovering on all but one satellite: ![Screenshot_2022-02-20_135238](https://storj-s3.bcdn.literatehosting.net/original/2X/6/6c34bb2a803aaad9c302932cddcbf144361700af.jpeg) ```auto docker logs storagenode2 2>&1 | grep -E 'GET_AUDIT' | grep 'failed' | wc -l 0 docker logs storagenode2 2>&1 | grep -E 'GET_REPAIR' | grep 'failed' | wc -l 0 ``` Node3 which seems to be recovering on some and doing worse on other satellites: ![Screenshot_2022-02-20_135732](https://storj-s3.bcdn.literatehosting.net/original/2X/2/2def6479f46c6c3a619af3dbc0e2521ca0008a60.jpeg) ```auto docker logs storagenode3 2>&1 | grep -E 'GET_AUDIT' | grep 'failed' | wc -l 0 docker logs storagenode3 2>&1 | grep -E 'GET_REPAIR' | grep 'failed' | wc -l 2 ``` Errors are: 1x `use of closed network connection` and 1x `broken pipe` Maybe this? [Load tests on Storage nodes?](https://forum.storj.io/t/load-tests-on-storage-nodes/17356) ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [February 20, 2022, 3:53pm UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/40 "2022-02-20T15:53:44Z")
What I see frequently now is this: ```auto "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked ``` I see that only for bandwidthdb, no other database affected it seems. Also, disk load does not seem to be so high that it should turn into locking. ---
**Author:** ![SGC](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/sgc/32/1132_2.png) [@SGC](https://forum.storj.io/u/SGC)\ **Post date:** [February 20, 2022, 4:37pm UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/41 "2022-02-20T16:37:31Z")
i use to see that when my storage was heavy loaded, it might not seem like its loaded, but it most likely is… its because the write io might not look like a lot of work on the disk, you should check its latency with something like netdata ---
**Author:** ![unrealSpeedy](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/unrealspeedy/32/1122_2.png) [@unrealSpeedy](https://forum.storj.io/u/unrealSpeedy)\ **Post date:** [February 20, 2022, 6:34pm UTC](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239/42 "2022-02-20T18:34:20Z")
This. Had the Suspensions too on nodes without QUIC enabled but HDD was in 100% load. [Previous page](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239.md?page=1) [Next page](https://forum.storj.io/t/now-it-hit-me-your-node-has-been-suspended/17239.md?page=3)