# All nodes crash at the same time

**URL:** <https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791>\
**Category:** troubleshooting\
**Tags:** windows-gui, multiple-nodes, hardware\
**Created:** [September 7, 2024, 5:38pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791 "2024-09-07T17:38:21Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![batelis](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/batelis/32/1136_2.png) [@batelis](https://forum.storj.io/u/batelis)\
**Post date:** [September 7, 2024, 5:38pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/1 "2024-09-07T17:38:21Z")

</div>

Hey so I set up few nodes on a new pc and I see this thing happening everyday on all 3 nodes 2 disks are used and 1 brand new and all of them crash at the same time  
all of them with same fatal error

| 2024-09-04T19:57:56+03:00 | ERROR | failure during run | {error: piecestore monitor: timed out after 1m0s while verifying readability of storage directory, errorVerbose: piecestore monitor: timed out after 1m0s while verifying readability of storage directory\n\tstorj.io/storj/storagenode/monitor.(\*Service).Run.func1.1:153\n\tstorj.io/common/sync2.(\*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(\*Service).Run.func1:140\n\tgolang.org/x/sync/errgroup.(\*Group).Go.func1:78} |
| --- | --- | --- | --- |
| 2024-09-04T19:57:56+03:00 | FATAL | Unrecoverable error | {error: piecestore monitor: timed out after 1m0s while verifying readability of storage directory, errorVerbose: piecestore monitor: timed out after 1m0s while verifying readability of storage directory\n\tstorj.io/storj/storagenode/monitor.(\*Service).Run.func1.1:153\n\tstorj.io/common/sync2.(\*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(\*Service).Run.func1:140\n\tgolang.org/x/sync/errgroup.(\*Group).Go.func1:78} |
| I know Alexey will say disks are too slow but there is nothing else running on them they running on almost 0% all day after they finish filewalkers | | | |
| this is windows btw. | | | |

---

<div class="post-metadata">

**Author:** ![Knowledge](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/knowledge/32/10681_2.png) [@Knowledge](https://forum.storj.io/u/Knowledge)\
**Post date:** [September 7, 2024, 5:51pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/2 "2024-09-07T17:51:20Z")

</div>

Well, it says it timed out waiting for a response back from the drive. So, the drive didn’t return the data in one minute of time. Perhaps your drive is going to sleep at that time due to some power setting? Or your drive is indeed slow.

---

<div class="post-metadata">

**Author:** ![Stez](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/stez/32/1130_2.png) [@Stez](https://forum.storj.io/u/Stez)\
**Post date:** [September 7, 2024, 5:53pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/3 "2024-09-07T17:53:24Z")

</div>

Something in your pc (assuming the disks are internal) is a bottleneck. You could check Process Monitor (from MS) to check the I/O and/or CPU/RAM usage. That might give you some clues.

---

<div class="post-metadata">

**Author:** ![batelis](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/batelis/32/1136_2.png) [@batelis](https://forum.storj.io/u/batelis)\
**Post date:** [September 7, 2024, 5:53pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/4 "2024-09-07T17:53:38Z")

</div>

But on all nodes at the same time not like 10minutes difference, same minute same second.  
Sleep settings set to never sleep too

---

<div class="post-metadata">

**Author:** ![Julio](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/julio/32/1132_2.png) [@Julio](https://forum.storj.io/u/Julio)\
**Post date:** [September 7, 2024, 6:27pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/5 "2024-09-07T18:27:14Z")

</div>

There are only two devices that can freeze/seize up Windows, GPU driver & HDD. In case of the latter, you may not necessarily notice your entire disk subsytem to be frozen, as long as there is ram the kernel will continue to service, system will appear fine - but upon any disk access, will freeze until the bus /flush/clears/reconnects.

#1. Check your OS drive for errors;  
#2. Check your three remaining drives, any one drive can freeze a system.  
#3. If any of those are dynamic .vhdx, and you have previously expanded any volumes on them.

2 cents

---

<div class="post-metadata">

**Author:** ![batelis](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/batelis/32/1136_2.png) [@batelis](https://forum.storj.io/u/batelis)\
**Post date:** [September 7, 2024, 6:29pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/6 "2024-09-07T18:29:10Z")

</div>

I did have few bluescreens before that might be it, and my taskbar doesnt work right now aswell…

---

<div class="post-metadata">

**Author:** ![Julio](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/julio/32/1132_2.png) [@Julio](https://forum.storj.io/u/Julio)\
**Post date:** [September 7, 2024, 6:31pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/7 "2024-09-07T18:31:21Z")

</div>

Blue screen, be nice to know the specific error. Nevertheless, DISM then SFC scan.

---

<div class="post-metadata">

**Author:** ![batelis](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/batelis/32/1136_2.png) [@batelis](https://forum.storj.io/u/batelis)\
**Post date:** [September 7, 2024, 6:32pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/8 "2024-09-07T18:32:34Z")

</div>

it was karnel power I think might be ram might be psu but I only got 1 bluescreen I think

---

<div class="post-metadata">

**Author:** ![Julio](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/julio/32/1132_2.png) [@Julio](https://forum.storj.io/u/Julio)\
**Post date:** [September 7, 2024, 6:32pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/9 "2024-09-07T18:32:35Z")

</div>

> [@batelis](#):
>
> my taskbar doesnt work right now

Task Manager then re-start explorer.exe tasks.

---

<div class="post-metadata">

**Author:** ![Julio](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/julio/32/1132_2.png) [@Julio](https://forum.storj.io/u/Julio)\
**Post date:** [September 7, 2024, 6:34pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/10 "2024-09-07T18:34:46Z")

</div>

Do DISM online repair first, then SFC /scannow after that, on your OS drive.

GL

---

<div class="post-metadata">

**Author:** ![batelis](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/batelis/32/1136_2.png) [@batelis](https://forum.storj.io/u/batelis)\
**Post date:** [September 7, 2024, 6:35pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/11 "2024-09-07T18:35:07Z")

</div>

ran chkdsk and ut didnt find anything

---

<div class="post-metadata">

**Author:** ![batelis](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/batelis/32/1136_2.png) [@batelis](https://forum.storj.io/u/batelis)\
**Post date:** [September 7, 2024, 6:38pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/12 "2024-09-07T18:38:43Z")

</div>

With scannow it did find corrupt files, lets see if that works. Thank you.

---

<div class="post-metadata">

**Author:** ![Julio](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/julio/32/1132_2.png) [@Julio](https://forum.storj.io/u/Julio)\
**Post date:** [September 7, 2024, 6:45pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/13 "2024-09-07T18:45:06Z")

</div>

If you don’t do DISM repair first, even though SFC finds errors it may not solve your problems. If you don’t go back and do DISM online repair, then SFC. The DISM repairs the repository that SFC uses to do it’s repairs.

But sounds good enough for now… 🙂

---

<div class="post-metadata">

**Author:** ![batelis](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/batelis/32/1136_2.png) [@batelis](https://forum.storj.io/u/batelis)\
**Post date:** [September 7, 2024, 6:59pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/14 "2024-09-07T18:59:25Z")

</div>

dism check didnt find any errors.

---

<div class="post-metadata">

**Author:** ![daki82](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/daki82/32/1128_2.png) [@daki82](https://forum.storj.io/u/daki82)\
**Post date:** [September 8, 2024, 8:24pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/15 "2024-09-08T20:24:59Z")

</div>

You may find more information here,

> [@Fatal Error on my Node / Timeout after 1 min](https://forum.storj.io/t/fatal-error-on-my-node-timeout-after-1-min/22052/1):
>
> Hello Everyone, a week ago i got several errors on my main node. Maybe someone can help me to solve the problem. The Node runs on Windows 10. Thanks for your Help. 2023-03-27T05:11:25.897+0200 INFO piecestore download started {Piece ID: 7WMQ364BMF672RLAQWDD76IQ6EORNXMYXIL5PGN46I4VUT4P4JXA, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET, Offset: 110336, Size: 469504, Remote Address: 184.104.224.98:48702} 2023-03-27T05:11:25.916+0200 ERROR services unexp…

my tip is higher timeouts and defragmentation of the mft.  
temporarily(?) loglevel to fatal

oh, and maybe update bios, network and mainboard driver.

---

<div class="post-metadata">

**Author:** ![JWvdV](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jwvdv/32/7814_2.png) [@JWvdV](https://forum.storj.io/u/JWvdV)\
**Post date:** [September 8, 2024, 9:28pm UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/16 "2024-09-08T21:28:50Z")

</div>

My steps would be:

1. CHKDSK
2. Make sure the drive had enough power / disable APM
3. Disable AAM or set to performant node, if the disk supports it
4. Change to badger cache.
5. Increase timeout to 5m0s or something

To be sure: are all nodes running on the same disk?

---

<div class="post-metadata">

**Author:** ![batelis](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/batelis/32/1136_2.png) [@batelis](https://forum.storj.io/u/batelis)\
**Post date:** [September 9, 2024, 6:08am UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/17 "2024-09-09T06:08:07Z")

</div>

You see its a diffrent problem here all nodes crash at the same time with same error. This had to be something else and in my case sfc /scannow helped and nodes didnt crash in 2 days.

---

<div class="post-metadata">

**Author:** ![batelis](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/batelis/32/1136_2.png) [@batelis](https://forum.storj.io/u/batelis)\
**Post date:** [September 9, 2024, 6:09am UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/18 "2024-09-09T06:09:45Z")

</div>

sfc /scannow helped while chkdsk didnt find anything wrong and I do have my pc on power saver. My cpu goes down to 1.5 ghz sometime that might be too slow.

---

<div class="post-metadata">

**Author:** ![JWvdV](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jwvdv/32/7814_2.png) [@JWvdV](https://forum.storj.io/u/JWvdV)\
**Post date:** [September 9, 2024, 6:25am UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/19 "2024-09-09T06:25:06Z")

</div>

> [@batelis](#):
>
> You see its a diffrent problem here all nodes crash at the same time with same error.

Yeah, so I think it might be underpower. Si hoe are they powered? How many disks/ hoe many Watt?  
And are all nodes running at different drives?

---

<div class="post-metadata">

**Author:** ![JWvdV](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jwvdv/32/7814_2.png) [@JWvdV](https://forum.storj.io/u/JWvdV)\
**Post date:** [September 9, 2024, 6:26am UTC](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791/20 "2024-09-09T06:26:04Z")

</div>

> [@batelis](#):
>
> My cpu goes down to 1.5 ghz sometime that might be too slow.

Don’t think so, have about 35 nodes on one CPU (N100) idling 20% of the time.

[Next page](https://forum.storj.io/t/all-nodes-crash-at-the-same-time/27791.md?page=2)
