# Facing dying hdd **URL:** \ **Category:** Node Operators\ **Tags:** hardware\ **Created:** [December 29, 2023, 3:28pm UTC](https://forum.storj.io/t/facing-dying-hdd/24792 "2023-12-29T15:28:39Z")\ **Posts on this page:** 9\ **Page:** 2
**Author:** ![node1](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/node1/32/6289_2.png) [@node1](https://forum.storj.io/u/node1)\ **Post date:** [December 31, 2023, 6:25pm UTC](https://forum.storj.io/t/facing-dying-hdd/24792/21 "2023-12-31T18:25:26Z")
Now i see even more interesting error: "Got a signal from the OS: “terminated” Any idea what is going on? (currently drive is connected to another machine, than before). 2023-12-31 17:52:12,883 WARN received SIGTERM indicating exit request 2023-12-31 17:52:12,890 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die 2023-12-31T17:52:12Z INFO Got a signal from the OS: “terminated” {“Process”: “storagenode-updater”} 2023-12-31 17:52:12,897 INFO stopped: storagenode-updater (exit status 0) 2023-12-31T17:52:12Z INFO Got a signal from the OS: “terminated” {“process”: “storagenode”} 2023-12-31T17:52:12Z INFO piecestore downloaded {“process”: “storagenode”, “Piece ID”: “IGMBY5JYUVXKVPIBRGVZ5XM27LCB732OLEAUN63A7BTWBH5SU74A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 290048, “Remote Address”: “5.161.64.186:44440”} 2023-12-31T17:52:12Z INFO piecestore downloaded {“process”: “storagenode”, “Piece ID”: “TRRDJZ7UB3CPJUG2TGCTMOFME77755IYR2SALDR5VC6VSXWJ4TOA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “Offset”: 2061824, “Size”: 4864, “Remote Address”: “158.180.29.230:47726”} 2023-12-31T17:52:12Z INFO piecestore upload canceled {“process”: “storagenode”, “Piece ID”: “27ZOTVSKG4L2IAX7U6RQWIWIFZFXWY5MIHGVEHXLXZNBT233CNKA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 327680, “Remote Address”: “5.161.128.79:23270”} 2023-12-31T17:52:12Z INFO piecestore upload canceled {“process”: “storagenode”, “Piece ID”: “QTHGYC2LJVCE5H2ON2G4PDRPILI3P26FJDOTKSXLOWW47473OSEA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “79.127.205.225:51272”} 2023-12-31 17:52:13,431 INFO stopped: storagenode (exit status 0) 2023-12-31 17:52:13,434 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM) ---
**Author:** ![node1](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/node1/32/6289_2.png) [@node1](https://forum.storj.io/u/node1)\ **Post date:** [December 31, 2023, 6:25pm UTC](https://forum.storj.io/t/facing-dying-hdd/24792/22 "2023-12-31T18:25:58Z")
Currently i only have ubuntu and mac os 🙂 But thank you for your suggestion 🙂 ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [January 1, 2024, 3:20am UTC](https://forum.storj.io/t/facing-dying-hdd/24792/23 "2024-01-01T03:20:20Z")
You need to check logs before. It either a new version installed by storagenode-updater or a FATAL error. ---
**Author:** ![node1](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/node1/32/6289_2.png) [@node1](https://forum.storj.io/u/node1)\ **Post date:** [January 1, 2024, 9:52am UTC](https://forum.storj.io/t/facing-dying-hdd/24792/24 "2024-01-01T09:52:31Z")
Alexey could you give me more information on what exact log to see? In the previous message i showed what is visible in the log (storagenode container log), but maybe you are talking about different log? ---
**Author:** ![JWvdV](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jwvdv/32/7814_2.png) [@JWvdV](https://forum.storj.io/u/JWvdV)\ **Post date:** [January 1, 2024, 11:34am UTC](https://forum.storj.io/t/facing-dying-hdd/24792/25 "2024-01-01T11:34:32Z")
I’m on Debian, which is running fine. I once did a trial with Ubuntu; which gave me a lot of trouble over and over, with also dieing nodes and turning mount points in read only mode and so on. Since I couldn’t get grip on the root cause but knew it worked fine on Debian, I reverted back to Debian and never left that path since ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [January 1, 2024, 12:34pm UTC](https://forum.storj.io/t/facing-dying-hdd/24792/26 "2024-01-01T12:34:38Z")
You can search for FATAL errors in the logs ```auto docker logs storagenode 2>&1 | grep FATAL ``` ---
**Author:** ![node1](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/node1/32/6289_2.png) [@node1](https://forum.storj.io/u/node1)\ **Post date:** [January 1, 2024, 12:43pm UTC](https://forum.storj.io/t/facing-dying-hdd/24792/27 "2024-01-01T12:43:58Z")
I run nodes on ubuntu for many years. Never had any problem other than hdd failure. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [September 11, 2024, 4:41am UTC](https://forum.storj.io/t/facing-dying-hdd/24792/29 "2024-09-11T04:41:36Z")
All these backups are useless for storagenode. As soon as you restore it from the backup and bring online, it will be disqualified for lost data since the last backup, it will be quick. It’s possible to make rsync more often and there is a small chance, that the missing data would be less than 4%, but I doubt. If you have doubtful HDDs, then perhaps it would be better to either run a normal RAID with autocorrection, like raidz or mirror (zfs too), or run several nodes, each on own HDD. Due to a node selection algorithm these nodes would be treated as a one big node and the ingress traffic for your subnet would be distributed between them. Thus if one disk die, you would lose only this one piece of the common data, not all as in case of mergefs or other RAID0 solutions with zero fault tolerance. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [September 11, 2024, 6:51am UTC](https://forum.storj.io/t/facing-dying-hdd/24792/31 "2024-09-11T06:51:33Z")
From their manual > [@](#): > > The main one is that if a disk fails, and you haven’t recently synced, you may be unable to do a complete recover. More specifically, you may be unable to recover up to the size of the changed or deleted files from the last sync operation. This happens even if the files changed or deleted are not in the failed disk. This is why SnapRAID is better suited for data that rarely change. It would not help to recover data which were changed since the last sync. The same as for rsync or any other backup programs. Or you need to do a sync as frequently as possible. There are other limitations, including the long running sync while files are still modifying, so not so useful for storagenodes’ as you may think. But I would be glad to hear how did you manage it and how did you fix the failure of the one disk, and would the node(s) survive after that. Also, it looks like you’re losing disk space for parity anyway, just like with regular RAID with parity. [Previous page](https://forum.storj.io/t/facing-dying-hdd/24792.md?page=1)