Hi, I’m running storj on a 18TB Toshiba HDD in Linux. I’ve scheduled a monthly SMART long test of all disks in the system, which is generally a good idea™.
Anyway, the disk in question is very busy doing storj, so the test takes very-very-very long. Normally, if being idle, it should take 25.5 hours (it is reported by smartctl as 1531 minutes). But with storj it takes 15 days! Today is 6th of January and the smartctl reports: Self-test routine in progress… 60% of test remaining. Test started on 1st January.
Is there a way to temporarily slow down the storj disk activity to allow the SMART test to be finished sooner? The SMART test running so long also decreases the disk performance, which is not good for storj.
Is the node maybe still running used-space-filewalker: and that’s what is competing with smartctl? (ps -ef | grep used-space-filewalker)
If so… then that’s a housekeeping task that runs at node startup by default: so it will complete eventually… and as long as the node isn’t restarted it won’t clobber smartctl again.
But remember that automatic node upgrades will restart things periodically… so…
There is no good reason to run long smart tests. Run short tests, they will update Smart Control parameters, and report detected bad sectors encountered during normal operations, from sectors accessed organically. There is no benefit in force-checking entire disk; results are non-actionable anyway — both in a single disk scenarios and large redundant array. Even less so for storj that can tolerate a lot of data loss by design.
Separately, storj produces minuscule IO today; if your disk is overwhelmed it’s likely because you don’t have enough ram available to fit hot metadata. You may want to consider adding ram or some sort of tiered SSD storage setup to facilitate fast path to metadata.
I drive 10km to work a couple times per week, at no more than 50km/h. And sometimes take the car to get groceries on weekends. Very gentle use.
But once per month I still hold the pedal to the floor and let it bounce off the rev-limiter for an hour. You know, to test the engine, which is generally a good idea™. My friends only depend on the check-engine light: but I like to be thorough.
It keeps getting low on coolant, and make a knocking noise, and the oil looks like milk. Can anyone help?
Well, the short-test is useless in my opinion. It does the same as the long-test, but only at the beginning and at the end of the disk. It’s not updating smart control parameters, whatever it means.
The unreadable sectors are recorded automatically without need to run any test. But only if something is trying to read that sector (and fails). In my case there are storj files on the disk (I don’t care), but also my backup files (I care a lot), which are not read periodically.
So how am I supposed to know early that the disk is failing? From my own experience the periodic long tests have alerted me about failing HDD many times already.
I’m not sure about minuscule IO. Just tried to do iotop and reading is very low, but writing shows almost permanent activity around 3MB/s. I don’t think this is metadata problem, is it? Computer has 16GB RAM. What disk activity do you see? (that’s iotop from package iotopc, a better alternative to the old iotop in python)
Funny, except not exactly my case. Not sure how reading the whole disk compares to flooring the gas pedal . I run monthly long self-tests on all disk I have. It was very useful in the past, because I could tell that otherwise problem-free disk is going to fail soon. I have quite a lot of old disks which do not seem to be damaged by the monthly long testing.
The problem is just with exactly one disk with storj on it. Just that one disk is taking approximately 15x the time needed for the self-test. I don’t see that in your funny story though
Offline for 30 days without disqualification? That’s 30 days in what period? I’d need 1 day in a month offline just for the testing.
Maybe it is better to temporarily decrease allocated space as you suggested. As I saw with iotop, it looks like storj is heavily writing at the moment. Not reading that much.
There’s a couple questions in there, and you’ll probably get a lot of opinions. But for Storj… to the network… any particular disk is disposable. So I think many listen to the general hints they get from instantaneous smart data (since it’s ‘free’)… but otherwise run-to-failure. Then recover what they can, or start a new node.
Like if smart tells me a Storj disk is unhealthy… so what? It may keep running for years… so let it run. Storj is a great place for disks with potential issues to live out their dying days
For important data, you use mirrored/parity configs to keep the data available, and automated backups to make it recoverable. But in the end you do the same thing: run the HDD to failure. Because you can survive failure. (Storj isn’t important data: but other files on the same drive may be)
I still let things like zfs scrubs run once per month though: that catches data-rot (and may indirectly detect hw issues too).
Why running smart test on StorJ disk? Why give the disk more work than needed? Won’t it shorten the disk lifespan? And if the disk gets errors, a short test or even the system will tell you.
I have a storage node running on a bad disk and I see sometimes “bad I/o” errors on my login screen.
Edit: The bad disk is running for almost a year now without any problem. (Maybe some degraded performance, but on the StorJ dashboard all looks fine). So even if it shows errors, the disk might survive way longer than anticipated
Yes, Hard drives are stronger then some people think. My unhealthy disk is even making funny clicking noises, but it keeps running and won’t die. I am scared it might even survive longer than some of my other disk that are fine at the moment.
Funny if youre in bed and hearing every 5-10 minutes a clickclick sound. It’s like a heartbeat
I’d say it has no negative influence on the lifespan of the disks. Short test isn’t telling you anything, except if you have bad sectors in the very beginning or the very end of the disk.
I see. Good to know. I think the zfs scrub is just another user disk read, whereas the smart test reads the disk only when the disk is idle. And that’s not happening with storj.
If it is only running when the disk is otherwise idle… then even if it takes a week (or longer)… that would be OK? You’re not sitting around waiting for the results.
Well, it is like that for a year or so. Self tests takes two weeks. Then after another two weeks it starts new tests (new month).
What is a bit annoying is that I e.g. can’t restart the computer when the self-test is running, as the test will be stopped and will show as Interrupted in the SMART log.
I’m also aware that the disk performance is a bit lower during the self-test, which now is 15 days of a month. That contradicts a bit what I said earlier about the self-test running only if the disk is completely idle. But it seems both are right.
You should not assume this is possible. For drives in the last 20-30 years there were never dependable indicators of future HDD failures. The best statements I’ve seen in literature, and I’ve spent quite a bit of time reading papers about this topic, is that you can maybe have a predictor with sensitivity of ~75%, meaning you’d still fail to predict ¼ of failures—this at a non-zero specificity, meaning you’d throw away drives that are fine.
Besides, regular SMART parameters are simple to collect and tend to give most information used in these papers through parameters like reallocated sector count.
It got even worse recently with manufacturers explicitly stating drives are rated for a specific amount of data transfer, also consumed by scrubs and long tests, which make them even less practical.
And, as others say, Storj as a system doesn’t care. Four years ago I added a HDD with a number of bad blocks to my Storj setup. It still works.
Why would you use your backup drive for Storj? If you care about your data, don’t put Storj on it.
I can see the huge wave of criticism coming to my way, but I don’t care.
I don’t even use my personal NAS for Storj. I use dedicated systems and NASes, with dedicated drives for Storj.
I won’t try to convince you my way is better, or safer or whatever, but ever since I started runing Storj nodes 4 years ago, I didn’t mixed nodes with personal data.
Regarding drive long test, I see it’s usefulness only when buying the drive, for a first time complet check up. After that, I wouldn’t stress runing it again ever.
Keeping nodes offline for long periods of time, even if you don’t get DQed, you loose a lot of data, starting after 4h of offline. The pieces that are requested by clients, if are not received from your node, they will be repaired on other nodes, and when you come back online and receive the bloom filter, those pieces are removed from your node.
To reduce activity on the drive, if you still want to do the test, reduce the allocated space to a minimum and stop the startup file walker, or enable badger cache, set noatime and stop indexing on drive, and reduce the log level. Or move the log to another drive. Regarding databases, they are not so IO intensive, so you can ignore them.
. I run monthly long self-tests on all disk I have. It was very useful in the past, because I could tell that otherwise problem-free disk is going to fail soon. I have quite a lot of old disks which do not seem to be damaged by the monthly long testing.
