I plan to use ddrescue to migrate 7TB of data to a new disk. Is there any way to do this without affecting my online score too much?
If your node started healthy it can be offline for more than 10 days before it even gets suspended: and it takes even longer to become disqualified. You should have lots of time to bring it back online.
I assume you’re using ddrescue because of damage: because if not you can migrate your 7TB with rsync and only be offline about 10 minutes.
Yes, as described in documentation and discussed on the forum countless times.
For advice beyond what’s in documentation you would need to provide way more details, at least the source and target filesystems to start with.
It will take 14 days to fully clone the disk. I am looking for ways to keep my node running during that time.
Currently, my drive keeps triggering hard link resets. Using data across 19 minutes from dmesg, only 18% of the time can be used for transfering data. The system also forced UDMA/33, which lowered the disk’s speed from 200MB/s to 33MB/s maximum. This is how I calculated it will take about 14 days to fully clone the disk.
Check if your HDD is not simply overheating, that it has a good SATA cable, that your SATA controller is not overheating, and that your power supply is good. All might result in this kind of behavior.
And if it is triggering these hard link resets so often, you’re already past non-violent options. Take the node offline (because you’re making it worse by having multiple workloads at the same time!), then do ddrescue, and hope that you’ll be able to recover enough data that in 30 days past restart you recover your online score.
This is not normal. 7TB at 250MB/s shall take about 8 hours. I would investigate why does not cloning go at full speed.
On the other hand, you don’t need to clone. You can incrementally send filesystem or, in the worst case, files, and only shut down node for the last sync that shall not take more than a minute.
Yeah, this needs to be dealt with first. Check controllers, cables, all the usual stuff, including thermal issues mentioned above. Likely cables.
I have tried all the usual stuff:
replacing sata cables
cleaning sata connectors
using other sata ports on the motherboard
The temperature is stable at 40C, it shouldnt be a problem.
But I have discovered another thing: When Sequentially reading the disk, the average speed is actually 200KB/s (including waiting for link reset)
I have another disk on the same device and it is perfectly fine. So, the PSU and motherboard should have no problem
I had a case where a drive connected to the first SATA power connector on a PSU ribbon cable worked, but not on the second one. Checked with a multimeter, turned out the PSU was already failing to provide enough power on the first connector, but HDD was somehow dealing with it. But the drop of voltage past the first connector/first HDD was already too much 
Maybe…? But the two drives of mine are the same model. I assume that if the PSU causes voltage drops, both will be affected. I’ve also tried swapping power cables but it had no effect.
You could also potentially use something like overlayfs to redirect writes to the new drive (the upper layer) while also cloning the data (the lower layer) to the new drive. This will keep the node online but all write activity will effectively go to the new drive.
I believe that the original drive’s condition will get worse by time. Months ago it didn’t reset link that often. I highly doubt whether it can stay in the current condition for over 2 months…
The one further away will be affected more.
When some unexplained nonsense is happening PSU is always the first suspect.
But, the one having issues now is the closest from the PSU (even for only a few cm)
The other disk had no error at all at all times, while this one triggers hard link reset even when not mounted. Are these enough to prove that the issue is caused by the disk itself? (I’d be happy to test out, but I don’t have other PSUs)
EDIT: Fixed by switching power cables. The first switch had no effect because only the connector (on the same cable) was switched. Now both disks share the same healthy cable and the issue is fixed.
Nice. On a tangent, I don’t know where to buy sata/sas/power cables these days. Everything is cheap chinese horseshit, overpriced same cheap Chinese horseshit, or left kidney level costing server gear.
Same goes for power supplies. Unreal.
My PSU is Apevia SFX-AP500W, so i cant change its cables. If the currently healthy one (also the last one) also fails, I will need to change the entire PSU…
Found PSU Tier List rev. 17.0g - Cultists Network to be a quite interesting benchmark, BTW…
My idea is using “RSync” with archive and deletion flags to catch all changes between sync steps, if it will be possible. So:
Step 1. RSync all data to the spare drive, Node is still running, no downtime, this will take long time
Step 2. RSync all data to the spare drive again, Node is still running, no downtime, this will take short or little time, depends on changes on the live system
Step 3. RSync all data to the spare drive again, but with Node down, now you will have all the data on both drivers 1:1 with all changes
Step 4. Replace damaged drive with spare drive and you are up again
This will not preventing downtime! But reducing the downtime.
some offline time wont hurt.
offline cloning is fast (often under 1h) and safest
i love the hardware cloning station tho like Unitek Cloning Station
What do you mean? This is literally what documentation describes. How do I migrate my node to a new device? - Storj Docs as a default approach.
They are still comparing different flavors of crap. Which crap is more flavorful is immaterial.
They are comparing consumer devices by some metrics like ripple and voltage and components. This says nothing about long term stability, or transient suppression or decoupling or anything else that matters.
Consumer devices are designed for price point. All the money is spent on marketing, stupid modular connectors (yes, we need more failure points), RGB lighting, and impressive box with unhinged specs, all while also figuring out where can they cut 30 cents more without getting caught – is money not spent on reliability and simplicity.
Those that are half-decent – are ridiculously overpriced. They have to be, because if crappy devices are overpriced – half-decent ones have to be ridiculously overpriced.
I solved this problem by buying enterprises power supplies in brown bags, with no outrageous claims about gold plated capacitors, from vendors that don’t need to impress anyone, don’t have to compete with Chinese alphabet soup brands, and which will work for decades.
Power delivery is not the area where any compromise is acceptable.