Hello - i am looking to move a very big node - maybe about 10TB from one disk to another…
What is the best approach?
its installed with the windows gui and will continue to run on that same machine - i just need to upgrade the disks.
the machine cannot be powered off because then it will affect other nodes.
what is the best way to get the data on a new drive?
i have tried robocopy but stopped because it takes wayyy to long.
i have thought about a cloning software but it requires a restart of the machine and into some weird safe mode or something - so i dont know if i can run the node meanwhile…
Also cloning software doesn’t need a reboot if you’re not trying to do something like replace the boot drive. The trial version of something like Macrium Reflect should work fine.
If you’re using something like rsync then you only need to shut it down for your last sync. If you’re cloning the disk then yes you need to shut the node down.
(I think you can be offline something like 12 days before issues?)
It should be possible to use cloning software as long as the partition you try to clone is not the system partition. Though, I do not have practical experience with this kind of software under Windows, so I cannot give you any specific advise on this topic, sorry.
10TB cloning can take few hours, but still a lot faster than robocopy (1-2 weeks)
Unfortunately its safe to turn off the computer, some things You just cannot jump over.
You can try just disconnect the HDD, if hotswap is ON, but if You never tried before, You can easily cause some problems and need to turn off the computer anyway so, why complicate, just turn it off in first place.
Edit: i used this inateck docking station offline, just put 2 disks and press for 3 sec, and then 1 quick press to clone it. Paid like $20 for this, and im super happy. it just clones offline without need of any PC or software.
Clone will run almost at full serial write speed of your drive.
There may be some Windows programs that can do it while you are booted into Windows, personally I would put the old and new drive into a second machine and boot into clonezilla (from USB). The drive you want to clone can’t be mounted during the process, the affected node needs to be turned off either way.
Speed should be near 200mb/s, but even if you calculate with 150mb/s for safety reasons it should be done in 18.5 hours. Shouldn’t take more than one day for the whole process (e.g. not sitting in front of monitor waiting for it to finish or other delays)
The other way would be robocopy or rsync (does this work on Windows?). Will take weeks, but for the first few runs the node can be kept online. For the last run the node needs to be shutdown too.
I wouldn’t bet on that the last run of robocopy will be faster than cloning the entire disk in the first place.
Just clone it, turn off the PC if you must, no problem. You over thinking it too much. If you stay off line more than 4 hours, you loose some data; big deal… I was in vacation and I lost internet to 4 nodes for 3 days; after a day or two, they got back all the data lost.
Both works under Windows, robocopy is a native tool, it can work faster than rsync, the rsync can be run under WSL. Perhaps there is a Windows version of rsync, but why, if you have robocopy (unless you want to migrate to Linux)?
Nice. I have something similar but have never tried it. I use a small linux box and DD then resize the disk as its always a bigger disk. Keeps the same Blkid so sort of plug and play.
I usually start off by running rsync while the node is running. When it’s done, I turn off the storj node and rsync again to make sure I have an exact replica of how the filesystem was when the node was last online. At this point, I usually run async a third time to check that 0 bytes differ between the storage drives. Now I’m ready to start storj again with the new drive.
Every rsync run takes a considerably lower amount of time to finish than the previous ones since it’s copying less and less data in each step. With this strategy, I can ensure low downtime, while still ensuring all data is intact. I sometimes even throw in an extra rsync job before turning the storj node off if the first rsync job took a long time (several hours or even days), just to minimize downtime.
The last rsync should be with a --delete flag to remove excess files from the destination since the last sync. This is not only to save a little space, but also to remove outdated database journals, otherwise you may end up with a corrupted database.
You may also just reduce the allocation below the usage, this will allow to do not accept new data while you syncing data, and you would need less sync cycles.
