I have a task to migrate small (1Tb) node to another country.
As i know (please correct if I’m wrong), that there still no down time disqualification if node for more then 5 hours offline.
First idea was to turn of the node and ship the HDD by post. But in this case node will be offline for probably a week or so. Even if there are no disqualification, reputation probably will go down dramatically. To prevent that i decide, that probably i can make a copy of that 1TB drive. Leave the node here online and ship the copy drive by post. Once the drive will arrive to destination country, i can arrange rsync remotely. The problem that on the destination country node will be only on 100Mbps pipe. If the changes on data folder will be huge, it might take ages to synchronize.
right now with the traffic on the network i’m seeing is near zero… and lets say you needed to updated 500gb… the 12mb/s at 3600sec an hour so lets call it 36gb an hour… so in less than a day you can at 100mbit fill half the drive…
so i would go the remote rsync route… and if you want to give it a head start you can always send it off with a copy of the node on it, then its up an update which will be anywhere from a few hours to a day max at decent speed.
and i know my math is rough around the edges… but we are doing estimates here…
Of course if i send HDD, there will be a copy of the NODE (in rsync option) and i only rsync changes.
But as i had this experience in the topic bellow, where i counted up to 200Mb/s write to new HDD, but in reality i’ve got only 25MB/s (because of milions tiny files), i believe that 12mb/s is also just very theoretical math, but reality can be 2Mb/s or so
well either you got 100mbit bandwidth or you don’t
but yeah Iops can also be limitations, tho network vs hdd’s in iops networks takes that with ease… no moving parts
and like i said network is currently very quite… in the last week i’ve gotten like 200gb ingress
even at 2mb/s thats 7200mb an hour, so 144gb in 20 hr and +1/3
still less than a day if you get it shipped already because some day in the near future that will stop and we will most likely see a lot of ingress… atleast for a couple of weeks if the last 2 months are any indication…
seems we get 1 or 2 good ingress weeks and then some low traffic weeks only to repeat the pattern next month.
looks like some cache or buffer or maybe even io get overloaded and the system then has to wait for it to catch up… and then when it starts again it again overloads it and has to wait…
i recently had a similar issue when trying to migrate my own node… turned out to be blocksize related… recordsize or what… i think it’s called blocksize in windows…
i was running a mixture on my zfs pool… started out at 128k, then set it to 256k and then went back to 32k … basically testing it out a bit… then when i was migrating it, copying the 8-9 tb node took 5 days and i then copied it to 16k recordsize (blocksize) then to verify my data was okay i wanted to run a scrub on the node data… and that wouldn’t start even after working on it for like 3 hours…
so i figured it was the small recordsize of 32k i had been running the longest that caused the 5 day copy time, and the 16k that just made it much more horrible…
so i set my recordsize to 512k now… not sure if it really is better… but time will tell when i eventually decided to move it again…
ofc there are some issues to running larger recordsizes… such as some workloads can require a ton of RAM and other workloads might also take a very very long time to run due bandwidth usage amplification…
goes a bit like this… the system needs to 1kb from a 100kb record/block then it will read the 100kb use the 1kb and throw out the 99kb
in some cases this can be an issue… but usually this is mitigated by cache or ram which will keep the data in case it’s needed again to avoid repeated reads…
128k is a good choice for block/recordsize … afaik it is a nice middle point where one gets best avg across most hardware… you might not want to run something like an OS on 512k recordsizes… but should be great for data storage of larger files…
My use case is similar to yours, in terms of having minimum downtime.
Only difference being that both my servers are local (I just want to be able to run pacman -Syu and reboot).
As I run rync -lazzr --progress --delete-during /server1/path/ /server2/path/ because of the large number of files, each subfolder under blobs takes about 10min to be scanned. so the total time to finish rsync is about 40min at the moment for around 2TB.
setting up a mount using /server1/path/ and /server2/path/
initially as the storj is running on server1, /server2/path/ would be empty. then becase mergerfs would (hopefully) basically merge both folders and present that to storj docker container, I can move the files from server1 to server2, while the node is operating.
at the very end, i can stop the node, move the DB files as they would be active; restart the container on server2 with the same mount; confirm storj is running ok in server2, and then finally update and reboot server1
Let me know if you do try any of this first.
I’ll report back when I get to it
i always finish with shutting down the storagenode, running 1 final rsync to be sure everything is perfect… not sure how i feel about moving the db… i’m sure it works…i’m sure 99.99% is correct
but it could give me a few failed audits i would think, while the other method of running rsync after shutdown shouldn’t give any issues…
ofc running rsync yet another time may take yet another couple of hours… but well i had rsync running for 14-15 days last time… an extra hour doesn’t really bother me, and i would like to spare my node for any possible errors i can, so that it can survive if it runs into issues…
not saying that your method isn’t perfectly valid… i just don’t see how you can get a 100% accurate copy of the entire storagenode that way, but maybe thats just me not understanding the command parameters you use.
This is rsync speed over openvpn (tcp).
Source: 300Mbps
Destination: 100Mbps (other side of the Europe, ~3000km away)
Looks like to transfer 1Tb node it will take nearly week or so.
