Unfortunately after about 26 hours, the command failed with the following errors in the log:
tar: -: Wrote only 4096 of 10240 bytes
tar: Error is not recoverable: exiting now
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
sh: line 1: /usr/bin/xargs: Argument list too long
Sometimes it would throw such kind of errors, mostly when 2 processes attempt to access the same file.
If you can just repeat it or continue with different folder.
Wat will happen with the existing files in the destination? Will it ignore those and just continue with the files it hasn’t copied yet?
It will ignore them.
After multiple attempts it kept on failing, if any copies were running at the same time, they would also fail at the same time, all with the above error or this one:
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
tar: -: Cannot write: Broken pipe
tar: Error is not recoverable: exiting now
xargs: cannot fit single argument within argument list size limit
Because I don’t have the time to debug further at the moment, i’ve resumed the sync using rsync for now
This is why I doubt that it could work. Just use a usual rsync without using of xargs.
That is sad to hear. I have not been able to fully understand such a break. It sounds like either the reading file is not readable or accessible. Maybe even the directory empty and then there is nothing that gets passed to xargs. However empty folders shoudl not be a problem as the directories are supposed to be submitted as well. So I dont’t really know what it seems that xargs is complaining a bout here.
When I had this error it never stopped at the same folder so at the end everything got copied.
Normally I would agree, but the strange thing is that I had 2 or 3 copies running at the same time (of different directories ofcourse) and they all failed at exactly the same time. This happened multiple times and never during the same file copy. It has to be something with either memory, or the operating system.
Unfortunately the directories are to large to keep experimenting with, as I would like to be done on time. So I’ll have to settle for the slower rsync for now 
Maybe it needs a fs check.
The fs checks are clean, like I expected. I think it was unlikely they all had issues that would pop up at exactly the same time. I’ve resumed running the rsyncs, the first node is almost ready.
For the next node I might try the xargs method without nohup. And just run it directly in screen.
Either that or you simple tail -f the nohup file.
What’s really missing is what the error is. It would be nice if it would not exit then but simply go to the next file and process it.
I think the command has issues when the file is not available, accessible, readable and the tar therfore remains empty. But that is only a guess.
The error messages are directly from the nohup file, there is nothing else above that except for the progress list:
./2c/oabrhrwce66x3wrjb6im7s5kg45xi524ktmvacrd23562rvsla.sj1
./2c/oacalcmliw6l6y4qtquszoaa5evdv7musbmtpx666q5sdh3dia.sj1
./2c/ob2clf7uch7jgxaxqmfzhx6nwfsylqconhkqlw6unuvc7g2dxq.sj1
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
tar: -: Cannot write: Broken pipe
tar: Error is not recoverable: exiting now
xargs: cannot fit single argument within argument list size limit
xargs is usually eaten all log output or make it a not usable (you cannot determine one thread from another).
I would suggest to use a simple plain rsync without trying a parallelism this kind. You likely wasted much more time to try this suggested approach already.
The thing is, Rsync is a pain in the ***.
While it was working on one single satellite folder, I was able to successfully move 1 complete different satellite folder plus all trash folders to the new destination. All while Rsync was chewing slowly on one foldert.
Use the LVM’s pvmove or zfs send and zfs receive. Or dd.
See the example for LVM:
ZFS makes this super easy with send / receive. On truenas the replication ui makes it super easy to do this.
If you have to move a node without that and don’t want to use some strange sector copy thing, then rclone is by far the fastest way I found.
You can do “rclone sync source destination --progress” and it does its thing with the node running. Gives a nice indication of how much it has copied.
Once the first pass completes just stop the node and run the command again to mop up anything that was missed.
Other useful command is “rclone purge folder” to remove node data if you have other stuff on the disk and don’t want to nuke it.
old topic, but I concur that rclone seems a bit faster than rsync. It seems to run more parallel processes, which helps a bit. Still very slow (days/weeks) to migrate a large node of course.
You are correct, rclone sync is working faster because it uses 4 parallel transfers by default, it also can copy several chunks of each file in parallel.
And the sync command of rclone works like rsync --delete by default.