Migrate node from Pi3 to Synology / rsync problem

Hey guys,

First off topic - I miss the chat! Am I the only one?

Second - I finally have my Synology DS1019+ and want to migrate from my Raspberry Pi3… So my idea was (and I remembered it from chat) to make sure Docker is running on the Synology (done), I also enabled SSH and created a user who is part of admin group so I can login.
Created a specific folder etc.

Now: I then looked in the old chat for rsync and tried a bit with it to copy the files while the node is still running. (As in doing it once to get most stuff, then again to get the latest ones. Then stop node and doing it once more to have it all). Then restart the node on the Synology.

So here’s my problem, I tried it from both sides (pi to synology and vice verca) but somehow I don’t get it correct with the rights or so:

sudo rsync -rtuvP username@192.168.x.x:/home/pi/DROBO /volume1/Storj/
pi@192.168.x.x's password: 
receiving incremental file list
rsync: opendir "/home/pi/DROBO/storage" failed: Permission denied (13)
rsync: send_files failed to open "/home/pi/DROBO/kademlia": Permission denied (13)
rsync: send_files failed to open "/home/pi/DROBO/revocations.db": Permission denied (13)

So I do understand that as on the pi itself to access the files I need to ‘sudo ls’ so to speak.

Now if I go to the pi3 and do it from there:

sudo rsync -rtuvP home/pi/DROBO username@192.168.x.x:/volume1/Storj/
username@192.168.x.x's password: 
Permission denied, please try again.
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.2]

As I’m also a N00b I always run into these wonderful problems. My interpretation for case 1 is: Right issue hence rsync can’t read it.
For the 2 case it might be that I either have an right issue on the Synology or something with rsync doesnt work?

Anyone any ideas?

Thanks!

1 Like

sudo rsync -rtuvP /home/pi/DROBO username@192.168.x.x:/volume1/Storj/

/ was missing maybee?

Hah good catch, copied the one where I had it wrong but doesnt change:

sudo rsync -rtuvP /home/pi/DROBO username@192.168.x.x:/volume1/Storj/
username@192.168.x.x's password: 
Permission denied, please try again.
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.2]

So if i ssh into the synology I can create a folder etc. so should work?!

I think you may need to force it to use ssh instead of rsh for synology.

Try:

sudo rsync -rtuvP -e ssh /home/pi/DROBO username@192.168.x.x:/volume1/Storj/

Or if you use a non-standard ssh port (example 2222)

sudo rsync -rtuvP -e "ssh -p2222" /home/pi/DROBO username@192.168.x.x:/volume1/Storj/
1 Like

The problem could be the permissions. Docker creates the files as root so you might have problems copying them as username.
You should do this first: sudo chown -R username 192.168.x.x:/volume1/Storj/

Tried that but doesnt look like that was the issue. Same error message. I stop removing username and IP, doesnt matter anyhow :slight_smile:

sudo rsync -rtuvP -e ssh /home/pi/DROBO tankmann@192.168.2.119:/volume1/Storj/
tankmann@192.168.2.119's password: 
Permission denied, please try again.
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.2]

So on the PI where the data is permissions are root, which could be the reason why I cant get it from the other side.

sudo ls -lah /home/pi/DROBO
total 796K
drwxrwxrwx 3 root root 4.0K Sep 16 22:37 .
drwxr-xr-x 7 pi pi 4.0K Sep 6 17:21 …
-rwxrw-rw- 1 root root 567K Apr 11 22:03 .VolumeIcon.icns
-rw-r–r-- 1 root root 750 Sep 16 22:37 config.yaml
-rw-r–r-- 1 root root 750 Sep 3 15:56 config.yaml~
-rw------- 1 root root 256K Sep 21 11:09 kademlia
-rw------- 1 root root 32K Sep 19 16:37 revocations.db
drwx------ 7 root root 4.0K Sep 19 16:37 storage

But on the synology it looks like this:

tankmann@Gynology:/volume1$ ls -lah
total 4.0K
drwxr-xr-x   1 root     root   164 Sep 19 22:08 .
drwxr-xr-x  23 root     root  4.0K Sep 14 14:08 ..
drwxrwxrwx+  1 root     root    12 Sep 19 22:08 docker
drwx--x--x   1 root     root   108 Sep 19 22:08 @docker
drwxrwxrwx+  1 tankmann root    22 Sep 21 12:05 Storj

Storj is the folder I want to copy into.
Don’t know why but I’m lost somehow.

Sorry for the obvious question, but tankmann is an admin user and can log in over ssh right?

Yes, I can login with ssh and it’s admin.

Haven’t used rsync on synology myself, but I have a feeling this page is going to be useful.
https://www.synology.com/en-global/knowledgebase/DSM/help/DSM/AdminCenter/file_rsync

Thanks - I had it enabled already but this might do the trick now, I didnt check the box and added an account (I though it takes the ones already created). Let’s see…

It looks like something is happening now:

sudo rsync -rtuvP -e ssh /home/pi/DROBO tankmann@192.168.2.119:/volume1/Storj/
tankmann@192.168.2.119's password: 
Could not chdir to home directory /var/services/homes/tankmann: No such file or directory
sending incremental file list
DROBO/kademlia
        262,144 100%   24.31MB/s    0:00:00 (xfr#1, ir-chk=1038/1043)
DROBO/revocations.db
rsync: recv_generator: failed to stat "/volume1/Storj/DROBO/storage/info.db": Permission denied (13)
rsync: recv_generator: failed to stat "/volume1/Storj/DROBO/storage/info.db-shm": Permission denied (13)
rsync: recv_generator: failed to stat "/volume1/Storj/DROBO/storage/info.db-wal": Permission denied (13)
rsync: recv_generator: failed to stat "/volume1/Storj/DROBO/storage/piecestore.db": Permission denied (13)
         32,768 100%  410.26kB/s    0:00:00 (xfr#2, ir-chk=1037/1043)
DROBO/storage/blob/
rsync: recv_generator: mkdir "/volume1/Storj/DROBO/storage/blob" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
DROBO/storage/blobs/
rsync: recv_generator: mkdir "/volume1/Storj/DROBO/storage/blobs" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: mkstemp "/volume1/Storj/DROBO/.kademlia.o6RNBX" failed: Permission denied (13)
rsync: mkstemp "/volume1/Storj/DROBO/.revocations.db.0uwlv5" failed: Permission denied (13)

Hm it didn’t do a lot:

sudo rsync -rtuvP -e ssh /home/pi/DROBO tankmann@192.168.2.119:/volume1/Storj/
tankmann@192.168.2.119's password: 
Could not chdir to home directory /var/services/homes/tankmann: No such file or directory
sending incremental file list
DROBO/kademlia
        262,144 100%   24.31MB/s    0:00:00 (xfr#1, ir-chk=1038/1043)
DROBO/revocations.db
rsync: recv_generator: failed to stat "/volume1/Storj/DROBO/storage/info.db": Permission denied (13)
rsync: recv_generator: failed to stat "/volume1/Storj/DROBO/storage/info.db-shm": Permission denied (13)
rsync: recv_generator: failed to stat "/volume1/Storj/DROBO/storage/info.db-wal": Permission denied (13)
rsync: recv_generator: failed to stat "/volume1/Storj/DROBO/storage/piecestore.db": Permission denied (13)
         32,768 100%  410.26kB/s    0:00:00 (xfr#2, ir-chk=1037/1043)
DROBO/storage/blob/
rsync: recv_generator: mkdir "/volume1/Storj/DROBO/storage/blob" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
DROBO/storage/blobs/
rsync: recv_generator: mkdir "/volume1/Storj/DROBO/storage/blobs" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: mkstemp "/volume1/Storj/DROBO/.kademlia.o6RNBX" failed: Permission denied (13)
rsync: mkstemp "/volume1/Storj/DROBO/.revocations.db.0uwlv5" failed: Permission denied (13)
DROBO/storage/garbage/
rsync: recv_generator: mkdir "/volume1/Storj/DROBO/storage/garbage" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
DROBO/storage/temp/
rsync: recv_generator: mkdir "/volume1/Storj/DROBO/storage/temp" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
DROBO/storage/trash/
rsync: recv_generator: mkdir "/volume1/Storj/DROBO/storage/trash" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***

sent 10,751,121 bytes  received 14,192 bytes  57,722.86 bytes/sec
total size is 380,839,798,896  speedup is 35,376.57
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]

Is it me or does it now looks like an issue on the synology side / where to copy… hmmm…

Try without the -e ssh, the docs seemed to suggest the rsync accounts were for rsh.

I don’t know whether it’s related to your problem, but you need the trailing / on the source AND destination. Otherwise you copy the folder DROBO into the destination and not only the content of that folder.

Furthermore you should also use the --delete option to delete the files from the destination which don’t exist in the source anymore.

Ah I think you’re right, the / was missing. I also added the --delete command but then it showed:

sudo rsync -rtuvP --delete /home/pi/DROBO/ tankmann@192.168.2.119:/volume1/Storj/
tankmann@192.168.2.119's password: 
Could not chdir to home directory /var/services/homes/tankmann: No such file or directory
sending incremental file list
./
rsync: opendir "/volume1/Storj/DROBO/storage" failed: Permission denied (13)
rsync: delete_file: rmdir(DROBO/storage) failed: Permission denied (13)
rsync: delete_file: unlink(DROBO/config.yaml~) failed: Permission denied (13)
rsync: delete_file: unlink(DROBO/config.yaml) failed: Permission denied (13)
rsync: delete_file: unlink(DROBO/.VolumeIcon.icns) failed: Permission denied (13)
cannot delete non-empty directory: DROBO
deleting @eaDir/@tmp/
deleting @eaDir/
.VolumeIcon.icns
        579,844 100%   13.38MB/s    0:00:00 (xfr#1, ir-chk=1041/1043)
config.yaml
            750 100%   12.85kB/s    0:00:00 (xfr#2, ir-chk=1040/1043)
config.yaml~
            750 100%   12.41kB/s    0:00:00 (xfr#3, ir-chk=1039/1043)
kademlia
        262,144 100%    3.29MB/s    0:00:00 (xfr#4, ir-chk=1038/1043)
revocations.db
         32,768 100%  410.26kB/s    0:00:00 (xfr#5, ir-chk=1037/1043)
storage/
IO error encountered -- skipping file deletion
storage/info.db
    181,764,096   7%   15.82MB/s    0:02:10  ^C
rsync error: unexplained error (code 130) at rsync.c(644) [sender=3.1.2]

I cancelled it … and then changed command to --delete-after

At the moment it’s looking better:

sudo rsync -rtuvP --delete-after /home/pi/DROBO/ tankmann@192.168.2.119:/volume1/Storj/
tankmann@192.168.2.119's password: 
Could not chdir to home directory /var/services/homes/tankmann: No such file or directory
building file list ... 
169393 files to consider
kademlia
        262,144 100%   41.54MB/s    0:00:00 (xfr#1, to-chk=169388/169393)
storage/
storage/info.db
    336,223,344  14%    4.35MB/s    0:07:20  

Not sure why the info.db copies so slow but at least I’m copying stuff.
Let’s see when files should be deleted later in case they’re ot at the source any longer - I still feel there’s a permission problem somehow.

Thanks for the great help so far!

Oh and sure… now I didn’t mount the drobo directly with gbit, it’s now going through the Pi3 :slight_smile: So 100mbit/s.

YAY. Just was able to move the Node to the Synology… it’s running fine as well, only problem now is, it has been paused :frowning: one or two days ago I assume from the logs on the pi3

1 Like

Just opened a ticket for it as well. FYI

So what was the problem in the end? Might be helpful for other people in the future.