Trying to migrate node from a server to a Raspberry Pi 4 8gb, struggling with rsync

Node Operators
1

I’ve been running a node for approximately 1 year and it is now full at 4.5Tb. It’s running on a 20c/40t rackmount server that is entirely unnecessary, so I’m trying to move it to my 6Tb disk attached to my Rpi. I followed the installation directions exactly for the installation on the server and the pi.

I have gotten to the point where I can SSH into either machine from the other, and at the point in the installation-for-rpi directions where I would create an identity. Rsync is installed on both machines. I think my problem is mostly my inexperience with Linux in general, but when I SSH into either machine from the other and run the command ‘rsync -aP /mnt/storj/identity/storagenode/ /mnt/storj2/storagenode-new/identity/’, I get

rsync: change+dir “/mnt/storj/identity” failed: No such file or directory (2)
rsync: mkdir “/mnt/storj2/storagenode-new/identity” failed: No such files or directory (2)
rsync error: error in file IO (code 11) at main.c(681) [Receiver=3.1.3]

I’m hoping someone can explain what’s going on, as I’m not sure why the files wouldn’t be there since I didn’t make any changes during installation. Am I missing part of the command that I’m supposed to fill in myself? I did some searching here and it seems like nobody is using the command specified in the documentation, and I don’t know enough to know what will work for me.

2

You don’t first connect via ssh and then rsync. After connection, you are on the other machine. It doesn’t know about filesystem of the one that you are connecting from. What you need instead is to specify a “network path” like this:
rsync -aP ~/SourceDirectory/* username@192.168.1.XXX:~/Destination

Take a look here (just a first link from google) for caveats and options:

You do know that since you already have a node, you shouldn’t create new identity, right?

1 Like
3

Please execute the command

df -T --si

on each of the host and post result here.

Basically if your data and identity are in /mnt/storj/storagenode/ on the server, and you do want to copy data to RaspberryPI to /mnt/storj/storagenode, you need to:

  • ssh to the server
  • run screen or tmux
  • run command like described by @mars_9t:
rsync -aP /mnt/storj/storagenode/ username@192.168.1.XXX:/mnt/storj/storagenode/

here username is a user on your Raspi, and 192.168.1.XXX its IP.

4

Why don’t you connect the usb harddrive to the server and transfer it directly over usb then connect it to your rpi when done.
This should be a lot faster than trying to transfer it over network.

3 Likes
5

Results from the pi-
:~ $ df -T --si
Filesystem Type Size Used Avail Use% Mounted on
/dev/root ext4 15G 1.6G 13G 12% /
devtmpfs devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs tmpfs 4.1G 0 4.1G 0% /dev/shm
tmpfs tmpfs 1.7G 1.2M 1.7G 1% /run
tmpfs tmpfs 5.3M 4.1k 5.3M 1% /run/lock
/dev/mmcblk0p1 vfat 268M 33M 236M 13% /boot
/dev/sda2 ext4 6.0T 29k 5.7T 1% /mnt/storj2
tmpfs tmpfs 820M 0 820M 0% /run/user/1000

results from the server-
~$ df -T --si
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 34G 0 34G 0% /dev
tmpfs tmpfs 6.8G 2.5M 6.8G 1% /run
/dev/sda2 ext4 5.0T 4.6T 139G 98% /
tmpfs tmpfs 34G 0 34G 0% /dev/shm
tmpfs tmpfs 5.3M 0 5.3M 0% /run/lock
tmpfs tmpfs 34G 0 34G 0% /sys/fs/cgroup
/dev/loop0 squashfs 132k 132k 0 100% /snap/bare/5
/dev/loop1 squashfs 9.5M 9.5M 0 100% /snap/canonical-livepatch/164
/dev/loop2 squashfs 11M 11M 0 100% /snap/canonical-livepatch/190
/dev/loop3 squashfs 67M 67M 0 100% /snap/core20/1828
/dev/loop7 squashfs 53M 53M 0 100% /snap/snapd/18357
/dev/loop11 squashfs 49M 49M 0 100% /snap/snap-store/599
/dev/loop4 squashfs 123M 123M 0 100% /snap/core/14946
/dev/loop6 squashfs 67M 67M 0 100% /snap/core20/1852
/dev/loop5 squashfs 49M 49M 0 100% /snap/snap-store/638
/dev/loop8 squashfs 86M 86M 0 100% /snap/gtk-common-themes/1534
/dev/loop13 squashfs 364M 364M 0 100% /snap/gnome-3-38-2004/119
/dev/loop12 squashfs 53M 53M 0 100% /snap/snapd/18596
/dev/loop9 squashfs 123M 123M 0 100% /snap/core/14784
/dev/loop14 squashfs 97M 97M 0 100% /snap/gtk-common-themes/1535
/dev/loop10 squashfs 367M 367M 0 100% /snap/gnome-3-38-2004/137
/dev/sda1 vfat 536M 6.4M 530M 2% /boot/efi
tmpfs tmpfs 6.8G 78k 6.8G 1% /run/user/1000

I seem to have figured out the path to the original identity just by copying the stated location from the original installation directions ~/.local/share/storj/identity/storagenode/*

But I cannot figure out what needs to be the path for the destination. user@192.168.XXX:~/mnt/storj2/storagenode-new/identity does not work ‘No such file or directory’ I’m obviously making some simple mistake with the path here but I don’t know what.

6

Why don’t you connect the usb harddrive to the server and transfer it directly over usb then connect it to your rpi when done.
This should be a lot faster than trying to transfer it over network.

If there was documentation explaining that, then I would, but I’m not familiar enough with Linux, Storj, or any part of this to diverge from the documentation. I would not be able to get it to work. Waiting for a sync also just isn’t much of a problem for me, but losing the 4.5tb node would be pretty annoying.

7

I suggest to create an identity folder on your data location, i.e. /mnt/storj2/storagenode1/identity with the command on the raspi

mkdir -p /mnt/storj2/storagenode1/identity

and copy your identity to there. Later you will use it in your docker run command.

You usually need to create this directory on a destination, like this:

mkdir -p /mnt/storj2/storagenode-new/identity

However, make sure that your drive is actually mounted to /mnt/storj2 or /mnt/storj2/storagenode-new

8

However, make sure that your drive is actually mounted to /mnt/storj2 or /mnt/storj2/storagenode-new

lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 5.5T 0 disk
├─sda1 8:1 0 1007K 0 part
└─sda2 8:2 0 5.5T 0 part /mnt/storj2
mmcblk0 179:0 0 14.5G 0 disk
├─mmcblk0p1 179:1 0 256M 0 part /boot
└─mmcblk0p2 179:2 0 14.2G 0 part /

You usually need to create this directory on a destination, like this:
mkdir -p /mnt/storj2/storagenode-new/identity

I tried that, it seemed to successfully create the directory but when I tried rsync it still said no such directory.

9

Ommit the ~ as this gets substituted by the user’s home directory /home/user. So the path would be /home/user/mnt/storj2/storagenode-new/identity in your example, which does not exist. What you want ist the right absolute path /mnt/storj2/storagenode-new/identity. Make sure that the user can write into that folder.

1 Like
10

the “~” character means the home directory of current logged in user, for example if you using the root user it will be “/root/”, if you using user “alan” then it’s “/home/alan/”.

you need to remove the “~” character and use an absolute path.

11

Ommit the ~ as this gets substituted by the user’s home directory /home/user.

you need to remove the “~” character and use an absolute path.

That worked! Sort of. It now seems to be making the attempt correctly, but still failed because ‘Permission denied’. The command I am using is ‘sudo rsync -aP ~/.local/share/storj/identity/storagenode/* user@192.168.xxx:/mnt/storj2/storagenode-new/identity/’

When I run that command it asks for sudo password for the machine I’m ssh’d into and then asks for password for the pi. When I enter the passwords it then shows some lines with download rates, all with times of 0:00:00, and tells me they all failed on the reciever side because permission denied.

1 Like
12

The user “user” needs write permissions on the folder /mnt/storj2/storagenode-new/identity/.

1 Like
13

Success! I also created the /mnt/storj2/storagenode-new/orders/ directory and /mnt/storj2/storagenode-new/storage/ directory and given user permissions, but I can’t figure out where the actual orders and data are on the original machine. Can I get some help finding those?

14

you need to check your config.yaml file for this

1 Like
15

From what I’m reading, the config.yaml file is in the same directory that I can’t find though. This feels impossibly complicated.

16

Please show your current docker run command (you may mask the private info).
Or you may execute this command on the source server:

and post result here.

17

I already generated a new identity for a new node. I’ve put over a week into trying to figure this out and I’m still not there, it’s not worth struggling anymore.

If I follow the installation directions for raspberry pi exactly, am I going to end up losing the identity and node in a year when the sd card fails?

18

If you place your identity to the disk with data instead of SD card, it should survive.
In the worst case you would need to re-flash the SD card, install docker and run your node back.
Please never run the setup step again for the existing worked node and it should be ok.