Storage node migration with mergerfs?

Node Operators
1

I’ll soon need to migrate some of my nodes to different drives. The usual way would be to just run rsync multiple times, the last time with the node turned off. But this means downtime for the last rsync run, which I recall to take few hours.

I wonder how reasonable would be to first set up mergerfs between the source and target blobs directories, setting the target as the only writable directory (ie. the source directory as «no-create»). Then move older data with mergerfs.consolidate while the node is operating as usual. After this step the target directory would contain an up-to-date version of the blobs directory, so all I would need to do while the node is down is to manually move the database/trash files, which are orders of magnitude smaller to copy.

What do you think?

2

few hours downtime for last sync? my 1tb node sync about 5 min last time. about 2 million files

1 Like
3

I would not recommend to use mergefs. If you have used LVM to create partitions on your disks, you can use it to migrate your data

  1. Create PV on your new disk
  2. Add a new PV to VG
  3. Start move data from source PV to a new one (pvmove).
  4. Mark the source PV as readonly to force to place all new data only to a new PV
  5. After the move would be done - you can remove the source PV from VG and re-use it somewhere else.

The node can run during the whole migration process. Of course both disks would be under a higher load than with rsync, but no downtime at all.

4

I have, but the logical volume hosts more nodes, and I want to migrate only one of them.

5

Maybe you have enough RAM for all that metadata to be cached from one rsync run to another. My box doesn’t, and needs to do millions of IO operations to go over all the files again.

6

My node run in 775 celeron and 2gb ddr2 ram :slight_smile:
2gb enough to run about 4-5 parralel rsync tasks. in 775 socket slowest part is processor, not hdd

7

You don’t exactly have to boast how powerful your machine is. It’s off-topic anyway.

8

I had the same idea for moving files to new hdd using mergerfs. Which solution did you use at the end?

9

Just a classic rsync. Slow, but reliable.

1 Like