Syncing files from several other computers but i want to put them into the storj bucket automatically and not to local storage

Product Discussions
1

i use resilio sync to sync some data from two other computers onto my computer.
and i want to put some of that data into the storj bucket currently mounted with rclone, automatically instead of making a local copy first then copying them manually.

for each synced file resilio creates first a temporary file where it downloads the data, then rename-moves that file into the user dir so that it appears to the user instantly.

and the problem with “all that directly onto cloud”, is that the temp file is a sparse file of course, but cloud storage has no idea about sparse files and rclone begins to upload that file with zeroes in place of gaps and creates a lot of useless upload traffic for data that is still being synced and zeroes that will be soon overwritten with actual data.

there is no way in resilio to specify temp location on a drive or folder different than the .sync folder inside the user dir, so i have to periodically copy those files manually if i want them on storj and cannot make resilio do this automatically.

any ideas how i could automate this, maybe with some other software involved?

2

If you ultimately are sending data from “two other computers” to Storj… why the temporary step of sending their data to the rclone-mounted-bucket? Can’t you sync directly from the two other computers to Storj? (You can still keep the rclone mount for other uses)

3

on more thought, i think i can afford making a local copy first with resilio, only need to automate syncing with the cloud a resilio user folder with the .sync subfolder excluded/ignored.

4

please more details which software to use, and i will compare putting that onto all computers instead of resilio.

5

Rclone works fine for periodic backups to S3 (sync, copy): you don’t need to leave it running as a live mount.

But you can pick any backup program that can talk directly to S3. There are many, even free/open-source. Restic (or +Backrest) is popular.

6

thanks.
but i still think that a very simple cross-folder syncing tool may suffice, only if it permits excluding/ignoring a specified subfolder.
in this case i would also keep local copies for faster access, and just make one extra copy on storj for offsite considerations.

7

any suggestions for the simplest cross-folder syncing tool?

8

You could use rsync for your local copy, and rclone to push things to S3.

9

great, thanks.
but are there any alternatives that run in the background and sync that automatically and in near-real time insted of a scheduled periodicity, similar to what resilio does?

10

also a limitation of resilio is that it allows only one instance of a folder on each instance of resilio.
and it can sync only across different computers and never just two local folders! :slight_smile:

11

So you’re looking for live-sync, and not backups? For example… if you accidentally delete some files a backup will let you restore them from the past… but a live-sync tool will happily delete the files from everywhere and they’re gone. (But you can use live-sync tools… to build a backup system).

Syncthing would be one option. Or…you mentioned syncing several computers: is this to protect VMs so they can always run somewhere? If so, have you looked at Ceph?

12

nah, it is mainly for syncing software install kits so i can have them handy on all computers and in case of failures i may need them to reinstall something while having access only to local storage and not cloud because the entire internet access may be unavailable in those cases.
and i would prefer to also have a offsite storj copy of them.
yes, it is more in the realm of live-sync than periodic backup.

13

that so far it seems that a periodic cloud sync might suffice, and leave resilio as it is now, live-syncing locally.

14

locally i mean within my lan.

15

back in time, when i was evaluating syncthing vs resilio, choice fell on resilio because it is more user-friendly than syncthing and it satisfied my then-needs.

but i already dont remember, can syncthing sync just two local folders with a subfolder excluded?
this actually never even fell into what i’ve looked for back then.

16

and to advantages of resilio is that they have relay servers for cases when two clients are behind nat and none of them can portmap.
of course slower than when direct connections are possible, but it works all the time as soon as there is at least some form of internet access.

17

For your original question—I see the discussion went in a different direction—you might want to check rclone’s --vfs-write-back parameter together with --vfs-cache-mode full, it is explicitly designed to delay upload while file is modified. Obviously this means you need to explicitly wait for rclone to finish all uploads, otherwise syncing is not actually done… but seems it would for your use case.

18

Effectively,

rclone nfsmount remote:path /mount/point \
    --vfs-cache-mode full  \
    --vfs-cache-max-size 1024G \
    --vfs-write-back 5s  \
    --daemon \
    --daemon-wait 30s

will provide the “data lives in the cloud while fully accessible locally” behavior.

Thai does not magically solve sync conflicts but you have to deal them with the sync as well.

The key is to use nfmount and not fuse.

I’ve used this setup for a while but then abandoned it to 180 opposite approach — keeping source of trust of the data locally.

Before they resort to relaying they do all sort of NAT traversal techniques, in my experience I had never witnessed them using relay. I’m not sure which parts come from the BitTorrent-derived design and which parts are Resilio’s own infrastructure, but the result is very good.

Answering your other question, for the actual replication resilio is far ahead of syncthing in about any metric.

Your usecase specifically

Is a prime usecase for resilio. You already have the best possible setup.

If you also want that in storj (why?) I would run periodic rclone sync on one the local stores. (Do not give rclone mounted folder to resilio; resilio writes intermediate state into the folder that you don’t want propagated)

19

Hmmmm… out of curiosity, why? I never complained about fuse for this kind of use case.

20

very good question in parenthesis and after a night of sleep i am heavily scratching my head whether i really need that, given that the data is intended only for local use.

and also a good question would be that how likely is that data to be gone locally from all 3 places where it resides.
and also a good question would be how critical is that data to bother with an offsite copy, given that they are install kits that i can reobtain from the net.