Tardigrade connector for Borg Backup

Ideas & Suggestions Tardigrade feature requests - voting
#1

Hi Storj enthusiasts :slight_smile:,

Duplicati has now a built-in Tardigrade connector. That’s very good news :slightly_smiling_face:
But there also a lot of other very powerful open-source backup software.

Among those, there is Borg Backup.
I tested several backup software. Several times each. Back and forward. Borg Backup is the one that best fits with my own needs.

But this is not the place to compare Duplicati and Borg (both are very good anyway).
I would like to suggest considering the option of creating a Tardigrade connector for Borg Backup.
In my opinion, this software is more script-oriented than Duplicati and allows to easily create complex Backup tasks, using your favorite scripting language, with backup externalization, compression, deduplication and history/archiving features.

Borg backup can already send backup to Cloud providers, including Tardigrade, by using the Tardigrade S3 gateway and mounting the bucket. But it would be very nice if a built-in connector would be implemented. It would be as easy as sending backup to a Borg server.
Plus, Borg Backup doesn’t have a native GUI, it targets directly developers, system admins and so on. They would probably be interested by Tardigrade.

So what do you think about creating a Tardigrade connector for Borg backup?

#2

I’m using borgbackup myself. Borgbackup doesn’t really have a concept for connectors, so it will likely not be a trivial task; first you’d probably need to make borgbackup provide some kind of connector API.

Restic would be easier, it already has the right abstractions and has mostly similar features.

1 Like
#3

Hey @jeremyfritzen!

To easily get started, you could use rclone’s mount to make a local mounted directory, which you then can use as a backup repository. It supports our network as storage option already and thus would make an initial test fairly easy.

1 Like
#4

Borg isn’t really ideal for making backups to cloud storage. It is optimized to require as little bandwidth as possible by running a server program on the backup target over SSH, that’s not possible with standard cloud storage. You could mount a filesystem locally, but at that point you should switch to a backup program that supports cloud storage properly like Restic or Duplicati.

Rsync.net has a borg service: https://www.rsync.net/products/borg.html. I would strongly recommend running your own off site backup server though, or in case of home use, back up to a friend and let them back up to you (https://hub.docker.com/r/derkades/borg-serve)

1 Like
#5

I have the same need, and did some tests.
I’m currently successfully doing nightly backups over a ssh-fuse-mounted remote NAS with borg, it works well.

However, I tested the mounting backends goofys, s3fs-fuse, rclone to mount cloud storage (S3 and in the case of rclone also tardigrade), reasoning that the behaviour of "S3 over fuse" might be a good preview of how a native storj-fuse-mount might perform.

It appears that borg's requirements on the “good” behaviour of the file system are quite high (also confirmed by borg code analysis), and I would get I/O errors along the way, the only one working in the end was s3fs-fuse.

In the end, the only reasonable solution IMHO would be to implement a kind of backend plugin structure directly in borg, and implement a plugin based on libuplink. But this is far from trivial, there have been discussions about doing S3 in borg since 2016 but is until now perceived by borg insiders to remain quite challenging.

As a stop-gap measure a good tardigrade-fuse mount could be envisioned. There was some noise that people are working on this (e.g. Storj’s Jessica), but I haven’t seen results yet. Given the instability in this respect of goofys and rclone, it also seems not trivial, but it might be interesting to base the solution off s3fs-fuse. Actually it’s on my list potentially, but I won’t get around to it in the near future.

1 Like