Hi,
I had some Cloudsync backup tasks in TrueNAS Core migrated to Scale too as Cloudsync task. Now I see TrueCloud in Scale that also takes snapshots and uploads that to Storj.
I have pointed the new task to the same bucket, but it seems it uploaded everything again as the data used is twice the amount.
Is my assumption correct?
On the Storj website I can browse my bucket, but I don’t see snapshots (because I view the project with my passphrase I guess). So I think I am seeing the “old” cloudsync data and not the TrueCloud data?
Is there a way to see the TrueCloud data?
Should I just remove the data on the website as it is the Cloudsync data?
And is there an easy way to move from Cloudsync to Truecloud without reuploading data? I have more tasks, but kept them in cloudsync for now.
Use the same passphrase with the second CloudSync task as you used with the first.
To see old data in the bucket - use it with the old passphrase. Storj bucket can contain multiple objects encrypted with different passphrases, if the provided passphrase does not match the one object was uploaded with it won’t be visible.
Thanks, so I need to enter the “old passphrase” into this field? In Core/Cloudsync, there was no passphrase, in the website there is (specific for the buckets, next to the login)
I don’t use Scale, but I don’t think it’s the right password. You can click that “?” next to it to learn more what it does – perhaps it encrypts the data before uploading, independently of the remote?
You want to configure credentials (see Remote → Credentials) in the right top corner on your screenshot. As part of configuring those credentials (in Core it was under System → Cloud Credentials) you would either need to provide a storage grant, or S3 credentials, or whatever else; somewhere in the process of generating or inputting the credentials you would by necessity need to specify the passphrase. I don’t know whether Scale uses native integration or S3; if they use S3 then the passphrase would be needed at the stage of generating the s3 credentials on the satellite web site.
This version has two separated methods of backup - the CloudSync (which I wouldn’t call a backup) and TrueCloud (restic).
So, basically they are incompatible.
@etienneb I would recommend you to remove data, uploaded with CloudSync (you should use rclone purge storj:my-bucket or uplink rm --recursive --parallelism 30 --encrypted sj://my-bucket and run the TrueCloud backup - the first one would be a full backup, all others are snapshots (by default).
However, you can keep both. You may also generate a new S3 credentials for the TrueCloud backup using a different encryption phrase. And then these backups will be visible only with that encryption phrase and any credentials created with it.
The old one CloudSync backup will be visible only with the old encryption phrase or any credentials which are generated with it.
Or you may use the same credentials in both. Perhaps they wouldn’t fight for your data in the same bucket , and you may also use a prefix (“subfolder”) for any of them or both to separate them in the bucket.
No, this is an additional encryption above the Storj encryption. @arrogantrabbit meant the encryption phrase used when you created your S3 credentials in the Storj Console. So, if you would use the same S3 credentials as for CloudSync, both backups would be visible with that encryption phrase in the Storj Console or using uplink, rclone, Cyberduck, etc.
According to the Truenas manual, that password is for restoring the backup task on a new install/machine, so you have access to the bucket etc. There is no separate passphrase for encryption when creating a backup task.
Thanks Alexey, that is clarifying. I think starting with new bucket inside my project for the truecloud backup is the easiest and then afterwards remove the “old” buckets. Unfornunately, that means reuploading 2TB of data.
@Alexey
However, you can keep both. You may also generate a new S3 credentials for the TrueCloud backup using a different encryption phrase. And then these backups will be visible only with that encryption phrase and any credentials created with it.
The old one CloudSync backup will be visible only with the old encryption phrase or any credentials which are generated with it.
As far as I can see, there is only 1 (or more) passphrase for buckets and that is set in the storj web console. In Truenas you setup the S3 keys created on the Storj site, but no separate passphrases for encryption.
I started with new S3 credentials in storj, loaded them into TrueNAS Scale. In Truenas I created a new bucket. It started uploading.
But I am unable to see the contents in the bucket on the Storj Site, the bucket has data but nothing is visible. That is intended for the Truenas TrueCloud? That only Truenas can see it? Or where is that passphrase? The Storj manual has a different screen with passphrase when you set up a S3 service for TrueNAS, it is not there when I created the S3.
Thanks, there is one downside to this: you won’t be able to quickly download a single backup file via the Storj website (since you have to use a TrueNAS server).
Yes and no. You won’t see your data plaintext on the satellite portal (which is never a requirement and is never useful; web portal does not even show you all the files you have in the bucket and is not intended for primary access) but you don’t need TrueNAS to restore it either. You can use rclone (if synced) or restic (if backed up) on any other machine. Both tools are mature and exist since forever.
And very importantly, a sync is not a backup, as Alexey pointed out above. Most backup solutions that create versioned, often deduplicated, backup will store data in some form of opaque container on the destination. It’s not specific to restic.
Does the bucket look empty? Then the passphrase is wrong.
Again: there are two pasphrases involved. One is that storj requires. Depending on the method of authentication with storj that passphrase may or may not be wrapped into credentials — like S3 secret. Or handled separately, like when using api key. That is setup where you create a connection.
Then tools TrueNAS uses — restic and rclone — have their own encryption support, because they don’t just work with storj, and other remotes may not be end-to-end encrypted like storj. That one is setup on a task page.
That is my issue. I can’t set nor find a second passphrase when you create S3 credentials (new) in or via TrueNAS Scale.
The S3 credentials you create only give you an Access key and secret key. And In Truenas Scale you create a password for the specific TrueCloud task, for later restoration of the task on a new system. There is just no passphrase possibility as far as I can find.
In TrueNAS I can browse the bucket, but on the Storj website it shows an object count etc, but the folder has no contents as it asks for the unknown passphrase (and is empty with the other passphrase I use to the other buckets from Cloudsync).
Do TrueNAS Scale and Storj create their own passphrase and store it somewhere locally/serverside out of reach of the user?
I prefer the TrueCloud backups as it uses snapshots too, like I do with replication to another server.
Somewhere between you logging in to satellite web interface and generating the s3 credentials you were asked for the passphrase. That passphrase is then save d on the satellite protected with the S3 secret. It’s not possible to create s3 credentials without specifying the passphrase.
It’s the last step in the wizard creating the s3 credentials:
I see, I just tested it by creating another s3 without entering the passphrase. As you said, then you need to enter or create a passphrase.
I will wait until the backup is finished, though there are objects in there already the folder remains empty with my “old” passphrase.
Effectively, you can’t do anything with that bucket used with TrueCloud via the Storj Website, except for deleting it totally.
If you can’t find a passphrase used for data in that bucket — yes, by design. The same is true with the new passphrase — if you lose it your new data is also poof.
This Password not only to restore your backup, restic uses it to encrypt your backup (this is an additional client-side encryption before the Storj server-side encryption, since you use S3).
Because you need to use the same encryption phrase which you have used when generated S3 credentials for that task. You would see a bunch of files, but they are not your raw files like in a CloudSync, they are encrypted (by restic on TrueNAS) chunks of data. These encryptions are separate - the first one is used by restic in the TrueCloud Backup, the second one is encryption phrase for objects stored in Storj buckets.
It’s under the project menu on the left sidebar, it’s called Manage Passphrase. When you generate your S3 credentials it uses the encryption phrase provided when you open a project or any bucket in it. This encryption phrase is not bound to a specific bucket, it’s used to encrypt/decrypt any object in any bucket of that project. However, it’s stored in your local browser session only, so if you would refresh the page it will gone. If you didn’t remember/save it and didn’t save your S3 credentials/access grant, then you may consider your data as lost. Without encryption phrase it would be useless and can be only removed.
Since you provided these credentials in the TrueNAS, then they would work, however, they would work while this instance is exist and working. I do not know, where it stores these credentials, but they should store them in rclone config. And if they didn’t enable the config encryption, then they would be available in a plain text.
, and you may also use a prefix (“subfolder”) for any of them or both to separate them in the bucket.