Is there any form of lifecycle management of files stored in a bucket?
Backblaze has it - as does Amazon
1 Like
You can set an expiration date for objects. If that is what you mean. Developers can then build such a service around Storj DCS if their requirements call for it.
2 Likes
Through the GUI?
I don’t see a way of doing this.
I guess there might be in the CLI - but I am not using that
Storj GUI is not meant to be full featured. The interface provided is mainly to allow developers and such to get an understanding of the tools at a high level. Most of the functionality is available at the Command Line level or via S3 connectivity to existing (and new) applications that leverage Storj, and which features they utilize is determined by the developers and what they need.
1 Like
No, the GUI doesn’t have this feature, you need to use uplink CLI for example:
The other way is to generate an access grant/S3 credentials with provided TTL, i.e.
uplink share --max-object-ttl 24h --readonly=false --not-after=none sj://my-bucket
then use this access grant in any of your tools. All objects uploaded with this access grant will expire after 24h.
You may also generate an S3 credentials with the same behavior:
uplink share --max-object-ttl 24h --readonly=false --not-after=none --register sj://my-bucket
2 Likes
We’ve just hit this problem when trying to migrate customers from both S3 and Backblaze. Both have lifecycle policy’s on the buckets that delete the files automatically after 30 days.
We are a big user of rclone and this doesn’t seem to support the expiry feature described above.
Hello @dogsbody,
Welcome to the forum!
You have two options:
- Provide the TTL header when you use rclone to upload data and specify a delta or the exact date when uploaded objects should be auto deleted
- Create the access grant/S3 credentials with TTL. All objects uploaded using this access grant or these S3 credentials would have this TTL.
See
1 Like
I would like to awaken this request. I have objects that need to expire after a certain amount of time has passed after the object is marked as deleted in a versioned bucket. Currently I have to manually clear out old/deleted objects using rclone. I don’t have a way to predict when the objects should expire at upload time as they may be arbitrarily deleted. All objects have a 30d compliance lock applied at upload time and extended as needed.
With Backblaze this deletion happened automatically based on the defined object lifecycle rule such as “delete old versions after x days”. This is the feature that I am missing.
1 Like
We have a feature to upload objects with TTL, you can also create an access grant/s3 credentials which would have this TTL, docs are linked above.
However, Object Lock and TTL are Mutually Exclusive.
So, you should either use an Object lock or TTL.
Please also note, that rclone didn’t delete versions, it puts a deletion marker as described in the article Working with delete markers - Amazon Simple Storage Service.
To delete versions you need to specify this option, e.g.
rclone delete --s3-versions --s3-version-deleted --min-age 30d --rmdirs -P storj-s3:my-bucket/my-prefix/
You can still delete the object and all its versions even if they were uploaded with TTL. But if you use an Object lock feature, you will not be able to upload with a TTL. And depending on a Object lock mode (Compliance, Governance or Legal Hold) you might be not able to delete the not expired version at all.
Thank you for the rclone command, that is actually very helpful. I was using
rclone backend cleanup-hidden storj-s3:backup
before and it errors on every object still locked because it doesn’t check if it is locked before trying to delete it. The software I am using is kopia backup. It automatically extends the object locks to prevent backups being deleted, and puts delete markers on objects that are no longer needed. It does not remove hidden objects after their lock expires, so I have to use rclone for that step. This is where the lifecycle management feature would normally remove the deleted (hidden) objects after the lock expires.
I’m aware of the TTL feature, but it doesn’t work in this case as the object lock and TTL are mutually exclusive as you said. It’s also impossible to tell how long a certain object will need to exist at upload time, so no TTL.
1 Like
I’m putting this out here for anyone else who decides to try these commands.
I used the rclone command rclone delete --s3-versions --s3-version-deleted --min-age 30d --rmdirs -P storj-s3:my-bucket/my-prefix/ you gave me and it deleted every object that was 30 days old (by adding a delete marker to them). This broke my backup. I was able to recover by removing all the delete markers to restore the versioned objects. Just putting this out here in case anyone else has the same problem.
I believe the issue here is that the rclone delete command automatically puts a delete marker on versioned objects when using the delete command. Then it chose objects that were 30 days old and deleted them. It was not removing already deleted (hidden) objects.
rclone backend cleanup-hidden does sort of work, but rclone doesn’t support object locks so it fails in a safer way. It will delete the unlocked objects with delete markers only, but then it tries to delete objects still locked, starting with the latest version which is always the delete marker, then it gets a 403 when it tries to remove the underlying object. So it actually ends up undeleting all the objects that are still locked. I’m looking into scripting this myself as rclone can’t handle it. rclone hasn’t implemented this because every other S3 provider has basic lifecycle management features that take care of this edge case automatically.
You are correct, rclone puts a deletion marker with the delete command, unless all versions would be requested to be deleted (which is not the case, when you use a filter like older than 30 days). However, I did expect, that it will delete all previous versions as well. Seems I was wrong.
I do not have any suggestions how to overcome this at this time and will share your feedback with the team.