# Atomic snapshots for buckets **URL:** \ **Category:** Product Discussions\ **Created:** [November 29, 2022, 11:19pm UTC](https://forum.storj.io/t/atomic-snapshots-for-buckets/20510 "2022-11-29T23:19:49Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\ **Post date:** [November 29, 2022, 11:19pm UTC](https://forum.storj.io/t/atomic-snapshots-for-buckets/20510/1 "2022-11-29T23:19:49Z")
I’ve talked to a heavy AWS S3 user today. He complained to me that they’d love to see a form of atomic snapshots for whole buckets, and AWS S3 does not offer anything like that. The apparently keep tens of millions of objects in a single bucket, which keep changing. Any iteration over all items in a bucket is bound to take hours. And they’d love to be able to make an atomic snapshot of a bucket’s state for backup and investigation purposes. Is this possible with server-side copies on Storj? ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [November 30, 2022, 8:33am UTC](https://forum.storj.io/t/atomic-snapshots-for-buckets/20510/2 "2022-11-30T08:33:25Z")
Snapshot yes, but it would be not instant, thus not atomic. You may increase `--transfers` though ```auto ./uplink mb sj://test2 ./uplink cp --recursive --transfers 20 sj://test sj://test2 ``` ---
**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\ **Post date:** [November 30, 2022, 10:09am UTC](https://forum.storj.io/t/atomic-snapshots-for-buckets/20510/3 "2022-11-30T10:09:04Z")
Yeah, for the use case he was describing, he really insisted on atomicity. He is working around his problem now by setting up an EBS volume with LVM and does LVM snapshots instead, but he was clearly annoyed he has to do so. ---
**Author:** ![Vadim](https://storj.bcdn.literatehosting.com/letter_avatar/vadim/32/5_5575768a8748004e209b776fc1b2916d.png) [@Vadim](https://forum.storj.io/u/Vadim)\ **Post date:** [November 30, 2022, 10:22am UTC](https://forum.storj.io/t/atomic-snapshots-for-buckets/20510/4 "2022-11-30T10:22:45Z")
I thin the problems can be solved easily, as any change in file for storj is a new file, then old files just need to keep every instance, it takes space(lot of space) but you can recover every change. ---
**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\ **Post date:** [November 30, 2022, 10:23am UTC](https://forum.storj.io/t/atomic-snapshots-for-buckets/20510/5 "2022-11-30T10:23:50Z")
He stores hundreds of terabytes this way. Storing every past instance would be quite expensive. ---
**Author:** ![Vadim](https://storj.bcdn.literatehosting.com/letter_avatar/vadim/32/5_5575768a8748004e209b776fc1b2916d.png) [@Vadim](https://forum.storj.io/u/Vadim)\ **Post date:** [November 30, 2022, 10:25am UTC](https://forum.storj.io/t/atomic-snapshots-for-buckets/20510/6 "2022-11-30T10:25:06Z")
you dont need to hold it forever, also usually people change something like 10% of files or even less. ---
**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\ **Post date:** [November 30, 2022, 10:26am UTC](https://forum.storj.io/t/atomic-snapshots-for-buckets/20510/7 "2022-11-30T10:26:14Z")
As I understand it, the oldest files in his dataset are ~3 months old. As in, the past storage would grow by hundreds of terabytes every 3 months. ---
**Author:** ![Vadim](https://storj.bcdn.literatehosting.com/letter_avatar/vadim/32/5_5575768a8748004e209b776fc1b2916d.png) [@Vadim](https://forum.storj.io/u/Vadim)\ **Post date:** [November 30, 2022, 10:26am UTC](https://forum.storj.io/t/atomic-snapshots-for-buckets/20510/8 "2022-11-30T10:26:49Z")
Also backups are making copy of all files, but this way it will make files that chenges. Every backup will twice this amount. ---
**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\ **Post date:** [November 30, 2022, 10:30am UTC](https://forum.storj.io/t/atomic-snapshots-for-buckets/20510/9 "2022-11-30T10:30:31Z")
The idea was to make a snapshot to be able to debug the state of application and to make a consistent offline backup. So these snapshots would be short-lived. ---
**Author:** ![Vadim](https://storj.bcdn.literatehosting.com/letter_avatar/vadim/32/5_5575768a8748004e209b776fc1b2916d.png) [@Vadim](https://forum.storj.io/u/Vadim)\ **Post date:** [November 30, 2022, 10:31am UTC](https://forum.storj.io/t/atomic-snapshots-for-buckets/20510/10 "2022-11-30T10:31:58Z")
Client can always make some time thing, like after 3 months it delete all old version and stay only last and 1 previous version. More over if they are short time, then client just delete them. When you change file in storj you always need to upload new file, then you deside delete or not delete old, it not happen itself. ---
**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\ **Post date:** [November 30, 2022, 10:39am UTC](https://forum.storj.io/t/atomic-snapshots-for-buckets/20510/11 "2022-11-30T10:39:08Z")
The difficulty is not in storing files, the difficulty is in knowing what was exactly the state of the bucket at a snapshot time. That is, this file existed and had this content, that file didn’t exist yet. Manually querying all files would be too slow. ---
**Author:** ![Pentium100](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pentium100/32/6082_2.png) [@Pentium100](https://forum.storj.io/u/Pentium100)\ **Post date:** [November 30, 2022, 11:28am UTC](https://forum.storj.io/t/atomic-snapshots-for-buckets/20510/12 "2022-11-30T11:28:37Z")
Snapshots take up space as well, since they work the same way (some way of keeping what has changed). But yeah, since in Storj, every change is a new file, this means that Storj is basically COW, adding snapshots to that should not be too difficult - just keep the old data until the snapshot is deleted. ---
**Author:** ![Jacob](https://storj.bcdn.literatehosting.com/letter_avatar/jacob/32/5_5575768a8748004e209b776fc1b2916d.png) [@Jacob](https://forum.storj.io/u/Jacob)\ **Post date:** [December 1, 2022, 4:31am UTC](https://forum.storj.io/t/atomic-snapshots-for-buckets/20510/13 "2022-12-01T04:31:23Z")
@Toyoo Would this user be willing to send us more details or get on a call to talk to a product manager at Storj? We’d love to learn more details and exactly the problem they are trying to solve. Send me a PM if they are willing to chat. ---
**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\ **Post date:** [December 1, 2022, 8:14am UTC](https://forum.storj.io/t/atomic-snapshots-for-buckets/20510/14 "2022-12-01T08:14:30Z")
> [@Jacob](#): > > @Toyoo Would this user be willing to send us more details or get on a call to talk to a product manager at Storj? Hmmm… I’ll send him a message, we’ll see. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [December 2, 2022, 3:36am UTC](https://forum.storj.io/t/atomic-snapshots-for-buckets/20510/15 "2022-12-02T03:36:49Z")
Maybe they can use [Using Restic for Backups - Storj Docs](https://docs.storj.io/dcs/how-tos/backup-with-restic) It has snapshots feature and uses not 1:1 space. > [@Toyoo](#): > > He stores hundreds of terabytes this way. Storing every past instance would be quite expensive. By the way, server-side copy on Storj DCS doesn’t uses 100% for the copy, until you change the original, then it will start to consume space.