How to delete bucket faster than for 12h?

Developer Discussions DCS
,
1

I’ve bucket with 270 files. Time to delete 22 minutes :scream:
How I can speedup delete process?

2

If you’re not on the latest uplink version it could help to update. I know there have been improvements to delete performance in recent updates.

3

Using lates version, delete process super-slow. Upload speed seems good but delete process… Seems like tardigrade still unusable or good to store few big files

4

latest version, 140 files
real 10m40.263s
user 0m0.228s
sys 0m0.038s

5

Yeah, that’s still not great. It’s not my expertise so I’ll leave it to someone else to respond further. Just wanted to make sure it wasn’t just an old still slow version.

6

I thought they implemented a delete cache on the satellite for uplinks. Why would it be still taking such a long time?

7

WOW! Satellite so slow)

failed to delete encrypted object, cannot empty bucket "filekormeemzgzisrbjpydjvpmbecqusocvj":     uplink: metainfo error: context canceled
            storj.io/uplink/private/metainfo.(*Client).BeginDeleteObject:683
            storj.io/uplink/private/storage/streams.(*streamStore).Delete:426
            storj.io/uplink/private/storage/streams.(*shimStore).Delete:61
            storj.io/uplink/private/metainfo/kvmetainfo.(*DB).DeleteObject:136
            storj.io/uplink.(*Project).DeleteObject:99
            storj.io/storj/cmd/uplink/cmd.deleteBucket:81
            storj.io/private/process.cleanup.func1.4:359
            storj.io/private/process.cleanup.func1:377
            github.com/spf13/cobra.(*Command).execute:840
            github.com/spf13/cobra.(*Command).ExecuteC:945
            github.com/spf13/cobra.(*Command).Execute:885
            storj.io/private/process.ExecWithCustomConfig:88
            main.main:16
            runtime.main:203
8

I did it like that, then all files are gone!

C:\Uplink>uplink rb sj:// test / --force

9

I do same: uplink rb --force sj://name

10

Well there’s your problem, you have a typo in there.

11

Yes, typo only at this forum, not command in shell.

12

Ah I see.
Well keep us informed on any delete performance improvements in any case.

13

Hey, we took that task and working on improving performance of file deleting from nodes themselves and from the databases. Will be done soon )

7 Likes
14

Hello, do you have any updates? Still have this problem to…

1 Like
15

Hello @anon68609175, we are still making improvements to the delete speed on the network. Our performance team has been working on this problem for some time and hopefully, some of the improvements we are making will give it a big performance boost. You can always check out what we are working on in our monthly development updates or on our “view into our sprints” at the bottom of our development updates.

3 Likes
17

add a damn file that just contains a list of the files that should be deleted but hasn’t been processed yet.
just because something looks like it’s deleted for the customer doesn’t mean it has to be deleted at that time… just like a regular hdd or ssd or file system doesn’t actually delete a file when a user deletes the file… it simply pulls the name tag off the file and thus it’s marked as a deleted file…

if you don’t want to have a file for keeping track of it, using certain special file names that is hidden for the user and is then deleted when the system has time for it.

like say it sends a list of stuff that should be deleted to the satellite and then the …

i guess one of the issues one runs into is the live issue… because the system is secure and only the customer has access to the files then how does one know which files to delete without the customer computer telling us…and thus all deletions has to go through the user… but again this could be solved by removing the time factor…the customer must be able to log on for 1 sec, send a task which the network then processes afterwards… like say delete these “strings” pieces whatever its called
one only needs the customer to translate what should be deleted and record a list of that…

ofc that would mean if a piece can contain multiple files… then one runs into the whole … some files are deleted, but not others in a certain piece (string) and since some are suppose to be deleted and others are not, then… one is locked into reprocessing the whole thing which requires the customer to be online i suppose…

anyways… i duno it’s a confusing topic and sure does seem like it might require some additional trickery to make work… but i think i touched upon a few important points, that if not already, should be taken into account.

and i don’t have much knowledge about the tardigrade coding, so just thinking out loud to the general conceptual terms that relates to all such things, in my understanding of the topic.
in hopes that it might inspire a good idea.

3 Likes
18

Hi everyone

I’ve been backing up my 365 cloud via Veeam to the S3 Gateway since November 2021. Recently, I lost the keys to the bucket and had to recreate.

In the process of trying to delete the old bucket, I’m encountering what seems to be slowness. Originally, bucket size was 850 GB in 120k segments. Attempts to remove it from the web UI didn’t seem to work. I was able to browse and delete small folders of specific files…

Fast forward, I used the uplink CLI to delete the bucket but that has happened between errors.

I am now on my 3rd iteration of issuing the rb --force command. I continue to see a reduction in the segment count on the web. The CLI seems to timeout after a while with the following error. I reissue the rb command and it appears to continue to empty the bucket. Is this normal behavior? Is there another way to go about this from the user side?

Bucket has NOT been deleted
uplink: bucket: metaclient: context canceled
Error: uplink: bucket: metaclient: context canceled

Looking forward,
refined

19

You may try this way:

uplink rm --recursive --encrypted --parallelism 100 sj://my-bucket

then after successfully deleted objects you can try to remove the bucket:

uplink rb --force sj://my-bucket