I’m in the process of copying my buckets from another S3 storage to Storj.
Some copies go smoothly, with thousands of files being copied without a single error.
However, in some of these copy processes (like the one I’m running at the moment) errors repeatedly occur. In this case, there are about 2 thousand files to be copied, and every few dozen files copied the process is interrupted with this error:
Failed to upload the file [redacted]: InternalError: We encountered an internal error, please try again.: cause(uplink: stream: metaclient: metabase: object already exists)
status code: 500, request id: 16FE0A34025FD8A6, host id:
(The error occurs with different files on each try)
The performance of successive copy runs gets worse until it becomes impractical, with only a few files copied per run:
Files to copy: 2295, to skip: 23302, total: 25597
Copied file [redacted] (1/2295) 1.12MB/s 02:35:02 0.0%
Copied file [redacted] (2/2295) 2.64MB/s 01:30:46 0.1%
Copied file [redacted] (3/2295) 466KB/s 00:36:04 0.1%
Failed to upload the file [redacted]: InternalError: We encountered an internal error, please try again.: cause(uplink: stream: metaclient: metabase: object already exists)
status code: 500, request id: 16FE187C3B471588, host id:
The other times it happened, with other buckets, I usually - not always - managed to solve it only when I completely deleted the bucket itself and recreated it.
But there must be a better way to fix this error, once the root cause is identified…
The files are chunks of backups generated by Duplicacy, so I have to use Duplicacy itself to do the copy, I can’t use Cyberduck or something like that.
I can’t use Rclone because the chunks change during copying. Duplicacy reads the source chunk, decrypts it, encrypts it with the new storage (DCS) key, and uploads it.
(There is an option used in the creation of storage by Duplicacy that would make the chunks not need to be decrypted in the copy, and so I could use Rclone, but as I’m changing the encryption key, I cannot use this option.)
Thanks! One case we discovered is that when new uploads are created concurrently for the same object key sometimes the “object already exists” error is thrown. I was only able to replicate this synthetically by creating many uploads with the same key in parallel. We are investigating how and why some S3 clients send requests this way that can lead to this error.
I noticed there’s a -threads option to Duplicacy. I’m not sure what it defaults to, but does setting it to 1 help you at all for the moment?
Yes, that was one of the first options I thought of, and even with -threads 1 the error occurs
It defaults to 4.
Edit: complementing, it seems that using threads 1 option, the copy advances a little further, something like a dozen files copied, in the several attempts I made, but it is not something completely determined.
Another point to consider is that in some other buckets, even larger ones and with more files, I was able to copy without any problem. I couldn’t identify any characteristic that would determine this (success or failure), they are all buckets with Duplicacy backup files.
The other times this error occurred, the only solution I found was to completely delete the bucket, recreate it with the same name, and redo the copy. Sometimes it worked on the second try, sometimes on the third.
Thank you for this! That will give me something to go on in trying to replicate the problem. Also, it’s great you were able to successfully finish the copy at least!
Could this perhaps have something to do with how asynchronous deletes are implemented? Like if a delete and upload for the same filename happened in quick succession?
Not sure if duplicacy would even have that behavior though, but it might if there is an update to a file.
I waited a day and looked for this file (chunk) in bucket in various ways (web interface, Rclone, Cyberduck) and… it’s not there… In other words, there’s no way to even manually delete it and resume the backup. The only way would be to completely erase the bucket and restart, which is obviously not practical.
It looks like only the metadata of the file is in a database (?) somewhere.
No, chunks are immutable once uploaded. They can only be deleted when a prune is performed. And Duplicacy only saves the backup snapshot (something like an index of the chunks) after all chunks from that snapshot have been uploaded. That is, chunks are not updated.
Have you tried using s3c:// instead? That backend uses different S3 client code in Duplicacy that has a delay between retrying requests, which may help you to avoid these errors.
We currently have an initiative “object consistency” which will allow for multiple pending objects under the same key which will get rid of this error.
