Since yesterday ive almost 100% been able to retrieve anything i upload. Im using the latest version of the gateway with Cloudberry Explorer. Also tried S3 Explorer today and experiencing the same issues…
Also seeing some errors in the gateway too but not all the time. Perhaps more concerning is that the files appear to upload correctly with no errors. Screenshots below:
Those errors say that not enough uploads were successful to store the piece. So the upload didn’t finish correctly. Are you using an older bucket? I think buckets were created with low RS settings on older versions, making these issues much more prevalent. If you create a new bucket in the latest version it should use the current RS settings. This may fix your problem.
Though I thought this update would also overwrite the bad RS settings for older buckets. Apparently not.
That’s interesting because my storage node upload failure rates are through the roof since the latest update. I was around 86% successful before this update and now I’m down to almost 30% successful uploads. Wonder if there is some correlation.
I assume you mean on the storagenode side?
Yes, there is a correlation. I believe before the update 95 uploads were started and after 80 were done the remaining 15 were cancelled. Now 110 are started and 30 of them cancelled. So that’s already a doubling of the context cancelled amount. But for some nodes the impact may be much higher. If you consistently came in as around the 20th to last node to finish the transfer, you would have succeeded with the previous settings but you fail with the new settings.
Have you tried skipping the third party clients and just using the AWS CLI? Do you see the same issues that way?
That makes sense, I did not think about that change making this impact. And yes it is on the storage node side. Figured there wasn’t so much a problem as a change at some point resulting in this, so thanks for the FYI!