Due to I tred to setup a QNAP internal HBS§ backup job. With Gateway MT it works like a charm with self hosted Gateway ST version 1.2.0 via docker it works also with small amount of data.
When I increase the backup data e.g. to 1.25 GB it never end the backup. I see then a lot of messages in the Gateway ST logs
2021-08-03T11:12:24.147Z INFO ecclient Upload to node canceled by user {“Node ID”: “1UXJaq3XfBpgwBkLiFxBvtAdusWY5FoHG4r1fo7jcmxygMKnU”}
2021-08-03T11:12:24.148Z INFO ecclient Upload to node canceled by user {“Node ID”: “142k8NaRwiKi2dWUgdf4HS56E8AnYr88BChDgC1gDuEdJWfeYy”}
2021-08-03T11:12:24.148Z INFO ecclient Upload to node canceled by user {“Node ID”: “12pNbFFj7G6Y5C9dgzRe5GtA1cb8ucQSeH6qpoTstKwCJm2GnR5”}
According to one of our gateway engineers, Gateway-ST and Gateway-MT have diverged significantly over time, with Gateway-MT receiving significantly more attention. Currently there is an effort to more closely unify them moving forward. Both are currently based on Minio for S3 support, but at the moment they reference different versions of Minio. We hope to unify the two as much as possible over the next 2-8 weeks; as much as possible using the more permissive Apache2 license (Minio recently changed their license to AGPL).
In the mean time, perhaps multipart POSTs instead of PUTs would fix address timeout issue?
Those would be features of the S3 API. Since you’re using a QNAP backup solution, see if there is an option to turn on multipart. Preferably with a part size of 64MB.
Hi @jensamberg! Just recently, my team and I based Gateway-ST and Gateway-MT on a common object layer. This means that the behaviour of operations, e.g. uploading an object, should be exactly the same between these. Would you mind testing the newest version of Gateway-ST (v1.3.1)? Let me know if you run into any trouble.
However I can not test the download speed and reliability due to I already have production data and test the Gateway ST will cost me some extra money. Do you have a coupon for me then I can test reliably ?
I will be happy to help figure this out! Would you mind dumping logs from this container and attaching them here or sending them through, e.g. linksharing? Also, what happens when the download stops? Does it stall, upload speed goes down to 0 B/s and never gets back or something else?
Not really. Gateway-ST shouldn’t log access grants, and if it logs Access Key ID/Access Secret Key—it doesn’t matter outside of local context. That being said, it’s still a good practice to review logs yourself before handing them to someone else. By default, there shouldn’t be many logs, so it should be easy to review them.
From the screenshot you posted, I can see that Gateway-ST runs in a Docker container. I usually check logs of Docker containers by running docker logs <container ID or name>, but a more robust way might be getting the log file directly (docker - Where is a log file with logs from a container? - Stack Overflow). There also might be an option to export the logs in the UI of the application that you use to view running Docker containers.
I don’t have a QNAP drive, unfortunately, but @sean from my team has (if I remember correctly), so I will ask him if we could reproduce it there. Thanks!
Is it possible to post logs after or while you are uploading via Gateway-ST and the upload stalls? There might be some errors or information on what’s happening with the upload there. Enabling debug mode might spew out more information, too.
Hello @jensamberg, I was able to reproduce this exact problem on my QNAP NAS as well attempting to run backup of some 64MB test files, and I’m currently looking into why it’s happening now.
From an initial observation, the backup appears to be work if I disable the “QuDedup” feature in the backup settings in HBS3.
