Gateway ST and Gateway MT are different?

Does Gateway ST and Gateway MT behave different.

Due to I tred to setup a QNAP internal HBS§ backup job. With Gateway MT it works like a charm with self hosted Gateway ST version 1.2.0 via docker it works also with small amount of data.

When I increase the backup data e.g. to 1.25 GB it never end the backup. I see then a lot of messages in the Gateway ST logs

2021-08-03T11:12:24.147Z INFO ecclient Upload to node canceled by user {“Node ID”: “1UXJaq3XfBpgwBkLiFxBvtAdusWY5FoHG4r1fo7jcmxygMKnU”}
2021-08-03T11:12:24.148Z INFO ecclient Upload to node canceled by user {“Node ID”: “142k8NaRwiKi2dWUgdf4HS56E8AnYr88BChDgC1gDuEdJWfeYy”}
2021-08-03T11:12:24.148Z INFO ecclient Upload to node canceled by user {“Node ID”: “12pNbFFj7G6Y5C9dgzRe5GtA1cb8ucQSeH6qpoTstKwCJm2GnR5”}

Those lines look like just longtail cancellation, but I could be wrong. That’s normal behavior when enough pieces are uploaded.

According to one of our gateway engineers, Gateway-ST and Gateway-MT have diverged significantly over time, with Gateway-MT receiving significantly more attention. Currently there is an effort to more closely unify them moving forward. Both are currently based on Minio for S3 support, but at the moment they reference different versions of Minio. We hope to unify the two as much as possible over the next 2-8 weeks; as much as possible using the more permissive Apache2 license (Minio recently changed their license to AGPL).
In the mean time, perhaps multipart POSTs instead of PUTs would fix address timeout issue?


What does this mean ?

Those would be features of the S3 API. Since you’re using a QNAP backup solution, see if there is an option to turn on multipart. Preferably with a part size of 64MB.


I can not found such kind of option.

Have sombody else ever bring it up to work?

Hi @jensamberg! Just recently, my team and I based Gateway-ST and Gateway-MT on a common object layer. This means that the behaviour of operations, e.g. uploading an object, should be exactly the same between these. Would you mind testing the newest version of Gateway-ST (v1.3.1)? Let me know if you run into any trouble.

@artur cool thanks for info. Will test it and provide feedback.

@artur good work. Upload does now work as expected. Also from my feeling performance is higher in upload speed.

However I can not test the download speed and reliability due to I already have production data and test the Gateway ST will cost me some extra money. Do you have a coupon for me then I can test reliably ?

@artur sorry command back. It stops working after a while so ST is not exactly the same as MT

I will be happy to help figure this out! Would you mind dumping logs from this container and attaching them here or sending them through, e.g. linksharing? Also, what happens when the download stops? Does it stall, upload speed goes down to 0 B/s and never gets back or something else?

@artur Yes I will help. I have a better feeling when I host the gateway on my own server.

  1. When I send you my logs. do you see my secrets?
  2. How to read out the logs. Is there something special?
  3. If you have a QNAP Drive you can also reproduce the Issue when you are using HBS3

Not really. Gateway-ST shouldn’t log access grants, and if it logs Access Key ID/Access Secret Key—it doesn’t matter outside of local context. That being said, it’s still a good practice to review logs yourself before handing them to someone else. By default, there shouldn’t be many logs, so it should be easy to review them.

From the screenshot you posted, I can see that Gateway-ST runs in a Docker container. I usually check logs of Docker containers by running docker logs <container ID or name>, but a more robust way might be getting the log file directly (docker - Where is a log file with logs from a container? - Stack Overflow). There also might be an option to export the logs in the UI of the application that you use to view running Docker containers.

I don’t have a QNAP drive, unfortunately, but @sean from my team has (if I remember correctly), so I will ask him if we could reproduce it there. Thanks!


  1. the command docker logs only say this

  2. Have I to activate debug in the config.yaml file. I see there some entries.

Is it possible to post logs after or while you are uploading via Gateway-ST and the upload stalls? There might be some errors or information on what’s happening with the upload there. Enabling debug mode might spew out more information, too. :slight_smile:

How exactly can I enable the Debug mode ?

Setting one of these or both of these flags should do the trick:

  • --log.development
  • --log.level=debug

Depending on your configuration, these might already have been the default values.

Note: I added an issue to our bug tracker for this particular problem: Gateway-ST's (running in Docker on QNAP) upload stalls while using HBS3 · Issue #38 · storj/gateway-st · GitHub. Of course, feel free to add more details, if any, here or there. :slight_smile:

Hello @artur here are the messages when it stop working.

Hello @jensamberg, I was able to reproduce this exact problem on my QNAP NAS as well attempting to run backup of some 64MB test files, and I’m currently looking into why it’s happening now.

From an initial observation, the backup appears to be work if I disable the “QuDedup” feature in the backup settings in HBS3.