"List objects" with a delimiter is horrendously slow

Thanks. Unfortunately I can’t use the native integration so we’ll have to stay on B2 for the time being. I’m interested in hearing about future performance improvements in this area, for sure.

By the way, seems the slowness is related only to US1 satellite. EU1 for the same dataset work in 19x faster (this is not a typo):

$ time uplink ls --access eu1-demo --recursive sj://data | wc -l
265589

real    0m32.035s
user    0m7.235s
sys     0m3.141s

$ time uplink ls --access eu1-demo sj://data/tmp | wc -l
2

real    1m18.186s
user    0m0.089s
sys     0m0.068s

$ time uplink ls --access eu1-demo sj://data/tmp/ | wc -l
2

real    0m0.211s
user    0m0.016s
sys     0m0.008s

So, I believe that something is not right for US1 only. I passed this info to the team as well.

S3 integration:

for US1

$ time rclone ls us1-demo-gw-mt:data | wc -l
265588

real    5m6.404s
user    0m7.421s
sys     0m1.525s

$ time rclone ls us1-demo-gw-mt:data/tmp | wc -l
17

real    0m0.799s
user    0m0.110s
sys     0m0.017s

$ time rclone ls us1-demo-gw-mt:data/tmp/ | wc -l
17

real    0m0.544s
user    0m0.071s
sys     0m0.053s

for EU1

$ time rclone ls eu1-demo-gw-mt:data | wc -l
265588

real    0m40.035s
user    0m7.243s
sys     0m1.458s

$ time rclone ls eu1-demo-gw-mt:data/tmp | wc -l
17

real    0m0.407s
user    0m0.107s
sys     0m0.013s

$ time rclone ls eu1-demo-gw-mt:data/tmp/ | wc -l
17

real    0m0.360s
user    0m0.106s
sys     0m0.013s

So, with an S3 integration EU1 only in 8x times faster rather than 19x for native, but it’s still a significant difference.

I would suggest to open an account on EU1 and try to use it instead of US1. Nodes would be selected the same way - closest to your location, so only a metadata would be stored on EU1.

2 Likes

That’s kind of wild. I’m hesitant to just switch satellites; if this issue is caused by some update that was applied to US1 but not yet EU1, then using EU1 could put us in a bad spot later whenever this change comes to that satellite.

I suspect that it is related to a not migrated database (we recently started to migrate to a different database backend, AP1 and EU1 are migrated, US1 not yet).

But you can switch the satellite without losing the performance, however, they are separated from each other by a metadata (but shares the same nodes), so you need to re-upload data.

I would hope that we would finish the migration after Holidays.

We reverted the change related to the listing, could you please re-evaluate for your US1 account?