I don't see most of my Folders anymore

Product Discussions
1

Hey Folks,

I’m kinda nervous. About 2 hrs. ago it first appeared, from one moment to the other I wasn’t able to list like 95% of my folders anymore, despite that they are still there, let me explain.

No matter if I use the uplink CLI or the gateway-st software, as soon as I list a Bucket I’m currently missing a lot of subfolders, or at least the listings for them.

Take a look at this:


uplink ls sj://resource
KIND    CREATED    SIZE    KEY
PRE                        Sprites/
PRE                        Artists/

As you can see, inside the resource bucket I only have two (2) folders according to the ls output, but if I do this:


uplink ls --recursive sj://resource/Visual_Analytics/
KIND    CREATED                SIZE        KEY
OBJ     2025-01-06 12:39:38    14060960    Visual_Analytics/ca8826c4-3a7e-45e1-9f1e-06a32923f1c0.tar.xz
OBJ     2025-01-05 22:03:00    6232904     Visual_Analytics/795d3f6c-1a23-4828-aa59-52c0dc174d44.tar.xz

We can clearly see that Visual_Analytics folder is inside the resource bucket…
Can please somebody explain to me why I don’t see the Visual_Analytics folder anymore when I do a uplink ls sj://resource

My uplink version:


uplink version
Release build
Version:            v1.116.7
Build timestamp:    08 Nov 24 23:46 CET
Git commit:         a94c0b66fa910ccab9e6c2c12ad7d0ea46d54783-dirty

PATH                      VERSION
storj.io/storj            (devel)
storj.io/common           v0.0.0-20241014122736-47dfa836934a
storj.io/drpc             v0.0.35-0.20240709171858-0075ac871661
storj.io/eventkit         v0.0.0-20240415002644-1d9596fee086
storj.io/infectious       v0.0.2
storj.io/monkit-jaeger    v0.0.0-20240221095020-52b0792fa6cd
storj.io/picobuf          v0.0.3
storj.io/uplink           v1.13.2-0.20240913115229-e8ce0e91b41f

And with the gateway-st I also tried version 1.9.0 and 1.10.0
It’s the same result everywhere with multiple host machines I run.

2 Likes
2

Seeing the same issue - folders are not showing up using uplink ls

root@REP-11-2-1:~# ./uplink version
Release build
Version:            v1.119.12
Build timestamp:    03 Jan 25 15:59 UTC
Git commit:         2e018b768bb7bb4acf9606ac100768bb63e9d16e-dirty
PATH                      VERSION
storj.io/storj            (devel)
storj.io/common           v0.0.0-20241216122958-b7b1937644d2
storj.io/drpc             v0.0.35-0.20240709171858-0075ac871661
storj.io/eventkit         v0.0.0-20240415002644-1d9596fee086
storj.io/infectious       v0.0.2
storj.io/monkit-jaeger    v0.0.0-20240221095020-52b0792fa6cd
storj.io/picobuf          v0.0.3
storj.io/uplink           v1.13.2-0.20241209213014-e5f3beed1a59

root@REP-11-2-1:~# ./uplink ls
CREATED                NAME
2024-10-02 01:37:41    backups
2024-10-02 01:37:42    backups-archive

root@REP-11-2-1:~# ./uplink ls sj://backups-archive/
KIND    CREATED    SIZE    KEY
PRE                        2024-12-31/
PRE                        2024-12-30/

root@REP-11-2-1:~# ./uplink ls sj://backups-archive/2025-01-07/
KIND    CREATED    SIZE    KEY
PRE                        BD_-_Backups_Copy_to_REP-11-2-1_and_Storj/

Our rclones suddenly started to upload the missing files/folders again @2100 CEST …

image
image793×329 43.1 KB

Don’t know if it’s a coincides, but my nodes suddenly has more ingress, around the same time our rclones started uploading again.

image
image658×302 22.3 KB

Th3Van.dk

3 Likes
3

I still see 2x ingress, @Alexey can you forward this client problem to devs, as it can be a big problem for clients

4

Same problem here, missing most of my folders when listing with either S3, native or webpage endpoint.

5

Question to all in this thread, did you folks recently also upgraded your gateway-st from 1.9.0 to 1.10.0 ??? Currently trying to figure out if it maybe has anything to do with this. I’m asking because I upgraded my gateway-st yesterday, and about 8 hrs. after, this file/folder listing issue started to happen …

6

It seems to be an issue from storj side. I contacted the support and the problem was solved.

1 Like
7

Same here, the StorJ support solved the issue also on my side.

1 Like
8

Hello @V_O92,
Welcome to the forum!

As @HighDoping is stated, we fixed the issue. We tried a new version with a more fast listing, but it was not work so well despite dozens tests, so currently this feature is disabled.
We would work on it to implement a better approach.

Sorry for inconveniency!

@Th3Van it was the issue related to listing, it’s returned not all prefixes/objects, thus your rclone jobs noticed a difference and tried to reupload what was already uploaded. If you would use the exact path you would see that the object is still there and untouched.

1 Like