Uplink ls and uplink ls --recrusive results are different

[stonematt]

Continuing the discussion from Delete nested directory (without S3 gateway):

@Alexey

You can use this command

uplink ls --recursive sj://duplicatibackups/myfiles/ | sed -E 's/(.* ){4}//g' | xargs -P64 -Ifullname uplink rm "sj://duplicatibackups/fullname"

@Alexey, This cool. I should have though of that (although a native rm --recursive would be awesome – I’ll look for a feature request forum).

I don’t quite understand the result set from uplink ls. I’m sure there’s a permission thing going on that I’m missing.

Check this out:

recursive – includes /backups folder

(base) StoneBLMBPro-15:~ $ uplink ls --recursive sj://stoneblmbpro-15/
OBJ 2020-09-18 05:33:34          979 /backups/duplicati-iee04b0cb5cf648e9bd81e9f631f90ec3.dindex.zip
OBJ 2020-09-15 09:38:52     52370612 /backups/duplicati-bcc289dcf4b834444b387d54c2eba87b3.dblock.zip
OBJ 2020-09-15 09:46:31     52418103 /backups/duplicati-bda14b82edd104bdbbeca3d052bc189a7.dblock.zip
<snip>
OBJ 2020-09-15 06:43:49       349792 /backups/duplicati-b97761f2576534d688a37ea8d8d0f59fb.dblock.zip
OBJ 2020-09-17 15:39:27         3043 testJunk-duplicati-config.json
OBJ 2020-09-14 15:59:57          554 storj_start

not recursive – no /backups folder listed

(base) StoneBLMBPro-15:~ $ uplink ls sj://stoneblmbpro-15/
PRE /
OBJ 2020-09-17 15:39:27         3043 testJunk-duplicati-config.json
OBJ 2020-09-14 15:59:57          554 storj_start

looking specifically at /backups

(base) StoneBLMBPro-15:~ $ uplink ls  sj://stoneblmbpro-15/backups/
(base) StoneBLMBPro-15:~ $ uplink ls --recursive sj://stoneblmbpro-15/backups/
(base) StoneBLMBPro-15:~ $

Any idea why I can’t see the contents of /backups/ unless I’m doing a total bucket list?

[nerdatwork]

@stonematt

no /backups folder listed

Create folder on storj

Storj is like AWS S3, where there are objects, and you can list objects with shared prefixes. They act like folders because the prefixes are often delimited with slashes, but there are no folder objects and they cannot be empty.

[Alexey]

You managed to create an empty prefix with only slash

@stonematt

PRE /

The normal prefix should not start with /.

[stonematt]

I ran a test that was really pedantic to see that I got behavior that I general expected (and I did --see below), I’m not sure how I need up with an PRE named /. It seems like I’m missing an PRE called /backups

Is there an uplink command to make that missing backups PRE?

Also, I tried listing contents of PRE / with uplink ls sj://stoneblmbpro-15// -> I didn’t expect it to work, and it didn’t. How do I get rid of it?

Test Run for my reference.
“normal” file ops seem to work like I expect

Put a file the bucket:

$ uplink cp uninstallD sj://testme

$ uplink ls sj://testme
OBJ 2020-09-26 08:20:20          490 uninstallD

Put a file in a bucket with a prefix (acts like a folder):

$ uplink cp test.txt sj://testme/afolder/test.txt
Created sj://testme/afolder/test.txt

List to see them both

$ uplink ls sj://testme
PRE afolder/
OBJ 2020-09-26 08:20:20          490 uninstallD

$ uplink ls --recursive sj://testme
OBJ 2020-09-26 08:21:19           61 afolder/test.txt
OBJ 2020-09-26 08:20:20          490 uninstallD

Put a couple more files with new and existing PRE

$ uplink cp plugin.html sj://testme/afolder/anotherfolder/plugin.html
Created sj://testme/afolder/anotherfolder/plugin.html

$ uplink cp wcplugin.html  sj://testme/bfolder/wcplugin.html
Created sj://testme/bfolder/wcplugin.html

List and see everything as expected

$ uplink ls sj://testme
PRE afolder/
PRE bfolder/
OBJ 2020-09-26 08:20:20          490 uninstallD

$ uplink ls sj://testme/afolder
PRE afolder/anotherfolder/
OBJ 2020-09-26 08:21:19           61 afolder/test.txt

$ uplink ls --recursive sj://testme/
OBJ 2020-09-26 08:22:59        42022 afolder/anotherfolder/plugin.html
OBJ 2020-09-26 08:21:19           61 afolder/test.txt
OBJ 2020-09-26 08:23:58        54566 bfolder/wcplugin.html
OBJ 2020-09-26 08:20:20          490 uninstallD

$ uplink ls --recursive sj://testme/afolder
OBJ 2020-09-26 08:22:59        42022 afolder/anotherfolder/plugin.html
OBJ 2020-09-26 08:21:19           61 afolder/test.txt

[Alexey]

I’m unable to create an empty prefix via uplink, seems it’s possible only via third party tools.
You can’t add a missed prefix, but you can upload data again with a right prefix.
Perhaps in the tool where you created a backup you specified a prefix started with /.

There is two ways how to solve this problem:

1. Setup an Tardigrade S3 Gateway
   1.1. Install aws cli
   1.2. Setup aws cli to use the Tardigrade S3 Gateway
   1.3. Operate with aws s3 --endpoint http://localhost:7777 with an empty prefix, i.e.

aws s3 --endpoint http://localhost:7777 ls s3://stoneblmbpro-15//

2. Remove the bucket and create a new one and upload with a right prefix

uplink rb --force sj://stoneblmbpro-15/
uplink mb sj://stoneblmbpro-15

I created an issue

There is no way to list or copy objects with an empty prefix

[stonematt]

thanks for directing me to --force I’ll do that.

I was setting up duplicati for backups to this bucket. It was successful (based on the list of objects above). I may have done something like use / as the folder path for the backup job:

I’m not positive b/c I have removed that backup job and I was doing some POC type work, and later decided to use /backups as a folder path.

[Alexey]

Then please, use a backup path instead of /backup.
Because the /backup has an empty prefix. The first / will be automatically added by library (or S3 Gateway) and your path will be sj://stoneblmbpro-15//backup instead of sj://stoneblmbpro-15/backup thus you will receive the same result with empty prefix.