After some successful testing with the free account, I recently moved to the paid pro tier and ran my initial uploads into three buckets. My primary use case is to backup data from my TrueNAS SCALE machine to Storj. I created S3 credentials with a passphrase and verified that when I upload files from TrueNAS using the S3 credentials, I can then log into the Storj DCS satellite website and view them using the encryption passphrase. I can also connect to gateway.storjshare.io using WinSCP and my S3 credentials to view files that I have uploaded.
So before anybody suggests it, yes I am using the correct passphrase when viewing the bucket on the website. I only have ever created and used one passphrase and one S3 credential.
Problem: There are some files I uploaded that do not show up when browsing the bucket via the Storj website (https://us1.storj.io/). But I can see them when connected via WinSCP.
In the last screenshot you can see several other files missing comparing WinSCP vs. Storj web view (I have sorted by name in reverse order in all screenshots). Is this a known issue? I need to be confident that files I upload are available and not hidden
Yes, the uploaded files do show up in both the uplink CLI and rclone, just not in the satellite admin console. Here are the PowerShell commands that show the file available with uplink CLI and rclone:
I’m looking into the issue, my knee jerk reaction is that its relative to passphrases and access grants. The S3 connection and the web ui are using different methods of authentication and the tools you are using to verify are appear to be dependent on the S3 connection. But you seem like you tried to run that down already and seem confident that you aren’t experiencing that issue.
I am going to do some testing and see if I can reproduce similar issues and generate a ticket as needed. In the interim can you verify that its not a sorting issue in the WebUI. It appears you have alot of files so you can use the search feature to narrow it down. Also can you check the consol in the browser and see if there are any errors or info messages that seem like they relate to the issue you are observing.
@Code_Breaker, yes all my files were uploaded from TrueNAS SCALE with a task that uses the same Storj S3 credentials. However, it remains that there are files missing in the web UI that are visible both using the S3 credential (via WinSCP and rclone) AND using uplink CLI with a Storj access grant. Here is another example of a file missing in the Storj Web UI (VID_20201107_162232740.mp4):
WinSCP connected to gateway.storjshare.io using the Storj S3 credential used for upload:
Uplink CLI connected using the access grant (NOT S3 credential):
It is true that both examples I have provided have many files in the “Camera” directory. In this example, the “Camera” directory has 1127 files/directories present.
In the above screenshots, I have sorted by filename in the same order in both WinSCP and the web UI. In the web UI if I search the page using CTRL-f for “VID_20201107_162232740.mp4” the file is not found.
Here is the console when I inspect the page in Chrome:
One other piece of info that might be relevant is how I created my buckets. I created my account via us1.storj.io and not the partner signup at ix.storj.io. Buckets created via the Storj web UI are not available to the TrueNAS SCALE Storj cloud sync task. But, I was able to create usable buckets from the TrueNAS SCALE gui that have the required attribution metadata (discussed here: TrueNAS backups and differences between Storj and Storj (iX)). So I take this to mean the buckets were created using the Storj S3 credentials, since this is how TrueNAS connects to Storj.
@ElectraFish thank you for reporting this issue! It looks like the Web UI is currently limited to showing 1000 objects. Additionally the object listing does not return a traditionally sorted list because keys are stored encrypted. This makes the bucket appear to be randomly missing objects that were not among the first 1000 encrypted keys.
I want to assure you that the files you have uploaded are available and are only missing from the Web UI because of the 1000 object limitation. We’re working on what to do about fixing this, I’m sorry for the trouble this has caused you. I’ll keep you updated here.
