How to add Storj to Nextcloud "as external storages"?

[kevink]

You follow the documentation until the point of installing nextcloud: https://documentation.tardigrade.io/how-tos/set-up-tardigrade-with-nextcloud-primary-storage

So you have your gateway running.
Then in nextcloud you configure your external storage like this:

But be aware that currently there is a bug preventing you from succesfully uploading files >64MB to nextcloud. We’ll have to wait until that one is fixed. Smaller files work fine and all other operations too.
But because of rate limiting, doing many uploads/downloads/deletes will only make parts of them succeed… Also have to wait for a proper rate limiting implementation here too.

So my advice currently is to only use it for testing and “proof-of-concept” but nothing else until those bugs are fixed.

[Andisers]

Thanks for you reply.

I managed to set up the external storage on my existing Nextcloud instance.

I then tried to upload a small file (“Hello world.txt”) from a browser, and it failed soon due to the error below.

An exception occurred while completing a multipart upload: Error executing “CompleteMultipartUpload” on “http://localhost:7777/sample/Hello%20world.txt?uploadId=Upload2”; AWS HTTP error: Server error: POST http://localhost:7777/sample/Hello%20world.txt?uploadId=Upload2 resulted in a 500 Internal Server Error response: <?xml version="1.0" encoding="UTF-8"?> InternalErrorWe encountered an internal error, pleas (truncated…) InternalError (server): We encountered an internal error, please try again. - <?xml version="1.0" encoding="UTF-8"?> InternalErrorWe encountered an internal error, please try again./sample/Hello world.txt3L1373L137

Would you please let me know how to solve the issue?

Thanks in advance,

[BrightSilence]

@kevink

Also have to wait for a proper rate limiting implementation here too.

Please don’t. Storjlabs seems to be basing their decisions on this rate limit on who requests an increase and encourages you to request that. So pleas go to support.tardigrade.io and click the new request button to ask for your rate limit to be increased if you run into this. They will need your feedback in order to determine the correct setting for everyone. If you wait for them to fix it, they also don’t get the signal that the current limit is too low. Don’t hesitate to ask for this increase on your account.

[Andisers]

Hi, I think that issue should be discussed on another discussion.

[Alexey]

Seems your gateway is down or inaccessible. Please, verify that either by a minio browser http://localhost:7777/ , or aws s3 CLI.
I think your gateway could be running on other PC rather than the PC with Nextcloud

[Andisers]

Hi @Alexey

Based on the post indicated above (https://documentation.tardigrade.io/how-tos/set-up-tardigrade-with-nextcloud-primary-storage) I set up gateway via SSH on the same machine (VPS) as Nextcloud is hosted.

Next I tried to add a file manually on the folder on Windows 10, and then on SSH the error below is returned:

2020-03-04T18:04:34.210+0900    ERROR   gateway error:  {"error": "Storj Gateway error: pending upload \"Upload1\" missing", "errorVerbose": "Storj Gateway error: pending upload \"Upload1\" missing\n\tstorj.io/storj/pkg/miniogw.(*MultipartUploads).Remove:230\n\tstorj.io/storj/pkg/miniogw.(*gatewayLayer).CompleteMultipartUpload:114\n\tstorj.io/storj/pkg/miniogw.(*layerLogging).CompleteMultipartUpload:147\n\tgithub.com/minio/minio/cmd.objectAPIHandlers.CompleteMultipartUploadHandler:1404\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:212\n\tgithub.com/minio/minio/cmd.securityHeaderHandler.ServeHTTP:657\n\tgithub.com/minio/minio/cmd.rateLimit.ServeHTTP:642\n\tgithub.com/minio/minio/cmd.pathValidityHandler.ServeHTTP:602\n\tgithub.com/minio/minio/cmd.httpStatsHandler.ServeHTTP:541\n\tgithub.com/minio/minio/cmd.requestSizeLimitHandler.ServeHTTP:66\n\tgithub.com/minio/minio/cmd.requestHeaderSizeLimitHandler.ServeHTTP:91\n\tgithub.com/minio/minio/cmd.crossDomainPolicy.ServeHTTP:51\n\tgithub.com/minio/minio/cmd.redirectHandler.ServeHTTP:243\n\tgithub.com/minio/minio/cmd.minioReservedBucketHandler.ServeHTTP:301\n\tgithub.com/minio/minio/cmd.cacheControlHandler.ServeHTTP:270\n\tgithub.com/minio/minio/cmd.timeValidityHandler.ServeHTTP:371\n\tgithub.com/rs/cors.(*Cors).Handler.func1:200\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tgithub.com/minio/minio/cmd.resourceHandler.ServeHTTP:492\n\tgithub.com/minio/minio/cmd.authHandler.ServeHTTP:300\n\tgithub.com/minio/minio/cmd.reservedMetadataHandler.ServeHTTP:134\n\tgithub.com/minio/minio/cmd/http.(*Server).Start.func1:108\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tnet/http.serverHandler.ServeHTTP:2802\n\tnet/http.(*conn).serve:1890"}

[kevink]

Ah yes I can reproduce this if you try to upload multiple files. Never tried that before. Guess I can add that to my already open github issue about Nextcloud.

[Andisers]

Mind sharing the URL of the issue on GitHub?

[kevink]

Nextcloud: AWS gateway fails uploading

[kevink]

Only weird thing is that I was trying to reproduce your first error and was only able to do that once…

[Andisers]

Hi there @Alexey, would you please check my post above about the error I still get?

How to add Storj to Nextcloud "as external storages"?

Hi @Alexey Based on the post indicated above (https://documentation.tardigrade.io/how-tos/set-up-tardigrade-with-nextcloud-primary-storage) I set up gateway via SSH on the same machine (VPS) as Nextcloud is hosted. Next I tried to add a file manually on the folder on Windows 10, and then on SSH the error below is returned: 2020-03-04T18:04:34.210+0900 ERROR gateway error: {"error": "Storj Gateway error: pending upload \"Upload1\" missing", "errorVerbose": "Storj Gateway error: pending u…

Thanks in advance,

[kevink]

The issues with nextcloud aren’t completely fixed yet. Just wait patiently. Alexey won’t be able to help either.

[Andisers]

I understand, thanks.

[jocelyn]

We’ve flagged this to the engineering team, and will post updates when we hear back - thank you!

[Andisers]

So basically the manual on https://documentation.tardigrade.io/how-tos/set-up-tardigrade-with-nextcloud-primary-storage has been completely rewritten and the steps look quite different than before-

Are steps explained there applicable for adding Tardigrade to Nextcloud as an external storage? Any help appreciated. Thanks,

[kevink]

Yeah this new connector and how-to is quite a mystery to me too…

So the “old way” is: set up the gateway on the machine where your nextcloud is running. Then in nextcloud set up an external aws3 storage, the config should look like the picture I posted a month ago: How to add Storj to Nextcloud "as external storages"?

Note that uploading files >64MB is still not working but everything else seems to be fine.

[Andisers]

yet since a couple of the recent versions gateway_linux_amd64.zip (which was available on 0.34.6, for example) is gone.

Nextcloud on Storj?

Hi! I managed to get it worked with an existing Nextcloud installation If you are interested in getting tardigrade as an external storage, just follow these steps: 1. Follow the official documentation until the step “Run the S3 gateway”, included. nb: there is an error in the command to run the gateway. It shouldn’t be “./uplink_linux_amd64 setup” but rather “./gateway_linux_amd64 setup” 2. Go to “Settings > Administration > External Storage” page 3. Add an S3 storage [im…

[kevink]

The gateway moved to a different repository: https://github.com/storj/gateway/releases

[Andisers]

Ah it looks like I got it working. Here are steps I did:

1 - Downloading files (uplink and gateway):

1. curl -L https://github.com/storj/gateway/releases/latest/download/gateway_linux_amd64.zip -o gateway_linux_amd64.zip && unzip gateway_linux_amd64.zip
2. curl -L https://github.com/storj/storj/releases/latest/download/uplink_linux_amd64.zip -o uplink_linux_amd64.zip && unzip uplink_linux_amd64.zip
3. chmod 755 gateway
4. chmod 755 uplink

2 - Setting up (uplink):

1. ./uplink setup
2. Create a bucket by running ./uplink mb sj://example (See /dcs/api-reference/uplink-cli/ for other commands like deleting/removing a bucket, listing files inside a bucket, etc.)

3 - Setting up (gateway):

1. screen (to let gateway run on another screen)
2. ./gateway setup (See /dcs/api-reference/s3-gateway/ for more detail)
3. ./gateway run
4. Make sure to copy Endpoint, Access key, and Secret key to somewhere safe (like a password manager)

4 - Setting up (Nextcloud):

1. Open External storages setting page
2. Input a folder name
3. Select Amazon S3 for External storage
4. Select Access key for Authentication type
5. Input example for Bucket (See above: 2.2)
6. Input 127.0.0.1 for Hostname (See above: 3.4)
7. Input 7777 for Port (See above: 3.4)
8. Leave Region blank
9. Check Enable Path Style
10. Paste your Access key (See above: 3.4)
11. Paste your Secret key (See above: 3.4)
12. Click three-dot-icon for advanced settings
13. Click check mark to complete the configuration

And the external folder should be available!

If you want to sync the folder to your local machine, open the Nextcloud client and follow the instruction displayed there.

Please make sure that syncing the external storage with other machines multiplies the bandwidth. Also storing a bunch of tiny files can be costly (See: Nextcloud on Storj? - #17 by kevink).

On Tardigrade there are three ways of calculating fees (Object Storage fee, Per Object Fee, Bandwidth Fee). I would recommend you to read https://documentation.tardigrade.io/pricing/billing-and-payment carefurelly before you move your files to the external folder on your Nextcloud instance.

While Storj have arrived at 1.0 the procedures written above are subject to change after future release. Please follow https://documentation.tardigrade.io/ periodically for available changes.

Also according to @kevink:

@kevink

Note that uploading files >64MB is still not working

[silaxe]

Hello!
I’m looking into linking my nextcloud server with storj dcs. Are the above steps still valid and/or is there an up-to-date howto to do so?
Thanks in advance!