How to add Storj to Nextcloud "as external storages"?

Developer Discussions FAQ
1

Hello, this is my first post here.

I have followed the projects since a couple of years, and I am considering to use Tardigrade on my existing Nextcloud instance as an external storage.

I have read the posts below and could not find how to do so.

https://forum.storj.io/t/storj-as-a-free-data-replication-tool-for-home-server-such-as-dappnode/2687/3

Implementing Nextcloud Using Storj - Storj Docs

I have set up and used the Nextcloud instance with Snap package on a remote server (Ubuntu on VPS).

I am willing to upload files from my local machine (Windows 10) via that VPS to Tardigrade.

Would anyone please let me know if it is possible, and if so, how?

Kind regards,

2

FYI: I have already created the account and made API key (I wrote it down). I downloaded the Uplink CLI from the GitHub release page too. My Satellite is Asia-East-1 and I don’t add my credit card or STORJ on my accout yet. I have never used Amazon S3 so I actually am not familiar with its concept or usage.

3

You follow the documentation until the point of installing nextcloud: https://documentation.tardigrade.io/how-tos/set-up-tardigrade-with-nextcloud-primary-storage

So you have your gateway running.
Then in nextcloud you configure your external storage like this:

grafik
grafik1348Ă—398 7 KB

But be aware that currently there is a bug preventing you from succesfully uploading files >64MB to nextcloud. We’ll have to wait until that one is fixed. Smaller files work fine and all other operations too.
But because of rate limiting, doing many uploads/downloads/deletes will only make parts of them succeed… Also have to wait for a proper rate limiting implementation here too.

So my advice currently is to only use it for testing and “proof-of-concept” but nothing else until those bugs are fixed.

1 Like
4

Thanks for you reply.

I managed to set up the external storage on my existing Nextcloud instance.

Clipboard01
Clipboard01709Ă—361 14.8 KB

I then tried to upload a small file (“Hello world.txt”) from a browser, and it failed soon due to the error below.

An exception occurred while completing a multipart upload: Error executing “CompleteMultipartUpload” on “http://localhost:7777/sample/Hello%20world.txt?uploadId=Upload2”; AWS HTTP error: Server error: POST http://localhost:7777/sample/Hello%20world.txt?uploadId=Upload2 resulted in a 500 Internal Server Error response: <?xml version="1.0" encoding="UTF-8"?> InternalErrorWe encountered an internal error, pleas (truncated…) InternalError (server): We encountered an internal error, please try again. - <?xml version="1.0" encoding="UTF-8"?> InternalErrorWe encountered an internal error, please try again./sample/Hello world.txt3L1373L137

Would you please let me know how to solve the issue?

Thanks in advance,

5

Please don’t. Storjlabs seems to be basing their decisions on this rate limit on who requests an increase and encourages you to request that. So pleas go to support.tardigrade.io and click the new request button to ask for your rate limit to be increased if you run into this. They will need your feedback in order to determine the correct setting for everyone. If you wait for them to fix it, they also don’t get the signal that the current limit is too low. Don’t hesitate to ask for this increase on your account.

1 Like
6

Hi, I think that issue should be discussed on another discussion.

7

Seems your gateway is down or inaccessible. Please, verify that either by a minio browser http://localhost:7777/ , or aws s3 CLI.
I think your gateway could be running on other PC rather than the PC with Nextcloud

8

Hi @Alexey

Based on the post indicated above (https://documentation.tardigrade.io/how-tos/set-up-tardigrade-with-nextcloud-primary-storage) I set up gateway via SSH on the same machine (VPS) as Nextcloud is hosted.

Next I tried to add a file manually on the folder on Windows 10, and then on SSH the error below is returned:

2020-03-04T18:04:34.210+0900    ERROR   gateway error:  {"error": "Storj Gateway error: pending upload \"Upload1\" missing", "errorVerbose": "Storj Gateway error: pending upload \"Upload1\" missing\n\tstorj.io/storj/pkg/miniogw.(*MultipartUploads).Remove:230\n\tstorj.io/storj/pkg/miniogw.(*gatewayLayer).CompleteMultipartUpload:114\n\tstorj.io/storj/pkg/miniogw.(*layerLogging).CompleteMultipartUpload:147\n\tgithub.com/minio/minio/cmd.objectAPIHandlers.CompleteMultipartUploadHandler:1404\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:212\n\tgithub.com/minio/minio/cmd.securityHeaderHandler.ServeHTTP:657\n\tgithub.com/minio/minio/cmd.rateLimit.ServeHTTP:642\n\tgithub.com/minio/minio/cmd.pathValidityHandler.ServeHTTP:602\n\tgithub.com/minio/minio/cmd.httpStatsHandler.ServeHTTP:541\n\tgithub.com/minio/minio/cmd.requestSizeLimitHandler.ServeHTTP:66\n\tgithub.com/minio/minio/cmd.requestHeaderSizeLimitHandler.ServeHTTP:91\n\tgithub.com/minio/minio/cmd.crossDomainPolicy.ServeHTTP:51\n\tgithub.com/minio/minio/cmd.redirectHandler.ServeHTTP:243\n\tgithub.com/minio/minio/cmd.minioReservedBucketHandler.ServeHTTP:301\n\tgithub.com/minio/minio/cmd.cacheControlHandler.ServeHTTP:270\n\tgithub.com/minio/minio/cmd.timeValidityHandler.ServeHTTP:371\n\tgithub.com/rs/cors.(*Cors).Handler.func1:200\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tgithub.com/minio/minio/cmd.resourceHandler.ServeHTTP:492\n\tgithub.com/minio/minio/cmd.authHandler.ServeHTTP:300\n\tgithub.com/minio/minio/cmd.reservedMetadataHandler.ServeHTTP:134\n\tgithub.com/minio/minio/cmd/http.(*Server).Start.func1:108\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tnet/http.serverHandler.ServeHTTP:2802\n\tnet/http.(*conn).serve:1890"}

Clipboard01

9

Ah yes I can reproduce this if you try to upload multiple files. Never tried that before. Guess I can add that to my already open github issue about Nextcloud.

1 Like
10

Mind sharing the URL of the issue on GitHub?

11
12

Only weird thing is that I was trying to reproduce your first error and was only able to do that once…

13

Hi there @Alexey, would you please check my post above about the error I still get?

Thanks in advance,

14

The issues with nextcloud aren’t completely fixed yet. Just wait patiently. Alexey won’t be able to help either.

1 Like
15

I understand, thanks.

16

We’ve flagged this to the engineering team, and will post updates when we hear back - thank you!

1 Like
17

So basically the manual on https://documentation.tardigrade.io/how-tos/set-up-tardigrade-with-nextcloud-primary-storage has been completely rewritten and the steps look quite different than before-

Are steps explained there applicable for adding Tardigrade to Nextcloud as an external storage? Any help appreciated. Thanks,

18

Yeah this new connector and how-to is quite a mystery to me too…

So the “old way” is: set up the gateway on the machine where your nextcloud is running. Then in nextcloud set up an external aws3 storage, the config should look like the picture I posted a month ago: How to add Storj to Nextcloud "as external storages"?

Note that uploading files >64MB is still not working but everything else seems to be fine.

19

yet since a couple of the recent versions gateway_linux_amd64.zip (which was available on 0.34.6, for example) is gone.

20

The gateway moved to a different repository: https://github.com/storj/gateway/releases

1 Like