How to add Storj to Nextcloud "as external storages"?

Developer Discussions FAQ
7

Seems your gateway is down or inaccessible. Please, verify that either by a minio browser http://localhost:7777/ , or aws s3 CLI.
I think your gateway could be running on other PC rather than the PC with Nextcloud

8

Hi @Alexey

Based on the post indicated above (https://documentation.tardigrade.io/how-tos/set-up-tardigrade-with-nextcloud-primary-storage) I set up gateway via SSH on the same machine (VPS) as Nextcloud is hosted.

Next I tried to add a file manually on the folder on Windows 10, and then on SSH the error below is returned:

2020-03-04T18:04:34.210+0900    ERROR   gateway error:  {"error": "Storj Gateway error: pending upload \"Upload1\" missing", "errorVerbose": "Storj Gateway error: pending upload \"Upload1\" missing\n\tstorj.io/storj/pkg/miniogw.(*MultipartUploads).Remove:230\n\tstorj.io/storj/pkg/miniogw.(*gatewayLayer).CompleteMultipartUpload:114\n\tstorj.io/storj/pkg/miniogw.(*layerLogging).CompleteMultipartUpload:147\n\tgithub.com/minio/minio/cmd.objectAPIHandlers.CompleteMultipartUploadHandler:1404\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:212\n\tgithub.com/minio/minio/cmd.securityHeaderHandler.ServeHTTP:657\n\tgithub.com/minio/minio/cmd.rateLimit.ServeHTTP:642\n\tgithub.com/minio/minio/cmd.pathValidityHandler.ServeHTTP:602\n\tgithub.com/minio/minio/cmd.httpStatsHandler.ServeHTTP:541\n\tgithub.com/minio/minio/cmd.requestSizeLimitHandler.ServeHTTP:66\n\tgithub.com/minio/minio/cmd.requestHeaderSizeLimitHandler.ServeHTTP:91\n\tgithub.com/minio/minio/cmd.crossDomainPolicy.ServeHTTP:51\n\tgithub.com/minio/minio/cmd.redirectHandler.ServeHTTP:243\n\tgithub.com/minio/minio/cmd.minioReservedBucketHandler.ServeHTTP:301\n\tgithub.com/minio/minio/cmd.cacheControlHandler.ServeHTTP:270\n\tgithub.com/minio/minio/cmd.timeValidityHandler.ServeHTTP:371\n\tgithub.com/rs/cors.(*Cors).Handler.func1:200\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tgithub.com/minio/minio/cmd.resourceHandler.ServeHTTP:492\n\tgithub.com/minio/minio/cmd.authHandler.ServeHTTP:300\n\tgithub.com/minio/minio/cmd.reservedMetadataHandler.ServeHTTP:134\n\tgithub.com/minio/minio/cmd/http.(*Server).Start.func1:108\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tnet/http.serverHandler.ServeHTTP:2802\n\tnet/http.(*conn).serve:1890"}

Clipboard01

9

Ah yes I can reproduce this if you try to upload multiple files. Never tried that before. Guess I can add that to my already open github issue about Nextcloud.

1 Like
10

Mind sharing the URL of the issue on GitHub?

11
12

Only weird thing is that I was trying to reproduce your first error and was only able to do that once…

13

Hi there @Alexey, would you please check my post above about the error I still get?

Thanks in advance,

14

The issues with nextcloud aren’t completely fixed yet. Just wait patiently. Alexey won’t be able to help either.

1 Like
15

I understand, thanks.

16

We’ve flagged this to the engineering team, and will post updates when we hear back - thank you!

1 Like
17

So basically the manual on https://documentation.tardigrade.io/how-tos/set-up-tardigrade-with-nextcloud-primary-storage has been completely rewritten and the steps look quite different than before-

Are steps explained there applicable for adding Tardigrade to Nextcloud as an external storage? Any help appreciated. Thanks,

18

Yeah this new connector and how-to is quite a mystery to me too…

So the “old way” is: set up the gateway on the machine where your nextcloud is running. Then in nextcloud set up an external aws3 storage, the config should look like the picture I posted a month ago: How to add Storj to Nextcloud "as external storages"?

Note that uploading files >64MB is still not working but everything else seems to be fine.

19

yet since a couple of the recent versions gateway_linux_amd64.zip (which was available on 0.34.6, for example) is gone.

20

The gateway moved to a different repository: https://github.com/storj/gateway/releases

1 Like
21

Ah it looks like I got it working. Here are steps I did:

1 - Downloading files (uplink and gateway):

  1. curl -L https://github.com/storj/gateway/releases/latest/download/gateway_linux_amd64.zip -o gateway_linux_amd64.zip && unzip gateway_linux_amd64.zip
  2. curl -L https://github.com/storj/storj/releases/latest/download/uplink_linux_amd64.zip -o uplink_linux_amd64.zip && unzip uplink_linux_amd64.zip
  3. chmod 755 gateway
  4. chmod 755 uplink

2 - Setting up (uplink):

  1. ./uplink setup
  2. Create a bucket by running ./uplink mb sj://example (See /dcs/api-reference/uplink-cli/ for other commands like deleting/removing a bucket, listing files inside a bucket, etc.)

3 - Setting up (gateway):

  1. screen (to let gateway run on another screen)
  2. ./gateway setup (See /dcs/api-reference/s3-gateway/ for more detail)
  3. ./gateway run
  4. Make sure to copy Endpoint, Access key, and Secret key to somewhere safe (like a password manager)

4 - Setting up (Nextcloud):

  1. Open External storages setting page
  2. Input a folder name
  3. Select Amazon S3 for External storage
  4. Select Access key for Authentication type
  5. Input example for Bucket (See above: 2.2)
  6. Input 127.0.0.1 for Hostname (See above: 3.4)
  7. Input 7777 for Port (See above: 3.4)
  8. Leave Region blank
  9. Check Enable Path Style
  10. Paste your Access key (See above: 3.4)
  11. Paste your Secret key (See above: 3.4)
  12. Click three-dot-icon for advanced settings
  13. Click check mark to complete the configuration

And the external folder should be available!

If you want to sync the folder to your local machine, open the Nextcloud client and follow the instruction displayed there.

Please make sure that syncing the external storage with other machines multiplies the bandwidth. Also storing a bunch of tiny files can be costly (See: Nextcloud on Storj? - #17 by kevink).

On Tardigrade there are three ways of calculating fees (Object Storage fee, Per Object Fee, Bandwidth Fee). I would recommend you to read https://documentation.tardigrade.io/pricing/billing-and-payment carefurelly before you move your files to the external folder on your Nextcloud instance.

While Storj have arrived at 1.0 the procedures written above are subject to change after future release. Please follow https://documentation.tardigrade.io/ periodically for available changes.

Also according to @kevink:

1 Like
22

Hello!
I’m looking into linking my nextcloud server with storj dcs. Are the above steps still valid and/or is there an up-to-date howto to do so?
Thanks in advance!

23

Hi @silaxe nobody has told me anythin about an update to those instructions. Therefore, I think they should still work. What happens when you step through it? thanks!

1 Like
24

Thanks Jocelyn! I will do this in the next days and will get back to you with my feedback.

2 Likes
25

Hi all,
I have managed to set up storj as external storage for nextcloud thanks, among others, to Andisers’ explanations. I am using it for backup purposes.
Is anyone using storj for nextcloud as primary storage? I’d be interested to do this in the long run but would like to know first how fast and reliable this option is.
Thanks!

26

I’ve worked on an app for Nextcloud so that you don’t need the S3 gateway. You can install it through the app store Storj - Apps - App Store - Nextcloud

Perhaps you can give it a try and see if you like it.

Is anyone using storj for nextcloud as primary storage? I’d be interested to do this in the long run but would like to know first how fast and reliable this option is.

I have not done a thorough comparison but I have some decent guesses

  • You’ll use more Storj bandwidth from things like tumbnails and generated assets and whatever apps decide to put there.
  • File browser will be faster because all the metadata is in the local SQL database
  • Other parts of Nextcloud may be slower.
  • Less chance of bugs because Nextcloud is treating Storj as a simple key-value store rather than as a file system.

It doesn’t sound like the best fit for Storj but should work fine.

4 Likes