Seems your gateway is down or inaccessible. Please, verify that either by a minio browser http://localhost:7777/ , or aws s3
CLI.
I think your gateway could be running on other PC rather than the PC with Nextcloud
Hi @Alexey
Based on the post indicated above (https://documentation.tardigrade.io/how-tos/set-up-tardigrade-with-nextcloud-primary-storage) I set up gateway via SSH on the same machine (VPS) as Nextcloud is hosted.
Next I tried to add a file manually on the folder on Windows 10, and then on SSH the error below is returned:
2020-03-04T18:04:34.210+0900 ERROR gateway error: {"error": "Storj Gateway error: pending upload \"Upload1\" missing", "errorVerbose": "Storj Gateway error: pending upload \"Upload1\" missing\n\tstorj.io/storj/pkg/miniogw.(*MultipartUploads).Remove:230\n\tstorj.io/storj/pkg/miniogw.(*gatewayLayer).CompleteMultipartUpload:114\n\tstorj.io/storj/pkg/miniogw.(*layerLogging).CompleteMultipartUpload:147\n\tgithub.com/minio/minio/cmd.objectAPIHandlers.CompleteMultipartUploadHandler:1404\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:212\n\tgithub.com/minio/minio/cmd.securityHeaderHandler.ServeHTTP:657\n\tgithub.com/minio/minio/cmd.rateLimit.ServeHTTP:642\n\tgithub.com/minio/minio/cmd.pathValidityHandler.ServeHTTP:602\n\tgithub.com/minio/minio/cmd.httpStatsHandler.ServeHTTP:541\n\tgithub.com/minio/minio/cmd.requestSizeLimitHandler.ServeHTTP:66\n\tgithub.com/minio/minio/cmd.requestHeaderSizeLimitHandler.ServeHTTP:91\n\tgithub.com/minio/minio/cmd.crossDomainPolicy.ServeHTTP:51\n\tgithub.com/minio/minio/cmd.redirectHandler.ServeHTTP:243\n\tgithub.com/minio/minio/cmd.minioReservedBucketHandler.ServeHTTP:301\n\tgithub.com/minio/minio/cmd.cacheControlHandler.ServeHTTP:270\n\tgithub.com/minio/minio/cmd.timeValidityHandler.ServeHTTP:371\n\tgithub.com/rs/cors.(*Cors).Handler.func1:200\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tgithub.com/minio/minio/cmd.resourceHandler.ServeHTTP:492\n\tgithub.com/minio/minio/cmd.authHandler.ServeHTTP:300\n\tgithub.com/minio/minio/cmd.reservedMetadataHandler.ServeHTTP:134\n\tgithub.com/minio/minio/cmd/http.(*Server).Start.func1:108\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tnet/http.serverHandler.ServeHTTP:2802\n\tnet/http.(*conn).serve:1890"}
Ah yes I can reproduce this if you try to upload multiple files. Never tried that before. Guess I can add that to my already open github issue about Nextcloud.
Mind sharing the URL of the issue on GitHub?
Only weird thing is that I was trying to reproduce your first error and was only able to do that once…
Hi there @Alexey, would you please check my post above about the error I still get?
Thanks in advance,
The issues with nextcloud aren’t completely fixed yet. Just wait patiently. Alexey won’t be able to help either.
I understand, thanks.
We’ve flagged this to the engineering team, and will post updates when we hear back - thank you!
So basically the manual on https://documentation.tardigrade.io/how-tos/set-up-tardigrade-with-nextcloud-primary-storage has been completely rewritten and the steps look quite different than before-
Are steps explained there applicable for adding Tardigrade to Nextcloud as an external storage? Any help appreciated. Thanks,
Yeah this new connector and how-to is quite a mystery to me too…
So the “old way” is: set up the gateway on the machine where your nextcloud is running. Then in nextcloud set up an external aws3 storage, the config should look like the picture I posted a month ago: How to add Storj to Nextcloud "as external storages"?
Note that uploading files >64MB is still not working but everything else seems to be fine.
yet since a couple of the recent versions gateway_linux_amd64.zip
(which was available on 0.34.6, for example) is gone.
Ah it looks like I got it working. Here are steps I did:
1 - Downloading files (uplink and gateway):
curl -L https://github.com/storj/gateway/releases/latest/download/gateway_linux_amd64.zip -o gateway_linux_amd64.zip && unzip gateway_linux_amd64.zip
curl -L https://github.com/storj/storj/releases/latest/download/uplink_linux_amd64.zip -o uplink_linux_amd64.zip && unzip uplink_linux_amd64.zip
chmod 755 gateway
chmod 755 uplink
2 - Setting up (uplink):
./uplink setup
- Create a bucket by running
./uplink mb sj://example
(See /dcs/api-reference/uplink-cli/ for other commands like deleting/removing a bucket, listing files inside a bucket, etc.)
3 - Setting up (gateway):
-
screen
(to let gateway run on another screen) -
./gateway setup
(See /dcs/api-reference/s3-gateway/ for more detail) ./gateway run
- Make sure to copy
Endpoint
,Access key
, andSecret key
to somewhere safe (like a password manager)
4 - Setting up (Nextcloud):
- Open External storages setting page
- Input a folder name
- Select
Amazon S3
forExternal storage
- Select
Access key
forAuthentication type
- Input
example
forBucket
(See above: 2.2) - Input
127.0.0.1
forHostname
(See above: 3.4) - Input
7777
forPort
(See above: 3.4) - Leave
Region
blank - Check
Enable Path Style
- Paste your
Access key
(See above: 3.4) - Paste your
Secret key
(See above: 3.4) - Click three-dot-icon for advanced settings
- Click check mark to complete the configuration
And the external folder should be available!
If you want to sync the folder to your local machine, open the Nextcloud client and follow the instruction displayed there.
Please make sure that syncing the external storage with other machines multiplies the bandwidth. Also storing a bunch of tiny files can be costly (See: Nextcloud on Storj? - #17 by kevink).
On Tardigrade there are three ways of calculating fees (Object Storage fee, Per Object Fee, Bandwidth Fee). I would recommend you to read https://documentation.tardigrade.io/pricing/billing-and-payment carefurelly before you move your files to the external folder on your Nextcloud instance.
While Storj have arrived at 1.0 the procedures written above are subject to change after future release. Please follow https://documentation.tardigrade.io/ periodically for available changes.
Also according to @kevink:
Hello!
I’m looking into linking my nextcloud server with storj dcs. Are the above steps still valid and/or is there an up-to-date howto to do so?
Thanks in advance!
Hi @silaxe nobody has told me anythin about an update to those instructions. Therefore, I think they should still work. What happens when you step through it? thanks!
Thanks Jocelyn! I will do this in the next days and will get back to you with my feedback.
Hi all,
I have managed to set up storj as external storage for nextcloud thanks, among others, to Andisers’ explanations. I am using it for backup purposes.
Is anyone using storj for nextcloud as primary storage? I’d be interested to do this in the long run but would like to know first how fast and reliable this option is.
Thanks!
I’ve worked on an app for Nextcloud so that you don’t need the S3 gateway. You can install it through the app store Storj - Apps - App Store - Nextcloud
Perhaps you can give it a try and see if you like it.
Is anyone using storj for nextcloud as primary storage? I’d be interested to do this in the long run but would like to know first how fast and reliable this option is.
I have not done a thorough comparison but I have some decent guesses
- You’ll use more Storj bandwidth from things like tumbnails and generated assets and whatever apps decide to put there.
- File browser will be faster because all the metadata is in the local SQL database
- Other parts of Nextcloud may be slower.
- Less chance of bugs because Nextcloud is treating Storj as a simple key-value store rather than as a file system.
It doesn’t sound like the best fit for Storj but should work fine.