Object storage provider for Mastodon instance

Andisers · January 28, 2021, 8:45am

Is it possible to use Tardigrade as a object storage provider?

Running your own server - Mastodon documentation

Optional: Object storage provider. Mastodon can save files that you and your users upload on the hard disk drive of the VPS it runs on, however, the hard disk drive is usually not infinite and difficult to upgrade later. An object storage provider gives you practically infinite metered file storage.

How to get: Amazon S3, Exoscale, Wasabi, Google Cloud, anything that exposes either an S3-compatible or OpenStack Swift-compatible API. Comes with a monthly cost based on the amount of files stored as well as how often they are accessed.

I would like to know if Tardigrade provides S3-compabible API.

Thanks in advance,

Pac · January 28, 2021, 12:25pm

I think that’s the whole point of Tardigrade.

https://documentation.tardigrade.io/api-reference/s3-gateway

Alexey · January 29, 2021, 6:17am

You also can use a Gateway MT:

In that case you will not need to host the Tardigrade S3 Gateway yourself, you can use a hosted version. But it’s beta and you can use only server-side encryption at the moment.

Andisers · January 29, 2021, 8:05am

Yeah that’s a totally stupid question. I was trying to ask where to find them on the documentation. My bad.

Andisers · February 10, 2021, 12:24pm

I cannot have a clue to set up the configuration properly on Mastodon yet.

I am digging around these pages, still no clue.

The related configs look to me:

    S3_ENABLED
    S3_BUCKET
    AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY
    S3_REGION
    S3_PROTOCOL
    S3_HOSTNAME
    S3_ENDPOINT
    S3_SIGNATURE_VERSION
    S3_OVERRIDE_PATH_STYLE
    S3_OPEN_TIMEOUT
    S3_READ_TIMEOUT

Would anyone please help me out? Thanks!

Alexey · February 12, 2021, 7:00pm

You need to configure these options:

    S3_ENABLED=true
    S3_BUCKET=mastodon
    AWS_ACCESS_KEY_ID=Awsekrijrifjeioiojrfojeorjfmoerf
    AWS_SECRET_ACCESS_KEY=43oij4nmfo4ifjorfm4i9jfio3j2i9i2djoi3jrfuihduiw
    S3_ENDPOINT=http://localhost:7777
    S3_PROTOCOL=http

Substitute your values. I assumed that you run a local Tardigrade S3 Gateway

Andisers · February 13, 2021, 3:01am

Thanks, I will try later

Andisers · February 14, 2021, 4:01pm

Hi, first I was going to set up gateway on https://documentation.tardigrade.io/api-reference/s3-gateway#set-up, but I cannot even set up the gateway because the documentation is very confusing. It looks to me that the procedure on the page is not updated or not well organized…

Should Set Up section be ignored (it was copied from the deprecated article, do we have the concept of API for initial setup yet?), and is Configure using an Access Grant the first step? Also, is Reconfiguring the AWS CLI to interface with the Tardigrade Network required in every case? Not quite sure what must be done and what may be done, at lease for me.

Also when you run gateway run after doing Configure using an Access Grant, you get the error below under Secret Key:

DeploymentID: fef77d00-69be-4507-b62b-1c2229caa56d
Error: WARNING: Expected number of all hosts (0) to be remote +1 (0) (*errors.errorString)
       4: /go/pkg/mod/github.com/minio/minio@v0.0.0-20201028162317-be7f67268d1e/cmd/peer-rest-client.go:866:cmd.newPeerRestClients()
       3: /go/pkg/mod/github.com/minio/minio@v0.0.0-20201028162317-be7f67268d1e/cmd/notification.go:1292:cmd.NewNotificationSys()
       2: /go/pkg/mod/github.com/minio/minio@v0.0.0-20201028162317-be7f67268d1e/cmd/server-main.go:160:cmd.newAllSubsystems()
       1: /go/pkg/mod/github.com/minio/minio@v0.0.0-20201028162317-be7f67268d1e/cmd/gateway-main.go:304:cmd.StartGateway()

Is it fine to ignore the error?

Thanks in advance,

Alexey · February 15, 2021, 5:59am

Thank you to point out to the wrong link for API keys. The usage of API keys is a deprecated mechanism and we want to remove it and leave only access grants.
However, while it is not removed you can configure the Tardigrade S3 Gateway in two different ways:

Setup Tardigrade S3 Gateway with an interactive setup, for that you will need a token, compatible with API key format and address of chosen satellite in format SatelliteID@hostname:port (or simple select a number of the satellite in the interactive menu).
The second way is to create an access grant and import it to the gateway.

This warning could be ignored, it doesn’t prevent the Gateway from working. We have this in the backlog, but it has a low priority.

Andisers · February 15, 2021, 6:08am

Thanks for the information! I will try again soon and let you know how it works out

Andisers · February 15, 2021, 12:24pm

I created another discussion to even setting up Gateway with Access Grant to begin with.

Gateway set up: Unauthorized API credentials

teutates · April 7, 2021, 1:02pm

Hello @Andisers ,

I’m curious about setting up Mastodon with Storj as file storage as well. Did you manage to setup a working Mastodon server with Storj?

I stumbled across this in the Mastodon docs, which made me wonder if it would work the same with the long share links of Storj out of the box? (My guess: probably not, modifications needed?)

In fact, if you are using external storage like Amazon S3, Minio or Google Cloud, you will by default be serving files from those services’ URLs.

Maybe it would work out of the box when using Storj in combination with something like goofys and letting Mastodon access the local directory managed by goofys like described here for Matrix?

jennifer · April 7, 2021, 3:14pm

hi @teutates - are you looking for shorter URLs for linksharing? We have made an update that will generate much shorter URLs. If not, could you give me more info on what you’re trying to accomplish?

teutates · April 8, 2021, 8:33pm

Thank you @jennifer! How nice, I totally missed that. I’m going to try this out and report back!

iks77 · November 24, 2022, 8:59am

I am currently also trying to use storj as Object Storage for Mastodon.
Unfortunately i can not access the page tardigate dot io (sorry, only 2 links for new users)

Do you really need some middle-man services to get this running?

Basially yesterday i was following these instructions that are made for Amazaon AWS cybrespace-meta/s3.md at 62ecf7dfc550e9025a49a3457a593ddce6e22b9f · cybrespace/cybrespace-meta · GitHub
I created a bucket and able to sync the files to bucket with aws-cli command line tool.
Configured Mastodon like mentioned in post above
S3_ENABLED=true
S3_BUCKET=test
AWS_ACCESS_KEY_ID=[blubb]
AWS_SECRET_ACCESS_KEY=[blubb]
S3_ENDPOINT=gateway dot storjshare dot io
S3_PROTOCOL=https

Mastodon now assumes the file can be accessed via HTTP on link like this: gateway.storjshare.io/test/cache/media_attachments/files/109/394/283/839/423/575/original/6f8d30f905e6ff79.jpeg (which is obviously not the case)

I think i am missing a basic thing, but i am a bit lost at this point. Maybe someone has a hint for a good tutorial or other ressources?

mitexleo · November 26, 2022, 4:49am

This forum seems dead … I can’t even send any support requests … That says I don’t have permission to access their support page …

mitexleo · November 26, 2022, 4:51am

I’m in same trouble too …

As far as I’ve been able to figure out, We need to run our own Gateway in order to get URLs … Which I already did … Then I have to setup Nginx Reverse Proxy … Which I’m trying to do now …

Alexey · November 26, 2022, 5:29am

Our GatewayMT (Storj-hosted S3-Compatible Gateway) doesn’t support globally public buckets, so you need to use either a GatewayST in the website mode and NGINX reverse-proxy to support the domain-like access (http://my-bucket.localhost:7777 instead of http://localhost:7777/my-bucket), or use a CDN like Fastly with our GatewayMT.

mitexleo · November 26, 2022, 5:42am

Is there any problem If I wanna stick with localhost:7777/my_bucket?

Alexey · November 26, 2022, 5:43am

no. Only if your software requires domain-like access instead of path-style and cannot be configured this way (Mastodon seems cannot).