Object storage provider for Mastodon instance

Developer Discussions DCS
#1

Is it possible to use Tardigrade as a object storage provider?

https://docs.joinmastodon.org/user/run-your-own/#so-you-want-to-run-your-own-mastodon-server

Optional: Object storage provider. Mastodon can save files that you and your users upload on the hard disk drive of the VPS it runs on, however, the hard disk drive is usually not infinite and difficult to upgrade later. An object storage provider gives you practically infinite metered file storage.

How to get: Amazon S3, Exoscale, Wasabi, Google Cloud, anything that exposes either an S3-compatible or OpenStack Swift-compatible API. Comes with a monthly cost based on the amount of files stored as well as how often they are accessed.

I would like to know if Tardigrade provides S3-compabible API.

Thanks in advance,

#2

I think that’s the whole point of Tardigrade.

https://documentation.tardigrade.io/api-reference/s3-gateway

#3

You also can use a Gateway MT:


In that case you will not need to host the Tardigrade S3 Gateway yourself, you can use a hosted version. But it’s beta and you can use only server-side encryption at the moment.

1 Like
#4

Yeah that’s a totally stupid question. I was trying to ask where to find them on the documentation. My bad.

#5

I cannot have a clue to set up the configuration properly on Mastodon yet.

I am digging around these pages, still no clue.

The related configs look to me:

    S3_ENABLED
    S3_BUCKET
    AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY
    S3_REGION
    S3_PROTOCOL
    S3_HOSTNAME
    S3_ENDPOINT
    S3_SIGNATURE_VERSION
    S3_OVERRIDE_PATH_STYLE
    S3_OPEN_TIMEOUT
    S3_READ_TIMEOUT

Would anyone please help me out? Thanks!

#6

You need to configure these options:

    S3_ENABLED=true
    S3_BUCKET=mastodon
    AWS_ACCESS_KEY_ID=Awsekrijrifjeioiojrfojeorjfmoerf
    AWS_SECRET_ACCESS_KEY=43oij4nmfo4ifjorfm4i9jfio3j2i9i2djoi3jrfuihduiw
    S3_ENDPOINT=http://localhost:7777
    S3_PROTOCOL=http

Substitute your values. I assumed that you run a local Tardigrade S3 Gateway

1 Like
#7

Thanks, I will try later :smile:

#8

Hi, first I was going to set up gateway on https://documentation.tardigrade.io/api-reference/s3-gateway#set-up, but I cannot even set up the gateway because the documentation is very confusing. It looks to me that the procedure on the page is not updated or not well organized…

Should Set Up section be ignored (it was copied from the deprecated article, do we have the concept of API for initial setup yet?), and is Configure using an Access Grant the first step? Also, is Reconfiguring the AWS CLI to interface with the Tardigrade Network required in every case? Not quite sure what must be done and what may be done, at lease for me.

Also when you run gateway run after doing Configure using an Access Grant, you get the error below under Secret Key:

DeploymentID: fef77d00-69be-4507-b62b-1c2229caa56d
Error: WARNING: Expected number of all hosts (0) to be remote +1 (0) (*errors.errorString)
       4: /go/pkg/mod/github.com/minio/minio@v0.0.0-20201028162317-be7f67268d1e/cmd/peer-rest-client.go:866:cmd.newPeerRestClients()
       3: /go/pkg/mod/github.com/minio/minio@v0.0.0-20201028162317-be7f67268d1e/cmd/notification.go:1292:cmd.NewNotificationSys()
       2: /go/pkg/mod/github.com/minio/minio@v0.0.0-20201028162317-be7f67268d1e/cmd/server-main.go:160:cmd.newAllSubsystems()
       1: /go/pkg/mod/github.com/minio/minio@v0.0.0-20201028162317-be7f67268d1e/cmd/gateway-main.go:304:cmd.StartGateway()

Is it fine to ignore the error?

Thanks in advance,

#9

Thank you to point out to the wrong link for API keys. The usage of API keys is a deprecated mechanism and we want to remove it and leave only access grants.
However, while it is not removed you can configure the Tardigrade S3 Gateway in two different ways:

  1. Setup Tardigrade S3 Gateway with an interactive setup, for that you will need a token, compatible with API key format and address of chosen satellite in format SatelliteID@hostname:port (or simple select a number of the satellite in the interactive menu).
  2. The second way is to create an access grant and import it to the gateway.

This warning could be ignored, it doesn’t prevent the Gateway from working. We have this in the backlog, but it has a low priority.

1 Like
Known bug - Unauthorized API credentials
#10

Thanks for the information! I will try again soon and let you know how it works out :grinning:

1 Like
#11

I created another discussion to even setting up Gateway with Access Grant to begin with.

Gateway set up: Unauthorized API credentials

#12

Hello @Andisers ,

I’m curious about setting up Mastodon with Storj as file storage as well. Did you manage to setup a working Mastodon server with Storj?

I stumbled across this in the Mastodon docs, which made me wonder if it would work the same with the long share links of Storj out of the box? (My guess: probably not, modifications needed?)

In fact, if you are using external storage like Amazon S3, Minio or Google Cloud, you will by default be serving files from those services’ URLs.

Maybe it would work out of the box when using Storj in combination with something like goofys and letting Mastodon access the local directory managed by goofys like described here for Matrix?

#13

hi @teutates - are you looking for shorter URLs for linksharing? We have made an update that will generate much shorter URLs. If not, could you give me more info on what you’re trying to accomplish?

1 Like
#14

Thank you @jennifer! How nice, I totally missed that. I’m going to try this out and report back!

1 Like