Please help - Backing up database used for Rocket.Chat and uploading to Tardigrade.io

Developer Discussions DCS
1

Hello,

At a community project I introduced Rocket.Chat for hosting talk sessions.

I set up a Rocket.Chat instance with Docker, following the official manual here (https://rocket.chat/docs/installation/docker-containers/) and want to back up a database weekly and upload to Tardigrade automatically.

I am not familiar with either Docker or mongodb so basically I don’t know anything here at all and am all confused (also the instruction there is not quite sufficient… it does not explain how to create a backup).

What I would like to know is:

  • How to create a mongodb database backup inside these Docker containers (run a mongodb dump?)
  • How to set up to upload the backup periodically (write a cron?)

This is actually not quite related with Tardigrade itself… still I believe an instruction would help others too in this time, which also benefits SNOs as a result. Fund is already secured for my case.

My docker-compose.yaml was copied from: https://rocket.chat/docs/installation/docker-containers/#6-create-the-docker-composeyml-file--local-directories

Version: '2'

services:
  rocketchat:
    image: rocket.chat:latest
    command: bash -c 'for i in `seq 1 30`; do node main.js && s=$$? && break || s=$$?; echo "Tried $$i times. Waiting 5 secs..."; sleep 5; done; (exit $$s)'
    restart: unless-stopped
    volumes:
      - ./uploads:/app/uploads
    environment:
      - PORT=3000
      - ROOT_URL=http://chat.inumio.com
      - MONGO_URL=mongodb://mongo:27017/rocketchat
      - MONGO_OPLOG_URL=mongodb://mongo:27017/local
      - Accounts_UseDNSDomainCheck=True
    depends_on:
      - mongo
    ports:
      - 3000:3000

  mongo:
    image: mongo:4.0
    restart: unless-stopped
    volumes:
     - ./data/db:/data/db
     - ./data/dump:/dump
    command: mongod --smallfiles --oplogSize 128 --replSet rs0 --storageEngine=mmapv1

  # this container's job is just run the command to initialize the replica set.
  # it will run the command and remove himself (it will not stay running)
  mongo-init-replica:
    image: mongo
    command: 'bash -c "for i in `seq 1 30`; do mongo mongo/rocketchat --eval \"rs.initiate({ _id: ''rs0'', members: [ { _id: 0, host: ''localhost:27017'' } ]})\" && s=$$? && break || s=$$?; echo \"Tried $$i times. Waiting 5 secs...\"; sleep 5; done; (exit $$s)"'
    depends_on:
      - mongo
....

Any indication would be appreciated :pray:

Edit: I suspect that docker exec [$DOCKER_CONTAINER_ID] mongodump ... would run the command for backing up, still no clue.

2 Likes
2

There is actually a specific connector for MongoDB backups on Tardigrade. I haven’t used it myself, but I hope this link can get you started.
https://documentation.tardigrade.io/v/staging/how-tos/backup/mongodb-backup-to-tardigrade

3 Likes
3

Thank you very much for the link!

Would you let me know if it works for mongodb inside a Docker container?

Thanks in advance,

4

We are working on an integration with the MongoDB Ops Manager product to back up to the Tardigrade service. We will be providing details in the next few weeks.

4 Likes
5

That’s great to hear! Until the integration happens I am going to back up the database manually and wait for it :smile:

It would be very appreciated if you would release an automated script (or service?) as well :pray: I am looking forward to it!

1 Like
6

I managed to backup the mongodb database for Rocket.Chat on Docker. What I did on VM was:

sudo docker exec -t -i <your_mongodb_container_id> bash
mongodump --archive="mongodb_database_archive" --gzip
sudo docker cp <your_mongodb_container_id>:/mongodb_database_archive.gz mongodb_database_archive.gz

And then copy the compressed archive file to VM with scp:

scp <your_username_on_VM>@<your_VM's_ip_address>:/home/<your_username>/mongodb_database_archive.gz localcopy_mongodb_database_archive.gz

That’s it. I know it’s not elegant, still it works at least.

I wish that whole process would be automated by the integration with MongoDB :wink:

2 Likes
7

You also can use a mongoDB connector:

And I checked it exactly in the docker.
You also can link the docker container with modgoDB connector to the docker conatiner with MongoDB to have a direct connection to it.

Here is a Dockerfile for connector

FROM golang:1.14

RUN git clone https://github.com/utropicmedia/storj-mongodb && \
         cd storj-mongodb && \
         go get github.com/utropicmedia/storj-mongodb

COPY ./db_property.json ./config/
COPY ./storj_config.json ./config/

ENTRYPOINT ["storj-mongodb"]
CMD ["-h"]

Create needed json files: https://documentation.tardigrade.io/v/staging/how-tos/backup/mongodb-backup-to-tardigrade#set-up-files

Build the image:

docker build . -t storj-mongodb

Run it:

docker run -it --rm --link <your_mongodb_container_id> storj-mongodb store

Also, you can use the option --network <mongodb_network> instead of using the option --link <your_mongodb_container_id>

The network you can see from the command:

docker network ls
2 Likes
8

I noticed that it should work to run rsync the mounted volumes for persistent data storage and tar with gzip option, and then upload the tar.gz archive file with uplink. This way you would not have to connect storj with the docker container on which MongoDB runs, and be able to restore the database simply by downloading the archive with uplink and extracting it.

9

This is how I set up a bash script to run rsync to back up the docker’s volume and compress it with sudo crontab.

#!/bin/bash

date=$(date +%Y%m%d-%H.%M.%S)
home=/home/temp

# Stop Rocket.Chat container
docker-compose -f /var/www/rocket.chat/docker-compose.yml stop rocketchat

# Run rsync data folder
rsync -a /var/www/rocket.chat/data/ $home/rocketchat_data.bkp.$date && \

# Restart Rocket.Chat container
docker-compose -f /var/www/rocket.chat/docker-compose.yml up -d rocketchat

# Archive and zip the folder with max compression rate
tar -I 'gzip -9' -cvf $home/rocketchat_data.bkp.$date.tar.gz $home/rocketchat_data.bkp.$date/ && \

# Remove the folder
rm -r $home/rocketchat_data.bkp.$date && \

# Move the archive file inside another folder
mv $home/rocketchat_data.bkp.$date.tar.gz $home/rocketchat_data.bkp && \

# Upload the archive to Tardigrade.io
$home/uplink cp $home/rocketchat_data.bkp/rocketchat_data.bkp.$date.tar.gz sj://rocket.chat

# Remove archive files more than 60 days old
find $home/rocketchat_data.bkp/* -mtime +60 -exec rm -rf {} \;

This script removes anything but tar.gz files to save up data usage on your hard drive.

For me it uploads (only) ~300 MB daily to Tardigrade, and since two months it has costed me only under a dollar. Uploading is fast, data should be reliable (I haven’t tested downloading yet)…

The great experience with Tardigrade so far. SNOs and the development team, please keep up the nice work :smile:

4 Likes