My company are looking for a way to secure our databases (SQL, No-SQL) on DCS. But it seems that DCS isn’t well suited for transactional database. Any suggestion on how to utilize this?
Hello @tariusagi ,
Welcome to the forum!
I would say none of transactional databases are currently suited to use a remote storage, especially the object storage.
The main problem that our platform do not allow objects modification, you need to upload a whole object again if it were modified, you cannot modify it in place.
With other object storages you will have another problem - consistence, see amazon s3 - Mysql Data Directory on S3 - Stack Overflow
Our platform has consistence, but you cannot modify an object in place.
We are working on implementing a feature of the server-side copy of an object and perhaps this will allow to use modification in place.
For example, you can try to mount an object storage as a filesystem with rclone mount (see Sync Files With Rclone - Storj DCS) or with s3fs (see https://docs.storj.io/dcs/how-tos/how-to-connect-s3fs-to-storj-dcs), but the database engine will not work, because you cannot modify a file in place.
From the other side - if you use a local storage and do rclone sync to the bucket - this will work.
There are plenty of drivers supports S3. For example, you can use an S3-compatible protocol via JDBC/ODBC driver Amazon S3 Drivers - Online Documentation or Amazon S3 ODBC and JDBC Driver with SQL Connector | Simba - Magnitude or similar, however they are not tested.
To emulate S3 you can use either Storj-hosted S3 Compatible Gateway - Storj DCS or Self-hosted S3 Compatible Gateway - Storj DCS
Storj DCS has an S3 API, which can be used by e.g.:
- Using the S3 Storage Engine - MariaDB Knowledge Base
- S3-compatible · Apache Druid
- GitHub - pgspider/parquet_s3_fdw
As you can note, in all cases this is for read-only storage of data. The S3 protocol isn’t really designed for much more than that.
1 Like