I want to propose an idea for improving the pmoxs3backuproxy project, which I’ve submitted as a feature request in Issue #67. This project enables the use of S3-compatible storage for backing up virtual machines and containers in Proxmox. My suggestion is to add native support for Storj as one of the backend options.
Why is this worth implementing?
Proxmox’s Popularity: Proxmox VE is one of the most widely used free tools for managing virtual machines and containers. Integrating Storj as a native backend for backups would expose Storj to a large and active user base, potentially driving adoption.
Increased Usage of the Storj Network: Supporting Storj in this tool could significantly boost data stored on the network by enabling Proxmox users to leverage decentralized storage for their backup needs. This would generate additional demand for the Storj ecosystem.
Organic Promotion for Storj: Being natively supported in a widely used project like this would serve as a form of organic advertising. Proxmox users would become more aware of Storj’s benefits, such as reliability, security, and cost-effectiveness.
Technical Benefits: Developing this integration could also help identify and address potential performance challenges when using Storj for tasks like handling large numbers of small files—common in backup scenarios—thereby improving its functionality for similar use cases.
If anyone from the development team or community is interested in contributing to this idea, it would be a valuable enhancement for both the pmoxs3backuproxy project and the growth of the Storj network. Let’s collaborate to make this happen!
Storj is already continuously improving their S3-api support (see their announcements about object-lock). So when you say “native support” are you talking about using the lower Storj-specific APIs (and moving away from S3)? And would you be doing that for… maybe better performance than using the S3 layer?
Or are you asking to support Proxmox internals: so it doesn’t have to use S3? Or for the Storj Satellites to directly support PBS emulation?
Lots of options!
Right now the S3 layer is working well (even having Veeam ‘Ready for Object’ status) - but if PBS users would pay to bypass S3 maybe there is a market for it. Cool idea!
As Storj determined very early: few customers want to integrate with an API used by a single vendor. Especially in a market as commoditized as object storage. Without S3 support… they would have failed as a business.
It may be cheaper for them to promote native APIs, but it’s not what the market wants to buy.
When I refer to “native support,” I mean enabling the pmoxs3backuproxy project to connect directly to the Storj network using the uplink library, bypassing the S3 compatibility layer. This approach would allow the proxy to fully leverage the unique benefits of Storj, such as:
Decentralization: Directly interacting with the Storj network ensures data is distributed across nodes without relying on a centralized S3 gateway.
Performance: By avoiding the S3 layer, we can potentially reduce overhead and improve performance, especially for operations like handling small files or high-frequency backups.
Cost Efficiency: Eliminating the S3 layer could reduce costs associated with API compatibility while maximizing Storj’s inherent economic advantages.
To address your other points:
I am not suggesting implementing PBS (Proxmox Backup Server) support directly on Storj Satellites. Instead, my proposal is focused on enhancing pmoxs3backuproxy to work natively with Storj via uplink.
This would not replace S3 support but rather add an additional option for users who want to directly utilize Storj’s decentralized architecture.
While the S3 layer works well and is a great option for many use cases (as evidenced by Veeam’s certification), a direct integration using uplink could unlock further benefits for Proxmox users who value performance, decentralization, and cost savings.
I believe this would make pmoxs3backuproxy even more versatile and attractive, especially for users already invested in Storj’s ecosystem. Thank you again for considering this idea!
As a cloud backup user I prefer to stick with the de facto standard S3. Only a significant performance gain could make me using something else, but does it really exist here?
This is true. The problem is that an S3-compatible API is fundamentally misaligned with the Storj architecture.
What might make sense is to allow customers to self-host the S3 gateway, so that they still get S3 compatibility, but the additional cost isn’t on Storj.
If the Storj S3 Gateway is open-source, and it speaks uplink on its back-end: then wouldn’t it be working sample-code for the pmoxs3backuproxy developers?
I guess you’re… asking OP, not me? Though I do also wonder why devs working on pmoxs3backuproxy as a S3 shim… would spend time adding support for a non-S3 backend?
It would be nice addition and it could make a concurrency for
which is already integrated with Storj (of course via S3, because Object lock and Object versioning are S3 features, they are not available for uplink yet as far as I know).