The business also relies on Railway, a cloud infrastructure provider that is generally regarded to be ‘friendlier’ than the likes of AWS. However, Crane reckons this pair created a recipe for disaster.
Briefly, the cloud provider’s API allows for destructive action without confirmation, it stores backups on the same volume as the source data, and “wiping a volume deletes all backups.”
Maybe talk to PocketOS and/or Railway. It sounds an external backup solution is direly needed. Maybe Storj could be integrated for that.
Also, this is a good reminder that many storage providers (including Storj) offer Object Lock functionality to prevent unauthorized modification or deletion to backups (be it from human error, rogue AI, ransomware, etc.). It can be extremely beneficial in situations like this.
A daily backup on Storj with immutable settings and AI agents can’t srew your companies data. And your provider who “backups” on th same volume that AI agents can delete any time, can’t screw your data too.