The suggestion has been brought up by @Vadim to have special S3 nodes that handle S3 proxying rather than one central S3 gateway or forcing customers to run a local S3 gateway. How might such a thing work?
Some interesting aspects of the S3 protocol that could be used to construct a more distributed S3 gateway service:
in the S3 API, HTTP connections are made to a specific bucket, ie, the host you connect to is: bucket.s3.us-west-2.amazonaws.com
or in the case of Storj: bucket.gateway.storjshare.io
This directly allows S3 traffic to be distributed across many hosts by assigning buckets to specific S3 nodes (static DNS config) or using a load-balancing DNS server that returns multiple/varying IP addresses for a single bucket. This DNS link is the reason S3 bucket names have to be globally unique: they are resolved to IP addresses by Amazon’s DNS servers.
S3 can return HTTP redirect responses. In the example above, if storage for a bucket actually “lives” in the us-west-1 region, a connection to that bucket using the us-east-2 region can return a redirect response:
This forces the S3 client to re-issue their request to a new host/location. For a connection to bucket.gateway.storjshare.io, Storj’s DNS service could map that request to a distributed S3 node’s IP address as mentioned above, or if buckets aren’t part of the DNS service at Storj (they use a wildcard DNS A record), the main S3 gateway could issue an HTTP 3xx redirect with the distributed S3 node’s IP address in the Location: header.
S3 already has the concept of multipart uploads. First there is a call to initiate a multipart upload. This would be made to the main S3 gateway. Then each part (Storj segment) is uploaded in separate S3 API calls, and each can be on a different HTTP connection talking to different S3 nodes concurrently. I’m guessing each S3 node would have to make a request to the main S3 gateway to start a part transfer with uplink. When all parts are uploaded, another S3 API request to the main S3 gateway completes the multipart upload, specifying the complete list of parts and checksums. The important thing is that the bulk data traffic flows directly between the S3 client (backup program for example) and the S3 nodes, and uplink runs on the S3 nodes, not the main S3 gateway.
one thing not so great is that the S3 nodes (but not storage nodes) would get access to customers’ S3 credentials, just like the current S3 gateway. I don’t think there is any way around this while staying compatible with the S3 API. Maybe a node wanting to be an S3 node would have to be bonded, be part of Select, have to sign legal agreements with Storj, etc. Sort of the way businesses say “we only disclose your data to our business partners”.
as far as encryption goes, that doesn’t seem like a big deal to me. Amazon’s S3 as launched didn’t do encryption so clients had to do it themselves if that was a concern. Then Amazon added encryption at rest, now they have encryption with Amazon-managed keys and encryption with customer-managed keys. I’m guessing Storj’s S3 service doesn’t support all these S3 extensions. It’s nice that Storj encrypts everything end-to-end without needing the S3 extensions, but S3 has been around for so long without the extensions that most client software already handles encryption.
Obviously a lot more thoughts and ideas are needed!
we also need to handle one serious problem, S3 is not encrypted data, also that node can make it encrypted or decrypted, node will need to get encryption passwords.
So this should be addressed somehow. Some matter of TRUST.
moreover need to think about security, that node will be not hacked.
so minimum security standards have to be made.
So it will need some kind of contract. Also some security standards
Maybe an immutable lightweight OS that’s completely locked down (no ssh, web, console, etc) and lives mostly in memory. I most like would prefer a virtual appliance.
S3 node should get network related settings like IP, gateway, DNS, etc from a local DHCP server (reserved address), this way you can put it in every vlan, subnet or segmented network you want.
Because the node gets a local DHCP address, it makes it also possible to create FW rules etc.
The node frequently (automatic) updates SSL certificates, keys, local config/database with the latest settings, security updates, etc.
A node should every x-minutes check into Storj homebase. If it misses its checkin period, it should be put onto hold until its establish that the node is ready to return to regular work after a specified successful check-ins.
I think that creating a complete locked down S3 node is possible.
The rest about how to balance traffic among the nodes, and this without creating chokepoint is far from my expertise.
Indeed, a lot. Like, what’s the compensation scheme, and how do you issue (shared?) TLS certificates to these nodes? How do you ensure enough throughput? How do you deal with multipart uploads where each part is received by a different node? How do you deal with unwanted traffic, like storage/distribution of malware (or worse)?
I don’t really think this is feasible; S3 has not been designed for this.
I may only add to the S3-only approach, that it’s could be possible to encrypt the uploaded object using the customer’s access key, then it would store the whole object encrypted with it. This might help to avoid contacting an auth service and use the access grant and uplink, but it also means, that this approach will not be faster, because you will upload/download an object only to/from the one node.
Uploading different chunks to different nodes will not help - you need to assembly them back on download, how the first contacted node may figure out, which other nodes it should contact? Only if the each node will register their part on the satellite and might figure out whom to contact to provide a requested part using customer’s S3 credentials.
And this is could work only if we assume that every S3 client requesting a different node for each part.
Next question - the redundancy, how to do that if you don’t want to use libuplink under the hood and want to reimplement it based on S3? So, either the node should replicate their part to other nodes or it should use erasure codes, but in the last case it should be orchestrated from the satellite, otherwise each node might start to use their own encoding and it will be incompatible with each other.
The audit and repair. Perhaps workers can audit parts on the nodes or more likely erasure stripes. And perhaps repairers could reconstruct erasure stripes and re-upload.
The streaming, how to implement seeks?
So, for me it seems this reimplementation will build a gateway-mt at the end.
Need to solve security problems with S3 credentials and/or access grants/customer’s data leaking on the Byzantine node.
I tried to understand what your explaining. Eventually I dove into the Storjv3 whitepaper trying to get some clarity. Although I’m stil trying the wrap my head around some aspects, a picture displaying specific network flows would help me a lot (reason why I like whiteboard sessions when designing new systems, I’m not that fond of large amounts of text when a network flow diagram can also give a lot of clarity).
Why cant we repurpose current Gateway ST as a distributed solution? (someone will point out if this is a stupid question )
I can see why this would be difficult (or imposible) when you use the Self-Managed Project Encryption
Would it not be doable when a customer uses Storj-Managed (Automatic) project encryption? This way a customer only needs a Access and Secret Key
Option 1: We do not offer encryption for data at rest and make customer responsible for encryption (client/application side encryption). Uplink must possibly be altered by removing the mandatory encryption passphrase.
Option 2: Uplink gets an distributed version which will lookup the encryption passphrase from the satellite (or is the encryption passphrase stored on the gateway-mt?? I’m not sure, someone will tell/correct me). But uplink wil only use it while uploading/downloading data and purge this information after the transaction (upload/download) completed.
If (for instance) option 2 is feasible (I dont like option 1), both options should be part of a more basic plan than the current pricing plans. This makes using the Storj hosted GW a feature for the current pricing plans for customers who want more security/safety.
Oh, forgot. I don’t think it is a good idea to make every SNO also participate in a distributed gateway setup. It should be handled as a completely separate thing. Handling large amount of storage is something else then handling a lot of traffic/connections. So lets introduce a GNO
Because gateway-st is a single tenant gateway, it will work only with a single access grant.
But you want to provide a service for multiple customers (tenants), so the only option is to use a gateway-mt (multi-tenant S3 gateway).
The difference between a self-managed and automatic encryption is who own your encryption phrase. It’s doesn’t affect how it would be processed on nodes. In the usual scenario the node will get one of the pieces of the erasure coded encrypted segment. How the encryption was done - doesn’t matter here.
uplink doesn’t request an encryption phrase, it uses the provided access grant. The access grant includes the satellite address, the API key to get access to your metadata on the satellite (where pieces are located), the encryption key (derived from your encryption phrase, there could be many encryption keys, which could be derived) and caveats (what you can write, read, list, delete).
The same as a gateway-st (only for one access grant), or gateway-mt (for any provided access grant, but the access grant itself is stored on the auth service encrypted with the access key. So to get it decrypted you need an access key and, if it’s not public, also a secret key to decrypt the access grant, which then give a temporary access to your data from the gateway-mt instances accordingly caveats in the decrypted access grant).
The gateway-mt service is already a distributed service. So, until there is a bottleneck - it wouldn’t have more instances.
However, I believe, that the idea is to make it more distributed using already existing 30k nodes. If not - then I don’t know, what’s the problem we are trying to solve now.
I could be separate nodes, with contract from storj, to gain some level of trust.
with separate identity and signed from storj.
may be 100-200 nodes with Gbit connection. compensated by traffic, with some logical price for both sides.
some hardware requirements. like xeon cpu 10 cores, 64gb ram-need to be specified by storj.
I understand that it is hard in today reality, but if storj plan get mor clients s3 will cost more and more.
Storj need to plan things today to make something work tomorrow or next year
)