It is said that Storj is S3 compatible. From what I understand, I can stream data from S3 directly to a browser securely. By saying securely, what I mean is no-one can access the video directly but only through the website the video is being streamed to. My question: Is this possible in Storj.
I am working on an ed-tech platform like Udemy and I want to store videos of courses on Storj. Also, for mitigating the risks of piracy, I am using Storj because of the default encryption system which is not there in IPFS/Filecoin.
Will I be able to do this? If yes, can you point out how?
If you start fresh, I would like to suggest to use a native integration instead. This way your clients will receive a stream from nodes nearest to their location, unlike s3, where you have only few regions.
See Host a Static Website - Storj DCS for example.
What i do is make an account on us1 or other sat then upload video to bucket and then get the share url for bucket and copy the download link and embedded it in your HTML it works great for me
Reviving the thread due to some news coverage about a local streaming platform failing to handle traffic of a sport event, with comments that “a decent CDN would do, something the platform failed to ensure” (-:
Streaming live events like that at scale seems to usually work by means of the HTTP Live Streaming protocol, i.e. splitting the video into 10-second chunks, each a separate file that is uploaded to a CDN, then downloaded by each customer as soon as it shows up on the CDN. This requires upload latency to be as low as possible, and then having the nodes actually cope with the traffic.
Has any HTTP Live Streaming setup been tested on the Storj network?
This is quite a complex use case for Storj to handle. First you would need a gateway like implementation that can recombine segments into HLS and offer it to viewers. Now that should technically be possible, but the real complication comes in when you consider the fact that everyone will be requesting the same segments at the same time. And each segment is available on about 80 nodes. Meaning each node would have to roughly serve 29/80 ~= 36% of demand for these files. Some of this could be mitigated by using relatively small segments and small time shifts from viewers might be enough to not demand those pieces at the exact same time. Different segments would then be spread out over the entire network.
It’s possible, but there are limits. Ideally you would want demand based scaling on the network. Not only that, but a really responsive version of demand based scaling. Preferably seeded by more than 80 pieces per segment to begin with.
On demand scaling has been part of the V3 white paper as a future possibility since the start. You can think of it like repair workers accept instead of replacing lost pieces they would create additional pieces on demand to spread the load.
It wouldnt technically be live if storj has to store it first then stream it would it?
I have tested opencast and it works so far but its not storing data right off the bat. So live streaming isnt live technically.
Im also testing with plex for streaming it works great though.
Yeah, this part will probably be the actual bottleneck here. As long as there is no web browser-compatible implementation, HLS won’t benefit much from Storj.
It’s live “enough”, i.e. tens of seconds of delay. This seems to be acceptable even for sports events, I’ve seen local stations using HLS there. Sure, your neighbour will probably shout GOOOOAAALLLLL!! few seconds before you see it
Reminds me of Joost. The old P2P video project that the founders of Skype created after they sold Skype.
Large scale content distribution of live video like this might be possible but I doubt the current implementation would handle it ideally. It may be something where the protocols could be expanded to use the nodes as more immediate relays, for instance keeping a certain number of pieces in memory (cache) temporarily to service constant requests for the same bits. In other words, some enhancements to the model would be possible to foster a healthier experience for specific use cases. We’ll get there, just takes time.
The question here is whether the current implementation is scalable enough for this use case, and as @BrightSilence pointed out, there might be some problems. The event that led to my question had 300k viewers. If viewers used libuplink, this would mean that each piece would initiate ~100k uploads at almost the same time, maybe within few minutes tops. If viewers used a gateway, then the gateway needs all that bandwidth too.
Even if you could somehow spread it equally over all nodes. 1080p30 video of decent quality can be about 8Mbps. 300k viewers spread over 13k nodes would be about 185Mbps for each node. Assuming no overhead (which there will be). Plenty of nodes will not be able to keep up with that.
Many of those nodes will share the same connection
Many of those nodes are full and won’t participate
So, to support this kind of application for just a single big event, we’d probably need 100× the amount of nodes. Or a large number of well-colocated nodes, ready to accept big traffic spikes. Frankly, makes me think of putting nodes in proper data centers.
for this thing my nodes has NVME cash 500-1000GB it helps make user experience better, and makes less reading from HDD and give more time for writing to HDD.
I wouldn’t go that far. This is clearly not something that’ll happen tomorrow. But with time a lot of benefits will make this easier. Internet connections will become faster, the network will grow and Storj would be able to implement intelligent scaling. Meaning that it could become aware of which nodes actually have high throughput capabilities. My nodes could already serve this amount of data, through the use of a 1TB R/W cache and a 1gbps synchronous connection. Over time this will become more of a norm. Now consider a new kind of capability for satellites to rank nodes based on upload capabilities (not as reported by the user, but as measured. For example based on long tail cancellation rates). The node selection method could than be changed to take this upload capability in mind and select faster nodes more frequently. Automatic scaling could prefer those faster nodes. This would also be a very valuable use case for node operators, because of the high egress income. So it might become worth upgrading your internet connection. Or adding SSD cache.
You wouldn’t start with a 300k viewer scenario of course, but you can build there over time. By the time the network gets to that point, i don’t think you’d need more than 10x the number of nodes we have right now. But at the same time, incentives may become good enough that running nodes becomes more profitable and a lot more node operators will jump on board.
