This should exactly be the reason to store backups in a distributed way, like Tardigrade:
Storj should send out a Twitter immediately…
5 Likes
Very careful trying to be seen to capitalise on some else’s misfortune, especially whilst it’s still happening…
Let the dust settle.
15 Likes
Of course not. For empathy and support. This could turn into hell for OVH.
2 Likes
100% agree there.
A supportive message looks very good in the public eye.
The post mortem would be the time to say “this wouldn’t happen to us” (we’ll skirt over the fact that it could very well happen to the satellites…) 
1 Like
Satellites? Which satellites??? 
2 Likes
Don’t the satellites use highly replicated databases (CockroachDB) [1], so it would be possible to survive with just downtime and no data loss if this happened to where they’re hosted?
[1] https://storj.io/blog/2020/08/choosing-cockroach-db-for-horizontal-scalability/
2 Likes
Yes, but down time is a BIG DEAL.
2 Likes
I thought replication is specifically so that unless a lot of machines go down together across many availability regions, there’s not even downtime. Anyone to clarify?
1 Like
They would not have lost data. But OVH is not about storing data only, they provide servers too.
Unless they’re scattered in different geographic locations, a fire like the one at OVH’s would completely kill a satellite, even if it has a redundant database. But maybe they’re not in the same place, I dunno. Hopefully not.
Do you guys think satellites get disqualified if they stay offline for too long? 
5 Likes
Hah! Hadn’t thought of that!
Also, we can pay our Tardigrade bill only when it reaches a certain threshold
2 Likes
Yes. That’s the plan. Otherwise Community satellites can abuse the system.
You already do so 
You will be charged once a month. If you have a coupon or promo credits, you will not actually pay.
And also the US2 is free until go in Production
2 Likes
You are right, they are multiregional. So, downtime currently should be only in specific migrations, which cannot be done without downtime, like
1 Like
I found this:
Does this mean that there is not 1 satellite in one specific data center but 1 satellite is spread over several data centers?
It would be really nice if Storj Labs would let users and SNOs know, if their data would be safe in case of such an incident like at OVH and how the network would be affected in case of such an incident. For example we know that the repair jobs are hosted at Hetzner only. What if they experience a similar incident and the repair servers burn down?
As far as I know - yes, they are in several datacenters, at least databases.
1 Like
It seems OVH builds their data centers from wood? Someone who actually understand French can comment? https://nitter.cc/MedecineLibre/status/1369618937063817219
1 Like
Well this is interesting: https://nitter.cc/olesovhcom/status/322224148925521921#m
If I understand correctly they used a custom liquid for their fire protection system.
And it does look like wooden floors. But that’s in Roubaix not Strasbourg.
SBG2 was built in 2016 (tweets are from 2013), so it could have been built the same way.
1 Like
Looks like we’re seeing the impact in the # of active nodes: http://storjnet.info/
“There are ~11,282 active nodes, -205 during the last day”
Someone who collects all active storage nodes, like storjnet.info, could probably simply check which storage node IPs are owned by OVH. That would indeed be interesting!
2 Likes
We’re in a bit of a transition point over the next month or two (and this is the reason for some of the migrations @Alexey referenced). Some of our Satellites, like @Alexey says, are multiregion, some are in a middle point, and some are not. This is an active area of work for us, which is why we haven’t advertised it a bunch yet. We do intend to make a big deal about it once we’re done.
Currently:
- us2.tardigrade.io (our Beta satellite) is fully multiregion. It will continue to run with no downtime if an entire datacenter goes down.
- asia-east-1.tardigrade.io is partially multiregion. The database (CockroachDB) was migrated two weeks ago to be multiregion (the hard part), but the API servers are not yet (the easy part).
- europe-west-1.tardigrade.io is single region right now, but the database migration is happening this Sunday (check status.tardigrade.io).
- us-central-1.tardigrade.io is having a database migration soon thereafter.
In the next month, we expect all Satellites to be completely multiregion, like us2.tardigrade.io is. Once that’s done, we are planning a larger marketing push to announce multiregion, and in addition, hosted S3 gateways, multipart support, etc. You’ll hear about it!
15 Likes