OVH datacenter fire

This should exactly be the reason to store backups in a distributed way, like Tardigrade:

10 Likes

Storj should send out a Twitter immediately…

5 Likes

Very careful trying to be seen to capitalise on some else’s misfortune, especially whilst it’s still happening…
Let the dust settle.

15 Likes

Of course not. For empathy and support. This could turn into hell for OVH.

2 Likes

100% agree there.
A supportive message looks very good in the public eye.
The post mortem would be the time to say “this wouldn’t happen to us” (we’ll skirt over the fact that it could very well happen to the satellites…) :wink:

1 Like

Satellites? Which satellites??? :innocent:

2 Likes

Don’t the satellites use highly replicated databases (CockroachDB) [1], so it would be possible to survive with just downtime and no data loss if this happened to where they’re hosted?

[1] https://storj.io/blog/2020/08/choosing-cockroach-db-for-horizontal-scalability/

2 Likes

Yes, but down time is a BIG DEAL.

2 Likes

I thought replication is specifically so that unless a lot of machines go down together across many availability regions, there’s not even downtime. Anyone to clarify?

1 Like

They would not have lost data. But OVH is not about storing data only, they provide servers too.

Unless they’re scattered in different geographic locations, a fire like the one at OVH’s would completely kill a satellite, even if it has a redundant database. But maybe they’re not in the same place, I dunno. Hopefully not.

Do you guys think satellites get disqualified if they stay offline for too long? :joy:

5 Likes

Hah! Hadn’t thought of that!
Also, we can pay our Tardigrade bill only when it reaches a certain threshold :wink:

2 Likes

Yes. That’s the plan. Otherwise Community satellites can abuse the system.

You already do so :slight_smile: You will be charged once a month. If you have a coupon or promo credits, you will not actually pay.
And also the US2 is free until go in Production

2 Likes

You are right, they are multiregional. So, downtime currently should be only in specific migrations, which cannot be done without downtime, like



1 Like

I found this:

Does this mean that there is not 1 satellite in one specific data center but 1 satellite is spread over several data centers?
It would be really nice if Storj Labs would let users and SNOs know, if their data would be safe in case of such an incident like at OVH and how the network would be affected in case of such an incident. For example we know that the repair jobs are hosted at Hetzner only. What if they experience a similar incident and the repair servers burn down?

As far as I know - yes, they are in several datacenters, at least databases.

1 Like

It seems OVH builds their data centers from wood? Someone who actually understand French can comment? https://nitter.cc/MedecineLibre/status/1369618937063817219

1 Like

Well this is interesting: https://nitter.cc/olesovhcom/status/322224148925521921#m
If I understand correctly they used a custom liquid for their fire protection system.
And it does look like wooden floors. But that’s in Roubaix not Strasbourg.
SBG2 was built in 2016 (tweets are from 2013), so it could have been built the same way.

1 Like

Looks like we’re seeing the impact in the # of active nodes: http://storjnet.info/
“There are ~11,282 active nodes, -205 during the last day”

Someone who collects all active storage nodes, like storjnet.info, could probably simply check which storage node IPs are owned by OVH. That would indeed be interesting!

2 Likes

We’re in a bit of a transition point over the next month or two (and this is the reason for some of the migrations @Alexey referenced). Some of our Satellites, like @Alexey says, are multiregion, some are in a middle point, and some are not. This is an active area of work for us, which is why we haven’t advertised it a bunch yet. We do intend to make a big deal about it once we’re done.

Currently:

In the next month, we expect all Satellites to be completely multiregion, like us2.tardigrade.io is. Once that’s done, we are planning a larger marketing push to announce multiregion, and in addition, hosted S3 gateways, multipart support, etc. You’ll hear about it!

13 Likes