Destroyed By Fire, Millions of Websites Offline at French Cloud Services Firm

Destroyed By Fire, Millions of Websites Offline at French Cloud Services Firm, Following the fire of March 10, 2021 which destroyed part of the OVHCloud datacenters in Strasbourg, a group legal action was initiated by several customers. Reason given: the provider would have breached its contractual obligations.

Seven OVH customers who saw their data go up in smoke in the fire at the group’s data centers in Strasbourg on March 10 have come together to launch a joint legal action (or classaction). They are represented by Ziegler & Associés. “Being bound by a contractual responsibility for storing and securing data with its customers, OVHCloud has committed a contractual breach. The companies affected by this incident are entitled to compensation”, explains master Jocelyn Ziegler to JDN, before entrusting : “A dozen other companies are in the process of evaluating the interest they might have in joining the procedure.” Who are these companies? These are mainly SMEs that do not have the means to restore lost data. Having subscribed to the Hosted Private Cloud offer, all had their data hosted in the SBG1 data center destroyed in the fire. Problem: The service’s standard backup option simply replicated data to the same data center in an adjacent room. “As soon as we have brought together 20 companies, we will start the procedure,” says Jocelyn Ziegler. “This action will be a force of opposition, reduce legal costs, and improve the chances of obtaining compensation.”

OVH Data Center in France Destroyed

The roadmap for this class action? Initially, Ziegler & Associés intends to assess the damage suffered by each client as well as the corresponding damages. “We consider that the fire is not a case of force majeure, in particular insofar as OVHCloud has not put everything in place to deal with it. We can mention, for example, the absence of an automatic extinguishing system in the data center.” As for the damage, it would be assessed in terms of the means necessary to recover the lost data, but also in terms of images for the organizations affected vis-à-vis their own customers and prospects. “Some of them have been sued by their customers for having lost their data”, observes Jocelyn Ziegler. Among the SMEs involved in the grouped action are players in the tourism, SEA/SEO and medical sectors. Structures that wish to remain anonymous to avoid tarnishing their brand image more broadly. “Some have lost billing data, which causes them payment problems,” admits Jocelyn Ziegler. “For others, in the field of health, it is medical data of patients, in particular suffering from cancer.” Players who had nevertheless activated the HDS option (health data hosting certification) of the Hosted Private Cloud offer.

The fire that hit the OVH campus in Strasbourg on March 10, 2021 caused a small earthquake in the data center market. “Customer tenders now require information on infrastructure insurance, but also APSAD certifications (which relate to the way a building is constructed and operated in terms of safety, editor’s note)”, indicates Arnaud de Bermingham, president of Scaleway, in a podcast at JDN. What about insurer policies? “Risk premiums are reviewed, even terminated customers. Contracts will be conditioned by additional certification prerequisites. This movement does not only concern France”, notes Arnaud de Bermingham.

This information comes as an expert report from Bureau Veritas on the fire safety of the Roubaix site, where the operator hosts more than 130,000 servers (compared to 62,000 for the Strasbourg site), reveals the lack of security historic settlement fire. As in Strasbourg, the Roubaix campus is not equipped with an automatic extinguishing system (PJ N°4 of the impact study): “the installations are not equipped with active RIAs, water mist systems or automatic extinguishing by sprinkler.” No mention of gas extinguishing either (read the article A Bureau Veritas report reveals the lack of fire safety at the Roubaix site).

Following a first fire on March 10 which destroyed one of OVHCloud’s data centers in Strasbourg and damaged a second, a new fire broke out on Friday March 19 at 7 p.m. on the site. It declared itself in a de-energized container in the Strasbourg 1 (SBG 1) data center, acting as a battery storage room. 300 batteries of 25 kg were impacted. “No injuries were reported among the teams of OVHcloud or its partners. Two security personnel who were inconvenienced by the fumes were examined by health professionals”, specifies the group. The fire was quickly brought under control by firefighters. Mobilizing 130 people, operations aimed at putting the infrastructure back into production following the first fire were interrupted for the night. The power supply to SBG 1, whose servers were being restarted, was cut off as a precaution. In the process, OVHCloud announced that all SBG 1 servers would eventually be moved to other datacenters located on the Strasbourg site, or on its Gravelines and Roubaix campuses, with a passage through its Croix factory to be cleaned up following the smoke release. A decision that seems to have been taken in conjunction with his insurer. The origin of this second incident is still undetermined.

OVH had started on the night of March 17 to gradually reboot the machines still functional on Strasbourg 3 and 4 after checking the status of each. On March 22, the Cnil published a note on its website on Monday March 22. Its title is unequivocal: “OVH fire: should the CNIL be notified?” . In this document, the commission recalls that a notification is necessary “if personal data have been permanently lost or if they have remained unavailable for a long enough time, such that this has created a risk for individuals.” She adds: “If the breach is likely to create high risks for individuals, they must also be informed directly by the data controller.”

Six Months Free In Case Of Data Loss

On March 22 in the afternoon, Octave Klaba posted a third video on Twitter to take stock of the situation. Regarding the ongoing investigation into the fire, the CEO insists on the number of stakeholders: judicial police, BEA, insurers, independent experts, bailiffs… “This investigation will take several months. We will obviously share all the conclusions. “, insists the CEO. But Octave Klaba intends to learn the lessons of the disaster without delay. “We decided to create a laboratory to work on the different cases of fire starting within a data center, on the ability of doors and partitions to keep the fire going, and on the most effective extinguishing methods depending on the case. We intend to publish these methods in open source so that the maximum number of companies can benefit from them”.

In a previous video released on March 16, Octave Klaba said customers would get three months free in the event of a service outage and six months free in the event of data loss.

The Private Cloud Offer Touched In The Heart

The group has drawn up an inventory of non-recoverable data backups or those under investigation depending on the data center used and the service subscribed to. Crucial information to allow its customers to restore their website and other cloud applications without delay. Main surprise: within SBG1, OVH’s private cloud offer (Private Cloud) was hosted in one room, and its backup in another room in the same datacenter. Both were destroyed in the fire. Unless you have taken the precaution of implementing a second backup with another provider, the data is therefore lost.

Remember that the Private Cloud solution is historically positioned at the top of the range. In addition, 20% of VPS/PCI backups based on OVH’s Alsatian infrastructure also went up in smoke. Customers whose backups are part of its 20% can only hope that their virtual private server has been spared (read the article How to react if my website is impacted by the OVH fire?).

Many Unrecoverable Backups

Billing was also suspended on the date of the fire for all customers using services at OVH data centers in Strasbourg. Pending their return to service, alternative infrastructures were offered to them free of charge (bare metal server, Hosted Private Cloud and Public Cloud) on the group’s data centers in Roubaix and Gravelines. OVH has also put a question and answer page online for its customers to help them deal with the situation.

The Fire Would Have Taken In An Inverter

French Cloud Services Firm

The group’s CEO, Octave Klaba, spoke for the first time on Twitter on March 11. An 8-minute video in which he discusses the circumstances of the event. The CEO apologizes to the customers and promises that such an event will not happen again. Here’s what to remember:

  • The fire started in the SBG2 data center which was completely destroyed. The fire alarms worked well. But the fire spread too quickly for on-site officers to intervene.
  • OVH provides impacted customers with alternative resources (dedicated servers, Public Cloud, Private Cloud) in its Roubaix and Gravelline datacenters.
  • The production rate in the OVHCloud factory in Croix increases to 2,500-3,000 servers per day to meet demand.
  • The thermal cameras of the firefighters who arrived on site 15 minutes after the first alert detected two inverters on fire within SBG2, one of which had been the subject of interventions the same day in the morning before being returned to service. in the afternoon.
  • SBG2 is based on technology that dates back to 2011. It is a self-ventilated tower that works by the pressure difference between the top and bottom of the building. Generation 2016, SBG3 is based on a technology that has avoided its conflagration.
  • OVH intends to use the videos from the 300 surveillance cameras on its Strasbourg site to trace the history of the fire, to understand precisely what happened in order to learn all the lessons.

A Blast For French Tech

Following the fire that broke out on the night of March 9 to 10 on the OVHCloud site in Strasbourg, 3.6 million HTTP servers representing 464,000 domain names went offline. The figure is published by Netcraft, an American specialist in Internet monitoring. “More than 18% of the IP addresses assigned to OVH in our last Web Server Survey published two weeks ago were unresponsive on March 10 between 7am and 8am,” says Netcraft. For its part, Downdetector identified several hundred error reports at the same time. OVHCloud mentions 12,000 to 16,000 customers affected. As the GAFAM penetrate Europe with merciless competitive strength, it is a real explosion for French Tech, whose image will de facto be tarnished. Its torchbearer suffered the biggest industrial disaster in its history.

The Strasbourg 2 (SBG2) data center of the French cloud was completely destroyed by the flames. On the SBG1 side, four server rooms were destroyed. Eight escaped the fire, as well as the network rooms. SBG3’s servers were also unaffected.

A press release was issued by the company in the hours following the declaration of the fire: “This Wednesday March 10, 2021, at 00:47 a.m., a fire broke out in a room of one of our four Strasbourg data centers, SBG2. We specify that the site is not subject to a Seveso classification. The firefighters immediately intervened on site in order to protect the teams and limit the progression of the fire. They thus carried out the complete isolation of the site and its perimeter as early as 2:54 a.m. At 4:09 a.m., the fire destroyed SBG2 and continued to pose risks to nearby data centers until firefighters took complete control of the fire.” The CEO of OVHCloud immediately recommended to all his customers to activate the “Disaster Recovery Plan”.

115 Firefighters Mobilized

OVHCloud’s location in Strasbourg is located on a former site of the steelmaker ArcelorMittal in the Port du Rhin district. Mobilizing 115 firefighters, 43 vehicles, six cannon launchers and two ladders, the fire was finally contained around 5:30 am on March 10. “Operational means have also been mobilized by the German authorities”, underlines the prefecture of Bas-Rhin in a press release. The few employees present on site were quickly taken care of by the firefighters.

Alongside SBG2 which was completely destroyed, the SBG1 data center will therefore have been largely affected. “Firefighters were able to protect SBG3. No impact on SBG4,” Octave Klaba tweeted.

SBG2, the data center where the fire broke out, has five floors and covers a total of 1900 m2. The flames rose to several tens of meters. Inaugurated in 2012, it has a capacity of 14,000 servers. It houses part of the very strategic Hosted Private Cloud offer targeting major OVHCloud customers. This IaaS solution based on VMware virtualization technology is fully managed by the group’s teams. SBG2 also hosts dedicated servers used by many French websites.

Speaking on Twitter, some customers who failed to back up their data found themselves in big trouble. This is the case of the Rust game, or even of the bailiffs firm Leroi & Associés which indicated that it had lost its emails following the fire. Like this firm, many users from almost all regions of France were worried on March 10 on the Downdetector forum about the impossibility of accessing their email at OVH. The group’s Mail service was put back into service on March 11 at 1:22 a.m.

Consequences that demonstrate the importance of subscribing to a backup service in a datacenter located in another geography, or even with another provider to mitigate failures that could affect all of the provider’s datacenters, such as a network crash affecting richochet to all services.

A disaster recovery plan was immediately deployed. On the program: the recommissioning of the power supply of the SBG3 data center (20 KV), but also those of SBG4 (240 KV). A temporary network room has been set up for the SBG5. Finally, the fiber optic connection to the OVHCloud data centers in Paris and Frankfurt has already been checked and was not affected by the fire.

Multiple Websites Affected

We have obviously identified many sites affected: data gouv (which has since been put back online), the sites of Esri France, the Center Pompidou in Paris, CREPS Rhône-Alpes, the Federation of Doctors of France, Generation 5, the Kiosk So Press, the Colmar Tourist Office…

Our DNA colleagues have also identified the following sites: those of Strasbourg airport, of the ENT (digital work space) ONE & NEO in Sarreguemines, of the city of Cherbourg, of the works council of Peugeot-Sochaux , the Meteor brewery, the Cityscoot electric scooter reservation site (in the Paris region), the city of Vichy, the Clermont-Ferrand rugby club, the UPR site (the political party of François Asselineau), the Caen stadium, the Saverne tourist office, the Recyclivre stadium, the European People’s University. Note that German, Italian, Spanish, Polish or Turkish sites are also affected by the outage.

BFM for its part mentioned the following sites and services: those of the cities of Arras or Saint-Ouen, several media, including Le Nouveau détective and Front populaire, the media launched by Michel Onfray. The channel’s website also cited sports clubs, including ASM Rugby, US Créteil Handball and AS Saint-Priest.

An Ineffective Fire-Fighting System

It remains to be seen whether the fire protection system deployed in the OVH data centers in Strasbourg worked well. A fire detection device is well installed there, and fire drills are carried out there every 6 months. But were the first flames that broke out in the SBG2 data center identified by this system? Have the planned procedures been correctly implemented? Beyond these questions, one certainty remains: OVHCloud data centers in Strasbourg are not equipped with extinguishing networks. They are not equipped with water sprinklers as is the case in the OVHCloud datacenters in Beauharnois, Canada, nor with high-pressure foggers to absorb the flames while protecting the machines against water, nor with gas inert, like most data centers on the market. Gases which are intended to empty the server rooms of their oxygen to smother the fire, again avoiding causing damage to the equipment.

On Friday March 26, the entire SBG4 data center was functional again, which was not yet the case for SBG3.

The information comes as OVHCloud announced on March 9 its intention to go public. Octave Klaba and his family nevertheless intend to retain the majority of shares following the operation.

Leave a Reply

Your email address will not be published. Required fields are marked *