Microsoft had three staff at Australian data centre campus when Azure went out – Cloud – Storage | Windows

20210318104400 Microsoft Azure Sign 2.jpg

Microsoft had “insufficient” staff levels at its data centre campus last week when a power sag knocked its chiller plant for two data halls offline, cooking portions of its storage hardware.

The company has released a preliminary post-incident report (PIR) for the large-scale failure, which saw large enterprise customers including Bank of Queensland and Jetstar completely lose service.

The PIR sheds light on why some enterprises lost service altogether: so many storage nodes were gracefully shut down – or had components fried – in the incident that data, and all replicas of it, were offline.

In addition, after storage nodes were finally recovered, a “tenant ring” hosting over 250,000 databases, failed – albeit with uneven impact on customers.

Chillers offline

Microsoft said the cooling capacity for the two affected data halls “consisted of seven chillers, with five chillers in operation and two chillers in standby (N+2)”.

A power sag – voltage dip – caused the five operating chillers to fault. In addition, only one of the standby units worked.

Microsoft said the onsite staff “performed our documented emergency operational procedures (EOP) to attempt to bring the chillers back online, but were not successful.”

The company appeared to be caught out by the scale of the incident, with not enough staff onsite, and its emergency procedures not catering for the size of the issue.

“Due to the size of the data centre campus, the staffing of the team at night was insufficient to restart the chillers in a timely manner,” the company said.


read more

We use income earning auto affiliate links. More on Sponsored links.

Summer Heat
There's something about the balminess of summer that makes you want to let loose. And the right soundtrack makes all the difference. Mixing hip-hop and R&B with Afrobeats, dancehall, and more, here’s a playlist to take you through those warm nights. Our editors regularly update this playlist, so if you hear something you like, add it to your library and keep the party going.
Listen to Apple Music

Top 100: Global
The most-played songs around the world, updated every day. Listen to Apple Music.

Ad Amazon
  1. Home and Kitchen
  2. Home Decor
  3. Men's Clothing
  4. Women's Clothing

Related Posts