Story Technology Microsoft

Microsoft outage exposed how much of the digital economy depends on a few big tech platforms

Technology
The brief

A major outage hit Microsoft 365 last week, leaving large numbers of users unable to send and receive email. The company has since acknowledged that at the worst point, more than 80% of its servers were unresponsive. Microsoft said the outage was caused by a technical issue rather than a cyber attack, and it denied any link to the thousands of layoffs it had announced the previous month.

Microsoft's platform head, Charles Lamanna, said the defect was encountered across the company's fleet in less than two hours and rendered servers unresponsive. The root cause combined two factors. Microsoft maintains two independent network gateway systems; one failed, and the remaining gateway took the full load. Each gateway was designed to handle all traffic on its own, but a latent defect made servers overwhelmed and unresponsive under certain load conditions, caused by lock contention inside an internal process. That defect stemmed from a change prepared specifically for Microsoft 365 networking gateway servers, which began deploying in July and was rolled out globally over several weeks without observed issues.S1

Microsoft said its normal load testing failed to catch the problem because test labs used a different traffic and hardware mix. The contention issue worsens on machines with more cores and newer hardware, while testing focused on weaker hardware with fewer cores, meaning the update was not tested under the conditions where it was most likely to fail.S1

Services including Outlook, Teams and Copilot, used by more than 430 million people worldwide, went down, and email and internal communications froze for governments and businesses. Commentators argue the episode shows how dependent the digital economy has become on a handful of big technology companies, which they describe as too big to fail. Microsoft faces reputational damage and contractual penalties that could reach up to $800 million per day for services it failed to deliver.S2

Analysts trace the vulnerability to a shift over the past two decades from selling software once for an upfront cost to renting access through software as a service. That model moved operational control toward providers and locked customers into long-term contracts, so a single failure can ripple across millions of users. Some observers have drawn parallels to the 2008 financial crisis, when the public was left to absorb the fallout from banks deemed too systemically important to fail.S2

Timeline · newest first
20m 9to5mac

Microsoft outage saw 80% of servers fail – here’s what happened

Last week saw a truly epic outage to Microsoft 365, with a huge number of users unable to send and receive emails. The company has now admitted that at the worst point, more than 8…

43m Phys

Microsoft outage shows that big tech can't be treated as too big to fail

When a massive outage hit Microsoft late last week, much of the digital world ground to a halt. The outage wiped out key Microsoft services including Outlook, Teams and Copilot, wh…

Microsoft outage exposed how much of the digital economy depends on a few big tech platforms

2 outlets 2 reports
The brief

A major outage hit Microsoft 365 last week, leaving large numbers of users unable to send and receive email. The company has since acknowledged that at the worst point, more than 80% of its servers were unresponsive. Microsoft said the outage was caused by a technical issue rather than a cyber attack, and it denied any link to the thousands of layoffs it had announced the previous month.

Microsoft's platform head, Charles Lamanna, said the defect was encountered across the company's fleet in less than two hours and rendered servers unresponsive. The root cause combined two factors. Microsoft maintains two independent network gateway systems; one failed, and the remaining gateway took the full load. Each gateway was designed to handle all traffic on its own, but a latent defect made servers overwhelmed and unresponsive under certain load conditions, caused by lock contention inside an internal process. That defect stemmed from a change prepared specifically for Microsoft 365 networking gateway servers, which began deploying in July and was rolled out globally over several weeks without observed issues.S1

Microsoft said its normal load testing failed to catch the problem because test labs used a different traffic and hardware mix. The contention issue worsens on machines with more cores and newer hardware, while testing focused on weaker hardware with fewer cores, meaning the update was not tested under the conditions where it was most likely to fail.S1

Services including Outlook, Teams and Copilot, used by more than 430 million people worldwide, went down, and email and internal communications froze for governments and businesses. Commentators argue the episode shows how dependent the digital economy has become on a handful of big technology companies, which they describe as too big to fail. Microsoft faces reputational damage and contractual penalties that could reach up to $800 million per day for services it failed to deliver.S2

Analysts trace the vulnerability to a shift over the past two decades from selling software once for an upfront cost to renting access through software as a service. That model moved operational control toward providers and locked customers into long-term contracts, so a single failure can ripple across millions of users. Some observers have drawn parallels to the 2008 financial crisis, when the public was left to absorb the fallout from banks deemed too systemically important to fail.S2

Timeline · newest first

Sources · 2 citations

S1 9to5mac 1 report · EN
S2 PhysLead 1 report · EN