
Microsoft has traced last week’s global Microsoft 365 outage to a July update for its network gateway servers. The bug lay dormant for more than three months before a surge in load and a drop in capacity set it off, according to the company’s preliminary post-incident report.
Microsoft published the report to customers in the Microsoft 365 admin centre on 15 November, The Register reported. Charles Lamanna, Microsoft’s executive vice president for Copilot, agents and platform, posted an abridged version on X on Monday, ZDNET reported.
“A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions,” Lamanna wrote.
Three things at once
The July update was built specifically for the Microsoft 365 network gateway servers. It passed load testing and ran worldwide without problems, Microsoft said. But the defect caused lock contention inside an internal Windows process, which exhausted a server’s CPU and memory.
Testing missed it, because the load tests focused on weaker hardware with fewer cores, according to the report. The contention is worse on newer machines with more cores.
Two other conditions then met. Teams failed over to the gateway that also carries Exchange Online, which raised the load. At the same time, an unrelated issue from the day before had taken nearly 14% of the gateway’s server capacity out of service.
When a server stopped responding, traffic moved to the next healthy one, which then hit the same bug. Within two hours, more than 80% of the servers were unresponsive at the lowest point, the report says.
Why recovery took so long
Waiting traffic overwhelmed servers again as soon as they came back online. Microsoft said its automated recovery was not designed for such a broad failure.
“Since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change,” Lamanna wrote.
Engineers instead wrote an update that returned the Windows code path to its previous state. They applied it to groups of servers in turn. Teams, OneDrive, SharePoint, and the Office apps could move off the faulty gateways and came back sooner. Exchange Online could not, for scale reasons, which is why Outlook was down longest.
The report’s timeline, as published by The Register, starts with the Teams failover at 01:07 UTC on 12 November. The first support case came in at 16:33 UTC. Microsoft applied its fix from 07:20 UTC on 13 November and sent its final customer update at 04:15 UTC on 14 November.
More than 1,000 engineers and other staff joined outage calls led by Lamanna and cloud chief Scott Guthrie, Bloomberg reported. On 13 November, Microsoft apologised and said the cause was not a cyberattack or AI agents.
We are deeply sorry for the disruption this incident has caused our customers and partners. We know the trust they place in our services and take that responsibility extremely seriously. We are seeing broad recovery and stability across our services.
This was a technical issue not related to cyber or AI / agents. The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change. We have addressed the underlying issue and continue to restore service to remaining customers and regions.
Given the scale of demand, we restored services gradually, validating stability at each stage. While this took longer, it helped ensure a stable restoration.
We will provide a full post-incident report once our review is complete.
Questions in Washington
The outage hit governments, courts, hospitals, and businesses. Downdetector logged reports from 74 countries and territories, according to the Associated Press. TNW covered the restoration of services and Europe’s sovereignty debate that followed.
Last month, a Google Cloud outage in its London region hit services including Revolut.
House Homeland Security Committee chair Andrew Garbarino told Politico he was seeking more information from Microsoft and CISA.
“We need to understand what happened and how to make these systems more resilient,” Garbarino said.
Microsoft plans to publish its final post-incident report on 20 November, according to The Register.