A hidden defect in Microsoft’s cloud gateway, dormant since July, was to blame for one of the largest outages in the company’s history last week. A preliminary Post-Incident Report (PIR) released by the company outlines how the latent bug, changes in traffic, and reduced server capacity combined to knock much of Microsoft 365 and Exchange infrastructure offline at the end of last week. Microsoft’s report lists the incident’s final status as “a technical issue not related to cyber or AI/agents.”
Microsoft’s preliminary report places the outage between November 12 at 16:20 UTC and November 13 at 16:10 UTC, just under 24 hours total, and additional monitoring remained in place for any affected systems through November 14.
The outage started on November 12 and quickly rippled through businesses and public offices that depend on Microsoft’s cloud and email exchange infrastructure. The outage is covered in Microsoft’s incident IDs EX1550543 and MO1550821. You need Microsoft 365 admin credentials to see the full report on the company’s status pages, but Office Watch has reproduced the preliminary report, and the University of Pennsylvania maintains an archive of Microsoft’s service-health updates.
A perfect storm
Microsoft attributes the outage to three factors. A change deployed to Microsoft’s cloud gateway starting in July created a bug where, under certain conditions, CPU and memory resources could become exhausted and unresponsive. Microsoft contends that the defect was never discovered in pre-release testing because the hardware used was less powerful than some systems already deployed in production, where higher CPU core counts on newer machines apparently contributed to the lockout.
The second factor was server load. Microsoft said that “Traffic shifts during standard network maintenance and upgrades exposed a latent bug.” The company keeps two independent gateway systems in place to access Microsoft 365 for resilience. If one system encounters trouble and users can’t get through the front door, the other is opened to keep traffic flowing. According to the report, about 15 hours prior to the outage, traffic from Teams was shifted from the alternate gateway to the primary, where Exchange Online access is maintained. The switchover was a standard fail-safe, but it left the gateway handling more traffic, with a different mix of requests than before. As requests continued to pile up, affected servers exhausted their capacity and refused new connection requests.
The traffic shift exposed a third factor: infrastructure capacity. An existing fault meant that Microsoft’s gateway server fleet was operating at 14 percent below optimal conditions. That missing headroom could have normally absorbed the demand, but when combined with the changed traffic patterns and underlying defect, it left the gateways with lock contention as requests kept piling up.
As US business operations began hitting their daily peaks, the gateway servers began to freeze. Backup systems that shifted requests away from unresponsive servers did their job, but were met with the same lockouts on the supposedly healthy backup gateways. This created a snowball effect, leaving more than 80 percent of Microsoft’s affected gateway servers frozen in the sweep. Mechanisms in place to isolate affected systems by region seem to have failed. Microsoft’s current report doesn’t address why these isolation systems may have failed.
No easy way out
Recovery took longer than normal because the fix required a targeted system update from Microsoft engineers. Attempts to restart the gateway servers were ineffective; when the servers came back online, large request queues were still waiting for access, immediately triggering the same resource exhaustion that contributed to the outage in the first place. Microsoft says that services like Teams, OneDrive, and other Office applications could be routed away from the affected infrastructure, but scaling limitations stopped Exchange Online from making a similar move.
Since the bug from July was still undiscovered, system rollbacks were equally ineffective. Simply returning to a previous state where things were “working” couldn’t happen, because the ghost was already in the machine. Servers were patched and returned to service gradually, and Microsoft says it has added additional capacity and traffic-distribution guardrails to try to avoid anything similar happening again. Microsoft also extended delivery timeouts on affected emails to stop delayed messages from bouncing back as undeliverable.
Microsoft expects to publish its full final incident report on November 20. This final report may provide additional clarification on why the regional isolation systems failed and what corrective actions will be taken to ensure this type of outage can be avoided in the future. All of Microsoft’s cloud services appeared online and operational as of Monday morning.

