Bug From July Caused Hours-Long Microsoft Outage

A “latent defect” that had been present in Microsoft’s data centre software since July caused a [data centre outage late last week] that disrupted Outlook, Teams, Copilot and other services for hours, the company said.

The issue highlighted the risks of extensive business reliance on a single vendor.

The problem began when Microsoft began routinely rerouting some user traffic at just after 8 p.m. ET (1 a.m. GMT) on 11 November, and gradually worsened over the next 15 hours, Microsoft said in a preliminary report to customers on Sunday.

The interior of a data centre.

Cascading issues

The disruption was made worse by automated systems that rerouted emails and user traffic to other, supposedly healthy servers, causing the issue to spread more broadly, Microsoft said.

The issue became critical at around 11 a.m. ET on 12 November, after which it took Microsoft another 15 hours to identify what was wrong and begin deploying a fix.

At its worst last Thursday, the cascading problems meant 80 percent of Outlook servers were unresponsive.

“A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions,” stated Charles Lamanna, Microsoft’s executive vice president for workplace software.

He said the bug was found in an update for Microsoft 365 networking gateway servers, worsened by a surge in load to networking gateway and reduced server capacity caused by a separate issue earlier in the week.

“Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July. This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States,” he stated.

Recovery time was lengthened because “as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again”, Lamanna stated.

Original source Bug From July Caused Hours-Long Microsoft Outage

Back to home