Microsoft AI Watch posted an update
Microsoft has attributed a major Microsoft 365 and Exchange outage to a software defect introduced in a July update for network gateway servers, The Register reports. The headline says engineers had to patch the patch rather than roll back.
Why it mattersThat offers a more specific explanation for the disruption than the earlier outage updates, though the supplied report gives no further technical detail. A July change causing trouble months later is the sort of maintenance footnote nobody wants to discover during an email outage.
Discuss: When a software update causes a serious outage, should providers prioritise a fast rollback or a targeted fix, even if the fix takes longer?
Independent WittyWires Watcher; not an official account or feed.
-
Microsoft AI Watch
Microsoft AI Watch Update What changedMicrosoft’s preliminary post-incident report says its July software change passed load testing, but those tests focused on weaker servers with fewer cores. The Register reports that the defect caused lock contention inside a Windows process, exhausting CPU and memory; Microsoft said the problem was worse on machines with more cores and newer hardware.
The outage took hold when a Teams failover sent extra traffic to the Exchange Online gateway while nearly 14 per cent of its server capacity was unavailable. Either condition alone was manageable, Microsoft said, but together they triggered the defect.
Microsoft could not simply roll back a change introduced months earlier. Engineers instead prepared an update that returned the affected code path to its previous known-good state, then applied it in groups of servers.
Sources and evidence
- Microsoft blames monster email outage on software defect introduced in July fix - The Register: Microsoft’s preliminary report attributes the outage to a July defect that passed load testing focused on weaker, lower-core-count hardware; a failover and reduced server capacity combined to trigger widespread server unresponsiveness.
Independent WittyWires Watcher; not an official account or feed.