Microsoft says the issue may still persist for a small number of users despite its resolution.
The cloud computing platform was reported to be experiencing a major global outage on Wednesday
Photo Credit: Reuters
Microsoft on Thursday announced that its Azure services have been restored across affected regions. The cloud computing platform was reported to be experiencing a major global outage on Wednesday, impacting not only Microsoft-owned platforms like Microsoft 365, Minecraft, Outlook, and Xbox Live, but also Alaska Airlines, Starbucks, and other services. The Redmond-based tech giant said that it impacted services leveraging Azure Front Door (AFD), its modern cloud content delivery network (CDN).
Microsoft Azure services experienced an outage between 3:45pm UTC (9:15pm IST) on October 29 and 12:05am UTC (5:35am IST) on October 30. Customers and Microsoft services that use AFD experienced latencies, timeouts, and errors, the company explained on a support page.
Outage tracking platform Downdetector started tracking problems with Azure services at 9:17pm IST, and the complaints peaked at 9:47pm IST, with more than 16,000 reports. Issues were reported with both server connections and domains.
As per Microsoft, the affected services included App Service, Azure Communication Services, Azure Virtual Desktop, Microsoft Defender External Attack Surface Management, Microsoft Purview, and Microsoft Sentinel.
Soon after the outage was reported, Microsoft issued a statement on its X (formerly Twitter) page, saying, “We're investigating an issue impacting several Azure services. Customers may experience issues when accessing services”.
According to Microsoft, the outage occurred due to an inadvertent tenant configuration change within Azure Front Door (AFD). It is said to have triggered a widespread service disruption that affected not only Microsoft services but also customer applications that leverage AFD for global content delivery.
The configuration change is reported to have caused a “significant” number of AFD nodes to fail to load properly, as a result of an invalid or inconsistent configuration state. This led to increased latencies, timeouts, and connection errors for downstream services.
Microsoft further explained that the trigger was traced back to a faulty tenant configuration deployment process. The company acknowledged that its protection mechanisms, which are in place to validate and block any erroneous deployments, failed due to a software defect. It allowed the deployment process to bypass the safety validations that are in place.
The tech giant said that it immediately blocked future configuration changes once the outage was reported, to prevent additional propagation of the faulty state. It re-deployed the previous healthy configuration across its global fleet. The recovery process involved reloading configurations across several nodes and a gradual rebalance of traffic to mitigate overload conditions once normal services were resumed.
“This deliberate, phased recovery was necessary to stabilise the system while restoring scale and ensuring no recurrence of the issue”, the company explained on its support page.
However, customers' configuration changes to AFD are temporarily blocked, and they will be notified once it has been lifted. The company clarified that while the issue has been resolved, it may still persist for some users.
Microsoft said that it has reviewed safeguards and implemented additional validation and rollback controls to prevent similar issues in the future. It will also carry out an internal review to further understand the incident, the findings of which will be shared with impacted customers within 14 days, as part of a final Post Incident Review (PIR).
Get your daily dose of tech news, reviews, and insights, in under 80 characters on Gadgets 360 Turbo. Connect with fellow tech lovers on our Forum. Follow us on X, Facebook, WhatsApp, Threads and Google News for instant updates. Catch all the action on our YouTube channel.