{"id":478203,"date":"2023-01-30T11:14:01","date_gmt":"2023-01-30T09:14:01","guid":{"rendered":"https:\/\/mybroadband.co.za\/news\/?p=478203"},"modified":"2023-01-30T12:11:47","modified_gmt":"2023-01-30T10:11:47","slug":"why-microsofts-whole-cloud-was-broken-for-hours","status":"publish","type":"post","link":"https:\/\/mybroadband.co.za\/news\/cloud-hosting\/478203-why-microsofts-whole-cloud-was-broken-for-hours.html","title":{"rendered":"Why Microsoft&#8217;s whole cloud was broken for hours"},"content":{"rendered":"<p>Microsoft says the major hours-long global outage of its cloud services resulted from packet forwarding issues caused by a planned router IP address update.<\/p>\n<p>In a recent <strong><a href=\"https:\/\/status.azure.com\/en-us\/status\/history\/#:~:text=Preliminary%20Post%20Incident%20Review%20(PIR)%20%E2%80%93%20Azure%20Networking%20%E2%80%93%20Global%20WAN%20issues%20(Tracking%20ID%20VSG1%2DB90)\" target=\"_blank\" rel=\"noopener\">post-incident review<\/a><\/strong>, the tech giant explained\u00a0that the packet forwarding problem affected all routers on its Wide Area Network (WAN).<\/p>\n<p>&#8220;As part of a planned change to update the IP address on a WAN router, a command given to the router caused it to send messages to all other routers in the WAN, which resulted in all of them recomputing their adjacency and forwarding tables,&#8221; Microsoft said.<\/p>\n<p>&#8220;During this re-computation process, the routers were unable to correctly forward packets traversing them,&#8221; it explained.<\/p>\n<p>&#8220;The command that caused the issue has different behaviours on different network devices, and the command had not been vetted using our full qualification process on the router on which it was executed.&#8221;<\/p>\n<p>According to Microsoft&#8217;s outage report, customers began experiencing issues at 07:05 UTC (09:05 SAST). While most regions and services had recovered by 09:00 UTC (11:00 SAST), it only fully recovered its services by 12:43 UTC (14:43 SAST).<\/p>\n<p>&#8220;Between 07:05 UTC and 12:43 UTC on 25 January 2023, customers experienced issues with networking connectivity, manifesting as long network latency and\/or timeouts when attempting to connect to resources hosted in Azure regions, as well as other Microsoft services including Microsoft 365 and Power Platform,&#8221; it said.<\/p>\n<p>The tech giant said its networks started recovering automatically by around 08:10 UTC (10:10 SAST).<\/p>\n<p>However, the automated systems responsible for maintaining its WAN were paused due to the impact on the network.<\/p>\n<p>The incident also impacted Azure Government cloud services dependent on its public cloud.<\/p>\n<p>The worldwide outage occurred on Wednesday, 25 January 2023, and <strong><a href=\"https:\/\/mybroadband.co.za\/news\/cloud-hosting\/477607-huge-microsoft-outage-affecting-teams-outlook-microsoft-365-azure.html\">affected Microsoft Teams, Outlook, and Azure<\/a><\/strong>, in addition to the Microsoft 365 service.<\/p>\n<hr \/>\n<h2 class=\"my-4\">Now read: <a href=\"https:\/\/mybroadband.co.za\/news\/cloud-hosting\/476603-microsoft-adding-chatgpt-to-azure-cloud-service.html\" rel=\"bookmark\">Microsoft adding ChatGPT to Azure cloud service<\/a><\/h2>\n","protected":false},"excerpt":{"rendered":"<p>Microsoft has provided a preliminary post-incident review of the issue that took down its whole cloud infrastructure on 25 January 2023.<\/p>\n","protected":false},"author":341076,"featured_media":465441,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[25],"tags":[35,44440,24957,83965,39614,17198],"class_list":["post-478203","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloud-hosting","tag-headline","tag-microsoft-365","tag-microsoft-azure","tag-microsoft-outage","tag-microsoft-teams","tag-outlook"],"_links":{"self":[{"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/posts\/478203"}],"collection":[{"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/users\/341076"}],"replies":[{"embeddable":true,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/comments?post=478203"}],"version-history":[{"count":1,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/posts\/478203\/revisions"}],"predecessor-version":[{"id":478249,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/posts\/478203\/revisions\/478249"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/media\/465441"}],"wp:attachment":[{"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/media?parent=478203"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/categories?post=478203"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/tags?post=478203"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}