{"id":619005,"date":"2025-11-19T10:06:19","date_gmt":"2025-11-19T08:06:19","guid":{"rendered":"https:\/\/mybroadband.co.za\/news\/?p=619005"},"modified":"2025-11-19T15:34:44","modified_gmt":"2025-11-19T13:34:44","slug":"internet-outage-that-took-down-many-south-african-websites-caused-by-file-that-was-too-big","status":"publish","type":"post","link":"https:\/\/mybroadband.co.za\/news\/internet\/619005-internet-outage-that-took-down-many-south-african-websites-caused-by-file-that-was-too-big.html","title":{"rendered":"Internet outage that took down many South African websites caused by file that was too big"},"content":{"rendered":"\n

Cloudflare says its investigation into what caused Tuesday\u2019s outage across its global network revealed that the problem was triggered by a database permission change that caused a critical file to double in size.<\/p>\n\n\n\n

\u201cThe issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind,\u201d co-founder and CEO Matthew Prince said<\/a> in a post on Cloudflare\u2019s website.<\/p>\n\n\n\n

\u201cInstead, it was triggered by a change to one of our database systems’ permissions, which caused the database to output multiple entries into a \u2018feature file\u2019 used by our Bot Management system,\u201d Prince explained.<\/p>\n\n\n\n

\u201cThat feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.\u201d<\/p>\n\n\n\n

Several of South Africa\u2019s largest websites were impacted by the incident on 18 November, including BusinessTech, Daily Maverick, IOL, and MyBroadband.<\/p>\n\n\n\n

Other local services were also down, including the SABC\u2019s streaming service, SABC+, and various radio station websites such as Jacaranda FM, East Coast Radio, and Primedia+ (702 and CapeTalk).<\/p>\n\n\n\n

Outage tracker Downdetector was offline, as were various AI platforms, including OpenAI\u2019s ChatGPT and Anthropic\u2019s Claude.<\/p>\n\n\n\n

Twitter\/X was also impacted. Its website loaded, but feeds would not populate with tweets. Users also couldn\u2019t post to the social media network.<\/p>\n\n\n\n

Prince explained that the software running on Cloudflare\u2019s machines to route traffic across its network reads the \u201cfeature file\u201d to keep its Bot Management system up to date with ever-changing threats.<\/p>\n\n\n\n

\u201cThe software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail,\u201d he said.<\/p>\n\n\n\n

Cloudflare\u2019s Bot Management system is a tool that enables website owners to stop harmful bot activity without relying on something like CAPTCHA, which adds friction and degrades the user experience.<\/p>\n\n\n\n

Prince said the Bot Management system relies on a machine learning model that Cloudflare uses to generate bot scores for every request traversing its network.<\/p>\n\n\n\n

\u201cOur customers use bot scores to control which bots are allowed to access their sites \u2014 or not,\u201d he said.<\/p>\n\n\n\n

Cloudflare uses data from the millions of Internet properties using its services to help train the machine learning model and make it more accurate.<\/p>\n\n\n\n

The machine learning model takes a \u201cfeature file\u201d as input. A feature, in this context, refers to an individual trait used by the model to estimate the probability that a request was automated or not.<\/p>\n\n\n\n

The feature configuration file is a collection of individual features, which is refreshed every few minutes and published to Cloudflare\u2019s entire network.<\/p>\n\n\n\n

\u201cIt allows us to react to new types of bots and new bot attacks. So it\u2019s critical that it is rolled out frequently and rapidly as bad actors change their tactics quickly,\u201d said Prince.<\/p>\n\n\n\n

Outage initially misdiagnosed<\/h2>\n\n\n\n
\"\"<\/a>
Volume of 5xx error HTTP status codes served by the Cloudflare network. Normally, this should be very low, and it was right up until the start of the outage.<\/figcaption><\/figure>\n\n\n\n

Part of the reason Cloudflare took so long to resolve yesterday\u2019s outage was that it had initially misdiagnosed the problem as a cyber attack.<\/p>\n\n\n\n

Microsoft Azure and Cloudflare itself had recently suffered attacks from the Aisuru botnet, with Microsoft describing it as a Turbo Mirai-class IoT botnet that frequently causes record-breaking DDoS attacks.<\/p>\n\n\n\n

Aisuru exploits compromised home routers and cameras, mainly in residential ISPs in the United States and other countries.<\/p>\n\n\n\n

\u201cThrowing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare\u2019s status page went down,\u201d said Prince.<\/p>\n\n\n\n

\u201cThe status page is hosted completely off Cloudflare\u2019s infrastructure with no dependencies on Cloudflare.\u201d<\/p>\n\n\n\n

While it turned out to be a coincidence, it led people diagnosing the issue to believe that an attacker may be targeting Cloudflare\u2019s systems as well as its status page.<\/p>\n\n\n\n

\u201cAfter we initially wrongly suspected the symptoms we were seeing were caused by a hyper-scale DDoS attack, we correctly identified the core issue,\u201d said Prince.<\/p>\n\n\n\n

Once the issue was identified, Cloudflare was able to stop the propagation of the larger-than-expected feature file and replace it with an earlier version.<\/p>\n\n\n\n

\u201cCore traffic was largely flowing as normal by 14:30 (16:30 SAST). We worked over the next few hours to mitigate increased load on various parts of our network as traffic rushed back online,\u201d Prince said.<\/p>\n\n\n\n

\u201cAs of 17:06 (19:06 SAST), all systems at Cloudflare were functioning as normal.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"

Cloudflare says its investigation into what caused Tuesday\u2019s outage across its global network revealed that the problem was triggered by a database permission change that caused a certain critical file to double in size.<\/p>\n","protected":false},"author":15,"featured_media":619006,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[18,27],"tags":[723,27543,29694,37230,123,24957],"class_list":["post-619005","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-internet","category-security","tag-amazon","tag-amazon-web-services-aws","tag-cloudflare","tag-matthew-prince","tag-microsoft","tag-microsoft-azure"],"_links":{"self":[{"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/posts\/619005"}],"collection":[{"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/comments?post=619005"}],"version-history":[{"count":3,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/posts\/619005\/revisions"}],"predecessor-version":[{"id":619096,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/posts\/619005\/revisions\/619096"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/media\/619006"}],"wp:attachment":[{"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/media?parent=619005"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/categories?post=619005"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/tags?post=619005"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}