Cloudflare explains outage that took down many websites

mylesillidge

mylesillidge

Journalist
Joined
Jul 29, 2021
Messages
4,273
Reaction score
4,573
Internet outage that took down many South African websites caused by file that was too big

Cloudflare says its investigation into what caused Tuesday’s outage across its global network revealed that the problem was triggered by a database permission change that caused a critical file to double in size.

“The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind,” co-founder and CEO Matthew Prince said in a post on Cloudflare’s website.
 
“Instead, it was triggered by a change to one of our database systems’ permissions, which caused the database to output multiple entries into a ‘feature file’ used by our Bot Management system,” Prince explained.
Click to expand...
wtf_5.gif
 
  • Haha
Reactions: Jan
A system designed to be resilient and unbreakable then taken and put on the back of a few companies making is extremely susceptible to being taken down by one mistake. Yip, make perfect sense.
 
Bad Ballie said:
A system designed to be resilient and unbreakable then taken and put on the back of a few companies making is extremely susceptible to being taken down by one mistake. Yip, make perfect sense.
Click to expand...
The reason Cloudflare exists at all is DDoS attacks. If lone Internet hosts weren't so outgunned, they wouldn't be so dependent on DDoS mitigation providers like Cloudflare.
 
1763565488533.png


These cretins were boasting how amazing their switch to Rust was a few months ago

We weren’t starting from scratch. We’ve previously blogged about how we replaced another one of our legacy systems with Pingora, which is built in the Rust programming language, using the Tokio runtime. We’ve also blogged about Oxy, our internal framework for building proxies in Rust. We write a lot of Rust, and we’ve gotten pretty good at it.

We built FL2 in Rust, on Oxy, and built a strict module framework to structure all the logic in FL2.
.
Click to expand...


FL2 is also more secure by design than FL1. No software system is perfect, but the Rust language brings us huge benefits over LuaJIT. Rust has strong compile-time memory checks and a type system that avoids large classes of errors. Combine that with our rigid module system, and we can make most changes with high confidence.

Of course, no system is secure if used badly. It’s easy to write code in Rust, which causes memory corruption. To reduce risk, we maintain strong compile time linting and checking, together with strict coding standards, testing and review processes.
Click to expand...
https://blog.cloudflare.com/20-percent-internet-upgrade/


Overconfidence is the downfall of Rust devs IMO. It is not trivial to rewrite stuff into a language that is very different. This is why Microsoft made the very responsible decision to rewrite the Typescript compiler into Golang rather than Rust or their own languages. Golang most resembled the structure of old language.
 
I was going to ask whether this Bot Management is agentic-powered, but then I took a read, and well:

Cloudflare’s Bot Management includes, among other systems, a machine learning model that we use to generate bot scores for every request traversing our network. Our customers use bot scores to control which bots are allowed to access their sites — or not.
Click to expand...

There is a lot to dig into, since it is a descriptive blog post. The above is normal; I use a similar alternative. The question I have is: what triggered the permission change?

Queries to the distributed tables run through a shared system account. As part of efforts to improve our distributed queries security and reliability, there’s work being done to make them run under the initial user accounts instead.

Before today, ClickHouse users would only see the tables in the default database when querying table metadata from ClickHouse system tables such as system.tables or system.columns.

Since users already have implicit access to underlying tables in r0, we made a change at 11:05 to make this access explicit, so that users can see the metadata of these tables as well. By making sure that all distributed subqueries can run under the initial user, query limits and access grants can be evaluated in a more fine-grained manner, avoiding one bad subquery from a user affecting others.
Click to expand...

There is a process to these permissions and privileges. Someone decided to implement this as an improvement, but they don't explain that decision-making process. Symptoms, cause... This change was made live, knowing that the system has several limitations in place. I am curious whether this was agentic? They won't admit it, though.

Anyhow:

Remediation and follow-up steps​

Now that our systems are back online and functioning normally, work has already begun on how we will harden them against failures like this in the future. In particular we are:
  • Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input
  • Enabling more global kill switches for features
  • Eliminating the ability for core dumps or other error reports to overwhelm system resources
  • Reviewing failure modes for error conditions across all core proxy modules
Click to expand...

Not going to name them, but I know that there are cloud/CDN/mitigation providers that use AI proactively to make changes. Now I can't say whether AI is involved in the above, but AI doesn't always make the best decisions.
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X