How CrowdStrike bricked systems worldwide

Microsoft releases recovery tool to help repair Windows machines hit by CrowdStrike issue - The Verge​

IT admins now have a quicker way to get machines up and running again after CrowdStrike’s faulty update.

1721572152620.png

Microsoft has released a recovery tool that’s designed to help IT admins repair Windows machines that were impacted by CrowdStrike’s faulty update that crashed 8.5 million Windows devices on Friday. The tool creates a bootable USB drive that IT admins can use to help quickly recover impacted machines.

While CrowdStrike has issued an update to fix its software that led to millions of Blue Screen of Death errors, not all machines are able to automatically receive that fix. Some IT admins have reported rebooting PCs multiple times will get the necessary update, but for others the only route is having to manually boot into Safe Mode and deleting the problematic CrowdStrike update file.

Microsoft’s recovery tool now makes this recovery process less manual, by booting into its Windows PE environment via USB, accessing the disk of the affected machine, and automatically deleting the problematic CrowdStrike file to allow the machine to boot properly.

This avoids having to boot into Safe Mode or a requirement of admin rights on the machine, because the tool is simply accessing the disk without booting into the local copy of Windows. If a disk is protected by BitLocker encryption, the tool will prompt for the BitLocker recovery key and then continue to fix the CrowdStrike update.

Microsoft also has separate recovery steps available for Windows Virtual Machines running on Azure, and the company has also published recovery steps for all Windows 10 and Windows 11 devices over at its support site.

 

This isn’t the first time CrowdStrike’s CEO has been involved with a worldwide computer outage - BGR​

1721572736691.png

The faulty software update that led to massive outages around the world was so destructive, some cybersecurity experts marveled at how it took down a greater number of machines than any malware creator could have ever dreamed of.

The global disaster also sent IT teams around the world into crisis mode, since the mishap required a fix to be deployed on a machine-by-machine basis. A tedious recovery for a sudden crash.

What I’ve just described was, in fact, not anything recent — but, rather, a 2010 global Windows PC meltdown sparked at the time by a bungled McAfee update.

Of course, that incident certainly sounds quite similar to CrowdStrike’s routine update of its cybersecurity software on Friday, which ended up setting off a snowballing disaster that affected the computer systems of governments, banks, hospitals, media organizations, and much more around the world. Microsoft said the disaster affected 8.5 million Windows devices.

For the time being, I can’t even use my Starbucks mobile app to place orders as usual, that’s how pervasive the fallout from the CrowdStrike outage remains.

I didn’t bring up the McAfee situation, however, because of how similar it sounds to CrowdStrike’s. Instead, I simply want to point who was McAfee’s CTO at that time: George Kurtz, the man who’s now the CEO of — that’s right! — CrowdStrike.

Obviously, I’m not suggesting anything untoward here. At the very least, though, it’s pretty coincidental (at a minimum!) to be so adjacent to not one but two global computer system meltdowns.

“We’re deeply sorry for the impact that we’ve caused to customers, to travelers, to anyone affected by this,” Kurtz said in an appearance on NBC’s “Today.”

I’ll leave it to smarter people than me to draw any additional insights from the reality of the current situation. All I can say is that it raises certain questions in my mind about corporate governance and about the kind of executives the cybersecurity industry is producing.

And for more along these lines, I’d refer you to recent Glassdoor entries that blast things like that “Good pay, great product, iffy management” at CrowdStrike, as well as the fact that its “culture is toxic and can be damaging to one’s health.” Also: “KPIs driving behavior more than building relationships.”

For what it’s worth, Kurtz also reportedly left McAfee to launch CrowdStrike in response to what he perceived as the slow pace of evolution in security technology.

He’s also cited, as an inspiration for launching CrowdStrike, watching a passenger on a flight wait for 15 minutes for McAfee software to load on his laptop.

Was a “move fast and break things” attitude responsible for what happened yesterday?

Maybe. Given all of the above, it certainly sounds like Friday’s software update was probably rushed, had too few checks, and was the classic definition of an accident waiting to happen.

 

CrowdStrike has a new guidance hub for dealing with the Windows outage - BGR​

The company also warns about a spreading malware threat.

1721573305074.png

CrowdStrike has published a new “Remediation and Guidance Hub” that collects details related to its faulty update that crashed 8.5 million Windows computers across the globe on Friday.

The page includes technical information on what caused the outage, what systems are affected, and CEO George Kurtz’s statement. It contains links to Bitlocker key recovery processes and to various third-party vendor pages about dealing with the outage, as well.

The page points to a knowledge base article (which only logged-in customers can access) for using a bootable USB key. Microsoft released such a tool yesterday that automatically deletes the problematic channel file that caused machines to blue screen.

CrowdStrike also published a blog yesterday warning that threat actors have been taking advantage of the situation to distribute malware, using “a malicious ZIP archive named crowdstrike-hotfix.zip.”

The ZIP archive contains a HijackLoader payload that, when executed, loads RemCos. Notably, Spanish filenames and instructions within the ZIP archive indicate this campaign is likely targeting Latin America-based (LATAM) CrowdStrike customers.

Later in the blog:

Following the content update issue, several typosquatting domains impersonating CrowdStrike have been identified. This campaign marks the first observed instance in which a threat actor has capitalized on the Falcon content issue to distribute malicious files targeting LATAM-based CrowdStrike customers.

CrowdStrike says organizations should only be working directly with CrowdStrike’s representatives using official channels, and should use only the guidance its support team provides.

 
This is going to be such a problem for teams in the future:

Security Team: We really need to implement X
Business: But what about the risk of impact to systems, its not a good idea!
Security Teams: It tested, its got no impact and it is really needed to protect us.
Business: CrowdStrike July 2024
Security Teams: Silence.
Business: Silence
Everyone: Silence
 
This is going to be such a problem for teams in the future:

Security Team: We really need to implement X
Business: But what about the risk of impact to systems, its not a good idea!
Security Teams: It tested, its got no impact and it is really needed to protect us.
Business: CrowdStrike July 2024
Security Teams: Silence.
Business: Silence
Everyone: Silence
Nah the reality is, especially in regulated industries, if it can be showed the organisation didn't take every reasonable step to protect customers and data, C suite people go to jail.

It will be a very brave CEO or CIO or CTO that goes against the advice of his CISO when it comes to security.
 
And what does Capitec has to say about this?
Ah, sorry neh. Better luck next time.

I would not blame Capitec only for using MSWindows on their systems, but you have international banks that also had problems with backend, frontend...
 
Top
Sign up to the MyBroadband newsletter
X