{"id":530843,"date":"2024-04-02T15:31:39","date_gmt":"2024-04-02T13:31:39","guid":{"rendered":"https:\/\/mybroadband.co.za\/news\/?p=530843"},"modified":"2024-04-02T15:44:34","modified_gmt":"2024-04-02T13:44:34","slug":"a-superhero-hacker-saved-the-internet-this-weekend","status":"publish","type":"post","link":"https:\/\/mybroadband.co.za\/news\/security\/530843-a-superhero-hacker-saved-the-internet-this-weekend.html","title":{"rendered":"A superhero hacker saved the Internet this weekend"},"content":{"rendered":"<p>A long con social engineering attack that could have resulted in a large number of Internet servers being backdoored was detected and blocked this weekend thanks to the heroic work of one hacker.<\/p>\n<p>Postgres developer Andres Freund, who is currently employed by Microsoft, stumbled on the issue a few weeks ago when he noticed performance degradation \u2014 measured in milliseconds \u2014 in a core Linux tool.<\/p>\n<p>Freund noticed that Secure Shell (SSH), specifically OpenSSH, a tool used by just about every server on the Internet for secure administrative access, was performing poorly in a pre-release version of Debian.<\/p>\n<p>Debian is a widely used Linux distribution on which many others are based, including Ubuntu.<\/p>\n<p>Ubuntu, in turn, forms the basis for many other Linux distributions.<\/p>\n<p>After further digging, Freund determined that the issue was not with OpenSSH but with specific customisations Debian and other distributions patched into it.<\/p>\n<p>Fortunately, not all distributions or even Debian derivatives like Ubuntu use this customisation. However, many do.<\/p>\n<p>These customisations rely on a compression format called LZMA. One of this standard\u2019s widely-used open source implementations is the XZ project.<\/p>\n<p>Freund\u2019s <strong><a href=\"https:\/\/www.openwall.com\/lists\/oss-security\/2024\/03\/29\/4\" target=\"_blank\" rel=\"noopener\">analysis<\/a><\/strong> revealed that XZ\u2019s implementation of LZMA contained a backdoor that granted attackers some level of access to infected systems.<\/p>\n<p>At the very least, the backdoor allowed the attacker to remotely execute arbitrary code on a target machine.<\/p>\n<p>Although not full access or an authentication bypass, remote code execution can be as good as having total administrative control of a machine.<\/p>\n<p>Subsequent <strong><a href=\"https:\/\/bsky.app\/profile\/filippo.abyssdomain.expert\/post\/3kowjkx2njy2b\" target=\"_blank\" rel=\"noopener\">analyses<\/a><\/strong> of the attack suggest that the backdoor enabled remote code execution.<\/p>\n<p>The vulnerability is being tracked as <strong><a href=\"https:\/\/nvd.nist.gov\/vuln\/detail\/CVE-2024-3094\" target=\"_blank\" rel=\"noopener\">CVE\u20132024\u20133094<\/a><\/strong> with a CVSS score of 10 \u2014 indicating the highest possible severity.<\/p>\n<p>Freund\u2019s initial detective work also found that the backdoor was not in the XZ Project\u2019s source code \u2014 it only existed in already-compiled versions of the software that distributions like Debian used to create its packages.<\/p>\n<p>XZ used Github for version control, and it revealed that the malicious packages were all uploaded by the same person, \u201cJia Tan\u201d, with the username JiaT75.<\/p>\n<p>At this point, other hackers took the baton from Freund and investigated Jia Tan\u2019s activity on Github, revealing what appeared to be an incredibly sophisticated supply chain attack.<\/p>\n<p>Software developer Evan Boehs has published a <strong><a href=\"https:\/\/boehs.org\/node\/everything-i-know-about-the-xz-backdoor\" target=\"_blank\" rel=\"noopener\">summary<\/a><\/strong> of the investigation\u2019s findings so far.<\/p>\n<p>Boehs and other hackers found that Tan created his Github account in 2021 and submitted his first patch to the XZ Project in 2022.<\/p>\n<p>Tan became a regular contributor to the project, building trust over several months.<\/p>\n<p>XZ creator Lasse Collin, under increasing pressure to add more maintainers to the project to help speed up development, gave Tan greater privileges and responsibilities as he demonstrated commitment to the project.<\/p>\n<p>Tan pushed the final parts of the backdoor to GitHub on 23 February and 9 March 2024.<\/p>\n<p>Shortly after this, there was a flurry of activity by Tan and another unknown moniker, \u201cHans Jansen\u201d, to get the new versions of XZ included in various Linux distributions.<\/p>\n<p>According to <strong><a href=\"https:\/\/orca.security\/resources\/blog\/critical-xz-utils-supply-chain-compromise-affects-multiple-linux-distributions-cve-2024-3094\/\" target=\"_blank\" rel=\"noopener\">Orca Security<\/a><\/strong>, Tan got his backdoored code into pre-release versions of Debian and Fedora, and into OpenSUSE Tumbleweed.<\/p>\n<p>As a result of getting into Debian\u2019s unstable branch, the compromised version of XZ also found its way into the security-focused Kali Linux distribution.<\/p>\n<p>While the deception was discovered shortly afterwards, a lot of it was luck.<\/p>\n<div id=\"attachment_530857\" style=\"width: 1210px\" class=\"wp-caption aligncenter\"><a  data-lightbox=\"post-image\" href=\"https:\/\/mybroadband.co.za\/news\/wp-content\/uploads\/2024\/04\/Analysis-of-authentication-log-files-SSH.jpg\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-530857\" class=\"size-full wp-image-530857\" src=\"https:\/\/mybroadband.co.za\/news\/wp-content\/uploads\/2024\/04\/Analysis-of-authentication-log-files-SSH.jpg\" alt=\"\" width=\"1200\" height=\"800\" srcset=\"https:\/\/mybroadband.co.za\/news\/wp-content\/uploads\/2024\/04\/Analysis-of-authentication-log-files-SSH.jpg 1200w, https:\/\/mybroadband.co.za\/news\/wp-content\/uploads\/2024\/04\/Analysis-of-authentication-log-files-SSH-600x400.jpg 600w, https:\/\/mybroadband.co.za\/news\/wp-content\/uploads\/2024\/04\/Analysis-of-authentication-log-files-SSH-800x533.jpg 800w, https:\/\/mybroadband.co.za\/news\/wp-content\/uploads\/2024\/04\/Analysis-of-authentication-log-files-SSH-768x512.jpg 768w\" sizes=\"(max-width: 1200px) 100vw, 1200px\" \/><\/a><p id=\"caption-attachment-530857\" class=\"wp-caption-text\">Analysis of authentication log files<\/p><\/div>\n<h2 class=\"my-4\">The Internet dodged a bullet \u2014 but not because it was watching for one<\/h2>\n<p>In a post on <strong><a href=\"https:\/\/mastodon.social\/@AndresFreundTec\/112191135508298352\" target=\"_blank\" rel=\"noopener\">Mastodon<\/a><\/strong>, Freund explained that he wouldn\u2019t have spotted the issue had it not been for several coincidences.<\/p>\n<p>Firstly, he happened to be micro-benchmarking changes being implemented in Postgres, a relational database management system.<\/p>\n<p>He also happened to be running a pre-release version of Debian to ensure compatibility.<\/p>\n<p>He also happened to have seen a complaint about a related but separate issue a few weeks earlier, and happened to have an option enabled that caused that complaint to surface again during his benchmarks.<\/p>\n<p>\u201cJust to be clear: I didn\u2019t mean that I didn\u2019t do good \u2014 I did. I mean that we got unreasonably lucky here, and that we can\u2019t just bank on that going forward,\u201d he said.<\/p>\n<p>Senior Google engineer Damien Miller agrees. Miller has over 17 years of experience at Google, chiefly in information security.<\/p>\n<p>\u201cThis is the nearest of near-misses. Anyone who suggests this was any kind of success is a fool,\u201d Miller said on <strong><a href=\"https:\/\/cybervillains.com\/@djm\/112192731055910711\" target=\"_blank\" rel=\"noopener\">Mastodon<\/a><\/strong>.<\/p>\n<p>\u201cNo system caught this, it was luck and individual heroics. That\u2019s not acceptable when unauthorised access to [nearly] every server on the Internet is on the table. We need to find a way to do better.\u201d<\/p>\n<p>Miller said few of the software supply-chain defences people have been proposing would have prevented this compromise.<\/p>\n<p>\u201cThe attacker was a (relatively) long-term maintainer, was not averse to using sockpuppet accounts and was careful to hide their exploit from automated tools,\u201d he explained.<\/p>\n<p>\u201cWorse, many of the solutions being offered increase the workload on maintainers. But maintainer burnout was another key factor in this incident.\u201d<\/p>\n<div id=\"attachment_530851\" style=\"width: 470px\" class=\"wp-caption aligncenter\"><a  data-lightbox=\"post-image\" href=\"https:\/\/mybroadband.co.za\/news\/wp-content\/uploads\/2024\/04\/Damien-Miller.jpg\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-530851\" class=\"size-full wp-image-530851\" src=\"https:\/\/mybroadband.co.za\/news\/wp-content\/uploads\/2024\/04\/Damien-Miller.jpg\" alt=\"\" width=\"460\" height=\"460\" srcset=\"https:\/\/mybroadband.co.za\/news\/wp-content\/uploads\/2024\/04\/Damien-Miller.jpg 460w, https:\/\/mybroadband.co.za\/news\/wp-content\/uploads\/2024\/04\/Damien-Miller-400x400.jpg 400w\" sizes=\"(max-width: 460px) 100vw, 460px\" \/><\/a><p id=\"caption-attachment-530851\" class=\"wp-caption-text\">Damien Miller, senior staff engineer at Google<\/p><\/div>\n<p>Following Freund\u2019s disclosure of the backdoor, an <strong><a href=\"https:\/\/www.mail-archive.com\/xz-devel@tukaani.org\/msg00567.html\" target=\"_blank\" rel=\"noopener\">email<\/a><\/strong> sent by XZ Project creator Lasse Collin in 2022 has reignited discussion about how critical software infrastructure is built and maintained by volunteers whose conscientiousness often leads to burnout.<\/p>\n<p>In the email, Collin was responding to criticism from what is now suspected to be sockpuppet account created by the attackers.<\/p>\n<p>\u201cProgress will not happen until there is a new maintainer\u2026 Submitting patches here has no purpose these days. The current maintainer lost interest or doesn\u2019t care to maintain anymore. It is sad to see for a repo like this,\u201d a person calling themselves Jigar Kumar wrote.<\/p>\n<p>\u201cI haven\u2019t lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things,\u201d Collin replied.<\/p>\n<p>\u201cRecently I\u2019ve worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we\u2019ll see.\u201d<\/p>\n<p>Part of the sophistication of this supply-chain attack was that enemy hackers did not go after OpenSSH directly but instead targeted a weaker link in the chain.<\/p>\n<p>Based on the available evidence, they specifically targeted a project that had become an important dependency in critical security infrastructure where they could see a single vulnerable maintainer struggling.<\/p>\n<p>They then heaped more pressure on someone already struggling with their mental health to amplify the feelings that he wasn\u2019t living up to people\u2019s expectations or failing people who were depending on him.<\/p>\n<p>\u201cWe need to find a way to support maintainers without being proscriptive or parentalistic,\u201d Miller stated.<\/p>\n<p>Miller also predicted that this wouldn\u2019t be the last sophisticated and methodical open source software supply-chain attack.<\/p>\n<p>\u201cThe actor(s) behind XZ are probably already learning their lessons ahead of their next attempt,\u201d he said.<\/p>\n<p>Indeed, XZ might not be the only attack they had in progress, Miller warned.<\/p>\n<p>\u201cThe next one is going to be more carefully operated and harder to spot. How are we going to stop it?\u201d<\/p>\n<hr \/>\n<p><em>Author\u2019s Note: The term \u201chacker\u201d is used here in the word\u2019s original meaning. Freund is not an information security professional, but he is a highly skilled programmer, an enthusiastic tinkerer, and when he noticed something strange he did not rest until he understood what was happening.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Another hacker, believed to be a state actor, got frighteningly close to having a backdoor that would allow them to spy on just about any server on the Internet \u2014 and that\u2019s for starters.<\/p>\n","protected":false},"author":15,"featured_media":530847,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_sma_x_autopost_status":"idle","_sma_x_autopost_error":"","_sma_x_post_id":"","_sma_facebook_post_id":"","_sma_instagram_post_id":"","_sma_x_attempts":0,"footnotes":""},"categories":[27],"tags":[93317,25953,93323,2886,93327,7135,93331,35,93325,93329,1799,93321,32072,32224,30044,70563,93319],"class_list":["post-530843","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-security","tag-andres-freund","tag-backdoor","tag-damian-miller","tag-debian","tag-evan-boehs","tag-fedora","tag-hans-jansen","tag-headline","tag-jia-tan","tag-lasse-collin","tag-linux","tag-lzma","tag-openssh","tag-postgresql","tag-ssh","tag-supply-chain-attack","tag-xz"],"_links":{"self":[{"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/posts\/530843"}],"collection":[{"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/comments?post=530843"}],"version-history":[{"count":1,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/posts\/530843\/revisions"}],"predecessor-version":[{"id":530855,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/posts\/530843\/revisions\/530855"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/media\/530847"}],"wp:attachment":[{"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/media?parent=530843"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/categories?post=530843"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mybroadband.co.za\/news\/wp-json\/wp\/v2\/tags?post=530843"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}