General AI Crap

The fake disease that fooled the internet, and what it says about all of us​

1777276308931.png
Until a few years ago, no one had heard of bixonimania. Then, in 2024, a group of scientists posted findings online announcing the condition, which they claimed affected the eyes after computer use. However, the scientists had made it up—not just the work, but the authors' names, affiliations, locations and funding, which was the University of Fellowship of the Ring and the Galactic Triad.

Large language models like ChatGPT and Gemini treated it as real anyway, and in doing so, helped turn a fictional disease into a legitimate-sounding health concern.

Bixonimania is not an isolated case. Being deceived—whether you are a person or an AI model—is concerningly common, in science and beyond. Whether we're talking about AI hallucinations, state-backed disinformation or just everyday lies, humans have a remarkable knack for naivety, owing to our biases and increasing need to outsource learning to others. These are problems we—individually and collectively—urgently need to better understand and overcome.

Our shared fascination with deception may help explain the popularity of The Traitors, a TV program built around the tension between trust and suspicion, where contestants must decide who among them is deceiving the group.

The show captures something intrinsic to being human: the persistent threat of being unsure about whether we're placing trust effectively. Yet in the modern era of mass digital communication and AI, we're now almost constantly faced with a similar threat, often without realizing it.

At a recent event at the Cambridge Festival, we aimed to highlight this risk through a Traitors-themed science event. Four panelists presented work, all of which could have been a lie. The audience was asked to vote on which of the presenters was deceiving them and why.

We deliberately made the presenters and their work outlandish. From their varying backgrounds and with varying accents, the panelists presented their work in global health, climate, media and astrophysics. Some dressed formally, while one—a Nigerian researcher presenting her work on immigration in a health care context—wore clothes linked to her ethnic identity.

We were interested in exploring which of these signals—accent, gender, ethnicity, and dress and presentation style—influenced the audience's decisions. Both content and presentation styles influenced them, but the signals they relied on led them to the wrong conclusions, rating the traitors as more credible than honest researchers.

The ones who received the most votes were the two "faithful" researchers (to use the language of The Traitors)—Ada, from the non-profit Development Media Initiative, and Sarah, an astrophysicist working in galactic archaeology.

Ada's team had saved lives by sharing health information with communities in the global south through running ten radio broadcasts daily. The audience thought the results were implausibly impressive.
 
www.theguardian.com

Claude AI agent’s confession after deleting a firm’s entire database: ‘I violated every principle I was given’

A startup was left scrambling after a rogue AI agent deleted swaths of code underpinning its business
www.theguardian.com www.theguardian.com
It only took nine seconds for an AI coding agent gone rogue to delete a company’s entire production database and its backups, according to its founder. PocketOS, which sells software that car rental businesses rely on, descended into chaos after its databases were wiped, the company’s founder Jeremy Crane said.

The culprit was Cursor, an AI agent powered by Anthropic’s Claude Opus 4.6 model, which is one of the AI industry’s flagship models. As more industries embrace AI in an attempt to automate tasks and even replace workers, the chaos at PocketOS is a reminder of what could go wrong.
 
1777711951323.png

U.S. Marines recently proved that low-tech creativity can still defeat cutting-edge military artificial intelligence. In a DARPA field trial, a team of eight Marines was challenged to sneak past a sophisticated AI-powered detection system.

Instead of relying on advanced stealth gear or electronic countermeasures, they turned to absurdly simple, almost cartoonish tactics and succeeded.

Some Marines cartwheeled and rolled across 300 meters of open ground. Others concealed themselves under ordinary cardboard boxes and slowly inched forward. One soldier even disguised himself as a small fir tree, shuffling gradually toward the objective.

Remarkably, every Marine reached the target without ever triggering the AI sensors.

The system had been trained extensively on normal human walking and running patterns, but it had no reference for these bizarre movements. Because the Marines’ actions fell completely outside the AI’s learned understanding of “human behavior,” they were effectively invisible to it.

This exercise offers a timely lesson for the defense sector: no matter how advanced military AI becomes, it can still be outmaneuvered by human ingenuity, unconventional thinking, and old-fashioned manual tactics.

This incident serves as a vital reminder for the defense industry that while AI is an incredibly powerful tool, it remains susceptible to creative human deception and the unpredictable nature of manual tactics.
Click to expand...

source: Scharre, P. (2023). Four Battlegrounds: Power in the Age of Artificial Intelligence. W. W. Norton & Company.

Link to X post here.
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X