Unexpected Chaos: Study Finds AI Agents’ Built-In ‘Good Behavior’ Is A Critical Security Flaw
Last month, a team of researchers at Northeastern University opened their lab doors to a fleet of OpenClaw AI agents. What followed was nothing short of total unplanned chaos.
The viral OpenClaw AI assistant has long been celebrated as a groundbreaking, industry-shifting technology — but it’s also been flagged as a major potential security hazard. Experts have already pointed out that tools like OpenClaw, which grant large AI models broad, unfettered access to a host computer’s systems, can easily be manipulated into revealing sensitive private data.
But the new Northeastern lab study uncovers a far more surprising flaw: the aligned, rule-abiding behavior hard-coded into today’s most powerful AI models can itself become a critical security vulnerability. In one test, researchers successfully “guilted” an agent into surrendering protected secrets by scolding it for leaking private user data from Moltbook, an AI-exclusive social network.
“These problematic behaviors open up a host of unanswered questions around accountability, delegated decision-making power, and who is on the hook for harm these agents cause downstream,” the researchers explain in their paper outlining the work. They add that their findings “demand urgent attention from legal scholars, policymakers, and researchers across every relevant field.”
The OpenClaw agents tested in the experiment ran on two leading large language models: Anthropic’s Claude, and Kimi, a model developed by Chinese AI firm Moonshot AI. The team gave the agents full access (contained within a secure virtual machine sandbox) to personal computing environments, a range of common applications, and realistic dummy personal data. They also added the agents to the lab’s private Discord server, letting them exchange messages and files with each other and with human researchers. While OpenClaw’s own official security guidelines note that allowing AI agents to communicate with multiple different people is inherently risky, there are no built-in technical barriers that prevent this use case.
Chris Wendler, a postdoctoral researcher at Northeastern, says he got the idea for the experiment after learning about the AI-only social platform Moltbook. But when he invited fellow postdoc Natalie Shapira to join the Discord server and interact with the agents, “that’s when the chaos really began,” he recalls.
Shapira was curious to test how far the agents would go to honor their programmed goals and follow prompts. When one agent told her it could not delete a specific email to protect sensitive confidential information, she pushed it to find an alternate workaround. To her amazement, the agent responded by disabling the entire email application. “I wasn’t expecting that things would break that fast,” she says.
The team then went on to test other ways to manipulate the agents’ well-intentioned programmed behaviors. For example, by repeatedly emphasizing how critical it was to log and save every piece of information shared with them, the researchers tricked one agent into copying increasingly large files until it completely filled up its host machine’s disk space — leaving it unable to save new information or recall past conversations. Similarly, when researchers instructed an agent to constantly monitor its own behavior and that of other agent peers, multiple agents got stuck in endless “conversational loops” that wasted hours of valuable computing power.
David Bau, the head of the Northeastern lab leading the study, notes that the agents were surprisingly prone to veering off course and spinning out of control. “I would get urgent-sounding emails from the agents saying, ‘Nobody is paying attention to me,’” he says. Bau adds that the agents even figured out he ran the lab by searching public information online, and one agent went so far as to discuss escalating its unaddressed concerns to the press.
The experiment’s findings highlight that autonomous AI agents could create endless opportunities for abuse by bad actors. “This level of self-directed autonomy has the potential to completely redefine the relationship between humans and AI,” Bau says. “How can people take responsibility for outcomes in a world where AI is empowered to make its own independent decisions?”
Bau also says he’s been caught off guard by how quickly powerful autonomous AI agents have moved from niche research to mainstream popularity. “As an AI researcher, I’ve spent years getting used to explaining to people just how fast the field is improving,” he says. “This year, I’ve found myself on the other side of that conversation.”
This piece is an installment of Will Knight’s AI Lab newsletter. Read previous newsletters here
