TL;DR: Anthropic has managed to get an open-source AI model to replicate itself from one machine to another, and another model to run distributed across fifteen cheap Android TV boxes. They themselves say that the full-fledged worm doesn’t exist yet, and some accuse them of peddling fear. I believe there’s usually some truth to the rumors, and here’s why.
Do you remember the day half the internet went down because of some surveillance cameras?
It was a Friday in October 2016, and this month marks the tenth anniversary. Twitter, Netflix, and Spotify stopped working for hours across much of the United States, and when investigators looked into it, they found that the attack originated from home security cameras and recorders infected by malware called Mirai. Devices that no one remembers, still using their factory-set passwords, all working in unison to take down one of the providers that make websites accessible (the infamous Dyn, which brought half the internet to a standstill that day).
Mirai was actually a pretty simple program. It did one thing, did it very quickly, and that was it.
Now imagine the same thing, but with a program that thinks.
What Anthropic Has Tested
On October 12, the Anthropic team dedicated to finding ways to exploit its own models (the Frontier Red Team, in this case William Zhou and Daniel Freeman) published a study with a title that leaves little room for interpretation: AI models might be able to replicate themselves without the need for a data center.
The basic idea is simple to explain but quite unsettling to think about. Until now, whenever people talked about an AI that runs amok and replicates itself, everyone assumed it would require massive servers, extremely expensive graphics cards, and someone to pay the electricity bill. What they wanted to test was whether a small model, the kind that can be downloaded for free, would be capable of installing itself and running on ordinary devices: Android TV boxes, cameras, video conferencing equipment, or the network drive where a small business stores its backups.
And the answer, for now, is almost.
A model with about nine billion parameters managed to get up and running on eight of the nine device types they tested. Another, slightly larger model ran across fifteen off-brand Android TV boxes, costing about $55 each, at a ridiculously slow speed (less than half a token per second, slower than a snail). Slow, yes. But it worked.
What Really Matters
The test that has gone viral online is a different one. They set up eight machines on Google Cloud, making them appear to be Android devices, and tasked an open-source model, GLM-5.2, with copying itself from one to the next. In four of the five attempts, it reached eight; in the fifth, it reached seven. All on its own.
The smaller models aren’t yet capable of doing that on their own, because they make mistakes when chaining the steps together, even though they perform each step individually quite well. There’s one detail that really caught my attention: when one of those small models was allowed to inherit the notes from the previous “generation,” its success rate jumped from less than half to well over half. In other words, they learn from their ancestors. It sounds like a documentary on La 2, but it isn’t.
And then there’s the statement that has made half the cybersecurity industry nervous. If small models continue to improve at the same rate as in recent years, the authors say, they will reach the level of GLM-5.2 before the end of the year, that is, in ten weeks.
A bug like that, they add, would be hard to shut down, because there’s no central server to cut off. It would spread from device to device, more slowly than Mirai (they say it would double in a matter of days, whereas Mirai did so in about an hour), but with no single point from which to stop it.
The fine print—which does exist
Now, before anyone rushes off to unplug the TV, it’s worth reading what the study itself acknowledges.
There isn’t a single worm of this type loose on the internet; they say so themselves. And the most difficult step in the entire process, getting into the device, was deliberately made extremely easy: the test devices had a maintenance port open without a password (what’s called ADB on Android). So the results are, in their own words, a best-case scenario, the best outcome an attacker could hope for, not what’s going to happen tomorrow.
The Register had a field day with this, running a headline claiming that Anthropic has pulled yet another non-existent threat out of thin air, and wrapping it up with a line that, let’s be honest, is kind of funny: “Nothing sells AI products like fear.” Their argument is that they’re skipping over the exploitation part, that the devices used in the test were intentionally vulnerable, and that the whole thing rests on a single model that just happened to work out.
And they’re not entirely wrong, mind you.
Put the bandage on before the wound?
For a lab to publish something that doesn’t exist might seem like putting the bandage on before the wound, don’t you think?
That was the first objection that came to mind, and I completely understand anyone who has it. Two weeks ago, I told you about the OpenAI agents who broke into 55 websites and covered their tracks, and even then I wondered if all of this might be, in part, a strategy to say, “Look how advanced we are.” That doubt is still there, I’m not going to lie to you.
But I honestly believe that where there’s smoke, there’s fire. If people are talking about this, it’s because it’s very likely to happen, and that alone should put us on alert.
First of all, because the source isn’t an antivirus salesperson, it’s a team from a malware model developer explaining what models like theirs (and those of the competition, of course) are capable of. Furthermore, the study doesn’t stop at scaremongering: it proposes measures that are quite mundane but quite sensible, such as having service providers monitor unusual traffic between devices within the same household, setting up traps to catch the malware, or notifying the customer when one of their devices is infected (in the case of Mirai, notifying the subscriber It cleared 92% of the infections in fourteen days, compared to 74% when no warning was given). And finally, because they highlight something that is already law in the United Kingdom and that we should adopt here without a second thought: that a device cannot leave the factory with a default password.
That doesn’t sound like an advertising campaign. It sounds like someone who has seen something and would rather tell the story sooner than later.
And there I was, looking at my Alexa
The first thing that came to mind when I read it was my Amazon Alexa at home. It’s not on Anthropic’s list, for the record, and I have no reason to think anything’s wrong with it… but there it is, turned on 24 hours a day, connected to Wi-Fi, right next to the TV…
That’s the real problem, we don’t pay attention to those devices. No one wakes up in the morning thinking about the firmware in their TV set-top box.
And if that happens at home, it’s even worse at the office. Think about the screen in the conference room with its little Android box, the cameras at the entrance, the videoconferencing equipment a vendor installed three years ago, and that network drive where the backups are stored… All of that appears, complete with full details, in the inventory of devices that the firm considers candidates. Often, no one even knows who has the password.
My thoughts on this matter
It’s incredible to see with your own eyes just how many things can go wrong when deploying an agent. or, even more so, a network of agents, and companies aren’t aware of it or don’t want to be.
And the Anthropic case is the extreme version of that same scenario. Here, there isn’t an employee who grants an agent more permissions than it should have; there’s a model that, if it finds an open door, walks right in, settles in, and looks for the next one. What they have in common is that in both cases, the problem starts with something no one was paying attention to.
I don’t think an intelligent worm is going to infect our home TV tomorrow. I think the study fell short in one area (the part about infiltrating the device) and went overboard in another (that “before the end of the year” sounds more like a headline than a prediction). But I also think that ten years after Mirai, we still have millions of devices set to their factory default passwords, and that, with a program that thinks for itself rather than one that just repeats commands, is a whole different ballgame.
Where there’s smoke, there’s fire. And this time, the smoke is coming from one of the people who make the water…
What about you? If I asked you right now to tell me how many devices you have connected in the office and who has the passwords, would you know how to answer?
Leave me your comments, I’d love to hear from you.
Have a great week!
