It’s advice as old as tech support. If your computer is doing something you don’t like, try turning it off and on again. When it comes to the growing concern that a highly advanced artificial intelligence system could become so catastrophically rogue that it could cause a risk to society, or even humanity, it is tempting to resort to this type of thinking. An AI is just a computer system designed by people. If it starts to malfunction, can’t we just turn it off?
- A new analysis from the Rand Corporation looks at three possible courses of action to respond to a “catastrophic loss of control” incident involving a rogue AI agent.
- All three possible responses—designing a “hunter-killer” AI to destroy the rogue, shutting down parts of the global Internet, or using a nuclear-initiated EMP attack to take out electronic devices—have mixed chances of success and carry a significant risk of collateral damage.
- The conclusion of the study is that we are woefully unprepared for the worst-case risks of AI and that more planning and coordination is needed.
At worst, probably not. This is not just because a highly advanced AI system might have an instinct for self-preservation and resort to desperate measures to save itself. (In versions of Anthropic’s large language model, Claude resorted to “blackmail” to preserve himself during pre-launch testing.) It’s also because malicious AI may be too widespread to disable. Current models like Claude and ChatGPT already run across multiple data centers, not on one computer in one location. If a hypothetical rogue AI wanted to avoid being shut down, it would quickly copy itself to the servers it has access to, preventing hapless, slow humans from disconnecting the connection.
In other words, killing a rogue AI might require killing the Internet, or much of it. And that is no small challenge.
This is the challenge that worries Michael Vermeer, a senior scientist at the Rand Corporation, the California-based think tank once known for its pioneering work on nuclear war strategy. Vermeer’s recent research has addressed the potential catastrophic risks of hyperintelligent AI, telling Vox that when these scenarios are considered, “people dismiss these wild options as viable possibilities” for how humans might respond without considering how effective they would be or whether they would create as many problems as they would solve. “Could we really do that?” he wondered.
In a recent article, Vermeer considered three of experts’ most frequently suggested options for responding to what he calls a “catastrophic AI loss of control incident.” He describes this as rogue AI that has excluded humans from accessing key security systems and created a situation “so threatening to the continuity of government and human well-being that the threat would require extreme actions that could cause significant collateral damage.” Think of it as the digital equivalent of the Russians letting Moscow burn to defeat Napoleon’s invasion. In some of the more extreme scenarios that Vermeer and his colleagues have imagined, it might be worth destroying a good portion of the digital world to wipe out the corrupt systems within.
In (arguable) ascending order of potential collateral damage, these scenarios include deploying other specialized AI to counter the rogue AI; “shut down” large portions of the Internet; and detonate a nuclear bomb in space to create an electromagnetic pulse.
You don’t come away from the article feeling particularly good about any of these options.
Option 1: Use an AI to kill her
Vermeer envisions the creation of “digital vermin,” self-modifying digital organisms that would colonize networks and compete with rogue AI for computing resources. Another possibility is so-called hunter-killer AI designed to disrupt and destroy the enemy program.
The obvious downside is that the new killer AI, if advanced enough to have any hope of accomplishing its mission, could go rogue. Or the original AI could exploit it for its own purposes. By the time we’re considering options like this, we may no longer care, but the potential for unintended consequences is high.
Humans don’t have a great history of introducing one pest to wipe out another. Consider the cane toads introduced to Australia in the 1930s, which never really did much to kill off the beetles they were supposed to eat, but killed many other species and continue to wreak environmental havoc to this day.
Still, the advantage of this strategy over the others is that it does not require destroying actual human infrastructure.
Vermeer’s article considers several options for shutting down large sections of the global Internet to prevent AI from spreading. This could involve manipulating some of the basic systems that allow the Internet to function. One of them is “border gateway protocols” or BGP, the mechanism that allows information to be shared between the numerous autonomous networks that make up the Internet. A BGP bug was what caused a massive Facebook outage in 2021. In theory, BGP could be exploited to prevent networks from communicating with each other and shut down swaths of the global Internet, although the decentralized nature of the network would make this complicated and time-consuming.
There is also the “domain name system” (DNS) that translates human-readable domain names, such as Vox.com, into machine-readable IP addresses and is based on 13 globally distributed servers. If these servers were compromised, it could cut off access to websites for users around the world and potentially our unauthorized AI as well. However, once again, it would be difficult to take down all the servers quickly enough to prevent the AI from taking countermeasures.
The document also considers the possibility of destroying the Internet’s physical infrastructure, such as the undersea cables through which 97 percent of the world’s Internet traffic travels. This has recently become a concern in the world of human national security. The alleged cable sabotage has disrupted Internet service on the islands surrounding Taiwan and on the Arctic islands.
But globally, there are simply too many cables and too many built-in redundancies for a shutdown to be feasible. This is a good thing if you’re worried about World War III destroying the global Internet, but it’s a bad thing if you’re dealing with an AI that threatens humanity.
Option 3: Death from above
In a 1962 test known as Starfish Prime, the United States detonated a 1.45-megaton hydrogen bomb 400 kilometers over the Pacific Ocean. The explosion caused an electromagnetic pulse (EMP) so powerful that it knocked out street lighting and telephone service in Hawaii, more than 1,000 miles away. An EMP causes a voltage surge powerful enough to burn out a wide range of electronic devices. The potential effects in today’s world, much more dependent on electronics, would be much more dramatic than in the 1960s.
Some politicians, such as former House Speaker Newt Gingrich, have been warning for years about the potential damage that an EMP attack could cause. The issue was in the news again last year, thanks to US intelligence that Russia was developing a nuclear device to launch into space.
Vermeer’s article imagines the United States intentionally detonating warheads in space to cripple Earth’s telecommunications, energy, and computing infrastructure. It is estimated that between 50 and 100 detonations in total could be needed to cover the United States landmass with a pulse strong enough to do the job.
This is the ultimate blunt tool where you want to be sure that the cure is not worse than the disease. The effects of an EMP on modern electronics (which may include surge protection measures in their design or may be protected by buildings) are not well understood. And in the event that AI survived, it would not be ideal if humans had crippled their own power and communications systems. There is also the alarming prospect that if other countries’ systems are affected, they might retaliate against what would, in effect, be a nuclear attack, no matter how altruistic their motivations.
Given how unappealing each of these courses of action is, Vermeer is concerned about the lack of planning he sees from governments around the world for these scenarios. However, he notes that only recently have AI models become smart enough that policymakers have begun to take their risks seriously. He points to “smaller cases of loss of control of powerful systems that I think should make it clear to some decision makers that this is something we should prepare for.”
In an email to Vox, AI researcher Nate Soares, co-author of the controversy-inducing best-seller, If someone builds it, everyone diesHe said he was “encouraged to see elements of the national security apparatus beginning to get involved in these thorny issues” and broadly agreed with the article’s conclusions, although he was even more skeptical about the feasibility of using AI as a tool to keep it under control.
For his part, Vermeer believes that an extinction-level AI catastrophe is a low-probability event, but that loss-of-control scenarios are likely enough that we should be prepared for them. The conclusion of the article, as far as it is concerned, is that “in the extreme circumstance that a globally distributed, malevolent AI exists, we are not prepared. We are left with only bad options.”
Of course, we must also consider the old military maxim that in any question of strategy, the enemy gets one vote. All of these scenarios assume that humans were to retain basic operational control of government and military command and control systems in such a situation. As I recently reported for Vox, there are reasons to worry about the introduction of AI into our nuclear systems, but AI actually launching a nuclear bomb, at least for now, probably isn’t one of them.
Still, we may not be the only ones planning ahead. If we know how bad the available options would be for us in this scenario, the AI will probably know too.
This story was produced in partnership with the Outrider Foundation and Journalism Funding Partners.