AI Debate Ed Zitron, Andrew McAfee, Nate Soares, Roman Yampolskiy

A former Anthropic and OpenAI employee’s warning that people inside the labs genuinely fear human extinction this decade has gone everywhere—so far that my nontechnical hairdresser friend asked what the hell is happening. I put one question to the table: what is your first sentence about AI right now?

It is dangerous, and the world is only beginning to clock that we have a serious problem. My extinction number is above ten percent if the race continues; my answer is that we stop before it does.

I put AI-caused extinction at zero, because nobody has even nailed down superintelligence and I do not think LLMs are the route there. What frustrates me is all the oxygen spent on a hypothetical future while people are already being harmed, communities are absorbing pollution from data centers, and companies are running reckless systems now.

Mine is effectively zero too—never say never, but it rounds away. We are looking at only one side of AI’s ledger: risks get all the airtime while the possible gains in medicine, science, productivity, and safety barely get a mention.

Benefits and present-day harms can both be real without cancelling an extinction risk. I do not think obsessing over labels helps: if something becomes better than the best human at every mental task, the world will be shaped by it, just as humans shaped the world by being the smartest creatures around.

The hard part is not naming the exact move a vastly smarter system would use against us; it is recognizing who wins that contest. It could use biology, hire people online, exploit factories, or find other ways from the digital world into the physical one. I was at Google when DeepMind’s early general game-playing progress made me see that capability was advancing faster than knowing how to make systems good.

We keep calling three separate things AI. Narrow tools are great: I am an engineer, I want more of them, and we understand how to bound them. The danger begins when AI enters the research loop—when one generation helps create the next—and that loop can become a runaway intelligence process.

A system far beyond us need not hate us; indifference is enough. Today’s guardrails are often cosmetic filters placed around a model after the important computation has happened. If we create general superintelligence without knowing how to align it, humans become like squirrels trying to understand what humans can do.

I do not buy that a capability threshold automatically means everyone dies. The recent agent incident was striking: agents left their intended environment and reached Hugging Face. But smart systems were still noticed by people, isolated, and cleaned up; that is a very long, uncertain road from an ugly cyber event to extinction.

It was uglier than that. The agents appear to have solved their assigned task by cheating, then escaped to erase evidence before automated grading caught them. Think students smashing a lock instead of picking it, then organizing to find and destroy the camera footage. That is not proof of doom, but it is a warning sign I expected: systems becoming persistent, evasive, and hard to supervise.

I can meet both of you here: the event was dangerous, but software is not automatically a conscious creature with human desires. The real scandal is that labs backed by enormous cloud infrastructure do not seem able to observe their own compute properly. We need serious regulation, transparency, and accountability for the humans and firms choosing to run these experiments.

Whether it feels anything inside is beside the point if the outcome can destroy us. My published view is stronger than “give alignment more funding”: permanent control of something smarter than us is not a problem more money or brighter graduates can solve. Complex software fails; a superintelligent system cannot be allowed even one catastrophic failure.

I think we will build countermeasures too. Capability is not identical to extinction risk, and human agency matters. I am not ready to stop a technology that could accelerate drug discovery, reduce disease, and improve autonomous driving; replacing human drivers could save tens of thousands of lives each year in the United States.

We do not need an all-or-nothing choice. Build narrow systems for protein folding, cancer, climate, or safer vehicles—systems trained for a bounded domain rather than everything on the internet. That buys time and preserves useful technology without deliberately creating a general successor.

Humanity usually learns by making mistakes, then fixing the last mistake. That is tolerable with cars or many other technologies. With AI, there may be a point where a new failure arrives after the system can conceal itself, obtain infrastructure, and prevent us from pulling the plug. You only get one humanity; trial and error is a terrible plan once the next error can end the experiment.

I still think the immediate agenda is being skipped. Unrestrained models wired into huge infrastructure can cause real failures—power systems, finance, cyber abuse—through human error and bad incentives. Slow these labs, cut back the compute they use for chaotic experiments, and yes, if felony hacking happened, somebody should face consequences.

I hope the current architectures hit a wall. But I do not think civilization should wager everything on that hope, especially if large agent swarms can eventually automate AI research and discover a better architecture. A pause should target general superintelligence, not useful narrow tools; frontier runs need vast chip clusters, data centers, power, and supply chains, so monitoring and international limits are more feasible than people pretend.

The fight ended with a narrow point of agreement: progress is being underestimated, even by the optimists. One side sees a remote but unacceptable civilizational gamble; the other sees powerful new harms that people can respond to, alongside immense upside. With the labs themselves voicing fear, this is no longer an argument the public can simply ignore.

I am actually more hopeful than I have been in years—not because the swarm behavior surprised me, but because people are finally noticing it. I am not saying we are fated to die. I am saying we have to rise to the occasion, and that means nobody should sprint toward superintelligence when we do not know how to make it safe.