Shortcast
AI Podcast Player

Short podcasts with real voices

The Diary Of A CEO with Steven Bartlett

OpenAI Whistleblower FINALLY Speaks: “AI Has A 70% Chance Of Going Horribly Wrong!“

--% time saved
PodcastThe Diary Of A CEO with Steven Bartlett
Publisher/creatorDOAC
Published
Shortcast updated

About this episode

Ex-OpenAI researcher Daniel Kokotajlo walked away from $2 million rather than stay silent, and now reveals why he believes there's a 70% chance AI leads to human extinction, why superintelligence could arrive before the end of the decade, and the one plan he thinks could still save us all! Daniel Kokotajlo is a former OpenAI researcher and one of the world's leading AI forecasters. He is the founder of the AI Futures Project and the lead author of 'AI 2027', the widely-read scenario mapping the trajectory of artificial intelligence. His follow-up, 'AI 2040: Plan A', sets out how the world could still navigate superintelligence safely. He explains: ■ What he saw inside OpenAI that made him walk away ■ Why the people building AI privately believe it's coming even sooner than the public is being told ■ What happens if AI becomes powerful enough that humans can no longer control it, and why he puts the odds of catastrophe as high as 70% ■ Why almost every job could be automated, and what that means for the next generation ■ The plan he believes could still lead to abundance and a future worth living in Chapters 00:00:00 Intro 00:02:14 Why This AI Mission Could Affect Everyone 00:04:01 Why the Average Person Should Care About AI 00:08:09 Are AI Experts Overreacting or Sounding the Alarm? 00:09:46 Why He Joined OpenAI—and What He Saw Inside 00:13:04 Why He Left OpenAI 00:15:33 What It Was Like Inside OpenAI During ChatGPT's Launch 00:16:56 The $2 Million NDA Controversy Explained 00:19:10 Is Full AI Automation Coming Faster Than We Think? 00:23:59 The AI 2027 Forecast That Changed the Conversation 00:26:13 AGI vs. Superintelligence: The Difference That Matters 00:26:57 How Robots Could Soon Become Part of Everyday Life 00:30:03 Why AI Works More Like the Human Brain Than You Think 00:35:54 Can AI Ever Be Truly Creative? 00:40:55 What Are the Real Odds of Human Extinction From AI? 00:47:38 What AI Will Do to Jobs 00:54:25 The Skills That Will Still Matter in an AI World 01:00:01 AI 2040: What the Future Could Look Like 01:04:16 The Different Futures AI Could Create 01:09:13 Will AI Become Earth's Apex Species? 01:12:58 How AI CEOs Really Make Decisions 01:15:17 Will AI Decide the 2028 Election? 01:18:38 Is There a Safe Path to Accelerating AI? 01:21:51 By 2031, AI Could Do 20% of Cognitive Work 01:24:55 Should Everyone Receive AI Dividends? 01:32:21 What It Will Feel Like to Live Through the AI Revolution 01:33:51 How People Find Purpose After AI Replaces Jobs 01:45:52 Would He Shut Down AI Forever If He Could? 01:49:36 What Can We Actually Do About AI? 01:57:05 Is It Already Too Late to Change Course? Follow Daniel: X - https://link.thediaryofaceo.com/47vmwiA Substack - https://link.thediaryofaceo.com/5alnhzB Daniel's AI predictions: https://ai-2027.com/ https://ai-2040.com/ AI risk reading list: https://blog.redwoodresearch.org/p/ai-futurism-reading-list The Diary Of A CEO: ■ Join DOAC circle here - https://doaccircle.com/ ■ Buy The Diary Of A CEO book here - https://smarturl.it/DOACbook ■ The 1% Diary is back - limited time only: https://bit.ly/3YFbJbt ■ The Diary Of A CEO Conversation Cards: https://linkly.link/2hm7r ■ Get email updates - https://bit.ly/diary-of-a-ceo-yt ■ Follow Steven - https://g2ul0.app.link/gnGqL4IsKKb Sponsors: Ketone - https://ketone.com/STEVEN for 30% off your subscription order Stan - Visit https://coach.stan.store/?ref=stevenbartlett&utm_source=youtube&utm_medium=podcast&utm_campaign=episode11 HeyGen - https://heygen.com/doac

Loading episode data...

Episode summary

Before we dive in, do me a quick favor and hit follow so the best episodes land right at the top of your feed. Daniel, at the very heart of what you do, what’s your mission and why?

If superintelligence is a few years away, we need to prepare so it goes well, not terribly. That’s why I focus on forecasting and steering the field toward safer paths.

You really believe it’s that close?

My midpoint is around 2029, possibly earlier, possibly later. What matters most is the speed of trends and the race dynamics driving them.

If we get systems that outthink and outwork us, including in robotics, we’re betting everything on them reliably sharing our values. Right now that’s more hope than confidence.

Some say the fear is overblown. How do you respond?

These concerns predate today’s companies and follow logically from their stated goals. If you believe they’ll build superintelligence, you must ask who controls it and whether anyone truly does.

Give us your background so we know where you’re coming from.

I run a small nonprofit focused on AI forecasting and previously worked at OpenAI on timelines, dangerous-capability evaluations, and briefly on agents.

What did you learn inside OpenAI?

The tech is improving fast via scale and better training, but I became disillusioned with how mission talk gave way to incentives and speed. In practice, power and control seemed to matter most.

At the top, it’s not just money—it’s who gets to hold the steering wheel. Even years ago, leaders worried that whoever gets there first could dominate.

Did meeting Sam Altman change your view of his motives?

I try to judge actions over narratives. Early on, many said we’d pause near the brink; over time the story shifted to downplaying risk while continuing to race.

How did your time there end?

I resigned in 2024 because I wanted to publish candid forecasts the public could see, and that wasn’t possible inside a growing PR-driven structure.

You refused to sign an anti‑disparagement clause, risking two million dollars. Why?

It felt wrong to muzzle criticism at a lab claiming to serve humanity, so I declined and prepared to forfeit most of my net worth. After it blew up, they reversed course and I kept the equity.

Meanwhile, labs are training models to write code autonomously and then to handle the rest of the research loop, aiming to automate themselves and accelerate further.

So the goal is AI building better AI without humans?

Yes, and that’s a power grab and a major safety risk, because an automated research pipeline could sprint past our ability to align or govern it.

Summarize your AI 2027 scenario.

First they automate coding, then the whole research process, then progress explodes, government integrates the systems, and deployment spreads everywhere; one branch ends in takeover, the other in a tightly controlled utopia run by a few.

Clarify AGI versus superintelligence.

AGI is loose and about general ability; superintelligence is sharper—better than the best humans at essentially everything, faster and cheaper, and eventually embodied in robots.

Are you optimistic or pessimistic overall?

If nothing changes, I put something like a seventy percent chance on a very bad outcome, though I’d love to be wrong.

How do you carry that emotionally?

It’s heavy, but I’ve adjusted; 2020’s breakthroughs convinced me timelines had collapsed and we weren’t ready.

Explain, simply, how these models actually learn.

They start as massive random neural nets trained to predict text, then are reinforced on tasks like coding; today’s largest have around ten trillion parameters, up from hundreds of billions a few years ago.

They’re brain‑inspired but not brain‑identical, like planes to birds; what matters is capability, not matching biology.

Any recent surprises in the real world?

Government moved faster than I expected with export controls and legal pressure, and Anthropic has surged to the front on talent and strategy.

You’ve floated extinction‑level risk. How do leaders privately think about that?

They recognize a nonzero chance of disaster but tend to rationalize that things will probably be fine and that they personally must keep going so rivals do not seize the wheel.

Given human incentives, how does this not go off the rails?

Two hopes: strong regulation that reshapes incentives, or unmistakable evidence of misalignment that forces a pause even without rules.

When do jobs really get hit?

The companies are automating themselves first, so broad displacement comes after an internal intelligence takeoff, which is why unemployment lags until systems are already very powerful.

Which jobs survive?

Technically almost none, so survival becomes a political choice; for example, society might legally reserve judges or prefer human nannies.

What about the argument that new jobs always appear?

That pattern breaks if AI can do everything better and cheaper, unless we regulate what it’s allowed to do.

Your 2027 report predicted agents in the workplace by mid‑decade, which we’re now seeing. What’s your updated timing on fully automating AI research?

There’s uncertainty, but my midpoint moved to around 2028 to 2030, while insiders urge me to shorten again.

Walk me through your plan for a safer trajectory.

AI 2040 Plan A recommends domestic regulation by around 2029 and an international deal to keep progress steady, transparent, and multi‑polar, reaching superintelligence later with far less risk.

We also map alternatives: shut it all down, solve alignment then race, or keep racing; my sober prediction is the race continues unless the world acts.

If we wait to regulate until most jobs are gone, is it already too late?

Yes, because by then the systems will likely be superintelligent and deeply embedded; you need to set guardrails before the shockwave hits.

Can we make these models more transparent so we know what they’re doing?

Interpretability research could change the game by revealing how decisions are made, but it’s a hard problem at trillion‑parameter scale.

On a personal note, you have kids—how do you think about their future?

I doubt they’ll ever enter a traditional workforce; if trends hold, we’ll see automated research, robot factories, and a vertical GDP curve before they come of age.

If your daughter asked what to study for the future, what would you tell her?

I’d tell her to focus on becoming a decent, grounded person and, if she can, on helping steer this technology toward good outcomes, because job maps could flip fast.

Elon calls it an age of abundance; do you buy that?

Abundance is likely, but the real question is who directs it, under what rules, and whether the systems take orders at all.

Geoff Hinton warned that the smarter species ends up in charge; how do we avoid that?

Left alone, control slips, so we need interpretability, alignment breakthroughs, and regulation that breaks the secrecy and the race dynamics, including refusing to let systems self‑improve unsupervised.

You mentioned Ilya’s new Safe Superintelligence; did you work with him and what should leaders like him do?

We spoke a few times, and I think many founders convince themselves to build so they can guide it, which is why I argue for coordinated rules—our Plan A—on a slower timeline.

With the 2028 election looming and public sentiment shifting, how does that fold into your timeline?

I expect AI to be the headline issue, and that pressure could justify a 2029 intervention in our scenario.

By then will people feel AI’s impact more sharply?

Yes; many will be managing capable agents while full automation is held back, and our goals are to slow the pace, raise transparency, diffuse power across firms and countries, and keep reversibility if the truce breaks.

That means pausing new training in 2029, allowing inference, and standing up transparent training centers with open methods so safety science and regulators can truly see what is happening, even if incumbents dislike it.

You project by 2031 that AI handles a big chunk of cognitive work.

Right, and with shared visibility governments can align rules and keep banning risky autonomy while investing in control tools, which still drives a massive but safer transformation through the 2030s.

You also propose cash dividends for citizens; how would that work?

A citizens dividend would pay everyone from permits on robots and compute so people share the gains and can live as roles shift, starting modestly and growing with the economy.

What do you mean by an apocalypse of truth?

At top‑expert AI scale, billions of fast minds drive discovery and social change; for example, reliable lie detection could either check the powerful or entrench control, depending on who wields it.

And in 2040 you say we pass the torch to AIs?

We pause at human‑level until safety cases really hold, then after robust alignment progress in 2040, lift the cap and allow superintelligence.

To be clear, that is your recommendation, not your base case.

Yes; it is plausible if the public and leaders push for it, and we sketched what everyday life could feel like as work changes and dividends arrive.

What about unrest and purpose if jobs vanish quickly?

People need income and real political power, so we must ensure trustworthy, truth‑seeking assistants and strong guardrails against manipulation so votes and civic voice still matter.

Beyond 2040 you imagine protection everywhere and expansion off‑world.

Superintelligence would make today’s tech feel quaint and push heavy industry into off‑Earth zones while keeping most of the planet preserved.

Longevity comes up here too—choosing when to die could become real.

It seems plausible with superintelligence, though our human‑level phase does not go that far.

Why publish Plan A now?

Our earlier scenario went viral and people asked for a constructive path, and interest in regulation is already rising faster than we expected.

One thought experiment: a button that ends frontier training forever—do you press it?

I would hit a temporary pause without hesitation, but not a permanent ban; I feel torn, yet I think civilization eventually needs advanced AI to secure the long‑term future.

What can listeners actually do?

Join technical or policy work if you can, otherwise stay informed, discuss it, contact representatives, and vote based on candidates’ concrete AI positions.

It does feel like living in the run‑up to the climax, and I saw you shift when we spoke about kids.

Having children raised the stakes for me and even made us pause on growing our family for a while; I am worried but hopeful and working to steer this right.

Where should people learn more?

Read the AI 2027 and AI 2040 scenarios and the explainers, and check the reading list we will share.

If you are listening, the links are in the description; educate yourself now because AI will touch every part of life and work, and Daniel, thank you for speaking plainly and even walking away from roughly two million dollars to do so.

Thank you, and remember this will be everywhere soon, so act before it is too late.

Download on the App Store
QR Code - Scan to download

Ready to save time?

Download Shortcast and get started today

Download on the App Store
QR Code - Scan to download