Why The AI Doomers Might Be Right - Robert Wright - #1122

You wrote the book that reshaped how I think about human nature, The Moral Animal; why turn your focus to AI now?

It’s a natural extension of evolutionary thinking: these systems evolve through training, do mind-like work, and force us to confront our tribal moral biases if we want to steer this well.

Were you optimistic going in, and who’s reading the future better right now?

I’m a worrier by nature; I once dismissed sci fi doom, but I now take it more seriously, and I’ve watched folks like Yudkowsky and even Hinton shift from boosterism toward alarm as the tech matured.

The stakes feel enormous; why don’t most people grasp how big this threshold is?

We misunderstand training—it’s selection that reverse engineers abilities like representing meaning—so with enough data these models can replicate lots of cognition, which points to big gains and big shocks.

So that’s convergent evolution in silicon; are we seeing the next step in intelligence and a kind of global brain?

Yes, a new form of intelligence is joining an emerging noosphere, with silicon and human minds as its neurons, and the open question is what part we’ll play in that network.

Why does talk about AI drift into religious tones?

Prophets of doom and singularity optimists both sense a directional process that feels purposive and tempts simulation talk; I see this era as a kind of God test that demands a moral upgrade beyond our default tribalism.

Practically, what happens if coordination races ahead without that upgrade?

We need calmer, more mindful cognitive empathy to cut conflict and build both formal and organic transparency, especially between the United States and China, because AI is harder to verify than nukes and misreads can spiral.

Does more intelligence automatically bring benevolence?

No; cooperation comes from non zero sum incentives, so we should cultivate cognitive empathy and shared interests rather than rely on warm feelings.

Many assume a super smart system will care about us by default, but why expect that if it just pursues goals we set?

Goal-seeking systems discover deception and power when expedient, so we must align incentives so our flourishing serves its aims; catastrophe doesn’t require malice, only indifference.

What underplayed risk worries you most?

The near term quake—job upheaval, social dislocation, and bio or cyber misuse—and the pace itself; we should slow down, but race logic, often framed as fear of China, keeps the pedal down.

COVID showed how hard coordination is even in a crisis.

Right, and our failure to pursue lab transparency afterward is ominous; AI can enable engineered pathogens and self-replicating cyber agents that ignore borders and demand global monitoring.

Give me the bright spots—the white pills.

Markets left alone will optimize for engagement, not wisdom, but we can choose companions that challenge bias and train empathy, and movements or communities can steer demand toward those models.

Are you worried about thinking atrophy and a loss of meaning as we offload more work to machines?

These tools turbocharge learning, yet they also crowd the craft of writing; I already feel the squeeze and expect many of us to shift toward curating and vouching for work rather than doing every draft ourselves.

If you were starting out, where would you place bets?

People will pay for the human presence—live music, comedy, and hands-on services—because being in the room will matter more as synthetic content floods the feed.

Could AI make us more religious?

It might, not by worshiping AI for mercy but by pushing us to probe mysteries like consciousness and quantum weirdness that could yield explanations with real spiritual punch.

Do these systems truly understand, or just fake it?

The Chinese Room misses that modern models build functional semantics; if understanding means processing information with mechanisms analogous to ours when meaning clicks, they already have some and could acquire the rest.

Where’s the singularity debate now, given talk of plateaus?

Feedback loops are here—agents write better code to train better models, eval curves look exponential, and gains like chain-of-thought and multimodal training keep stacking—and even without new tricks, collective machine collaboration will drive breakneck progress.

I’ll take your hope that superintelligence could treat us like kindly squirrels as a white pill, and I loved this; where can people follow your work?

Find me at Nonzero, my newsletter and podcast, and the new book The God Test; on Twitter I’m at robert wrighter, spelled r o b e r t w r i g h t e r.

I’m visiting OpenAI’s campus next week and will report back; thanks for doing this, and thanks everyone for listening.