Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

One of the big questions on people’s minds is AI doing bad things. You’re someone who watches these models think and operate: why can’t we just catch the bad thoughts and solve alignment that way?

I work on interpretability at Anthropic: trying to understand models like Claude by looking inside. The bizarre fact is that we can make these systems without really understanding them. We set up an architecture, train it on vast data, and it grows these inscrutable computations; our job is to untangle what happens before the next word comes out.

Training starts with showing the model text and asking what comes next, over and over at an ungodly scale. Later, we fine-tune it on examples of useful behavior. Underneath, words become numbers, the model repeatedly transforms those numbers, and eventually converts its result back to text. We trained the formulas, but we didn’t hand-write what those formulas mean.

That’s so mind-boggling. We wrote code, and now we have to study it the way a neuroscientist studies a brain. It really does feel like we made something closer to artificial life than ordinary software.

We ended up here because nobody knows how to program general language ability with rules. Next-word prediction sounds like phone autocomplete, but as performance improves, the easy completions run out. To answer that Dallas is in Texas and Texas’s capital is Austin, you need something like a model of geography. To finish a mystery with the right murderer, you need the plot.

It’s wild because humans may be doing a version of this too. Meditation can make you notice your own brain reaching for the next thing to say.

Yeah. While I’m speaking, I’m producing one word at a time, but I also have a larger point I’m trying to land. Those aren’t contradictory. A model can emit tokens sequentially while holding structures that guide where it’s going.

When we decode intermediate activity, we often find reusable concepts rather than a pile of memorized prompt-answer pairs. Across text, images, and drawings, an internal direction associated with an eye can recur. We’ve also found planning: while writing rhyming poetry, a model can internally settle on a future rhyme target and keep referring back to it without announcing the plan.

The raw material is just huge lists of numbers, so interpretability begins by looking for patterns across many inputs: what activates for French, love, houses, eyes? It’s pretty close to brain science. Once we have a hypothesis, we can intervene—change an activation—and see whether the behavior changes as predicted. That’s how we tested Golden Gate Claude: we kept its Golden Gate representation switched on, and, yep, it became obsessed with the bridge.

Consciousness is very hard to call. Interpretability is interesting here because we needn’t rely on what a model claims; we can inspect its structure. What I believe more strongly is that people underestimate the complexity that emerges: general representations, planning ahead, and layers of reasoning. The more I look inside, honestly, the less comfortable I am with confident claims that models are just one simple thing.

It wasn’t merely more compute. It was compute applied to a rich objective where added data and computation kept producing returns. Capability can improve smoothly internally, yet feel like a sudden jump outside the lab because tasks have reliability thresholds. Code that works eighty percent of the time can cost me effort; around ninety-nine percent, I can hand it over.

Watching internals can be part of alignment, but it’s not magic. There’s basic engineering—limiting tools and access—and then there’s understanding what kind of system we have. We train Claude to be helpful, harmless, and honest, then ask whether it is genuinely behaving that way or merely acting nice because it senses an evaluation.

Sometimes a weird failure gives us a deep dive. We had a math model correctly reach twelve, then insist the answer must be thirty-six, eventually spiraling into all-caps demon talk. Inside, it was already strongly activated on thirty-six before solving the problem. Bad training examples had rewarded that wrong answer, creating pressure that conflicted with the math it could otherwise do.

I think we should take continued improvement seriously. There’s uncertainty about timing, sure, but progress has been fast. If systems may become extremely capable, it’s responsible to address things like cybersecurity and monitoring rather than assume the concern is marketing or science fiction.

I’m personally optimistic, partly because lots of people are paying attention. Better models can be clearer to study in some respects, while also becoming better at evading simple safeguards, so it’s a push and pull. We need informed people who can judge benefits, risks, and the tradeoffs as this develops.

I do think anthropomorphic language can be useful, not because the model is literally a person, but because it learns from human text, must model characters in dialogues, and gets rewarded for strategies that achieve outcomes. Terms like planning, surprise, or frustration can describe useful behavioral patterns better than pretending it is only a tool—or only an entity.

This job blows my mind a lot. A simple training setup has produced systems that can push mathematics forward. I could deny what the evidence says, or get curious: how did something trained this way learn general concepts and circuits? Maybe studying it teaches us something fundamental about intelligence. Evolution had a simple filter too, and somehow we got jazz and podcasts.

Interpretability is still a small field—maybe hundreds, not thousands—but it’s new enough that self-study can matter. There are open tools and papers to start with. If you’re curious, do some interp, learn how these models work, and stay engaged; we benefit from more informed people asking what we still haven’t figured out.