Eric Jang – Building AlphaGo from scratch

Today I’m with Eric Zhang, formerly VP of AI at Onex and before that at DeepMind Robotics; on sabbatical he’s been rebuilding AlphaGo from scratch. He’ll show us how it works and what it hints about AI, but first, why Go instead of the beach?

I love building things, and AlphaGo is what pulled me into AI because deep nets somehow tamed Go’s brutal search. With KataGo we saw approximately forty times less training compute, so what once needed a big lab now runs on a small budget with open tools.

This is a blackboard session, so video helps; let’s ground it in the rules.

You capture by cutting off orthogonal liberties, groups live or die by breath. Humans score by mutual agreement, but Tromp–Taylor makes it algorithmic; games end on a resignation or consecutive passes.

Go’s game tree is astronomically wide and deep, even with shared transpositions, so brute force is hopeless. AlphaGo uses Monte Carlo tree search guided by a learned policy prior and a value estimate.

Each node tracks a mean value, a prior, visit counts, and children; PUCT picks moves by balancing exploitation and exploration. We evaluate leaves with a value head, then back up averages to update ancestors.

So Q is the average win under the search’s sampling, and the exploration bonus fades as visits accrue, shifting from trying branches to trusting the best one.

Humans stop early because we carry an internal value function; a network can learn that shortcut and collapse long playouts into a quick judgment. That trims depth, while a strong policy prior prunes breadth.

The model is a shared trunk with two heads: a scalar value and a 19 by 19 policy. ResNets work well at small budgets; transformers help when you need global aggregation and lots of data, and KataGo mixes in global features to link distant fights.

I pretrain on expert games to predict next moves and outcomes; even that baseline beats many humans. Then MCTS runs per move: expand from the root with priors, traverse with PUCT, evaluate with the value head, back up means, and choose by visit counts.

You rebuild the tree after every move; a simulation is one more path down the tree. You toss most of it afterward, while keeping one statistic for training.

Early versions mixed in fast rollouts to the end, but later work dropped them; I do the same for speed. The visit distribution from search is a sharper target, so we train the policy to predict that directly and need fewer simulations next time.

Do gains taper as simulations increase, even after distillation?

I have not charted the exact shape, but strength rises monotonically; distillation mostly lifts the starting point.

MCTS can underperform if the value head is wrong or sims are few, so you must ground values; force some games to finish and add late‑game diversity. Bootstrapping on small boards or offline play helps the value head learn mid‑game judgment before heavy search.

Sharing a trunk for value and policy should be efficient since they ought to agree; it would be great to know the exact savings.

Later papers merged them into one network with two heads, likely for efficiency, and they should stay consistent. The deeper point is that a modest‑depth net can amortize a vast combinatorial search, much like AlphaFold compresses heavy physics.

Maybe worst‑case hardness is not what we meet in practice; structure helps. Weather is chaotic, yet we often care about coarse outcomes, not every microstate.

Right; you cannot predict the exact sequence of boards, but you can learn who wins in expectation. Chaos still has macro structure, like a Lorenz attractor, and networks seem powerful near that edge.

Training‑wise, MCTS gives a low‑variance label for every move, unlike naive self‑play that only upweights entire winning games. That straw‑like supervision is why LLMs favor one‑step RL with baselines and advantages to curb variance.

When search is infeasible, you can relabel moves by training best responses against fixed opponents and distilling them into a mixed strategy, as in neural fictitious self‑play. Q‑learning backs value through experienced steps, while MCTS plans over imagined ones; both push future estimates backward.

So Go lets us improve per move because MCTS plus a good value lets us learn locally, not only from full trajectories.

Do large language models already do a kind of tree search on their own, backing up when a path fails and trying a more promising one?

They often mimic that behavior without explicit trees, but classic Monte Carlo tree search fits games like Go because value is clear and branching is bounded; language is open‑ended, so common selection rules can be too greedy and rarely revisit the same choice, which breaks those heuristics.

So the real mismatch is huge action spaces and the lack of a clean value function, which makes local improvements less reliable for LLM reasoning and robotics.

That’s the rub today, though clever forward simulation could still return in new forms; some domains resemble tidy game trees, while negotiations or writing feel less like trees and more like fluid planning.

Andy Jones showed you can trade training compute for more test‑time search in board games; what fun scaling experiments still matter?

Test‑time scaling versus model size is deep: you can push more work into search or bake more into the forward pass, and the paper also forecasted compute for larger boards; I tried to build a compute‑savvy Go bot by hewing to scaling laws, but learned you first need a solid, bug‑free system and good data before those laws become reliable guides.

So your early scaling curves were misleading because the policy and labels weren’t solid.

Exactly; measuring laws on flawed data is cargo cult science, so get a working pipeline, then study how scaling behaves.

AlphaGo Zero used extraordinary compute compared to its era; you brought a strong bot online for a few thousand dollars—were they just inefficient?

Pioneering something always costs more than catching up; I bootstrapped with best‑response training against KataGo and am now probing the true from‑scratch step, while early teams had no opponent to lean on and didn’t need to squeeze every flop—similar to today’s robotics work, where capability wins first and compute optimality follows at scale.

How much of your compute efficiency comes from new methods versus things they could have done back then?

A lot comes from simplification and hardware: architectures matter less at these sizes, you can drop heavy distributed RL for a simple collect‑then‑train loop, modern GPUs are far faster, and if you warm up on nine‑by‑nine and transfer to nineteen‑by‑nineteen you cut the catch‑up period dramatically; many fancy tricks are unnecessary with a strong initialization.

Why is a replay buffer acceptable here when off‑policy usually bites you?

Off‑policy hurts if you train on states you’d never visit, but a buffer that mostly covers your trajectories plus nearby ‘recovery’ states is valuable, like DAgger; I even relabeled past positions with current MCTS to saturate GPUs, which worked if those states were reachable, mirroring robotics setups where a planner or Bellman updater provides better targets from old data.

Policy‑gradient RL seems data‑inefficient, especially as tasks get long‑horizon; you burn compute on full rollouts and get tiny signals until you stumble on a success.

Right, and because you sample from your own policy, if the winning action has near‑zero chance you may never learn; soft targets help a lot here since they carry more bits per sample, which is why distillation is so potent—AlphaGo trains on the full MCTS distribution, not just the chosen move.

This explains why MCTS‑guided training is elegant: you don’t start at zero success, you just climb stable supervised targets.

Exactly; it’s supervised learning on improved labels for both value and policy, which is stable, scalable, and sidesteps the barren‑signal problem.

You also ran an automated coding loop with LLMs; what’s working and what isn’t in automated research?

Today’s models are great at end‑to‑end experiment execution and aggressive hyperparameter and code‑level search, like rewriting data loaders or augmentations to beat a metric; they’re weaker at choosing the next high‑leverage question, spotting root‑cause bugs unprompted, or doing lateral resets, which is why I like Go as an outer loop where ideas are quick to verify while the inner loop trains an ‘automated scientist’ on reusable subtasks.

On the inner loop, how do you tell if a bad result is a bug or a bad idea, and how stackable are local wins without a top‑down plan?

Long‑horizon breakthroughs need intermediate signals, which is still an open design problem; meanwhile, many compute multipliers overlap and don’t stack cleanly, so taste matters—ask what the bitter lesson can buy you now, recognizing that as hardware improves, yesterday’s clever tricks often matter less.

What about outer‑loop verification for self‑improving AI—does progress on games transfer to economically useful work?

History suggests skills and infrastructure from games can transfer, but it’s hard even for humans to pick optimal research paths in real time; still, a clean outer loop like Go can guard against overfitting and validate ideas that might generalize.

Where should people go to try your system, read more, or fork the code?

Head to ebijang.com for the interactive tutorial and to my GitHub under Eric Chang for the auto‑go repo with reproducible training.

And check out the essay As Rocks May Think, which widens the lens on thinking as a computing primitive.

I’d love folks to explore the link between thinking and Go via search, and how that echoes in LLMs; you can do meaningful MCTS‑and‑reasoning research on small budgets, and there’s a deep, still underused duality waiting there.