I want to build a machine small enough that it has nowhere to hide.

Not a language model. Not a trained transformer. Just the smallest browser-sized object that makes one piece of in-context learning feel mechanical.

The trick is this:

... red moon fish ... red moon ?

If earlier in the prompt the pair red moon was followed by fish, then a useful next-token guess after the second red moon is fish. A model that does this is not retrieving a fact from weights. It is using the prompt as temporary memory.

Transformer Circuits calls one version of this behavior an induction head: an attention head that recognizes a repeated pattern and copies the token that came after the earlier occurrence.1 In its simplest slogan:

\[[A][B] \ldots [A] \rightarrow [B].\]

The slogan is almost too small. Good. Small enough means we can put it under glass.

Visible Lookup

The transformer paper introduced scaled dot-product attention as a way to map queries, keys, and values into a weighted average of values.2

\[\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V.\]

In a trained model, \(Q\), \(K\), and \(V\) are learned projections of residual stream activations. In this post, I will cheat. I will hand-code a tiny attention score so we can see the algorithm without waiting for training:

score(position j)
  = current-token match
  + prefix-token match
  + a small recency bias

The value at position \(j\) is the token after \(j\). If the current query is red moon, then positions where the prompt previously contained red moon receive high attention, and the value copied from that position is whatever came next.

This is not meant to be a faithful implementation of a particular model. It is a glass box version of a circuit motif. The Transformer Circuits framework makes this kind of decomposition useful by studying attention-only transformers and asking what algorithms individual heads can implement.3

Prompt Trapdoor

The lab below creates a synthetic sequence.

It first chooses a hidden rule:

prefix current -> target

For example:

red moon -> fish

Then it scatters several demonstrations of that rule into a longer stream. It also adds distractors: places where moon is followed by something other than fish. The query at the end is red moon, and the model must guess the next token.

Three predictors compete:

  • unigram prior: guess the most common token in the context;
  • copy current: find previous moon tokens and copy what followed them;
  • induction: find previous red moon pairs and copy what followed them.

The difference between the second and third predictor is the whole point. The current token alone can be ambiguous. The short prefix turns a noisy lookup into a more specific one.

Induction predictor Copy-current predictor Query tokens Attention mass

Deterministic synthetic experiment. Each setting evaluates 240 generated prompts and redraws one representative prompt with its attention weights.

Try breaking it.

Turn prefix strength to zero. The induction predictor becomes almost the same as the current-token copy rule, so distractors matter much more. Increase distractors. The current-token rule gets confused because the token after moon is no longer stable. Raise the temperature. Attention spreads out, and the copied answer becomes less decisive. Lower demonstrations to one. The circuit can still work, but it has no redundancy.

This is the feeling I wanted: in-context learning as a data structure. The prompt contains little records. Attention searches the records. The value path copies an answer out.

Keep the Claim Small

This toy circuit does not prove that all in-context learning is induction. It does not contain MLPs, learned embeddings, multiple layers, semantic features, optimizer dynamics, or long-horizon reasoning. Real models can do many things that this toy cannot.

Olsson et al. are careful about that distinction. They give strong causal evidence for induction heads in small attention-only models, and more correlational evidence in larger models with MLPs.1 Later work continues to debate which heads matter, when they matter, and how semantic variants of induction behave. The safe claim is not “this explains transformers.” The safe claim is “this is one concrete algorithm transformers can learn.”

That is still useful. Mechanistic interpretability becomes less mystical when the target is an algorithm one can implement in a few lines. A circuit is not a vibe. It should have inputs, intermediate quantities, outputs, failure modes, and ablations.

A Habit for Tiny Circuits

When a model succeeds on a few-shot prompt, ask a blunt question:

Which earlier token, span, example, or latent feature did the answer come from?

Sometimes the answer may be “nowhere obvious.” But when it is a copy, a lookup, a nearest-neighbor effect, or a repeated template, the answer should leave a trace. Induction heads are one trace: attention pointing to the earlier place where the current pattern already happened.

The next useful lab would not be larger for its own sake. It would add just one complication at a time: noisy embeddings, two competing rules, semantic aliases, multiple heads, then a learned version trained from scratch. The point is to keep the phenomenon visible while adding reality.

Small machines are good teachers. They do not explain everything. They explain one thing without hiding behind scale.

Paper Trail

  1. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah, “In-context Learning and Induction Heads,” 2022. Transformer Circuits, arXiv. ↩ ↩2

  2. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention Is All You Need,” NeurIPS 2017. arXiv. ↩

  3. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah, “A Mathematical Framework for Transformer Circuits,” 2021. Transformer Circuits. ↩