Beyond the Model

Each next token depends on the context so far.

The context changes with every generated token—so the next prediction changes, too.

Attention weights mix information within the model; they are not the final next-token probabilities.

Explore: 3Blue1Brown · Attention, step by step · Transformer Explainer · interactive model

Memorization and generalization can arrive at different times

Animation: Pearce et al., “Do Machine Learning Models Memorize or Generalize?” · phenomenon: Power et al., “Grokking.”

Examples in the prompt teach a temporary task

No retraining: the examples shape behavior only while they remain in the prompt.

In-context learning: Brown et al., “Language Models are Few-Shot Learners” · Garg et al., “What Can Transformers Learn In-Context?”

An induction head finds a pattern and continues it

An attention head is a small part of a transformer that decides which earlier words are useful for predicting the next one.

Mechanistic evidence: Olsson et al., “In-context Learning and Induction Heads.”

Tool calls request execution outside the model.

A calculator tool can be an ordinary local Python function—no HTTP request required.

Weather API: the same request–execution–result loop, with a remote service instead of local arithmetic.

Tool semantics: Anthropic · Tool use overview · Handling calls and results · calculator example is schematic.

The harness manages what reaches the next model call.

The model can choose retrieval and tool calls; the harness implements the mechanisms and policies.

Design synthesis: Anthropic · Context engineering · Zhou et al. · Externalization, §6.

Agentic RL trains over trajectories—not single responses

Having tools differs from learning to use them well. Evaluate whether goals and evidence survive across steps.

Loop foundation: Microsoft, MAI-Thinking-1, Fig. 18 / §3.3, p. 40 · continuity examples added here, not Microsoft implementation claims.

Distinguish three RL learning targets

A capability is available machinery; a learned policy decides how to use it. Memory behavior can be part of the model policy.

Agent Lightning v1.0 · MemexRL · Offline harness-control RL · separate research settings, not a single combined system.

Evaluate outcomes and continuity

METR · definition and methodology · Apple · LOOP · Chen et al., §5 / App. A, E.

CoALA: organizing a language agent as cognitive architecture

A cognitive architecture specifies how memory, reasoning, action, and decision-making work together within an agent.

Definition: our synthesis. Diagram adapted from Sumers et al., CoALA, Figs. 4–5 and §4. A conceptual framework for language agents.

From cognitive functions to externalized machinery

Zhou et al., §1–2, 6–7 names CoALA its closest conceptual bridge. Comparison is our synthesis.

Capability spans weights, context, and runtime

Zhou et al., §2 · Teaching summary.

Memory types preserve different kinds of value

CoALA, §4 · Zhou et al., §3 · Taxonomy synthesis.

Preserve experience; consolidate what it teaches

Tang et al., WikiSkill, §3 / Fig. 2 · Memory interpretation is ours.

Reusable skills need an acceptance test

Zhou et al., §4 · WikiSkill, §3.2 / Algorithm 1.

A tool call needs more than argument types

Zhou et al., §5 · Report-search example is ours.

The harness makes the pieces work together

Zhou et al., §6.1–6.4 · Three-question grouping is ours.

One task connects memory, skills, and feedback

Design question: what should persist externally, and what will selection, loading, and maintenance cost?

Zhou et al., §7.1–7.3 · WikiSkill · Example is ours.

An RLM operates on context as data

Symbols refer to objects
Reference large objects without copying them into context.

Subcalls are functions
Delegate bounded inspection; return selected findings.

The model chooses the path
Code filters and computes; the controller sees selected evidence.

Original diagram: Zhang and Khattab, RLM introduction, Fig. 1 · Madura, “It’s Tokens All The Way Down”.

A small experiment tests document scaling

20 sampled questions · 10–1,000 documents · supporting evidence included.
Perfect performance at 1,000 documents applies to this subset, not the full benchmark.

Original chart: Zhang, Fig. 5 · Benchmark: Chen et al., BrowseComp-Plus. Historical API costs.

Prime Agent learns through harness refinement

Karten et al., Prime Agent, §2.2 / §2.5 · Connection to WikiSkill is our synthesis.

Review a data room; produce a diligence memo

Data room: a company’s document collection for an acquisition review.
Hard part: connect scattered evidence, notice missing support, and produce a cited assessment.

Tool-loop → RLM harness: 23.3% → 62.4%
Mean rubric criteria pass rate across seven models · 50 held-out synthetic data rooms.

Original diagram, Fig. 5, and reported results: Harvey, “Post-Training RLM Agents for End-to-End M&A Diligence.”

Training improves root coordination

Train the root with RL; hold subagent models fixed.
Matched base and trained checkpoint · 50 held-out synthetic data rooms · reported by Harvey.

Harvey, Reinforcement Learning · Fig. 12–13 / Table 2. Behavioral changes accompany the gain; they do not isolate its cause.

Induce patterns, propose explanations, derive predictions

Investigation loop: observe → propose explanations → derive predictions → check evidence → revise.