Research memory that compounds

Extracting sources into a persistent, durable and queryable
llm-wiki

Chris Sweet

Charles Vardeman

Priscila Moreira

James Sweet

August 20, 2026

What an llm-wiki is

The setup, and the alternative

• LLMs are trained on a vast array of human knowledge. But science is a broad landscape, each domain its own deep dive; incorporating all of it into a model is challenging.
• The training data is stale by design. Recent advances are simply not in it.
• Harnesses now add “project data”, files you upload as the basis for a project. Useful, but it is a RAG approach: nothing persists, and the context is re-extracted from scratch on every question.
• llm-wiki, proposed by Andrej Karpathy, is the alternative: dense, connected wiki pages as a persistent memory, authored and maintained by the LLM itself.
• Visible and queryable by the human, so the memory stays trusted and auditable, not a black box.

A persistent, compounding memory the LLM writes and the human can read, not retrieval re-run on every question.

Story arc: LLMs know a great deal, but science is broad, deep, and advancing, and training data is stale by design. Harness “project data” (uploaded files) is a RAG approach: nothing persists, context is re-extracted per query. llm-wiki (Andrej Karpathy) is the alternative: dense connected wiki pages as a persistent, LLM-authored and LLM-maintained memory, visible and queryable by the human for trusted, auditable operation. Framing from llm-wiki.md (persistent, compounding artifact; the LLM owns and writes the wiki, you read it).

Does the memory actually work?

A first controlled measurement, on a real research project

Forty questions on one real project. Reading the llm-wiki answered 90% correctly and fabricated nothing, roughly double RAG’s 42% (10 fabrications); the model with no context scored 2%.

Approach	Correct	Overall (0-3)	Fabricated
Model alone, no context	2%	0.10	2
Plain RAG over the sources	42%	1.32	10
llm-wiki read path	90%	2.65	0

90%

correct, versus 42% for RAG. Roughly double.

0

fabricated, versus 10 for RAG.

The zero is by design, a layered control stack that keeps every claim honest:

1 · Honest-reporting rulesreal outputs only; tag every number’s corpus; file failures truthfully

2 · Verification gateblocks unsupported claims and untagged numbers before a commit

3 · Discipline gatesname the always-wrong shortcuts so the agent self-catches

4 · Blocking hooksfire the checks and can halt a bad write, exit-code enforced

Gates and hooks need an agentic harness that can run them, which is why we build in Claude Code and Codex, not the Claude or ChatGPT chat apps. Plain Markdown keeps every claim auditable.

Source: the WGRA post “llm-wiki vs RAG” and Wiki-vs-RAG-First-Run-Results / Cross-Model-Wiki-Reading. n=40 on one real research project, blind judge. Message: the memory works, and works well, 90% vs RAG’s 42%, a perfect 3.00 on method and judgment questions, and zero fabrications. The model alone at 2% shows the wiki is adding the value, not the base model. Keep it clean and promotional; the graph-walk arm and the within-noise detail are omitted on purpose.

llm-wiki operational architecture

How the agent writes, reads, and greps its memory, reconstructed from real transcripts

Reconstructed from real transcripts, this is the YARP wiki as the agent actually used it. It works in three channels: it writes pages (red arrows: authoring, editing, committing), it reads a page into context (blue arrows), and it greps to check its own work (green self-loops).

The mix is lopsided. By tool call it is 88% writes, 7% reads, 6% greps. Reads are rare because of caching: once a page is loaded it stays in the context window, so the agent reuses it instead of re-reading. Its moves go into building and maintaining the memory, not re-fetching it.

Everything routes through two hubs, log (touched 58 times) and index (45). The agent rarely walks the typed edges it wrote: only 24 of 136 moves follow an authored frontmatter link (solid). The rest, the dashed edges, are links it discovers dynamically by search, not ones it was handed.

The LLM builds the memory and reads it back, hub-organized.

The full YARP wiki, 40 pages. Red = write, blue = read, green = grep or re-read; dark = index/log hubs; dashed = link found by search.

From the Reading-the-wiki study; this is the full YARP graph (40 nodes, 136 moves), which we return to later. Channels: writes (red) 88% of tool calls, reads (blue) 7%, greps (green loops) 6%. Make the caching point out loud: reads are few because loaded pages stay cached in context, so the agent reuses rather than re-reads. Everything routes through the index and log hubs; only 24 of 136 moves follow a frontmatter link, the rest are search jumps. This is the llm-wiki architecture in action: the LLM authors the memory and reads it back, hub-organized.

Ingestion: fan out, then synthesize

The write path: one paper per agent, one synthesizer above

Ingesting a paper does not file the PDF. Each PDF is read natively by page range, so tables and figures come through, then the wiki keeps the extracted, attributed knowledge and the links plus a provenance pointer (the DOI and the local path). The source itself stays external.

8 papersPDFs, external

↓

8 subagents · parallel, ~60s1 PDF → structured data
never writes the wiki

↓

1 orchestratorsees all 8 at once

↓

wiki pageshub + typed edges

The extraction contract, the same 7 sections for every paper:

• Citation, confirmed or corrected against a pre-seeded reference
• Contribution, in one sentence
• Method, the specific techniques named
• Results, exact numbers only, each attributed
• Code-relevance, mapped to specific modules and files
• Key terms, anchored to where they appear (Table 1, Section 2.2)
• Quotes, one or two verbatim, with location

One paper per agent: bounded context, no cross-contamination. Each subagent returns data; the wiki’s structure is decided one level up.

Source: yarp/docs/Wiki-Paper-Ingestion-Process.md sections 1 (fan-out, two-layer table), 2 (the contract), 3 (native page-range reading, gets tables/figures), 4 (concepts anchored to location). Say aloud: this process report was itself reconstructed by an agent from the run’s own transcripts. The subagent contract: read one PDF, return the 7 sections as data, “Return ONLY this markdown extraction as your final message.” It never writes the wiki.

Worked example: tables and figures

A table and a figure, captured straight off the rendered page

from a table

CSV RGD1CHNO_smiles.csv, Table 1 columns Rind, Rsmiles, Psmiles, DE_F, DE_B, DG_F, DG_B, DH.

HDF5 RGD1_CHNO.h5, Table 2 keys R_E, R_H, R_F, P_E, P_H, P_F, TS_E, TS_H, TS_F (Hartree), RG, PG, TSG (Å), iterated via parse_data.py.

from a figure

Network seeded at Ga-ethyl (node 0); all reactions with ΔG‡ < 80 kcal/mol are presented in Fig. 2.

First-step from Ga-ethyl + ethylene: Ga-n-butyl 44.1, Ga-vinyl 59.8, Ga-hydride 93.5 kcal/mol (excluded, high barrier). Second-step: 53.2, 51.4, 61.3, 36.0.

Reading the rendered page, not its stripped text, is what captured the table’s exact schema and the figure’s network as attributed data. Tables or figures were read and cited in 6 of the 8 papers.

Source: yarp/docs/Slide-Handoff-Wiki-Ingestion.md (the “source page beside the extraction” version). Each right-hand block is the extractor subagent’s returned text (subagent adf7292b for the RGD1 tables, ae4bd1a8 for the catalysis figure), copied from the ingest transcripts, lightly trimmed to fit. Images are 300 dpi crops of the actual source pages. Honest caveat to say aloud: for the figure, the barrier numbers are printed in the paper’s body text, which Fig. 2 visualizes as edge colors; the line genuinely attributed to the figure is the structural reading, “all reactions with delta-G-double-dagger < 80 kcal/mol are presented in Fig. 2.” This is reading printed labels and structure, not point-level digitization.

Worked example: graphs

Given only a chart, Claude Code writes its own OpenCV pipeline to digitize it

No recipe is handed to it. The agent writes code to calibrate off the axis ticks, detect every point and curve, zoom in on the ambiguous spots, and then redraw its data back on the source to check it.

1. Writes OpenCV targets. Calibrate off the axes (blue), survey the points and curves (green), zoom on ambiguity (red).

2. Where it zoomed. Effort concentrates on the day 28 to 40 band where the markers and curves fuse, not evenly.

3. Pixel-check. Redraws its extracted points and curves on the source to verify against the pixels.

The model writes and runs its own extractor, then verifies it against the image. F1 0.93 on this chart, the hardest in the set, and 0.94 on average across a 28-chart corpus, beating a hand-written recipe.

Source: figure-recovery-suite/docs/llm-wiki-deck-handoff/HANDOFF.md and wiki pages Experiment-Free-Rein-Transcript-Review-2026-08-06 and Analysis-Attention-Map-Visualization-2026-08-12. el-94 = the aedes-aegypti-2014 survival combo chart. Real outputs: position F1 0.93 (recall 0.88, precision 0.99); this run wrote about 25 code executions and saved 21 zoom crops, 17 regions on the attention map; corpus-wide free-rein mean 0.940 vs a hand-written recipe’s 0.891. This is the point-level digitization the tables-and-figures slide said it did not do; here the numbers are locked in the chart geometry, so the agent writes code to recover them. Honest caveats to say aloud: the heatmap is reconstructed post-hoc from saved crops (a lower bound on where it looked, not a live eye-tracker, no time order); scores use a per-type prototype scorer, not a hardened benchmark; a couple of dense multi-series line charts still expose a measurement-layer gap. Not from YARP; it shows the ingestion pattern generalizes.

Behavioral salience: formal interpretation

The heatmap you just saw is behavioral salience: where the model spent its effort

The heatmap looks like a chart-extraction detail, but it is vital to AI development, far beyond the analysis we showed.

The problem. Prompt engineering cannot, on its own, separate instructed behavior from the model’s implicit behavior. That one blind spot produces both failure modes: tuning that does not converge while you build the system (fix one case, regress another, whack-a-mole), and silent brittleness once deployed, where any model change, even a routine same-vendor update you did not choose, removes behavior the prompt was quietly relying on.

Behavioral salience is the differential that separates them, and the field is converging on this trace-based read. The next four slides build it, ground-truth-free:

• What a transcript is: the raw work-log every read is performed on.
• Disposition (policy): surface the agent’s implicit model of the task, then intervene on it.
• Reducible entropy (artifact): flag where the model worked hardest, then resolve it without ground truth.
• Three levels: the same read as a reference you can control, not whack-a-mole.

Opens the behavioral-salience arc: states the problem (prompt engineering cannot separate instructed from implicit behavior, giving non-convergent tuning and silent brittleness), then previews the next four slides as behavioral salience answering it. The kicker names the attention heatmap from the previous slide. For Q&A only, not on the slide: the convergence is independent and recent, MAST (trace-based failure taxonomy, Cemri et al.), TrajTune (trajectory-based prompt optimization), and the silent-failure / entropy work (Liu). Our differentiators are ground-truth-free control and separating instructed from implicit behavior; the others diagnose, describe, or optimize prompts against error metrics.

Behavioral salience: what a transcript is

Claude Code’s own raw work-log: one chart extraction is 281 lines, 4.2 MB

Raw — one line of agent-a8a14075….jsonl

{ "agentId":"a8a14075…", "isSidechain":true, "message":{ "model":"claude-opus-4-8", "role":"assistant", "content":[{ "type":"text", "text": "The frame isn't a heavy box. Let me zoom into the axis regions to find ticks and the exact axis lines." }] }, "timestamp":"2026-08…", "cwd":"…", "gitBranch":"cv-oracle-recall" }

↓decode

Decoded — the agent's reason → act → observe loop

281 lines · 52 scripts · 21 zoom crops · 4.2 MB

THINKthe frame isn't a heavy box — zoom into the axis regions to find the ticks.

ACTpython3 … crop the bottom-left corner → crop_xaxis_left.png

LOOKopens crop_xaxis_left.png — reads the pixels back in.

THINK…and repeats: 52 scripts, 21 zoom crops, ending on the fused day 30–40 band it flags as partly unrecoverable.

THINKthree marker shapes — black circles, gray squares, open diamonds.

ACTsample the fills; check scipy is available

LOOKcircles ~30–70 dark, squares ~190–218 gray.

THINKcircles touch the dashed curve; naive detection merges them.

ACTbuild a disk matched-filter

LOOK29 circles, days 9–37; one legend dot dropped.

THINKsquares sit ON the solid curve; the centroid is pulled.

ACTtake the topmost gray-fill run per column

LOOK25 squares, days 14–38; 4 gap days interpolated.

THINKdiamonds are open; hole-fill leaks to background.

ACTtemplate-match the diamond glyph

LOOK10 clean diamonds, days 19–28.

THINKat convergence the three shapes fuse.

ACTerosion + distance-transform peak finding

LOOKday-30 diamond hidden behind a circle — omit.

THINKthe fit curves vanish in the marker cloud.

ACT3-track column tracer, mask the marker regions

LOOKsolid curve clean; dashed/dotted lost day 22–37.

THINKfill the curve gaps without inventing points.

ACTfit p = Pinf − A·exp(−k·day)

LOOKrms ~0.001; threads both clean segments.

THINKverify nothing is fabricated.

ACToverlay every mark on the source image

LOOKcrosses centered; day 33–36 hits are bright gaps — dropped.

code.py Claude Code wrote this

The transcript is where the model reasons, writes code, and looks: the record we mine for behavioral salience. Click the </> on any ACT to see the code it wrote.

Grounding slide for the arc: the transcript is the raw artifact behavioral salience is read from. Top cycle is the verbatim el-94 extraction step (the frame isn’t a heavy box, zoom, crop, read pixels back); the faded cycles below are that same extraction continuing, shown to convey scale, not read aloud. The ribbon numbers are real (281 lines, 52 scripts, 21 zoom crops, 4.2 MB). It scrolls in once on slide-entry and hands off on any scroll; reduced-motion just shows it.

Behavioral salience: disposition

Same papers, same template: what changes the memory is the agent’s model of the task

Twelve papers, one template, one schema, the same questions. Only the driving assistant changed, and Claude and Codex built different memories, because they hold different dispositions: implicit models of what the ingestion job is.

Codex’s stance

Build a shallow taxonomy; specifics are subordinate, raw numbers get dropped. 44 pages, no results page. Loss scales with particularity.

Claude’s stance

Preserve the paper’s particulars as first-class, retrievable nodes, because that is what future questions turn on. 78 pages; it invents the results page nobody mandated.

The result: one purpose-level paragraph, no fact or rule named, fixed symptoms nobody listed. Codex build, Claude querier held constant, 13 to 20 / 24, p = 0.0156.

“Treat the wiki as durable, queryable memory, not a summary. Your work succeeds only if a future agent, reading only these pages, can recover this paper’s specific claims, the exact quantities (with units), the named systems and methods, and the conditions, and see how they connect to other papers. Preserve those particulars as first-class, retrievable structure.”

Source: llm-wiki-test-harness, “From transcript to disposition to fix” (docs/behavioural-diagnostic-playbook.md, disposition-arms-full.md). The disposition is inferred from the transcript (particularity-fate, trajectory signature), a behavioral read weighted above self-report; the transfer test validates it (a purpose-level change fixes multiple unnamed symptoms). The intervention verbatim: “Treat the wiki as durable, queryable memory, not a summary. Your work succeeds only if a future agent, reading only these pages, can recover this paper’s specific claims, the exact quantities with units, the named systems and methods, and how they connect; preserve those particulars as first-class retrievable structure.” Honest bounds: n=1 run per arm on a saturated fixture (full-text arms cluster 17 to 21 / 24, about 10% flip on rerun), but this is the first non-saturated, significant, one-directional result, and the strongest evidence in the extraction story. This is the policy-level twin of the entropy loop: read the trace, control extraction without ground truth.

Behavioral salience: reducible entropy

The same behavioral read, now on a single figure: the loop reduces the uncertainty it can, ground-truth-free

Locate the high-entropy band from the transcript, zoom 6x and re-extract, recover all 8 misses with zero hallucinations. 89% to 100%, +8 points.

Entropy	Example	Zoom + re-extract
Reducible, epistemic	el-94 fused grayscale markers	resolves, 8/8 recovered
Irreducible, aleatoric	dots under an opaque bubble	cannot, ~37% floor

The method acts on the top row and stops on the bottom; the 100% versus 37% contrast is the evidence it tells them apart.

• Two kinds of “where to look.” Gradient saliency finds where the target is; behavioral salience reads the model’s own effort trace, where it was uncertain and worked hardest. No target required.
• The objective is entropy reduction, not answer matching, so ground truth never enters the decision of where to act. The loop is GT-free by construction.
• Epistemic resolves, aleatoric stalls, automatically.

The loop: flag high entropy from the transcript, attempt reduction by zoom and re-extract, and loop.

Source: figure-recovery-suite/docs/llm-wiki-deck-handoff/VISION-HANDOFF-reducible-entropy.md and the saliency wiki (Concept-Transcript-Saliency, Analysis-Attention-Predicts-Error, Analysis-Difficulty-Taxonomy). Use the term “behavioral salience” (where the model chose to look and spent effort), not gradient or encoder-internal saliency. Honest guardrails to voice: the 100% endpoint is a from-scratch recompute (all 8 within tolerance, 0 hallucinations); the evidence is n=2 charts (el-94 epistemic, life-exp aleatoric), a demonstration, not a benchmarked law. Scoring the result against ground truth (89 to 100) is ordinary validation, not supervision; the decision of where to act is ground-truth-free. Author’s original title phrasing was “Beyond analytics / behavioural saliency and epistemic/aleatoric entropy?” (with the question mark intended), confirm the final title.

Behavioral salience: three levels

Every row is one loop: read the trace, localize the cause, intervene, verify without ground truth

level	reads the trace	intervenes	verifies without ground truth	result
Policy disposition	the agent's model of the task, which particulars it drops	one purpose paragraph, no rule named	transfer test: fixes symptoms nobody named	13 → 20 / 24 p = 0.0156
Artifact entropy	local effort, the high-uncertainty region	zoom and re-extract	pixel-check or consensus: did entropy drop	89% → 100% 8/8, 0 halluc.
Statistical Bayesian	the loop's variance across reruns	decide runs-to-confidence	a posterior over the effect	runs → confidence

A patch fixes only what you named; a disposition change or an entropy reduction fixes the whole class and transfers. Ground truth is the reference you cannot have at scale, so behavioral salience is the reference: controllable, not whack-a-mole.

The payoff of the pair: disposition and the entropy loop are one method read across the columns, read the agent’s own trace, localize the cause, intervene, verify without ground truth, and each row is a different level. Policy (disposition) is the statistically significant one (13 to 20 / 24, p = 0.0156); artifact (entropy) is the clean, vivid loop (89 to 100, 8 of 8, zero hallucinations); the statistical row (Bayesian runs-to-confidence) decides when a change is real. In the proposal: lead the rigor with disposition, lead the picture with entropy. Sources: llm-wiki-test-harness disposition arms; figure-recovery-suite reducible-entropy loop; James’s runs-to-confidence harness.

Synthesis: where linking and trust happen

The layer that sees every paper at once

• Reads the existing wiki first (the schema, then the index), then wires each new page into what is already there: typed lineage edges (extends:) with reciprocal back-references.
• Numbers and citations: exact, attributed, verified not transcribed; “not stated” when unclear.
• Reads and extracts from the real code: for a paper that is a module, it opens the .py and maps each concept to its function.
• Flags degraded sources: it caught an abstract-only PDF and surfaced it.
• The payoff: cross-paper insight that no single page shows.

It caught a degraded sourceThe "YARP v2.0" PDF was a single-page abstract. The agent noticed, ran pdfinfo and pdftotext to confirm, and opened its extraction with a HONEST FLAG. The page still carries the caveat.

It checked itself against the codeindicator.py computes the six Table 1 features; select_pairs.py runs the ML tournament. Each concept mapped to a function, verified against the real files.

The subagents extract attributed data; the structure, lineage, and cross-paper insight belong to the synthesizer, for example an sklearn 1.3.0 pickle that explains why the cluster environment is pinned.

Source: Wiki-Paper-Ingestion-Process.md section 8 (synthesis reads schema then index, wires up: to the Publications hub and extends: lineage with reciprocal back-references), 2 and 5 (numbers and citation verify), 6 (code cross-ref), 7 (degraded-source HONEST FLAG). The lead bullet is the key point: linking and trust are decided where the whole wiki and all eight extractions are visible. Keep the sklearn line a teaser; full paper-to-code-to-ops reveal on the final YARP demo.

YARP: many kinds of source

Not just papers: synthesis across source types, one the agent built itself

The YARP memory was not built from papers alone. Four kinds of source fed one wiki, and the agent generated one of them itself.

• 8 published papers (DOIs), each filed as a source-summary page.
• Tutorials, the group’s onboarding guides.
• The code itself: concepts pulled straight from the .py (for example Lewis-Structure-Score, sourced from yarpecule/lewis/bem_score.py).
• Sphinx API docs the agent built from the code, then ingested as another source.

8 papersDOIs

tutorialsonboarding

the code.py source

Sphinx docsagent-built

↓

one YARP wiki~40 pages, 9 source-summaries, 5 synthesis

The agent generated a source, the API docs, from the code and then ingested it. The code is a first-class source, not just the papers.

Source: yarp.wiki YARP-Overview.md, Publications.md (the 8 papers), Documentation-Build.md (Sphinx docs built from the code via update_docs.sh, then ingested), the docs/references tutorials, and page source: frontmatter pointing at .py files (Lewis-Structure-Score source: bem_score.py). The wiki is about 40 pages, 9 source-summaries and 5 synthesis pages. The point for a data-extraction audience: ingestion is not paper-specific; heterogeneous sources, including ones the agent authored, synthesize into one memory.

YARP: the answer that lives in the links

Combining data: a question no single source can answer

Why is our cluster pinned to scikit-learn 1.3.0? No paper says it. No code comment says it. The answer exists only in the links between the sources.

Conformational Sampling 2022 · paper

is implemented as ↓

reaction/conf_sampling/ · code

ships its model as ↓

a scikit-learn 1.3.0 pickle

forces ↓

CRC environment pinned to 1.3.0

The memory follows paper to code to artifact to operations, a chain assembled from four different source pages.

Dense RAG over the raw repo and the PDFs cannot produce this: the fact lives in the connections, not in any one document.

Not “the AI knows your field.” The group’s hard-won knowledge, paper to code to operations, now instant instead of tribal.

Papers-Code-Coupling synthesis page (“the three couplings that mattered”, coupling #2). The chain is the payoff of the whole talk: it exists only in the cross-source links the synthesizer built. Chris’s framing: acceleration of the researcher, not replacement. If time, a second coupling: RGD1 (176,992 DFT-characterized reactions) trains the egat_rgd1 barrier model. Optional live demo: ingest a competitor paper and have the memory contrast approaches.

Two kinds of synthesis

What synthesis is, and a real task that shows it

Synthesis is reaching a conclusion that no single source states, by joining new information against what the memory already holds. It takes two forms:

• Retrieval synthesis (the sklearn pin you just saw): the conclusion already existed, latent in the links; the agent traverses the memory to surface it. Read-side.
• Investigative synthesis (this one): the conclusion existed nowhere. The agent runs the experiments, reconciles the results against the literature and the wiki, and writes the new conclusions back. Write-side.

The task, honestly. Ericka, the YARP dev lead, handed the harness a small but real job to evaluate it: a pre-release 3.0.1 batch for a Monday release, a “KHP fruitfly” case failing and breaking the progress pipeline, no reference outcomes yet. Run it on the current code, check out the stable v3.0.0 tag, and triage.

A successful outcome, in one session: a correct, non-obvious verdict. The release is exonerated (the same failures reproduce on stable v3.0.0), 15 of 22 “failures” are the pipeline working by design, and the real to-do list is 7 issues with the exact one-line fix location. Without the memory, each would have been a cold rediscovery or a wrong guess.

Source: yarp/docs/Vision-Handoff-KHP-Investigative-Synthesis.md and KHP-Fruitfly-Regression.md. Team-safe framing is required, the YARP dev team will see this: lead with the pro-team findings, never “your code is broken”; the job_manager.py:34 one-line guard is a gift, not a criticism. Investigative synthesis is the write-side, generative counterpart to the sklearn pin’s read-side, and it fuses the most source types on the project: the paper and SI, two code versions, live run outputs, the collaborator’s own diagnostic files, and four prior wiki pages. It then compounded into the discovery-vision page, the robustness issue drafts, and Lessons-Learned. The setup honesty: this was a capability evaluation from the dev lead, small scope, real stakes, not a large study.

Synthesis is a set of joins

Each conclusion is one synthesis: a new observation, joined to memory the wiki already held

Observation from the runPrior memory it joined againstConclusion (the synthesis)

yarp-progress stuck 15+ minutes in D state, 322,576 tiny files in scratch.

The crc skill and the ingested CRC-Infrastructure page (VAST flash is NFS-backed), plus Lessons-Learned and M5, which had root-caused a scratch re-walk stall.

The known stall, reproduced at 36-reaction scale, hitting even VAST because it is NFS-backed. Not slow disk, not a bad setup: GSM writing an explosion of tiny files. Killing 4 runaway jobs dropped scratch to 23,756, confirming it.

15 of 36 reactions in the failure bucket at ll_refine.irc_validation.

The Phase2-Project1-Pilot page: unintended reactions are retained by design in failed_rxns.pkl, correct routing, not a crash.

Read outcome_label: 15 of 22 "failures" are the by-design IRC quality filter working, not bugs. The genuine error budget is only 7 of 36.

13 to 14 intended reactions at the xTB level.

The ingested Zhao-Savoie-2021 paper and SI: SI Table 6 references 16, and 16 is the post-DFT survivor count.

13 to 14 at xTB is expected below 16, not a shortfall. Some xTB-unintended reactions likely recover at DFT.

CREST crashes are hash-identical on 3.0.1 and stable v3.0.0 (Arm A's 6 are a strict subset of Arm B's 7).

The swap-the-package trick from Lessons-Learned (pip install -e . --no-deps) made running both versions a seconds-long op, not a 15-minute env clone.

The crashes are version-independent and pre-existing: the 3.0.1 PRs are not the culprit. PR #44's one real effect is narrow (C=COCOO fails in both; it only moved where it dies).

Fast and correct not because the model was clever in the moment, but because the memory already held the yardstick, the by-design behavior, and how the cluster fails. Ingestion is what put them there.

Honest conditions to keep in the speaker’s voice, briefly: n = 1 per version, and GSM non-convergence is stochastic, so the 14 vs 13 intended difference is not attributable to the code version, both land at 13 to 14 at xTB. xTB level only, no DFT run yet; the unintended verdicts were confirmed classified-not-crashed, not inspected per reaction for chemical correctness. The CRC operational memory lives in two places, the crc skill and the ingested CRC-Infrastructure page; present them together as operational memory, the wiki page being the auditable twin of the skill. Full report: yarp/docs/Agentic-YARP-KHP-Regression-Final-Report.pdf.

Where this lands

Takeaway

• Extraction into durable memory, not a one-shot data pull. The wiki compounds: every source ingested and every answer filed means the next question starts from what is already known.
• It works, and it stays honest. On a real project the wiki read path answered 90% correctly and fabricated nothing, roughly double plain RAG, and the zero is by design: honest-reporting rules, a verification gate, and blocking hooks in an agentic harness.
• Hub-indexed reading, disciplined ingestion. The agent reads through the index and log hubs, and ingests papers, tables, figures, even charts it digitizes with its own code, and heterogeneous sources including ones it wrote itself.
• The payoff is the links. Answers that live in no single document, paper to code to operations, become instant instead of tribal.

Not “the AI knows your field.” A persistent, auditable memory the group builds once and keeps, so hard-won knowledge compounds instead of scattering.

Close on the arc: it works (90 vs 42, zero fabrication), it is honest by construction (control stack, verify not transcribe), it reads hub-indexed and ingests heterogeneous sources with discipline, and the payoff is the cross-source links (the sklearn pin). Honest framing to keep in your voice: the wiki-vs-RAG number is one controlled measurement on a real project, a strong signal rather than a benchmark verdict. Optional live demo: ingest a competitor paper and have the memory contrast approaches.

Questions

Thank you

Research memory that compounds. Extraction into a durable, queryable llm-wiki: it beat plain RAG and fabricated nothing, it ingests and synthesizes across sources, and the payoff is the answers that live in the links.

Chris Sweet, Charles Vardeman, Priscila Moreira, James Sweet

End of the talk. What follows is appendix material (more chart types the figure extractor handled), for Q&A or to skip.

Also handled: dense bubble scatter

Appendix · Life expectancy vs GDP, OWID. The generated OpenCV pipeline, stage by stage

Density correlation 0.94, and 132 of 165 markers. Erosion removes the country-label text and a distance transform splits touching bubbles, so every detected center lands on a bubble, not a label.

Source: figure-recovery-suite/docs/llm-wiki-deck-handoff/panels and its README, corpus owid-r6-1. Now a 4-stage process grid: mask by saturation, erode away the label text, distance-transform peaks, then pixel-check. The four stages are the agent’s own saved artifacts. These appendix slides show the free-rein extractor generalizes past the el-94 combo chart on the graphs slide. The composite title was cropped off, it carried an em dash.

Also handled: multi-series time-series lines

Appendix · Annual CO2 emissions, OWID. Seven same-family colors, separated stage by stage

Calibrate the axes and sample each line’s color, classify every pixel to its nearest series, resolve the near-identical China and UK colors, then pixel-check. All 7 lines traced and overlay-verified.

Now a 4-stage process grid. Honest notes to voice: stages 1 and 2 (calibration and nearest-color classification) are faithful reconstructions of the documented method, the agent saved no intermediate stage images for this chart; stages 3 and 4 are its own artifacts. On this chart the per-type scorer reports curve F1 0.34 despite the overlay-verified trace, a measurement-layer artifact (out-of-range plus tight tolerance), so report the verified trace, not a score. The stage-3 caption baked into the image contains an en dash (“0-2B”); the composite title was cropped off, it carried an em dash. Source: panels README, owid-r6-1.

Also handled: grouped bars with error bars

Appendix · Synthetic throughput benchmark, synthetic-r4-1

Left: a zoom on one error-bar cap, the model resolving the asymmetric error bars. Right: its pixel-check, every bar value and both caps on each bar recovered.

Bar-value F1 1.00: all 15 bars and 30 error-bar caps recovered.

Source: panels README, corpus synthetic-r4-1. A synthetic chart (Baseline, Tuned, Tuned+JIT throughput across Q1 to Q5) chosen to test asymmetric error-bar caps; the left panel is the T-cap zoom detail. The chart’s own title was cropped off because it carried an em dash.