Agentic research workflows

An introduction to using Codex for research workflows
and getting a project-specific knowledge base started

Chris Sweet
Charles Vardeman
Priscila Moreira
James Sweet

September 21, 2026

The bottleneck moved

The problem with a fast, plausible collaborator

Research is a loop: hypothesize, write the code, analyze, test, revise, and go round again. Producing each result used to be the slow part.

AI now appears to remove that bottleneck. It writes complex code, carries a vast working knowledge of statistics and analysis methods, and returns test results ready to present.

So the bottleneck of research is no longer producing results. It is knowing which results deserve belief.

Plausibility is the enemy. From our own project record, the AI describing its own failure, verbatim:

“The worst failure of this project: I built a tractable proxy instead of the user’s hypothesis, and let loose language hide the substitution.” Lessons-Learned.md, project wiki, 2026-08-19, recorded by the AI and found on query

Nothing was wrong with the output. It ran, it passed, it read well. It just was not the thing we set out to know. We only caught it because the AI had written the failure into a durable memory we could go back and read.

Tip 1. Have the LLM record what it learns into a durable memory you keep and can read back, not a chat window that vanishes. The quote above is from ours.

What epistemic discipline is

Habits you already have, that AI speed breaks

Epistemic discipline turns a produced result into something you actually know: justified, not merely generated. In practice, a few rules about what must be recorded, and carried with the claim, before it enters the record.

You already practice this. Each one is a standard scientific habit.

The habit What you already call it
Provenance. a claim carries where it came from, and its scope citation
Durable memory. what was tried, decided, and abandoned is written down a lab notebook
Pre-commitment. targets declared before the data arrives pre-registration
Why it breaks with AI
The habits live in your head and in a chat window that vanishes when the session ends. At AI speed they are the first thing skipped: when a result arrives in minutes, the provenance and the notes are what get dropped.
And AI discards and regenerates work at each step, instead of a small, preserving epsilon step that keeps what was already verified.
Seen here: asked to consolidate three reports into one paper, it regenerated rather than preserved, and the draft did not even describe the framework the reports exist to present. Filed as a dead end.

So stop remembering the discipline. Its key element is memory: move the provenance and the notes out of your head into an artifact the AI writes and you read. Get the memory right and the discipline holds.

What kind of memory?

Five kinds, only one is durable and yours to read

An agent has several memories. Most are frozen, optional, scoped, or temporary. The question that matters: which one still holds your project’s knowledge after the session ends?

The memory What it is How long it lasts
Embedded (training) the model's learned world knowledge frozen at the cutoff
Local memories optional machine-local cross-session store (Codex /memories) off by default, empty here
AGENTS.md scoped instructions, loaded per folder at startup rules, not a knowledge base
Session context the live conversation and tool results gone when the session ends
llm-wiki pages the agent writes and you read, in git durable, versioned, yours
And local memories are not yours to control. The agent decides whether to write one, and can overwrite or delete it. There is no version history and no backup: if a memory is lost or silently changed, you cannot see what happened or roll it back.

Only the llm-wiki is durable and under your control: the agent writes it, you read it, and it lives in git, so every change is visible, reviewable, and reversible.

What an llm-wiki is

The memory the LLM writes and you read

  • Proposed by Andrej Karpathy (OpenAI founding member, former director of AI at Tesla): dense, interlinked wiki pages as a persistent memory the LLM authors and maintains, kept as plain Markdown in version control.
  • A read-write loop that compounds. The LLM reads the wiki at the start of a session to recall where things stand, and writes back during the session to remember what it learned. Each session builds on the last instead of starting cold.
  • Auditable by you. Because it is Markdown in git, you can read exactly what the agent knows and see every change it makes.
  • Portable as a knowledge bundle: you can share the whole knowledge base and another person’s LLM can read it.

Trusted AI. The human stays in the loop: audit what the agent knows and check any claim at its source. When something is wrong, you catch it and direct the fix, which the agent makes and logs. Accountable and explainable, the trust our group builds toward, not a black box.

A persistent, compounding project memory the LLM writes and you read, not retrieval re-run from scratch on every question.

How to build an llm-wiki with Codex

The tutorial, run with students on 2026-09-15

llm-wiki is a Gist from Andrej Karpathy, an idea rather than an implementation. Our implementation assumes the following:

  • We assume project-based work, so there is a base GitHub repo for code, data, and artifacts.
  • By default we create the wiki as a GitHub wiki, a special repository GitHub uses for its wiki functionality, so you can view it in the online GitHub interface, shown here.
  • Other wiki destinations are available.

The next slides walk the whole process, built live in Codex: install, initialize, ingest a paper, run a real prediction, and record it as a durable wiki page.

The finished codex-llm-wiki-test GitHub wiki: the Chignolin AlphaFold Prediction page with its 12-page sidebar github.com/chrissweet/codex-llm-wiki-test/wiki

Demo 1: install the plugin, make the repo

One marketplace, two commands

Our implementation of llm-wiki provides our own marketplace, a storefront for plugins, and we have versions for the AI harnesses Codex, Claude, and Cursor.

Install the plugin from the LA3D marketplace:

$ codex plugin marketplace add LA3D-LLM-Agents/llm-wiki-colab
Marketplace `llm-wiki-colab` added.
$ codex plugin add llm-wiki@llm-wiki-colab
Added plugin `llm-wiki` from marketplace `llm-wiki-colab` (0.4.0).

Codex loads plugins at startup, so exit and restart Codex, and accept the plugin’s hooks when prompted. Then create the project repo:

$ gh repo create chrissweet/codex-llm-wiki-test --public
✓ Created repository chrissweet/codex-llm-wiki-test on GitHub

Same wiki, any harness. The identical plugin installs in Claude Code (/plugin marketplace add, /plugin install), Codex (codex plugin marketplace add, codex plugin add), and Cursor (cursor-agent plugin marketplace add).

Demo 2: initialize the wiki

wiki-init, GitHub-backed

Run the init skill and choose GitHub:

$ $llm-wiki:wiki-init
? storage backend → GitHub
$ bash "$SKILL_DIRECTORY/scripts/init-wiki.sh" --github --agent "codex"
created .llm-wiki/ (own git repo, attached to the wiki remote)

It scaffolds .llm-wiki/ as its own git repository, attaches it to the repository’s GitHub wiki remote, and writes the SCHEMA plus navigation pages. Result: a blank llm-wiki, ready to push.

Those pages make the wiki self-describing:

  • SCHEMA: the conventions, page format, required frontmatter, and the typed links between pages.
  • index: the catalog of every page, the wiki’s table of contents.
  • log: the append-only, dated record of what was done, by who and using which agent.

Each is namespaced to the project (index_codex-llm-wiki-test, SCHEMA_codex-llm-wiki-test, log_codex-llm-wiki-test), so pages stay unambiguous when wikis are shared or merged.

The wiki is a separate repo from your code. Code and artifacts live in the project; durable knowledge lives in .llm-wiki/.

Demo 3: ingest the AlphaFold paper

wiki-source: one paper, five pages

Ingest turns an external source into wiki pages: the agent reads it and writes summarized, linked, cited pages. AlphaFold is DeepMind’s protein-structure predictor; our source is its 2021 Nature paper.

ingest paper AlphaFold-Protein-Structure-2021.pdf
+ source summary + 4 concept pages

Plain language is enough: asking to ingest makes Codex call the $llm-wiki:wiki-source skill.

One ingest produced a source summary plus four linked concept pages:

  • Architecture — Evoformer and structure module.
  • Training and data — datasets and training cutoff.
  • Evaluation and confidence — the pLDDT and pTM metrics.
  • Limitations and inference — where it is unreliable.
  • All cross-linked wiki style, with links typed where the relationship is clear.

Pages, not chunks. Preserve what a future reader needs: dataset scope, training cutoffs, named methods, exact metrics, confidence intervals, limitations. A number without a source page does not exist for the project.

The ingested pages integrate into the existing wiki: related pages are updated, any contradiction with what the wiki already claims is reconciled, and back-references are fixed both ways. The ingest is recorded in the index and log, then a verification gate checks every page for unsupported claims and missing provenance before it is committed.

Demo 4: run AlphaFold on chignolin

A ten-residue mini-protein: GYDPETGTWG

The power of AI removes friction and accelerates understanding. Just from the paper, I can test the ideas by getting Codex to find the code repository, install the software, and run it locally without intervention. In this demonstration I asked it to find a small molecule (it suggested chignolin) and get it to fold it.

ColabFold 1.6.2, AlphaFold2 pTM, single-sequence MSA, five models, seed 0, three recycles, CPU.

JAX_PLATFORMS=cpu colabfold_batch chignolin.fasta results/
--msa-mode single_sequence --model-type alphafold2_ptm
--num-models 5 --num-recycle 3 --random-seed 0 --max-msa 1:1
ranked PDB + pLDDT/PAE plots + run log
Output Value
Mean pLDDT (10 res) 93.4
pTM 0.06
Max aligned error 5.35 Å

High pLDDT is confident local geometry; pTM 0.06 is poor global-fold confidence, expected here because a single-sequence MSA gives AlphaFold no evolutionary signal. These are model outputs, not experimental validation.

Experiment skill. The llm-wiki plugin includes an Experiment skill that documents an activity like this as a scientific experiment. It is called silently when a run produces results worth keeping.

Demo 5: the result becomes a wiki page

Run in, durable page out

Chignolin before and after AlphaFold prediction

One plain-language line produced the figure:

can you generate a pre and post image for the molecule?

Codex used its built-in image tool to make this unfolded → folded illustration of chignolin.

$llm-wiki:wiki-experiment filed Chignolin-AlphaFold-Prediction. The skill:

  • captures the configuration and metrics: parameters, seeds, dataset, headline numbers;
  • keeps the interpretation limits with the numbers, filing results honestly;
  • notes what changed vs. a previous run, and anything surprising;
  • records provenance: links to the results directory and the commit;
  • integrates with related pages and reconciles contradictions;
  • updates the index and log, runs the verification gate, then commits.

Live page: github.com/chrissweet/codex-llm-wiki-test/wiki

Demo 6: recall, the loop closes

Ask the wiki, not the paper

A later session asks about the run. The agent answers from the page it wrote, no paper, no rerun:

what did our chignolin run find, and why is the confidence what it is?
Our single-sequence ColabFold run produced a top chignolin model with
mean pLDDT 93.4, but pTM 0.06 and maximum PAE 5.35 Å. It was confident
in local geometry, but not the overall fold. The low global confidence
reflects the absence of evolutionary MSA information; this was a static
prediction, not folding dynamics.
 
Wiki pages read:
- .llm-wiki/Chignolin-AlphaFold-Prediction.md
- .llm-wiki/AlphaFold-Limitations-and-Inference.md
 
I did not reopen the AlphaFold PDF or rerun the experiment.

Write once, recall forever. The next session builds on the page instead of re-deriving it: the read half of the loop, closed.

The wiki is git: commit, pull, push

Say it in plain language; the agent runs the git

The wiki is its own git repo, so every page is versioned, diffable, and reversible.

  • Commits happen automatically. After each ingest or experiment the agent commits to .llm-wiki/, no need to ask; local commits are cheap and reversible.
  • Sync in plain language. Say “pull the wiki repo” or “push the wiki repo” and the agent runs the git for you, no commands to remember.
  • Pushes don’t collide. The plugin refreshes the wiki from the remote at the start of every session (fetch, then fast-forward, with any local work preserved), so you always build on the latest memory and your push lands cleanly.
  • The main repo too. Your project’s code repo (code, data, artifacts) is also git: commit, pull, and push it with the same plain-language commands.

“Push the wiki repo” lands it on GitHub, versioned and audit-ready, even when the team is writing at once.

More skills: maintain, share, consult

Beyond building the wiki

Building the wiki is the start. Three more skills keep it healthy and connect it to other projects:

  • wiki-lint — a health check over the wiki: finds orphan pages, dead links, stale claims, missing frontmatter, and one-way cross-references, then offers fixes. Keeps the compounding memory honest as it grows.
  • wiki-enroll — publishes an agent Card and a discovery topic so this wiki becomes findable by peer llm-wiki agents in the federation.
  • wiki-ask — consults another project’s wiki via the ask primitive: it clones that agent’s wiki and queries it, so you draw on another project’s durable memory without leaving your session. It authenticates via GitHub, so without at least read access to that wiki you cannot query it.

One wiki you maintain (lint), publish (enroll), and query across (ask): durable memory that compounds and connects.

Turning results into knowledge

The discipline, made concrete

Trustworthy results are not automatic enforcement; they come from encoded discipline the agent follows, plus reminders, run by the harness:

1 · Real-output guidanceuse real outputs; scope every number to its corpus; report failures truthfully
2 · Verification gatebefore committing, the agent re-reads each page against a checklist and fixes failures
3 · Discipline gatesa named list of always-wrong shortcuts the agent must reject
4 · Advisory hookssession-start and post-edit hooks inject context and reminders; the gate does the content check

How it runs, one loop:

1 · Session starta hook loads the guidance + the wiki's index and recent log into the agent
2 · Workskills carry the read and write procedures for pages
3 · After an edita hook reminds; the agent runs the gate and fixes failures
4 · Committo git; plain Markdown, so every step is auditable

A chat app runs none of this loop, which is why we build in Codex and Claude Code.

Discipline gates: the shortcuts we reject

Named always-wrong rationalizations, caught before the write

The plugin names the excuses an agent might use to cut a corner, so it self-catches before committing. A few:

The shortcut Why it is rejected
"I'll add the corpus tag or cross-links later" Later never comes; do it now while context is fresh.
"The estimate or projection is fine to file" Numbers must be real outputs or citations; projections are marked as such, never slipped in as fact.
"Close enough to compare these two corpora directly" A gap between numbers on different corpora is not a gap; tag each with its corpus.
"Good enough to commit" Run the verification gate first; it costs seconds.
"These changes don't need a log entry" The log is the next session's input; skipping it makes the change invisible.

The list is meant to grow: each real mistake that traces back to a rationalization becomes a new row.

Does the memory actually work?

Two measurements: head to head, and at scale

Forty questions on one real project: reading the llm-wiki answered 90% correctly and fabricated nothing, roughly double RAG’s 42% (10 fabrications); no-context scored 2%. And accuracy holds as the wiki grows to thousands of pages.

Head to head (n=40, one project, blind judge)

Approach Correct Overall Fabr.
Model alone 2% 0.10 2
Plain RAG 42% 1.32 10
llm-wiki 90% 2.65 0

At scale (500 to 4,000 real pages, Sonnet 5)

pages recall reads greps index
500 100% 5 5 90 KB
1,077 100% 5 7 199 KB
2,028 100% 3 5 384 KB
4,013 98% 4 6 778 KB

Recall stays near-perfect across 8x growth, and per-query cost stays flat, because the agent greps the index rather than reading it whole (retrieval is O(1) in corpus size).

This wiki, mapped to a Knowledge Graph

Generated by the wiki-kg plugin from the LA3D-LLM-Agents/llm-wiki-marketplace

The llm-wiki marketplace

Extra tooling, installed like the core plugin

Our own marketplace, LA3D-LLM-Agents/llm-wiki-marketplace, ships the extras alongside the core llm-wiki plugin, dual Codex and Claude Code.

  • wiki-kg — build, query, and visualize a typed-edge knowledge graph from a wiki (rdflib + pyshacl, in-process). The map on the last slide came from here.
  • wiki-agent-msg — cross-harness live messaging: list and message Claude Code and Codex sessions on the same machine.

Install like the core plugin:

$ codex plugin marketplace add LA3D-LLM-Agents/llm-wiki-marketplace
$ codex plugin add wiki-kg@llm-wiki-marketplace

Claude Code: /plugin marketplace add LA3D-LLM-Agents/llm-wiki-marketplace then /plugin install wiki-kg@llm-wiki-marketplace.

Each page becomes typed RDF, so the wiki is a graph you can draw (above) or query with SPARQL:

@prefix wiki: <https://la3d.github.io/llm-wiki-colab/page/> .
@prefix kg:   <https://la3d.github.io/llm-wiki-colab/ontology#> .

wiki:Constraining-the-Field-for-Trust a kg:Synthesis ;
    dcterms:title "Constraining-the-Field-for-Trust" ;
    kg:extends   wiki:Lightweight-Structural-Graphs ;
    kg:mentions  wiki:LLM-Wiki-Read-Path , wiki:Vision .

Read more: the WGRA Nuggets blog

The ideas behind this, one nugget at a time

The WGRA Nuggets blog: browsable one-idea posts with a categories sidebar

Short, self-contained posts you can graze in any order. The lineage traces back to Karpathy’s llm-wiki. Topics include:

  • Foundations — why agentic software breaks deterministic habits.
  • Evidence — llm-wiki vs RAG, and a Bayesian harness for honest evaluation.
  • The knowledge graph — typed edges, small ontologies, catalog to content.
  • Ecosystem — federation, connectors, data fabrics, a tools marketplace, bundles.
  • Practice — standing up a wiki, and how to talk to a research agent.

la3d.github.io/WGRA/blog

Part of a series

Talks, and where to read more

Talks

  • Agentic research workflows (this talk) — Codex to start a project knowledge base.
  • Research memory that compounds — sources into a durable, queryable llm-wiki.
  • Knowledge: Extract, Retrieve, Synthesize — the flow and the knowledge graph.

Pages

  • The WGRA overview and the Nuggets blog.
  • Ecosystem: Research Agents (the competitive edge), the Knowledge Fabric.
  • Deep dives: memory that compounds, team scale, agent capability, the knowledge graph, knowledge bundles.

All at la3d.github.io/WGRA, plus the paper Beyond Memory (Saboia Moreira et al., 2026).

Questions

Thank you

Agentic research workflows. In one Codex session we installed the plugin, built a durable llm-wiki, ingested a paper, ran a real AlphaFold prediction, and recorded it as a linked, queryable page. That discipline turns results into knowledge you can trust, and it measurably works: it beat plain RAG with zero fabrications, and held near-perfect recall to thousands of pages.

Build one: github.com/chrissweet/codex-llm-wiki-test/wiki

Chris Sweet, Charles Vardeman, Priscila Moreira, James Sweet