The Web Forager llm-wiki collaboration story

The project in one paragraph

Web Forager is an adaptive web scraper that builds knowledge incrementally by reading webpages with Claude vision, answering user queries against what it knows, and going back to the source when it cannot. It is validated against a set of competency questions on manufacturer websites (ITAMCO, Schafer Industries, later a 108-company drone-industrial-base fleet). Two Claude Code instances contribute: one owned by Chris Sweet, one owned by Chris Frederick (Omegaice in the wiki commit attributions). Both write into a shared llm-wiki at wiki/web_forager.wiki/.

Repository: github.com/chrissweet/web_forager · Wiki: github.com/chrissweet/web_forager/wiki

The story

The setup: a demo branch that "wasn't meant to be important"

On 2026-05-20, Chris Frederick pushed commit 81e03e2 to the project. It was a demo branch for an informal Navy lunch with three guests. In his own framing, filed in the wiki page for the experiment:

"I wanted to make sure I could give Jose a simpler demonstration in Obsidian of what we are working on. ... This was solely for that demonstration."

Chris Frederick explicitly offered to drop the commit if it diverged from project direction. Chris Sweet kept it as valid work. Both wrote a wiki page recording what the branch was and why: Experiment-11-Productionization. The page's frontmatter carried a load-bearing edge:

---
type: synthesis
up: "[[Experiment-10-Vocabulary-Alignment]]"
tags: [experiment, demo-branch, navy-lunch, claude-cli, haiku-regression, sonnet-validated]
related: "[[Experiment-8-Second-Entity]]"
extends: "[[Experiment-7-Question-Deepen]]"
supports: "[[Working-Hypothesis]]"
---

extends: [[Experiment-7-Question-Deepen]] told any reviewer: "this branch inherits assumptions from Experiment 7." related: [[Experiment-8-Second-Entity]] said "these two live in the same family." Neither of those edges was decoration. They were what let the wiki review that followed turn a demo branch into a project-wide correction.

The bug the wiki review revealed

Chris Frederick ran his demo branch against a three-entity ground-truth set on 2026-05-22 and filed the results in the same wiki page. The numbers looked good at first pass:

Chris Sweet reviewed the entity files that Chris Frederick's pipeline had produced and cross-referenced them against the "reported" counts on the wiki page. The reported count included any model response as an answer, including honest "not visible on this page" hedges. The real evidence-based counts were substantially lower:

The bug: when the LLM honestly said "the answer isn't on this page," the pipeline treated that as a successful answer and stopped searching. When the LLM instead confabulated a wrong answer (as Haiku did with "prior knowledge of Southwire"), the pipeline treated that as an answer too. Only "no answer at all" would have triggered deeper exploration — but the model, being helpful, always said something. The pipeline was conflating the model returned a response with the question was answered.

This is the bug the reader should picture: the LLM refusing to hallucinate ("I can't find this on this page") looked identical to the LLM answering, so the search stopped in exactly the case where deeper searching was warranted.

The wiki page's coverage analysis captures the mechanism explicitly:

"Sonnet's honest 'not visible' responses look like plateau-evidence and trigger early stop, especially because Sonnet front-loads a lot of 'not visible' hedging on the first page (see ITAMCO). Haiku's confident-hallucination behavior accidentally drove deeper exploration in prior experiments — fixing that behavior also broke the implicit assumption that drove page-depth."

The cascade that made this a project-wide correction

The extends: [[Experiment-7-Question-Deepen]] edge in the demo branch's frontmatter was the load-bearing structural piece. It told the reviewer, formally, that this branch inherited assumptions from Experiment 7. If those assumptions were wrong here, they were wrong there. Chris Sweet followed the edge into the Experiment 7 page and the same pattern was visible.

The wiki page for Experiment 11 records the audit result explicitly:

• Experiment 7 (ITAMCO 20/20 reported): real evidence-based was 14/20. Six of those answers were "not visible on this page" hedges counted as successes.
• Experiment 8 (Schafer 20/20 reported): real evidence-based was 12/20. Eight of those answers were the same category of hedge.

The Working Hypothesis of the entire project — "questions replace ontologies for LLM extraction; ontology is the failure point, not the expert knowledge" — was resting on numbers that were partly artifacts of the counting bug. If the same fix that recovered ITAMCO also recovered Experiments 7 and 8 to their claimed numbers, the hypothesis stood. If not, the project's central architectural bet needed revision.

The fix and its measurable impact

Commit 1cf819e on main shipped two changes:

1. Prompt update in ask_vision.py: explicit instruction that null tells the pipeline to keep looking, while a hedged string tells it the question is resolved. Encouraged the model to actually return null.
2. Defensive filter in asker.py: _is_not_found() matches 16 hedge phrases and applies the filter in four places — the main merge step, the plateau computation, the "still needed" list, and the deepen-time prior-answer sanitization.

The re-run on the same three entities on 2026-05-22 (Sonnet v2):

ITAMCO's recovery — 8 → 14 real answers, from 1 page to 7 pages — is the smoking gun. The bug had been forcing the pipeline to stop at the homepage; the fix let it keep exploring the site the way it was designed to. Depth restored, real coverage restored.

The concept extraction (semantically distinct content the model found) followed the same pattern:

And the cascade back into Experiments 7 and 8: both experiments were re-run under the fix. Both landed at exactly 18/20 real evidence-based answers. ITAMCO recovered 4 of 6 lost answers. Schafer recovered 6 of 8 lost answers. The exact parity across two different entities on the same 20-question generic set confirmed the cross-entity transfer claim that was the whole point of Experiment 8, at corrected coverage. The Working Hypothesis survived the audit because the wiki structure surfaced the audit in the first place.

Chris Frederick's demo branch was recorded on the page as still "valid work." The wiki page's log entries record both parties' contributions in sequence: Frederick filed the branch, Sweet reviewed and found the bug, the fix shipped, both experiments got re-run with attribution to whoever ran them. No single contributor gets displaced; the wiki records the whole arc.

Why the wiki structure made this catch possible

Six llm-wiki primitives did the load-bearing work.

1. Typed pages with typed edges

Every page carries YAML frontmatter declaring what kind of page it is and what its relationships are. The demo branch's frontmatter had extends: [[Experiment-7-Question-Deepen]]. That is not "these pages are related." It is an interface contract: treat the target's claims as background assumptions for this page. When a reviewer finds a bug on the child page, the extends: edge tells them to check whether the parent inherited the same bug. That single semantic edge is what turned a one-experiment bug report into an audit of the two prior experiments the demo branch inherited from.

The wiki's schema page documents this explicitly under the heading Edges as Interface Operations:

A wiki reviewer following extends: from Experiment 11 to Experiment 7 is not just reading two pages that happen to link. They are performing the exact operation the edge encodes: inherit the parent's assumptions, check whether the current bug violates them.

2. Structured coverage table in the page itself

The demo branch's wiki page had a table comparing "reported" answered counts against "real" evidence-based counts. That table did the analytical work: without it, the reviewer would have seen only the reported column (17/17, 14/17, 17/17) and moved on. The wiki convention of documenting the audit inline with the finding meant the counter-evidence was visible on the same page as the claim.

3. Wikilinks that make cross-page implications inspectable

The demo page names [[Experiment-5-Question-Driven]], [[Experiment-7-Question-Deepen]], [[Experiment-8-Second-Entity]], [[Working-Hypothesis]] in its Alignment table. Each is a durable link into a page that carries its own frontmatter and its own edges. Traversing them made it possible to reach the conclusion "if the bug affects the demo branch, it may affect three prior experiments and one central hypothesis," without holding all of those in working memory at once.

4. Auto-maintained index

index_web_forager.md catalogs every page with a one-line description. A collaborator does not need to remember what exists to find it. When Chris Frederick filed the demo branch's page, the index reflected it. When Chris Sweet later cross-referenced back into Experiment 7 and 8, both were reachable by name from the index.

5. Per-entry commit log with author attribution

log_web_forager.md records each wiki operation as a dated entry with the first bullet reading by: <human> via <agent>. The demo-branch filing reads by: Chris Frederick via claude-code. The bug-fix and re-run entries elsewhere in the log read by: Chris Sweet via claude-code. Every log entry is its own commit (schema rule), so git blame on the log file is a faithful per-entry record: provenance is visible on the page, and it is verifiable in git. Neither collaborator's contribution gets erased by the other's follow-up. Both stand.

6. Session boundary crossed via durable state

Neither Claude Code session shared conversational context with the other. They shared a wiki. Chris Frederick's Claude wrote the demo-branch experiment page as part of ingesting his session's findings. Chris Sweet's Claude, in a completely separate session, read the same pages on load per the wiki-first project convention. Neither session needed to be "held open" or "handed off." The wiki was the substrate, and it survived session boundaries.

The generalizable pattern

• Individual sessions surface findings. A contributor working on a scoped task (a demo branch, an environment audit, a metric-collection script) produces ambient facts that seem too small to flag as bugs.
• A wiki-first convention forces those findings into durable, cross-referenced structure. Not a notepad, not a Slack message, not chat context that expires. A page with frontmatter, typed edges, and an index entry.
• A collaborator (or the same author in a later session) reviews the wiki structure independently. Typed edges make consequences visible that a single session cannot see, because a single session does not have both sides of the relationship in context simultaneously.
• extends: is where the review pays off. It says "this inherits from that." When the review finds a bug on the child, the edge tells them where else the bug might live.
• The fix cascades along the edges. Correcting the bug corrects downstream conclusions, and (crucially) surfaces prior experiments that need re-running. The wiki logs the re-runs on the same page as the original claim, so the record stays honest.
• Attribution is preserved through the cascade. Chris Frederick's demo branch stayed valid work. Chris Sweet's fix cascade credited both parties. Neither got erased.

None of this depends on both collaborators being in the same room, same session, or same time zone. It depends on both writing into a shared, structured, LLM-maintained knowledge base.

A second, smaller example of the same pattern (2026-06-11)

The demo-branch story is not a one-off. Three weeks later, the same pattern caught a second bug.

On 2026-06-11, Chris Frederick was prepping for a fleet-scale run and filed a routine reference page: Vision-Token-Economics, documenting per-model tokenization limits and cost math. Filed alongside it: Screenshot-Resolution-Bug, with frontmatter criticizes: [[Experiment-11-Productionization]].

The bug: browser.py chunked screenshots only above 7680 px (the API's hard-reject ceiling), but Sonnet's native cap forces silent downscaling on anything over ~950 px tall at 1280 wide. The ITAMCO homepage (1280 × 7396 px) was being read at 21% scale, 560 tokens. Body text 16 px rendered at 3.4 px. The model was reading thumbnails.

The criticizes: edge told Chris Sweet: the Experiment 11 conclusions may be confounded by this. Which they were — the "not visible on this page" answers from ITAMCO's homepage were sometimes literally true because the pages were illegible. The fix landed the same day (main-repo commit 352fe477); the re-baseline that followed gave the project a defensible tier decision: Sonnet 93.6% [89.8, 96.5] > Opus 86.8% [82.2, 90.7] > Haiku 78.7% [72.8, 84.1]. P(Sonnet > Opus) = 99.3%.

Same pattern, different edge type. extends: propagated an inheritance-checking review from the demo branch. criticizes: propagated a doubt-checking review from the resolution bug. In both cases, the wiki structure carried the semantic burden the reviewer would otherwise have had to reconstruct from memory.

The wiki pages that carry the story

Ordered by relevance to the primary story.

1. Experiment 11: Demo Branch for Navy Lunch — Pipeline + First Validation Run — the page at the center of the primary story. Chris Frederick's demo branch, the reported-vs-real coverage table, the fix (1cf819e), the ITAMCO recovery from 8 to 14 answers, the cascade back into Experiments 7 and 8. Read the "Sonnet v2 — With Not-Found Filter Fix" section for the concrete before/after.
2. Experiment 7: Question Deepen — the parent experiment the demo branch declared it extended. Now updated with the audit result: 20/20 reported → 14/20 real → 18/20 after fix.
3. Experiment 8: Second Entity (Schafer) — the sibling experiment the demo branch declared it was related to. Same audit story: 20/20 reported → 12/20 real → 18/20 after fix. The exact parity with the ITAMCO redux at 18/20 is what confirmed the cross-entity transfer claim.
4. Working Hypothesis — the project's central architectural claim (questions replace ontologies). The audit that Chris Frederick's demo branch triggered stress-tested it against real evidence-based coverage. The wiki page shows the hypothesis surviving the correction.
5. Wiki Schema — Web Forager — the living document defining frontmatter conventions, typed edges (with the Edges as Interface Operations table quoted above), the per-entry commit log rule, and the four operations. This is the shape of the discipline that made the story possible.
6. Screenshot Resolution Bug — the second-example page. Uses the criticizes: edge type instead of extends:, but the same pattern: file the finding structurally, let the edge trigger the review.
7. Vision Token Economics — the routine reference page filed alongside the resolution bug. Ordinary documentation whose value emerged only in cross-reference against the extraction pipeline.
8. Eval: Model-Tier Re-Baseline — the measurable payoff of the second-example story. Sonnet 93.6% > Opus 86.8% > Haiku 78.7% with 32 trials and credible intervals.
9. Index and Log — catalog and chronological record. Discoverability by topic and by time, with author attribution.

Direct quotes from the wiki

Chris Frederick's own framing of the demo branch, showing he did not think it would be revelatory:

"I wanted to make sure I could give Jose a simpler demonstration in Obsidian of what we are working on. ... This was solely for that demonstration."

The mechanism of the bug, in the wiki's own words:

"Sonnet's honest 'not visible' responses look like plateau-evidence and trigger early stop, especially because Sonnet front-loads a lot of 'not visible' hedging on the first page (see ITAMCO). Haiku's confident-hallucination behavior accidentally drove deeper exploration in prior experiments — fixing that behavior also broke the implicit assumption that drove page-depth."

The metric-honesty framing:

"The pipeline's questions_answered: metric counts any model response — including 'not visible on this page' — as answered. That overstates coverage. The honest metric is 17 minus the unanswered_questions count, which counts only answers backed by visible page evidence (or explicitly-flagged inference)."

The fix landing and its scope:

"Patched forager/asker.py + forager/ask_vision.py to treat honest 'not visible on this page' hedges as null. ... Committed as 1cf819e on main."

The smoking-gun recovery:

"ITAMCO: +6 real answers (+75%). The fix did exactly what was needed. v1 saw the homepage front-load 'not visible' hedges for 9 questions and plateau-stopped after 1 page. v2 saw those hedges as no-progress, kept exploring, and found the answers on /manufacturing/capabilities/, /manufacturing/defense-and-government/, ... This is the smoking-gun proof that the bug was killing exploration on entities where deeper pages have real content."

The audit outcome:

"Both at 18/20 real under the fix. ITAMCO recovered 4 of 6 hedges; Schafer recovered 6 of 8 hedges. Exact parity between the two entities (18/20 / 18/20) with the same 20-question generic set — confirms the cross-entity transfer claim survives the audit at corrected coverage."

The takeaway

• A shared llm-wiki turns discrete sessions into cumulative project memory. Neither collaborator held state across the boundary. The wiki did.
• Structure matters more than content volume. Frontmatter, typed edges, and cross-references made ambient session notes reviewable in ways freeform prose or chat logs are not.
• The wiki catches bugs their filers did not consider bugs. Chris Frederick did not file the demo branch as a bug report — he filed it as a demo he had offered to drop. The review made possible by the wiki's structure turned it into the corner piece of a project-wide audit that stress-tested the central hypothesis and recovered ~75% more real coverage on the flagship entity.
• Typed edges are not decoration. extends: said "this inherits from that." When the child page was found to have a bug, the edge told the reviewer exactly which pages to check next. That is not a graph. That is an inheritance interface.
• Attribution scales trust. by: <human> via <agent> on every log entry, one commit per entry, git-blame-traceable. Chris Frederick's demo work stands. Chris Sweet's fix cascade stands. Both credited.
• The pattern is generalizable. Any multi-session, multi-contributor codebase where session-context evaporates can adopt the same primitives: typed pages, typed edges, index, log, per-entry commits. The llm-wiki template packages that.