Retrieval as trajectory
The context envelope framing, and why dense RAG breaks when the envelope lives in a graph.
Public writing about retrieval-augmented generation treats retrieval as “find the chunk that answers the question,” with a carve-out for small corpora where one chunk is enough. Scientific work does not fit that carve-out: a claim is supported by evidence, not answered in one chunk, and the supporting chunks are inherently spread across the corpus with provenance chains that cross documents. What retrieval actually has to assemble is a context envelope: the local neighborhood of concepts, layout cues, and cross-document evidence around the question. Dense similarity assembles the envelope well when it fits inside a small top-K. When it does not, the assembly job becomes a trajectory problem: a walk over a graph the embedding does not encode.
The claim
RAG is not about answering questions. It is about assembling a context envelope: the local neighborhood of concepts, layout, and cross-document evidence around the question. When the envelope is missing, the language model cannot answer, even when the strictly correct chunk is in the top-K. Dense retrieval assembles the envelope well when it fits inside a small K. Once the envelope is spread across a graph that similarity cannot traverse, the assembly job is not a retrieval problem anymore. It is a trajectory problem.
The rest of this page unpacks what that means, why it matters, and what an alternative looks like.
The failure mode
Dense RAG at K = 10
Two constants set the operating envelope for production RAG:
- K = 10 is the modal production cutoff. It keeps latency and cost down and it puts every retrieved chunk in a position the language model actually attends to. This is the number to design against.
- K = 30 is the upper bound, the point at which lost-in-the-middle effects (Liu et al., 2024) start eating into answer quality faster than extra recall helps. The U-shaped attention curve means chunks in the middle of a long context are systematically harder for the model to use than chunks at the ends. This ceiling matters because it closes the escape hatch of “just retrieve more.”
What plain dense retrieval delivers to the language model at K = 10 on a 21,100-chunk MuSiQue (Trivedi et al., 2022) corpus, per hop group (fraction of TEST questions whose slot-i gold chunk lands inside the top 10):
| Hop group | g1 | g2 | g3 | g4 |
|---|---|---|---|---|
| 2-hop | 100 % | 17 % | ||
| 3-hop | 97 % | 37 % | 0 % | |
| 4-hop | 87 % | 23 % | 3 % | 0 % |
Read the table by row. g_1 is the gold chunk the retriever ranks highest for a question, g_2 the next, and so on. Dense retrieval reliably delivers the direct-match chunk (g_1) at every hop count. Everything else falls off a cliff.
The sharper reading is diagonal. A multi-hop question can only be answered when every gold chunk it needs is inside K, so the effective retrieval quality per hop group is bounded by the deepest slot’s coverage:
| Hop count | Effective retrieval at K = 10 |
|---|---|
| 1-hop | 100 % |
| 2-hop | 17 % |
| 3-hop | 0 % |
| 4-hop | 0 % |
Dense retrieval delivers single-hop reliably and multi-hop essentially never at the K the attention curve lets you use. The 1-hop row is inferred from g_1’s 100 % coverage across all hop counts; MuSiQue does not include 1-hop questions, but this is what dense would give you.
The failure is not that the deep chunks are missing from the corpus. It is that dense retrieval ranks them into the thousands: the 4-hop g_4 median rank is 1,796 out of 21,100. Raising K past the K = 30 ceiling does not save it, because a chunk ranked at 1,796 is nowhere near the pool a reranker would even see. This is a ranking gap, not a recall gap, and turning the retriever dial does not close it.
This is not a claim about dense retrieval being bad. It is a claim about what dense retrieval is for. It is a one-shot query against a similarity manifold. If the answer lives one hop away in similarity space, it finds it. If the answer lives across a graph that the embedding does not encode, it cannot.
On terminology
Two kinds of “hop”
A lot of confusion in the multi-hop literature is because the word “hop” is doing two different jobs.
Reasoning-hops are a property of the question. How many inference steps does the question demand? “What year was the director of Blade Runner born?” is a two-reasoning-hop question because the model has to resolve a director and then a birth year. This is the sense of “hop” used in MuSiQue and elsewhere in the QA-benchmark tradition.
Retrieval-hops are a property of the index. How many graph edges does the supporting evidence live across? A retrieval-hop is a step across a link in the corpus itself: a citation, a shared concept, a cross-document reference. This is the sense of “hop” used in HopRAG (Zhang & Others, 2024) and IRCoT (Trivedi et al., 2023).
The two are correlated but not the same, and mistaking one for the other leads to the objection we heard most often at talks:
“If the question is 3-hop, why not just decompose it into three 1-hop questions?”
Because question decomposition operates on the question side. It reduces the reasoning-hop count. It does not help you assemble the context envelope, which lives on the index side. If the supporting evidence is spread across a graph, running three separate similarity queries against that graph gets you three separate one-hop retrievals, none of which is the walk-informed neighborhood that would have surfaced the envelope in one shot. Decomposition composes with graph retrieval; it does not replace it.
The mechanism
Retrieval as a trajectory in graph space
The shift from dense RAG to graph-informed retrieval is a shift of the space the retriever works in.
Dense retrieval lives in document space. Every chunk is a point on a continuous embedding manifold, and “close” means small cosine distance along that manifold. When the manifold happens to align with the reasoning structure the answer needs, this works. When the answer chain requires connecting two chunks that use different surface language to describe the same underlying thing, the manifold cannot help. The chunks are far apart in embedding space because their tokens differ, and cosine has no memory of the corpus-level relationships that would tell you otherwise.
Graph-informed retrieval lives in graph space. Nodes are chunks and the entities extracted from those chunks. Edges are structural: a chunk is linked to every entity that appears in it. Two chunks are “close” not because their surface tokens are similar but because they mention the same entities. Retrieval becomes a walk on this discrete structure. The relationships the embedding did not encode are the edges the walk follows.
Formally, a random walk with restart produces a stationary distribution over the graph:
\[ \boldsymbol{\pi} = c \cdot \boldsymbol{\pi}_0 + (1 - c) \cdot \mathbf{P}^{\top} \boldsymbol{\pi} \]
The walker restarts at a seed distribution \(\boldsymbol{\pi}_0\) with probability \(c\), otherwise follows an outbound edge, and iterates until \(\boldsymbol{\pi}\) stops moving. The result is a personalized-PageRank score over every node, telling you how much neighborhood a node absorbs from the seed. This is a local computation in graph space in the same way a top-K cosine query is a local computation in document space: both surface the neighborhood near a probe, but they measure the neighborhood on completely different structures.
The animation below runs this. Yellow nodes are the seed set; darker nodes have accumulated more probability mass through the walk; the graph structure controls where the mass ends up.
Live simulation by Priscila Moreira. Refresh the page or click a node to reseed. What matters visually: mass flows along dense sub-graphs, and short paths through shared entities pull mass across document boundaries that similarity search would treat as unrelated.
Why the walk happens before the question arrives
The RWR is not run at query time. The stationary distribution is computed once, offline, and cached against the corpus. When a question comes in, the retrieval step is:
- Seed a distribution \(\boldsymbol{\pi}_0\) from the question.
- Read off the walk-informed neighborhood around that seed.
- Re-rank the neighborhood locally against the question.
This is why “just decompose the question” is not a substitute. The graph the walk discovered is already there. The question is used to choose which walk-informed neighborhood to pull, not to run a fresh walk from scratch. Decomposition helps you seed more precisely; it does not replace the trajectory.
Concepts as closeness
Entity similarity as a surrogate for entity resolution
If graph edges are what make retrieval work, the graph has to encode conceptual closeness somehow. Two chunks that discuss the same underlying thing should be linked even when they use different surface words for it.
The classical way to do this is entity resolution: cluster every surface mention of “Barack Obama,” “President Obama,” and “Obama” into one canonical entity node. Do this hard-clustering step across the whole corpus, once, and now chunks about the same entity share a graph neighbor no matter which surface form they use.
Entity resolution is expensive, brittle, and rarely worth it at scale. It requires a global pass, disambiguation rules, and re-runs whenever the corpus changes. What we do instead is skip the hard clustering and let the initial conditions do the resolution work.
The entity layer of the graph keeps surface forms distinct: “Barack Obama,” “President Obama,” and “Obama” are separate nodes. The seed distribution \(\boldsymbol{\pi}_0\) is a softmax over question-vs-entity similarity, computed against every entity label the corpus contains. A question about the Obama administration seeds mass across every surface form of Obama at once, weighted by how close each label sits to the question in embedding space. The walk propagates that mass along chunk-entity edges, and mass accumulates at chunks that any of the seeded surface forms can reach.
Entities that mean the same thing get similarly seeded, and they reach similar chunk neighborhoods through the walk. Entities that share nothing but a surface token (Washington the person vs Washington the state) get seeded together only if the question is genuinely ambiguous, and the walk pulls them into separate neighborhoods. The soft resolution happens at query time, driven by the question, without ever committing to a hard clustering the corpus may not support.
This is what “entity similarity as initial conditions” means concretely, and it is what makes the approach cheap to deploy: no preprocessing pass over the corpus, no per-domain disambiguation rules, no re-runs when new documents arrive.
Why the framing matters
What this changes
The failure table above says nothing about whether any graph-informed retriever solves the problem in practice. That is an empirical question with an empirical answer, and it is a follow-up. What the framing changes is what you are looking for when you build a retriever in the first place.
- Stop optimizing embeddings.
g_1is at rank 1 for every hop count already. Another point of MTEB is not going to move the 4-hopg_4from rank 1,796 to rank 10. - Start thinking about the graph the evidence lives on. Wikis and citation networks give you one for free. Corpora that do not come with links get an entity layer, and question-seeded walks over that layer substitute for hard entity resolution.
- Treat retrieval as a two-step: assemble the envelope in graph space (offline, once), then re-rank the envelope against the question (online, per query). Question decomposition composes with this at the second step; it does not replace it.
This is a design shift, not a metric shift. The evaluation of a specific graph-informed retriever against the K = 10 table above is the follow-up study. The point of this page is that the table above is the target to aim at, and dense-alone cannot hit it.
For scientific work
Evidence-based reasoning is envelope assembly
The “small corpus, single-hop” carve-out later on is a real case. It does not describe scientific work.
A scientific claim is not answered by a chunk; it is supported by one. Answering “was gene X shown to affect phenotype Y?” is not “find the sentence that says so.” It is finding the primary paper, its methods, its sample size, its dataset, any preprint or replication attempt, and the chunks in citing papers that agree or disagree. Evidence-based reasoning is multi-source by construction, which makes evidence-based retrieval multi-hop by construction.
Four shapes that put a research corpus in the envelope regime the moment it exists:
- A claim needs provenance. The chunk stating a finding and the chunk describing how it was measured are different chunks, sometimes in different sections of the same paper, often in different documents (paper, supplementary, preprint, dataset paper).
- Replication is a citation walk. “Has this been replicated?” is a question about the citation graph. Dense similarity does not encode citations.
- Method reuse crosses papers. “What else uses this assay?” links chunks that share a method entity, not chunks that share topic words.
- Datasets are cited, not embedded. Modern papers depend on datasets described elsewhere. Retrieving an analysis without the dataset description retrieves incomplete evidence.
A 500-paper reference library is 5,000 to 10,000 chunks, and every question against it carries an implicit evidence chain whether the user articulates it or not. Single-hop is not a scientific way of asking questions. If you are building RAG over a research corpus, the failure table above is the ordinary operating regime, not a distant edge case.
The natural artifact for scientific work in this regime is a project wiki that grows alongside the research: pages that record findings with sources, decisions with reasons, and links between related ideas as they emerge. A wiki-style memory carries the provenance chain scientific reading needs, and it is the graph that graph-space retrieval walks. Every page name is a canonical entity; every link is an edge; every sourced claim is a chunk with provenance attached. Writing the wiki as you go replaces both the graph-construction step and the entity-resolution pass with a single writing habit, and it turns the corpus that the retriever walks into the same artifact that the group is already reading and citing. See the llm-wiki-memory-template for a scaffold that instantiates this pattern for a research project.
Practical takeaways
What to do with all this
- If your corpus is on the order of a thousand chunks and each question needs one chunk to answer, dense RAG is fine. You do not have an envelope problem.
- If your corpus is on the order of ten thousand chunks and your questions cross-reference multiple documents, envelope assembly matters. Dense RAG will get first-hop evidence and miss most of the rest. Scientific corpora are almost always in this regime, and a project wiki (see the section above) is the artifact that turns the envelope problem into a memory problem.
- GraphRAG-style community summaries (Edge et al., 2024) solve a related problem via hierarchical summarization instead of random walks. If your corpus is well-suited to Leiden communities, that path exists.
- If the corpus does not come with links, plan for a graph-construction step. Concept extraction, entity linking, and typed-edge frontmatter all work; pick whichever fits your pipeline.
Learn more
Where to go from here
- The WGRA talk: full slides at chrissweet.github.io/wgra-talk-2026 for the visual walk-through.
- The wiki-memory template: github.com/crcresearch/llm-wiki-memory-template for the pattern that makes graph construction a side effect of writing.
- The Markov RAG source: github.com/tcflab/markov_embeddings_and_rag for the embedding-and-walk pipeline.
Methods
How the K = 10 coverage numbers were produced
MuSiQue (Trivedi et al., 2022) answerable split, 21,100 candidate paragraphs across the whole benchmark, 90 held-out TEST questions (30 each at 2-, 3-, 4-hop). Retriever is plain dense: cosine similarity between question and paragraph embeddings, mpnet, L2-normalized. No entity layer, no walk, no sub-questions. This is the out-of-the-box baseline; that is the point.
The coverage table is a numpy.count_nonzero on rank arrays from a single JSON artifact:
python -m experiments.musique.sparql.design.vanilla_dense_positions --scope full
# writes cache/p3v5/vanilla_dense_21k.jsonThe standard production pattern of retrieve a large candidate pool with a cheap dense or BM25 hybrid, then rerank down to K = 10 with a cross-encoder recovers shallow multi-hop chunks the base retriever surfaces late but does not reach chunks it does not surface at all. A chunk ranked at 1,796 is not in any candidate pool a reranker would receive.
References
Contact
Comments, corrections, and pointers to related work are welcome as issues on this repo, or by email to csweet1@nd.edu.
This site is versioned. When a claim changes, the page changes, and the commit history records why. It is meant as a living reference, not a frozen artifact.