Retrieval’s bottleneck is moving from finding to selecting

For a given query, the task of a retriever is to find the most relevant chunks from documents in a corpus. Modern retrievers usually perform at least two steps. First, they find a pool of likely candidates which they retrieve from the corpus. Then, they select the most relevant chunks from the candidates. For example, a common approach in classic RAG is to retrieve candidate chunks via hybrid search (BM25 + vector search) and then refine the selection via a specialized reranker, returning only the top-k.

Until recently, the biggest failure mode in retrieval was finding the right candidates. That is, the retriever would often end up with candidates that made it impossible to select the right chunks for the downstream task. Over the past year or so, much of this gap has closed: embedding models have improved, LLMs are now good at rewriting and decomposing queries before searching, and agentic retrievers search iteratively until they have gathered enough.

This suggests a hypothesis: the bottleneck in retrieval is moving from finding to selecting. Search now surfaces most of what a query needs, and the loss happens afterwards, when the pipeline decides which retrieved chunks the model gets to see. This post tests that hypothesis by tracing every gold chunk (a chunk that a complete, correct answer actually needs) through a production retrieval pipeline on questions from our Company Knowledge Bench. The shift is already under way: search still misses some of the gold chunks, but of the ones it did surface, almost a third were never shown to the model.

Finding got much better

The clearest indication of how far finding has come is code search. Models have become good enough at using grep that the semantic search path no longer helps coding agents in a meaningful way. A lexical tool, driven by a capable model, is now sufficient to navigate large codebases.

Grep, however, does no selection. It returns every match, so all filtering happens inside the model's context window. The selection problem doesn't go away; it just moves into the model's context. Further, code is a special case: it has a strict syntax, identifiers are distinctive and spelled the same way everywhere, with no typos or synonyms to account for, so a capable model can refine exact patterns until it finds the right file. This is also why grep isn't enough for Kapa, which serves technical companies across many fields from dev tools to semiconductors. Company knowledge is spread across docs, tickets, forums, and release notes, each with its own vocabulary, and one question can touch many of them. In a large, varied corpus, a literal match can return more than a model can take in its context, and the same concept may be phrased, or misspelled, in ways no pattern anticipates.

Where the gold goes

In our Company Knowledge Bench each test case consists of a user query and the chunks a complete answer requires: its gold chunks. Following the gold through a retrieval pipeline shows where retrieval failures happen.

For our analysis, we consider a simplified pipeline which involves two models: a retriever which searches the corpus and picks which chunks to keep, and an answer model which writes the response from that final set. Here we consider a retriever model similar to the one in our reranker analysis, which uses GPT-6 Luna for query decomposition and Voyage-3 for ranking the retrieved chunks. For each query, we retrieve 50 chunks and then feed up to 100 chunks into the reranker which returns the top 15. This setup is only for illustrative purposes; we will talk about more realistic agentic retrievers later. As we explain in our blog post, some gold chunks are interchangeable alternatives; we will ignore this here for the sake of simplicity.

There can be two reasons why a gold chunk didn't make it to the answer model:

1. The retriever never found the gold, i.e. the reranker could not select it as a top chunk.

2. The reranker saw the gold but decided it was not relevant enough to make it into the top-k chunks.

The first error means the retriever had trouble finding the gold, the second error means it had trouble selecting the gold. Historically, finding was always much harder than selecting. Roughly 18 months ago, about 80% of the failed eval cases were caused by the gold never entering the reranker, and only 20% by the reranker itself. Today, about half of the failed test cases are caused by search. The other half fail after search, when the reranker selected the wrong chunks for the answer model. The insight we can get from this analysis is simple: the difficulty in retrieval is shifting from finding to selecting.

Counted by chunk, the picture is sharper. Of all the gold chunks search found, 34% were never shown to the agentic retriever. The rest of this post calls this share the selection gap.

The hidden gold chunks are not necessarily buried deep in the reranked pool. Some are, but others ranked well in the search results, just not well enough to make the cut. That makes the obvious remedy tempting: show the agentic retriever more.

You can buy recall with tokens

Showing more chunks to the answer model obviously works. For example, going from the top 5 to the top 10 chunks per search increases the share of questions where the answer model receives the gold chunks by 14%. However, the returns shrink quickly. The distribution of missed gold chunks may be dense for low k but thins into a long tail further down, while every step down the ranking adds the same amount of text to the answer model's valuable context window. Precision, the share of returned chunks the answer actually needs, falls as fast as recall rises, and every extra chunk is text the answer model has to read.

“Just let the answer model ignore the noise”

A simple solution would be to just let the answer model deal with it. A case for returning more comes from SID: their technical report argues that a capable answer model can easily cope with a few irrelevant documents but cannot recover a missing one, so over-reporting is safer than under-reporting. A missing chunk is indeed worse than an extra one. But recall first is not recall at any price:

• Every chunk is paid for. Retrieved chunks make up about two-thirds of the cost of a query in Kapa's assistants (see our pruning post).

• Models are not infinitely powerful. Accuracy drops as inputs grow, and faster when the extra text resembles what the model is looking for (Liu et al., 2023; Chroma, 2025).

• The worst noise looks relevant. An outdated page or a similar feature in another product is exactly what a ranker places near the top.

Therefore, today's retrievers are often forced to balance recall and precision. This trade-off is one of the oldest findings in information retrieval, dating back to the late 1950s. Back then, in "Delegation of Classification", Robert Fairthorne had already named the two ends: a system can return all but not only the relevant documents, or only but not all of them. Getting all and only is the hard part, and a metric that only checks whether the gold was retrieved cannot tell the two apart.

Nevertheless, real retrievers often follow SID's argument and tend to drift toward the first case. For example, on Company Knowledge Bench, an agent that searches with grep performs comparably to Kapa's deep retriever when it comes to recall. However, for grep, only about one in twenty-five of the chunks it returns is needed, while Kapa's deep retriever returns eight times fewer text tokens (see our blog post).

This is why one needs to look at precision next to recall. A recall score on its own rewards returning more, so any comparison that ignores what came along with the gold favors the retriever that returns the most.

Agentic retrievers are only a partial solution

So far we have only looked at a simple one-shot retriever. Most modern pipelines use sophisticated agentic retrievers, which iteratively refine search results. However, this merely shifts the problem from the answer model's context to the agentic retriever's context and the same limitations apply. What makes selection hard is a loop. The only reliable way for the agentic retriever to judge whether a chunk is needed is to read it, and reading costs exactly the context that selection is meant to save. For this reason, typical agentic retriever pipelines put a reranker in front of the agentic retriever that filters the chunks before they enter its context. As a result, the decision about what the agentic retriever reads is made before it has read anything. Rerankers are not built for that decision. They judge each chunk on its own, while relevance often belongs to a set: a chunk can be useless alone and essential next to another (see our pruning post). And the right number of chunks differs for every question. So the selection needs the context that it is meant to save, and we have completed the circular argument.

There are methods that try to address this limitation. List-wise rerankers read the candidates together rather than scoring each one alone, so they can account for chunks that are redundant or only useful in combination, but they do so by handing the reading to a cheaper model with its own context limit. Hierarchical approaches instead show the agentic retriever a short preview of each chunk, such as a title, a summary or its first lines, and let it decide which ones to read in full. The retriever then judges for itself, at the cost of reading a preview of everything and an extra step for every chunk it opens. Here the problem shifts to writing previews that keep what makes a chunk useful.

Even though agentic retrievers aren't perfect, they offer a lot of freedom in their design and currently top most retrieval benchmarks. Kapa's current approach to agentic retrieval tries to address this shift from finding to selecting both in its search tools and in how it returns the chunks to the downstream model. Most retrievers return a fixed number of chunks, or a set that only grows as the agent keeps searching (append-only). Kapa's deep retriever instead returns as many chunks as a question needs, and can drop chunks that turn out not to be needed. On Company Knowledge Bench, the deep retriever scores higher than the grep agent while returning about an eighth of the text, at about six times the precision (benchmark results). However, it does not fully remove the trade-off, and open questions remain: does selection belong to the retriever or to the answer model? How much recall is worth giving up for a cleaner context? How can minimality be benchmarked when several sets of chunks can answer the same question?

What is clear is where the gap has moved. Search now finds most of what a question needs. The gold lost after search is the part of the problem that is growing fastest, and the part that recall-based metrics cannot see. Progress from here depends on efficient selection: keeping the gold without paying for everything around it.

Before improvement comes measurement. Building Company Knowledge Bench was the first step, because it made it possible to trace the gold through the pipeline. What's left is keeping it.

Follow the blog for more findings as the work continues, or reach out with questions.