Jev is NOT a state-of-the-art reranker (but it's close)

Jev, the new decision model from TypeSafe AI, is getting a lot of attention. It's fast, it's cheap, and people quickly started applying it to retrieval. Some teams are already swapping their dedicated rerankers for it. But the results published so far disagree. Testing at turbopuffer calls it a state-of-the-art search reranker, while DevX Labs found it the fastest and cheapest but twelfth on accuracy.

We run millions of retrieval queries a month at Kapa, and every one of them goes through a reranker, so we wanted to see for ourselves. We tested Jev against six other rerankers on Company Knowledge Bench, the retrieval benchmark we published earlier this month.

TL;DR: Jev is a good reranker, and it beat every Cohere and Qwen model we tried. But Voyage AI's rerank-3 came out on top, and it was also the cheapest and the fastest.

Benchmarking on real-world company knowledge

Company Knowledge Bench has 1,000 eval cases built from real production queries. Each one comes with its own knowledge base, frozen as it was when the question was asked, and a labelled criterion that says which chunks a complete answer needs.

These knowledge bases are what our customers actually index: product docs, Slack threads, support tickets, wikis and code. That means stale pages, contradictory facts and many ways to answer the same question. We built our own benchmark because academic benchmarks looked nothing like our production traffic. You can read more about how it works in the benchmark post.

The score measures how much of each criterion the top 10 reranked chunks cover, with partial credit when only some of the required chunks are found, averaged across all 1,000 cases.

How we set up the test

This is a test setup, not what we run in production. We picked a standard retrieval pipeline because it makes rerankers easy to compare. For each question, an LLM splits it into three search queries. We retrieve the top 30 chunks for each query, merge them with reciprocal rank fusion, and send the top 50 to the reranker. Then we score the top 10 chunks it returns.

We tested Voyage rerank-2.5 and rerank-3, Cohere rerank-v3.5, v4.0-fast and v4.0-pro, Qwen3 reranker 8B via OpenRouter, and Jev 1.13 in two modes. We capped the candidate set at 50 because Jev needs one call per chunk, and 50 is a common candidate set size for reranking anyway.

Two ways to turn Jev into a reranker

Jev isn't trained as a reranker. It's a general-purpose decision model: you give it a state (here, the query and one chunk), a question and a set of labels, and it returns a probability distribution over those labels. To turn that into a ranking, you pick a question and a way to collapse the distribution into one number per chunk. There are two standard ways to do it, and we tried both.

Noul (binary). We ask a yes or no question: is this chunk relevant to the query? Jev returns the probability of yes and of no, and we sort the 50 chunks by the probability of yes. It's a pointwise classifier, with one call per query and chunk pair.

Score (graded). We ask how relevant the chunk is on a scale of relevance levels, from unrelated to a perfect match, each mapped to a number. Jev returns a probability for each level, and we sort by the expected score: each level's value times its probability, summed. This is the method turbopuffer used. Because it uses the whole distribution, it can tell a chunk that is certainly partly relevant apart from one that is either a perfect match or unrelated.

Results: Voyage wins on score, cost and latency

Voyage rerank-3 scored 0.535, at $1.91 per 1,000 questions and 0.18s median latency. No other reranker beat it on any of the three.

Jev came in just behind. At 0.519, the Noul mode beat Cohere's newest model, rerank-v4.0-pro, on score (0.506), cost ($2.49 vs $3.42) and latency (0.30s vs 0.77s). The two Jev modes came out almost even: 0.519 for Noul and 0.516 for Score, with Score costing a bit more ($2.82 per 1,000 questions). On our data, the extra detail of the graded scale didn't improve the ranking.

The open model was the most expensive. Qwen3 reranker 8B cost 4.3x more than Voyage, took 14x as long, and ranked second to last.

Every model saw the same 50 chunks, and only the order changed. Even so, the best score was 18% higher than the worst (0.535 vs 0.454).

What we took away

First, results depend on the data. turbopuffer's benchmark and ours point in different directions, and both can be right. If you're choosing a reranker, test it on your own data.

Second, Jev is worth a look if you want a reranker you can steer with a rubric. For our domain, though, Voyage was the best, so we're sticking with it. Good job to Tengyu Ma and the rest of the Voyage team at MongoDB.

Cohere costs come from billed units. Qwen uses OpenRouter pricing, so self-hosting it would change the numbers.