We measured it, and the fancy thing lost

The uncomfortable part of building the harness first is that you then have to live with what it tells you, and over about three days at the start of July it told me, in four separate ways, that the more sophisticated version of our retrieval system was not better than the simple one we already had. I went into that stretch expecting to ship dense embeddings, an LLM reranker, and a wider context window, because all three are what the literature and the ecosystem would tell you to do, and all three had a clear story for why our known misses would improve. None of them survived contact with a scorecard. I want to walk through them in order, because the pattern that emerges matters more than any individual result.

#Dense embeddings, twice refuted

We had already found that lexical tf-idf beat dense retrieval by sixteen points on our corpora. But that test used Apple's word-level NLEmbedding, mean-pooled, which is about the weakest possible embedder and not a fair trial of the idea. So we ran the fair one. We wired a real retrieval-fine-tuned transformer – thenlper/gte-small, 384 dimensions, through swift-embeddings with MLTensor and no Python – as an actor behind our existing ChunkEmbedding protocol. It ships inert, with the semantic weight defaulted to zero.

Before spending a corpus run on it, I probed the embedder on three inputs that mirrored our actual misses, and the results were a preview of everything that followed. On paraphrase it was strong: an income query matched a passage containing the monthly figure over a calendar page by a margin of 0.12, which is exactly the upside dense is supposed to buy you on label-poor documents. On form numbers it was nearly blind, preserving the distinction between two similarly-named court forms by only 0.013, where lexical's exact token match is decisive. And on vendor names it was actively wrong, ranking a passage about the wrong company higher than the right one (0.708 against 0.692) because the two passages shared sentence structure and the distinctive vendor token had been compressed into near-interchangeability.

The full run on all three corpora, sixty-seven supporting-document lookups, came back as a wash: two borderline recoveries against two regressions, plus thirty cases that neither retriever finds. Those thirty are OCR or ingestion problems no retriever can touch. We also assessed adopting an off-the-shelf package wholesale, and the finding there was structural rather than numerical. The package's public search is dense-only, so adopting it would have meant shipping precisely the configuration we had already refuted, with its only advantage needing to overcome a sixteen-point deficit first. So we did not adopt it, and the embedder code ships inert as a validated seam for a future regime where dense might actually pay.

#The reranker, which could not lose and did

The reranker was the one I believed in most. The survey ranked it our top lever, and the external evidence is genuinely good – published work on contextual retrieval showed a reranker cutting top-20 failure from 2.9% to 1.9%. The argument for it in our system was almost airtight. It only reorders candidates we already retrieved, so it structurally cannot hurt recall, and it runs on the Qwen3-4B we already load, so it adds no dependency at all. A lever that cannot hurt and might help is about as easy a decision as this work offers.

It reordered a twenty-candidate lexical pool through one prompt and kept the top eight. On the legal corpus, against the shipped topDoc3 baseline, it scored 10 out of 15 both with and without: zero grade changes, at 2.3 times the latency, with median response going from 5.46 seconds to 12.82 and tokens per second halved. It was not silently falling back either, which I checked, because a zero-delta result usually means the feature never ran. On one income question the baseline cited the right form and the reranked run cited nothing at all, so the reordering was genuinely happening. It still recovered nothing.

Legal was the only clean A/B I had at the time, and I said so, because the one thing worse than a disappointing number is a general claim resting on a single corpus. The second corpus arrived at the end of August, when I re-ran the pair on an external academic benchmark, and it was worse than a wash: 79.55% without the reranker against 68.18% with it, at 1.75 times the latency, seven questions flipping and only one of them a gain. The gain was exactly the case the reranker was built for – a real retrieval miss where the plain pipeline had cited nothing, and the wider pool surfaced the golden document. The six losses were all questions the baseline already had right, where a topically-similar chunk got promoted and displaced the correct one, and one of them was a negative question where the promoted chunk gave the model just enough to answer instead of decline. That last one is the finding I would carry furthest: a reranker that costs an honest decline is strictly worse than no reranker at all.

The mechanism is the durable lesson and I have carried it into every ranking conversation since. A reranker reorders the pool it is handed, and every one of our misses lived outside that pool. On the income question the correct document was already retrieved and already cited, and the miss was the specific chunk – the model answered with a different line from the same form – which is a chunk-granularity problem that reordering documents cannot reach. On two others the document sat at lexical rank 24 and rank 79, far below a top-20 pool, so the reranker never saw them at all, which is a scoring problem. Reranking targets in-pool mis-ordering, and in-pool mis-ordering is the one failure we did not have.

We shelved it the way we shelve things here, which is to keep the code and default it off behind a no-op implementation, so the pipeline with the switch off is byte-identical to the pipeline that never had the feature:

public struct IdentityReranker: ChunkReranking {
    public nonisolated init() {}
    public nonisolated func rerank(query: String, candidates: [RetrievedChunkDTO]) async -> [RetrievedChunkDTO] { candidates }
}

Shelving rather than deleting matters, because the reranker is not wrong in general, it is wrong for the misses we currently have, and those two statements have very different shelf lives.

#Chunk selection: the experiment we planned to revert, promoted

Not everything lost, and the thing that won is instructive because I had already written it off. We ran a three-way A/B on chunk selection: flat (a per-document cap of two chunks, the proven baseline), topDoc3 (the single top-ranked document lifted to three chunks), and wholeTopDoc (the top document contributing all its chunks in reading order, bounded by a character budget). That third one was the handoff's number-one ranked lever, the "whole top doc" idea, the thing everybody expects to work.

topDoc3 won legal and presentations with zero regressions anywhere. It had been slated for revert as a no-op, and the A/B promoted it instead. wholeTopDoc's apparent tax gain decomposed under inspection into three separate things: one real honesty win, one grading fluke where a declining answer matched a loose needle, and one real regression where the wrong top-ranked document drowned out the right one and the system answered about the wrong county entirely. On many-document collections, concentrating on a single top document sacrifices the cross-document breadth that aggregation needs and amplifies ranking errors when the number-one document is wrong. We kept the code and shelved it, off by default, because its actual regime is a long-single-document corpus we do not have yet.

#Context volume is not a lever at all

The last one was the cleanest and the most deflating. I asked, in early July, whether we had reconsidered the reranker, the chunk count, the context size and the chunk size recently, because I was sure there was optimization left in them. We optimized for correctness with latency explicitly be damned, and we ran the full deterministic matrix:

Three modes, a four-fold range in context window, a six-fold range in latency, and an identical total. Each mode wins a corpus and loses another, which is the recall-versus-dilution trade playing out as pure zero-sum. Feeding the model a full 64k window and 48 sources does not help on net even when you have decided you do not care how long it takes.

That table also exposed something about our own defaults. The shipped default had been .wide, chosen on the first of July on stochastic data where it showed plus one on two corpora, which under greedy decoding was almost certainly sampling noise. Under deterministic measurement topDoc3 and wide are equal, and topDoc3 is three times faster and keeps tax at 17/17. For a vault that skews toward forms and statements rather than sprawling presentation decks, the faster mode is the better default, and it is the one in the shipped code today (defaultRetrievalMode = .topDoc3). We had been shipping the slower option on the strength of a coin flip.

Put the four results next to each other and the shape is hard to miss: lexical baseline 42/50, BM25 reorder net flat to slightly negative (legal held, presentations lost two), context volume 42/50 across all three modes, real-embedder semantic fusion 42/50 with a small honesty regression on top. Every retrieval axis we could think of converges on the same number, which is what a local optimum actually looks like when you measure it instead of arguing about it.

#What I take from it

I do not read any of this as "the fancy techniques are bad," and I want to be careful there. The survey also validated plenty of what we do: our chunk size, our grounded and cited prompt shape, our gated top-document expansion. The literature confirms that gated expansion is the correct form of the parent-document pattern, while the ungated version – the one we refuted – is documented as dropping entity extraction by ten to fifteen percent. External evidence was useful all the way through, and it was never decisive on its own.

The thing that changed how I work is smaller and more portable than any of the individual findings. Every one of these techniques arrived with a good mechanism story, credible outside evidence, and in the reranker's case an argument that it could not possibly make things worse. Each of those stories was a hypothesis about our particular regime rather than an answer to it. Our regime is titled, entity-rich documents, roughly thousand-character chunks, a 4B model on a laptop, no vector database, and no cloud. Best practice is written for someone else's regime almost by definition, and the only way to find out whether it transfers is to build the thing that can tell you it did not. After all of that measurement we chose to stop optimizing retrieval and go build validation corpora instead, which felt like an admission of defeat for about a day, and then felt like the first genuinely well-founded decision we had made in weeks.