The ceiling that wasn't

For most of a month I believed the engine had a hard ceiling, and I had a number to prove it: 42 out of 50, on the tuned corpora, sitting there unmoved no matter which retrieval knob I turned. We had run the A/B tests properly by then, one lever at a time, with a greedy deterministic decode so the same question produced the same answer every run, and the result was almost eerie in its flatness. Context volume did not matter (topDoc3, wide and max all tied at 42/50). A query-time LLM reranker was a net loss, zero recoveries at roughly twice the latency. Finer chunking with more overlap measured negative. When every knob on the console produces the same reading, you start to suspect the console. That took me longer than it should have.

The honest interpretation available to me at that point was that the retrieval-plus-small-model stack had simply reached the limit of what it could do on documents, and that further gains would require something structurally different, a bigger model or a real semantic index or both. That interpretation was comfortable, which should have been the tell, and it took me too long to hear it. What I actually had was a ceiling measured entirely on corpora that had been shaped, question by question, alongside the engine that was being measured – tax, legal, presentations, banking – and a corpus that grows up next to the system it grades is not a neutral witness, it is a member of the family.

So I built two corpora the engine had never seen, from my own real documents: an insurance set and a utilities set, 410 and 365 documents respectively, with questions written against what the documents actually say rather than against what I hoped the engine could reach. Ground truth was verified against the source PDFs (the orchestrator opened the highest-risk column-flattened statements by hand, because a flattened text layer will happily pair a label with the wrong neighbor's number). Same knobs as the tuned runs, greedy topDoc3, snapshot 20260704-165209, so the comparison would be clean. Then I ran it and read the table.

Fifty-three percent on insurance and sixty-nine on utilities, against a tuned set that had been sitting at 100 and 83. I want to be careful about how I describe my reaction to that table, because the honest version is not disappointment, it is something closer to relief mixed with a slightly sick feeling, the way you feel when a test you were about to skip turns out to catch something real. The 42/50 ceiling was not a property of the engine at all, it was a property of the corpora I had been grading it on, and the engine's true behavior on dense unseen real-world material was materially worse than the number I had been quoting to myself for weeks.

#What was actually breaking

The useful part was not the score, it was the shape of the misses. That shape was remarkably consistent across both new corpora. The dominant failure class turned out to have nothing at all to do with scoring. The query would name a specific dated or versioned document ("the March 2017 bill," "the 2019 homeowners policy"), retrieval would key on the signal the near-duplicates all share – the provider name, the policy type, the form vocabulary – and surface a different month's or a different year's copy of essentially the same document. There were many bills from the same telecom provider, one per month, all nearly identical apart from a date and a dollar figure, and multiple homeowners policies, one per renewal year. The distinguishing feature was a date, and neither the lexical nor the semantic side of retrieval was keying on the date with anything like enough weight.

Here is the part I did not expect, and the part that changed how I think about honesty as an engineering property. The engine did not answer those questions wrong. It declined, and it declined in a way that told me exactly what had happened. The phrasing is the point:

the March 15 2017 bill from that provider isn't in the sources – I see Sep 15 2017 …

That is a decline, which is scored as a miss, and it is also the single most useful diagnostic output the system produced that week. A confident wrong answer would have told me only that the engine was wrong, and nothing else. The decline told me what the engine had retrieved instead of the right thing, which named the failure mode outright: the retrieval had found the right provider and the wrong date. Every one of those misses came with its own explanation attached, and the taxonomy assembled itself out of the declines rather than out of my guesses about them.

That reframed the whole problem, because the documents have dates – in the filename, in the extracted date metadata, sitting right there in the store – and retrieval simply was not filtering or ranking on them. This was not the below-pool scoring problem I had already exhausted. BM25 weights, context volume, semantic embeddings, all of that had been measured and had gone nowhere. It was a metadata gap, which is a different axis entirely, and the first lever I had found in weeks that had a real mechanism behind it rather than a hope.

#The lever I was sure about, and was wrong about

Grounding the next phase produced its own correction, which is worth telling because it is the more embarrassing half. The model-free retrieval probe classified the dominant within-document miss class as "capped out" – the answer chunk was present in the full ranking, at rank 11, 24, 104 – and the probe's own remedy label was literally "relax cap." Representative ranks across the six corpora looked damning in exactly the right direction: an income chunk at 104 of 614, another at 24 of 873, tax at 12 through 111, insurance at 24 through 84, utilities near-duplicate bills at 14 through 106, banking as deep as 214. The obvious fix was to let the top-ranked documents contribute more chunks than the cap allowed.

So I implemented it as a live-tunable per-document cap relaxation. Five strengths, swept against the topDoc3 baseline at the shipped k=8:

baseline topDoc3 = 53 answer chunks surfacing
  5 chunks × 1 doc  → 49
  4 chunks × 2 docs → 49
  5 chunks × 2 docs → 48
  8 chunks × 1 doc  → 44
  8 chunks × 3 docs → 44

Every configuration lost ground, and not one recovered a single capped chunk anywhere in the sweep. The mechanism, once measured, is obvious in hindsight: at a fixed context budget the relaxed top document eats the extra slots and displaces chunks that were surfacing from the second and third documents, so the more aggressively you relax, the more cross-document breadth you burn. Worse, correlating each capped question's chunk rank against its document rank showed that roughly 26 of about 40 "capped" questions were not cap problems at all – the answer's document never reached the top ten distinct documents, so no amount of cap relaxation could structurally reach it. The probe's label had conflated two failure classes. I had spent a phase implementing the remedy for the smaller one.

The lever went in the drawer, the shipped default stayed topDoc3, and the date axis, which the validation corpora had pointed at from the beginning, turned out on closer measurement to be partly solved already – the shipped date self-query boost rescues roughly 12 of about 24 pure-lexical document-ranking misses into the top ten, while over-broadening on bare-year queries in a way that sinks number-dense forms, so a bare "the 2025 W-2" lifts every same-year document into one undifferentiated cluster and the sparse form drops out. That regression is now the top lever on the list, which is a strange place to end up. The highest-value next move was undoing damage done by the previous fix.

What I take from all of it is less about retrieval than about measurement. The 42/50 was never a lie, it was an accurate answer to a question I had stopped noticing I was asking, which was "how does the engine do on the documents I built for it." Corpora built from my own real material asked a different question and got a much less flattering answer, and the misses they produced were legible only because the engine, when it could not find the thing, said so plainly instead of guessing. I have come to think that an honest decline is not a failure the way the scorecard counts it – it is the system handing you its own bug report, and a system that guesses instead is one that costs you the diagnosis along with the answer.