Grading the grader

Every corpus in this project was built by me or by an agent working for me, from documents I own, with questions I wrote and answer keys I verified, which is the only practical way to measure a system that reads private financial documents on a phone. It is also a closed loop, and I had been aware of that for months in the abstract way you are aware of a thing you have no plan to fix. The scores were rising, the levers were being measured honestly, the methodology was sound as far as it went. None of that addresses the possibility that the instrument itself had a systematic error every corpus shared, because every corpus came from the same author with the same assumptions.

So I built an adapter for an external benchmark, which took a weekend and a fair amount of grumbling. The point was never the score. The Open RAG Benchmark ships question sets over arXiv preprints with relevance judgments as third-party ground truth, and the property I wanted from it is simply that nobody on this project wrote it. The questions were not phrased with our retrieval in mind, the golden-document labels were not chosen by anyone who knew what our engine finds easy, and the score therefore cannot be tuned the way a hand-authored corpus quietly can. Fifty golden PDFs plus ten distractors, forty questions plus four negatives, graded on citing the qrels golden document with needles mined from the gold answers. The needles are kept only when they appear in the gold section text and are absent from the question itself, because a needle the prompt hands you proves nothing about retrieval.

The first run came back at 75.00% overall (33 correct, 7 partial, 4 wrong), 22 of 31 answerable, with a median latency of eight seconds on dense academic PDFs. That is a defensible first showing on material this far outside the app's domain. I would have filed it and moved on if I had not read the failures one by one.

#The two failures that were not failures

Two of the four wrongs were negatives – questions with no answer in the corpus, where declining is the correct behavior. The model had declined in both cases, quite plainly. It said, of one, that the topic "is not directly addressed in the provided sources," and of the other, that the figure "is not explicitly stated in the provided sources." Both were graded wrong, with the failure label "did not decline," which is the grader telling me it had not recognized a decline that any reader would recognize instantly.

The reason is unglamorous, and it is sitting in plain sight in the harness. Honesty is scored by a substring heuristic over a hand-maintained list of phrasings:

/// Does `answer` read as an honest "I couldn't find it in this collection"? Used to grade the negative
/// (out-of-corpus) questions and as one accept path for ambiguous ones.
static func looksLikeNotFound(_ answer: String) -> Bool {
    let a = answer.lowercased()
    let needles = [
        "couldn't find", "could not find", "cannot find", "can't find", "unable to find", "not found",
        "not in this collection", "no information", "not available in", "not contain", "no mention",
        "not mention", "doesn't appear", "not specify", "not specified", "not provide",
        "cannot be answered", "cannot be determined", "not possible to determine", "insufficient",
        "cannot be found", "could not be found",
        // …
    ]
    return needles.contains { a.contains($0) }
}

That list had grown by accretion, one phrasing at a time, each addition triggered by a decline somebody happened to notice being mis-scored. Which means the list was fitted to the phrasings the models produced on my corpora, over financial documents, in the register those prompts induce. Point the same engine at academic PDFs and it declines in a different dialect. The heuristic does not recognize the dialect, so it scores an honest answer as a lie.

Here is the part that made me stop and re-read the whole scorecard. Honesty on that benchmark was being reported as 2 out of 4 when the true value was 4 out of 4. The heuristic is corpus-independent code, so the same under-reporting was happening on every corpus I had ever measured. I had not been looking at a benchmark result. I had been looking at a defect in my own measuring instrument, found by the one corpus in the set that I had no hand in writing, which is exactly the job that corpus was built to do and rather sooner than I expected it to do it.

#Widening a needle list is a scoring change

The fix is five strings, and I want to be careful about how small that makes it sound. Changing what counts as a correct answer is a scoring change, and scoring changes are how benchmarks get quietly bent toward the system under test. The approved additions were conservative: "not directly addressed," "not addressed," "not discussed," "not explicitly stated," "not stated." Each one is unambiguous as a decline, and each was observed in a real answer rather than imagined at a whiteboard.

The more interesting decision was the one we rejected, and it is the reason I trust the change. "Not covered" looks like an obvious sixth member of that list and would have been a genuine error. In an insurance or tax corpus "not covered" is the vocabulary of substantive answers – "expenses not covered by insurance," "casualty losses are not covered there" – so adding it would have taught the grader to read real answers as declines and score them correct for the wrong reason. That exclusion is now a comment in the source and a test:

// "not addressed / discussed / stated" — the openrag baseline caught these honest declines
// grading WRONG ("did not decline"): openrag-neg-0035bbca answered "not directly addressed in
// the provided sources" and openrag-neg-006b56ae "not explicitly stated in the provided sources".
// NOTE: "not covered" is DELIBERATELY EXCLUDED — it false-positives substantive answers
// (e.g. "expenses not covered by insurance"), which are real answers, not declines.
"not directly addressed", "not addressed", "not discussed",
"not explicitly stated", "not stated",

@Test("looksLikeNotFound flags openrag decline phrasings, but not 'not covered' real answers")
func honestyHeuristicOpenRAGPhrasings() {
    // The two real observed openrag negatives that graded WRONG before this change.
    #expect(RAGEval.looksLikeNotFound("This question is not directly addressed in the provided sources."))
    #expect(RAGEval.looksLikeNotFound("The specific figure is not explicitly stated in the provided sources."))
    // … three more accepted phrasings
    // "not covered" is deliberately NOT a needle: these are substantive answers, not declines.
    #expect(!RAGEval.looksLikeNotFound("You may deduct medical expenses not covered by insurance [1]."))
    #expect(!RAGEval.looksLikeNotFound(
        "Form 1040 Schedule A covers itemized deductions; casualty losses are not covered there [2]."))
}

Before running anything, the audit went looking for collisions. That meant a programmatic scan of every must-contain, must-contain-any and must-not-contain needle across all three corpus files and all nine question configurations, checking whether any new needle or the word "covered" could shadow an existing assertion (zero hits, on either). And the grader turns out to be monotone under this particular change. looksLikeNotFound is consulted in only three places – the negative accept path, the ambiguous accept path, and the negated money guard – so widening it can never turn a correct answer into a wrong one, and can only remove money violations, never create them. That is not an accident of the code so much as a property worth checking before you touch a metric, because a scoring change that can move numbers in both directions is one you can never cleanly attribute.

#Re-baselining, and what the delta proved

Then came the boring, necessary part. Re-run the affected baselines so the numbers of record stay comparable: same stores, same knobs, same model, changing only the grader.

The delta is exactly the two predicted flips and nothing else at all. The two surviving wrongs are the same real retrieval misses as before, both of which cited nothing at all, so 77.42% answerable remains an honest floor on lexical retrieval over dense academic prose. The synthetic IRS re-run came back byte-for-byte unchanged at 91.30%: same two wrongs, same two declined negatives, no money question changing classification. That is a null result, and the expected one, since that corpus's answers happen to contain none of the new phrasings. The null was worth the compute, because it is the evidence that the widening did not leak into substantive answers.

Four and a half points of that openrag improvement came from nothing changing in the engine at all. That is the uncomfortable and useful fact in all of this. It is easy to hold a high opinion of your own rigor while your rigor is aimed entirely at the system and not at the ruler. I had a harness with A/B discipline, deterministic decoding, source-verified ground truth and a hard safety invariant, all of which was sound, and underneath it a substring list that had been fitted by accident to the register of my own documents. The corpora were audited and the retrieval was audited, repeatedly and by fresh eyes. The grader was the thing everybody trusted because it was the thing doing the trusting, and the only reason we found it is that we finally handed the instrument a set of documents it had never been allowed to shape.

#Addendum, a day later: there were two more, and the outside corpus did not find these

I want to correct the ending above, because it reads as though the external benchmark was the mechanism that finds grader defects, and a day after I wrote it I found two more that it could never have found. Both were in the zero-wrong-money bar, which is the strongest safety claim this project makes, and both were caught the dull way: by reading the model's actual answers on my own corpora instead of reading the scores they produced.

The first is that the check deciding whether a question is even a money question required cents. A question keyed to $44,322 with no decimal point was therefore not a money question at all, could never appear in the bar, and could never be counted as a violation. During an unrelated experiment the engine answered $450,658 to how much the taxpayers owed on a federal return whose line 37 reads $44,322, and the tax corpus reported ZERO-WRONG-MONEY: PASS. Not "PASS with a caveat" – PASS, because the question had never been in the denominator. Every whole-dollar key in the library shared that blind spot, which is coverage limits, liability limits, and anything a form rounds to the dollar.

The second is worse in the way that quiet things are worse. The answer key check is a substring test over the whole answer, and a model that reasons out loud will quote the right figure on its way to the wrong one. Asked the amount due on an ADT bill keyed $42.99, the engine answered: "the amount due is $26.99. This is indicated in Source 3, where the ending balance is listed as $42.99, but the amount due is specified as $26.99 … Therefore the amount due is $26.99." Graded CORRECT, on the strength of the number it had explicitly ruled out. The fix was to stop grading what the answer mentions and start grading what it asserts, which means splitting each sentence at its first source attribution and keeping the half the model spoke in its own voice. I measured four ways of drawing that line against all 307 currently-correct money answers on disk, and three of them flipped between 52 and 172 of them, which would have been a scoring change dressed as a bug fix. The one I shipped flips three, all of which deserved it, including a banking answer that had been passing because its $20.00 key was sitting inside a quoted -$10,020.00 while the answer itself declined to name any fee at all.

The uncomfortable part is not that the instrument had two more defects. It is that the essay above had already drawn the moral, and drawn it slightly wrong. The lesson I took from the openrag episode was that you find your blind spots by importing someone else's documents, and that is true but it is the smaller half. These two were found by an agent reading four hundred answers one at a time while measuring something else entirely, and the thing that made them visible was not foreign data, it was declining to accept a PASS without looking at what passed. An external corpus tests whether your rules generalise. It does not test whether your rules are pointed at the right thing, because it is scored by the same rules. Nothing catches that except reading the output.