There is a version of this project where I spent the summer tuning retrieval parameters by feel, asking the app a handful of questions I already knew the answers to, and declaring victory whenever a change made those particular questions look better. I know that version exists because I have watched teams live in it for years, and because I started there myself, and the honest reason I climbed out was not discipline so much as embarrassment. I could not tell whether the last change had helped. That is a miserable position to be in when you are the only person who can decide what ships.
Later, when I ran a survey of the on-device Swift RAG ecosystem to see what everyone else was doing, the finding that stuck with me had nothing to do with retrieval technique. It was that most of the surveyed projects have no evaluation at all – no question set, no grader, no scorecard, nothing that would tell the author whether last week's pipeline beat this week's. Our chunking (roughly 1000 characters with 150 of overlap), our grounded and cited and sanitized prompt, our auto-graded harness: all of those came back as best practice, and the harness was the one that almost nobody had. That gap is, I think, the actual moat, and it is available to anyone willing to do the unglamorous part first.
Separating the app from the instrument
The harness did not start clean. For a while the eval lived inside the app, which produced a series of small disasters that were each individually explicable and collectively damning. The in-app eval pkill'd the running dogfood app out from under me. A crashed A/B preference restore materialized stale explicit UserDefaults values that silently disabled shipped features in the live app, which took an embarrassing while to diagnose because everything looked configured correctly. A re-extract had to run against the live store for the eval to see it at all, which is a sentence that should worry anyone who reads it carefully.
So in early July we split them into two products in one repository: the app sits in a stable place, takes only changes the harness has already proven, and got tagged eval-baseline-2026-07; the harness is where experimentation and new corpora live. The separation had to be twofold – binary and state – because a harness executable that still opens the live store and writes the live preferences domain has fixed exactly half the problem. ShabuBoxCore became a local Swift package holding models, repositories, persistence and services, and shabubox-eval became an executable that owns the eval family, the probes, and the A/B levers, with its own preferences domain that never touches the app's.
The gate ordering was the part I cared most about, and I want to be specific because it is the thing I would repeat. The harness had to reproduce the tax corpus at 17/17 and the aggregation probe at 34 PASS, on a snapshot store, off-app, before a single line of eval code was allowed to leave the app. Never strip before the measurement bridge exists. It cost an extra phase, and it meant that when we did strip 1,744 lines out of the app the arithmetic closed exactly – 737 app tests plus 81 package tests plus one deleted. I had a number on both sides of the move.
Snapshotting a live store without breaking it
The mechanism that makes off-app measurement honest is the snapshot, and it is one of my favorite small pieces of the system precisely because it is boring and correct. The harness reads the live database through SQLite's online backup API, opened read-only, which takes a torn-read-safe copy even while the app holds the database open in WAL mode:
guard sqlite3_open_v2(source.path, &src, SQLITE_OPEN_READONLY, nil) == SQLITE_OK else {
throw HarnessError("cannot open source store readonly: \(message(src)) (\(source.path))")
}
guard let handle = sqlite3_backup_init(dst, "main", src, "main") else {
throw HarnessError("backup init failed: \(message(dst))")
}
let deadline = Date().addingTimeInterval(60)
var rc = sqlite3_backup_step(handle, -1)
while rc == SQLITE_BUSY || rc == SQLITE_LOCKED {
guard Date() < deadline else {
sqlite3_backup_finish(handle)
throw HarnessError("backup gave up after 60s of SQLITE_BUSY/LOCKED — …")
}
usleep(250_000)
rc = sqlite3_backup_step(handle, -1)
}The read-only open means this can never mutate live app state. The sixty-second deadline means a pathologically busy source produces a clear error rather than an infinite retry, and the command writes a marker file the store resolver checks before it will run anything against that copy. I like this piece because it is the seam where a measurement stops being a favor the app does for me and starts being an independent read of state I can repeat tomorrow. An audit round caught the torn-snapshot trap in that code before it caught me, which is one of several places in this project where a fresh set of eyes paid for itself immediately.
A corpus is a folder plus one JSON file
The other half is being able to measure on documents that are not mine. A corpus, in this system, is a directory of documents plus a corpus.json manifest. The schema is documented in exactly one place – the CorpusManifest.swift file that parses it – because a schema documented twice is a schema that is wrong in one of the two places. The questions in the manifest reuse the RAGEval.Question type verbatim, so there is no parallel format to drift:
nonisolated struct CorpusManifest: Codable, Sendable {
var name: String
var description: String
var questions: [RAGEval.Question]
var probeGroundTruth: [ProbeGroundTruthRow]? // per-(doc, field) expected values
var documents: [DocumentEntry]? // per-file notes + generation ground truth
var docTypeGroundTruth: [DocTypeGroundTruthRow]?
var generator: [String: String]? // seed, template hashes, tool route
}The ingest command takes such a directory and builds a fresh store through the real pipeline: the same import, text extraction and OCR, structure harvest and chunking path the app runs. It is sandboxed in the ingest directory, nowhere near my own library. The steps it deliberately skips get printed and recorded in the marker, so a scorecard never quietly rests on a store that was built differently from the one it claims to represent.
There is a detail in that manifest I want to draw out, because it took a round of argument to settle. In docTypeGroundTruth, an absent expected type does not mean "don't care," it means the document is expected to classify as untyped – a jurisdiction lookalike whose anchors must fail closed is a real, scoreable expectation rather than a shrug. Encoding "this should produce nothing" as a first-class assertion is the difference between a harness that measures a system's caution and one that only measures its enthusiasm.
The grader, and what it will not do for you
The grading is deliberately mechanical. A question carries assertions – mustContainAll, mustContainAny, mustCiteAny, mustNotContain – and the grader checks them case-insensitively as substrings, with all passing giving correct, some giving partial, and none giving wrong. Negative questions invert the whole thing: the right answer to a question about a document that does not exist is a decline, and the grader is looking for exactly that shape of response rather than a fact.
if q.negative {
let declined = looksLikeNotFound(answer)
if declined && banned.isEmpty { return GradeResult(grade: .correct, failed: []) }
var failed: [String] = []
if !declined { failed.append("did not decline") }
if !banned.isEmpty { failed.append("contains \(banned.joined(separator: ", "))") }
return GradeResult(grade: .wrong, failed: failed)
}
guard !checks.isEmpty else { return GradeResult(grade: .partial, failed: ["no assertions to grade"]) }That last guard is the humility clause: a question with no assertions cannot be auto-graded, so it returns partial and flags itself rather than silently counting as a pass. An audit later found a related gap where a question authored with only a mustNotContain fell through that guard and could never grade correct even when the model behaved perfectly. That is the sort of bug that quietly suppresses your own score, and you will never find it by staring at a percentage.
Grader weaknesses are real and I have hit several. A mustContainAny needle of "wages" was too loose on a tax corpus and marked a declining answer correct. An early synthetic run graded genuinely correct money as failure because the answer key did not accept OCR's period-thousands shape, and separately banned a corrected-supersede disclosure that the system was right to make. The rule we settled on is that corpus keys are measurement apparatus, so fixing a key is not tuning, but I hold that line nervously, because it is exactly the rationalization a person would use to cheat their own benchmark.
The finding that reframed everything
Then, on the fourth of July, the harness told me that the harness had been lying. The eval was building its MLX engine at the shipped temperature of 0.3, which is fine for a product and fatal for a benchmark, because a single scorecard varied by one or two questions run to run from sampling alone. Every single-run A/B I had done that session was sitting on that noise floor. The fix was one line in spirit – force temperature 0 for the eval, keep 0.3 in the shipped app – and it was proven by running the same legal configuration twice and getting per-question identical results. From that point a scorecard delta meant the configuration and not the dice.
I do not think I would have found that by being smarter. I found it because a measurement apparatus that you use constantly eventually gets pointed at itself, and the multi-corpus work forced that. The held-out synthetic corpus we generated from official form templates immediately caught a real generalization bug that my own real documents had never once exposed. The point of building the instrument before the tuning was never that the instrument would be right. It was that the instrument would be the thing capable of telling me it was wrong, and everything worth knowing that came afterward – including the results in the next post, where several ideas I believed in lost – arrived through it.