I have been building a document vault that answers questions about your own files, entirely on your machine, and the question that shaped the architecture more than any other came out of a fairly ordinary conversation I had with myself in July. I had just watched the system answer a stack of tax questions perfectly, and I was pleased with it for about an hour. Then it occurred to me to ask whether the thing I was pleased with was actually an engine or just a very well-dressed special case. The exact way I put it to my agent that day was this: does tax have a different test mechanism, because we are supposed to be building the best single generic engine, and a user is never going to walk up and tell us they have a tax corpus. That question is the whole post, and the answer we landed on – one pipeline, two layers, no domain branch anywhere – has held up through every corpus we have thrown at it since. The shape of it is more interesting than the scoreboard.
The constraint underneath all of this is that everything runs on-device. The generation tier is a Qwen3-4B model at 4-bit running through MLX on Apple Silicon, with no cloud call, no API key, and no document ever leaving the machine. That is not a marketing posture, it is a design constraint with teeth. A 4B model is a genuinely limited reader, and you feel its limits the moment you ask it to do arithmetic across five forms. Everything that follows is downstream of that constraint.
Layer A is the engine, and it knows nothing
Layer A is the boring, general thing: retrieve chunks (lexical tf-idf, with an optional semantic weight that ships at zero), diversify and select them, ground and sanitize the text, and let the model generate a cited answer. It knows nothing about tax, or law, or invoices, or the file you imported ten minutes ago. It runs on any document you give it, which is exactly the property you want. The population of document types a real person keeps in a vault is unbounded, and I am never going to enumerate it.
The important consequence of that is where investment pays. Improving Layer A – retrieval recall, scoring, chunk selection – helps every corpus at once, without anyone writing a bespoke extractor, and so it is the "best single generic engine" play by definition. When we tried BM25 scoring behind a flag, legal and presentation recall jumped (answer chunks surfacing at k=8 went 2 to 8 on legal and 7 to 9 on presentations) and nothing about the tax path even noticed. That turned out to be a very clean piece of evidence for the two-layer story, for reasons I will get to.
Layer B is a precision bonus, and it fails closed
Layer B is the deterministic path, and when you ask a money question – a total, a comparison, a specific box on a specific form – RAGService does not hand the problem to the model at all. It enumerates the matching documents by type, extracts the field as an exact Decimal, sums or compares in code, and returns the number with citations, with no model math and no hallucinated money anywhere in the path. The reason we built it is written in the source as a measurement rather than an opinion. A 4B model asked to read N forms and total them reliably grabs one form's number, or the 1040's pre-summed total, instead. We measured that at 0 out of 6 before this code existed.
// Gated on detected INTENT; any shortfall (no typed members, nothing extracted) falls
// through to the normal retrieval path below, so non-aggregation behavior is byte-identical.
if docAggregation {
if let intent = RAGAggregation.detectIntent(in: query),
let aggregated = try await aggregatedAnswer(intent: intent, collectionID: collectionID) {
return aggregated
}
if let field = RAGAggregation.detectLookupField(in: query),
let looked = try await lookupAnswer(field: field, query: query, collectionID: collectionID) {
return looked
}
}The load-bearing detail, though, is not that Layer B exists, it is what triggers it. Layer B is not gated on "this is a tax corpus," because no such flag exists anywhere in the system and no user would ever set it. It fires on two generic signals, both derived without a single declaration from the person using the app: the phrasing of the query (a compute cue plus a field cue, or a field cue alone for a single lookup), and the type of the document, which the classifier reads off the document's own structure and title. Two users with the same documents asking the same question get the identical code path. There is no test shortcut and no eval-only branch – the harness runs the real thing.
The honest limit
Layer B only knows the form vocabularies we have actually written down: IRS 1099 variants, W-2, 1098 boxes, California property-tax coupons. That is a short list against the world. A medical bill, a European invoice, a bank statement – none of them type into a known form, so candidateTypes comes back empty, the lookup declines, and the question falls through to Layer A. It is never wrong on a foreign document, it just does not get the deterministic precision. Failing closed was the design requirement, and it is expressed in something as small as a regex boundary:
(#"\b1099[-\s]?composite(?![a-z0-9])"#, "1099-Composite"),
(#"\bw[-\s]?2[-\s]?g(?![a-z0-9])"#, "W-2G"),
(#"\bw[-\s]?2(?![a-z0-9])"#, "W-2"),
(#"\b1099[-\s]?int(?![a-z0-9])"#, "1099-INT"),
(#"\b1098(?![a-z0-9])"#, "1098"),That trailing negative lookahead is used instead of \b because Swift's Unicode word boundary treats "INT.pdf" as word-internal, so a file named 1099-INT.pdf would not have fired at all. It also stops W-2 from swallowing W-2G. It is a small thing that took a real debugging session to understand, and it is the sort of detail that separates a classifier that fails closed from one that fails confidently.
And it does not always fail closed cleanly, which I should say out loud. A held-out synthetic corpus we built later (a post of its own) caught a genuine defect our real tax documents had never exposed: a question about student loan interest on Form 1098-E routes into the plain 1098 lookup, because that same \b1098(?![a-z0-9]) treats the "-E" as a boundary, so the deterministic path confidently quotes mortgage interest at a student-loan question. That is exactly the failure mode Layer B is supposed to be immune to, it is on the board unfixed as of this writing, and I would rather publish the pipeline with that in it than write a post implying the design guarantees what it merely encourages.
Why the two-layer story is measurable rather than architectural
The reason I trust this framing is not that it is tidy on a whiteboard. It is that the two layers separate cleanly in the numbers. Our banking corpus – real bank statements, which do not type as any known form – runs pure Layer A, and at the shipped defaults it scores 16 out of 18 (12 out of 18 with the date-disambiguation boost switched off). The misses that remain are model misreads of column-flattened OCR. Our tax corpus runs A plus B and scores 17 out of 17, and it is the only one of the two that has never stated a wrong dollar figure. The gap between those numbers is, roughly, Layer B's contribution, and the banking column is the true face of the generic engine on financial-looking documents we have written no extractors for.
The BM25 experiment confirmed the same split from the other direction. Under BM25 the tax corpus stayed at 17/17 even though the retrieval reshuffle introduced three new tax retrieval "misses" – because those money answers ride Layer B, which bypasses chunk ranking entirely – while legal and presentation recall improved. A retrieval change that moves one layer's numbers and provably cannot move the other's is about as good a piece of evidence for a layering claim as you are going to get without a proof. (BM25 itself we shelved, for reasons that belong to a later post: the retrieval win netted to zero at generation, so the branch never merged.)
What "let the user get specific" actually means
The natural follow-up question is whether we should let people tell us more about their documents, and I think the honest answer has three rungs rather than one. The first rung already works with zero user input, because structure-driven typing generalizes to any titled form – an "Acme Invoice," a lab result – and feeds retrieval boosts without anyone hard-coding a family. The second rung is light-touch: let someone confirm or correct a document's type, or scope a chat to one collection. The third rung is the real feature, and it is the only one that genuinely extends Layer B: user- defined extraction fields. You tell the app that in your invoices the total is labeled "Balance Due," and the deterministic path picks up your forms without us shipping code for every form family in existence. That is a feature with a roadmap slot rather than a toggle, and I have filed it as a post-launch idea rather than pretending it is close.
What I keep coming back to is that the discipline here was refusing the special case at the exact moment the special case would have been cheapest. It would have been perfectly easy, that day in July, to notice that tax answers were excellent and to quietly build the rest of the product around a tax-shaped assumption. The eval scores would have looked wonderful for a while, right up until the first person imported a folder of medical bills. Separating the generic engine from the deterministic bonus cost more up front and it constrains what we are allowed to claim, but it means that when I look at a scorecard now I know which layer earned the number, and that is the thing that makes the next decision possible.