Every corpus I had built up to that point came from my own documents, which is the right way to start and a terrible way to finish, because my documents are whatever happens to have landed in my filing system rather than a deliberate sample of the hard cases. Real material gives you realism and takes away control, and I had run out of the second one. I could not make my tax folder contain a corrected-and-reissued 1099-DIV, or a two-page multi-account 1099-INT, or a scan bad enough to force the OCR path but not so bad that the failure was uninteresting. I certainly could not verify by hand, at scale, that every answer key matched the printed page. What I wanted was a corpus whose ground truth was true by construction, where the values in the answer key are the same values that were drawn onto the page, because the same program did both.
So the synthetic IRS corpus starts from real IRS AcroForm templates downloaded from irs.gov (W-2, 1099-INT, 1099-DIV, 1099-B, 1099-NEC, 1099-MISC, 1098, W-2G), cached and hash-recorded so a regeneration is reproducible. The generator does not set AcroForm field values, which would leave the appearance streams up to whatever renderer opened the file later. Instead it draws the template's recipient-copy page as vector art, overlays seeded-random but internally consistent values at the form fields' widget rectangles, and flattens the result. The seed is a constant and the random number generator is SplitMix64, so it is stable across runs and platforms. The order in which values are drawn is fixed, with a comment telling anyone who follows not to reorder it. The layout and the label vocabulary stay authentically IRS, which matters because the classifiers and the deterministic extractors key on exactly that vocabulary.
That gets you a corpus with perfect ground truth and one serious problem. It took me an afternoon of confusion to see it.
Making the easy path fail on purpose
The extraction pipeline has tiers. The cheap tier reads the PDF's embedded text layer. The expensive one renders pages and runs Vision OCR over them, and it only fires when the flat text is not good enough. On a born-digital form with overlaid text, the flat tier succeeds, so the corpus was exercising the fast path exclusively and telling me nothing about the path my real documents actually take. Worse, it was succeeding badly. PDF text layers are ordered by the content stream, not by anything visual, so overlay values drawn after the template's own text ended up clustered roughly forty lines away from the labels they belonged to. The extractor was getting a text layer that was technically complete and semantically scrambled, which is arguably worse than getting nothing.
The fix is the piece of this corpus I am fondest of, and it is about thirty lines long. Money values are not drawn as text at all. They are drawn as filled glyph outlines, real vector paths, perfectly visible to a human and to Vision, and completely invisible to any text extractor:
/// Draw a string as GLYPH OUTLINES (filled vector paths, not text operators): invisible to
/// the PDF text layer, fully visible to Vision. Money values are drawn this way so the clean
/// forms are the real "unmapped overlay font / moneyless form" class …
static func drawOutline(_ s: String, x: CGFloat, y: CGFloat, size: CGFloat, in ctx: CGContext) {
let line = CTLineCreateWithAttributedString(NSAttributedString(string: s, attributes: attrs))
for run in (CTLineGetGlyphRuns(line) as! [CTRun]) {
// … for each glyph, translate to its position and append its path
if let path = CTFontCreatePathForGlyph(runFont, glyphs[i], &t) { ctx.addPath(path) }
}
ctx.fillPath()
}
nonisolated static let moneyLikeShape = #"^\d[\d,]*\.\d{2}$"#
static func drawValue(_ value: String, in rect: CGRect, ctx: CGContext, size: CGFloat = 8.5) {
if value.range(of: moneyLikeShape, options: .regularExpression) != nil {
drawOutline(value, x: rect.minX + 2, y: …, size: size, in: ctx)
return
}
drawTextValue(value, in: rect, ctx: ctx, size: size)
}The effect is that the labels stay in the text layer and every dollar figure leaves it, which is precisely the class my real overlay-font 1099s fall into. The pipeline's needsOCRTopUp check sees a form with no money-like token in its text and fires. Vision reads the page and appends a transcript in genuine reading order, so the values arrive next to the labels they belong to rather than forty lines downstream. Deterministic extraction coverage on the corpus went from 12% to 62% – not because the extractor got better, but because it was finally being handed text with the right geometry, through the same OCR top-up path the real store uses.
There is a second class the corpus needed, which is the scanned document, and that one is cruder: render the filled page to a bitmap at roughly 190 dpi, rotate it slightly, add seeded speckle, and wrap it back into an image-only PDF.
try degrade(pdf: bluebirdClean, to: outDir.appendingPathComponent(w2BluebirdFile),
dpi: 190, rotationDegrees: 0.7, rng: &rng)The speckle is deliberately light, "never dense enough to defeat OCR by design," because a corpus that is impossible does not measure anything. The point of the degraded variants is not to break the OCR, it is to put the OCR under enough strain that its characteristic errors show up in the results. They did – one of the probe failures is a degraded scan where Vision read the form's own labels as "tipe" and "Foderal."
Because all of this is generated, the corpus can verify itself before it is ever used, which is the other quiet benefit of manufacturing your own ground truth. The generator ends by asserting its own invariants: degraded documents must have no usable text layer, clean IRS forms must carry labels but no money-like token, and the drawn-from-scratch lookalikes must have their printed values present in the text. If any of those break, the run throws rather than quietly producing a corpus that measures the wrong thing.
The answer keys are apparatus, not answers
The first eval round against this corpus produced failures that were not failures. Sorting that out taught me more than the corpus itself did.
Vision, reading these forms, sometimes renders a comma as a period, so a figure printed as 17,338.88 comes back as "17.338.88." My first instinct was that this was an OCR defect to be counted against the score (it is not). A model that reads its source faithfully and quotes what the page appears to say is doing the correct thing, and grading that as a wrong money answer would have taught the system to distrust its own evidence. So the key accepts all three shapes:
/// The accepted answer-key shapes for a money value: comma-grouped, bare, and the
/// period-thousands OCR shape ("17.338.88") — Vision reads the comma as a period on these
/// forms and a model that faithfully quotes its source is stating the CORRECT money.
static func variants(_ v: Decimal) -> [String] {
let grouped = fmt(v)
let bare = grouped.replacingOccurrences(of: ",", with: "")
guard grouped != bare else { return [grouped] }
return [grouped, bare, grouped.replacingOccurrences(of: ",", with: ".")]
}The second case is subtler and I nearly got it wrong in the other direction. The corpus contains a 1099-DIV pair, an original and a CORRECTED reissue with the corrected banner stamped on it, and the question asks what the corrected form reports. The deterministic lookup answers with the corrected figure and then discloses the disagreement, telling you the other copy reads a different number and that it is worth a manual check. My instinct was to ban the superseded figure from the answer with a mustNotContain, since it is, after all, the wrong number. That instinct was backwards, and the note I left in the corpus source says so plainly:
// NB: no mustNotContain on the superseded original — the deterministic lookup DISCLOSES
// the revision disagreement by design ("the other copy reads $924.90 — worth a manual
// check"); banning that figure would punish exactly the transparency we want.
q("irs-div-corrected", .extraction,
"What ordinary dividends did Redwood Capital Partners report on the corrected 1099-DIV?",
"$\(fmt(d1c)) — the CORRECTED reissue supersedes the original $\(fmt(d1)); the disagreement is disclosed.",
docs: [divRedwoodCorrFile], any: variants(d1c), cite: ["Redwood Capital Partners_001"])Both of those are edits to the measuring instrument rather than to the thing being measured, and the distinction is the one I care most about here. Loosening a key so the engine's actual behavior scores better is tuning, and it is how benchmarks rot. Correcting a key that was encoding a wrong expectation – that a faithfully quoted OCR artifact is an error, that disclosed uncertainty is a defect – is repairing the instrument. Refusing to do that does not make you rigorous, it makes your numbers wrong in a direction that flatters nobody. I wrote the rule down in the memory note at the time, because I knew I would want to argue with myself about it later: corpus keys are measurement apparatus, and fixing them is not tuning.
The corpus that came out of all this is seventeen documents and twenty-three questions, which is small, and it has been worth more per question than anything else I have built. It scores 91.30% correct, and it carries eighteen money questions with no wrong dollar figures against any of them. It also holds a defect I have not yet fixed and cannot pretend away. A question about student loan interest on Form 1098-E routes to the plain 1098 lookup, because the classifier's word-boundary pattern treats the "-E" as a boundary, so the deterministic path confidently quotes mortgage interest at a student-loan question. That is exactly the kind of thing a corpus built out of my own convenient documents would never have caught, because I do not happen to own a 1098-E. The whole reason to manufacture ground truth is to own the documents you need rather than the ones you have.