Notes from building the engine
These are working notes from building ShabuBox’s on-device document engine — an account of what was measured, what shipped, and what was measured and then thrown away. They are written in order, and the later posts assume the apparatus described in the earlier ones.
Nothing here is a product claim. The numbers come from our own eval harness run over our own corpora, and the methodology is described alongside them so you can judge what they are worth.
-
01
One engine, two layers
Whether the system was a real engine or a very well-dressed special case — and the layer split that turned that question into something measurable.
-
02
The eval harness came first
Building the instrument before tuning anything, so victory could never be declared by feel on a handful of questions whose answers were already known.
-
03
We measured it, and the fancy thing lost
Three days in July in which dense embeddings, an LLM reranker, and a wider context window each failed to beat the simple retrieval already shipping.
-
04
The ceiling that wasn't
A score of 42 out of 50 that would not move no matter which knob was turned, and the point at which the console became more suspect than the engine.
-
05
Manufacturing ground truth
Real documents give realism and take away control; a corpus whose answer key is true by construction gives both back.
-
06
Zero wrong money
The app is allowed to be unhelpful and is not allowed to be confidently wrong about a dollar figure — written as a check that can fail a build.
-
07
Grading the grader
Every corpus came from the same author with the same assumptions. What an outside benchmark found when it was pointed at the instrument itself.