Six values of the same metric
Self-hosted inference — written 1 September 2026, republished here 2 September 2026
Note on republication and correction. Written and published elsewhere on 1 September 2026, before this site existed. It appears as written, and it contains two errors I found the next day: the figure I cite was not measured on the corpus I attributed it to, and that corpus held 536 lines rather than 492. Its conclusion is also too simple. The full correction is here: A third of my failures were not failures. This text stays online because a correction is flagged, not erased.
Six values of the same metric, inside one week. The model never changed.
No retraining. No new prompt. Not a single setting touched.
Every time, the measurement was what was wrong.
Three causes, and none of them is peculiar to my case.
A test that encodes its own assumption tests nothing
My first measurement compared the system’s output against what I believed the right answer to be. Not a ground truth: an intuition, hard-coded into the test.
It confirmed my assumption, not the system.
Without ground truth there is no recall
492 statements adjudicated by hand, one at a time. The most thankless work of these six months.
And the only work that made every figure after it interpretable.
A number without a denominator travels
The same “60 percent” says different things depending on what sits underneath. The value circulated through four documents before I noticed it did not refer to the same population in each.
What I take from it
When a metric moves, the first instinct is to go look at the system. That is almost always the wrong place.
An organization that cannot measure whether it is improving cannot decide whether a tool is worth its cost. True for a personal lab, true for a seven-figure budget.
My register carries two blank lines: real unit cost, and latency percentiles along the application path. I have not measured them. I would rather write that down than estimate.