A third of my failures were not failures

Self-hosted inference — published 3 September 2026

Two days ago I wrote that recall on this system had moved through six values in a week without the model changing, and that the measurement had been wrong every time.

That was true, and short on three counts.

The figure I quoted was not measured on the corpus I attributed it to. A third of the recorded failures were not failures of the system. And part of the movement I blamed on measurement was a real improvement.

This note supersedes that reading. I am not deleting what I wrote: I am saying what was wrong and why.

The system, briefly

A deterministic extraction pipeline — normalization, segmentation, a sweep against a closed list of verb forms — followed by classification through a local model at zero temperature, with grammar-constrained output. The model returns identifiers, never prose.

The task is not to answer “what does this corpus say about X.” It is to surface every binding statement without missing any. Exhaustiveness is not something similarity search can deliver: no top-k guarantees it.

First correction: wrong denominator

The figure I had published was 96 out of 145. It was measured on one reference package, and I placed it next to a ground-truth corpus holding 421. Two separate measurements. The number was right; the attribution was not.

The corpus had also changed shape in the meantime. It held 492 lines in the version I quoted; it holds 536 today, eleven lines having been split so that each carries a full verbatim. The count of binding statements did not move — 421, no new requirements — but quoting 492 meant quoting a superseded state.

So I published a note about measurements with unstated scope, quoting a measurement with unstated scope.

What the actual corpus says

Measured 2 September 2026, across the 421 statements marked binding. Denominator stated.

Recall by inclusion: 312 of 421, or 74.1%.

On this same corpus, the 25 August measurement gave 59.1% by inclusion and 43.7% on exact match. The fifteen points gained since are not an artifact: they come from three real fixes — a wider verb vocabulary, removal of a section restriction, and a fix for non-breaking spaces.

That point matters, and I flattened it the first time. Part of the movement really was the system getting better.

Where the ceiling comes from

The 109 statements not found were compared mechanically, never by a model, against the full set of candidates the sweep actually produced. Each category below is derived from a verified mechanism rather than assumed in advance.

Truncated sentence boundary — 59 cases, 54.1%. The candidate stops at a colon while the statement continues into a list that is grammatically the object of the same verb. Or the reverse: a long verbless clause precedes the real verb and the candidate starts too late. In every one of these, the right verb was detected, in the right sentence. This is delimitation, not comprehension.

Non-unitary reference — 15 cases, 13.8%. The ground-truth line bundles several distinct occurrences: a compressed list of alternatives, separate quotations fused with ellipses. No sweep can match a candidate to a line that is not the exact copy of any single passage.

Glued header rejected — 8 cases, 7.3%. The sweep produced the expected sentence, word for word, with a section header stuck in front. The comparator tolerates that addition, but not beyond eight words and not if it contains sentence-ending punctuation. A section reference contains one.

Two clauses bundled — 13 cases, 11.9%. The line covers a triggering clause and then an action clause. The sweep produces one candidate per clause when each carries its own verb, but no candidate spans the whole line, and coverage lands just under a majority threshold set elsewhere.

No verb from the vocabulary — 14 cases, 12.8%. Passive construction, a verb outside the swept list, an imperative without a subject.

A third comes from the instrument

Three of those five categories do not describe a failure of the system.

Non-unitary reference is an artifact of the corpus doing the judging. Glued header is a narrow comparator defect, and the statement had in fact been found. Two bundled clauses fall under a threshold — and a threshold is a design decision about the measurement: move it, recall moves, the system has not changed.

That is 36 cases out of 109. A third.

The remaining 73 are genuine misses. So the published figure understates the system, and it understates it for reasons that belong to the instrument.

The category that is not a defect

The fourteen statements with no verb from the vocabulary deserve separating from the rest.

The sweep is a closed list of roughly twenty verb forms. A sentence using none of them is invisible to it by construction, not by defect. This is not a bug: it is the price of a choice, that of an enumerable and replayable mechanism over a statistical judgment.

Telling a design limit apart from a defect to be fixed is not an engineering skill. It is a decision-making one, and it settles where the budget goes.

Fixing reveals

The analysis includes an estimate: fixing the boundary detector would take recall by inclusion from 74.1% to roughly 88.1%, some fourteen points.

It is presented as an upper bound, not a floor. And the reason given is the most interesting thing in the report: the “glued header” and “two bundled clauses” categories exist precisely because an earlier fix exposed a second, finer defect.

The analysis of measurement errors produced measurement errors of its own, and knows it.

Two measurements, one number

While building a reconciliation table of every recall value in circulation, something surfaced that no method was looking for.

There is a second measurement equal to 74.1%.

It has nothing to do with the first. Pooled scope across three packages instead of one. Tolerance threshold at 0.25 instead of 0.5. Different date. Two unrelated measurements landing on the same number.

Pure numerical coincidence. And the only kind of collision a scope table catches, because a reader who meets “74.1%” twice across two documents will naturally conclude it is the same thing.

It is the shortest argument I know for stating scope: a number can collide with itself.

There is no single current state

The table produced a second result, more important than the first.

Six recall values circulate on this system. All six trace back to a source. None is wrong. And none is the recall of the system, because recall is measured per reference package, not globally.

Two are stale rather than competing: nothing in their definition changed, only the code being measured. One belongs to another package, not replayed since, so neither stale nor current with any certainty. And two are not candidates for current state at all: they use a relaxed tolerance threshold that production never adopted, produced by diagnostic tools deleted after use.

That last group is the one demanding most discipline, because its values are more flattering than the production measurement. A prettier number, unlabeled, sitting in an internal document, eventually turns up in a deck without its context.

So the question “what is this system’s recall” is malformed. That is an unwelcome finding and it is solid.

The lesson, revised

I had written: when a metric moves, the first instinct is to look at the system, and that is almost always the wrong place.

Too simple. Sometimes it is the measurement. Sometimes it is the system. Here it was both, in the same week, on the same figure.

Knowing which of the two moved is the real skill. No dashboard supplies it, and no isolated value lets you deduce it.

What I cannot conclude

The corpus covers a single reference package. It shows that the comparator measures the right thing and that the defects found are real. It shows nothing about other document types.

Two measurements remain absent from the register. Real unit cost, for which no component exists: no power draw, no applied rate, no amortized hardware. And latency percentiles along the application path — the logging table has existed since 30 August, but no real execution has ever fed it, everything running through the test harness.

A note from the Self-hosted inference series. The measurements cited appear in the register, with their dates. This note corrects a reading published on 1 September 2026.