It is almost never the model

Self-hosted inference — published 8 September 2026

When a system built around a language model misses something, the reflex is to blame the model. Bad prompt, context too short, not enough capability.

Across the 109 statements this system fails to surface, the model is behind none of them.

And the category everyone assumes is dominant — missing vocabulary — turns out to be the smallest of the five.

Where the 109 come from

The previous note set the scene: of 421 binding statements, 312 are surfaced and 109 are not. A third of those misses came from the measuring instrument rather than the system.

This note takes up the remaining two thirds, and above all one decision: where to spend the fixing effort.

The 109 were compared mechanically, never by a model, against every candidate the sweep actually produced. Five mechanisms come out of it.

The five mechanisms

Boundary trouble dwarfs everything else. The candidate stops at a colon while the statement runs on into a list that is grammatically the object of the same verb. Or the reverse: a long verbless clause comes first and the candidate starts too late.

In all fifty-nine, the right verb was detected, in the right sentence. What fails is knowing where the sentence begins and ends.

Non-unitary reference is an artifact of the judging corpus: the ground-truth line bundles several distinct occurrences, a compressed list of alternatives or separate quotations fused with ellipses. No sweep can match a candidate to a line that is not the exact copy of any single passage.

Missing vocabulary covers passive constructions, verbs outside the swept list, imperatives with no subject.

Two bundled clauses describes a line holding a trigger and then an action. The sweep does produce one candidate per clause, but none spans the whole line, and coverage falls under a majority threshold set elsewhere.

Glued header is a narrow comparator defect: the sentence was found word for word, with a section heading stuck in front, and the comparator refuses beyond eight words or if the addition carries sentence-ending punctuation. A section reference carries one.

The model appears nowhere

Read the list again. Sentence segmentation, the shape of reference lines, a verb list, a coverage threshold, comparator tolerance.

Five engineering decisions taken upstream, and not one of them turns on what the model does or does not understand. It never saw these hundred and nine statements: they were never submitted to it, because the stage before it did not produce them as candidates.

That is a consequence of the architecture, not an accident. A model that only receives candidates cannot recover what was never a candidate.

And it generalizes: in any system where a model sits behind a deterministic chain, some share of the failures people attribute to it is structurally out of its reach.

One category is not a defect

The fourteen missing-vocabulary cases deserve pulling out of the set.

The sweep is a closed list of roughly twenty verb forms. A sentence using none of them is invisible to it by construction, not by defect. This is not a bug to fix: it is the price of a choice, an enumerable and replayable mechanism over a statistical judgment.

The list can be lengthened. It cannot be made exhaustive, and claiming otherwise would be the system’s first false promise.

What this settles

Here the analysis stops being technical.

Fifty-nine cases come down to a single, localized fix in the sentence-boundary detector. Twenty-three — non-unitary reference and glued header — get fixed in the measuring instrument, not the product. Thirteen turn on a threshold, so on a judgment call. Fourteen are an accepted limit.

Put differently: one piece of work covers more than half of what is left to gain, two pieces concern only the measurement, and a quarter of the cases are not defects at all.

Without this taxonomy, the conversation would have been about the model. It would have cost time, it would probably have led to trying a bigger one, and it would have produced nothing.

Telling a design limit apart from a defect to be fixed is not an engineering skill. It is what decides where a budget goes.

What I cannot conclude

Two categories hold fewer than twenty cases. Proportions there are fragile and I would not present them as stable.

The analysis covers a single reference package. The split across mechanisms reflects the drafting style of those documents as much as the behavior of the system.

One estimate appears in the report: fixing the boundary detector would take recall by inclusion from 74.1% to roughly 88.1%. It is presented as an upper bound, not a floor — and the reason given is that two of the five categories exist precisely because an earlier fix exposed a second, finer one.

So the thing to do afterwards is measure again, not conclude beforehand.

A note from the Self-hosted inference series. The measurements cited appear in the register, with their dates.