The failure that raises no error

Data — published 10 September 2026

The case

An extraction pipeline receives a document. The input contract says: text. A native document carries a clean text layer, produced by the tool that created it.

A scanned document does not. Its text layer is reconstructed by optical character recognition. It exists, it is syntactically valid, it reads. It can also be partial: a flattened table, a lost column, a handwritten signature ignored, a stamp gone illegible.

The pipeline sees no difference. It received text. It produces an output.

That output is plausible. It is structured, it has the right format, it passes schema validation. It rests on a partial reading of the document, and nothing in the system says so.

No exception is raised. That is what makes the defect expensive: it does not ask for attention.

Why measurement does not catch it

On a reference corpus of 492 hand-labeled items, I measure an overall recall. That figure aggregates two populations: documents with a native text layer, and reconstructed documents.

If the second population fails systematically and accounts for a minority of the corpus, overall recall stays good. The defect is diluted by the average. It goes unseen because it is not looked for separately.

The exact share of reconstructed documents in this corpus: not measured to date. That is the first thing to instrument, before any improvement to the system.

A figure aggregated over two populations that do not behave the same way measures neither of them.

What this changes in the input contract

Text alone is not sufficient data. What must travel is the text together with its origin and a confidence indicator, per zone of the document rather than for the document as a whole — a page can be clean and a table illegible.

Three consequences:

1. Input type becomes a dimension of measurement. Recall per type, not overall recall.
2. The threshold becomes an explicit decision. Below a given confidence level, the system refuses and escalates the document to a human. It does not infer.
3. Refusing must be a legitimate output state, on the same footing as succeeding. A pipeline that only has succeeded and crashed has nowhere to say that it read partially.

The part that gets forgotten

When a scanned document is too degraded, the temptation is mechanical: send it to a more capable external service. That is precisely the moment when data we had decided to keep local leaves the infrastructure, and it happens without a decision.

The confidence indicator is therefore not only a quality metric. It is where the inference policy is wired in: what is processed locally, what is refused. Not what is escalated outside.

When the constraint is that data does not leave the infrastructure, this wiring is not a configuration option. It is an architectural decision, taken before the first call is written.

Observations from a personal project in progress, outside my professional activity. Corpus of 492 manually labeled items. Any correction will be flagged at the foot of the note.