Self-hosted inference

What a model costs, what limits it, and what it reveals when you run it yourself on hardware you bought.

This series started in February 2026. The argument fits in one line: most of the decisions you make about an AI system are taken before the model, and verified after it.

Three things keep coming back. The real ceiling is almost never the one you expect. A measurement with no stated scope spreads faster than it can be corrected. And exhaustiveness is not something similarity search can deliver, however good it gets.

No demos here. Readings, with the conditions under which they were taken, and the ones I have not managed to take.

Notes in this series

• August 25 Six months running a model on my own server

  Three lessons nobody sold me: the hidden cost is not the GPU, your quality metric lies until proven otherwise, and trust is earned through what the model is forbidden to do.

• August 27 A model that answers badly does not crash

  Errors at zero, latency at 98 ms, no alerts. And the answers are wrong. Conventional monitoring says everything is fine.

• September 1 Six values of the same metric

  Recall moved through six values in a week without the model changing. Three causes. This reading has since been corrected.

• September 2 Why these notes start midway

  The work began in February. Publishing begins today. What this site holds, what it does not, and why the early notes carry two dates.

• September 3 A third of my failures were not failures

  I published six values of one metric and wrote that the measurement was wrong. Looking for why, I found a third of the failures came from the instrument — and two unrelated measurements landing on the same number.

• September 8 It is almost never the model

  One hundred and nine missed statements, five mechanisms, and the model is behind none of them. The vocabulary gap everyone assumes is dominant turns out to be the smallest category.