Six months running a model on my own server
Self-hosted inference — written 25 August 2026, republished here 2 September 2026
Note on republication. Written and published elsewhere on 25 August 2026, before this site existed. It appears as written. The figures it cites have since been revised — see the register for their current state.
Six months running a language model on a server I own. No demos, no talking points. Measurements, mistakes, and disk alerts at two in the morning.
Three lessons nobody sold me.
The hidden cost is not the GPU, it is the prompt
Between 39 and 47 percent of my tokens were identical on every request.
You size an inference system thinking about compute. The share of work that repeats verbatim from one call to the next stays invisible until you instrument it, and that share is what decides whether a cache buys you anything at all.
Your quality metric lies until proven otherwise
38 percent of the items that mattered fell outside its scope.
A metric that does not cover what you believe it covers raises no flags. It reassures. That is worse than having no metric, because it occupies the slot where a real one would go.
Trust is earned through what the model is forbidden to do
On my system it never regenerates source text. Not once.
This is not stylistic caution. A quotation that has to remain citable cannot be rewritten by a model. Provenance is reconstructed from the source, never produced by the model.
It is an architectural constraint, and it is what makes the system defensible to someone who has no particular wish to believe it.