The inference cost curve is the only number that matters now

Every few months a new model tops a benchmark and the coverage treats it as the story. The more consequential number is what it costs to serve a token, and that has been falling on a curve steep enough to change what is buildable.

Why cost dominates capability

A capability that exists but costs too much to serve is a demo. Most of the products that failed in the last two years failed on unit economics, not on quality — the model could do the task and the margin did not survive doing it at scale.

When serving cost drops by an order of magnitude, a set of products that were uneconomic become economic overnight, without anyone shipping a smarter model.

What it changes

Per-seat pricing stops working where usage is unbounded. Vendors that priced per seat while paying per token have been quietly adding usage caps, and that will continue.

Batch beats interactive for a growing class of problems. If a task can wait an hour, it can run at a fraction of the cost, and a lot of “real-time” features never needed to be.

The build-versus-buy line moves. Fine-tuning a smaller model becomes rational at a lower volume than it did a year ago.

What to watch

Published price-per-million-token figures across providers, and the gap between the flagship and the small model. That gap, not the benchmark score, tells you what the next wave of products will look like.

Reported at TechCrunch and The Verge; analysis ours.