Small Models, Big Systems: The Case for Routing Instead of Scaling

The reflex when an AI feature underperforms is to reach for a bigger model. It is a reasonable first move and a terrible steady state. Most production workloads are a mix of tasks with wildly different difficulty, and paying frontier prices to classify a support ticket or extract three fields from an invoice is a choice, not a necessity.

Sort your traffic before you tune your model

Spend an afternoon reading a sample of real requests and you will usually find the same distribution. A large share are mechanical: extraction, classification, formatting, short rewrites. A middle band needs reasoning over a modest amount of context. A small tail is genuinely hard, requiring multi-step reasoning or long-context synthesis. Serving all three from one endpoint means the mechanical majority subsidises the difficult minority, and everyone waits in the same slow queue.

Route on features you can compute cheaply

A useful router does not need to be clever. Input length, task type from the calling code path, whether tools are required, and whether the last attempt failed will get you most of the way. Start with explicit rules per feature rather than a learned router; rules are debuggable, and you can read them in an incident.

Add one escalation rule: if the small model's output fails validation, retry once on the larger model. Now your worst case is bounded and your average case is fast. This single pattern, applied to the mechanical band of traffic, tends to cut cost dramatically while improving p50 latency, because most requests never touch the expensive path.

Validate structurally, not semantically

Escalation only works if you can tell that an output is bad without asking another model. That means designing tasks so failure is machine-detectable: require JSON that must parse against a schema, require an extracted value that must exist verbatim in the source, require a citation whose identifier must resolve. Tasks with checkable outputs are cheaper to run, easier to escalate, and safer to automate.

Latency is a product feature, not an infra metric

Users experience a fast small model as a better product than a slow accurate one for anything interactive. Autocomplete, inline suggestions, search reranking, and form assistance all live under a threshold where being right two hundred milliseconds later is worse than being nearly right now. Reserve the slow path for work the user has explicitly asked to wait on.

The operational payoff

• Cost becomes controllable. You can change the routing table without touching product code.
• Failures become local. A degraded provider takes out one tier, not the whole feature.
• Upgrades become testable. New models slot into a tier and get compared on that tier's tasks, rather than being swapped in globally and hoped over.

Treat models as interchangeable capacity behind a policy layer, not as a dependency you hard-code. The half-life of "the best model" is now measured in weeks.

Where it goes wrong

Two anti-patterns to avoid. Do not build a learned router before you have a rules-based one working, because you will have no baseline to justify it. And do not route on cost alone, ignoring which tier your evaluation set actually covers; a router that saves money by sending hard tasks to a weak model is not an optimisation, it is a quality regression with a nice dashboard.

The goal is not to use the smallest model. It is to stop treating model choice as a one-time architectural decision and start treating it as a runtime policy you own.