How we cut LLM spend 38% without losing quality
Model routing, prompt caching, and per-tenant budgets sound obvious. The hard part is proving quality held. Here's the playbook — and the evals that made it safe to ship.
Every team running LLMs in production eventually hits the same wall: the bill. The instinct is to reach for a cheaper model, ship it, and hope quality holds. That hope is exactly the problem. Here's how we took 38% off our LLM spend across a multi-tenant SaaS platform — with the evals that let us prove, not assume, that quality held.
1. Route to the smallest model that passes
Not every request needs a frontier model. We tiered requests across Haiku, Sonnet, and Opus based on task complexity, and let the routing decision itself be data-driven: classify the task, try the cheaper tier first, and escalate only when structured-output validation or a confidence check fails.
- Cheap-first with automatic escalation beats picking one model for everything.
- Structured-output retries are cheaper than a wrong answer that reaches a customer.
- Per-tenant token budgets stop one noisy tenant from blowing the whole bill.
2. Cache the prompt, not just the answer
Long system prompts and retrieved context dominate token cost. Prompt caching on the stable prefix — instructions, few-shot examples, tool schemas — cut input tokens dramatically on high-volume endpoints without touching quality at all.
3. Prove quality held with a golden dataset
This is the step teams skip, and it's the one that makes cost work safe. We maintained a golden dataset per product and wired regression evals into CI, so every prompt or model change had to clear a quality gate before it merged.
If you can't measure quality, you can't cut cost — you can only gamble with it.
The result: a 38% reduction in spend, with quality tracked on every change instead of discovered in a customer complaint.