· 2 min read · Zain Akram
Measure AI features by cost per resolved task, not tokens
Token counts are a vanity metric. The one number that aligned engineering, finance and product on our AI features, and the four levers that moved it.
- AI
- Cost
- Evals
- Product
The first dashboard we built for the support agent showed tokens per day. It went up and to the right, everyone nodded, and nobody could answer the only question that mattered: is this worth the money?
We replaced it with one number: cost per resolved ticket. Model spend for a conversation, divided over conversations the user did not reopen or escalate within 48 hours. That number is comparable to a human support cost, so finance understood it immediately. It also punishes a cheap model that fails and a pricey model that rambles equally, which is the behaviour you want.
The four levers
Once you have the number, cost optimisation stops being guesswork. (There is a calculator for this that shows what each lever does to your own numbers.) These moved it, in order of effort.
1. Prompt caching, properly. The system prompt, tool definitions and retrieval instructions are frozen and identical across requests. No timestamps, no per-user values near the top. Conversation history is appended, never edited. Cache hit rate went from near zero to most of every request's input, and that alone cut input cost by more than half.
2. Model routing by task. Classifying the ticket type and choosing retrieval filters does not need the strongest model. Writing the final answer to a frustrated brokerage admin probably does. We route by step, not by conversation, and the eval set tells us when a cheaper route is good enough.
3. Effort, not budgets. Modern models take an effort setting. Low effort for lookups and summaries, higher for reasoning over ambiguous tickets. This replaced the old habit of guessing a thinking budget.
4. Shorter outputs on purpose. Support answers were capped by instruction, not by truncation. Shorter answers resolved more tickets, not fewer. People want the fix, not the essay.
Guard the quality side
Cost per resolved task only works if "resolved" is honest. We track reopen rate and escalation rate separately, and any cost change that moves either one is rejected. The eval set, built from real tickets and including adversarial ones, runs on every change.
What this changed
Model choice stopped being a debate. Someone proposes a change, it runs on the eval set, we look at cost per resolved ticket and the quality guards, and the better one ships. It turned AI from a line item people were nervous about into an ordinary engineering system with a budget and a dashboard.