Shipping an LLM feature without a surprise invoice

Dashboard showing per-conversation token usage and cost for a production LLM assistant

The first production LLM feature I shipped cost roughly nine times what I had estimated in its first week. Traffic was as predicted. My mental model of the billing was wrong.

With a server, cost tracks requests. With an LLM, cost tracks tokens — and in a chat interface, tokens grow with the square of the conversation. Every turn resends the whole history, so turn ten does not cost the same as turn one. It costs about ten times as much.

The quadratic you did not plan for

A twenty-turn conversation where each turn adds 200 tokens does not send 4,000 tokens. It sends the accumulating history twenty times over, which is around 42,000 input tokens. Ten users having long conversations can cost more than a thousand users asking one-off questions.

diagram

flowchart TD
    R[Request] --> C{Exact cache hit?}
    C -->|yes| H[Return cached, 0 tokens]
    C -->|no| S{Semantic cache hit?}
    S -->|yes| H
    S -->|no| T{Task complexity}
    T -->|classify, extract, route| Small[Small model]
    T -->|reason, synthesise, write| Large[Frontier model]
    Small --> W[Write to cache]
    Large --> W

Two cache layers and a routing decision before any expensive call. Most production traffic never reaches the frontier model.

Cache in two layers

An exact-match cache on the normalised prompt is trivial and catches more than you would expect, because real users ask the same handful of questions constantly. A semantic cache — embed the question, return the stored answer if cosine similarity clears a threshold — catches the rewordings.

cache.pypython

import hashlib

SIMILARITY_FLOOR = 0.95


def cache_key(prompt: str, model: str) -> str:
    normalised = " ".join(prompt.lower().split())
    return hashlib.sha256(f"{model}:{normalised}".encode()).hexdigest()


async def answer(prompt: str, model: str):
    key = cache_key(prompt, model)

    if hit := await redis.get(key):
        return hit, "exact"

    # Semantic layer: catches rewordings the hash cannot.
    embedding = await embed(prompt)
    match, score = await vectors.nearest(embedding)
    if score >= SIMILARITY_FLOOR:
        return match.answer, "semantic"

    result = await llm.complete(prompt, model=model)

    await redis.set(key, result, ex=86_400)
    await vectors.add(embedding, answer=result)
    return result, "miss"

Route by task, not by default

Frontier models are worth their price for reasoning and synthesis. They are wasted on classification, extraction, and deciding which of four intents a message belongs to — jobs a model an order of magnitude cheaper does just as well. Most systems send everything to one model because that was easiest on day one.

Intent classification, routing, yes/no judgements — the smallest model that passes your evaluation set.
Structured extraction from a known schema — a small model with a strict output format.
Multi-step reasoning, long-form writing, ambiguous requests — the expensive model, and worth it.
Anything you have a cached answer for — no model at all.

Cap the history deliberately

Keep the system prompt, a rolling summary of everything older, and the last few turns verbatim. Conversation cost then flattens instead of compounding, and quality barely moves — because turn three of a twenty-turn conversation is rarely what determines the right answer to turn twenty.

Set a ceiling in code

Provider dashboards alert you after the money is spent. A hard per-user daily token budget, enforced in your own code, is the only thing that actually stops a runaway loop or an abusive client. Degrade to the cheap model at 80% of budget, refuse politely at 100%.

None of this is premature optimisation. Caching, routing and a history cap are perhaps a day of work, and they are the difference between a feature you can leave running and one you have to switch off at the end of the month.

LLMCostCachingFastAPIProduction

All articles