Templates
LLM Cost Control Toolkit
Two cache layers, model routing and hard budget ceilings.
The middleware I wrote after a production LLM feature cost nine times its estimate in week one. Exact and semantic caching, task-based routing between cheap and frontier models, conversation history capping, and per-user token budgets enforced in your own code rather than discovered on an invoice.
What is inside
- Exact-match and semantic response caching with a configurable similarity floor
- Task-based routing so classification never reaches a frontier model
- Rolling conversation summarisation that flattens the quadratic token cost of long chats
- Per-user daily token budgets, with graceful degradation before refusal
- Structured cost logging tagged by feature and user, ready for a dashboard
- Provider-agnostic adapters for OpenAI, Anthropic and local models
Built with
- Python
- FastAPI
- Redis
- OpenAI
- Anthropic
Next step
Have an AI problem worth solving?
Tell me what you are trying to build. I will tell you honestly whether AI is the right tool, what it would take, and how I would approach it.
Typical reply within 24 hours
Working from Dhaka, Bangladesh · remote worldwide