I keep seeing engineers get their first real LLM bill and quietly panic. None of these fix it alone, but stack a few and it adds up fast.
- Model routing — send easy tasks to a cheap model, save the expensive one for what actually needs it.
- Context trimming — strip anything irrelevant before it hits the prompt. Unused tokens are still paid tokens.
- Structured output — ask for JSON. Fewer tokens, fewer parsing failures, fewer retries.
- Cost monitoring — track usage and cost per request, and set budget alerts before the invoice surprises you.
- Agent guardrails — cap iterations, tool calls, and spend so nothing loops its way into a huge bill.
- Prompt caching — cache long system prompts instead of resending them every call.
- Semantic caching — if a question is close enough to one you've already answered, reuse that answer.
- Real RAG — retrieve only the relevant chunks, not the whole knowledge base.
- Output token limits — cap response length. You pay for output tokens too.
- Conversation summarization — compress older turns instead of resending full history every time.
None of these are hard individually. The hard part is doing all ten instead of the two that were easy to bolt on.
A few more worth knowing, once the basics are in place:
- Batch APIs — most providers give you ~50% off for anything that doesn't need a live response. Free money for async jobs.
- Skip the LLM entirely — a chunk of "AI" requests are really just classification or lookup. Catch those with a cheap check first.
- Fine-tune for repeat tasks — a small fine-tuned model beats a big general one on the one task you hit thousands of times a day.
- Dedupe and debounce — don't fire a call on every keystroke, don't run the same in-flight request twice.
- Self-host at scale — past a certain volume, a quantized open-source model can undercut API pricing. Only worth the ops overhead once volume justifies it.
- Fallback ladders — try cheap first, escalate only on failure or low confidence. Different from routing — this decides after seeing the result.
- Prompt compression — shrink few-shot examples and instructions with tools like LLMLingua, not just trim irrelevant context.