The Cost Problem
LLM API costs can spiral quickly. A modest app serving 10K users can cost $500-2000/month. Here's how to bring that down.
1. Cache Aggressively
- Cache identical or similar queries with semantic caching.
- Use Redis or in-memory LRU for hot paths.
- Cache rate: 30-60% for most applications.
2. Use Smaller Models for Simple Tasks
- Route easy queries to GPT-5-mini or Haiku ($10x cheaper).
- Reserve GPT-5 or Claude for complex reasoning.
- Use a classifier to determine routing.
3. Optimize Prompts
- Shorter system prompts = fewer input tokens.
- Remove redundant instructions.
- Use structured output to avoid verbose responses.
4. Batch Requests
- OpenAI and Anthropic offer batch APIs at 50% discount.
- Queue non-urgent requests and process in batches.
5. Limit Context Window
- Only send relevant context, not entire documents.
- Use RAG to retrieve just the needed chunks.
- Summarize long conversations before sending.
6. Monitor and Set Budgets
- Use usage dashboards to track spending.
- Set hard limits per user and per feature.
- Alert on unusual spikes.
7. Fine-Tune Instead of Prompt
- A fine-tuned small model can replace a large prompted model.
- Saves 5-10x on per-token costs at scale.
8. Use Streaming
- Streaming lets users cancel early, saving output tokens.
- Better UX too — users see partial results.
9. Self-Host for High Volume
- At >10M tokens/day, self-hosted open-source models are cheaper.
- Use vLLM or TGI for efficient serving.
10. Negotiate Enterprise Plans
- Most providers offer volume discounts.
- Commit to monthly spend for better rates.
Quick Win: Semantic Caching
Many API calls are duplicates or near-duplicates. A semantic cache stores embeddings of previous inputs and their responses. When a new request arrives, compute its embedding and check similarity against the cache. If cosine similarity exceeds 0.95, return the cached response. This alone can cut costs by 30–60% for many applications.
Model Routing: The 80/20 Rule
Not every request needs GPT-5 or Claude Sonnet. Route simple requests (formatting, short answers, classification) to cheaper models like GPT-5-mini or Claude Haiku. Reserve expensive models for complex reasoning, long-form generation, or tasks requiring high accuracy. A simple classifier at the entry point can sort requests by complexity and route accordingly.
Batching and Async Processing
If your use case doesn’t require real-time responses, batch requests. OpenAI’s batch API offers 50% discount. Process during off-peak hours for even better rates. Similarly, use streaming selectively—streaming is great for chat UIs but wastes resources for background processing.