The Cost Problem

LLM API costs can spiral quickly. A modest app serving 10K users can cost $500-2000/month. Here's how to bring that down.

1. Cache Aggressively

  • Cache identical or similar queries with semantic caching.
  • Use Redis or in-memory LRU for hot paths.
  • Cache rate: 30-60% for most applications.

2. Use Smaller Models for Simple Tasks

  • Route easy queries to a lower-cost tier such as GPT-6 Luna or Claude Haiku 5.5.
  • Reserve a frontier model such as GPT-6 Astra or Claude Sonnet 5.5 for complex reasoning.
  • Use a classifier to determine routing.

3. Optimize Prompts

  • Shorter system prompts = fewer input tokens.
  • Remove redundant instructions.
  • Use structured output to avoid verbose responses.

4. Batch Requests

  • OpenAI and Anthropic offer batch APIs at 50% discount.
  • Queue non-urgent requests and process in batches.

5. Limit Context Window

  • Only send relevant context, not entire documents.
  • Use RAG to retrieve just the needed chunks.
  • Summarize long conversations before sending.

6. Monitor and Set Budgets

  • Use usage dashboards to track spending.
  • Set hard limits per user and per feature.
  • Alert on unusual spikes.

7. Fine-Tune Instead of Prompt

  • A fine-tuned small model can replace a large prompted model.
  • Saves 5-10x on per-token costs at scale (rule of thumb, not a published figure: the break-even depends on your token volume, training cost, and the model pair, so measure it on your own traffic).

8. Use Streaming

  • Streaming lets users cancel early, saving output tokens.
  • Better UX too — users see partial results.

9. Self-Host for High Volume

  • At >10M tokens/day, self-hosted open-source models are often cheaper (rule of thumb, not a published threshold: it ignores engineering time, idle GPU capacity, and whether you already run the hardware, so compare against your own bill).
  • Use vLLM or TGI for efficient serving.

10. Negotiate Enterprise Plans

  • Most providers offer volume discounts.
  • Commit to monthly spend for better rates.

Quick Win: Semantic Caching

Many API calls are duplicates or near-duplicates. A semantic cache stores embeddings of previous inputs and their responses. When a new request arrives, compute its embedding and check similarity against the cache. If cosine similarity exceeds 0.95, return the cached response. This alone can cut costs by 30–60% for many applications.

Model Routing: The 80/20 Rule

Not every request needs a frontier model. Route simple requests (formatting, short answers, classification) to lower-cost tiers such as GPT-6 Luna or Claude Haiku 5.5. Reserve expensive models for complex reasoning, long-form generation, or tasks requiring high accuracy. A simple classifier at the entry point can sort requests by complexity and route accordingly.

Batching and Async Processing

If your use case doesn’t require real-time responses, batch requests. OpenAI’s batch API offers 50% discount. Process during off-peak hours for even better rates. Similarly, use streaming selectively—streaming is great for chat UIs but wastes resources for background processing.

Most of Your Bill Is Input Tokens You Did Not Need

The instinct is to negotiate the per-token price. The larger lever is usually the token count, and most of it is input. Long system prompts, retrieved context that includes whole documents instead of the relevant passage, and conversation history that grows without pruning all multiply your bill on every single request. Measure the input-to-output token ratio for your app; if it is far above what the task requires, you have found your first project.

Prune the history deliberately: keep the last few turns verbatim, summarise older ones, and drop the rest. Cache the static prefix if your provider supports prefix caching, and structure the prompt so the static part comes first and the variable part last. That ordering alone changes the cached fraction substantially.

Route by Difficulty

Not every request needs your largest model. A cheap classifier that sends simple requests to a small model and escalates only the hard ones can cut cost by a large factor while keeping quality where it matters. Measure the quality of the escalated path against sending everything to the big model; the usual finding is that the small model handles the majority of traffic acceptably and the escalation rate is low.

Watch Out for Retry Storms

A failing request that retries three times costs four times as much and produces the same failure. Cap retries, distinguish retryable from non-retryable errors, and log the retry rate. A sudden cost spike is more often a retry loop than a traffic increase.

Buy the Cheapest Thing That Passes Your Eval

Run your own evaluation set against a smaller model before assuming you need the frontier. The gap is often smaller than the price difference, and the decision should come from your data rather than from a general impression.

Model names were checked against the official documentation of OpenAI and Anthropic on 2026-11-10.