Why Serverless AI?
Traditional GPU servers are expensive and idle most of the time. Serverless inference lets you pay only for actual usage.
Platforms
Modal
- Python-native serverless cloud.
- GPU access with code changes.
- Great for ML pipelines and batch inference.
- Pricing: Per-second GPU billing.
Replicate
- Run open-source models via API.
- No infrastructure management.
- Huge model library (Llama, Stable Diffusion, Whisper).
- Pricing: Per-second GPU time.
Cloudflare Workers AI
- Edge-deployed models globally.
- Low latency for end users.
- Limited model selection but growing.
- Pricing: Per-request.
Banana / Together AI
- Specialized in LLM inference.
- Auto-scaling from zero to thousands of requests.
- Competitive pricing for high-volume workloads.
Architecture Patterns
- Cold Start Mitigation: Use model caching and pre-warming.
- Request Queuing: Buffer requests during scale-up.
- Edge + Cloud: Simple models at edge, complex in cloud.
When NOT to Use Serverless
- Consistent high-throughput workloads (>80% GPU utilization).
- Models too large for available serverless GPU memory.
- Latency-sensitive applications requiring <50ms response.
The most common pattern is API Gateway → Lambda/Cloud Function → Model endpoint. For GPU inference, use services like Modal, Replicate, or AWS Lambda with GPU (coming soon). For CPU inference (small models, quantized), standard Lambda functions work well.
Cold Start Optimization
Cold starts are the biggest challenge for serverless AI. Strategies:
- Provisioned concurrency: Keep warm instances ready (AWS) at a cost premium.
- Model caching: Store models in /tmp or EFS. Lambda can reuse files across invocations if the container is warm.
- Smaller models: A 100MB model loads in ~200ms; a 2GB model takes ~4 seconds. Use quantized models (GGUF Q4) to minimize size.
- SnapStart: AWS Lambda SnapStart reduces cold starts by 90% for Java-based inference servers.
Where Cold Starts Actually Hurt
A serverless function that must load a multi-gigabyte model on every cold start is not serverless in any useful sense; the first request after idle takes tens of seconds and the cost of keeping the model warm approaches the cost of a small VM. The practical split is to keep the model behind a managed inference endpoint and let the serverless function own only the orchestration: authentication, prompt assembly, retrieval, and response shaping. Those steps are milliseconds of work and scale to zero cleanly, while the expensive part stays on infrastructure designed to stay warm.
If you must run the model in-function, measure the p95 cold-start latency on your own payload, not the average from a blog post. Cold-start time depends on image size, layer caching, and whether the provider keeps your container warm after deploy. A reduced image and a smaller quantised model often cut it in half.
Streaming Changes the Economics
Token streaming is usually described as a UX feature, but it also changes how you budget time. A request that finishes in eight seconds feels broken when the user stares at a spinner and feels instant when the first token arrives in 300ms. Set your timeout on time-to-first-token rather than total duration, and let the stream continue past it.
Keep the Model Call Behind One Function
Whether you eventually move from serverless to a container, or from one provider to another, depends on how many places know about the model. One module that owns the client, the timeout, the retry policy, and the response shape turns a migration into an afternoon. Calling the SDK directly from five route handlers turns it into a rewrite.