Production Deployment
Checklists and operational practice for running LLM-backed agents in production: provider configuration, resilience, monitoring and key capacity.
1 min read
Pre-Deployment Checklist
Before deploying to production, verify:
Provider Configuration
- API keys stored in a secrets manager (not environment variables or code)
- Multiple keys configured per provider for redundancy
- Circuit breaker enabled on every provider with appropriate thresholds
- Rate limits set below provider quotas to avoid hard rejections
- Fallback providers configured for critical paths
Resilience
- Circuit breaker thresholds tuned to your traffic patterns
- Retry policies configured with exponential backoff and jitter
- Timeout values set on all LLM calls (recommended: 30s for completions, 60s for long-form)
- Fallback routing tested — verify traffic shifts when a provider goes down
Monitoring
- Request latency tracked per provider
- Error rates tracked per provider and error type
- Token usage tracked for cost management
- Circuit breaker state changes logged
- Key rotation events logged
Scaling Considerations
Key Capacity
Rule of thumb: configure at least 3 keys per provider for production workloads. This provides:
- Redundancy if a key is revoked or rate-limited
- Better rate limit distribution across keys
- Zero-downtime key rotation
Next Steps
- Security Best Practices — secure your deployment
- Architecture Overview — system design