Docs
Rate limits
Design clients that behave well under per-key quotas, provider capacity, and temporary 429 responses.
What is limited
Available throughput can vary by API key, model, account group, and upstream provider. The models endpoint and the response headers or error body are the best sources for the limits that apply to your request.
Handle 429 responses
When the gateway returns 429, pause before retrying. Prefer a bounded exponential backoff with jitter, cap concurrent requests, and avoid sending the same request in a tight loop. Check your available balance when the error indicates quota exhaustion.
import random
import time
for attempt in range(5):
try:
response = client.chat.completions.create(...)
break
except Exception as exc:
if getattr(exc, "status_code", None) != 429 or attempt == 4:
raise
time.sleep(min(30, 2 ** attempt) + random.random())Production throughput
Start with a small concurrency setting, measure latency and error rates, then increase gradually. If your workload needs higher sustained throughput, contact the operator with the models, request volume, and target concurrency so the applicable limit can be reviewed.