Engineering Notes
Fallback routing for predictable latency
Why a model gateway needs an explicit fallback path, and what it should and should not hide from your application.
What you will learn
- Retry only transient upstream failures; never replay invalid or unauthorized requests.
- Choose fallbacks by endpoint and workload, not by name alone.
- Log the selected route, retry count, and final upstream result.
Before you start
- Basic HTTP and API knowledge
Leave with a concrete implementation checklist and a testable starting point.
Key takeaways
- Retry only transient upstream failures; never replay invalid or unauthorized requests.
- Choose fallbacks by endpoint and workload, not by name alone.
- Log the selected route, retry count, and final upstream result.
The problem with a single upstream
A production feature can depend on one model provider while the provider is dealing with a busy region, a transient timeout, or a temporary capacity limit. If every request has one possible destination, that upstream incident becomes your incident too.
A gateway gives the application one stable address and keeps provider-specific decisions at the edge. The useful goal is not to pretend failures never happen. It is to make the failure mode bounded and observable.
What fallback should do
A practical fallback path starts after a request has been classified: endpoint, model, account group, and request policy are known. The router can then retry a safe transient failure or select an enabled alternative that matches the workload. Timeouts, authentication failures, and invalid requests should not be blindly replayed.
The application still receives a normal OpenAI-compatible response shape. This keeps retry logic close to the place that has the most context about available models and current provider health.
Designing for visibility
Fallback routing should leave a trace in gateway logs and metrics: selected model, upstream result, retry count, and final route. On the client side, keep your own request ID and latency measurement. Together these signals show whether a slow response came from your code, the gateway, or an upstream provider.
EasyAI exposes the current catalog and pricing through the marketing site and console. Check the live model list before making an alternative part of a production policy.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Classify the request by endpoint, model, account group, and timeout policy.
- 2
Define a small allow-list of compatible fallback models.
- 3
Retry once for an explicitly transient failure, then return a traceable error.
- 4
Measure p50/p95/p99 latency and fallback rate separately.
Copy-ready example
const candidates = ["deepseek-chat", "deepseek-reasoner"];
const retryable = new Set([429, 500, 502, 503, 504]);
for (const model of candidates) {
try {
return await client.chat.completions.create({ model, messages });
} catch (error) {
const status = error instanceof OpenAI.APIError ? error.status : undefined;
if (!status || !retryable.has(status) || model === candidates.at(-1)) throw error;
}
}Frequently asked questions
Should every timeout trigger a fallback?
No. Use a bounded timeout policy and distinguish a transient upstream timeout from client cancellation or an overloaded application queue.
Will fallback change the response quality?
It can. Record the final model and apply fallbacks only where the product requirement permits a quality trade-off.
Sources
- EasyAI documentationSource checked 2026-08-27