Model Economics
How to Measure AI Gateway Reliability: Latency, Error Rate, and Route Health
Define the metrics that explain whether an AI gateway is healthy and how to turn them into operational decisions.
What you will learn
- Segment reliability metrics by endpoint and model.
- Separate gateway time from upstream time.
- Every alert needs a threshold, owner, and runbook action.
Before you start
- Basic HTTP and API knowledge
Leave with a concrete implementation checklist and a testable starting point.
Key takeaways
- Segment reliability metrics by endpoint and model.
- Separate gateway time from upstream time.
- Every alert needs a threshold, owner, and runbook action.
Measure outcomes, not activity
Request count alone does not describe reliability. Track successful responses, user-visible errors, timeout categories, latency percentiles, and cost per successful request.
Always segment metrics by endpoint and model. An aggregate number can look healthy while one critical route is failing.
The core dashboard
A useful dashboard includes p50, p95, and p99 latency; 4xx and 5xx rates; upstream versus gateway time; retry and fallback rates; and active rate-limit responses.
Pair every metric with a threshold and an action. A chart without an owner or runbook is not an operational control.
Route health
Route health combines recent success, latency, and capacity signals for an eligible model. It should expire quickly and never override hard policy constraints.
When a route is degraded, record the reason and the alternative selected so the incident timeline remains understandable.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Define success and failure categories.
- 2
Instrument request ID, route, status, and latency.
- 3
Build percentile and error-rate dashboards.
- 4
Attach alerts to runbooks and review them after incidents.
Copy-ready example
p99_gateway = percentile(total_request_ms, 99)
p99_upstream = percentile(upstream_wait_ms, 99)
fallback_rate = fallback_requests / eligible_requestsFrequently asked questions
Why use p99 instead of average latency?
Averages hide the slow tail that users experience during overload or provider degradation.
Sources
- EasyAI documentationSource checked 2026-08-27