Skip to Content
AI GatewayFailover & routes

Failover & routes

When a provider is down or returns an error, the gateway can automatically try the next model in a chain you configure—primary → fallbacks—so your app keeps a single endpoint.

You set this up in the console under Marshal → AI Gateway → Management → Routes. Prefer console routes over hard-coding dual clients in application code.

Failover applies to non-stream Chat Completions and Responses when a route matches. Streaming, passthrough, and requests with no matching route use a single provider only.

When it runs

Runs when

  • The request is non-stream chat or responses
  • A route matches the operation and model
  • The primary fails with a retryable/fallbackable error (for example upstream 429, 5xx, or timeout after retries)

Does not run when

  • The request is streaming
  • You use provider passthrough (/gateway/openai, /gateway/anthropic, …)
  • No route matches (default provider/model routing only)
  • The gateway itself returns 429 for your org/API-key rate limit (that is terminal—see Rate limits)

Console setup

Open AI Gateway → Management .

Enable providers (credentials + allowlist)

On the Providers tab, confirm connectors have credentials ready, then add providers to the allowlist. Disabled providers are skipped in a failover chain.

AI Gateway Management Providers tab with connector credentials

Open Routes

Switch to the Routes tab. Without routes, traffic uses default model-prefix routing only. Click + Add route to define a primary target, retries, and fallbacks.

AI Gateway Management Routes tab with a primary → fallback route

Create a route (primary + fallbacks)

In the Add route dialog, fill in:

FieldWhat to set
NameFriendly id (e.g. chat-openai-failover)
Operationchat, responses, or *
PriorityHigher wins when several routes match
Match model prefix / exact modele.g. openai/ or openai/gpt-4o-mini
Primary targetProvider + model, retries, timeout
FallbacksOrdered list of next targets (same fields). Tried after primary retries are exhausted

Fallbacks run on 5xx, 429, or primary timeouts (same-target retries first, then the next fallback).

Edit route dialog with primary target, retries, and status-code policies

Save and verify

Save the route, then send a non-stream Chat Completions request for a matching model. If the primary fails, the gateway walks fallbacks. Check Analytics / audit to see which target served the call.

Creating and editing routes requires an admin role in the org.

Behavior notes

  • Retries happen on the same target; failover moves to the next fallback.
  • Unhealthy providers can be skipped temporarily until they recover.
  • Hard client errors (for example 400 validation) return immediately—they do not walk fallbacks.
  • Response shape can change across vendors (OpenAI → Anthropic). Prefer unified Chat Completions when clients need a stable schema.
  • Prefer non-stream for production failover coverage today.

A gateway rate limit rejection is not an upstream failure. Honor Retry-After and retry later—failover will not switch providers for tenant quota exhaustion.

Guidelines

  1. Configure failover in Management → Routes, not in app code.
  2. Use provider/model ids so routing and analytics stay clear.
  3. Put expensive models behind routes with cheaper fallbacks.
  4. Pair with response cache for repeated prompts (hits never reach failover).
  5. Pair spend controls with rate limits and Guardrails.
Last updated on
Rate limitsOverview