Skip to Content
AI GatewayRate limits

Rate limits

The gateway enforces requests-per-minute (RPM), tokens-per-minute (TPM), and request burst after tenant auth and before cache, failover, or upstream calls. Policies bind to an organization or API key, and optionally to a model.

Pair with product Guardrails for agent-level budgets and breakers. Dollar spend caps are not enforced yet—use RPM/TPM for hard stops today.

When it applies

PathGuardrails rate limit?
/gateway/v1/chat/completions, /completions, /responses, /embeddingsYes
Capability proxies (images, audio, video)Yes (when policy effective)
/gateway/cursor/... chatYes
Passthrough /gateway/{provider}/...Yes (subject + defaults; model peeked when body present)

A gateway-enforced 429 is terminal—it never enters the failover engine. Upstream provider 429s remain failover-retryable.

Console setup

Open AI Gateway → Management  → Rate limits.

The Management header shows how many policies are active (for example 0 rate limits).

Open the Rate limits tab

Switch to Rate limits. If none exist yet, you see an empty state with + Add rate limit.

AI Gateway Management Rate limits tab empty state

Add a policy

Click + Add rate limit. Fill in:

FieldWhat to set
NameFriendly label (e.g. Default org RPM)
ScopeOrganization (org-wide) or a single API key
PriorityHigher wins when several policies match the same scope
ModelExact model id, or blank / * for every model
RPMRequests per minute (0 = off for this metric)
TPMTokens per minute (0 = off for this metric)
BurstExtra request allowance; 0 → default RPM/10
EnabledOff policies are ignored at request time

At least one of RPM or TPM must be greater than 0.

Add rate limit dialog with org scope, RPM, TPM, and burst

Create and verify

Click Create. Send a Chat Completions request and inspect X-RateLimit-* response headers. When you exceed the limit, expect 429 with Retry-After—failover will not switch providers.

Creating and editing rate limits requires an admin role in the org.

Admission rules

CheckBehavior
RPMConsume 1 request; window max ≈ RPM + burst (burst defaults to RPM/10, min 1)
TPMAdmit if remaining token budget in the current sliding window is > 0; tokens are added after the response from actual usage (JSON or SSE)

Trade-off: The last admitted request under TPM can overshoot the budget by its own usage; subsequent requests are blocked until the window slides. Gateway response cache hits skip TPM debit.

Client-visible contract

On allow (when RPM and/or TPM apply):

  • X-RateLimit-Limit-Requests / X-RateLimit-Remaining-Requests (if RPM > 0)
  • X-RateLimit-Limit-Tokens / X-RateLimit-Remaining-Tokens (if TPM > 0)

On deny:

FieldValue
Status429
Body{"error":"rate limit exceeded: requests per minute"} or ...tokens per minute
Retry-AfterSeconds until approximate window relief
Remaining headersSet to 0 where applicable
# Example: inspect rate-limit headers on a successful call
curl -D - -o /dev/null https://production-api.exemplar.dev/gateway/v1/chat/completions \
  -H "Authorization: Bearer $EXEMPLAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [{"role": "user", "content": "ping"}]
  }'
# Look for: X-RateLimit-Limit-Requests, X-RateLimit-Remaining-Requests, …

Policy resolution

Precedence (most specific wins):

  1. API key + exact model
  2. API key + all models (* / empty)
  3. Organization + exact model
  4. Organization + all models
  5. Platform defaults

Within a tier, higher priority (from the Console policy) wins.

Guidelines

  1. Start with an org-wide RPM in Management → Rate limits; add TPM when you need token budget control.
  2. Prefer API-key + model policies for expensive models (openai/gpt-4o, large Claude variants).
  3. Treat gateway 429 as a client retry-after-window signal—do not expect failover to another provider.
  4. Expect one-request TPM overshoot; size budgets with that in mind.

Daily / monthly dollar spend caps are out of scope for enforcement today. Use RPM/TPM for hard stops.

Last updated on
Response cacheFailover & routes