Response cache

Reuse LLM answers for identical or near-duplicate prompts so you spend less and respond faster. Set an org default in the console, or override any single call with the x-exemplar-cache-config header.

Caching applies to non-stream Chat Completions and Responses only. Streaming always bypasses the cache. The Console tab sets org policy for exact-match and semantic modes.

Cache hits skip TPM debit—see Rate limits.

When it applies

Applies when

The request is non-stream chat/completions or responses
Org default type is not none, or the request sends an opt-in x-exemplar-cache-config
Caching is available for your org (exact-match and/or semantic, depending on platform enablement)

Does not apply when

"stream": true → response header x-exemplar-cache-status: bypass
Org default is off/none and the request does not opt in
You opt out with {"type":"none"} or {"enabled":false}

Console setup

Open AI Gateway → Management → Cache.

The Management header shows the current org cache state (for example cache off or cache exact-match):

AI Gateway Management status showing providers, routes, and cache state

Open the Cache tab

Switch to Cache. This panel is the org default for non-stream chat/responses. Per-request x-exemplar-cache-config still overrides.

AI Gateway Management Cache tab with Enabled, type, TTL, and similarity

Keep off until you are ready

Leave ENABLED off so the org default stays none. Callers can still opt in with the header.

Enable carefully

When enabling:

Start with Default type = exact-match.
Set TTL seconds (typical 600).
Use semantic only for FAQ-style near-duplicates—set Similarity threshold ≈ 0.9 and a tight namespace (for example faq:v1).

Save

Click Save cache. Toggle ENABLED off and Save again to disable caching for that org only.

Setting	Typical value	Notes
Enabled	off → on when ready	Off keeps org default as `none` (header can still opt in)
Default type	`exact-match`	Safer than semantic; `none` when disabled
TTL seconds	`600`	Entry lifetime
Similarity threshold	`0.9`	Semantic only (cosine ≥ threshold = hit)
Namespace	`default` or `faq:v1`	Isolates keys across products/versions

Saving cache settings requires an admin role in the org.

Per-request headers

Override (or opt in/out) with JSON in x-exemplar-cache-config:

Intent	Header value
Opt out	`{"type":"none"}` or `{"enabled":false}`
Exact match	`{"type":"exact-match","ttl":600,"namespace":"faq:v1"}`
Semantic	`{"type":"semantic","similarity_threshold":0.9,"ttl":300,"namespace":"faq:v1"}`

Response header x-exemplar-cache-status: hit · miss · bypass (and optionally x-exemplar-cache-similarity-score / x-exemplar-cached-trace-id on semantic hits).

curl — exact-match


curl https://production-api.exemplar.dev/gateway/v1/chat/completions \
  -H "Authorization: Bearer $EXEMPLAR_API_KEY" \
  -H "Content-Type: application/json" \
  -H 'x-exemplar-cache-config: {"type":"exact-match","ttl":600,"namespace":"faq:v1"}' \
  -D - -o /tmp/chat.json \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [{"role": "user", "content": "What is our P1 severity definition?"}]
  }'
# Look for: x-exemplar-cache-status: miss (first call), hit (identical second call)

curl — opt out (fresh answer)


curl https://production-api.exemplar.dev/gateway/v1/chat/completions \
  -H "Authorization: Bearer $EXEMPLAR_API_KEY" \
  -H "Content-Type: application/json" \
  -H 'x-exemplar-cache-config: {"type":"none"}' \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [{"role": "user", "content": "List open Sev-1 incidents right now"}]
  }'

Use opt-out for live data, tool-backed prompts, or personalized content.

SDK examples


import OpenAI from "openai";
 
const gateway = new OpenAI({
  apiKey: process.env.EXEMPLAR_API_KEY!,
  baseURL: "https://production-api.exemplar.dev/gateway/v1",
  defaultHeaders: {
    "x-exemplar-cache-config": JSON.stringify({
      type: "exact-match",
      ttl: 600,
      namespace: "faq:v1",
    }),
  },
});
 
await gateway.chat.completions.create({
  model: "openai/gpt-4o-mini",
  messages: [{ role: "user", content: "What is our P1 severity definition?" }],
});
 
// One-off bypass
await gateway.chat.completions.create(
  {
    model: "openai/gpt-4o-mini",
    messages: [{ role: "user", content: "List open Sev-1 incidents right now" }],
  },
  { headers: { "x-exemplar-cache-config": JSON.stringify({ type: "none" }) } },
);


import json
import os
from openai import OpenAI
 
gateway = OpenAI(
    api_key=os.environ["EXEMPLAR_API_KEY"],
    base_url="https://production-api.exemplar.dev/gateway/v1",
    default_headers={
        "x-exemplar-cache-config": json.dumps(
            {"type": "exact-match", "ttl": 600, "namespace": "faq:v1"}
        ),
    },
)
 
gateway.chat.completions.create(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "What is our P1 severity definition?"}],
)
 
gateway.chat.completions.create(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "List open Sev-1 incidents right now"}],
    extra_headers={"x-exemplar-cache-config": json.dumps({"type": "none"})},
)

For semantic FAQ near-duplicates, use "type":"semantic" with similarity_threshold ≈ 0.9 and a versioned namespace (same header shape as the table above).

Guidelines

Default org policy to off; enable exact-match first.
Keep namespaces versioned (faq:v1) so prompt pack changes do not serve stale hits.
Never cache streaming (stream: true → bypass).
Opt out when answers must reflect live systems or user-specific state.
Pair with rate limits for FinOps visibility.

Cache lookup runs after rate-limit admission and before failover. Hits never call upstream providers.

Failover & routes — runs on cache miss
Rate limits — hits skip TPM debit
Unified API — chat and responses surfaces