Evals

Marshal evaluates ingested agent sessions on two tracks. Both need sessions in the platform first (SDK ingest, framework hooks, or gateway-attributed traffic).

Track	What it answers	Granularity	Console
Insights (LLM judge)	What to optimize—tokens, tools, context, limits—with impact estimates	Whole session / run	Marshal → Insights
Session Evals	Did each turn stay relevant, faithful, and tool-aligned?	Per turn	Marshal → Session Evals

Marketing overview: exemplar.dev/marshal/evals .

Insights ≠ Session Evals. Judge Insights recommend product/ops changes. Session Evals score turn-level quality (DeepEval-style metrics). You can run either or both on the same session.

Prerequisites

An org API key (eis_*) — Tokens and API keys.
At least one ingested session with turns (input/output, optional tools/context).
Judge / session-eval services enabled for the org (platform default for production).


export EXEMPLAR_API_KEY=eis_…

Generate Insights (LLM judge)

Via SDK

Option A — auto-run on ingest (recommended for live agents):

Python


from exemplar_harness import Harness
 
harness = Harness.from_env(agent_id="support-bot")
session = harness.session(
    "sess-abc",
    agent_id="support-bot",
    source_app="my-app",
    auto_judge_run=True,  # queue a judge run after this ingest
)
session.ingest(
    "generic",
    event="turns",
    data={
        "turns": [
            {
                "input": "Summarize open incidents",
                "output": "Three Sev2s in payments…",
                "model": "gpt-4o",
            }
        ]
    },
)

Or pass flags on a one-shot ingest:


harness.ingest(
    "generic",
    session_id="sess-abc",
    event="turns",
    data={"turns": [{"input": "…", "output": "…", "model": "gpt-4o"}]},
    agent_id="support-bot",
    source_app="my-app",
    auto_judge_run=True,
)

Option B — trigger judge on existing sessions:

Python


from exemplar_harness import Harness
 
harness = Harness.from_env()
# Async by default; set sync=True to wait for completion in demos
harness.runs.trigger(session_ids=["sess-abc"], force=False, sync=False)
# Optional: judgement = harness.runs.get("<resultId>")

Framework post-hooks / callbacks that call harness.ingest(...) can pass the same flags—see Agent hooks that ingest below and Client usage → Session ingest.

Via UI

Open Insights

Console → Marshal → Insights (/marshal?subtab=insights).

Switch to Sessions

Use the Sessions view to list ingested sessions and their judge status.

Run judge

Click Run judge (or Judge again) on a session. Wait until status is completed, then open Findings / Runs for recommendations, evidence, and savings estimates.

Findings feed Cost Intelligence (/marshal?subtab=cost-intelligence) when impact is priced.

Agent hooks that ingest

Wire ingest into the agent runtime so every turn lands in Marshal automatically—optionally queueing Insights and Session Evals on the same call.

Python — Agno post-hook


pip install "exemplar-harness-sdk[agno]"


from agno.agent import Agent
from agno.models.openai import OpenAIChat
from exemplar_harness import Harness
from exemplar_harness.integrations.agno import harness_agno_post_hook
 
harness = Harness.from_env(agent_id="support-bot")
session_id = "sess-agno-1"
 
agent = Agent(
    name="support-bot",
    model=OpenAIChat(id="gpt-4o-mini"),
    post_hooks=[
        harness_agno_post_hook(
            harness,
            session_id=session_id,
            agent_id="support-bot",
            source_app="my-app",
            auto_judge_run=True,       # Insights after each run
            auto_session_eval=True,    # Session Evals after each run
        )
    ],
    instructions=["Be concise."],
)
agent.run("Summarize open Sev2 incidents.")
# Review: Marshal → Insights / Session Evals for sess-agno-1

Python — LangChain / LangGraph callbacks


pip install "exemplar-harness-sdk[langchain]"


from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
from exemplar_harness import Harness
from exemplar_harness.integrations.langchain import make_langchain_callback_handler
 
harness = Harness.from_env(agent_id="support-bot")
session_id = "sess-lc-1"
handler = make_langchain_callback_handler(
    harness,
    session_id=session_id,
    chain_name="support-bot",
    source_app="my-app",
    auto_judge_run=True,
    auto_session_eval=True,
)
llm = ChatOpenAI(model="gpt-4o-mini", callbacks=[handler])
agent = create_react_agent(llm, tools=[])
await agent.ainvoke(
    {"messages": [("user", "What is our refund window?")]},
    config={"callbacks": [handler]},
)

Python — OpenAI SDK wrapper


pip install "exemplar-harness-sdk[openai]"


from openai import OpenAI
from exemplar_harness import Harness
from exemplar_harness.integrations.openai import HarnessOpenAICallback, sdk_chat_completion
 
harness = Harness.from_env(agent_id="support-bot")
callback = HarnessOpenAICallback(
    harness,
    session_id="sess-oai-1",
    agent_id="support-bot",
    source_app="my-app",
    auto_judge_run=True,
    auto_session_eval=True,
)
sdk_chat_completion(
    harness,
    callback,
    OpenAI(),
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Draft a status update"}],
)

TypeScript — OpenAI Agents `wrapOpenAIAgentsRun`


npm install @exemplar-dev/exemplar-harness-typescript-sdk @openai/agents


import { Agent, run } from "@openai/agents";
import { Harness } from "@exemplar-dev/exemplar-harness-typescript-sdk";
import { wrapOpenAIAgentsRun } from "@exemplar-dev/exemplar-harness-typescript-sdk/integrations/openai-agents";
 
const harness = Harness.fromEnv({ agentId: "support-bot" });
const tracedRun = wrapOpenAIAgentsRun(harness, run, {
  sessionId: "sess-oai-agents-1",
  sourceApp: "my-app",
  autoJudgeRun: true,
  autoSessionEval: true,
});
 
const agent = new Agent({
  name: "Assistant",
  instructions: "Be concise.",
});
await tracedRun(agent, "Summarize open Sev2 incidents.");

TypeScript — Vercel AI SDK `onFinish`


npm install @exemplar-dev/exemplar-harness-typescript-sdk ai


import { generateText } from "ai";
import {
  Harness,
  ExemplarVercelAISession,
} from "@exemplar-dev/exemplar-harness-typescript-sdk";
 
const harness = Harness.fromEnv({ agentId: "support-bot" });
const session = new ExemplarVercelAISession(harness, {
  sessionId: "sess-vercel-1",
  sourceApp: "my-app",
  autoJudgeRun: true,
  autoSessionEval: true,
});
const prompt = "Summarize our refund policy.";
await generateText({
  model: yourModel,
  prompt,
  onFinish: session.onFinish(prompt),
});

Runnable peers: typescript/ingest/ · Python frameworks: python/frameworks/. Full adapter matrix: Frameworks with Marshal SDK.

Trigger Session Evals

Via SDK

Set auto_session_eval / autoSessionEval when ingesting so the platform scores turns after the session is stored:

Python


from exemplar_harness import Harness
 
harness = Harness.from_env(agent_id="support-bot")
session = harness.session(
    "sess-abc",
    agent_id="support-bot",
    source_app="my-app",
    auto_session_eval=True,  # queue per-turn session eval after ingest
)
session.ingest(
    "generic",
    event="turns",
    data={
        "turns": [
            {
                "input": "What is our refund window?",
                "output": "30 days with receipt.",
                "model": "gpt-4o",
            }
        ]
    },
)
 
# You can combine both tracks on the same session:
# auto_judge_run=True, auto_session_eval=True

On-demand re-runs for specific metrics are available from the console (below). Prefer the UI when you need to pick metrics or force a re-score after changing prompts.

Via UI

Open Session Evals

Console → Marshal → Session Evals (/marshal?subtab=session-evals).

Pick metrics

Use the metric picker (relevancy, faithfulness, tool alignment, and related DeepEval metrics). Selection is remembered for the next run.

Run eval

From a row, the detail pane, or bulk select: click Run eval (or Re-run eval). Open the session for per-turn scores, pass rates, and side-by-side compare.

What to do with results

Outcome	Next step
Insights: rate / budget / tool spam	Guardrails · gateway rate limits
Insights: context / skills gaps	Skills · Prompts · Context
Session Eval regressions	Hold prompt/skill promotion; re-run after the fix
Fleet $ opportunity	Marshal → Cost Intelligence

Client usage — Session ingest — direct ingest, session helper, and framework hooks
Evals → Agent hooks — Agno / LangChain / OpenAI Agents / Vercel AI with auto judge + session eval
SDKs & CLI — when to use SDK vs UI
Prompt management · Skill management · HITL
Live SDK examples — runnable samples repo

Evals

Prerequisites

Generate Insights (LLM judge)

Via SDK

Python

TypeScript

Python

TypeScript

Via UI

Open Insights

Switch to Sessions

Run judge

Agent hooks that ingest

Python — Agno post-hook

Python — LangChain / LangGraph callbacks

Python — OpenAI SDK wrapper

TypeScript — OpenAI Agents wrapOpenAIAgentsRun

TypeScript — Vercel AI SDK onFinish

Trigger Session Evals

Via SDK

Python

TypeScript

Via UI

Open Session Evals

Pick metrics

Run eval

What to do with results

Related

TypeScript — OpenAI Agents `wrapOpenAIAgentsRun`

TypeScript — Vercel AI SDK `onFinish`