Cases, Assertions, and Judges

Purpose and Scope

Evals in eve are executable checks for agent behavior. A case drives the agent, assertions record what should be true, and the runner computes a verdict from all recorded assertion results. This page focuses on the authoring model used by eval files: write one async test(t) function, send one or more turns through the target agent, then record deterministic or model-graded expectations inline. That structure keeps the test script close to the behavior being measured, which is especially useful for durable agents where tool calls, sessions, human-in-the-loop pauses, and streamed events all matter.

Sources: docs/evals/assertions.mdx

The key distinction is between deterministic assertions and judge assertions. Deterministic assertions are the first choice when the expected behavior can be checked from the run record: the run succeeded, a tool was called exactly once, no tools were used, structured output matches a schema, or events occurred in order. Judge assertions are reserved for subjective or semantic quality checks, such as factuality, summary quality, or whether a free-form answer satisfies a rubric. This split lets evals stay reliable where possible while still measuring qualities that are hard to express with exact matching.

Sources: docs/evals/assertions.mdx

Relevant Source Files

• docs/evals/assertions.mdx - Defines the first-party assertions documentation, including scoped assertions, value checks, matcher language, severity behavior, and how run, session, turn, tool, subagent, and event assertions are intended to be used.

Case Authoring Model

An eval case is normally a single .eval.ts file under evals/. Each file exports defineEval({ async test(t) { ... } }), and the runner executes that function against the configured target. The test context t is both the driver and the assertion recorder: it can send turns, create or attach sessions, inspect replies, and record assertions. Because the function is regular TypeScript, intermediate turns can be stored in local variables, compared later, or passed into judge calls before a later turn overwrites the final reply.

Before writing cases, add evals/evals.config.ts at the root of the eval tree. The config can be empty when no shared judge model, reporter, concurrency, or timeout behavior is needed. The official docs use defineEvalConfig({}) as the minimal starting point. From there, individual eval files can be grouped by directories, where path identity becomes the eval identity; for example, evals/weather/brooklyn-forecast.eval.ts is run as weather/brooklyn-forecast, and eve eval weather selects the whole directory group.

import { defineEvalConfig } from "eve/evals";
 
export default defineEvalConfig({});

A single-turn case sends one prompt and checks the settled result. t.send(input) waits until the turn settles, and t.reply refers to the last assistant message. Use this shape for smoke tests, regression checks, and focused behavior assertions. A text expectation can be written with t.check(t.reply, includes("Sunny")), while behavior-only evals can skip text and assert that the run succeeded or that a specific tool was not called.

import { defineEval } from "eve/evals";
import { includes } from "eve/evals/expect";
 
export default defineEval({
  async test(t) {
    await t.send("What is the weather in Brooklyn?");
    t.succeeded();
    t.check(t.reply, includes("Sunny"));
  },
});

Multi-turn cases are for stateful behavior. They can verify memory across turns, approval flows, structured output, attachments, or branching actions. A turn returned from t.send() is an immutable object that can be asserted immediately, which prevents later conversation state from hiding the result being checked. When control flow depends on a value-level check, use t.require(...) instead of a soft observation; it records the failure and stops the script so later steps do not run on invalid assumptions.

Deterministic Assertion Surfaces

Assertions record results instead of throwing on the first failure. That design makes an eval report more useful because one run can show every failing expectation. The assertions documentation identifies two deterministic surfaces: scoped methods and t.check for grading a specific value. Scoped assertions inspect a natural object: t for the whole run, a session for a session snapshot, or a turn for an immutable response. A scope is considered parked when it pauses on unanswered human-in-the-loop input.

Sources: docs/evals/assertions.mdx

The most common run-level assertion is t.succeeded(). It accepts a closed session or a healthy session that remains open for another user message, but it rejects protocol failures and unanswered human-in-the-loop pauses. Use t.parked() when the expected outcome is a clean HITL park. For text checks, t.messageIncludes(token) checks joined assistant text for a string or regular expression. For structured results, turn and session assertions such as outputEquals(value) and outputMatches(schema) check deep equality or Standard Schema validation where the output is unambiguous.

Sources: docs/evals/assertions.mdx

Tool and action assertions are central for agent evals because they measure behavior that may matter more than wording. t.calledTool(name, opts?) checks for a completed tool call and can constrain input, output, status, or count. t.loadedSkill(skill, opts?) is documented as sugar for a load_skill tool call. Negative and budget-style checks include t.notCalledTool(name), t.usedNoTools(), t.maxToolCalls(n), and t.noFailedActions(). These expectations are useful for preventing unnecessary side effects, enforcing tool discipline, and catching regressions in capability routing.

Sources: docs/evals/assertions.mdx

Event and delegation assertions cover more complex runs. t.calledSubagent(name, opts?) verifies that delegation happened, with constraints such as remoteUrl or output. t.event(type, opts?) and t.notEvent(type, opts?) check typed event presence, data, and count, while t.eventOrder([...matchers]) verifies event sequencing. For cases that need custom inspection, t.eventsSatisfy(label, predicate) is the escape hatch over the typed event stream. Use the narrow built-in assertions first, then fall back to predicates only when the event relationship cannot be expressed directly.

Sources: docs/evals/assertions.mdx

Value Checks and Severity

Use t.check(value, matcher) when the expected result is about a specific value rather than the whole run. Official examples pair t.check with matchers from eve/evals/expect, such as includes for substring-like checks or equals for exact value comparison. This style is a good fit for final replies, intermediate draft messages, parsed outputs, IDs, and session continuity checks. It also keeps the assertion attached to the value that motivated it, instead of forcing all checks into broad run-level methods.

Severity determines whether an assertion gates the verdict or simply records a score. The assertions source describes scoped assertions as gate-by-default, which means failures affect the eval verdict unless explicitly softened by the assertion API. Judge assertions are different: the official judge docs describe them as soft by default, with thresholds attached to the returned assertion handle. This means deterministic checks should normally define correctness boundaries, while judge scores can start as tracked metrics and later become gates after the team trusts their signal.

Sources: docs/evals/assertions.mdx

Judge-Based Expectations

A judge assertion uses a separate model to score output when deterministic checks cannot capture quality. The judge model is not the agent under test; it is resolved separately and used only for scoring. This matters because changing the judge should not silently change the target agent behavior. Use judges for factual correctness, summary quality, SQL semantic equivalence, or broad criteria like professional tone. Avoid them for simple checks such as whether a tool ran, a reply included a token, or structured output matched a schema.

The official judge namespace is t.judge.autoevals, reflecting the Braintrust autoevals grader family. Available graders include factuality(expected), summarizes(expected), closedQA(criteria), and sql(expected). Each grader scores t.reply by default, but accepts an options object where on can point at an intermediate value such as draft.message. Per-call model and modelOptions can override the judge configuration when one eval needs a different scoring model from the project default.

import { defineEval } from "eve/evals";
 
export default defineEval({
  async test(t) {
    await t.send("Explain quantum tunneling to a 10-year-old.");
    t.succeeded();
    t.judge.autoevals.closedQA("uses no math beyond arithmetic").atLeast(0.8);
  },
});

Judge thresholds live on the returned handle. With no threshold, the score is tracked in reports and artifacts but does not fail the eval. .atLeast(threshold) sets a soft bar; below-threshold results are marked as scored and become fatal when running in strict mode. .gate(threshold) promotes the judge to a hard gate that fails the eval outright. Because judge calls burn tokens and run asynchronously during finalization, treat them as targeted instruments rather than the default way to assert behavior.

Practical Authoring Flow

Start every eval by naming the behavior you want to protect, then choose the least subjective assertion that proves it. For a weather agent, a deterministic case might send a Brooklyn forecast request, assert t.succeeded(), require t.calledTool("get_weather"), and check that the reply includes the expected mocked condition. For a greeting case, the strongest assertion might be t.notCalledTool("get_weather"), because the important behavior is avoiding an unnecessary external action rather than producing one exact phrase.

For richer workflows, write the eval as a miniature user journey. Send the first prompt, assert the immediate turn, store any important message or session ID, then continue. If the agent drafts before sending, judge the draft before the next turn changes t.reply. If the agent should ask for approval, use parked-session assertions to verify the HITL pause. If the agent should delegate or emit typed events, combine subagent and event-order assertions so the eval checks both final outcome and operational trace.

Compact Reference

Next Steps

After drafting a case, run it locally with the eval runner and inspect every recorded assertion rather than only the final verdict. Tighten deterministic checks before adding judges, and make judge thresholds soft until they have proven stable for your task. Organize cases by directory so targeted commands can run one capability at a time, then add project-level judge configuration only when several evals share the same scoring model. For broader context, read the evals overview, running evals and reporters, and the TypeScript API reference for exported eval helpers.