Evals Overview

Purpose and Scope

Evals are eve's repeatable, scored checks for agent behavior. An eval runs an agent against real sessions, records what happened, and grades the result so a prompt, tool, model, or integration change does not silently break expected behavior. This is intentionally broader than a unit test for one helper. The runner boots or targets a real agent server, drives sessions through the TypeScript client protocol, and grades the responses that come back. A passing eval therefore proves the agent accepted a request through the same public surface users hit and produced the outcome the case asserted. Sources: docs/evals/overview.mdx

Use evals when a behavior is important enough to preserve across releases. A weather agent might need to call a weather tool for forecast questions, avoid that tool for greetings, and include the expected forecast text in the reply. A warehouse-analysis agent might need to choose a connection-backed query capability, summarize the result, and avoid exposing credential material. Because evals exercise sessions and turns, they are best for observable product behavior: the run completed, the right tool or connection operation happened, the reply matched expectations, or the agent parked cleanly for human input when approval was required. Sources: docs/evals/overview.mdx

This overview is for maintainers who are adding evaluation to an eve project for the first time and for reviewers who need to understand what an eval result actually proves. The core mental model is simple: write a small TypeScript program that talks to the agent like a user, then score the transcript, events, tool calls, and final output that the agent produced. That placement makes evals a release-safety layer between ordinary development feedback and production deployment, especially for agents whose behavior emerges from instructions, runtime tools, channels, connections, and model selection.

Relevant Source Files

• docs/evals/overview.mdx - Defines the reader-facing eval model, the eval directory convention, defineEval, evals.config.ts, reporter defaults, CLI precedence, and deterministic mockModel fixtures.
• docs/channels/overview.mdx - Explains the channel layer and the default eve HTTP channel, which matters because evals exercise the same HTTP session API used by frontend hooks, local tooling, SDK clients, and curl.
• docs/connections/overview.mdx - Explains MCP and OpenAPI connections, qualified connection tool names, credential handling, and app versus user authentication, which matters when evals validate external-system behavior.

Core Primitives

The first primitive is the eval file. eve discovers evals under the app-root evals directory, and files ending in .eval.ts are eval cases by default. The file path is the eval identity, so authors do not create a separate id or name field. A file such as evals/weather/brooklyn-forecast.eval.ts becomes the id weather/brooklyn-forecast, and directories become the natural grouping mechanism. This keeps evaluation filesystem-first, matching the rest of eve's authoring model where the project layout communicates intent and capabilities are found from predictable locations. Sources: docs/evals/overview.mdx

The second primitive is defineEval. Each eval centers on one asynchronous test function that receives a test object. That object is both the driver and the assertion recorder: it can send a message to the target agent, wait for the turn to settle, inspect the reply, and record expectations about the run. The overview example sends a Brooklyn weather question, requires the run to succeed, checks that get_weather was called, and verifies the final reply includes Sunny. Only test is required; optional description, judge, tags, metadata, timeoutMs, and reporters add context and control. Sources: docs/evals/overview.mdx

import { defineEval } from "eve/evals";
import { includes } from "eve/evals/expect";
 
export default defineEval({
  description: "Basic message and tool-usage coverage for the weather agent.",
  async test(t) {
    await t.send("What is the weather in Brooklyn?");
    t.succeeded();
    t.calledTool("get_weather");
    t.check(t.reply, includes("Sunny"));
  },
});

The third primitive is evals.config.ts. Every evals directory needs exactly one config file at its root, and that file declares defaults shared by evals in the tree. Everything in the documented config is optional. A deterministic suite can define an empty config, while a model-graded suite can set a default judge model and shared reporters such as Braintrust. Config-level reporters observe every eval in the run, so common reporting belongs there instead of being copied into each case. CLI flags for max concurrency and timeout, plus per-eval values, take precedence over config defaults. Sources: docs/evals/overview.mdx

import { defineEvalConfig } from "eve/evals";
import { Braintrust } from "eve/evals/reporters";
 
export default defineEvalConfig({
  judge: { model: "openai/gpt-5.4-mini" },
  reporters: [Braintrust({ projectName: "my-agent" })],
});

The fourth primitive is the deterministic fixture model. mockModel lets an eval fixture exercise eve's runtime without calling a model provider. A static fixture can return the same text every time, which is useful for local and CI runs that should not vary with provider availability or model sampling. A callback fixture can derive its response from eve's prompt view, including the last user message, all user messages, the user message count, available tools, and prior tool results. The callback may also return tool calls and usage, enabling deterministic coverage of tool loops and token accounting. Sources: docs/evals/overview.mdx

System-to-Code Mapping

At the project level, evals are consumers of the running agent rather than agent capabilities themselves. The documented tree places agent and evals as sibling directories under the app root, with evals.config.ts at the root of the eval tree and grouped cases underneath. That shape is important for maintainability. Agent files define instructions, tools, channels, connections, schedules, and runtime behavior; eval files launch or target that behavior from outside and grade what it does. The init template includes evals/**/*.ts in TypeScript configuration, so eval helpers and cases can type-check alongside the application code. Sources: docs/evals/overview.mdx

Channels explain the ingress side of that mapping. A channel is the edge adapter between a platform and the agent: it normalizes platform input into a user message, owns the continuation token used to resume a conversation on that surface, and decides how responses are delivered. The eve HTTP channel is enabled by default and is the route family used by the terminal UI, frontend hooks, SDK clients, and curl. Since evals exercise the same HTTP surface, a passing eval gives concrete evidence that the default session API is alive and that the agent can process traffic through the interface real applications use. Sources: docs/channels/overview.mdx

Connections explain the outbound capability side. A connection wires an agent into an external server the agent author does not implement, either through MCP or through an OpenAPI document. eve discovers the remote tools, surfaces matched capabilities to the model, and brokers authentication while keeping URLs and credentials out of model-visible context. The model discovers tools through connection_search and calls them by qualified names such as linear__list_issues. An eval for an integration-heavy agent can assert that the agent chooses the expected connection-backed capability, handles the returned information, and still avoids exposing secret or provider-specific details. Sources: docs/connections/overview.mdx

Because channels and connections sit on opposite sides of the agent boundary, good evals should name which boundary they are validating. An eval that sends a user message through the default HTTP channel and expects a connection tool call is testing both ingress and outbound capability selection. That does not replace provider contract tests for Slack, Linear, GitHub, a warehouse, or another external system. It does, however, verify the agent behavior that eve owns: the session was accepted, the model saw the right available capabilities, the run made the expected calls, and the final answer matched the product promise.

Authoring Workflow

A practical workflow starts with one smoke case. Create evals/evals.config.ts, then add a single .eval.ts file that sends a representative user message and records a few deterministic expectations. The first expectation is usually that the run succeeded. From there, add the smallest behavior-specific assertion that would catch the regression you care about: a required tool call, a forbidden tool call, an expected phrase in the reply, or a structured output match. The overview's weather example combines success, a tool-call assertion, and a reply-content check, which is a strong default pattern for tool-driven agents. Sources: docs/evals/overview.mdx

As the suite grows, organize by user-visible behavior rather than by implementation file. Since identity comes from the file path, directories become the vocabulary used in execution, reporting, and review. A weather directory can contain a Brooklyn forecast case, a no-tools-for-greetings case, and shared helper code in sibling TypeScript files that do not end in .eval.ts. A single eval file can also default-export an array to fan out over a dataset, which is useful when the same assertion pattern should run against many inputs. This keeps each case readable while allowing broad scenario coverage. Sources: docs/evals/overview.mdx

For bug reproduction, convert the smallest failing transcript into an eval before changing the agent. Send the same user prompt, assert the corrected behavior, and then make the prompt, tool, connection, or model change that causes the eval to pass. Keeping that case in the suite turns a production issue into a permanent regression guard. When the bug involves nondeterministic model behavior, start with deterministic assertions on events or tool calls before adding judge-based checks. When the bug involves runtime wiring, use mockModel to isolate session, channel, and tool behavior from provider variability. Sources: docs/evals/overview.mdx

Local, CI, and Deployment Signals

During local development, evals are most valuable after edits that change the agent's behavior surface: instructions, tool schemas, tool implementations, connection filters, channel authorization, model configuration, or output expectations. Fast deterministic evals can run repeatedly while an author iterates, while real-model or judge-based evals are better suited to higher-signal checkpoints. The command described by the docs is eve eval, and the overview notes that the runner can boot or target a real agent server. That gives developers a realistic loop without requiring them to manually replay conversations in the terminal UI every time. Sources: docs/evals/overview.mdx

In CI, evals occupy the agent-behavior layer of the validation stack. Conventional build, typecheck, unit, integration, scenario, and end-to-end checks can prove source correctness and fixture health, but evals answer a different question: did the agent still behave correctly through its session protocol? A common policy is to run deterministic evals on every pull request, run provider-backed or judge-graded evals on protected branches, and send reporter output to a shared destination for review. Braintrust is documented as a reporter option, and placing it in evals.config.ts ensures the whole run is observed consistently. Sources: docs/evals/overview.mdx

Before deployment, evals help answer an operational question: can the built agent still accept a request and produce the expected behavior through its public interface? This matters especially when a release changes route auth, channel selection, connection credentials, or external operation filters. For agents that depend on user-scoped connection auth, remember that eve can resolve a user token only when the active session has an authenticated user principal. Eval authors should therefore validate the externally visible contract without expecting the model to see URLs, bearer tokens, or other credential material, because the connections model keeps those out of conversation history. Sources: docs/connections/overview.mdx

Compact Reference

Next Steps

Start with one high-value eval that protects the agent's main promise. Give it a clear path under evals, drive the agent with one realistic message, and record assertions that would have failed for the most likely regression. Then add only the shared defaults you need in evals.config.ts: an empty config for deterministic checks, or judge and reporter settings when you are ready for model grading and external reporting. As coverage grows, keep directory names aligned with product behavior because those paths become the eval identities reviewers and operators will see. Sources: docs/evals/overview.mdx

After the first smoke case passes, add boundary coverage. If users reach the agent through a browser, SDK client, local tool, or curl, keep exercising the default eve HTTP channel. If the agent depends on Slack, Discord, GitHub, Linear, MCP, OpenAPI, or a warehouse, add behavior-level evals that verify the agent chooses the right capability and handles the result. Then move to the detailed eval references for cases, assertions, judges, reporters, and targets so the suite can grow from a smoke check into a release gate.