Running Evals, Reporters, and Targets

Purpose and Scope

This page explains the operational side of eve evals: how to run eval files, choose a target application, and export results for humans or automation. In eve, an eval is an end-to-end check that drives an agent through the same HTTP-facing runtime used by local development and deployments. The target is therefore not a mock object or an in-process function call; it is an HTTP URL that the runner can health-check, inspect, authenticate to, and exercise through sessions, channels, schedules, or webhooks. Sources: docs/evals/targets.mdx

Use this page after you understand how to author eval cases and assertions. The authoring APIs define what to test, while the runner decides where to test it, how much to run, what the process exit code means, and where evidence is stored. That separation is important for CI: the same eval files can run against the local dev server during development or against an existing deployment with eve eval --url <url>, so teams can reuse behavioral checks across laptops, preview environments, and production-like validation. Sources: docs/evals/targets.mdx

Relevant Source Files

• docs/evals/targets.mdx - Defines the target model for eval execution, including local versus remote URLs, target health and info checks, t.target helpers, schedule dispatch, session attachment, and authentication behavior for Vercel and explicit bearer tokens.

Running Evals from the CLI

The core command is eve eval. By default, it discovers eval files under evals/, starts a local development target, runs the selected cases, and prints a concise summary. You can run all discovered evals, pass positional ids such as weather or weather/smoke, list evals without executing them, filter by tag, or tune concurrency and timeouts. Positional ids are designed around filesystem organization: an id can match exactly or by directory prefix, which makes directory layout part of the eval selection interface.

eve eval
eve eval weather smoke
eve eval --tag fast
eve eval --list
eve eval --verbose
eve eval --json
eve eval --timeout 60000
eve eval --max-concurrency 4

For remote validation, pass --url. That changes the target from the local dev server that eve starts for you to an existing server or deployment. The important constraint is that an eval target is still always an HTTP URL, and the runner treats both local and remote execution through the same target abstraction. Before tests run, the runner polls /eve/v1/health, verifies /eve/v1/info, and then exposes the live target through t.target inside each eval's test function. Sources: docs/evals/targets.mdx

eve eval --url https://example.vercel.app
eve eval --url https://example.vercel.app --tag smoke --strict

Exit codes make the command safe for automation. A successful run exits with 0 when every non-skipped eval passes its hard gates, and soft thresholds also pass when --strict is enabled. A failing gate, execution error, or strict threshold miss exits with 1. Configuration errors exit with 2. An eval that calls t.skip(reason) is reported as skipped and does not count as pass or fail, so it should be used for intentional unavailability rather than flaky failure handling.

Selecting and Exercising Targets

The target abstraction is what lets evals cover more than a simple chat turn. Inside test(t), t.target represents the verified HTTP application under test. The documented helpers support authenticated fetches into the target, dev-only schedule dispatch, and attaching externally-created sessions back into the eval's assertion stream. This matters for agent systems because many important behaviors begin outside a direct t.send(...) call: a webhook can arrive through a channel, a recurring schedule can create a session, or an integration can produce events that the eval should still grade. Sources: docs/evals/targets.mdx

import { defineEval } from "eve/evals";
 
export default defineEval({
  async test(t) {
    const { sessionIds } = await t.target.dispatchSchedule("heartbeat");
    const session = await t.target.attachSession(sessionIds[0]!);
    session.succeeded();
    session.calledTool("send_report");
  },
});

Use t.target.fetch(path, init) when the test needs to hit target routes directly, especially channel ingress or webhook endpoints. The fetch uses the same authentication context as the session protocol, so a route exercised through this helper sees the same credentials the runner would use for the target. Use t.target.dispatchSchedule(id) when an eval should trigger a schedule through the dev-only schedule route. That helper works only when dev routes are enabled, which includes the local eve eval server and deployments intentionally running in development mode. Sources: docs/evals/targets.mdx

Use t.target.attachSession(sessionId, { startIndex? }) when a channel, webhook, or schedule creates a session outside the eval's primary conversation. Attachment consumes one turn from that existing session so its events participate in run-level assertions. The optional startIndex prevents replaying earlier stream events when the session is already partway through execution. Attached sessions are full EveEvalSession instances, so you can continue driving and asserting on that specific session while aggregate assertions on t still read across every attached session. Sources: docs/evals/targets.mdx

Authentication for Remote Targets

Local eval targets do not need outgoing auth because the runner owns the dev server it boots. Remote targets are different: eve eval --url <url> may be pointed at a Vercel deployment or at an arbitrary HTTP origin. For Vercel-aware targets, eve resolves the expected owner and project from VERCEL_ORG_ID and VERCEL_PROJECT_ID when both are set, otherwise it reads .vercel/project.json. It then resolves the exact HTTPS origin and sends ambient credentials only when the project identity matches. Sources: docs/evals/targets.mdx

After verification, eve sends available Vercel credentials to authenticated remote targets. The resolved OIDC token is sent as both the bearer token and the Vercel trusted-IDP header, and VERCEL_AUTOMATION_BYPASS_SECRET is sent as the Protection Bypass for Automation header when configured. For non-Vercel authorization, EVE_EVAL_AUTH_TOKEN provides an explicit bearer override. Credential-bearing clients do not follow redirects, which prevents sensitive headers from being forwarded to a different origin. Sources: docs/evals/targets.mdx

Reporters and Artifacts

The console summary is the default reporter surface: it keeps routine output short and points failures toward richer artifacts. The runner also writes per-run evidence under .eve/evals/<timestamp>/, including summary and result indexes plus per-eval assertion results, verdicts, event streams, and t.log output. Use --verbose when local debugging benefits from streaming per-eval log lines to stdout, and use --json when another tool needs machine-readable output rather than human-oriented terminal text.

eve eval --junit .eve/junit.xml
eve eval --json
eve eval --verbose
eve eval --skip-report

JUnit output is intended for CI systems that already understand test reports. Braintrust reporting is configured through eval reporters rather than by outsourcing grading: eve still runs the evals and computes assertion results, while reporters ship those results elsewhere. A shared reporter normally belongs in evals/evals.config.ts so it observes every eval in the run. A per-eval reporter belongs on an individual defineEval(...) when only that case, or a small group sharing the same reporter instance, should export to a destination.

import { defineEvalConfig } from "eve/evals";
import { Braintrust } from "eve/evals/reporters";
 
export default defineEvalConfig({
  reporters: [Braintrust({ projectName: "weather-agent" })],
});

Be intentional about exported data. Reporters can carry eval metadata, assertion results, gate outcomes, soft scores, prompts, outputs, or other captured behavior depending on the destination. That makes them useful for experiment tracking and regression review, but it also means the team running the evals is responsible for making sure the observability or eval provider is approved for the data being sent. Use --skip-report when a local or CI run should execute and score normally without invoking configured external reporters.

Compact Reference

Recommended Workflow

Start locally with broad discovery: run eve eval, inspect the console summary, and open artifacts only when a case fails or needs deeper debugging. Once the suite grows, organize evals by directory so positional prefixes map to product areas such as weather, channels, or schedules. Add tags for speed tiers like fast and nightly, then use --max-concurrency and --timeout to keep local feedback predictable while preserving enough time for multi-turn or integration-heavy cases.

For CI, prefer the same eval files against a deployed or preview URL. Configure Vercel project identity or an explicit EVE_EVAL_AUTH_TOKEN before using --url, then emit JUnit XML for the CI test tab and enable external reporters only for approved destinations. When an eval needs to validate schedules, channel ingress, or webhook behavior, reach for t.target.dispatchSchedule, t.target.fetch, and t.target.attachSession instead of rewriting the behavior as a synthetic chat-only test. Sources: docs/evals/targets.mdx

What to Read Next

Read eval-cases-assertions-and-judges to design high-quality eval files before running them at scale. Read schedules when using t.target.dispatchSchedule, and read channels-overview when using t.target.fetch to exercise ingress paths. For deployment-specific CI flows, continue to deployment and auth-and-route-protection so remote eval authentication matches the way your application is protected.