Evals

Purpose and Scope

Evals are the repeatable safety net for changes to agent behavior. A Flue agent can produce different output when you revise its instructions, change its model, add or remove tools, or alter the surrounding application logic. Instead of relying on one-off manual prompts, an eval suite gives those changes a stable set of scenarios and expectations. The goal is not to freeze all natural-language output, but to decide whether a new behavior is acceptable before it reaches users or an automated workflow.

Sources: apps/docs/src/content/docs/guide/evals.md

Flue deliberately does not embed an evaluation framework inside the runtime. The guide recommends Sentry's vitest-evals, which builds on Vitest and lets teams keep evals close to ordinary TypeScript tests. This is an important boundary: Flue supplies the agent, workflow, HTTP surface, SDK client, and runtime behavior, while vitest-evals supplies eval harnesses, judges, normalized reports, and reporting workflows. That separation lets application teams choose assertion style, CI policy, and report retention without coupling eval logic to Flue internals.

Sources: apps/docs/src/content/docs/guide/evals.md

Relevant Source Files

• apps/docs/src/content/docs/guide/evals.md — First-party guide for evaluating Flue agents with vitest-evals, including the blueprint command, harness behavior, example eval case, execution commands, and reporting flow.

Core Evaluation Primitives

The first primitive is the tooling blueprint. The guide tells users to run flue add tooling vitest-evals, which adds the eval dependencies, a dedicated configuration, a reusable Flue harness, and an application-specific starter case. Treat the generated files as the beginning of your evaluation surface rather than as a finished test strategy. The blueprint gives you the HTTP-based shape; your team still chooses which agents or workflows should be exposed, which credentials are safe to use, and which behaviors deserve gates.

Sources: apps/docs/src/content/docs/guide/evals.md

The second primitive is createFlueAgentHarness(...). The generated harness prompts an HTTP-exposed agent through @flue/sdk and converts what happens into the normalized result format expected by vitest-evals. That normalized result includes the response text, model usage, costs, and tool calls. Each eval case receives a fresh agent instance, so persisted conversation history from one case cannot influence another case. For workflow evaluation, the guide recommends creating a harness around client.workflows.invoke(...) and returning the workflow result as the eval output.

Sources: apps/docs/src/content/docs/guide/evals.md

The third primitive is the assertion or judge. Some requirements are deterministic enough for direct Vitest assertions: an answer contains a required phrase, a specific tool was called, or token usage was reported. Other requirements, such as clarity, faithfulness, or summary quality, need scored evaluation instead of exact matching. The guide points readers to vitest-evals judges and toSatisfyJudge(...) for those cases, with the important caution that the grading model should be independent from the model being evaluated.

Sources: apps/docs/src/content/docs/guide/evals.md

Write an Eval

A useful eval starts with a narrow behavior requirement. The guide's example evaluates a service-status agent by asking whether the checkout service is operational, then checking three separate signals: the final output mentions operational status, the agent used the get_service_status tool, and the run reported token usage. This pattern is valuable because it verifies both the user-visible answer and the process that produced it. For agentic systems, tool use can be as important as text quality because it proves the agent consulted live application state.

Sources: apps/docs/src/content/docs/guide/evals.md

import { expect } from 'vitest';
import { describeEval, toolCalls } from 'vitest-evals';
import { createFlueAgentHarness } from './harness.ts';
 
const harness = createFlueAgentHarness({ agentName: 'service-status' });
 
describeEval('Flue service status agent', { harness }, (it) => {
  it('checks live service status before answering', async ({ run }) => {
    const result = await run('Is the checkout service currently operational?');
 
    expect(result.output).toContain('operational');
    expect(toolCalls(result).map((call) => call.name)).toContain('get_service_status');
    expect(result.usage.totalTokens).toBeGreaterThan(0);
  });
});

Use describeEval(...) to bind the harness to the suite and call run(...) explicitly inside each case. That explicit call makes the prompt and normalized result easy to inspect, and it keeps ordinary Vitest assertions available for exact requirements. toolCalls(result) is useful when the behavior depends on application capabilities, because it lets the test assert that the agent used the intended tool rather than guessing. For workflows, mirror the same shape but make the harness output the workflow's returned value.

Sources: apps/docs/src/content/docs/guide/evals.md

Run and Report Evals

Run evals against the same public boundary that the application exposes. Start the Flue application in one terminal with pnpm exec flue dev, then run the eval suite in another terminal with pnpm run evals. The application process must have its normal model-provider credentials, because the eval is exercising real agent or workflow behavior. If an assertion or gated judge fails, the eval command exits non-zero, which makes the same command suitable for CI gates.

Sources: apps/docs/src/content/docs/guide/evals.md

pnpm exec flue dev
pnpm run evals

For deployed or protected targets, configure the SDK client rather than bypassing authentication. The guide notes that protected deployments should provide the required token or headers, and the ecosystem tooling docs describe using FLUE_BASE_URL for local versus deployed applications. This preserves the production-facing contract: the eval prompts the same HTTP-exposed agent or workflow that a real client would reach. Never treat eval credentials as sample data; they can expose prompts, tool arguments, model usage, and application-specific metadata in reports.

Sources: apps/docs/src/content/docs/guide/evals.md

The reporting path is designed for both local inspection and automation. The guide documents pnpm run evals:json as the command that saves normalized runs into vitest-results.json. That artifact can be opened locally with vitest-evals serve vitest-results.json or passed into GitHub reporting. Because reports may include prompts, outputs, tool calls, errors, usage, costs, and metadata, review retention and access policies before uploading them from CI. Use detailed reports when diagnosing regressions and compact output when guarding pull requests.

Sources: apps/docs/src/content/docs/guide/evals.md

System-to-Code Mapping

In the Flue docs spine, evals sit between authoring and operating agents. Building-agent guides explain how an agent receives models, instructions, tools, skills, and route exposure; the eval guide explains how to exercise that exposed behavior repeatedly. The recommended harness uses @flue/sdk, so evals do not import runtime internals or test private implementation details. This keeps the suite focused on the public behavior users and integrations observe, while still preserving process-level evidence such as tool calls and usage when vitest-evals normalizes the run.

Sources: apps/docs/src/content/docs/guide/evals.md

Practical Next Steps

Start by adding the tooling blueprint, then keep the first suite small. Choose one agent route or one workflow, write two or three cases that represent real user or system tasks, and assert outcomes that are concrete enough to fail usefully. Add judges only after direct assertions are not expressive enough. Once the suite is stable, run it before model upgrades, instruction rewrites, tool changes, and release branches. For deeper context, read the pages on Building Agents, Workflows, Tools, CLI Add, and the Vitest Evals tooling integration.