Evals

Purpose and Scope

Evals are feedback loops that let an agentic application inspect model behavior, judge whether a task is complete, and route the next action. In this repository’s public documentation, that pattern is built from AI SDK Core rather than a separate evaluator-only API. The important idea is that generation calls already expose the artifacts an evaluator needs: text, tool calls, tool results, structured output, usage, warnings, step details, and final-step metadata. An eval can therefore be implemented as a normal model call, a tool-backed loop, or a structured classifier that scores previous output and decides what to do next.

Sources: content/docs/03-ai-sdk-core/01-overview.mdx, content/docs/03-ai-sdk-core/05-generating-text.mdx

The reader problem is usually not “how do I run a benchmark” but “how do I make an agent improve or stop safely while it is working.” AI SDK agents are described in the first-party docs as language models that use tools in a loop, with stopping conditions and context management controlling progress. The Core docs supply the lower-level mechanics for that loop: generation, streaming, tools, structured outputs, and multi-step execution. Treat evals as another participant in the loop: they can be a prompt, a schema-validated judgment, a tool result, or a follow-up model step.

Sources: content/docs/03-ai-sdk-core/15-tools-and-tool-calling.mdx, content/docs/03-ai-sdk-core/10-generating-structured-data.mdx

Relevant Source Files

• content/docs/03-ai-sdk-core/01-overview.mdx - Defines AI SDK Core as the standardized layer for text generation, structured data generation, and tool usage, which are the building blocks used by evaluator and feedback-loop patterns.
• content/docs/03-ai-sdk-core/05-generating-text.mdx - Documents result metadata from generation, including all-step content, tool calls, tool results, usage, warnings, steps, final step details, and performance fields that evaluators can inspect.
• content/docs/03-ai-sdk-core/10-generating-structured-data.mdx - Shows how structured output is generated and validated with schemas, making it the natural mechanism for typed evaluator decisions and scoring payloads.
• content/docs/03-ai-sdk-core/15-tools-and-tool-calling.mdx - Defines tools, schemas, execution, strict mode, dynamic descriptions, and multi-step stopping behavior used by agents that evaluate and act repeatedly.
• content/docs/03-ai-sdk-core/16-mcp-tools.mdx - Explains MCP clients and transports, which allow feedback loops to call external tools, services, resources, and prompts through a standardized interface.
• content/cookbook/01-next/72-call-tools-multiple-steps.mdx - Provides an end-to-end Next.js example where a client streams a conversation and a server allows dependent tools to run over multiple steps.

Core Primitives

The foundation for an eval loop is a generation call. The Core overview names generateText for non-interactive automation and agents that use tools, and streamText for interactive experiences such as chatbots and content streaming. The generating-text guide then describes the result object in a way that is directly useful for evaluation: the final text is only one field. A feedback loop can also examine all generated content, every tool call and result, warnings from providers, usage across steps, and per-step performance. That makes evaluation observable without changing the public generation surface.

Sources: content/docs/03-ai-sdk-core/01-overview.mdx, content/docs/03-ai-sdk-core/05-generating-text.mdx

Structured output turns an evaluator from prose into a typed contract. The structured-data guide explains that the AI SDK standardizes object generation through the output property on generation and streaming calls, with schemas supplied through Zod, Valibot, or JSON Schema. For evals, that means a judge can return a validated shape such as a score, a pass or fail decision, a reason, and suggested next action. The docs also note that structured output is part of the generation flow and can be combined with tool calling, so evaluator decisions can sit inside the same task loop as the agent’s work.

Sources: content/docs/03-ai-sdk-core/10-generating-structured-data.mdx

Tools are the action side of the feedback loop. The tool-calling guide defines a tool as an object the model can call for a specific task, with a description, input schema, optional execute function, and optional strict behavior. In an eval design, tools can fetch ground truth, run code, call an internal policy service, query a retrieval system, or store the result of a scoring pass. Because input schemas are consumed by the model and used for validation, tools also create a boundary between model reasoning and trusted application behavior.

Sources: content/docs/03-ai-sdk-core/15-tools-and-tool-calling.mdx

System-to-Code Mapping

A practical evaluator maps cleanly to the source-backed Core concepts. The task producer uses generateText or streamText to produce an answer, plan, or tool-using attempt. The evaluator then uses another generation call with structured output to classify the attempt, or it uses tool results and step metadata already returned by the first call. If the evaluator says the task is incomplete, the application can continue the loop with additional context, a different prompt, or a tool call. If it says the answer passes, the application returns the final text or structured payload to the caller.

Sources: content/docs/03-ai-sdk-core/05-generating-text.mdx, content/docs/03-ai-sdk-core/10-generating-structured-data.mdx, content/docs/03-ai-sdk-core/15-tools-and-tool-calling.mdx

Execution Flow

A common eval flow starts with a user request and a generation call. The model may answer directly, call a tool, or produce structured output depending on the configuration. The application records the response fields that matter for evaluation: final text, tool calls, tool results, source references where available, finish reason, provider warnings, and step performance. Next, a judge prompt or structured output schema evaluates whether the result satisfies the task. The application then either returns the answer, asks for a repair, calls a tool for more evidence, or starts another bounded step.

Sources: content/docs/03-ai-sdk-core/05-generating-text.mdx, content/docs/03-ai-sdk-core/10-generating-structured-data.mdx

The multi-step cookbook is the clearest task-flow example in the supplied evidence. A React client uses a chat hook with a transport pointed at an API route, and the server route calls streamText with tools for location and weather. The docs explain that dependent tools can execute in sequence during the same generation flow, and the server uses a stopping condition to allow multiple consecutive tool calls. An evaluator loop follows the same shape: each step may gather information, judge progress, and decide whether another step is justified.

Sources: content/cookbook/01-next/72-call-tools-multiple-steps.mdx

import { generateText, Output, isStepCount, tool } from 'ai';
import { z } from 'zod';
 
const evaluation = await generateText({
  model,
  output: Output.object({
    schema: z.object({
      passed: z.boolean(),
      score: z.number().min(0).max(1),
      reason: z.string(),
      nextAction: z.enum(['return', 'revise', 'call-tool']),
    }),
  }),
  prompt: `Evaluate this answer against the task: ${answer}`,
});

API Components and Configuration

The most important configuration choice for evals is the boundary between model judgment and application execution. A structured judge is appropriate when the decision is semantic, such as checking whether a summary follows instructions. A tool is better when the decision depends on trusted computation, policy, permissions, or external data. The tool-calling docs also describe dynamic descriptions that can depend on tool context and an experimental sandbox, which matters when an evaluator should expose different capabilities per tenant, project, environment, or workspace.

Sources: content/docs/03-ai-sdk-core/15-tools-and-tool-calling.mdx

When an eval loop needs tools outside the local process, MCP provides the integration surface. The MCP docs describe creating a client with HTTP transport for production, alternatives such as server-sent events, and stdio for local development only. They also explain session reattachment for Streamable HTTP sessions and the behavior of cached initialize metadata. For evaluator systems, MCP is useful when grading requires standardized access to external services, resources, or prompts without packaging every capability as a local function tool.

Sources: content/docs/03-ai-sdk-core/16-mcp-tools.mdx

Implementation Guidance

Keep eval loops bounded and observable. The tool-calling and cookbook examples both emphasize stopping conditions for multi-step behavior; without them, a feedback loop can keep asking for more work instead of returning a result. Account for structured output as a step when combining it with tools, because the structured-data docs explicitly call out its place in the multi-turn execution model. Store enough metadata to debug failures: warnings, finish reasons, usage, tool execution timing, response timing, and the final-step details are all exposed by the generation result.

Sources: content/docs/03-ai-sdk-core/05-generating-text.mdx, content/docs/03-ai-sdk-core/10-generating-structured-data.mdx, content/docs/03-ai-sdk-core/15-tools-and-tool-calling.mdx

A good next step is to decide which evaluator shape your agent needs. Use a structured output judge for scoring and routing, a tool-backed evaluator for deterministic checks, MCP when external systems should provide capabilities, and streaming when the user should see progress while the loop runs. Then read the adjacent pages on tools, loop control, runtime context, MCP tools, and structured generation. Those pages explain the same primitives from the angle of execution, state boundaries, external integrations, and typed model output.