Evals

Purpose and Scope

Evals are the repeatable checks used to decide whether a model, prompt, or agent is producing acceptable output. In the product documentation, evaluation is framed around comparing outputs with ground-truth data, applying metrics such as relevance, coherence, similarity, or F1 score, and optionally using a judging model. In this repository, the most direct implementation evidence is the Copilot testing service that asks a chat endpoint to judge a response against criteria and returns a pass or failure message. That makes evals both a user workflow and a developer-facing primitive for validating AI behavior before it is trusted in an editor scenario.

Sources: extensions/copilot/src/extension/testing/node/aiEvaluationService.tsx

The important distinction is that an eval is not just another chat request. A normal chat request tries to help a user complete work, while an eval request measures the quality of a response against an explicit standard. The testing service encodes this by taking a response string, a criteria string, and a cancellation token, then rendering an examiner prompt. The examiner is instructed to think through the response and end with a clear PASS or FAIL line. The implementation then parses only that final decision line into a compact result that tests can assert on.

Sources: extensions/copilot/src/extension/testing/node/aiEvaluationService.tsx

Core Primitives

The repository exposes three practical primitives around evaluation: the candidate response, the criteria, and the evaluator endpoint. The candidate response is the AI output being judged. The criteria is a natural-language contract describing what the response must satisfy. The evaluator endpoint is obtained from the endpoint provider as copilot-utility-small, which keeps the evaluation flow separate from the feature that produced the original answer. Prompt rendering uses prompt-tsx components with a system message and user payload, so the evaluator itself is structured as a reusable prompt element rather than ad hoc string concatenation.

Sources: extensions/copilot/src/extension/testing/node/aiEvaluationService.tsx

Before evaluating or even processing a document, Copilot code paths can apply repository policy. The document validation helper asks the ignore service whether a parsed document URI is blocked for Copilot. If policy blocks it, the helper returns an invalid status with the reason that the document is blocked by repository policy; otherwise it returns valid. That check matters for eval design because a high-quality eval should not train, judge, or assert behavior over content that the workspace policy excludes. Eligibility checks are therefore part of the evaluation boundary, not a cosmetic preflight step.

Sources: extensions/copilot/src/extension/completions-core/vscode-node/lib/src/util/documentEvaluation.ts

Code references add another boundary around AI evaluation. The CodeReference class registers a token listener outside test runtime, reads whether public code references are enabled, disposes engagement tracking when they are disabled, and creates a CodeRefEngagementTracker when they are enabled. Evals that involve generated code or code suggestions need to respect this feature boundary because a response can be functionally correct while still requiring policy-aware handling of public code references. The source shows enablement is driven by Copilot token state, so tests and evaluation fixtures should not assume the feature is always active.

Sources: extensions/copilot/src/extension/completions-core/vscode-node/extension/src/codeReferencing/index.ts

Relevant Source Files

• extensions/copilot/src/extension/testing/node/aiEvaluationService.tsx - Defines IAIEvaluationService, AIEvaluationService, EvaluationResult, and EvaluationPrompt for model-judged test evaluation of AI responses.
• extensions/copilot/src/extension/completions-core/vscode-node/lib/src/util/documentEvaluation.ts - Checks whether a document URI is valid for Copilot processing or blocked by repository policy.
• extensions/copilot/src/extension/completions-core/vscode-node/extension/src/codeReferencing/index.ts - Enables or disables public code reference tracking from Copilot token state and runtime mode.
• src/vs/editor/contrib/codeAction/common/types.ts - Defines code action kinds, trigger sources, auto-apply choices, and filtering helpers that shape editor action behavior an eval may inspect.
• cli/src/bin/code/main.rs - Routes integrated and standalone command-line invocations, including agent subcommands used to host, inspect, stop, kill, or view logs for agents.
• src/vs/code/electron-main/main.ts - Starts the Electron main process and wires foundational application services that host the desktop workbench where evaluation-backed features run.

System-to-Code Mapping

The official evaluation workflow asks users to run prompts or agents against a dataset, rate responses, add evaluators, choose a judging model when needed, and run an evaluation job. The repository test service represents the same concept in a minimal programmable form: tests provide one response and one criterion, the service asks a judging endpoint to examine them, and the result is normalized to either success or an error message. That lower-level contract is intentionally small, which makes it useful for smoke and integration tests that need a yes-or-no quality signal rather than a full metrics dashboard.

Sources: extensions/copilot/src/extension/testing/node/aiEvaluationService.tsx

Editor actions supply another evaluable surface. The code action common types define quick fixes, refactor categories, notebook actions, source actions, organize imports, fix all, and surround-with refactors. They also define trigger sources such as lightbulb, problems view, on save, and quick-fix hover. Filtering helpers decide whether a provided action kind may be included, whether source actions require explicit inclusion, and whether only preferred actions should pass. Agent evals that verify edit behavior can use these categories as observable expectations: the right action kind, trigger path, and preferred status are often as important as the text edit itself.

Sources: src/vs/editor/contrib/codeAction/common/types.ts

Runtime entrypoints matter because evals are often executed from automation rather than by a person clicking through the interface. The Rust command-line main parses legacy, integrated, and standalone forms, builds a command context with HTTP client, launcher paths, logging, and arguments, and then dispatches commands. The agent command family includes process listing, hosting, stopping, killing, and log access. For agent evaluation, these commands are the operational surface for starting a host, observing its behavior, and collecting logs when a judged response fails or an autonomous workflow becomes stuck.

Sources: cli/src/bin/code/main.rs

API Components

Component	Contract	Behavior
IAIEvaluationService	evaluate(response, criteria, token)	Returns an EvaluationResult or throws when the evaluator request cannot be completed.
EvaluationResult	errorMessage optional	Empty result means pass; errorMessage explains a failure.
EvaluationPrompt	response plus criteria props	Renders examiner instructions and user content for the judging endpoint.
isDocumentValid	accessor plus text document identifier	Returns valid or invalid with a policy reason.
CodeActionFilter	include, excludes, includeSourceActions, onlyIncludePreferredActions	Controls which editor actions are considered by action collection.

The parse behavior is deliberately strict. After the judging model replies, the service scans from the bottom of the response and accepts a line that starts with PASS as success or a line that starts with FAIL as failure. A failure line is trimmed and returned as the error message. If neither marker appears, the service throws because the evaluator did not follow the required protocol. This is a useful pattern for repository tests: allow the model to reason in prose, but make the final machine-readable verdict narrow, stable, and easy to report in test output.

Sources: extensions/copilot/src/extension/testing/node/aiEvaluationService.tsx

Execution Flow

A typical repository-backed eval begins by producing or capturing the candidate answer from an AI feature. The test or harness states the acceptance criteria in plain language, then calls the evaluation service with a cancellation token so the work can be interrupted. The service chooses the utility judging endpoint, renders the prompt, sends a chat request named testEvaluation at a non-chat feature location, and parses the reply. If the result contains an error message, the calling test can fail with the model-provided explanation; otherwise it can continue to additional assertions such as edits, code actions, or policy checks.

Sources: extensions/copilot/src/extension/testing/node/aiEvaluationService.tsx, src/vs/editor/contrib/codeAction/common/types.ts

For desktop features, the Electron main process is the outer application host rather than the evaluator itself. Its imports and startup responsibilities include command-line parsing, configuration, file service, diagnostics, lifecycle, logging, product information, IPC, and the CodeApplication. That context is useful when interpreting evaluation failures from full VS Code runs: a failed agent or prompt eval may be caused by endpoint behavior, prompt quality, workspace policy, or by the application environment that launched the workbench. Keep those layers separate when designing tests and when deciding where to collect logs.

Sources: src/vs/code/electron-main/main.ts, cli/src/bin/code/main.rs

Next Steps

Use the product evaluation workflow when you need dataset management, evaluator selection, version history, and comparison across prompt or agent versions. Use the repository evaluation service when you need a small programmable quality gate inside tests. When adding a new eval, write the acceptance criteria as an observable contract, check document eligibility before processing workspace content, and include logs or agent process information when the eval depends on a hosted agent. For adjacent concepts, continue with the pages on Copilot and AI Overview, Chat Tools and Approvals, Custom Agents, Skills, and Prompts, and AI Troubleshooting.