Evals reference
Purpose and Scope
This page is a reference for the Evals surface in the OpenAI TypeScript and JavaScript SDK. Evals are structured tests for model or application behavior: you define what should be measured, run that definition against representative inputs, and inspect the run results. In agent and application workflows, this is the repeatable stage after ad hoc debugging, because it lets teams compare prompts, model choices, tool behavior, and regressions over time. The SDK exposes this surface as generated resource classes and typed request and response objects, so a TypeScript project can create, list, update, run, cancel, and inspect evaluations without constructing raw REST requests by hand.
The repository evidence shows three layers that matter to SDK users. The top-level Evals barrel exports the evaluation resource and its public type names. The nested Runs barrel exports run operations and the output item subresource. The generated API reference is the canonical inventory for the full SDK surface, while the generated tests demonstrate call shape, argument ordering, request options, and response wrapper behavior. Together, those files establish that Evals are not a standalone helper pattern; they are part of the normal generated client resource tree under the OpenAI client.
Sources: api.md, src/resources/evals/index.ts, src/resources/evals/runs/index.ts, tests/api-resources/evals/runs/runs.test.ts
Relevant Source Files
- api.md — Generated SDK API reference for the public client surface, including the Evals resource family and its typed methods.
- src/resources/evals/index.ts — Barrel export for the Evals namespace. It re-exports the main Evals class plus create, retrieve, update, list, delete response types, parameter types, data source config types, and pagination page types.
- src/resources/evals/runs/index.ts — Barrel export for nested run resources. It re-exports Runs, run request and response types, run data source types, cancellation and deletion types, and the OutputItems subresource types.
- tests/api-resources/evals/runs/runs.test.ts — Generated resource tests for run operations. These tests show concrete invocation patterns for creating, retrieving, listing, deleting, and canceling eval runs, plus SDK response wrapper helpers and request option forwarding.
Resource Model
The SDK organizes Evals as a generated resource family. The exported top-level names include the main resource class and typed objects for evaluation creation, retrieval, update, listing, and deletion. The same export file also exposes two data source configuration types: one for custom data sources and one for stored completions. That split matters when designing an evaluation workflow, because the eval definition describes the task and grading intent, while the run uses a data source to provide examples for execution. In a typed project, these exported parameter and response types are the stable contract for constructing calls and interpreting returned objects.
Runs are nested beneath Evals because a run belongs to an evaluation definition. The tests call the run methods through the client with an evaluation identifier and, for operations on a particular run, a run identifier plus the enclosing evaluation identifier. That shape reflects the API hierarchy: first choose the evaluation, then create or inspect executions of that evaluation. The run export list includes response types for create, retrieve, list, delete, and cancel operations, so callers can keep compile-time awareness of each operation instead of treating every result as an untyped object.
Run output items form the inspection layer after an evaluation has executed. The nested runs index exports an OutputItems resource with retrieve and list response types, request parameter types, and a paginated page type. Conceptually, output items are the per-example or per-result records that let a team move beyond a single aggregate score. They are where an evaluation becomes actionable: inspect failures, compare unexpected model outputs, and identify whether the issue is data quality, prompt wording, tool selection, or a grader definition. The SDK exposes these items as first-class generated resources rather than requiring callers to parse them from a run object.
Sources: src/resources/evals/index.ts, src/resources/evals/runs/index.ts
Method and Type Reference
| Surface | Public names visible in source | Use |
|---|---|---|
| Evals | Evals, EvalCreateParams, EvalRetrieveResponse, EvalUpdateParams, EvalListResponsesPage, EvalDeleteResponse | Define, inspect, update, enumerate, and remove evaluation definitions. |
| Eval data sources | EvalCustomDataSourceConfig, EvalStoredCompletionsDataSourceConfig | Describe where evaluation examples or stored model outputs come from. |
| Runs | Runs, RunCreateParams, RunRetrieveParams, RunListParams, RunDeleteParams, RunCancelParams | Execute an eval and manage individual run lifecycle operations. |
| Run data sources | CreateEvalCompletionsRunDataSource, CreateEvalJSONLRunDataSource | Provide data when creating a run, including completions-based and JSONL-based sources. |
| Run responses | RunCreateResponse, RunRetrieveResponse, RunListResponse, RunDeleteResponse, RunCancelResponse | Typed responses returned by run lifecycle methods. |
| Output items | OutputItems, OutputItemRetrieveParams, OutputItemListParams, OutputItemListResponsesPage | Retrieve or page through detailed run result items. |
The generated test file provides the clearest concrete call signatures for runs. Creating a run is shown with an evaluation identifier as the first argument and an object containing a data source as the second argument. Retrieval, deletion, and cancellation are shown with the run identifier first and an options object that contains the evaluation identifier. Listing is shown with the evaluation identifier first and optional list parameters after it. This ordering is important for developers migrating from direct HTTP calls, because the SDK separates path identifiers from body or query parameters in a way that matches the generated resource method signatures.
The visible list parameters include pagination and filtering controls. The test passes an after cursor, a limit value, ascending order, and a queued status filter when listing runs. That demonstrates that run enumeration is intended for operational workflows, not just one-off scripts: a service can page through recent runs, filter by lifecycle state, and combine SDK request options with typed parameters. The same test intentionally overrides the request path to trigger a not found error, which verifies that per-request options are forwarded by the method and that generated resource calls still participate in the SDK-wide error model.
Sources: src/resources/evals/index.ts, src/resources/evals/runs/index.ts, tests/api-resources/evals/runs/runs.test.ts
Execution Flow
A typical code-first workflow begins by defining the evaluation task, then running it against known inputs, and finally inspecting the results. The official OpenAI evaluation guidance frames this as describing the task, running the eval with test data, and analyzing results before iterating. In SDK terms, the definition phase maps to the Evals resource and its create, retrieve, update, list, and delete types. The execution phase maps to the Runs resource. The analysis phase maps to retrieving run state and then using output item listing or retrieval to understand individual examples.
When creating a run, the test uses a JSONL-style data source with inline file content. Each content entry contains an item and may also include a sample object. The required-parameter case includes the source and type, while the optional-parameter case adds metadata and a name. That distinction gives implementers a practical baseline: the smallest call needs enough data source information to execute, while production calls can add descriptive metadata and a readable name so later list and audit workflows are easier to understand. Metadata is especially useful when attaching application version, prompt version, or deployment information to an evaluation run.
After creation, the run lifecycle is managed with retrieve, list, delete, and cancel operations. Retrieve is for checking a specific run by identifier within an eval. List is for operational dashboards, polling workers, and batch review scripts. Cancel is for stopping work that should no longer continue, such as a run launched against the wrong dataset or obsolete prompt. Delete is a cleanup operation. The SDK exposes separate response types for each operation, which helps calling code distinguish between terminal status checks, cancellation acknowledgements, and deletion results even when all calls are made through the same nested client namespace.
Sources: tests/api-resources/evals/runs/runs.test.ts, src/resources/evals/runs/index.ts
Response Handling and Request Options
The run tests also document an SDK-wide response pattern that applies to these generated methods. A resource call returns an API promise that can be awaited for the parsed data, converted to a raw web Response through the response helper, or awaited with both parsed data and raw response together. This is useful in evaluation infrastructure because a team may want typed results for application logic while still recording headers, status, or raw response metadata for observability. The tests assert that the raw response is a Response instance and that the parsed data is distinct from that raw object.
Request options are passed separately from resource parameters. The list test demonstrates this by supplying list filters and then a request option object with a path override that intentionally fails. In normal code, that same position is where callers can use SDK request configuration patterns such as timeouts, retries, headers, or other per-call behavior supported by the client. The key reference point is that Evals run methods behave like other generated resources: method parameters describe API inputs, while the trailing request options object customizes transport behavior for that single request.
Sources: tests/api-resources/evals/runs/runs.test.ts
Practical Usage Notes
Use Evals when you need repeatable measurement rather than a single manual inspection. For agent workflows, start with traces while debugging live behavior, then move to datasets and eval runs when you have a clear definition of good behavior and need to compare changes over time. The Evals platform is in a deprecation transition according to current OpenAI documentation, so new projects should verify the latest platform guidance before investing heavily in long-lived Evals API automation. Existing code can still use the SDK reference surface described here during the available transition period.
A good production evaluation loop records enough context to make results explainable. Put human-readable names and metadata on runs, keep identifiers for the prompt or application revision being tested, and page through output items after each run rather than relying only on aggregate status. If a run is queued or long-running, list filtering can support polling and dashboards. If a run was started with the wrong data source, cancel it rather than waiting for stale results. If result inspection shows repeated failures, update the eval definition or the application behavior and create a new run so the comparison remains reproducible.
Next, read the broader agent workflow evaluation guide if you are deciding between traces, datasets, and eval runs. Read the client and resource model page if you want to understand generated namespaces, paginated pages, request options, and response helpers across the whole SDK. For workflows that evaluate model outputs created through Responses or Chat Completions, pair this page with the corresponding API reference pages so the inputs under test and the evaluation run records are documented together.
Sources: api.md, src/resources/evals/index.ts, src/resources/evals/runs/index.ts, tests/api-resources/evals/runs/runs.test.ts