Evaluate agent workflows
Purpose and Scope
Evaluation for agent workflows is about proving that an agent still behaves correctly after changes to prompts, model settings, tool definitions, guardrails, routing logic, or SDK versions. The OpenAI platform guidance frames this as a progression: start with traces while you are debugging individual workflow behavior, then move to datasets and repeatable eval runs when you need benchmarkable results. In this repository, the strongest source-backed signals are not a runnable eval script; they are the CI and release gates that keep the TypeScript SDK compatible with agent workflows and the generated admin surface that controls hosted tools used by those workflows.
Sources: .github/workflows/ci.yml, .github/workflows/detect-breaking-changes.yml, src/resources/admin/organization/projects/hosted-tool-permissions.ts
An agent workflow normally includes a model call plus runtime behavior such as tools, hosted tools, MCP servers, handoffs, guardrails, and structured outputs. When you evaluate that workflow, you are checking the whole path, not only the final text. A good trace can answer whether the right tool was selected, whether a handoff occurred at the right point, and whether the workflow followed instructions. The SDK repository reinforces that whole-workflow mindset by running a downstream Agents SDK regression job against a locally built openai-node package, so changes in generated API surfaces are checked against real agent package consumers before they reach users.
Sources: .github/workflows/detect-breaking-changes.yml
Relevant Source Files
- .github/workflows/ci.yml - Defines the main CI checks for linting, building, and testing the SDK across Node versions, which is the baseline quality gate before agent-evaluation behavior can be trusted.
- .github/workflows/detect-breaking-changes.yml - Adds API compatibility detection and a dedicated Detect Agents SDK regressions job that builds openai-node, links it into openai-agents-js packages, then builds, lints, and type-checks docs scripts downstream.
- .github/workflows/create-releases.yml - Creates releases only on main for openai/openai-node, then publishes to NPM and JSR after release creation, making release automation part of the evaluation-to-delivery path.
- .github/workflows/publish-jsr.yml - Provides a manual JSR publish workflow with dependency installation and a publish script under the publish environment.
- .github/workflows/publish-npm.yml - Provides a manual NPM publish workflow with the same publish environment pattern and package publish script.
- src/resources/admin/organization/projects/hosted-tool-permissions.ts - Implements the generated admin resource for retrieving and updating project hosted-tool permissions, including code interpreter, file search, image generation, MCP, and web search.
Evaluation Workflow
Use traces first when the question is diagnostic. A trace is the end-to-end record for a single run: model calls, tool calls, guardrails, and handoffs. During early development, inspect representative traces from the application and grade them with criteria that reflect the intended task. For example, a grader might check whether the agent called file search before answering a retrieval question, whether MCP was invoked only for external-service work, or whether a tool handoff matched a routing instruction. This phase produces fast feedback because it is tied to concrete workflow executions rather than an abstract unit test.
When behavior stabilizes, move from trace inspection to repeatable datasets and eval runs. A dataset captures the inputs and expected criteria that define good behavior; an eval run applies those criteria across many examples so prompt, model, or tool-surface changes can be compared over time. The repository CI provides the SDK-side analogue of that repeatability. The main CI workflow installs dependencies with pnpm, builds the package, and runs tests over a Node matrix that includes Node 20, 22, 24, and 26.2.0. That matrix matters for agent apps because streaming, multipart uploads, tool-enabled requests, and generated resource types must remain stable across supported runtimes.
Sources: .github/workflows/ci.yml
System-to-Code Mapping
The most direct repository-level evaluation signal for agent workflows is the Detect Agents SDK regressions job. It checks out openai-node, bootstraps and builds it, then checks out openai/openai-agents-js. The downstream workspace is configured to consume the local SDK build through file dependencies for @openai/agents-core, @openai/agents-openai, and @openai/agents. It then installs, builds all packages, runs the linter, and type-checks documentation scripts. This is an integration-style evaluation: instead of only confirming that this repository compiles in isolation, it verifies that a locally changed SDK still supports the agent packages that depend on it.
Sources: .github/workflows/detect-breaking-changes.yml
The same workflow also contains a detect-breaking-changes job for pull requests into main and next. It calculates fetch depth from the pull request commit count, checks out enough history to compare against the base SHA, restores a previous version of the breaking-change script when possible, and runs it in compatibility mode. For generated SDKs, that protects public API contracts such as method names, type shapes, request parameters, and response types. For agent workflow evaluation, that protection is important because graders and datasets often encode tool configuration and response-item expectations that can silently break if generated types or resource paths shift.
Sources: .github/workflows/detect-breaking-changes.yml
Hosted Tool Permissions and Evaluation Controls
Hosted tools are part of what an agent can do at runtime, so evaluation should account for whether the project is allowed to use those tools. The generated HostedToolPermissions resource exposes retrieve and update methods under client.admin.organization.projects.hostedToolPermissions. Both methods address /organization/projects/{projectID}/hosted_tool_permissions and require admin API key authentication through the request security metadata. The retrieved ProjectHostedToolPermissions object reports enabled booleans for code_interpreter, file_search, image_generation, mcp, and web_search. The update params allow the same permission groups to be changed individually or omitted when no change is intended.
Sources: src/resources/admin/organization/projects/hosted-tool-permissions.ts
This resource is useful before and after an agent eval run. Before running a dataset, retrieve the project permissions and confirm that the eval environment matches production assumptions. An agent that depends on file search, MCP, or web search can fail for reasons unrelated to prompting if the project permission is disabled. After a policy or deployment change, update permissions intentionally and re-run representative traces or dataset evals to measure the effect. Treat hosted-tool permission state as part of the evaluation fixture, just like model choice, instructions, tool list, and guardrail configuration.
Sources: src/resources/admin/organization/projects/hosted-tool-permissions.ts
CI, Release, and Delivery Signals
Evaluation results are only useful if the SDK version that passed them is the version users receive. The release workflow runs only for main in openai/openai-node, uses a publish environment, and triggers release creation before publishing to both NPM and JSR. Separate manual workflows publish to NPM or JSR with explicit workflow_dispatch triggers, dependency installation, and publish scripts. These gates separate routine validation from package distribution, which is important when evaluating agent workflows across package managers and runtime targets.
Sources: .github/workflows/create-releases.yml, .github/workflows/publish-npm.yml, .github/workflows/publish-jsr.yml
For application teams, the practical pattern is to mirror this layering. Use trace grading for day-to-day agent debugging, use datasets and eval runs for repeatable comparisons, and keep SDK upgrades behind CI that exercises the actual agent package or application. If a workflow uses hosted tools, record the permission state and include it in the eval setup. Before promoting an SDK upgrade, run the application eval suite on the same Node or edge runtime used in production, then check release notes and package provenance for the channel you deploy from.
Compact Reference
| Concern | Repository-backed signal | What to verify |
|---|---|---|
| SDK correctness | .github/workflows/ci.yml | lint, build, and tests pass across the Node matrix |
| Public API compatibility | .github/workflows/detect-breaking-changes.yml | generated methods and types remain compatible with the pull request base |
| Agent downstream compatibility | .github/workflows/detect-breaking-changes.yml | openai-agents-js builds, lints, and type-checks against local openai-node dist |
| Hosted tool availability | src/resources/admin/organization/projects/hosted-tool-permissions.ts | code_interpreter, file_search, image_generation, mcp, and web_search are enabled or disabled as intended |
| Package delivery | .github/workflows/create-releases.yml, .github/workflows/publish-npm.yml, .github/workflows/publish-jsr.yml | release and manual publish paths use the expected publish environment and scripts |
Next, read the agent definitions, tools and approvals, MCP, code interpreter, file search, and evals reference pages to connect these repository signals to the specific tool and eval APIs your workflow uses.