Computer use

Purpose and Scope

Computer use is an agent workflow pattern where a model helps operate software through a user interface. In the OpenAI platform documentation, that means the model may inspect screenshots, propose interface actions for your application to execute, or work inside a custom harness that combines visual state with programmatic controls. In the JavaScript SDK, the most relevant repository-backed surface for this page is the administrative hosted-tool permission API. That API lets an organization administrator inspect and update whether hosted tools are enabled for a project, which is part of the operational boundary around tool-capable workflows. Sources: src/resources/admin/organization/projects/hosted-tool-permissions.ts

A computer-use implementation normally has two halves. The first half is the harness: browser automation, desktop automation, remote VM control, screenshot capture, and the code that turns model-suggested actions into real UI events. The second half is platform and project policy: which tools are available, which project can use them, and what approvals or human review should surround high-impact actions. The official computer-use guidance emphasizes isolated browsers or VMs, a human in the loop for sensitive work, and treating page content as untrusted input. The SDK source shown here supports the policy side through generated project hosted-tool permission methods. Sources: src/resources/admin/organization/projects/hosted-tool-permissions.ts

Relevant Source Files

• src/resources/admin/organization/projects/hosted-tool-permissions.ts — Defines the generated HostedToolPermissions resource, its retrieve and update methods, the project permission response shape, and update parameter types for hosted tools.
• tests/api-resources/admin/organization/projects/hosted-tool-permissions.test.ts — Exercises the generated resource through an OpenAI client configured with both an API key and an admin API key, and verifies promise, raw response, and data-with-response behavior.

Core Concepts for Computer-Use Workflows

A computer-use harness should be treated as a controlled execution environment rather than a simple text-generation call. The model can reason over a UI state, but your code remains responsible for capturing screenshots, applying actions, enforcing allowlists, pausing for human approval, and protecting credentials. In JavaScript projects, Playwright or similar browser automation tools are a common way to prototype this harness. The model-facing request flow may use the Responses API or custom tools, while the safety envelope lives in your application and administrative configuration.

Hosted tools are OpenAI-managed capabilities that can participate in model workflows, such as code execution, file search, image generation, MCP-backed integrations, and web search. The generated ProjectHostedToolPermissions interface models each hosted tool as an object with an enabled boolean. Although computer-use harnesses often involve custom UI automation code outside the SDK, they commonly sit alongside hosted tools in a broader agent workflow. For example, a UI automation loop might use file search for context, MCP for external services, or code interpreter for analysis, and administrators may want those capabilities enabled or disabled per project. Sources: src/resources/admin/organization/projects/hosted-tool-permissions.ts

System-to-Code Mapping

The SDK exposes hosted-tool administration under the nested client path client.admin.organization.projects.hostedToolPermissions. That path mirrors the API hierarchy: administrative APIs, organization scope, project scope, then the hosted-tool permission resource. The generated class extends the SDK resource base and calls the client’s HTTP methods directly. retrieve(projectID, options) performs a GET against /organization/projects/${projectID}/hosted_tool_permissions, while update(projectID, body, options) performs a POST to the same path with a typed request body. Sources: src/resources/admin/organization/projects/hosted-tool-permissions.ts

Both methods attach __security: { adminAPIKeyAuth: true }, which is an important distinction for readers building production computer-use systems. A normal model request may use the standard API key configured on the client, but project-level administration requires an admin API key. That separation helps keep runtime agent execution and administrative permission management distinct. In practice, a service that runs a browser automation harness should not casually hold broad administrative credentials; instead, use administrative credentials only in setup, deployment, or a secured control-plane path. Sources: src/resources/admin/organization/projects/hosted-tool-permissions.ts, tests/api-resources/admin/organization/projects/hosted-tool-permissions.test.ts

Hosted Tool Permissions Reference

The hosted-tool permission response is represented by ProjectHostedToolPermissions. It contains code_interpreter, file_search, image_generation, mcp, and web_search, and each property contains an enabled boolean. The update body is represented by HostedToolPermissionUpdateParams, where each tool-specific property is optional and may be set to a corresponding update object or null. This shape lets callers update only the hosted-tool permissions they intend to change while leaving other tool states under server-side control. Sources: src/resources/admin/organization/projects/hosted-tool-permissions.ts

A compact administrative check might look like this when run from a trusted control-plane process:

import OpenAI from 'openai';
 
const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  adminAPIKey: process.env.OPENAI_ADMIN_API_KEY,
});
 
const permissions = await client.admin.organization.projects.hostedToolPermissions.retrieve(
  'project_id',
);
 
await client.admin.organization.projects.hostedToolPermissions.update('project_id', {
  code_interpreter: { enabled: permissions.code_interpreter.enabled },
  file_search: { enabled: true },
});

The example intentionally separates the administrative permission check from any browser-control loop. A computer-use worker should execute only the actions its harness permits, and it should receive narrowly scoped runtime configuration. The hosted-tool permission API is better used before or around that worker: validate the project’s tool state during deployment, render an administrative settings page, or enforce an operational preflight before enabling an agent experience that combines UI automation with hosted tools.

Execution Flow

A practical flow starts with a project owner deciding which hosted tools are allowed for the project. A privileged service then retrieves the current permission state and updates only the allowed capabilities. After that, the runtime agent application can make model requests and run its computer-use harness under separate runtime credentials. When the model proposes UI actions, your harness should validate them against local policy, execute safe actions in an isolated browser or VM, return updated observations, and interrupt for human approval when actions affect money, accounts, identity, production data, or external communications.

The official computer-use guidance also calls out untrusted page content. A web page can contain text that tries to manipulate the model or the automation harness, so the harness should not treat page instructions as developer instructions. Keep browser environment variables empty where possible, disable extensions and local file-system access, and run the browser in a sandboxed context. The SDK does not replace those controls; instead, it gives the application typed API access and administrative surfaces that can be combined with a disciplined harness design.

Testing Signals

The API-resource test constructs an OpenAI client with both apiKey and adminAPIKey, then points baseURL at TEST_API_BASE_URL or a local mock server. It calls retrieve('project_id') and update('project_id', {}), checks that asResponse() yields a Response, awaits the parsed response, and then verifies withResponse() returns the same parsed data with the raw response object. This test pattern is useful for SDK consumers because it demonstrates that generated resource methods support both high-level typed data access and lower-level response inspection. Sources: tests/api-resources/admin/organization/projects/hosted-tool-permissions.test.ts

Those testing signals matter for computer-use operations because control-plane code often needs auditable HTTP behavior. If an administrative permission update fails, an operator may need status codes, headers, or raw response details rather than just the parsed body. The SDK promise helpers shown in the test give you those options without changing the resource method call. For production rollout, pair these SDK-level checks with integration tests for your browser or VM harness, approval gates, action allowlists, and secret isolation.

Next Steps

Start by deciding whether your computer-use workflow uses a built-in Responses API computer tool, a custom UI automation tool, or a hybrid harness. Then inventory any hosted tools the workflow depends on and use client.admin.organization.projects.hostedToolPermissions.retrieve() from a trusted administrative context to confirm project policy. After permissions are in place, implement the harness with isolation, observation capture, action validation, and human approval for high-impact operations. For adjacent SDK concepts, continue with the pages on Responses API concepts, tools and approvals, MCP integrations, Code Interpreter, and deployment and data controls.