Audio and speech

Purpose and Scope

Audio support in the OpenAI TypeScript and JavaScript SDK is organized around the Audio API namespace on the generated client. This page explains how that namespace fits together for text to speech, speech to text, translation-oriented workflows, and local helper utilities. The SDK exposes audio as a typed resource family, so application code can call dedicated subresources instead of constructing raw REST requests. The practical result is a single client area for producing spoken output, uploading recorded audio for transcription, requesting streamed transcript events, and accessing the generated type definitions that describe response shapes.

Sources: src/resources/audio/audio.ts, src/resources/audio/speech.ts, src/resources/audio/transcriptions.ts

The first-party OpenAI audio docs describe four common modalities: audio input, audio output, text transcript, and text prompt. In SDK terms, those map to concrete calls such as creating speech from text, sending an uploaded audio file for transcription, or using the exported translation namespace for speech translation tasks. The request-based Audio API is most appropriate for bounded files and generated audio assets. For microphone-like low-latency sessions, the broader platform guidance points readers toward Realtime APIs, while this page stays focused on the generated Audio resource and helper functions exposed in this repository.

Sources: src/resources/audio/audio.ts, src/resources/audio/index.ts, src/helpers/audio.ts

Relevant Source Files

• src/resources/audio/speech.ts - Defines the generated Speech resource, create method, speech models, voices, output formats, speed, instructions, and streaming format options.
• src/resources/audio/index.ts - Re-exports the public audio resource classes and audio-related generated types from the package subdirectory.
• src/helpers/audio.ts - Provides Node-oriented helper functions for playing audio with ffplay and recording audio with ffmpeg into a File.
• src/resources/audio.ts - Re-exports the audio directory as the top-level generated audio module entry.
• src/resources/audio/audio.ts - Defines the Audio resource container, attaches transcriptions, translations, and speech, and exports shared audio model and response-format unions.
• src/resources/audio/transcriptions.ts - Defines the generated Transcriptions resource, overloads for response formats and streaming, multipart upload behavior, and transcription response types.

Core Primitives

The Audio class is the namespace object that gathers the audio subresources under client.audio. Its constructor properties instantiate transcriptions, translations, and speech with the same internal client, which means audio calls share the same authentication, base URL, retry, timeout, and request-option behavior as the rest of the SDK. The class also publishes static references for the generated subresource classes, allowing the type namespace to expose resource classes and generated data structures together. This pattern matches other generated SDK resource groups and keeps application code discoverable around a single audio entry point.

Sources: src/resources/audio/audio.ts, src/resources/audio/index.ts, src/resources/audio.ts

Speech generation is represented by client.audio.speech.create. The method posts to the audio speech endpoint, accepts a typed body, and declares a binary response so callers receive a web Response object rather than a parsed JSON object. The generated parameters capture the main choices a text-to-speech application must make: input text, model, voice, optional instructions, output format, playback speed, and streaming format. The implementation also sets an accept header for octet-stream content, which aligns with the endpoint returning audio bytes or streaming audio data.

Sources: src/resources/audio/speech.ts

Transcription is represented by client.audio.transcriptions.create, which accepts uploadable audio and sends multipart form data. The generated overloads are important because the returned value depends on request options. A default JSON transcription returns an object with a text field, verbose JSON returns richer segment information, text-like formats return a string, and streaming requests return a stream of transcription events. The resource also attaches request metadata for the selected model and sets the stream flag from the request body, so streaming and non-streaming calls share one public method while preserving distinct TypeScript return types.

Sources: src/resources/audio/transcriptions.ts

Task Flow

A typical text-to-speech flow starts by constructing the OpenAI client, choosing a speech model, selecting a voice, and passing the text to generate. For modern voice control, the SDK type includes instructions, but the source notes that instructions do not work with the older tts-1 or tts-1-hd models. The returned value is a Response, so application code can call arrayBuffer, blob, or stream APIs depending on the runtime. In Node, a common pattern is to convert the array buffer into a buffer and write it to an audio file.

Sources: src/resources/audio/speech.ts

import fs from 'node:fs/promises';
import OpenAI from 'openai';
 
const client = new OpenAI();
 
const audio = await client.audio.speech.create({
  model: 'gpt-4o-mini-tts',
  voice: 'coral',
  input: 'Today is a wonderful day to build something people love!',
  instructions: 'Speak in a cheerful and positive tone.',
  response_format: 'mp3',
});
 
await fs.writeFile('speech.mp3', Buffer.from(await audio.arrayBuffer()));

A request-based speech-to-text flow begins with an uploadable file stream or file-like object and a transcription model. The OpenAI docs call out transcription and translation as the two speech-to-text endpoints, and the SDK reflects that split through separate subresources under the audio namespace. For transcription, the generated create overloads let TypeScript narrow the response based on response_format and stream. That means a captions workflow can request text-like output, an analysis workflow can request verbose segment details, and a live-ish bounded workflow can consume stream events from the same method name.

Sources: src/resources/audio/audio.ts, src/resources/audio/transcriptions.ts

import fs from 'node:fs';
import OpenAI from 'openai';
 
const client = new OpenAI();
 
const transcription = await client.audio.transcriptions.create({
  file: fs.createReadStream('speech.mp3'),
  model: 'gpt-4o-transcribe',
});
 
console.log(transcription.text);

API Components

The compact reference above hides a few details that matter when designing application behavior. Speech input has a documented maximum length of 4096 characters in the generated type comments. Voice can be a built-in voice name or a custom voice reference object with an id. Response formats for generated speech include common compressed and uncompressed audio containers, while stream_format separates server-sent event style streaming from audio streaming. Transcription response formats are constrained by model family, with diarized output reserved for the diarization-capable model when speaker annotations are required.

Sources: src/resources/audio/speech.ts, src/resources/audio/audio.ts, src/resources/audio/transcriptions.ts

Helper Behavior

The audio helper module is not a generated REST resource; it is a convenience layer for local development and demos. playAudio accepts a Node readable stream, a web Response, or a File. In Node, it spawns ffplay, chooses the appropriate readable source, and pipes audio bytes into the player process. For Response bodies, it handles both Node streams and web readable streams by converting web streams when needed. For File inputs, it checks that file support is available and reads the file stream before piping it to playback.

Sources: src/helpers/audio.ts

Recording follows the same Node-first philosophy. The helper chooses an ffmpeg input provider from the current platform, records from a device index, forces a default sample rate of 24000 Hz and one channel, emits WAV bytes to standard output, and wraps the captured data in a File named audio.wav. The function accepts an abort signal, a device number, and a timeout. Both timeout and external abort signals terminate the ffmpeg process, allowing demos to capture bounded clips that can then be passed directly into transcription calls as uploadable audio.

Sources: src/helpers/audio.ts

There are a few operational constraints to plan for. The helper playback path requires ffplay, recording requires ffmpeg, and browser playback through this helper deliberately throws an error rather than attempting a partial browser implementation. On the API side, speech creation returns raw binary response content, so caller code must decide whether to save, stream, play, or forward those bytes. Transcriptions use multipart upload options, so callers should provide a supported uploadable value and choose output formats that match the selected model. These constraints are explicit in the resource signatures and helper implementation.

Sources: src/helpers/audio.ts, src/resources/audio/speech.ts, src/resources/audio/transcriptions.ts

Implementation Details and Next Steps

Use this page when choosing the right audio surface before writing code. If the application generates narration, announcements, or spoken assistant replies from text, begin with client.audio.speech.create and decide whether binary file output or streaming output is more appropriate. If the application processes an audio file into text, use client.audio.transcriptions.create and select a response format that fits downstream needs. If the application needs translation, start from client.audio.translations in the same namespace and then consult the dedicated reference page for method-level details.

Sources: src/resources/audio/audio.ts, src/resources/audio/index.ts

For live voice agents, continuous captions, or speech-to-speech interactions, treat this Audio API page as the request-based foundation rather than the complete architecture. The official audio docs distinguish file-oriented transcription from Realtime transcription and describe streaming as the way to exchange partial input or output while an interaction is active. In this SDK, request streaming appears in transcription event streams and speech stream formats, while full-duplex conversational voice belongs with the Realtime API pages. Good next reads are the Realtime API page, the streaming examples page, and the audio reference page for method-level signatures.