Realtime Models Realtime

Purpose and Scope

Realtime in the AI SDK is the experimental Core capability for browser-based, bidirectional audio and text sessions. Instead of making a one-shot server call such as text generation, a realtime application opens a WebSocket from the browser to a realtime model endpoint, then exchanges microphone audio, model audio, text, messages, and tool calls during a live session. The documentation positions this as a voice-conversation workflow where the server does not proxy every frame; it mints a short-lived token and the browser uses that token to connect directly to the provider or AI Gateway.

Sources: content/docs/03-ai-sdk-core/36-realtime.mdx

The central reader problem is safely combining a browser session with credentials that normally belong on the server. The documented pattern solves that by splitting setup from runtime. A server setup endpoint owns the long-lived provider credential and calls a token-minting API. The browser receives only a short-lived client secret and session metadata, then uses the React realtime hook to manage connection, microphone capture, playback, messages, and tool-call handling. This separation is the primary security boundary for realtime applications.

Sources: content/docs/03-ai-sdk-core/36-realtime.mdx

Relevant Source Files

• content/docs/03-ai-sdk-core/36-realtime.mdx — Defines the public Realtime documentation page, including the experimental status note, setup endpoint pattern, AI Gateway variant, client hook usage, token flow, session configuration, and realtime tool-definition flow.

Core Primitives

A realtime session starts with experimental_realtime.getToken(). Provider packages expose this under their realtime support, and the server calls it with a target model and optional sessionConfig. The example uses openai.experimental_realtime.getToken() with the gpt-realtime model. The returned token is serialized from an API route so the browser can authenticate to the realtime provider without seeing the application’s server-side API key.

Sources: content/docs/03-ai-sdk-core/36-realtime.mdx

The browser primitive is experimental_useRealtime from @ai-sdk/react. The hook is configured with a realtime model, an api.token endpoint, and session configuration such as instructions, voice, input audio transcription, and turn detection. The docs describe the hook as the client-side integration point for connecting to a realtime model, capturing microphone audio, playing model audio, sending text messages, and rendering messages. In practice, this makes realtime feel closer to a stateful UI transport than a single Core generation call.

Sources: content/docs/03-ai-sdk-core/36-realtime.mdx

Tools are supported through a conversion step rather than by sending the normal Core tool objects directly to the realtime service. The server defines AI SDK tools with tool() and input schemas, then calls experimental_getRealtimeToolDefinitions({ tools }). Those generated definitions can be attached to sessionConfig.tools when the token is minted and returned to the browser alongside the token. When the realtime model emits a tool call, the application handles it with onToolCall, preserving application control over execution.

Sources: content/docs/03-ai-sdk-core/36-realtime.mdx

Execution Flow

The documented flow has five phases. First, the browser calls a setup endpoint such as app/api/realtime/setup/route.ts. Second, that endpoint creates a short-lived realtime token using the provider’s experimental_realtime.getToken() API. Third, the browser opens a WebSocket connection to the provider or AI Gateway using the returned token. Fourth, the model streams audio, text, and tool calls back to the browser. Fifth, the application responds to tool calls through its onToolCall handler instead of letting the model execute arbitrary application behavior by itself.

Sources: content/docs/03-ai-sdk-core/36-realtime.mdx

A minimal setup endpoint also demonstrates how session configuration is merged. The route reads the request body, converts local tools into realtime tool definitions, then passes sessionConfig: { ...body.sessionConfig, tools: toolDefinitions } to the token call. That pattern lets the client influence session-level behavior while the server retains authority over the final tool list and token creation. The docs explicitly warn that production endpoints should authenticate and rate-limit this route because it creates realtime sessions using server-side credentials.

Sources: content/docs/03-ai-sdk-core/36-realtime.mdx

import { openai } from '@ai-sdk/openai';
import { experimental_getRealtimeToolDefinitions, tool } from 'ai';
import { z } from 'zod';
 
const tools = {
  getWeather: tool({
    description: 'Get the current weather for a city',
    inputSchema: z.object({ city: z.string() }),
  }),
};
 
export async function POST(request: Request) {
  const body = await request.json().catch(() => ({}));
  const toolDefinitions = await experimental_getRealtimeToolDefinitions({ tools });
 
  const token = await openai.experimental_realtime.getToken({
    model: 'gpt-realtime',
    sessionConfig: {
      ...body.sessionConfig,
      tools: toolDefinitions,
    },
  });
 
  return Response.json({ ...token, tools: toolDefinitions });
}

AI Gateway and Provider-Specific Support

AI Gateway is the provider-management path for realtime when you want the same browser client code to work across supported upstream providers. The Gateway example replaces the provider-specific token call with gateway.experimental_realtime.getToken({ model: 'openai/gpt-realtime-2' }), then uses gateway.experimental_realtime('openai/gpt-realtime-2') in the browser. The documentation emphasizes that Gateway normalizes realtime events server-side while still returning only a short-lived client secret to the browser.

Sources: content/docs/03-ai-sdk-core/36-realtime.mdx

This Gateway split is important because it keeps credential usage on the server but allows the browser to construct a safe realtime model reference. The docs state that gateway.experimental_realtime.getToken() must run on the server because it uses the Gateway credential to mint a vcst_ client secret. In contrast, creating the model with gateway.experimental_realtime() is safe in browser code. Tool definitions are not a separate Gateway concept: the same experimental_getRealtimeToolDefinitions() conversion is used in the setup endpoint, and the hook includes definitions in the session update after the WebSocket opens.

Sources: content/docs/03-ai-sdk-core/36-realtime.mdx

'use client';
 
import { experimental_useRealtime } from '@ai-sdk/react';
import { gateway } from 'ai';
 
export default function RealtimePage() {
  const realtime = experimental_useRealtime({
    model: gateway.experimental_realtime('openai/gpt-realtime-2'),
    api: { token: '/api/realtime/setup' },
    sessionConfig: {
      instructions: 'You are a helpful assistant. Be concise.',
      inputAudioTranscription: {},
      voice: 'alloy',
      turnDetection: { type: 'server-vad' },
    },
  });
 
  return null;
}

API Components Reference

The session configuration shown in the docs includes instructions, inputAudioTranscription, voice, and turnDetection. These fields define the conversational behavior, transcription behavior, synthesized voice, and turn-taking mode for the live session. The setup endpoint can also merge client-supplied configuration with server-controlled fields, but tool definitions and token minting should remain server-mediated. Treat the session config as the contract between the browser UI and the realtime provider rather than as a replacement for endpoint authorization.

Sources: content/docs/03-ai-sdk-core/36-realtime.mdx

Implementation Guidance and Next Steps

Build realtime features by starting with the setup endpoint, not the UI. Confirm that the route can authenticate the user, rate-limit session creation, choose the allowed realtime model, attach only approved tool definitions, and return the short-lived token shape expected by the client. After that boundary is in place, wire experimental_useRealtime in a client component and add the session settings needed for the user experience, such as a concise assistant instruction, a voice, transcription, and server-side voice activity detection.

Sources: content/docs/03-ai-sdk-core/36-realtime.mdx

Choose direct provider realtime when you are intentionally targeting one provider’s realtime model and want the provider package to own token creation. Choose AI Gateway when you want Gateway-managed routing and normalized realtime events across supported upstreams. In both cases, remember that Realtime is marked experimental, so isolate integration code behind a small endpoint and client wrapper. For adjacent concepts, read the provider and model guidance before selecting a model, then review tool calling if your realtime assistant needs to call application functions during the conversation.