Embedding and Multimodal Providers

Purpose and Scope

This page helps you choose provider integrations for embeddings and multimodal work without treating every provider package as a separate decision. In the AI SDK, a provider is a model factory or model identifier that can be passed into core functions. The same project-level value proposition applies across language, embedding, image, audio, video, and ranking use cases: application code calls a stable AI SDK API, while the selected model determines which capability is actually used. For embeddings, the repository reference centers this model around embed(), which accepts an EmbeddingModel and a value whose type depends on that model. Sources: content/docs/07-reference/01-ai-sdk-core/05-embed.mdx

The practical implication is that capability should be your first filter. If your workflow needs semantic search, clustering, retrieval, or similarity, start with an embedding model and the embed() or related embedding APIs. If it needs image generation, transcription, speech, video, or reranking, use the corresponding AI SDK Core capability and then select a provider that supports that model class. Provider packages and AI Gateway model strings are setup choices; the public API keeps the application-level operation separate from the vendor-specific model implementation.

Relevant Source Files

• content/docs/07-reference/01-ai-sdk-core/05-embed.mdx — defines the public embed() reference, including the import, example call, accepted parameters, provider options, telemetry options, lifecycle callback behavior, and embedding-specific result fields.

Capability-Oriented Provider Selection

Embedding providers are best understood as providers of vector-producing models. An embedding converts a single value into a numeric vector that downstream systems can compare, store, cluster, or use for retrieval. The reference example embeds the text value sunny day at the beach with model: 'openai/text-embedding-3-small', showing the Gateway-style model string path. The same parameter can also be a provider-created embedding model, because the documented parameter type is EmbeddingModel. Sources: content/docs/07-reference/01-ai-sdk-core/05-embed.mdx

A capability-oriented selection process starts by naming the shape of the data and the downstream task. Text search and retrieval typically need text embeddings. Image search or multimodal retrieval may require a model whose input value supports images or other media, because the documented value type is model-dependent. Reranking is a different capability: it evaluates candidate results rather than producing a reusable vector. Image generation, transcription, speech, and video generation are also distinct model families even when they come from the same vendor. Keeping these categories separate prevents provider choice from leaking into application architecture.

AI Gateway and direct provider packages are two access paths to the same conceptual interface. A Gateway model string is convenient when you want to avoid installing a provider package for every vendor. A direct provider package is useful when you need provider-specific setup, custom authentication, or typed provider options. In both cases, the core call remains focused on the operation: pass an embedding model to embed() for a single vector, and pass provider-specific options through providerOptions only when the selected provider documents them.

Embedding API Contract

The embed() API is intentionally narrow: it generates one embedding for one value. The reference describes it as ideal when you need to embed a single value to retrieve similar items or use the embedding in a downstream task. That means it is a good fit for request-time comparisons, query embedding before vector database lookup, lightweight classification features, or preparing a single document fragment. Bulk ingestion should use the companion embedding-many workflow documented elsewhere, but embed() is the smallest public contract to understand first. Sources: content/docs/07-reference/01-ai-sdk-core/05-embed.mdx

import { embed } from 'ai';
 
const { embedding } = await embed({
  model: 'openai/text-embedding-3-small',
  value: 'sunny day at the beach',
});

The required parameters are model and value. model is an EmbeddingModel, with the reference giving openai.embeddingModel('text-embedding-3-small') as an example provider-created model. value is generic in the reference because providers can define which input types their embedding models accept. Optional transport and reliability controls include maxRetries, abortSignal, and headers. Provider-specific behavior belongs in providerOptions, which lets provider packages expose model features without changing the AI SDK Core function signature.

Provider Options, Telemetry, and Lifecycle Signals

Provider-specific options are the main escape hatch for capability differences. The providerOptions field is passed through to the selected provider, so embedding code can stay portable while still enabling provider-defined settings when necessary. This matters most for non-text or highly configurable models, where dimensions, input modes, output formats, or vendor flags may be available only on specific providers. Treat those settings as a boundary: keep portable application logic around embed(), then isolate provider-specific fields in a small configuration layer. Sources: content/docs/07-reference/01-ai-sdk-core/05-embed.mdx

Telemetry is also part of the public embedding contract. The reference exposes a telemetry object with isEnabled, recordInputs, recordOutputs, functionId, and per-call integrations. These fields let production applications observe embedding calls without rewriting provider code. For retrieval systems, a stable functionId is especially useful because query embedding, document embedding, and evaluation embedding can be grouped separately. Input and output recording should be reviewed carefully for privacy because embeddings often encode user text, documents, or media-derived content.

The lifecycle callback onStart is documented as running when the embed operation begins, before the embedding model is called. It receives event data such as a unique callId and an operationId identifying the operation type as ai.embed. The reference also states that errors thrown in this callback are silently caught and do not break the embedding flow. Use that behavior for logging, tracing, or metering setup work, not for required validation that must stop the request.

Compact Reference

System-to-Code Mapping

Use the provider ecosystem as a catalog of capabilities, then map the capability to the AI SDK Core operation. Embeddings map to embed() for a single vector and to embedding-many APIs for batches. Reranking maps to rerank-oriented APIs, while image, audio, video, and file workflows map to their own media APIs. The important pattern is consistent: your application imports from ai, selects a model from Gateway or a provider package, passes request-level options, and receives a normalized result appropriate to that operation.

For multimodal systems, this separation lets you compose different providers by stage. A product search system might embed catalog text with one provider, rerank search results with another, generate image previews with a third, and transcribe user voice queries with a fourth. The AI SDK design keeps these as separate calls rather than forcing one provider to satisfy every requirement. Start with the smallest capability-specific API, confirm the model supports your input and output shape, then add provider options, telemetry, and cancellation only when the production workflow needs them.

Next Steps

If you are building retrieval, read the embeddings and reranking material next so you can decide when to store vectors and when to rerank candidates at query time. If you are building media workflows, continue to the image, audio, and video generation pages and compare providers by capability rather than package name. For setup decisions, read the provider overview and AI Gateway page to decide whether Gateway model strings or direct provider packages are the right access path for your deployment.