Serve CLI

Purpose and Scope

The Transformers Serve CLI is the project’s lightweight path for running a local or self-hosted model server without adopting a dedicated inference engine. It is intended for evaluation, experimentation, local application integration, and moderate-load deployments where the operational overhead of systems such as vLLM or SGLang is not required. The server exposes OpenAI-compatible endpoints, so clients that already speak OpenAI-style chat, completions, responses, audio transcription, or model-listing APIs can point at a local Transformers process instead of a hosted provider. Sources: docs/source/en/serve-cli/serving.md

This page focuses on how to launch the server, which API surfaces it provides, how to choose optimization flags, and how to connect common local clients. The Serve CLI documentation is split into a core serving guide, an optimization guide, and integration pages for Cursor, Jan, Open WebUI, and tiny-agents. Read it as a practical bridge between model loading in Transformers and interactive applications that expect an HTTP server rather than direct Python calls. Sources: docs/source/en/serve-cli/serving.md, docs/source/en/serve-cli/serving_optims.md

Relevant Source Files

• docs/source/en/serve-cli/serving.md — Defines the Serve CLI positioning, installation extra, default launch command, OpenAI-compatible REST routes, and client examples.
• docs/source/en/serve-cli/serving_optims.md — Documents throughput and memory options such as continuous batching, quantization, attention backends, torch compile, and reduced data types.
• docs/source/en/serve-cli/cursor.md — Explains how to expose a local server to Cursor, including CORS, ngrok tunneling, and OpenAI base URL configuration.
• docs/source/en/serve-cli/jan.md — Shows Jan’s native local-provider setup against the Serve CLI endpoint.
• docs/source/en/serve-cli/openweb_ui.md — Shows Open WebUI speech-to-text configuration using the local OpenAI-compatible endpoint.
• docs/source/en/serve-cli/tiny_agents.md — Connects the local server to tiny-agents and MCP tool servers through a JSON configuration.

Launching the Server

Install the serving extra before using the command-line server. The docs present the package extra as the required dependency bundle for this mode, then launch the server with a single command. By default, the server listens on localhost port eight thousand, and clients generally use the base URL with a versioned suffix when they expect OpenAI-compatible paths. This makes the first workflow simple: install the extra, start the server, then point an SDK or compatible application at the local base URL. Sources: docs/source/en/serve-cli/serving.md

pip install transformers[serving]
transformers serve

The core serving surface includes several routes with different compatibility goals. Chat completions are the main interface for conversational text and multimodal requests. Legacy completions accept a freeform prompt for older clients. The responses route supports the newer Responses API style. Audio transcriptions target speech-to-text workflows, models lists available models for integrations, and load_model streams model-loading progress with server-sent events. Together, these endpoints let the same local process serve chat applications, coding tools, agent loops, and transcription user interfaces. Sources: docs/source/en/serve-cli/serving.md

API Components

Use the OpenAI-compatible API contract when you want application interoperability, and use lower-level Transformers Python APIs when you need direct control over tensors, custom preprocessing, or model internals. The Serve CLI is not described as a replacement for specialized high-scale serving stacks; the docs explicitly position dedicated inference backends as the better option for large production deployments. The practical decision is therefore based on deployment size: choose Serve CLI for local and self-hosted convenience, then graduate to a dedicated backend when throughput, fleet management, or production hardening dominates the problem. Sources: docs/source/en/serve-cli/serving.md

Optimization Options

Continuous batching is the first throughput-oriented flag to consider. It dynamically groups and interleaves requests so multiple users can share GPU forward passes while individual sequences advance through prefill and decoding. New requests can join while existing requests are in progress, and completed requests can leave the batch after decoding. The result is better GPU utilization and higher throughput without making the server wait for a fixed batch to fill. Enable it with an attention implementation such as scaled dot product attention when you want a balanced default. Sources: docs/source/en/serve-cli/serving_optims.md

transformers serve --continuous-batching --attn-implementation sdpa

Quantization is the memory-reduction path. The documentation distinguishes pre-quantized models from runtime quantization: pre-quantized checkpoints can be passed as the requested model once the matching quantization library is installed, while runtime quantization is selected with the command-line quantization flag. Runtime quantization is especially useful for new checkpoints or fine-tunes that do not already have lower-precision weights. The Serve CLI optimization page specifically calls out bitsandbytes four-bit and eight-bit modes as the runtime quantization options. Sources: docs/source/en/serve-cli/serving_optims.md

transformers serve --quantization bnb-4bit

Attention backends, compilation, and data type selection provide additional knobs. Flash Attention can be selected with the attention implementation flag, and Apple Silicon users can install kernels so the server defaults to a Metal flash SDPA kernel on MPS, with an explicit SDPA override available when needed. The compile flag traces and compiles the decode loop for faster inference, but the docs note that compile is incompatible with continuous batching. Reduced data types such as bfloat16 and float16 save memory and can increase throughput when the hardware supports them. Sources: docs/source/en/serve-cli/serving_optims.md

transformers serve --continuous-batching --attn-implementation flash_attention_2
transformers serve --compile
transformers serve --continuous-batching --dtype bfloat16

Local Client Integrations

Cursor integration requires two extra considerations: CORS and reachability. CORS is disabled by default for security, so the docs start Cursor setup by enabling it on the server. Cursor also needs a public address to reach a local machine, so the guide uses ngrok to create a forwarding URL. In Cursor settings, deselect built-in models, add the desired local model name, enter any required API-key text, and set the OpenAI base URL override to the forwarding address with the versioned API suffix. Sources: docs/source/en/serve-cli/cursor.md

transformers serve --enable-cors
ngrok http localhost:8000

Jan has a simpler integration path because it supports the local server natively without tunneling. In Jan’s model provider settings, set the base URL to the local versioned endpoint and provide any text for the required API key field. The models section can then be refreshed, and a specific model such as Qwen can be added. This workflow is useful when the chat client and server run on the same workstation and do not need a public network bridge. Sources: docs/source/en/serve-cli/jan.md

Open WebUI uses the same compatibility idea for speech-to-text. Because it is a browser-facing self-hosted interface, the docs again start by enabling CORS on the local server. In the audio settings, choose the OpenAI speech-to-text engine, set the URL to the local versioned endpoint, and provide a transcription model such as a Whisper checkpoint. Once configured, speaking in a chat causes the local model served by Transformers to transcribe audio into the input field. Sources: docs/source/en/serve-cli/openweb_ui.md

tiny-agents shows the Serve CLI in an agentic workflow rather than a chat UI. The example configuration points the agent at the local model endpoint and also registers an MCP server delivered by a Hugging Face Space. The agent loop uses the local model for reasoning and tool selection, while the MCP server provides an external image generation tool. Keep the Transformers server running in the background before launching the agent configuration so model calls and tool calls can cooperate during the run. Sources: docs/source/en/serve-cli/tiny_agents.md

tiny-agents run path/to/your/config.json

Practical Next Steps

Start with the unoptimized server and a single client before adding performance flags. Once the request path works, enable continuous batching for concurrent generation, choose a reduced data type or quantized checkpoint when memory is the limiting factor, and select an attention backend that matches the accelerator. For browser or remote-editor integrations, enable CORS only when needed and prefer local-only settings when the client supports them. For broader background, read the continuous batching, quantization, inference optimization, and community integrations pages next.