Streaming

Purpose and Scope

Streaming in LlamaIndex is the set of interfaces that let an application consume model output incrementally instead of waiting for a complete final answer. It appears at several layers: low-level LLM chat responses, response synthesis objects returned by query-style workflows, chat engine responses that update chat history, and workflow agents that may stream token or event output while tools are being selected and executed. This page focuses on the public contracts visible in the core response and chat engine types, plus the deployment guidance that tells agent users when streaming is enabled and how to turn it off when a model cannot support it.

Sources: llama-index-core/llama_index/core/base/llms/types.py, llama-index-core/llama_index/core/chat_engine/types.py, llama-index-core/llama_index/core/base/response/schema.py, docs/src/content/docs/framework/module_guides/deploying/agents/index.mdx

Relevant Source Files

• llama-index-core/llama_index/core/base/llms/types.py - Defines foundational LLM message roles, content blocks, and the chat response generator types imported by chat engine streaming code.
• llama-index-core/llama_index/core/chat_engine/types.py - Defines chat engine response modes, regular agent chat responses, streaming agent chat responses, queues, stream generators, and stream instrumentation events.
• llama-index-core/llama_index/core/base/response/schema.py - Defines non-streaming and streaming response dataclasses used by query and synthesis paths, including source nodes and metadata.
• docs/src/content/docs/framework/module_guides/deploying/agents/index.mdx - Documents the agent loop, FunctionAgent construction, and the user-facing note that streaming is enabled by default but can be disabled for unsupported models.

Core Primitives

The lowest layer is the LLM message and response contract. The LLM types module defines message roles such as system, developer, user, assistant, function, tool, chatbot, and model, and it defines content block behavior for multimodal or structured inputs. Chat engine code imports ChatMessage together with synchronous and asynchronous chat response generator aliases from that module, so streaming chat implementations can work with a typed stream rather than an unstructured iterator. In practice, this means the same message vocabulary can be used for ordinary chat, multimodal agent inputs, and streamed chat output.

Sources: llama-index-core/llama_index/core/base/llms/types.py, llama-index-core/llama_index/core/chat_engine/types.py

For query-style flows, the important distinction is between Response and StreamingResponse. Response represents a completed answer when streaming is not enabled; it carries response text, source nodes, and optional metadata, and it can format source excerpts for display. StreamingResponse represents the streaming case and stores a token generator alongside the same source-node and metadata fields. That shape is important because downstream code can present tokens immediately while still retaining the retrieval evidence needed for citations, debugging, or user-facing source displays after the stream finishes.

Sources: llama-index-core/llama_index/core/base/response/schema.py

Chat Engine Streaming Interfaces

Chat engines are stateful conversation interfaces, and the official usage pattern streams with stream_chat and then iterates over response_gen. The core chat engine types show why that pattern works: AgentChatResponse is the non-streaming response container, while StreamingAgentChatResponse is the streaming chat response container that can receive ChatResponseGen or ChatResponseAsyncGen. The streaming class also stores the accumulated response text, unformatted response text, tool sources, source nodes, and metadata-like state. This lets a UI render partial output while the chat engine continues building the final answer and preserving evidence.

Sources: llama-index-core/llama_index/core/chat_engine/types.py

There is also a deliberate fake-streaming path for completed agent chat responses. AgentChatResponse exposes response_gen and async_response_gen only when is_dummy_stream is true. If a caller tries to use those generators on a normal non-streaming response, the implementation raises a ValueError explaining that the generator is only available for streaming responses unless dummy streaming is explicitly requested. This edge case matters for tools: a tool may return a complete Response or StreamingResponse, and the chat response object can derive source nodes from those tool outputs before any presentation layer starts iterating.

Sources: llama-index-core/llama_index/core/chat_engine/types.py, llama-index-core/llama_index/core/base/response/schema.py

Agent Streaming and Runtime Behavior

Agents add another layer because output is interleaved with reasoning steps and tool calls. The deployment guide defines an agent as a system that uses an LLM, memory, and tools to handle outside inputs. Its loop gathers the latest message plus chat history, sends tool schemas and history to the model, executes selected tool calls, appends tool results to memory, and repeats until the model answers directly. The same guide notes that FunctionAgent enables streaming by default, while users can pass streaming=False if a provider or model does not support streamed LLM output.

Sources: docs/src/content/docs/framework/module_guides/deploying/agents/index.mdx

The chat engine streaming implementation is designed for concurrent production of tokens and consumption by client code. StreamingAgentChatResponse includes a standard Queue, an optional asyncio queue, completion flags, events for new items, and fields that distinguish function-call messages from ordinary assistant text. It also imports instrumentation events for stream start, stream delta received, stream end, and stream error. Those pieces are not user-interface policy by themselves; they are runtime plumbing that lets a synchronous or asynchronous caller react as each delta arrives and still handle cancellation, completion, or errors cleanly.

Sources: llama-index-core/llama_index/core/chat_engine/types.py

Practical Usage Patterns

For chat engines, choose the streaming method when the user experience benefits from low latency or visible progress. A minimal pattern is to create a chat engine from an index, call the streaming chat method, and print or send each generated token to the client as it arrives. For query engines or response synthesis, expect the returned object to differ when streaming is enabled: code should consume the generator on StreamingResponse instead of assuming a complete response string is already present. After iteration, keep the response object around if the application needs source nodes or metadata.

Sources: llama-index-core/llama_index/core/base/response/schema.py, llama-index-core/llama_index/core/chat_engine/types.py

chat_engine = index.as_chat_engine()
streaming_response = chat_engine.stream_chat("Tell me a joke.")
 
for token in streaming_response.response_gen:
    print(token, end="")

For agents, start with the default streaming behavior when using FunctionAgent and a provider known to support streamed output. If a provider raises an error or the application needs a simpler blocking execution model, construct the agent with streaming disabled. This is especially relevant in heterogeneous deployments where some models support tool calling but not streaming deltas. The same agent loop still applies: memory and tool outputs are accumulated, and the final response can be printed or converted to a string, but the application no longer needs to manage a live token stream.

Sources: docs/src/content/docs/framework/module_guides/deploying/agents/index.mdx

Compact Reference

Next Steps

If you are wiring a user interface, decide first whether the application is chat-oriented, query-oriented, or agent-oriented. Chat engines expose the simplest token iteration pattern for conversations, query responses preserve retrieval sources around a token generator, and agents combine streaming with tool execution and memory updates. Read the chat engine, query engine, agents, callbacks, and instrumentation pages next if you need to connect streamed deltas to persistent chat history, source display, tracing, or deployment-specific error handling.