Deploying Chat Engines

Purpose and Scope

A chat engine is the deployment-facing interface to a conversational LlamaIndex application: instead of answering one isolated question, it keeps enough conversation history to answer follow-up questions in context. The official deployment guidance presents it as the stateful counterpart to a query engine, and the API reference exposes a dedicated chat engine type surface plus multiple concrete engine implementations. Use this page when you are turning an index-backed prototype into an application endpoint, UI handler, notebook service, or agent-facing component that needs multi-turn behavior rather than stateless retrieval.

Sources: docs/api_reference/api_reference/chat_engines/index.md, docs/src/content/docs/framework/community/faq/chat_engines.md

The central deployment decision is whether the user experience requires memory of previous turns. If users ask standalone questions over the same data, a query engine is usually the simpler fit. If users say “what about the second one?” or “summarize that in a different tone,” the application needs chat-specific state. The FAQ confirms this user problem directly: LlamaIndex provides chat engines to retain context and answer according to that context. That statefulness is the reason chat engines deserve separate deployment treatment from one-shot query endpoints.

Sources: docs/src/content/docs/framework/community/faq/chat_engines.md

Relevant Source Files

• docs/api_reference/api_reference/chat_engines/index.md — API reference entry point for llama_index.core.chat_engine.types, the shared typed surface for chat engines.
• docs/api_reference/api_reference/chat_engines/simple.md — Reference page selecting SimpleChatEngine from llama_index.core.chat_engine.
• docs/api_reference/api_reference/chat_engines/context.md — Reference page selecting ContextChatEngine from llama_index.core.chat_engine.
• docs/api_reference/api_reference/chat_engines/condense_question.md — Reference page selecting CondenseQuestionChatEngine from llama_index.core.chat_engine.
• docs/api_reference/api_reference/chat_engines/condense_plus_context.md — Reference page selecting CondensePlusContextChatEngine from llama_index.core.chat_engine.
• docs/src/content/docs/framework/community/faq/chat_engines.md — Community FAQ that describes retaining context and points users toward chat modes for data-agent-style chat behavior.

Deployment Model

Deploying a chat engine means exposing a multi-turn interaction loop around an index-backed or tool-backed LlamaIndex component. The smallest usage pattern is to construct a chat engine from an index and call chat() with a user message. In a web service, that call typically sits inside a route handler or websocket message handler; in an app backend, it sits behind a conversation API that also knows which user or session is speaking. The chat engine abstraction is important because it hides the mechanics of transforming user turns, prior history, retrieved context, and generated output into a single response object.

chat_engine = index.as_chat_engine()
response = chat_engine.chat("Tell me a joke.")

For production interfaces, treat the chat engine as a conversational runtime component rather than as a pure function. Each request belongs to a conversation, and the conversation state must be associated with a user, tenant, notebook cell, browser tab, or other caller identity. The source-backed FAQ only states the high-level capability, but that capability has concrete deployment consequences: retaining context is useful only when the application can reliably associate later turns with the right earlier turns. If you cannot persist or reconstruct that association, prefer a query engine endpoint and pass explicit context in each request.

Sources: docs/src/content/docs/framework/community/faq/chat_engines.md

Streaming changes the deployment shape because the caller receives partial output while generation is still in progress. The official usage pattern exposes this through stream_chat() and iteration over streaming_response.response_gen. In a CLI this can print tokens as they arrive; in a web app it often maps to server-sent events, websocket messages, or an HTTP streaming response. The important deployment constraint is that streaming handlers need cancellation, timeout, and cleanup behavior that a simple synchronous chat() route may not require.

chat_engine = index.as_chat_engine()
streaming_response = chat_engine.stream_chat("Tell me a joke.")
for token in streaming_response.response_gen:
    print(token, end="")

Chat Engine Modes and Runtime State

The API reference separates the chat engine family into a shared type entry point and implementation-specific reference pages. docs/api_reference/api_reference/chat_engines/index.md points at llama_index.core.chat_engine.types, while the implementation pages select named classes from llama_index.core.chat_engine: SimpleChatEngine, ContextChatEngine, CondenseQuestionChatEngine, and CondensePlusContextChatEngine. This organization tells deployers to design against the common chat engine contract where possible, then choose the implementation that matches the retrieval and prompt-management behavior they want.

Sources: docs/api_reference/api_reference/chat_engines/index.md, docs/api_reference/api_reference/chat_engines/simple.md, docs/api_reference/api_reference/chat_engines/context.md, docs/api_reference/api_reference/chat_engines/condense_question.md, docs/api_reference/api_reference/chat_engines/condense_plus_context.md

SimpleChatEngine is the obvious reference point for a minimal conversational service, especially when the app’s primary requirement is managing turns without adding a specialized retrieval transformation. ContextChatEngine signals an implementation that emphasizes conversational answers with contextual information. CondenseQuestionChatEngine and CondensePlusContextChatEngine indicate a common RAG chat pattern: rewrite or condense the current user turn into a clearer retrieval question, optionally combine that with context, then generate the final conversational response. Even when details live in the generated API pages, the class names and reference layout establish the deployable choices.

Sources: docs/api_reference/api_reference/chat_engines/simple.md, docs/api_reference/api_reference/chat_engines/context.md, docs/api_reference/api_reference/chat_engines/condense_question.md, docs/api_reference/api_reference/chat_engines/condense_plus_context.md

The FAQ also mentions data agents with chat engines and says to set the chat mode while initializing the chat engine. That matters for deployments where a conversational frontend is not just retrieving passages, but also dispatching to agent-like behavior. In those applications, chat mode is part of the runtime contract: a request handler should not silently change modes between turns, because the user’s expectations, available tools, and memory behavior may differ by mode. Treat the chosen chat mode as configuration for the deployed endpoint or session.

Sources: docs/src/content/docs/framework/community/faq/chat_engines.md

System-to-Code Mapping

The mapping should guide both code organization and deployment configuration. Keep the route, worker, or application service responsible for request authentication, session lookup, and transport concerns. Keep the chat engine responsible for the conversational LlamaIndex behavior. This separation makes it easier to swap SimpleChatEngine for a context or condensation-based engine without rewriting the whole endpoint. It also keeps operational features such as request logging, rate limiting, and streaming transport independent from the LlamaIndex chat implementation selected by the application.

Sources: docs/api_reference/api_reference/chat_engines/index.md, docs/api_reference/api_reference/chat_engines/simple.md, docs/api_reference/api_reference/chat_engines/context.md, docs/api_reference/api_reference/chat_engines/condense_question.md, docs/api_reference/api_reference/chat_engines/condense_plus_context.md

Compact API Reference

A practical deployment should expose two conceptual operations even if the underlying framework offers more methods: a blocking chat operation and a streaming chat operation. The blocking operation is appropriate for background jobs, short responses, tests, and simple JSON APIs. The streaming operation is appropriate for interactive products where perceived latency matters. The official usage pattern names these methods chat() and stream_chat(), and the API reference pages identify the chat engine family that implements this behavior. Document which operation your endpoint supports before clients integrate with it.

Sources: docs/api_reference/api_reference/chat_engines/index.md, docs/api_reference/api_reference/chat_engines/simple.md, docs/api_reference/api_reference/chat_engines/context.md, docs/api_reference/api_reference/chat_engines/condense_question.md, docs/api_reference/api_reference/chat_engines/condense_plus_context.md

Execution Flow

A typical non-streaming request starts when the client sends a message and a conversation identifier. The application validates the caller, loads or initializes the relevant chat state, resolves the selected chat engine configuration, and calls chat() with the new user message. The chat engine then uses its implementation strategy to produce an answer that reflects the conversation history and, where configured, application data. The response is returned to the caller and the updated state is associated with the same conversation so that the next turn can build on it.

Sources: docs/src/content/docs/framework/community/faq/chat_engines.md, docs/api_reference/api_reference/chat_engines/index.md

A streaming request follows the same state and configuration setup, but response delivery begins before the full answer is complete. The application calls stream_chat(), iterates over the response generator, and forwards chunks to the client transport. The server should still treat the turn as a single conversational update: if a client disconnects midway, the service must decide whether to discard, truncate, or store the partial assistant message. That policy sits outside the API reference, but it is a direct consequence of deploying a stateful chat surface rather than a one-shot query call.

Sources: docs/api_reference/api_reference/chat_engines/index.md, docs/src/content/docs/framework/community/faq/chat_engines.md

Testing Signals and Next Steps

Test chat deployments with sequences, not just single prompts. A useful smoke test asks an initial question, follows up with a pronoun or abbreviated reference, and verifies that the answer reflects prior context. A second test should exercise the configured chat mode when using data-agent-style behavior, because the FAQ calls out chat mode as part of initializing that path. A third test should cover streaming if the endpoint exposes it: confirm chunk delivery, final response completion, client cancellation, and state consistency after the stream ends.

Sources: docs/src/content/docs/framework/community/faq/chat_engines.md

Next, read the broader chat-engines page for chat engine concepts and the sessions page for conversation state design. If your deployment is retrieval-heavy, pair this page with retrievers, query-engines, and response-synthesis so the retrieval and answer-generation phases are explicit. If your deployment adds tools or data agents, continue to agents-overview, agent-configuration, and tools so chat mode, tool access, and user-facing behavior are configured deliberately rather than hidden inside an endpoint.