LlamaDeploy Runtime

Purpose and Scope

This page explains the practical runtime model a LlamaIndex developer needs when moving from a local RAG prototype to an operated application. In this context, “runtime” means the code path that receives user input, invokes LlamaIndex abstractions, manages conversational or agent state, streams output when requested, and returns a response to the caller. The source-backed deployment surfaces here are agents, chat engines, and query engines. They are distinct public interfaces, but they share an application shape: prepare data and indexes, expose an interaction endpoint, then run natural-language requests through LlamaIndex components backed by LLMs, tools, memory, retrieval, or response synthesis.

Sources: docs/src/content/docs/framework/module_guides/deploying/agents/index.mdx, docs/src/content/docs/framework/module_guides/deploying/chat_engines/index.mdx, docs/src/content/docs/framework/module_guides/deploying/query_engine/index.mdx

The page also covers llama-dev, the official development, testing, and automation CLI for the LlamaIndex monorepo. llama-dev is not the application runtime that serves user traffic; it is an operator and contributor tool for working with packages in the repository. That distinction matters in day-to-day work. Use the deployment abstractions to build and host applications, and use llama-dev when you need to inspect packages, run package commands, or test changed packages and dependents before shipping source changes.

Sources: llama-dev/README.md

Relevant Source Files

• docs/src/content/docs/framework/module_guides/deploying/agents/index.mdx — Defines the deployable agent concept, the FunctionAgent example, the agent action loop, tools, memory, multimodal messages, and the streaming note for agent output.
• docs/src/content/docs/framework/module_guides/deploying/chat_engines/index.mdx — Defines chat engines as stateful interfaces for conversations with data, shows index.as_chat_engine(), and documents chat and stream_chat usage patterns.
• docs/src/content/docs/framework/module_guides/deploying/query_engine/index.mdx — Defines query engines as generic interfaces for asking questions over data, shows index.as_query_engine(), and documents streaming query usage.
• llama-dev/README.md — Documents the llama-dev CLI, installation with uv, package inspection and execution commands, smart test selection, coverage options, parallel workers, and repository-root guidance.

Runtime Model

LlamaIndex deployment starts by choosing the interaction contract your application should expose. A query engine is the simplest request-response surface: it accepts a natural-language question and returns a rich response. The query engine is most often built on one or more indexes through retrievers, and the deployment documentation explicitly calls out that multiple query engines can be composed for more advanced capability. This makes query engines a natural fit for API endpoints where each request is intended to stand on its own, such as search, document Q&A, report lookup, or a backend route that answers one scoped question at a time.

Sources: docs/src/content/docs/framework/module_guides/deploying/query_engine/index.mdx

A chat engine adds a conversation runtime on top of data access. The chat engine guide describes it as a high-level interface for having a conversation with your data, with multiple back-and-forth turns rather than a single question and answer. It is a stateful analogy of a query engine because it keeps track of conversation history and can answer with past context in mind. When a product needs user-visible continuity, follow-up questions, or session-aware chat behavior, the chat engine is the better deployment surface than a stateless query endpoint.

Sources: docs/src/content/docs/framework/module_guides/deploying/chat_engines/index.mdx

An agent runtime is broader than both query and chat engines because it can decide whether to answer directly or call tools. The deployment guide defines an agent as a specific system that uses an LLM, memory, and tools to handle inputs from outside users. It also contrasts “agent” with “agentic”: agentic systems are a broader class of LLM decision-making systems, while an agent is a concrete runtime with these components. The example uses FunctionAgent, which relies on a provider’s function or tool-calling capability to execute Python tools and continue until a direct response is produced.

Sources: docs/src/content/docs/framework/module_guides/deploying/agents/index.mdx

Deployment Surfaces

Use a query engine when the runtime should treat each input as a standalone question over indexed data. The guide’s minimal pattern is to call index.as_query_engine() and then invoke query_engine.query(...). Streaming is configured at construction time with index.as_query_engine(streaming=True), after which the response can print its stream. In production terms, this maps cleanly to a request handler that builds or loads an index-backed engine, validates the incoming query, calls query, and serializes the rich response or streamed tokens back to the client.

query_engine = index.as_query_engine()
response = query_engine.query("Who is Paul Graham.")
 
query_engine = index.as_query_engine(streaming=True)
streaming_response = query_engine.query("Who is Paul Graham.")
streaming_response.print_response_stream()

Sources: docs/src/content/docs/framework/module_guides/deploying/query_engine/index.mdx

Use a chat engine when the runtime needs conversation state. The guide’s minimal pattern is index.as_chat_engine() followed by chat_engine.chat(...). For streaming chat, the deployed path calls chat_engine.stream_chat(...) and iterates over streaming_response.response_gen. That generator-oriented interface is important for web sockets, server-sent events, terminal clients, or any UI that should display text as it arrives. The deployment choice is not only about transport; it changes the semantics because the chat engine is expected to maintain history across turns.

chat_engine = index.as_chat_engine()
response = chat_engine.chat("Tell me a joke.")
 
chat_engine = index.as_chat_engine()
streaming_response = chat_engine.stream_chat("Tell me a joke.")
for token in streaming_response.response_gen:
    print(token, end="")

Sources: docs/src/content/docs/framework/module_guides/deploying/chat_engines/index.mdx

Use an agent when the runtime needs tool use, memory, or LLM-driven control flow. The documented FunctionAgent example creates a calculator tool as a Python function, passes it in tools=[multiply], selects OpenAI(model="gpt-4o-mini"), and supplies a system_prompt. Running the agent is asynchronous: response = await agent.run(...). That async shape is a signal for deployment design. An application server should treat the agent as a coroutine-based workflow, await its result, and plan for intermediate tool executions and potential streaming behavior rather than assuming a single synchronous LLM completion.

import asyncio
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI
 
def multiply(a: float, b: float) -> float:
    """Useful for multiplying two numbers."""
    return a * b
 
agent = FunctionAgent(
    tools=[multiply],
    llm=OpenAI(model="gpt-4o-mini"),
    system_prompt="You are a helpful assistant that can multiply two numbers.",
)
 
async def main():
    response = await agent.run("What is 1234 * 4567?")
    print(str(response))
 
if __name__ == "__main__":
    asyncio.run(main())

Sources: docs/src/content/docs/framework/module_guides/deploying/agents/index.mdx

Agent Loop, Memory, and Streaming

The agent guide describes a concrete execution loop. The agent gets the latest message and chat history, sends tool schemas and chat history over the API, then receives either a direct response or a list of tool calls. Each tool call is executed, the results are appended to chat history, and the agent is invoked again with the updated history. This loop repeats until the agent responds directly. Runtime operators should treat tools as part of the trusted execution boundary: tools may call external APIs, perform calculations, retrieve data, or modify state, so they deserve the same validation and observability as any other backend operation.

Sources: docs/src/content/docs/framework/module_guides/deploying/agents/index.mdx

Memory is a first-class agent runtime concern. The deployment guide states that all LlamaIndex agents use ChatMemoryBuffer by default, and shows how to customize memory by creating ChatMemoryBuffer.from_defaults(token_limit=40000) and passing it to agent.run(..., memory=memory). This means agent state can be supplied per run instead of being hidden inside the agent definition. In a hosted deployment, that gives you a practical boundary for session storage, tenant isolation, retention policy, and token-budget control. The agent object defines behavior; the memory object can represent a specific conversation context.

from llama_index.core.memory import ChatMemoryBuffer
 
memory = ChatMemoryBuffer.from_defaults(token_limit=40000)
response = await agent.run(..., memory=memory)

Sources: docs/src/content/docs/framework/module_guides/deploying/agents/index.mdx

Streaming should be treated as an explicit runtime capability, not an incidental output format. Query engines enable streaming through the streaming=True option on as_query_engine; chat engines expose stream_chat and a response_gen; agents have streaming enabled by default in the FunctionAgent guide, with an escape hatch of FunctionAgent(..., streaming=False) for models that do not support streaming LLM output. The consistent operational lesson is to decide at the boundary whether clients receive complete responses or incremental output, then configure the chosen LlamaIndex surface accordingly.

Sources: docs/src/content/docs/framework/module_guides/deploying/agents/index.mdx, docs/src/content/docs/framework/module_guides/deploying/chat_engines/index.mdx, docs/src/content/docs/framework/module_guides/deploying/query_engine/index.mdx

Monorepo Development CLI

llama-dev is the source-tree companion for developers and maintainers working in the LlamaIndex monorepo. The README calls it the official CLI for development, testing, and automation, with commands for managing packages, running tests, and automating common development tasks. Installation begins with a virtual environment created by uv venv, activation through source .venv/bin/activate, and editable installation with uv pip install -e .. After installation, llama-dev --help should be available on the path.

uv venv
source .venv/bin/activate
uv pip install -e .
llama-dev --help

Sources: llama-dev/README.md

Package commands are useful when runtime work requires source-level changes across llama-index-core or integration packages. The README recommends running llama-dev from the repository root; otherwise, callers must pass --repo-root on every command. llama-dev pkg info llama-index-core inspects a specific package, while llama-dev pkg info --all lists all packages. llama-dev pkg exec --cmd "uv sync" llama-index-core runs a command in one package, and llama-dev pkg exec --cmd "uv sync" --all applies it across packages. --fail-fast stops at the first error, which is useful in CI-like local validation.

llama-dev pkg info llama-index-core
llama-dev pkg info --all
llama-dev pkg exec --cmd "uv sync" llama-index-core
llama-dev pkg exec --cmd "uv sync" --all
llama-dev pkg exec --cmd "uv" --all --fail-fast

Sources: llama-dev/README.md

Testing commands support the feedback loop before deployment changes land. llama-dev test --base-ref main runs tests for packages changed compared to main, and the README notes that the tool detects changed packages and their dependents so only needed tests run. Coverage can be enabled with --cov, enforced with --cov-fail-under 80, parallelized with --workers 4, and stopped early with --fail-fast. The README also records core requirements: Python 3.10 or newer, the uv package manager, and Git. Together, these commands form the development runtime for validating source modifications before application deployment.

llama-dev test --base-ref main
llama-dev test --base-ref main --cov
llama-dev test --base-ref main --cov --cov-fail-under 80
llama-dev test --base-ref main --workers 4
llama-dev test --base-ref main --fail-fast

Sources: llama-dev/README.md

Compact Reference

Execution Flow and Next Steps

A practical deployment flow is to first decide whether the user experience is standalone Q&A, stateful chat, or agentic tool use. Next, build or load the index and supporting model configuration required by that surface. Then expose a boundary that matches the surface: a query endpoint for query, a conversation endpoint for chat or stream_chat, or an async workflow boundary for agent.run. Finally, validate source changes with llama-dev before promoting the application. If you are designing user-facing sessions, read the sessions and streaming pages next; if you are designing tool-using systems, continue with agents, tools, and workflow pages.