Models, Embeddings, and Settings

Purpose and Scope

The model layer in LlamaIndex is the place where an application chooses how language understanding, text generation, embedding, token accounting, callbacks, and prompt sizing should behave. The Settings guide defines this layer through a singleton settings object that supplies commonly used resources during indexing and querying, while still allowing local overrides at specific interfaces. In practice, this means a simple application can configure defaults once, and a more advanced application can replace the language model, embedding model, splitter, or transformations for one index, retriever, query engine, or agent path when needed.

Sources: docs/src/content/docs/framework/module_guides/supporting_modules/settings.mdx

This page is for readers deciding where to configure LLMs and embeddings in a LlamaIndex application. It explains the boundary between global defaults and local configuration, then connects those choices to agents and MCP-backed tools because those systems also consume an LLM and tool definitions at runtime. The official component guide navigation places model guides, loading guides, indexing guides, storing guides, and Settings in the same framework documentation area, so treating model configuration as a shared foundation helps keep RAG, chat, and agent examples understandable rather than isolated.

Sources: docs/src/content/docs/framework/module_guides/_meta.yml, docs/src/content/docs/framework/module_guides/supporting_modules/settings.mdx

Relevant Source Files

• docs/src/content/docs/framework/module_guides/supporting_modules/settings.mdx - Defines Settings as the global configuration bundle and lists configurable attributes for LLMs, embedding models, text splitting, transformations, tokenizer behavior, callbacks, and prompt helper values.
• docs/src/content/docs/framework/module_guides/mcp/llamaindex_mcp.md - Shows how MCP servers are converted into LlamaIndex tool definitions and then supplied to an agent that has an explicit OpenAI LLM instance.
• docs/src/content/docs/framework/module_guides/_meta.yml - Places the component guides under the framework documentation navigation, which is the documentation family containing Settings and model-related guides.
• docs/src/content/docs/framework/module_guides/deploying/_meta.yml - Identifies the deploying documentation group that contains runtime-oriented agent material.
• docs/src/content/docs/framework/module_guides/deploying/agents/_meta.yml - Identifies the deploying agents subgroup used by the agent deployment guide.
• docs/src/content/docs/framework/module_guides/deploying/agents/index.mdx - Defines an agent as a system using an LLM, memory, and tools, and shows FunctionAgent construction with OpenAI, tools, prompts, memory, and streaming notes.

Core Primitives

Settings is the global default registry for frequently used application resources. The Settings guide describes it as a simple singleton that lives throughout the application and is consulted when a component is not given a more specific value. The most important model primitives are the LLM, which responds to prompts and writes natural language answers, and the embedding model, which converts text into numerical representations for similarity search and top-k retrieval. These two choices usually determine both answer style and retrieval behavior, so they should be treated as first-class application configuration rather than incidental example code.

Sources: docs/src/content/docs/framework/module_guides/supporting_modules/settings.mdx

The same settings object also coordinates supporting primitives that affect model behavior indirectly. A node parser or text splitter converts documents into smaller nodes before indexing. Transformations run during ingestion and can replace or extend the default splitting behavior. A tokenizer counts tokens and should match the language model being used, while prompt helper values such as context window and number of output tokens reserve space for model input and generation. Callback managers observe events across LlamaIndex, making model and retrieval behavior measurable during development and production debugging.

Sources: docs/src/content/docs/framework/module_guides/supporting_modules/settings.mdx

Configuration Flow

A typical configuration flow begins by choosing a language model integration, then choosing an embedding integration, and then deciding whether chunking defaults are sufficient for the source data. The Settings guide demonstrates assigning an OpenAI chat model as the global LLM and an OpenAI embedding model as the global embedding provider. Once those defaults are installed, index construction and query execution can omit repeated model arguments unless they need special behavior. This is especially useful in tutorials and small applications, where repeating provider setup across every call would obscure the data loading, indexing, and querying steps.

Sources: docs/src/content/docs/framework/module_guides/supporting_modules/settings.mdx

Local configuration is the escape hatch when one part of the application needs different behavior. The Settings guide explicitly notes that local configurations, including transformations, LLMs, and embedding models, can be passed directly into the interfaces that use them. A common pattern is to keep a production embedding model as the global default for most vector indexes, while passing a different embedding model or transformation list to an experimental index. Another pattern is to keep a fast, inexpensive LLM globally, then provide a stronger LLM for a specialized query engine or agent workflow that requires better reasoning.

Sources: docs/src/content/docs/framework/module_guides/supporting_modules/settings.mdx

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.node_parser import SentenceSplitter
 
Settings.llm = OpenAI(model="gpt-3.5-turbo", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-small", embed_batch_size=100
)
Settings.text_splitter = SentenceSplitter(chunk_size=1024)
Settings.chunk_overlap = 20

API and Configuration Reference

These configuration names matter because they are shared vocabulary across the framework. When an index guide says retrieval uses embeddings, it is referring to the embedding model chosen globally or supplied locally. When a query or chat component sends prompts, it ultimately depends on the configured LLM and on token limits derived from the model or prompt helper overrides. When ingestion behavior changes, the node parser or transformation list changes what text reaches the index. Understanding these knobs helps diagnose issues such as poor recall, excessive chunk fragmentation, truncated prompts, or unexpected callback output.

Sources: docs/src/content/docs/framework/module_guides/supporting_modules/settings.mdx

Agents, Tools, and MCP Connections

Agent documentation reinforces that model configuration is not limited to classic RAG pipelines. The deploying agents guide defines an agent as a system that uses an LLM, memory, and tools to handle outside user inputs. Its example constructs a FunctionAgent with a list of tools, an OpenAI LLM, and a system prompt. That explicit LLM argument is a local model choice for the agent workflow, and it can differ from any global default used elsewhere. The guide also notes that some models may not support streaming output, so agent configuration can include disabling streaming when a provider cannot satisfy that behavior.

Sources: docs/src/content/docs/framework/module_guides/deploying/agents/index.mdx, docs/src/content/docs/framework/module_guides/deploying/agents/_meta.yml, docs/src/content/docs/framework/module_guides/deploying/_meta.yml

MCP tools add another important distinction: tools may come from an external connection rather than local Python functions. The MCP guide shows installing the MCP tools package, creating a BasicMCPClient, converting an MCP server into a tool list with McpToolSpec, and then passing those tools into a FunctionAgent with an OpenAI model. The supported connection forms include Server-Sent Events, streamable HTTP, and a local process. This means model selection and tool connectivity are separate decisions: the LLM reasons and calls tools, while MCP provides a transport-backed source of tool definitions and tool execution.

Sources: docs/src/content/docs/framework/module_guides/mcp/llamaindex_mcp.md, docs/src/content/docs/framework/module_guides/deploying/agents/index.mdx

from llama_index.tools.mcp import BasicMCPClient, McpToolSpec
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI
 
mcp_client = BasicMCPClient("http://127.0.0.1:8000/sse")
mcp_tool_spec = McpToolSpec(client=mcp_client)
tools = await mcp_tool_spec.to_tool_list_async()
 
agent = FunctionAgent(
    tools=tools,
    llm=OpenAI(model="gpt-5-mini"),
    system_prompt="You are a helpful assistant.",
)

Implementation Guidance and Edge Cases

Use global Settings for application-wide defaults, especially when many indexes, query engines, or examples should share one provider configuration. Prefer local overrides when the component is experimental, tenant-specific, latency-sensitive, or requires a model with different capabilities. Keep tokenizer and prompt helper settings aligned with the selected LLM, because token accounting influences how much retrieved context can be included before generation. For retrieval quality, treat the embedding model and text splitter as a pair: embeddings decide similarity space, while chunking decides what semantic units are embedded and later retrieved.

Sources: docs/src/content/docs/framework/module_guides/supporting_modules/settings.mdx

For agents, verify model capabilities before assuming a runtime feature works everywhere. The agent guide calls out streaming as enabled by default but potentially unsupported by some models, with an option to disable it on FunctionAgent. Also distinguish memory from Settings: the agent guide describes ChatMemoryBuffer as the default memory and shows passing a custom memory object at run time. That memory controls conversational state, while Settings or explicit LLM arguments control model behavior. Keeping those concerns separate makes it easier to debug whether a surprising answer came from model choice, prompt history, tool output, or retrieved context.

Sources: docs/src/content/docs/framework/module_guides/deploying/agents/index.mdx

Next Steps

After configuring the model layer, continue with pages that use these primitives directly. Read Documents and Nodes before tuning chunk size or transformations, Vector Store Indexing before changing embedding providers for semantic retrieval, Query Engines and Chat Engines before choosing response behavior, and Agents Overview before wiring LLMs to tools and memory. If your tools live outside the Python process, read MCP Tools next so you can decide whether to use local function tools, predefined tool specs, or MCP connections over Server-Sent Events, streamable HTTP, or a local server process.

Sources: docs/src/content/docs/framework/module_guides/supporting_modules/settings.mdx, docs/src/content/docs/framework/module_guides/mcp/llamaindex_mcp.md