LLM Integrations
Purpose and Scope
LlamaIndex applications use a large language model as the reasoning and generation layer for retrieval-augmented generation, chat, agents, response synthesis, and evaluation. This page explains how to think about the LLM integration family rather than documenting every provider class individually. The community integrations page places LLMs alongside vector stores, tracing, experiment tracking, structured output, storage, application frameworks, distributed compute, and other integration categories, which is the right mental model: an LLM provider is one replaceable boundary in a larger application stack. Sources: docs/src/content/docs/framework/community/integrations.md
The first decision is not only which model is best, but which operating environment your application must fit. OpenAI, Anthropic, Gemini/Vertex, Mistral, Ollama, Bedrock, Azure OpenAI, Hugging Face, and local model providers differ in authentication, latency, network boundaries, streaming behavior, tool-calling support, cost controls, and deployment constraints. LlamaIndex examples and guides present these providers as interchangeable model modules used by higher-level primitives such as indexes, query engines, chat engines, workflows, and agents, so provider choice should be evaluated against the behavior expected by those primitives.
Relevant Source Files
docs/src/content/docs/framework/community/integrations.md— Defines the community integrations landing page, including the LLMs category and its relationship to LlamaPacks, data loaders, agent tools, observability, structured outputs, storage, application frameworks, distributed compute, and other integrations.docs/src/content/docs/framework/community/integrations/_meta.yml— Places the integrations section in the documentation navigation as a collapsed group labeledIntegrations.docs/src/content/docs/framework/community/integrations/aws_bedrock_agentcore.md— Shows a concrete Bedrock-backed agent example usingBedrockConverse,FunctionAgent, AWS region configuration, AgentCore runtime serving, SSE streaming, sandboxed browser tools, and optional memory.docs/src/content/docs/framework/community/integrations/chatgpt_plugins.md— Documents interoperability with the OpenAI ChatGPT Retrieval Plugin, including/upsert,ChatGPTRetrievalPluginReader,ChatGPTRetrievalPluginIndex, and conversion of LlamaIndexDocumentobjects.docs/src/content/docs/framework/community/integrations/deepeval.md— Shows how evaluator integrations test LLM and RAG outputs with DeepEval metrics and LlamaIndex response objects.docs/src/content/docs/framework/community/integrations/fleet_libraries_context.md— Demonstrates an integration-heavy RAG workflow using OpenAI API keys, Fleet Context embeddings, Pinecone hybrid search, and LlamaIndex vector store integration.
Integration Model
The integrations landing page intentionally groups LLMs with other extension points instead of isolating them as a special subsystem. A LlamaIndex program often combines a loader, node parser, embedding model, vector store, LLM, evaluator, callback or tracing backend, and optionally an agent tool. The page also points users to LlamaHub for data loaders and agent tools, and to a dedicated observability page for tracing integrations. This means LLM selection should be made together with data access, retrieval, storage, and production monitoring choices, not after the rest of the stack is fixed. Sources: docs/src/content/docs/framework/community/integrations.md, docs/src/content/docs/framework/community/integrations/_meta.yml
In practice, most LLM integrations are consumed through higher-level LlamaIndex APIs. A query engine may call the model during response synthesis, a chat engine may use it for conversational turns, and an agent may combine it with tool calls. The official examples list provider-specific notebooks for OpenAI, Anthropic, Bedrock, Gemini/Vertex, Mistral, and Ollama, while the community docs link to the broader LLM module guide. Treat those examples as starting points for provider-specific constructor options, but keep application code organized around LlamaIndex abstractions so provider swaps do not require rewriting retrieval or agent logic.
Provider Selection Guide
Choose OpenAI or Azure OpenAI when you need mature hosted APIs, common examples, and broad ecosystem compatibility. Choose Anthropic when Claude behavior, context window, or model policy is a better fit for your task. Choose Amazon Bedrock when AWS identity, regional deployment, or managed enterprise controls matter; the Bedrock AgentCore guide shows BedrockConverse configured with model and region_name and then passed into a FunctionAgent. Choose Ollama or other local providers when local execution, privacy, or offline development matters more than a hosted managed service. Sources: docs/src/content/docs/framework/community/integrations/aws_bedrock_agentcore.md
Hugging Face and other open-model integrations are useful when you want control over model weights, fine-tuned variants, or custom inference infrastructure. They may require more operational work than hosted APIs, especially around serving, batching, quantization, and hardware capacity, but they fit teams that need model portability. Bedrock, Azure, Vertex, and similar cloud-provider integrations are often selected for governance and identity alignment. Local providers are often selected for developer feedback loops and data-boundary requirements. The common thread is that LlamaIndex lets these choices surface at the model configuration layer while indexes, retrievers, and engines remain application-level objects.
Agents, Tools, and Runtime Integration
LLM integrations become especially important in agent systems because the model must coordinate tool calls, streaming output, memory, and runtime deployment. The Amazon Bedrock AgentCore guide demonstrates this end-to-end: a BedrockConverse LLM is created with an AWS model identifier and region, a browser tool spec is converted with to_tool_list(), and both are supplied to FunctionAgent. The same guide then serves the agent with AgentCoreRuntime.serve(agent) or configures AgentCoreRuntime(agent=agent, stream=True, port=8080, debug=False, memory=memory). Sources: docs/src/content/docs/framework/community/integrations/aws_bedrock_agentcore.md
That example is valuable because it shows where model integration stops and runtime integration begins. BedrockConverse is the model adapter, FunctionAgent is the LlamaIndex agent abstraction, AgentCoreBrowserToolSpec supplies sandboxed tools, and AgentCoreRuntime maps the agent into a deployable service with POST /invocations and GET /ping. When streaming is enabled, the runtime emits event types such as agent_stream, tool_call, tool_result, done, and error. A production provider choice should therefore account for more than completion quality: it must support the interaction style your agent runtime expects.
Adjacent Integration Patterns
Not every integration that touches LLM behavior is itself an LLM provider. The ChatGPT Retrieval Plugin guide shows a retrieval interoperability path where LlamaIndex Document objects can be converted into JSON for a plugin /upsert endpoint, and where ChatGPTRetrievalPluginReader can load documents from a service implementing the plugin API. It also describes a ChatGPTRetrievalPluginIndex backed by a document store implementing the ChatGPT endpoint. This is model-adjacent infrastructure: it helps an LLM access external knowledge through a standardized retrieval service. Sources: docs/src/content/docs/framework/community/integrations/chatgpt_plugins.md
Evaluation and embedding workflows also influence provider choice. The DeepEval guide demonstrates unit tests for LLM and RAG outputs using LLMTestCase, AnswerRelevancyMetric, assert_test, and LlamaIndex-compatible evaluators such as DeepEvalAnswerRelevancyEvaluator, DeepEvalFaithfulnessEvaluator, and DeepEvalContextualRelevancyEvaluator. The Fleet Context guide uses an OpenAI API key, downloads embeddings for LlamaIndex documentation, and builds a Pinecone hybrid-search workflow with PineconeVectorStore(add_sparse_vector=True). These examples reinforce that LLM provider selection should be validated with evaluation and retrieval quality, not only with isolated prompts. Sources: docs/src/content/docs/framework/community/integrations/deepeval.md, docs/src/content/docs/framework/community/integrations/fleet_libraries_context.md
Compact Reference
| Need | Prefer | Source-backed signal |
|---|---|---|
| Hosted general-purpose examples | OpenAI, Anthropic, Gemini/Vertex, Mistral | Official examples list provider notebooks for common LLMs. |
| AWS-managed agents | Bedrock / BedrockConverse | BedrockConverse(model=..., region_name=...) used with FunctionAgent and AgentCoreRuntime. |
| Local development or private execution | Ollama or local model packages | Official examples include Ollama as a supported LLM integration path. |
| Enterprise cloud governance | Azure OpenAI, Bedrock, Vertex-style providers | Provider configuration usually aligns with cloud identity, regions, and managed infrastructure. |
| Retrieval service interoperability | ChatGPT Retrieval Plugin | /upsert, ChatGPTRetrievalPluginReader, and ChatGPTRetrievalPluginIndex bridge documents and retrieval APIs. |
| Quality gates for model changes | DeepEval and LlamaIndex evaluators | evaluate_response(query=..., response=...) checks response objects from RAG applications. |
Practical Next Steps
Start with the provider example closest to your deployment target, then wire it through the smallest LlamaIndex primitive that exercises your real workload: a query engine for RAG, a chat engine for conversation, or FunctionAgent for tool use. Add evaluation early so provider swaps can be measured with the same queries and retrieved context. If you are building an agent, confirm streaming, tool-call events, memory, and runtime deployment behavior before committing to a provider. Next, read the pages on models-settings, query-engines, chat-engines, agents-overview, tools, streaming, and evaluating to see where the selected LLM participates in the full application lifecycle.