Evals

Purpose and Scope

Evals are the measurement layer for agent and LLM application quality. In LangSmith terminology, evaluations break down what “good” means for a system and turn that definition into repeatable checks over examples, runs, and production traces. In this repository, the Python code exposes two complementary surfaces: modern LangSmith-aware runtime evaluation hooks in langchain_core, and classic evaluation chains in langchain_classic for grading strings, comparing outputs, checking criteria, and evaluating agent trajectories. Together, these surfaces let developers test changes before deployment and monitor behavior after release.

Sources: libs/core/langchain_core/tracers/evaluation.py, libs/langchain/langchain_classic/evaluation/init.py

The official LangSmith evaluation flow is organized around three concepts: a dataset of inputs and optional references, a target function or application path to exercise, and evaluators that score the resulting outputs or traces. The repository reflects that model directly. LangSmithLoader can load LangSmith dataset examples as LangChain Document objects, while EvaluatorCallbackHandler runs LangSmith run evaluators when traced runs complete. Classic evaluators remain available for direct, local scoring workflows, especially where a developer wants a chain-style evaluator with explicit prediction, input, and reference values.

Sources: libs/core/langchain_core/document_loaders/langsmith.py, libs/core/langchain_core/tracers/evaluation.py, libs/langchain/langchain_classic/evaluation/init.py

Core Primitives

A dataset is the curated set of examples used to define expected behavior. Official guidance recommends starting with a small number of manually curated examples for each critical component, such as retrieval quality in a RAG system, tool selection in an agent, or response quality in a chatbot. LangSmithLoader provides a LangChain-native way to consume those examples: it loads example inputs into Document.page_content and stores the full example, including inputs and outputs, in Document.metadata. This is useful when examples should become few-shot context, retrieval material, or test fixtures inside a LangChain workflow.

Sources: libs/core/langchain_core/document_loaders/langsmith.py

An evaluator is the scoring function or chain that decides whether a target behaved well. At runtime, EvaluatorCallbackHandler accepts a sequence of langsmith.RunEvaluator instances and applies them to top-level traced runs. In classic LangChain, evaluators are exposed through load_evaluator and load_evaluators, and the module documentation names concrete evaluator families for QA, pairwise model comparison, criteria checks, embedding distance, string distance, exact matching, and agent trajectory evaluation. These are different interfaces for the same engineering task: converting subjective expectations into repeatable measurements.

Sources: libs/core/langchain_core/tracers/evaluation.py, libs/langchain/langchain_classic/evaluation/init.py

A run is the execution record being evaluated. For offline evaluations, runs are usually generated by applying a target function to dataset examples. For online evaluations, runs come from production traffic and tracing. EvaluatorCallbackHandler is designed for the latter shape: it subclasses BaseTracer, has the tracer name evaluator_callback_handler, and schedules evaluation work after runs are persisted. It also supports an example_id, allowing evaluation results to be associated with a dataset example when the run originated from one.

Sources: libs/core/langchain_core/tracers/evaluation.py

Relevant Source Files

• libs/core/langchain_core/document_loaders/langsmith.py — Defines LangSmithLoader, the core loader for converting LangSmith dataset examples into LangChain Document objects, with filtering, split selection, versioning, pagination, and custom content formatting options.
• libs/core/langchain_core/tracers/evaluation.py — Defines EvaluatorCallbackHandler and wait_for_all_evaluators, the runtime tracing integration that runs LangSmith evaluators over completed runs.
• libs/langchain/langchain_classic/callbacks/tracers/evaluation.py — Re-exports the core evaluation tracer types from the classic callback tracer namespace for compatibility.
• libs/langchain/langchain_classic/chat_loaders/langsmith.py — Provides deprecated dynamic imports for LangSmith chat loaders that now live in langchain_community.chat_loaders.langsmith.
• libs/langchain/langchain_classic/evaluation/__init__.py — Documents and exports classic evaluation chains, loader helpers, evaluator types, and common evaluation use cases.
• libs/langchain/langchain_classic/evaluation/agents/__init__.py — Exports TrajectoryEvalChain for evaluating ReAct-style agent trajectories.

System-to-Code Mapping

LangSmith evaluation concepts map cleanly onto these repository modules. Datasets map to LangSmithLoader when examples need to be loaded into a LangChain application as documents. Runtime evaluators map to EvaluatorCallbackHandler, which receives langsmith.RunEvaluator objects, owns a LangSmith client, and manages executor-backed asynchronous evaluation work. Classic evaluator chains map to the exports in langchain_classic.evaluation, where developers can load or instantiate evaluators for strings, comparisons, criteria, embeddings, exact match, and agent behavior. The compatibility re-export keeps older callback imports working while centralizing implementation in langchain_core.

Sources: libs/core/langchain_core/document_loaders/langsmith.py, libs/core/langchain_core/tracers/evaluation.py, libs/langchain/langchain_classic/callbacks/tracers/evaluation.py, libs/langchain/langchain_classic/evaluation/init.py

The agent-specific path is important because agent quality is not only final-answer quality. Official LangSmith guidance calls out final response, trajectory, and single-step evaluation for complex agents. The classic package exposes TrajectoryEvalChain from langchain_classic.evaluation.agents, described as chains for evaluating ReAct-style agents. That makes trajectory scoring a first-class evaluation mode alongside simpler string scoring. For agents that call tools, the trajectory can capture whether the system selected the correct tool, passed valid arguments, and followed the expected path before producing its final answer.

Sources: libs/langchain/langchain_classic/evaluation/init.py, libs/langchain/langchain_classic/evaluation/agents/init.py

Execution Flow

A typical offline evaluation begins by defining a dataset of examples, then running a target function over those examples, and finally applying evaluators to the outputs. If examples already live in LangSmith, LangSmithLoader can retrieve them by dataset_id, dataset_name, explicit example_ids, as_of version tag or timestamp, splits, metadata, filter expression, offset, and limit. Its content_key option selects which nested input field becomes document content, and format_content controls conversion to text. Those options let the same dataset support regression tests, few-shot retrieval, and targeted component checks.

Sources: libs/core/langchain_core/document_loaders/langsmith.py

A typical online or trace-based evaluation uses the callback handler. EvaluatorCallbackHandler is initialized with evaluators, an optional LangSmith client, an optional example id, skip_unfinished, a project_name, and max_concurrency. By default it obtains a client through the LangChain tracer integration and uses the shared executor; with a positive max_concurrency it creates a bounded ThreadPoolExecutor; with non-positive concurrency it can run without an executor. The helper wait_for_all_evaluators() iterates over active evaluator tracers and waits for their futures, which is useful before process shutdown or test completion.

Sources: libs/core/langchain_core/tracers/evaluation.py

API Components

LangSmithLoader is the dataset bridge. Its constructor accepts dataset identity fields, example filters, version selectors, split selectors, pagination controls, metadata and structured filter constraints, plus a client or client keyword arguments. The implementation explicitly rejects providing both a client and client keyword arguments, because only one LangSmith client construction path should be used. By default, it JSON-encodes extracted input content, but callers can supply a formatter when examples contain messages, tool arguments, or other nested values that should be represented differently.

Sources: libs/core/langchain_core/document_loaders/langsmith.py

EvaluatorCallbackHandler is the runtime bridge. Its public attributes include example_id, client, evaluators, executor, futures, skip_unfinished, project_name, logged_eval_results, and a lock. skip_unfinished defaults to true, matching the practical expectation that failed or incomplete runs often should not receive normal quality scores. project_name defaults to evaluators, giving evaluation-chain runs an organized destination. The classic tracer module exposes the same EvaluatorCallbackHandler and wait_for_all_evaluators names, so older imports can move forward without changing behavior.

Sources: libs/core/langchain_core/tracers/evaluation.py, libs/langchain/langchain_classic/callbacks/tracers/evaluation.py

Classic evaluation APIs are useful when the target is not a traced run but a value you already have in memory. The package documentation shows load_evaluator("qa") followed by evaluate_strings(prediction=..., input=..., reference=...). It also documents the low-level evaluator interfaces: StringEvaluator for a prediction with optional input and reference, PairwiseStringEvaluator for comparing two predictions, and AgentTrajectoryEvaluator for the full sequence of agent actions. These interfaces are the source-level contract behind the higher-level evaluator names.

Sources: libs/langchain/langchain_classic/evaluation/init.py

Implementation Notes and Next Steps

When choosing an evaluation strategy, start with the artifact you need to score. Use classic string and criteria evaluators for direct answer grading, pairwise evaluators for model or prompt comparisons, embedding or string distance evaluators for similarity measurements, and trajectory evaluation for agent tool-use paths. Use the LangSmith tracer handler when the important artifact is a completed run with trace context, not just a single string. Use LangSmithLoader when dataset examples should flow back into LangChain as documents for retrieval, prompting, or test construction.

Sources: libs/core/langchain_core/document_loaders/langsmith.py, libs/core/langchain_core/tracers/evaluation.py, libs/langchain/langchain_classic/evaluation/init.py

For a practical next step, create or select a small LangSmith dataset, decide whether you are testing final responses, trajectories, or single steps, and attach evaluators where the run data is most complete. If you are building an agent, include at least one trajectory-oriented check so tool selection and argument formatting are measured separately from final wording. If you are modernizing older code, replace deprecated LangSmith chat-loader imports with their community package locations while keeping classic evaluation imports where chain-style scoring is still appropriate.

Sources: libs/langchain/langchain_classic/chat_loaders/langsmith.py, libs/langchain/langchain_classic/evaluation/agents/init.py