Callbacks and Observability

Purpose and Scope

Observability is the practice of recording enough structured runtime information to understand what an LLM application did, why it did it, and where it spent time. In the LangChain ecosystem, the first-party observability path is LangSmith tracing. LangSmith describes a project as a container for traces, a trace as the complete record for one operation, a run as an individual step such as an LLM call or retrieval call, and a thread as multiple related traces in a conversation. That vocabulary is useful when reading callback output, tracing agent behavior, or debugging a production application because it gives every unit of work a consistent place in the hierarchy.

Callbacks are the application-side hooks that let a LangChain program report lifecycle events such as model calls, tool calls, chain steps, and errors. Tracing hooks turn those events into durable records, while local handlers such as stdout-style logging are useful for quick inspection during development. Usage tracking is the complementary concern of measuring provider cost signals, token counts, latency, and other metadata that help teams compare prompts, models, and agent paths. The practical rule is to start with tracing early, then add local or custom handlers only where they clarify a specific development workflow.

The repository evidence for this page sits at an important boundary: classic compatibility modules that redirect or block Python REPL-related imports and delegate older Python loader imports through dynamic importers. Those files do not define observability handlers themselves, but they do define behavior that matters when observing agents that may call tools. A trace can show that an agent attempted to use a Python capability; the compatibility modules then determine whether that import resolves through a deprecated path, proxies to a community package, or raises a security-oriented migration error. Sources: libs/langchain/langchain_classic/agents/agent_toolkits/python/init.py, libs/langchain/langchain_classic/tools/python/init.py, libs/langchain/langchain_classic/python.py, libs/langchain/langchain_classic/utilities/python.py

Relevant Source Files

• libs/langchain/langchain_classic/agents/agent_toolkits/python/__init__.py - Defines the classic Python agent toolkit import boundary and raises an ImportError for create_python_agent, directing users to langchain_experimental and warning that the underlying Python REPL should be sandboxed.
• libs/langchain/langchain_classic/document_loaders/parsers/language/python.py - Exposes the legacy PythonSegmenter name through a dynamic deprecated import lookup into langchain_community, preserving older import paths for Python source parsing workflows.
• libs/langchain/langchain_classic/document_loaders/python.py - Exposes the legacy PythonLoader name through the same dynamic importer pattern, keeping Python document-loading imports compatible while centralizing deprecation behavior.
• libs/langchain/langchain_classic/python.py - Provides backwards compatibility for PythonREPL by delegating lookup to langchain_community.utilities.python, while keeping it out of __all__ so it is not advertised as a normal import surface.
• libs/langchain/langchain_classic/tools/python/__init__.py - Blocks classic Python tool imports with an AttributeError that explains the move to langchain_experimental and repeats the sandboxing requirement for REPL access.
• libs/langchain/langchain_classic/utilities/python.py - Mirrors the classic PythonREPL compatibility behavior from the utilities namespace, using create_importer and a deprecated lookup to the community package.

Observability Model

A useful mental model is to treat every user request as a trace and every meaningful internal operation as a run. If a request formats a prompt, retrieves documents, calls a chat model, parses structured output, and invokes a tool, each step should be visible as part of the same trace. When a chat application has multiple turns, each turn can become its own trace while a shared session_id or thread_id groups those traces into a thread. This mirrors the LangSmith observability docs and lets developers move from a high-level conversation view down to the exact run that caused an unexpected answer.

For LangChain and LangGraph applications, the official quickstart emphasizes enabling tracing through environment configuration rather than adding ad hoc print statements everywhere. A minimal development setup installs the necessary client package, sets LANGSMITH_TRACING=true, provides LANGSMITH_API_KEY, and optionally sets LANGSMITH_PROJECT to route traces to a named project. If the account is outside the default region, LANGSMITH_ENDPOINT must point at the regional API endpoint. This environment-first setup is especially valuable for agents because tool, model, and parser events can be correlated without changing business logic for every experiment.

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="<your-langsmith-api-key>"
export LANGSMITH_PROJECT="my-agent-project"

Callbacks and stdout handlers still have a role. A stdout handler gives immediate feedback while developing a chain or agent locally, and custom callbacks can attach domain-specific metadata, counters, or feedback collection to runs. In production, however, the durable trace is the more useful artifact: it supports filtering, run comparison, feedback, latency analysis, and debugging after the original process has exited. The official LangSmith tutorial frames observability as something to add during prototyping, not after launch, because early traces show exactly what prompts, documents, and model outputs are being produced as the application evolves.

System-to-Code Mapping

The Python compatibility files show how runtime behavior should be interpreted when old imports appear in traces or callback logs. The agent toolkit module implements __getattr__ and specially handles create_python_agent. Instead of returning a classic implementation, it raises an ImportError explaining that the agent moved to langchain_experimental, that it relies on a Python REPL tool under the hood, and that the REPL should be sandboxed. In an observed agent run, that failure is not just an import problem; it is a safety boundary that should be preserved when migrating legacy code. Sources: libs/langchain/langchain_classic/agents/agent_toolkits/python/init.py

The classic Python tool package applies the same safety posture at the tool namespace. Its __getattr__ always raises an AttributeError describing that the tool moved to langchain_experimental, that it has access to a Python REPL, and that best practices require sandboxing. This matters for observability because tool execution is one of the most important events to trace in an agent. If a trace or callback stream shows attempts to load a Python tool from the classic namespace, the correct remediation is not to suppress the error; it is to migrate deliberately and evaluate the sandboxing model before allowing code execution. Sources: libs/langchain/langchain_classic/tools/python/init.py

The document loader and parser modules use a different pattern. document_loaders/python.py maps PythonLoader to langchain_community.document_loaders.python, and document_loaders/parsers/language/python.py maps PythonSegmenter to langchain_community.document_loaders.parsers.language.python. Both create _import_attribute through create_importer and use module-level __getattr__ for dynamic lookup. For observed retrieval or indexing pipelines, this means an old import can still resolve through a deprecation-aware path. Traces can help distinguish import-time compatibility warnings from failures in downstream loading, parsing, splitting, embedding, or retrieval. Sources: libs/langchain/langchain_classic/document_loaders/python.py, libs/langchain/langchain_classic/document_loaders/parsers/language/python.py

The langchain_classic.python and langchain_classic.utilities.python modules are explicitly marked for backwards compatibility. Both configure a deprecated lookup for PythonREPL into langchain_community.utilities.python, and comments state that the code has also been removed from the community package while the proxy will raise the appropriate exception. They also avoid listing the name as importable through __all__. When a trace records a failure around PythonREPL, developers should read it as a compatibility and migration signal rather than as a normal tool execution failure. Sources: libs/langchain/langchain_classic/python.py, libs/langchain/langchain_classic/utilities/python.py

Execution Flow for Debugging an Agent

Start debugging from the outermost operation. In LangSmith terms, find the trace for the user request, then inspect the sequence of runs. If the trace contains a model run with an unexpected prompt, the issue may be instruction construction or context assembly. If it contains a retriever or loader run with poor inputs, the issue may be indexing or document preparation. If it contains a tool call or import error involving Python execution, compare the event to the classic Python compatibility behavior described above and decide whether the code should be migrated, removed, or isolated behind a sandboxed experimental dependency.

Next, add metadata that makes traces filterable. The official docs recommend linking multi-turn work with session_id or thread_id; the same idea applies to experiments, tenants, model variants, or release versions. Usage tracking should be attached at the same level of abstraction where decisions are made. For example, model token counts belong on model runs, while overall latency and success state belong on the trace or top-level chain run. Feedback should be collected against the run that a human or evaluator is actually judging so later analysis can connect quality signals to specific prompts, tools, and model choices.

Finally, keep local logging and durable tracing in balance. Stdout-style callbacks are fast and readable while iterating in a terminal, but they are ephemeral and hard to aggregate. Traces are structured, queryable, and more appropriate for production debugging. For risky tools, especially anything that can execute Python, observability should be paired with policy: the trace should reveal the attempted call, the code boundary should enforce migration and sandboxing requirements, and the deployment should decide whether that capability is allowed for the user, environment, and data involved.

Compact Reference

Use this page as the operational bridge between LangSmith observability concepts and the runtime behavior of legacy Python-related LangChain surfaces. The next useful step is to instrument a small agent with LangSmith tracing, reproduce one model call and one tool-related failure, and confirm that the trace contains enough metadata to explain both. Then review the agent tools, retrieval pipeline, and callback configuration pages so local handlers, durable traces, and security boundaries work together instead of competing with each other.