Caching and Rate Limiting

Purpose and Scope

Caching and rate limiting are two production controls for language-model applications that attack different cost and reliability problems. A cache stores completed model generations so repeated prompts with the same model configuration can return without another provider request. A rate limiter controls how frequently requests are sent, so an application can stay under provider limits or smooth out bursts from evaluations, workers, or user traffic. In LangChain Python, both concepts live in langchain_core as small interfaces that model integrations can accept without tying application code to a specific provider or storage backend.

Sources: libs/core/langchain_core/caches.py, libs/core/langchain_core/rate_limiters.py

The caching layer in libs/core/langchain_core/caches.py is explicitly described as an optional layer for LLMs and chat models, distinct from provider-side prompt caching. That distinction matters because LangChain cache implementations operate around the serialized prompt and serialized LLM configuration used by the LangChain call path, while provider prompt caching is a provider capability with provider-specific behavior. The module also marks the cache feature as beta, so production use should include normal precautions such as test coverage, invalidation strategy, and operational monitoring.

Rate limiting in libs/core/langchain_core/rate_limiters.py is centered on acquiring permission before a request proceeds. The base interface supports synchronous and asynchronous callers through acquire and aacquire, both of which accept a blocking flag. The in-memory implementation uses a token bucket algorithm where tokens represent request capacity, not LLM input or output tokens. This makes the abstraction useful for controlling request frequency, but not for enforcing limits based on prompt length, completion length, or provider billing units.

Relevant Source Files

• libs/core/langchain_core/caches.py — Defines BaseCache, the RETURN_VAL_TYPE alias, and the synchronous and asynchronous cache contract for language model generations.
• libs/core/langchain_core/rate_limiters.py — Defines BaseRateLimiter and InMemoryRateLimiter, including sync and async acquisition behavior plus token-bucket configuration semantics.

Cache API Components

The cache contract is intentionally narrow. BaseCache.lookup(prompt, llm_string) returns either None for a miss or a sequence of Generation objects for a hit. BaseCache.update(prompt, llm_string, return_val) stores the sequence of generations for the same key inputs. BaseCache.clear(**kwargs) gives implementations a common invalidation entry point while still allowing implementation-specific keyword arguments. Together, these methods define the minimum behavior a model call path needs: check before calling the provider, persist after a successful call, and clear cached state when an application or test needs a reset.

Sources: libs/core/langchain_core/caches.py

The cache key is described as being derived from a two-tuple: the prompt string and llm_string. The prompt is a string representation of the request sent to the language model; for chat models, this can be a non-trivial serialization of messages into the model input. The llm_string captures invocation parameters such as model name, temperature, stop tokens, and maximum token settings. This means a cache implementation should avoid keying only on user-visible text, because two calls with the same prompt but different model parameters should not necessarily share cached generations.

BaseCache also supplies asynchronous methods: alookup, aupdate, and aclear. The default implementation delegates to the synchronous method through run_in_executor, which is convenient for compatibility but not always ideal for performance. If a cache backend is naturally asynchronous, such as a networked service or async database client, the source comments recommend overriding the async methods to avoid unnecessary executor overhead. This is an important implementation detail for high-throughput applications that call chat models from async web servers or evaluation runners.

Rate Limiter API Components

BaseRateLimiter defines two public operations: acquire(*, blocking=True) and aacquire(*, blocking=True). Both return True when the caller successfully acquires the required capacity and False when capacity is unavailable in non-blocking mode. With blocking=True, the call waits until tokens are available. With blocking=False, the call immediately reports whether the request can proceed. Implementations may add initialization parameters, including timeout-style options, but the shared runtime contract is acquisition before an expensive operation.

Sources: libs/core/langchain_core/rate_limiters.py

InMemoryRateLimiter is the concrete implementation supplied by langchain_core. It is thread safe and works from both sync and async contexts, but it is explicitly in-memory and therefore does not coordinate limits across multiple processes. It only performs time-based request limiting, so it cannot account for input size, output size, or other request characteristics. The implementation is based on a token bucket: the bucket fills at a configured rate, each request consumes a token, and a request waits when the bucket does not contain enough tokens.

The main constructor options exposed in the example are requests_per_second, check_every_n_seconds, and max_bucket_size. requests_per_second controls the steady-state fill rate, check_every_n_seconds controls how often a blocking waiter wakes to check whether capacity exists, and max_bucket_size controls burst capacity. Official LangSmith guidance uses the same pattern for large evaluation jobs: attach an InMemoryRateLimiter to Python chat models when third-party model APIs would otherwise return rate limit errors during high-volume runs.

from langchain_core.rate_limiters import InMemoryRateLimiter
 
rate_limiter = InMemoryRateLimiter(
    requests_per_second=0.1,
    check_every_n_seconds=0.1,
    max_bucket_size=10,
 )

System-to-Code Mapping

Execution Flow and Operational Guidance

A typical cached model call first serializes the prompt and model configuration, checks lookup, and uses the returned generations when present. On a miss, the application or model wrapper calls the provider, receives generations, and writes them with update. This flow is best suited to deterministic or acceptably repeatable requests, testing, cost reduction, and latency reduction for repeated calls. Because the cached value is a sequence of Generation or subclasses, cache implementations should preserve the output shape expected by the model abstraction rather than reducing responses to plain strings.

A typical rate-limited model call attempts acquire or aacquire immediately before sending the provider request. If blocking mode is enabled, application latency includes both waiting for capacity and the provider request time. The source notes that rate limiting information is not surfaced in tracing or callbacks, so a slow model invocation may include hidden waiting time. When diagnosing latency, developers should therefore consider the configured rate limiter alongside provider latency, retries, and network behavior rather than assuming all time was spent inside the model provider.

Use caching and rate limiting together when both repeated work and provider quotas matter. Caching reduces the number of calls that need capacity at all, while rate limiting protects the remaining misses and non-cacheable calls. For a single process, InMemoryRateLimiter is the built-in starting point. For multi-process deployments, distributed workers, or quota policies based on token counts, the source-defined limitations imply that a custom BaseRateLimiter implementation or an external coordination service is the appropriate next step.

Compact Reference

• Import cache base class: from langchain_core.caches import BaseCache
• Cache return type: Sequence[Generation] via RETURN_VAL_TYPE
• Required cache methods: lookup, update, clear
• Async cache methods: alookup, aupdate, aclear
• Import in-memory limiter: from langchain_core.rate_limiters import InMemoryRateLimiter
• Rate limiter methods: acquire(*, blocking=True), aacquire(*, blocking=True)
• In-memory limiter options shown in source/docs: requests_per_second, check_every_n_seconds, max_bucket_size
• Important limitation: InMemoryRateLimiter is thread safe but not cross-process
• Important limitation: rate limiting wait time is included in total invocation time and is not separately surfaced in tracing or callbacks