Prompt Caching in LangGraph: Ordering State So the Prefix Actually Caches

Every time a LangGraph node calls the model, it sends the whole payload again. That means the system prompt, tool schemas, reference documents and history so far. The model keeps no memory between calls. A research agent that loops four times and hands off to a formatting agent can pay for the same 10,000-token instruction block five or six times in one user request.

Prompt caching fixes most of that. It is cheap to adopt, and it fails silently when you get it wrong. The code still runs and the answers still look right. The bill and the response time just never come down.

What the provider is caching

Before generating a word, the model processes your entire input. This step is called prefill. Its cost and duration grow with input length. Prefill is the main reason a long-context call takes seconds before the first token appears. That delay is called time to first token, or TTFT.

With prompt caching, the provider keeps the processed form of a prompt prefix for a short window. If your next request starts with exactly the same tokens, that part is read back instead of recomputed. You pay a reduced rate for it, and the model starts generating sooner.

The key word is prefix. Matching runs from the first token forward and stops at the first difference. Everything after a changed token is processed at full price. If one changed character sits at the top of your system prompt, nothing below it caches.

The two major API providers handle this differently.

Anthropic (Claude) uses explicit caching. You mark breakpoints with cache_control, and everything up to each breakpoint becomes a cacheable prefix. Per Anthropic's prompt caching documentation:

• ▸The default lifetime is 5 minutes, refreshed each time the cache is read. A 1-hour lifetime is available with "ttl": "1h".
• ▸Writing to the 5-minute cache costs 1.25 times the base input price. Writing to the 1-hour cache costs 2 times. Reads cost 0.1 times.
• ▸You get up to four breakpoints per request.
• ▸Prompts below a model-specific minimum length are not cached. The minimum ranges from about 1,024 to 4,096 tokens depending on the model.
• ▸The prefix is assembled in a fixed order: tools, then system, then messages. Changing a tool definition invalidates everything after it.

OpenAI (GPT) caches automatically for prompts of 1,024 tokens or more, with no markup needed. Per OpenAI's prompt caching guide, a cached prefix remains eligible for reuse for 30 minutes after its most recent write or reuse. Some models support an extended retention option of up to 24 hours. The cached-input discount varies by model, so check the current pricing page. You can also pass a prompt_cache_key so requests that share a prefix are routed to the same cache.

The "5 to 15 minutes" figure that circulates in older write-ups is outdated. Plan around 5 minutes for Anthropic and 30 minutes for OpenAI by default. Pay for Anthropic's longer retention only when your traffic has gaps.

Static first, volatile last

For caching purposes, LangGraph state splits into two groups.

Byte-stable across a session: tool schemas, the system prompt, policies, few-shot examples, and documents retrieved once at session start.

Changes every turn: new user messages, tool results, timestamps, user-specific context, and trace or request IDs.

The rule follows from the prefix mechanism. Stable content goes first and must be identical down to the byte. Volatile content goes last. History is append-only, so it acts as a growing stable prefix as long as you never rewrite earlier turns.

The most common failure is a framework default or helper that puts Current time: {now} or a trace ID into the system prompt. That one line forces a full-price system prompt on every call.

The message-assembly code

This node calls the Anthropic SDK directly, so every byte of the payload is visible. The same structure works through LangChain's Anthropic chat model, which accepts cache_control on content blocks.

import json
from typing import TypedDict, Annotated
import operator
import anthropic

client = anthropic.Anthropic()

# Loaded once at startup from versioned files. Never formatted per request.
STATIC_SYSTEM = open("prompts/support_agent.v12.md", encoding="utf-8").read()

def load_tools() -> list[dict]:
    tools = json.load(open("tools/support_tools.v12.json", encoding="utf-8"))
    # Deterministic order and key layout, so the serialized bytes never drift.
    tools = sorted(tools, key=lambda t: t["name"])
    return json.loads(json.dumps(tools, sort_keys=True))

TOOLS = load_tools()
# Breakpoint 1: caches the tool block on its own.
TOOLS[-1] = {**TOOLS[-1], "cache_control": {"type": "ephemeral"}}

class AgentState(TypedDict):
    # Anthropic-format turns. Append-only: earlier turns are never edited.
    messages: Annotated[list[dict], operator.add]
    session: dict  # tier, locale, trace_id: NOT sent in the system prompt

def ingest_user_turn(text: str, session: dict, now_iso: str) -> dict:
    """Freeze volatile context into the user turn once, when it arrives.

    The header is persisted with the turn, so it is identical on every
    later call. Re-generating it at send time would break the history cache.
    """
    header = (
        f"<context>\nlocale: {session['locale']}\n"
        f"tier: {session['tier']}\nreceived_at: {now_iso}\n</context>\n\n"
    )
    return {"role": "user", "content": [{"type": "text", "text": header + text}]}

def build_request(state: AgentState) -> dict:
    system = [
        # Breakpoint 2: tools + system prompt.
        {"type": "text", "text": STATIC_SYSTEM,
         "cache_control": {"type": "ephemeral"}},
    ]

    # Copy, never mutate state: stale breakpoints would pile up past the limit of 4.
    messages = [{**m, "content": [dict(b) for b in m["content"]]}
                for m in state["messages"]]

    # Breakpoint 3: rolling marker on the newest block caches the history so far.
    messages[-1]["content"][-1]["cache_control"] = {"type": "ephemeral"}

    return {"system": system, "tools": TOOLS, "messages": messages}

def agent_node(state: AgentState) -> dict:
    req = build_request(state)
    resp = client.messages.create(
        model=MODEL_ID,                     # pinned per environment
        max_tokens=1024,
        metadata={"user_id": state["session"]["user_hash"]},  # outside the prompt
        **req,
    )
    log_cache_usage(resp.usage, trace_id=state["session"]["trace_id"])
    return {"messages": [{"role": "assistant", "content": resp.content}]}

Three details carry most of the weight.

The trace ID never enters the prompt. It goes to your tracing layer, such as a Langfuse callback, LangGraph run config, or your own logger. The model does not need it to answer.

The timestamp is frozen into the user turn when that turn arrives. Each later call sees the same bytes. If you rebuild the header at send time instead, every past turn changes and the history cache never hits.

The rolling breakpoint moves forward each turn. Anthropic checks backwards from a breakpoint for the longest previously cached prefix. The previous turn's cache is still found when the marker moves on, within a limited lookback window of content blocks.

Prove it is hitting

Don't trust the architecture diagram. Read the usage fields.

def log_cache_usage(usage, trace_id: str) -> None:
    logger.info(
        "llm_usage",
        trace_id=trace_id,
        input=usage.input_tokens,                         # uncached, full price
        cache_write=usage.cache_creation_input_tokens,    # written this call
        cache_read=usage.cache_read_input_tokens,         # served from cache
    )

On OpenAI, the equivalent field is usage.prompt_tokens_details.cached_tokens.

Put an assertion in your test suite. Run the same session twice within a minute, and check that the second call's cache_read covers at least the tools and system block. This one test catches most regressions before they reach the invoice. It belongs alongside your other agent production checks.

The cost formula, with your numbers

In plain words: when ten calls in a session share the same instructions, caching means you pay for those instructions about twice instead of ten times. Here is where that comes from.

Use these variables for the static block:

• ▸S = static tokens (tools, system prompt, fixed documents)
• ▸N = model calls per session that reuse it, inside the cache lifetime
• ▸p = base input price per token

Without caching, the static cost per session is N × S × p.

With Anthropic's 5-minute cache, it is S × 1.25p + (N − 1) × S × 0.1p.

For N = 10, that is 2.15 × S × p against 10 × S × p. You pay about 21.5% of the uncached static cost. That figure follows directly from Anthropic's published multipliers. Your real saving depends on how much of each request is static, since history and tool output still grow at full price until cached.

Breakeven is easy. A 5-minute write pays for itself after one read: 1.35 against 2.0. A 1-hour write at 2 times needs at least two reads to beat no caching. Choose the 1-hour lifetime only when calls in the same session are often more than 5 minutes apart.

Latency follows the same logic, but don't take anyone's percentage on faith. The gain grows with the share of the prompt that is cached. Measure TTFT on your own traffic before and after the change.

Caching does nothing for retries or for evaluation overhead. The full cost model including retries and evaluation puts the three together.

What silently breaks the cache

This table is the main reason for this post. Each row is a pattern we check for when reviewing an agent codebase.

The fan-out row matters most for multi-agent graphs. A supervisor that dispatches five workers at once, each with the same 8,000-token preamble, writes that preamble five times. One warm-up call, followed by the parallel batch, turns four of those writes into reads. Caching lowers the price of each step; it does not bound how many steps a run takes. For that you still need token ceilings and cost caps as controls.

Where this fits in a larger refactor

Caching is the cheapest performance work available in an agent system. It changes no answers and needs no new models. If your graph currently builds one big string with f-strings and sends it as a single user message, the refactor has three parts. Move to a structured message array. Separate the static and volatile layers. Add the usage logging above.

The production patterns for LangGraph cover the state and reducer design this depends on. Get the state shape right first, and caching falls out of it.

Frequently Asked Questions

How long does a cached prefix last? On Anthropic, 5 minutes by default, refreshed on each read, with an optional 1-hour lifetime at a higher write price. On OpenAI, a cached prefix remains eligible for reuse for 30 minutes after its most recent write or reuse. Some models offer extended retention of up to 24 hours. Low-traffic systems should expect the first call after a quiet period to be a full-price miss.

Does this apply to self-hosted models? Yes. The mechanism lives in the inference server, not the model weights. vLLM and SGLang both support automatic prefix caching, so the same ordering rules decide your hit rate. There is no per-token discount on your own hardware. The payoff is lower TTFT and more concurrent sessions per GPU. Our on-prem serving guide covers the related settings.

Can caching leak one customer's data to another? Provider caches are scoped to your organization, so other customers cannot read them. The real risk is in your own code. If a shared static prefix contains one user's data, it is no longer static, and it should never have been there. Keep shared policy and tools in the prefix, and keep anything user-specific in that user's own turns.

This week: prove the cache is hitting

Add the cache_read assertion above to your test suite: run one session twice inside a minute and assert that the second call's cache_read covers the tools and system block. Then open one production trace and read the three usage fields. If cache_read is zero, one of the rows in the table is in your code, and finding it is usually an afternoon's work.