01.What a "complex chat invoke" really is

Picture the invoke behind a typical assistant. Conceptually it's one function that assembles everything and fires it at the model on every user turn:

# ❌ One invoke rebuilds and re-sends the whole world, every turn
def answer(user_msg, history):
    messages = [
        SystemMessage(BIG_SYSTEM_PROMPT),   # static — same every turn
        *FEW_SHOT_EXAMPLES,                 # static — same every turn
        *load_reference_docs(),             # static — often re-read from disk!
        *history,                           # grows over the conversation
        HumanMessage(user_msg),             # the only truly new bit
    ]
    return llm.invoke(messages)

Look at the ratio. The system prompt, the tool schema, the few-shot block, the reference docs — that can be thousands of tokens, and it is byte-for-byte identical on turn 2, turn 20, and turn 200. The genuinely new information is one message. Yet the model prefills the entire payload from scratch each time. You pay for it in three currencies: latency (time-to-first-token grows with the prefix), money (every static token is re-billed), and code clarity (setup and per-turn logic are fused into one function that's hard to test).

02.The one distinction that unlocks everything: static vs dynamic

Before optimizing anything, split the payload in your head into two buckets:

• Static context — decided at setup and constant for the life of the session: system prompt, persona, tool/function definitions, output schema, few-shot examples, and any reference material you inject up front.
• Dynamic input — everything that changes turn to turn: the user's latest message, and the rolling conversation history.

Once you see the payload this way, the redundancy is obvious: you are re-transmitting and re-computing the static bucket on every call. The whole pattern below is just "compute the static bucket once, reuse it."

03.Two phases: init_session(), then process()

Instead of one function that does setup and answering, write two. init_session() runs once and produces a reusable, pre-configured session that already carries the static context. process() runs per turn and appends only the dynamic input.

The shape is the same one that makes any system easier to reason about: a constructor that runs once, and a method you call many times. Setup happens in one place; the hot path stays tiny.

04.Why this is actually faster, not just tidier

The speedup is not hand-waving — it falls out of how transformer inference works. A model call has two phases. Prefill reads every input token and builds the attention key/value (KV) cache; its cost grows with the size of the prompt and it dominates time-to-first-token for context-heavy calls. Decode then generates the answer one token at a time. When your static prefix is huge, prefill is where the wall-clock goes.

Prompt caching (a.k.a. prefix caching) lets the provider recognize that the static prefix is identical to a recent call and reuse the computed KV state instead of prefilling it again. On Azure OpenAI and OpenAI this is automatic for prompts past a size threshold; the model skips the cached prefix and starts generating almost immediately.

You are not caching the answer. You are caching the expensive prefill of the context the answer is computed from. The model still thinks fresh every turn — it just stops re-reading the same book first.

05.Making it real in LangChain + Azure OpenAI

In LangChain the two phases map cleanly onto composition. init_session() builds the runnable once, with the static content pinned to the front of the prompt and a placeholder for the dynamic tail. process() just supplies the variables per turn.

from langchain_openai import AzureChatOpenAI
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

# --- Phase 1: initialize the session ONCE ---
def init_session():
    llm = AzureChatOpenAI(azure_deployment="gpt-4o", temperature=0)
    prompt = ChatPromptTemplate.from_messages([
        # static prefix — assembled once, byte-identical on every call
        ("system", BIG_SYSTEM_PROMPT + FEW_SHOT + REFERENCE_DOCS),
        MessagesPlaceholder("history"),   # dynamic
        ("human", "{input}"),             # dynamic tail — the only new tokens
    ])
    return prompt | llm                   # a reusable, pre-configured session

# --- Phase 2: process each turn CHEAPLY ---
def process(session, history, user_msg):
    return session.invoke({"history": history, "input": user_msg})

Because the static prefix is identical across calls, Azure OpenAI's automatic prompt caching matches it and skips reprocessing. You can confirm the cache is working by reading the standardized usage on the response:

session = init_session()
resp = process(session, history=[], user_msg="Summarize section 3.")

# cache_read > 0 means the static prefix was served from cache
print(resp.usage_metadata["input_token_details"])
# {'cache_read': 3072, ...}

Two things earn the win here. First, the static content is concatenated at the very start, so the cacheable prefix is as long as possible. Second, the session object is built once and held — you are not re-reading docs from disk or re-templating the system prompt on every turn. The conversation state itself (the history) lives in your app; wire it through a message-history wrapper or a LangGraph checkpointer so "the session" is a real object with a lifecycle, not a global.

06.The one rule that makes or breaks it: keep the prefix byte-stable

Prompt caching matches on an exact prefix. Change one character near the front and the cache misses — you silently pay full price again. The failure mode is subtle because nothing errors; latency just quietly regresses. Guard the prefix:

• Put dynamic content last, always. A timestamp, a user id, or a per-request note injected into the system block busts the cache for everyone. Move it into the human turn at the end.
• Make the prefix deterministic. Serialize tool schemas and any JSON with stable key order; a dict that renders in random order is a different prefix every time.
• Mind the size floor. Automatic caching on Azure OpenAI / OpenAI only kicks in past a threshold (on the order of ~1K tokens). If your static context is tiny, the split still improves code clarity but won't move latency — don't expect caching magic below the floor.
• Mind the TTL. Cached prefixes expire after a period of inactivity (minutes). Bursty, back-to-back turns benefit most; a session idle past the TTL pays the cold prefill again on its next turn.

A note on providers: on Azure OpenAI and OpenAI, caching is automatic — you just structure the prompt well. On Anthropic it's explicit — you mark the end of the static block with a cache_control breakpoint. Same idea, different switch. See the Azure OpenAI prompt caching docs for the current thresholds and supported models.

07.The complexity win, beyond latency

Latency is the headline, but the reason I reach for this split even when caching isn't in play is that it makes the code simpler:

• Separation of concerns. Expensive, do-once work (loading docs, building the system prompt, binding tools) lives in init_session(). The per-turn path shrinks to "append input, invoke." Each is easy to read on its own.
• Testability. You can build a session in a fixture and hammer process() with many turns, instead of reconstructing the whole world in every test.
• Consistency. The static context is assembled exactly once, so it can't drift between turns due to a code path that rebuilds it slightly differently.
• Throughput. Less compute per turn means each request holds resources for less time, so the same hardware serves more concurrent sessions.

08.When not to split

This is a pattern, not a law. If your static context is small (a one-line system prompt), if calls are truly one-shot with no shared setup, or if every request legitimately needs a different system prompt, the split adds a layer without buying you a cache hit. Reach for it when you have a large, stable context reused across many turns — long system prompts, injected documents, big tool schemas, RAG contexts — which is exactly where the naive single invoke hurts most.

09.Takeaway

A "complex chat invoke" is usually one function doing two jobs: setting up static context and answering a turn. Pull them apart. Initialize the session once, keep the static prefix at the front and byte-stable, and let each turn carry only the new input. You get lower latency from prompt caching, a smaller bill from tokens you stop re-sending, and — quietly — a codebase where setup and the hot path are two things you can reason about separately. Compute the constant once; spend your budget on what actually changes.