You pay for every token you send, and most tokens in a long prompt repeat from the last request. AI context optimization sends the repeated part once, caches it, and reads it back at a fraction of the price.

Part 1 picked the right model for each job. This part cuts the size of what you send to that model. Every step below works on a plain hosted API.

TLDR

Put static content first and dynamic content last, so a cache can match your prompt.
Anthropic reads cached tokens at 0.1 times the base input price on most models, and DeepSeek caches on its own.
A semantic cache answers a repeat question with no model call at all.
A 300-token summary can replace 5,000 tokens of old chat turns.
Compress long one-off inputs last, and test the ratio on your own content first.

A Support Bot Pays for the Same 3,000 Tokens on Every Turn

Take a support bot with a 3,000-token system prompt. Each turn re-sends that prompt and the full chat history, both at full price.

A support bot re-sends its prompt and history every turn until layout, a summary and a cache stop it.

After the changes below, the story reads differently:

The system prompt sits first and the cache reads it on repeat turns.
Old turns collapse into a 300-token summary.
"How do I reset my password?" returns from a vector cache, and the model never sees it.
Retrieval pulls one whole chunk instead of three broken ones.

Agents that run all night also re-send context all night, which is the habit Token Anxiety describes.

Step 1. Put Static Content First and Dynamic Content Last

A cache matches your prompt from the first token onward. It can only reuse a prefix that is identical to an earlier request.

A cache hits when the unchanging content comes first, and misses when a changing line leads.

Order the prompt in this sequence:

Tool definitions and the system prompt.
Few-shot examples and any long reference document.
The date, the user's name and other per-request details.
The user's question.

Rule of thumb. Anything that changes on every request goes after the last block you want cached.

Anthropic's docs show the trap. A timestamp in the system prompt changes the prefix on every request, so you pay a cache write each time and never get a read.

Step 2. Turn on Prompt Caching and Read the Hit Rate

Prompt caching stores the model's work on the start of your prompt. The large language model (LLM) then skips re-reading that part.

On Anthropic's API you add one cache_control field to the request, or mark explicit breakpoints. The prompt caching docs list the prices:

A 5-minute cache write costs 1.25 times the base input price.
A 1-hour cache write costs 2 times the base input price.
A cache read costs 0.1 times the base input price on most models, and 0.05 times on Claude Opus 5.5 and Claude Sonnet 5.5.
A prompt below the minimum length is not cached. The minimum is 512 tokens on the newest models and 1,024 to 4,096 on older ones.

Anthropic's launch post claims up to 90% lower cost and up to 85% lower latency on long prompts. Its example is a 100,000-token book, where time to first token fell from 11.5 seconds to 2.4 seconds.

DeepSeek needs no code change. Its pricing page lists deepseek-flash cache-hit input at $0.006 per million tokens. A miss costs $0.30. Both are peak rates as of 2026-10-13.

DeepSeek announced the feature as Context Caching on Disk, with no code changes and up to 90% lower API costs. Its docs call the system best-effort.

Then read the result. Both providers return cache counts in the response, so you can check your hit rate.

The usage block lists cached tokens, and cache reads divided by total input is your hit rate.

Watch out. The default cache lasts 5 minutes, and each use refreshes it for free. A longer gap empties it, and the next request pays the write price again.

A hit rate below 50% on a repetitive workload means your prefix is changing. Fix the layout from Step 1 before you tune anything else.

Step 3. Answer Repeat Questions Without Calling the Model

Prompt caching saves money on the tokens you send. A semantic cache saves the whole call.

A prompt cache matches the exact prefix, and a semantic cache matches what the question means.

Embed each incoming question, then search a vector store such as Redis or Pinecone for the closest stored question. If the match scores at or above your threshold, return the stored answer. If not, call the model and store the new answer.

A question with a stored match returns the stored answer, and a new question is stored after the model answers.

"How do I reset my password?" and "I forgot my password" map to one intent. A bot with 10,000 queries a day, where 6 in 10 repeat an earlier intent, answers 6,000 of them with no model call.

Tip. Start the threshold at 0.95 and test it on real question pairs. A threshold that is too low returns a confident wrong answer.

Give stored answers an expiry, so a changed policy does not keep serving the old reply.

James Briggs shows in Python how the sentence-transformers library scores the similarity of two sentences that share almost no words, which is the match behind the 0.95 threshold.

Step 4. Retrieve One Whole Chunk, Not Three Broken Ones

Retrieval-augmented generation (RAG) means the system fetches document chunks and adds them to the prompt. How you split documents decides how many tokens each fetch costs.

A fixed 512-token split cuts through a function definition or separates a conclusion from its evidence. The search then fetches three chunks to rebuild one idea.

Semantic chunking places a boundary where the topic shifts. It compares the similarity of adjacent sentences and cuts at a drop.

One 800-token semantic chunk answers a question that needs 1,536 tokens of fixed chunks.

Chroma's chunking report found strategies differ by up to 9% in recall, the share of needed text that retrieval finds. Its LLM-based chunker scored highest at 91.9%. The 1,536 against 800 tokens above is my calculation, not a benchmark.

Chroma's cofounder Anton Troynikov announced the report, an evaluation of how chunking strategies affect retrieval performance.

Step 5. Replace Old Chat Turns With a 300-Token Summary

In a long chat, the standard approach sends the whole history on every request. By turn 50, the old turns dwarf the new question.

Keep the last 3 to 5 turns as raw text. Summarize everything older into a 200 to 500 token note of decisions and open threads.

A 300-token summary replaces 5,000 tokens of old turns, which cuts that part of the prompt 94%.

Replacing 5,000 tokens with 300 removes 94% of the old history from each request. That figure is a calculation, so measure the token count of your own chat before and after.

The summary pass is a simple task, so use a cheap model. Claude Haiku 5.5 lists $0.10 per million input tokens for prompts up to 100,000 tokens and $0.50 per million output tokens on Anthropic's pricing page. For work that should stay on your machine, Ollama runs open-weight models locally.

Ollama sits in the AI and Agent Tooling section of atharvashah.com/tools.

A chat summary holds history inside one conversation. Facts that must outlive the chat are a different job, covered in the 5 files that give Claude Code permanent memory.

Step 6. Compress Long One-Off Inputs and Test First

A one-off research question or a new document gives a cache nothing to reuse. Compression shrinks that input before you send it.

A 2023 study from Liu and colleagues found that models use the start and end of a long prompt best and often miss the middle. Jim Fan called it the "Myth of Context Length": a 1M-token window means little if the model does not use the context well.

LongLLMLingua, from Microsoft researchers, scores each token with a small language model and drops the low-information ones. The paper reports two results:

Up to 21.4% better performance with around 4x fewer tokens on NaturalQuestions, using GPT-3.5-Turbo.
A 94.0% cost reduction on the LooGLE benchmark.

It also tests compression ratios of 2x to 6x on prompts of about 10,000 tokens. These are the paper's numbers, so treat them as a ceiling.

Example. A team sends 5 billion input tokens a month to Claude Opus 5.5 at $4 per million. That is $20,000. At a 4x ratio the bill falls to $5,000, a saving of $15,000 before the cost of running the compression model.

Structured input such as JSON or code compresses less than plain prose. Run the compressor on a sample of your real prompts and compare answers before you deploy it.

For work that can wait, a batch discount stacks on top of these savings. Part 3 covers batching and the other levers.

Five Questions a Newcomer Asks About Cutting Context Costs

Which step do I do first? Do Steps 1 and 2. They need no new infrastructure, and the response tells you whether they worked.

Does caching change the model's answer? Caching reuses the model's work on the prompt prefix. The model still generates a fresh answer, so output varies as before. DeepSeek's docs say temperature still applies.

How do I know a cache hit happened? Read the usage block. Anthropic returns cache_read_input_tokens, and DeepSeek returns prompt_cache_hit_tokens.

Do these steps need a GPU or my own model? No. Every step runs against a hosted API. Only the compression model and the local summarizer in Ollama are optional models you run yourself.

What if the semantic cache returns a wrong answer? Raise the threshold, add an expiry to stored answers, and log each hit so you can review it.

The Appendix Holds the Cache Breakpoint Code

Mark the end of your static content with an explicit breakpoint. The model name follows Anthropic's docs example.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LONG_STATIC_INSTRUCTIONS,       # unchanged on every request
            "cache_control": {"type": "ephemeral"}, # cache everything up to here
        }
    ],
    messages=[{"role": "user", "content": user_question}],  # changes every request
)

u = response.usage
total = u.input_tokens + u.cache_read_input_tokens + u.cache_creation_input_tokens
print("hit rate:", u.cache_read_input_tokens / total)

A semantic cache sketch, with embed, store and call_model standing in for your own functions:

vector = embed(question)
match, score = store.closest(vector)
if score >= 0.95:
    answer = match.answer            # no model call
else:
    answer = call_model(question)
    store.add(vector, answer)

AI Context Optimization. 6 Steps to Cut Your…