How to Cut Your AI Costs by 100x in 2026 (Part 2, Context Optimization and Memory Management)
Part 1 covered the model pricing spectrum and how to route work to the right tier. The takeaway was simple: stop sending every task to a frontier model. Use OpenRouter to route cheap tasks to DeepSeek, and reserve Opus for the work that needs it.
The right model is only half the equation. The other half is how much context you send with each request.
Context bloat is the silent budget killer of 2026. Context windows expanded to 10 million tokens, and developers got lazy. They now send entire codebases, full conversation histories, and bloated system prompts with every request. The model does not need your 200-page PDF. It needs three paragraphs to answer the question. You pay for all 200 pages anyway.
This article covers three mechanisms that fix context waste. Caching cuts repeat costs by 90%. Compression trims inputs by 2-6x. Better retrieval halves your token use per query.
The Two Types of Caching That Save Real Money
The largest cost reduction of 2026 is also the simplest one. Never process the same tokens twice. Two distinct caching mechanisms do that, and they solve different problems.
Prompt Caching Works When Your Prefix Is Stable
Every request to a large language model (LLM) makes the model process every input token from scratch. That includes the system prompt, the few-shot examples, and any static context. None of that content changes between calls. Prompt caching removes this waste. It stores the model's internal key-value state for token prefixes it saw before.
Anthropic's implementation lets you mark cache breakpoints in your prompt by hand. Any prefix before the breakpoint stays cached for a 5-minute time-to-live (TTL) window. Later requests that share the same prefix get a 90% discount on those cached tokens. The model skips the expensive prefill computation and jumps straight to the new content.
DeepSeek handles this automatically. Their API detects when a new request shares a prefix with a recent one. It then caches that prefix with no developer configuration. Input costs drop from $0.28 per million tokens to $0.028 per million, a 10x reduction on repeat prefixes.
Here is the practical implication. Take a system prompt longer than 1,000 tokens. Send it with every request and you overpay. Structure your prompts so the static content appears at the start and the dynamic content at the end. The cache then hits on every later request.
Take a 20-turn conversation with a 5,000-token system prompt on every turn. Prompt caching alone cuts the cost of those system tokens by 90%. Across thousands of daily conversations, that becomes a large line item.
Semantic Caching Bypasses the Model Entirely
Prompt caching saves money on the tokens you send. Semantic caching saves money because it sends no tokens.
The idea is simple. Convert every incoming query into an embedding vector. Check that vector against a local vector store such as Redis or Pinecone. If a previous query with similarity above 0.95 exists, return the cached response directly. The LLM never sees the request.
This works well for support bots, FAQ systems, and any application with repeated questions. Users ask the same thing in different words. "How do I reset my password?" and "I forgot my password, what do I do?" map to the same intent. A semantic cache catches both and returns the same pre-computed answer.
The saving is 100% on every cache hit. No model call means no token cost. Take a support bot with 10,000 queries a day. If 60% of those queries repeat an earlier intent, the bot returns 6,000 free responses per day.
Compressing Your Prompts Without Losing Meaning
Caching handles repeated content. Some requests are always unique: dynamic multi-document analysis, fresh data ingestion, one-off research queries. Those requests give you nothing to cache. Use compression instead.
The research here is encouraging. The "Lost in the Middle" effect (Liu et al., 2023) describes one finding. LLMs often ignore information in the middle of long prompts. They attend most strongly to the beginning and the end. So a large part of every long prompt is waste. The model processes those tokens, charges you for them, and then ignores them.
LLMLingua (Microsoft Research, ACL 2024) exploits this directly. It uses a small, fast model. That model calculates the perplexity of each token in a large prompt. It then strips out the tokens with low information content: grammatical filler, redundant phrasing, and boilerplate. The result is a compressed prompt. It keeps the meaning and cuts the token count.
The published results show 2x to 6x compression ratios, and the range depends on the content type. On the NaturalQuestions benchmark, LLMLingua achieved a 21.4% performance boost with 4x fewer tokens. The model performed better with the compressed input, because the signal-to-noise ratio improved. On the LooGLE long-context benchmark, it cut cost by 94% and held accuracy at a comparable level.
Here is a concrete example. An enterprise sends 5 billion input tokens per month through Claude Opus 4.1 at $15 per million. The monthly input bill is $75,000. LLMLingua at a conservative 4x ratio cuts that bill to roughly $18,750, a saving of $56,250 each month. The compression model runs on a small local GPU, and its compute cost is small against the saved API calls.
The key constraint is the content type. Compression works best on natural language prompts with redundant phrasing. Highly structured inputs such as JSON schemas or code compress less. Test your own content to find the right compression ratio before you deploy.
Building a Smarter RAG Pipeline with Semantic Chunking
In Retrieval-Augmented Generation (RAG), the most expensive decision happens before the model sees a token. That decision is how you chunk your documents.
Traditional fixed-size chunking splits documents into 512-token blocks regardless of content, and it is the default in most RAG frameworks. It is also bad for both cost and accuracy. A 512-token boundary splits a function definition in half. It separates a conclusion from its supporting evidence. The retrieval system then pulls three chunks to rebuild what one well-bounded chunk holds. That triples your input token cost.
Semantic chunking fixes this with embedding similarity to detect natural topic boundaries. Instead of slicing at fixed intervals, it measures the cosine similarity between adjacent sentences. A drop below the threshold marks a topic shift, so it creates a chunk boundary there.
The result is that each chunk contains a coherent unit of meaning. Retrieval accuracy improves, because the model gets complete and self-contained context rather than fragments. Published benchmarks show a 28% improvement in retrieval accuracy. You also retrieve fewer and more relevant chunks per query. Your input token cost per retrieval drops by roughly 50%.
A software company with RAG over its API documentation shows the difference well. With fixed-size chunking, a coding question retrieves three 512-token chunks, or 1,536 tokens. The split function definition forces that. With semantic chunking, the system finds the function boundary and retrieves a single 800-token chunk with the complete answer. The accuracy is the same and the token count is nearly half.
Managing Chat History Without Losing the Thread
The final context cost trap is the conversation history. In multi-turn applications, the standard approach includes the entire conversation history in every request. By turn 50, you send tens of thousands of history tokens in every API call. The model barely uses most of them.
The fix is hierarchical summarization. Keep the last 3-5 turns as raw text, so the model has full fidelity on the recent context. For everything older, run a periodic summarization pass. It condenses the conversation into a compact state summary of 200-500 tokens. That summary holds the key decisions, the preferences, and the open threads.
This replaces 5,000+ tokens of raw history with 300 tokens of compressed state. Per-request context costs drop by 94%. The model still refers to earlier parts of the conversation. Some applications run long sessions: customer support agents, coding assistants, research workflows. For those, this is the difference between $10 per conversation and $0.60.
The summarization pass itself runs on a cheap model such as DeepSeek or Haiku 4.5, because condensation is a simple task. You spend a fraction of a cent and save dollars on every later turn.
The Context Optimization Playbook
Every scenario below maps to one or more of the techniques above. Find yours and follow the recommendation.
Your chatbot re-sends a long system prompt on every turn. Use exact prefix caching. Put your system prompt and few-shot examples first, and the dynamic user content last. DeepSeek does this automatically. Input drops from $0.28/M to $0.028/M on cached tokens. Anthropic requires manual cache breakpoints and gives you the same 90% saving. On a 20-turn conversation with a 5,000-token system prompt, caching alone removes 95,000 redundant tokens from the bill.
Your FAQ or support bot handles thousands of similar questions daily. Deploy a semantic cache with Redis and vector embeddings. Convert every incoming query to an embedding and check its similarity against your cache. If the similarity is above 0.95, return the stored response and skip the LLM call. Take a bot where 60% of queries repeat an intent. This removes 6,000 model calls per day at a 100% saving each.
You are processing large, messy documents (legal PDFs, technical manuals, compliance reports). Run LLMLingua compression before you send the prompt to the model. A 10,000-token document with boilerplate, footers, and redundant phrasing compresses to 2,500-5,000 tokens at a 2-4x ratio. The model often performs better on the compressed version, because the noise is gone. An enterprise processing 5B tokens/month on Claude Opus 4.1 at $15/M input saves $56,250/month at 4x compression.
Your RAG pipeline retrieves too many chunks per query and costs are ballooning. Switch from fixed-size chunking (512 tokens) to semantic chunking. Fixed chunks split documents at arbitrary boundaries, so the retrieval system pulls multiple fragments to answer one question. Semantic chunks map to natural topic boundaries, so you retrieve one cohesive chunk instead of three fragments. Token cost per retrieval drops ~50%, accuracy improves ~28%.
Your agent or assistant conversations grow expensive after 30+ turns. Implement hierarchical summarization. Keep the last 3-5 turns as raw text. Summarize everything older into a compact 200-500 token state block. Use a cheap model such as DeepSeek or Haiku. This replaces 5,000+ tokens of raw history with ~300 tokens. Per-request context costs drop by 94%, and the model still holds the conversational thread.
You need all of the above: caching, compression, and smart chunking. Stack them, because the four techniques work together. Structure your prompts for cache-friendly prefixes (caching). Compress dynamic content with LLMLingua before injection (compression). Use semantic chunking in your RAG pipeline (retrieval), and summarize long conversations on a schedule (memory). Each layer compounds on the last. A system with all four cuts context costs by 95%+ against a naive implementation.
Part 2 Checklist
[ ] Put static content first in every prompt, such as the system prompt and few-shot examples. Dynamic content goes last, which maximizes cache hits
[ ] Enable prompt caching, automatic on DeepSeek, manual breakpoints on Anthropic, for any system prompt over 1,000 tokens
[ ] Deploy a semantic cache (Redis + embeddings) for any user-facing application with repetitive query patterns
[ ] Run LLMLingua (or equivalent) on long natural-language inputs before you send them to the model. Target 2-4x compression on documents with boilerplate
[ ] Replace fixed-size RAG chunking (512 tokens) with semantic chunking based on embedding similarity
[ ] Implement hierarchical summarization for multi-turn conversations: raw text for the last 3-5 turns, compressed state summary for everything older
[ ] Measure your cache hit rate weekly. A rate below 50% on a repetitive workload means your prefix structure or similarity threshold needs tuning
[ ] Test compression on your actual content before you deploy. Structured data (JSON, code) compresses less than natural language
Coming up in Part 3: Part 1 picked the right model, and Part 2 fixed your context. The last set of techniques goes past standard API usage. Speculative decoding doubles inference speed, and activation probes replace expensive LLM judges at 1/50th the cost. Logit bias stops verbose output, and batch processing gives a flat 50% discount on non-urgent work. Part 3 covers those advanced algorithmic and governance techniques. They take the last order of magnitude out of your AI spend.
Related Reading
Practical Tips for Reducing AI Costs - Part 1 Tactical Stack
Practical Tips for Reducing AI Costs - Part 3 Advanced Algorithmic and Internal Governance
