LLM prompt caching can make a premium API cheaper than an underused local GPU for repeated context. It can also produce no saving at all. The deciding variables are not model list prices. They are prefix reuse, cache-write cost, read discounts, expiration, eviction, utilization, and the cost of failed work.
Audience: Advanced practitioners responsible for LLM platform cost, latency, or infrastructure decisions.
The practical rule: measure cache reads and invalidation on your real workload before you buy GPU capacity. A new enterprise coding-agent case study makes that point with a striking 99.3% cache-hit result, but its authors studied one developer, two sequential 28-day periods, different model families, and one production codebase. Treat the paper as a measurement prompt, not a cloud-versus-local verdict.
What LLM prompt caching changes
An LLM processes input in two broad phases:
At the serving layer, prefix caching can store reusable attention or key-value states for a prompt prefix. A later request with the same eligible prefix can skip some repeated prefill work. The changing suffix still needs processing, and the model still decodes a new answer.
That distinction matters. Prompt caching is not response caching. It does not return a stored answer, shorten the output, repair a weak prompt, or guarantee lower end-to-end latency. It primarily targets repeated input computation and often improves time to first token.
The original Prompt Cache paper formalized reusable prompt modules and reported prototype time-to-first-token improvements ranging from 8× on GPUs to 60× on CPUs. Those are results from the paper's tested models, hardware, prompts, and implementation—not a forecast for current commercial APIs.
The 99.3% case study: useful evidence with narrow boundaries
A July 2026 preprint, *Inference Economics of Enterprise Coding Agents*, compared two contiguous 28-day periods on one production monorepo:
The authors analyzed LLM telemetry and Git history. They reported a 99.3% prompt-cache hit rate for the API period, an effective API cost of $0.573 per million processed tokens, and a $2.83 amortized unit cost for the shared on-premise allocation. The API processed 16.9 times more tokens, while the local hardware had low utilization, so those normalized token prices do not settle the infrastructure question.
The total-cost results point in different directions depending on allocation. Under the paper's Taiwan-market and labor assumptions, shared local capacity reduced estimated true total cost of ownership by 40.1%. Dedicated local reservation cost 43.8% more than the cached API. The local period was also associated with a higher fix-commit ratio: 74.9% versus 45.9%.
What the authors measured
What they inferred
What the study cannot establish
The design was non-randomized and sequential. The codebase, developer, task mix, and working practices could have changed between periods. Model capability, serving stack, harness, and quantization changed together. The first author knew the hypothesis. The fix-commit label is a proxy, not an independent defect audit. Raw production telemetry and the complete replay environment were not published.
The durable conclusion is narrower than the headline numbers: prompt reuse and GPU utilization can dominate list-price comparisons, while repair work can dominate token savings.
Define the measurement contract before calculating a hit rate
“Cache hit rate” is ambiguous unless the denominator is explicit. One dashboard may divide cached tokens by eligible prefix tokens. Another may divide requests with any cache read by all requests. A third may report cached tokens as a share of total input, including an uncached suffix.
Use separate metrics:
Keep provider-reported usage fields separate from metrics you derive. Token counts can also change with model tokenizers and templates; the LLM tokenizer tax explainer→ shows why raw character counts are a weak substitute.
Current API and self-hosted behavior is not uniform
The following snapshot was checked on July 30, 2026. Recheck the linked documentation before using it for procurement or billing.
| Platform | Cache control | Observable signal | Operational constraint |
|---|---|---|---|
| --- | --- | --- | --- |
| OpenAI API | GPT-5.6 supports implicit and explicit prompt caching | `cached_tokens` and `cache_write_tokens` | Current GPT-5.6 guidance prices explicit writes at 1.25× uncached input; reads are discounted |
| Anthropic API | Automatic caching or explicit block breakpoints | Separate cache creation and read usage | Prefix order is tools, system, then messages; default TTL is 5 minutes, with a paid 1-hour option |
| Gemini Interactions API | Implicit context caching is enabled by default for Gemini 2.5 and newer models | `usage.total_cached_tokens` | Common content belongs at the start; current minimums range from 2,048 to 4,096 tokens by model |
| vLLM | Automatic Prefix Caching can be enabled in the engine | Engine metrics and request timing | Your team owns capacity, eviction, isolation, upgrades, and observability |
OpenAI's current model guidance tells GPT-5.6 users to monitor cache writes and reads because explicit writes cost more than uncached input. Anthropic's prompt-caching documentation documents 5-minute writes at 1.25× base input, 1-hour writes at 2×, and cache reads at 0.1×. Google's context-caching guide says implicit caching is automatic in its Interactions API and reports cached tokens in usage. vLLM's Automatic Prefix Caching example shows the same shared-prefix idea in a self-hosted engine.
These implementations share a principle, not a portable billing contract. Minimum prefix lengths, cache scope, retention, storage charges, usage fields, and isolation can differ.
*Measure the cache before changing infrastructure: stabilize the prefix, run cold and warm tests, force invalidation, then compare total cost.*
A break-even formula for the reusable prefix
Let:
Ignoring storage charges, the average reusable-prefix charge per request is:
`(W + (N - 1) × R) / N`
Caching is cheaper than processing that prefix uncached when:
`N > (W - R) / (U - R)`
This equation isolates the reusable prefix. It excludes the changing suffix, output tokens, minimum cacheable length, storage fees, misses caused by eviction, concurrency effects, engineering labor, and failed tasks.
For one dated illustration, Anthropic's 5-minute multipliers on July 30, 2026 were `U = 1`, `W = 1.25`, and `R = 0.1`. The input-only threshold is `N > 1.28`, so the second use within the cache lifetime amortizes the higher write charge for that eligible prefix. That does not mean two total API calls make caching profitable. A short prefix, a long uncached suffix, a cache miss, or an expensive output can erase the saving.
Use the equation as a unit test for your cost model. Replace every variable with values from the current provider contract and your measured workload.
Why apparently stable prompts miss the cache
Most misses begin in prompt construction, not in the model.
Volatile data appears too early
A timestamp, request ID, random nonce, user name, or fresh retrieval result near the front changes every token that follows. Put stable tools, policies, templates, and reusable documents first. Move request-specific data after the reusable prefix or explicit breakpoint.
Serialization changes between requests
JSON key order, whitespace, tool ordering, and document ordering can change an otherwise equivalent prefix. Use deterministic serialization. Sort tool definitions and retrieved documents by stable identifiers where semantics allow it.
The context manager destroys prefix continuity
Aggressive pruning can save input tokens while invalidating cached prefixes. TokenPilot, a June 2026 work-in-progress preprint, treats this as a joint optimization problem. Its authors stabilize ingestion and delay eviction until context loses task value. They report cost reductions of 56% to 87% across two benchmarks and two execution modes while maintaining competitive task performance. Those results require replication beyond the paper's workloads and LightMem2 integration.
The cache expires or is evicted under real load
A warm local test can hide TTL boundaries and capacity pressure. Self-hosted prefix caching shares the same finite-memory reality as other KV-cache systems. The deeper KV-cache eviction guide→ covers why reuse can collapse when concurrency and sequence length rise.
The model or template changes
A model version, tokenizer, chat template, image detail setting, tool schema, or safety preamble may create a new prefix. Treat releases and prompt migrations as cache invalidation events.
Current vLLM engineering discussions illustrate the pressure. A context-aware retention proposal argues that concurrent agent workloads can evict valuable prefixes, while a semantic KV-cache proposal explores reuse beyond exact matches. These are open design discussions, not production guarantees or adoption measurements.
A reproducible prompt-cache test
Use the same task set you would use to benchmark AI models for real work→. The cache test adds controlled prefix mutations and infrastructure accounting.
1. Freeze a representative workload
Choose 30 to 100 tasks from production traffic or a privacy-safe replay set. Preserve the real distribution of prefix length, suffix length, output length, tools, documents, and concurrency. Define task success before running the test.
Do not optimize on one long demo prompt. A cache-friendly support workflow and a cache-hostile research workflow can have opposite economics.
2. Split stable and volatile content
Label each prompt segment:
Build a deterministic prompt assembler. Record an opaque prefix version or keyed hash so each request identifies the prefix it attempted to reuse. Do not put secrets or raw customer content in logs.
3. Run a controlled matrix
For every task class, measure:
| Trial | Change | Question answered |
|---|---|---|
| --- | --- | --- |
| Cold | New prefix with no reusable state | What is the baseline write or prefill cost? |
| Warm | Identical eligible prefix | Does the provider or engine report a read? |
| Early mutation | Change one token near the start | How much reuse disappears? |
| Late mutation | Change only the suffix | Does the stable prefix remain reusable? |
| TTL boundary | Repeat before and after expiry | How often will production traffic arrive in time? |
| Concurrency | Increase parallel requests in steps | Does eviction or scheduling reduce reuse? |
| Version change | Change model, tools, or template | Which deployments invalidate the cache? |
Run enough repetitions to report distributions rather than one latency number. Separate p50 and p95 time to first token from end-to-end latency.
4. Record billing and quality together
Capture:
Exact-prefix state reuse should avoid repeated prefill computation; it does not excuse a quality check. A model, quantization level, routing policy, or serving stack change can alter output quality even when the cache mechanism itself is correct.
5. Calculate three cost views
If the local option depends on sustained utilization, test the utilization assumption. A GPU purchase does not become economical because a spreadsheet assigns every idle hour to future demand.
Cloud API, local GPU, or hybrid routing?
The decision is rarely binary.
Prefer a cached API when
Prefer shared local capacity when
Use hybrid routing when
Free or subsidized endpoints can help with prototypes, but they do not remove the need for workload accounting. The free AI models API case study→ is a useful starting point for separating access price from production reliability.
Prompt-cache security needs its own test
Caching creates data-dependent timing: a reused prefix can return its first token faster than a miss. An ICML 2025 audit used timing measurements to test real APIs and reported evidence of cross-user cache sharing at seven providers during the study period.
That result is historical, provider-specific evidence. It does not establish how those services isolate caches today.
Ask current providers and internal platform owners:
For self-hosted systems, include cross-tenant timing and eviction tests. Do not log reusable sensitive prefixes merely to debug the cache. Store hashes, token counts, version identifiers, and policy-safe metadata.
The decision checklist
Do not approve a GPU purchase or an API migration from list prices alone. Require:
Prompt caching is valuable because repeated context is common. It is dangerous as a procurement shortcut because reuse is workload-specific. Measure the prefix, measure the misses, then compare total cost.
Claim checks
| Important claim | Evidence type | Check and limitation |
|---|---|---|
| --- | --- | --- |
| The coding-agent study reported a 99.3% prompt-cache hit rate | Primary research, July 2026 preprint | One developer, one codebase, two sequential periods; the result is not portable |
| The paper reported $0.573/M processed API tokens versus $2.83/M for shared local allocation | Primary research | API processed 16.9× more tokens and local utilization was low; total spend and TCO are stronger comparisons |
| Shared local capacity saved 40.1% TCO while dedicated capacity cost 43.8% more | Primary research | Depends on the paper's hardware, allocation, Taiwan-market, labor, and quality assumptions |
| Prefix caching reuses attention state for repeated prompt segments | Peer-reviewed primary research and official engine docs | It reduces repeated prefill work; decoding and changing suffix work remain |
| Anthropic's 5-minute cache writes cost 1.25× and reads 0.1× base input | Official documentation checked July 30, 2026 | Pricing and supported models can change; the example excludes output, suffix, and storage |
| Gemini implicit caching is default for Gemini 2.5 and newer in the Interactions API | Official documentation updated July 7, 2026 | Minimum token counts and explicit-cache support differ by API and model |
| TokenPilot reported 56%–87% cost reductions across its tested settings | Primary research, work-in-progress preprint | Two benchmarks and a specific integration; independent replication is needed |
| Prompt-cache timing can expose information when cache scope crosses users | ICML 2025 primary research | Historical audit of tested providers; do not infer current vendor behavior |
