Missing Cache

Signs and symptoms

An online store’s semantic search computes an embedding for every query and, for hard queries, calls a model to rewrite the search. A one-hour traffic sample (anonymized):

query                          times in the last hour   paid calls
"black sneakers"               1,204                    1,204 embeddings
"black sneaker"                388                      388 embeddings
"black sneakers for men"       301                      301 embeddings + 301 rewrites
"laptop for programming"       277                      277 embeddings + 277 rewrites

The same queries, typed again by different people, are processed from scratch every time. Meanwhile, the same site’s shopping assistant sends an identical 9,000-token system prompt with every message, with no prompt caching.

In a Brazilian marketplace this mixes with NF-e (electronic invoices) and shipping. The product sheet barely changes. “Do you have it in 220V?” repeats all day. Each repeat pays for an embedding, sometimes a rewrite, sometimes the answer model.

Signs of a Missing Cache:

• Identical or equivalent inputs are processed from scratch every time.
• The prompt’s fixed prefix (instructions, tools, documents) is large and never cached.
• The cache token fields in the API’s usage are always zero.
• Embeddings for text that never changes get recomputed.
• Cost grows linearly with traffic, even with lots of repetition.

flowchart LR
  Q1(["1. 'black sneakers'"]):::io --> M(["2. Processed from scratch"]):::llm
  Q2(["1. 'black sneakers' again"]):::io --> M
  Q3(["1. 'black sneaker'"]):::io --> M
  M --> C[("3. Paid three times")]:::store

1. Repeated or equivalent inputs arrive.
2. Each is processed as if it were new.
3. Cost and latency repeat every time.

Check +20 XP on first tryIn one hour, 'black sneakers' appears 1,204 times and each one generates an embedding from scratch. Which measurement confirms the smell?

Search p50 latency

The distribution of repeated queries and the cost per distinct query

The size of the vector index

Reasons

• Caching looks like premature optimization. Early on, volume is low and repetition does not show.
• Prompts built with variable data at the start. A date or an id at the top of the prompt prevents prefix caching.
• Nobody looks at the input distribution. Without measuring, nobody knows how much repetition exists.
• Fear of stale answers. A cache with no expiry rule can serve outdated data, so the team prefers no cache at all.

Common trap: turning on prompt caching and leaving the date at the top

Model providers offer prefix caching: the identical leading chunk of the prompt is billed cheaper on later calls. The condition is that the prefix is byte-for-byte the same. If the first line is Current date and time: 14:31:07, the prefix changes every second and cache_read_input_tokens stays at zero. You “turned caching on” and still pay for the whole prompt.

Cost of ignoring

• Money. Paying again for work already done, on every repeated request.
• Latency. An answer that could arrive in milliseconds takes seconds.
• Rate limits. Repeated calls eat the quota you will need at peak.

For scale only, suppose US$ 0.0001 per query embedding and US$ 0.01 per rewrite. In the sample hour, “black sneakers” and “black sneaker” alone add up to about 1,600 embeddings. With no cache, that repeats every hour of the day. The assistant’s 9,000-token prefix stacks on top, on every turn of every conversation.

The first cache that usually pays is the prefix: it does not serve a stale product answer, it only skips re-reading instructions the model has already seen. Measure cache_read_input_tokens after you move timestamps and ids below the stable block. If that field stays at zero, you have not cached anything yet.

Treatment

Refactoring	Choose it when
Introduce Cache	There is a large fixed prefix or repeated inputs. Prompt caching for the prefix; exact or semantic caching for answers to equivalent questions; an embedding cache for text that does not change.

Three layers, from safest to most delicate:

1. Prompt prefix. Stable instructions and tools at the front, changing data (time, balance, name) after. Check cache_read_input_tokens for real, rather than trusting that the flag is on.
2. Embeddings and search. The key is normalized text. Recompute when the document changes, not on every request.
3. Answers. Only for stable questions (delivery time, return policy). Never for balances, live prices, or order status.

Normalizing (“black sneakers” and “black sneaker”) catches the exact cache. True equivalents (“black sneakers for men” vs “men’s black sneakers”) need a semantic cache, with a threshold measured on an eval. Without the threshold, you serve the wrong answer with confidence.

Check +20 XP on first tryThe shopping assistant sends an identical 9,000-token system prompt with every message. Which cache do you turn on first?

Prompt caching on the fixed prefix

A semantic cache of every answer

An embedding cache of customer messages

Payoff

• Cost that grows slower than traffic.
• Instant answers for what repeats.
• Quota left over for peaks.
• A clear measure of repetition that shows where to optimize.

When to ignore

• Always-unique inputs. Different documents on every request, with no meaningful shared prefix, do not benefit from caching.
• Answers that change every second. Real-time prices, account balances. Caching answers here is dangerous; prompt caching may still pay off.
• Low volume. If the bill is small, the complexity does not pay. Prompt caching is often a single line, so start there.

Is it this smell?

Check +20 XP on first tryThe system prompt is 6,000 tokens and starts with 'Current date and time: 2026-10-02 14:31:07'. Prompt caching is on, but cache_read_input_tokens is always zero. Why?

The provider does not support caching

The prompt is too short to cache

The prefix changes on every call

Check +20 XP on first tryA support chat caches answers by exact question. 'What is the delivery time?' and 'what's the delivery time?' produce two calls. What should you consider?

Normalize the question or use a semantic cache

Nothing, the exact cache is enough

Cache the embedding of the whole catalog

Check +20 XP on first tryThe assistant answers with the customer's account balance, which changes with every transaction. The team wants to cache answers. What do you do?

Cache by question, for an hour

Cache the answer for 24 hours and invalidate at midnight

Do not cache the answer; cache only the prompt prefix

Check +20 XP on first trySearch recomputes the embedding of every product sheet on every query, because the text is loaded from the database each request. What do you do?

Cache the search response for 1 second

Store the embedding with the product and recompute only when the description changes

Stop using embeddings and search only by exact text