Key insights

• Prompt caching provides up to 90% discount on input tokens with GPT-5+ and up to 67% faster response times
• Use prompt_cache_key to route related requests to the same server. One customer went from 60% to 87% cache hit rate
• Warp doubled their cache hit rate by structuring caching in three levels: global, user, and task
• There is no quality difference between responses with and without caching. The output is mathematically identical

---

In Brief

OpenAI's Build Hour on prompt caching, a technique that lets AI models remember work they've already done, saving you money and delivering faster responses. Erica from OpenAI explains how it works under the hood, shows live demos with an AI styling assistant, and shares five concrete optimization tips. Siraj from Warp (a developer tool with 700,000+ users) shows how they more than doubled the percentage of requests hitting their cache.

---

What is prompt caching?

Every time you send a message to an AI model via an API (Application Programming Interface, the way your code talks to the AI), the model has to do heavy mathematical work to understand everything you've sent: instructions, previous messages, images, and so on. Prompt caching means the model remembers this work from last time. If you send the same content again, it doesn't have to redo the job. As Erica from OpenAI explains it: we reuse computation. When multiple requests share the same prefix (the identical part at the start of a prompt), we skip processing those tokens and only use resources on what's new (3:15).

Ground rules

Erica walked through the basic rules for how OpenAI's prompt caching works (3:43):

• Minimum 1,024 tokens: your prompt must be at least 1,024 tokens (roughly 750 words) before caching kicks in. Shorter prompts are not cached
• Blocks of 128 tokens: after the first 1,024 tokens, content is cached in blocks of 128 tokens
• Order must be identical. The model only recognizes work it's done before if everything comes in exactly the same order. Swap two paragraphs, and it has to redo everything
• Works for everything: text, images, and audio
• Happens automatically. You don't need to change your code. OpenAI caches automatically (this is called implicit caching) (4:14)
• Lasts 5-10 minutes. After that, the cache is cleared from memory. With extended prompt caching you can extend this to 24 hours (4:34)

---

What do you save? Price and speed

Price

When the model reuses cached work, those tokens cost much less. Erica showed how discounts have increased with newer OpenAI models (5:00):

With GPT-5, you pay only 10% of the normal price for the part of the prompt that's already cached. For the audio model, it's nearly free.

What does this mean in dollars and cents?

Imagine you're building a chatbot for an online store. It handles 1,000 conversations per day, and each conversation sends the same product descriptions, instructions, and FAQ. That's about 3,000 tokens that are identical every time.

That's the difference between a noticeable expense and something that barely shows on the bill. And this is just input tokens (what you send to the model). Output tokens (what the model sends back) cost the same regardless.

Speed (latency)

Latency is the wait time from when you send a request to when you get the first word back, often called time-to-first-token (TTFT). Erica tested 2,300 requests of varying length (5:38) and found:

• For short prompts: minimal difference (~7% faster with cache)
• For long prompts: up to 67% faster with cache (6:08)

The longer the prompt, the more time caching saves. That's because caching makes the wait time depend on how much new content the model needs to generate, not how long the entire conversation history is.

---

Under the hood: What's actually in the cache?

To understand why the tips later work, it helps to know a bit about how AI models actually process text. Erica spends a few minutes explaining this (6:32).

Attention: how the model "reads"

The heart of a modern AI model is a mechanism called attention. It lets the model decide which parts of the text are relevant to each individual word it needs to understand.

Think of it this way: when you read the sentence "she went to the bank to go fishing", you know that "bank" means a riverbank, because you see the word "fishing" in the context. Erica uses precisely this example to show how the model does something similar, but mathematically (8:59).

For each token, the model creates three representations (8:01):

• Query: "what do I need to know about the other words to understand myself?"
• Key: "what do I represent? Am I a verb, a place, a color?"
• Value: "if I'm relevant, here's the information I can contribute"

The model compares each token's query with all other tokens' keys to find out what's relevant, then retrieves information from the relevant tokens' values. This process happens in layers. A model can have 32 to 64 such layers stacked on top of each other, and the result is an increasingly rich understanding of the text.

What the cache actually stores

All this query-key-value work produces enormous amounts of numbers. Without caching, the model has to redo all this work for every API call, even if 90% of the text is identical to last time. As Erica says: what's stored in the cache is just a giant pile of floating-point numbers (decimal numbers like 0.7382). Not your words, not your text, but the finished mathematical representations that the attention process produced (9:14).

---

What happens when you send an API request to OpenAI?

Erica walked through the actual flow that happens behind the scenes (9:38):

---

Five optimization tips

Erica presented five concrete tips for maximizing cache hit rate (15:05).

1. Use prompt cache key

The problem: OpenAI uses the first 256 tokens to determine which server your request is sent to. But if you send thousands of requests with the same start, traffic must be spread across multiple servers to avoid overload, and then they don't hit each other's cache (15:36).

The solution: Include a prompt_cache_key, an optional parameter that helps OpenAI group related requests on the same server. Think of it as a mailing address: without it, your packages end up at random post offices. With it, everything goes to the right place. Erica compares it to a shard key (a label that tells a database which server should store a particular piece of data) in a database (20:20).

Results: A coding customer increased their cache hit rate (the percentage of requests that hit the cache) from 60% to 87% just by adding this parameter (16:23).

Strategies for choosing a key:

• Per user: when the same user works on the same codebase across multiple conversations
• Per conversation: when users have completely unrelated tasks
• Grouped: combine multiple users under one key to use server capacity more effectively (max 15 requests/min per server)

2. Be mindful of context engineering

Context engineering is about controlling what the model sees: removing irrelevant information, summarizing long conversations, etc. But here's a dilemma: context engineering changes the content, while caching requires that the content stays identical. Erica calls them "inherently at odds" (22:00).

Two approaches:

• Trimming: remove older conversation history entirely. Make large cuts infrequently rather than small cuts on every call. Think of it as cleaning your desk. It's better to do a proper cleanup once in a while than to move one thing every time
• Summarization (compaction): instead of deleting old context, use OpenAI's new compaction endpoint (responses/compact) to create a summary that replaces the full conversations

For the realtime API (voice) with its 32K token limit, Erica showed that a retention_ratio of 0.7 (keep 70% of the context, delete 30% when needed) can save 70% on 30-minute sessions (25:36), compared to trimming a little on every call.

3. Use Responses API with reasoning models

Reasoning models (like o3, o4-mini) think step-by-step internally before giving an answer. They generate hidden "thinking tokens" (the model's internal reasoning that the user never sees). If you use the older Chat Completions API, these thinking tokens are discarded between conversation turns. This means the model "forgets" its own reasoning, and the cache breaks.

With OpenAI's Responses API, thinking tokens are preserved. Erica showed that simply switching APIs can increase cache hit rate from 40% to 80%, plus smarter answers because the model retains its reasoning history (28:49).

4. Consider flex processing over Batch API

If you're processing large amounts of data (e.g., images overnight), you might already use Batch API (sending many requests at once instead of one by one) for a 50% discount. Flex processing gives you the same discount, but with more control (29:16):

• Set service_tier: "flex" on individual requests instead of sending everything as one batch
• Combine it with extended prompt caching and prompt cache key, something Batch API doesn't support

Erica tested with 10,000 requests and found 8.5% higher cache hit rate and 23% lower cost on input tokens with flex (30:23).

5. Use allowed_tools instead of changing the toolset

When you use OpenAI's API with tools (function calling), the tool definitions are part of the prefix that gets cached. If you remove or add a tool, the cache breaks (30:44).

The solution: define all your tools once, and use the allowed_tools parameter to control which tools the model can actually use on a given request. The tool definitions remain identical (cache is preserved), but the model only sees the tools you allow.

---

Customer case: Warp

Siraj, technical lead at Warp (a developer platform where AI agents write and debug code for 700,000+ developers), shares how they think about prompt caching in practice (35:15).

How a code agent works

Siraj explains that AI agents in Warp work in loops (37:05): the user gives a task (e.g., "fix this compilation error"), the agent reads files, runs commands, thinks about the result, and repeats, step by step. With each step, the prompt grows, but most of the content is identical from the previous step. This makes code agents a perfect use case for prompt caching. Tools like Claude Code, where sub-agents (smaller AI helpers launched by a main agent) run in parallel loops, multiply this effect further.

Three levels of caching

Warp structures caching in three layers, from broad to narrow (41:30):

The key: Warp removed all dynamic information from the system prompt and placed it in a separate message afterwards. This way, all users share the global cache, while each user gets their own cache at level two.

Golden rule: Never modify history

Siraj shares an important lesson (40:32): if a user changes their mind mid-conversation, it's tempting to go back and modify the original message. But that destroys the cache for everything that came after it. Warp's solution: add a new message at the end saying "the user has changed plans" — this preserves the entire existing cache.

Results

By introducing a task-scoped prompt cache key, Warp doubled their cache hit rate (43:41). Siraj emphasizes the balance: the key should be stable enough to give a high hit rate, but not so narrow that it spreads requests across too many servers.

---

No trade-offs, seriously

Erica is crystal clear: there is no quality difference between responses with and without caching. Given identical input, the mathematical representations in the cache will be exactly the same as if the model computed them from scratch. The responses are identical (52:09).

The only trade-off is architectural: if you design your entire system solely to maximize caching, you might end up giving the model too much irrelevant context (because you don't dare change it). It's about finding the balance between good context management and high cache utilization.

---

Checklist: Why am I not hitting the cache?

Erica wrapped up with a troubleshooting list (33:49). Experiencing lower cache hit rate than expected? Go through this:

• Are you changing the content? Even an extra space or a timestamp in the prompt breaks the cache
• Are you changing the toolset? Use allowed_tools instead (see tip 5)
• Has too much time passed? Standard cache lasts 5-10 minutes. Use extended prompt caching for 24 hours
• Too many requests? High volume spreads traffic across multiple servers. Use prompt cache key to control routing
• Prompt too short? Prompts under 1,024 tokens are not cached. Erica showed that even with just 50% cache hit rate, you save 33% on token costs by extending your prompt past the 1,024 threshold (51:01)
• Wrong API? Chat Completions with reasoning models gives lower cache hit. Switch to Responses API
• Batch API with older models? Pre-GPT-5 models don't support caching in Batch API. Try flex processing

Practical implications

Solo developers

Start with implicit caching. You don't need to change your code at all. Just use GPT-5+ and you automatically get up to 90% discount on repeated content. If your prompts are under 1,024 tokens, consider adding a detailed system prompt to push past the threshold. Even 50% cache hit rate saves 33% on token costs.

Teams building production apps

Follow the Warp pattern: structure your prompts in three layers (global system prompt, user context, task context) and use prompt_cache_key to route related requests to the same server. Move all dynamic information out of the system prompt and into a separate message. This single change can double your cache hit rate.

Cost-conscious projects

Combine prompt caching with flex processing for a 50% discount plus cache benefits. If you're currently using Batch API, test whether flex processing gives better total cost. Erica's benchmark showed 23% lower input token cost with flex compared to batch.

Test yourself

1. Architecture trade-off: Context engineering and caching are "inherently at odds." How would you design a system that balances both — and when would you prioritize one over the other?
2. Cost model: With a 90% discount on cached tokens in GPT-5+, at what point does it become cheaper to send too much context rather than invest in smart trimming?
3. Server routing: Prompt cache key controls which server a request lands on. What can go wrong if the key is too broad (too many users per key) vs. too narrow?
4. Developer behavior: Erica says caching doesn't affect quality. But can it affect developer behavior — e.g., avoiding prompt improvements because changes break the cache?
5. Transfer learning: Warp structures caching in three levels (global, user, task). How would this strategy look for a completely different application — e.g., a medical advisory service?

---

Glossary

---

Sources and resources