Home / Blog / Groundhog Day for AI agents: Prompt caching patterns for…
Tech News

Groundhog Day for AI agents: Prompt caching patterns for using OpenAI

Amazon Bedrock prompt caching reuses an OpenAI agent's tools and instructions, reading 4,120 cached tokens on later turns at a fraction of the input price.

By Dillip Chowdary • Oct 26, 2026 • Source: AWS Machine Learning Blog

Groundhog Day for AI agents: Prompt caching patterns for using OpenAI
Illustration · Pexels

At a glance

Product
Amazon Bedrock
Model
us.openai.gpt-6-astra
Minimum
1,024-token reusable prefix to qualify
Cache TTL
30m
Test result
4,120 tokens written, then read later
Response storage
off; prompt caching still works

Amazon Bedrock stores the input prompt, instructions, tools, and content, and reuses it on the next request at a fraction of the input price. AWS Machine Learning Blog's report shows the pattern for OpenAI models on Amazon Bedrock, model id us.openai.gpt-6-astra. With no internal memory between requests, the model predicts the next token only from that input, so each step resends tool definitions, instructions, every previous tool call, and every result. By turn 20 of a coding session, turns 1 through 19 are re-read and paid again.

This piece follows one agent from its first turn through new tools and helper agents. It is for developers who already have an Amazon Bedrock client, a TOOLS list, and a task, and who need that shared prefix to lower input cost while the agent stays responsive and consistent. It covers prefill, implicit and explicit marks, the 1,024-token minimum, a ttl of 30m, and the loop that appends tool results until a final answer.

01What actually changed

The report opens on the movie Groundhog Day. Bill Murray's character re-lives the same February 2, waking to the same song on the radio, in the same inn, in the same small town, for what some movie fans estimate to be 34 years.

Nothing he accomplished the day before carries over unless he works it out again from scratch. An agent is in the same loop: every tool call, result, and next decision has to send the entire story so far.

A first call to the large language model is four blocks. The tool list defines every callable function and changes when a new release ships.

The new turn is the latest tool result or user message. Figure 1 shows that payload on every turn. Prompt caching stores the input and reuses the matching prefix.

The new turn is the latest tool result or user message. Figure 1 shows that payload on every turn. Prompt caching stores the input and reuses the matching prefix.

02How it works

Those vectors are its working state, built before it writes the first token. Prompt caching saves that state, including the system prompt, tools, content, and history. A later request that starts with the same text loads the saved state instead of computing it again, then processes only what is new in the next payload.

Those vectors are its working state, built before it writes the first token. Prompt caching saves that state, including the system prompt, tools, content, and history. A later request that starts with the same text loads the saved state instead of computing it again, then processes only what is new in the next payload.

Caching can be implicit or explicit, and the report recommends using both. Implicit caching lets Amazon Bedrock choose where the reusable prefix ends as the conversation grows.

Explicit caching uses prompt_cache_breakpoint so tools and instructions are reused across sessions and helper agents. Both forms are not always required.

Combined, a new session reuses that shared start and keeps its own cached history. Figure 2 shows turn 1 saving the prefix up to the breakpoint and turn 2 paying full price only for the new question.

Advertisement

Tech Pulse Daily

Get tomorrow's pulse first

Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.

03Why it matters now

The first saving is static tools and instructions. Samples use the existing Amazon Bedrock client, TOOLS, and task, and replace the shortened INSTRUCTIONS string, "Your full support policies and examples...", with real policies and examples.

The reusable prefix must reach 1,024 tokens to qualify. Explicit mode chooses where the saved prefix ends.

Rendered order is provider context, then TOOLS, then INSTRUCTIONS, with the task after the boundary. The cache starts at the beginning, the marker picks the end, and TOOLS stay before the instructions.

The call is client.responses.create with model us.openai.gpt-6-astra, store set to False, and include set to reasoning.encrypted_content. store=False leaves response storage off while prompt caching still works. prompt_cache_key support-agent:v7:gpt-6-astra names which requests share a cache. prompt_cache_options uses mode explicit and ttl 30m. The developer role sends input_text with prompt_cache_breakpoint mode explicit, saving that block and TOOLS.

The user role's task is processed but not cached. One test wrote 4,120 tokens for a roughly 4,000-token prefix and read those 4,120 tokens on later requests.

04Who is affected

Model Context Protocol servers and other tool delivery need to be indexed so the list stays consistent. The date, the user's name, and a request ID go after the cached instructions. The key support-agent:v7:gpt-6-astra marks a versioned support agent on gpt-6-astra, shared by later sessions and helper agents.

Model Context Protocol servers and other tool delivery need to be indexed so the list stays consistent. The date, the user's name, and a request ID go after the cached instructions. The key support-agent:v7:gpt-6-astra marks a versioned support agent on gpt-6-astra, shared by later sessions and helper agents.

TOOL_HANDLERS maps names to those functions. The loop appends the model output and every tool result to history and resends that list until a final answer. In implicit mode, Amazon Bedrock reuses the unchanged prefix and caches added content through the last tool result when earlier messages and the initial TOOLS stay unchanged.

TOOL_HANDLERS maps names to those functions. The loop appends the model output and every tool result to history and resends that list until a final answer. In implicit mode, Amazon Bedrock reuses the unchanged prefix and caches added content through the last tool result when earlier messages and the initial TOOLS stay unchanged.

05What to watch

Implicit is the default. With no explicit marker, the user task is the latest eligible message, so the breakpoint at its end covers tools, instructions, and the task. store=False keeps history in the application and does not disable caching. That fits the next turn of the same task.

Implicit is the default. With no explicit marker, the user task is the latest eligible message, so the breakpoint at its end covers tools, instructions, and the task. store=False keeps history in the application and does not disable caching. That fits the next turn of the same task.

A different task does not match a prefix that runs through the old task, and the implicit write does not necessarily create a separate entry of only the tools and instructions. An explicit marker makes that shorter prefix reusable across tasks.

The sample sets prompt_cache_breakpoint mode explicit on the developer role, then adds the user role task, and calls ask_with_implicit_cache with request-level mode implicit. The service can also save through the latest eligible message, so the shared prefix remains for later turns, new tools, and subagents.

Frequently asked questions

How does prompt caching change what an OpenAI agent resends on Amazon Bedrock?

Amazon Bedrock stores the input prompt, including instructions, tools, and content, and reuses that prefix on the next request at a fraction of the input price. The model has no internal memory between requests, so an uncached step resends tool definitions, instructions, earlier tool calls, and results. A later request that starts with the same text loads the saved key-value state from prefill and processes only the new payload.

How large must a reusable prefix be, and what did the cache test measure?

The reusable prefix must reach 1,024 tokens to qualify. One test of a roughly 4,000-token tools-and-instructions prefix wrote 4,120 tokens on the first request and read those 4,120 tokens on later requests. The user role's task sits after the cache boundary, so it is processed but not cached.

When should callers use implicit caching, explicit caching, or both?

Implicit caching is the default and lets Amazon Bedrock choose where the reusable prefix ends as the conversation grows. Explicit caching uses prompt_cache_breakpoint so tools and instructions can be reused across sessions and helper agents. The report recommends both, though both forms are not always required: an explicit save on the developer block, with the request left in implicit mode so history can grow on that shared start.

Which model, cache key, and time-to-live do the samples use?

The calls use model us.openai.gpt-6-astra, prompt_cache_key support-agent:v7:gpt-6-astra, and a ttl of 30m on prompt_cache_options. include is reasoning.encrypted_content, and store is False. store=False leaves response storage off and keeps conversation history in the application, and it does not disable prompt caching.

Why does a changed tool list or a new task miss the cache?

Caching matches only exactly the same text, so instructions and tool lists must be serialized the same way every time. Model Context Protocol servers and other tool delivery need to be indexed so the list stays consistent. A different task does not match a prefix that runs through the old task, and values that vary, such as the date, the user's name, or a request ID, belong after the cached instructions.

Sources

Dillip Chowdary

Author

Dillip Chowdary

Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.

Related on Tech Bytes

Advertisement

5-min tech signal

Weekday briefing for engineers who skip the noise.

No spam · Unsubscribe anytime

Advertisement

✈️ CareerPilot

Your AI job-search copilot

Match your resume against live Ashby, Greenhouse & Lever openings — fit scores, job-specific resume optimization and email alerts.

Find matching jobs →

Free Tools

Browse all tools →