fix: split system prompt into 2 messages for proper Anthropic prompt caching by surapuramakhil · Pull Request #14203 · anomalyco/opencode · GitHub

Issue for this PR

Type of change

Bug fix
New feature
Refactor / code improvement
Documentation

What does this PR do?

In llm.ts lines 67-80, the entire system prompt (static agent/provider prompt + dynamic environment info + dynamic instruction files) is joined into a single string via .join("\n") and pushed as one entry to the system array.

This causes 100% cache miss because:

system.length is always 1 after the join, so the 2-part cache restructuring logic on lines 92-97 (if (system.length > 2 && ...)) never activates — it requires length > 2
For Anthropic's prompt caching, the applyCaching() function in transform.ts places a cache_control: ephemeral breakpoint on the entire system message. Since the static agent prompt (~thousands of tokens) is concatenated with dynamic content (environment with Today's date, instruction files read fresh from disk each call), any change in the dynamic portion invalidates the entire cache
InstructionPrompt.system() is called inside the while(true) loop in prompt.ts line 653, reading instruction files from disk on every iteration. If the agent modifies any file in the working directory, instruction file content can change mid-session

The fix splits the system prompt construction into two separate entries:

Entry 1 (static): The agent/provider prompt — identical across all calls within a session
Entry 2 (dynamic): Environment info + instruction files + user system prompt

This works with existing infrastructure:

applyCaching() in transform.ts already handles up to 2 system messages via .slice(0, 2), giving each its own cache_control: ephemeral marker
The static agent prompt (entry 1) gets cached by Anthropic independently, enabling cache HITs on subsequent calls
The plugin restructuring logic (system.length > 2) now correctly activates if a plugin adds entries, since the base length is 2 instead of 1

How did you verify your code works?

Traced the full code path from prompt.ts → llm.ts → transform.ts → applyCaching() to confirm:

The system array now has length 2 (static + dynamic) instead of 1
applyCaching() marks both system messages with independent cache breakpoints
The header variable on line 82 (system[0]) still correctly references the static prompt for plugin comparison
All downstream consumers (system.map(...) on line 230) handle arrays of any length

Screenshots / recordings

N/A — backend change, no UI impact.

Checklist

I have tested my changes locally
I have not included unrelated changes in this PR

The system prompt was being joined into a single string, which meant the static agent/provider prompt and dynamic content (environment info, instruction files) shared one cache breakpoint. Any change in dynamic content (e.g. instruction files modified by the agent, date rollover) invalidated the entire cache, causing 100% cache miss. Split into two separate system messages: - Message 1: Static agent/provider prompt (stable across all calls) - Message 2: Dynamic content (environment, instructions, user system) This allows Anthropic's prompt caching to cache the static prefix independently, enabling cache hits on subsequent calls within a session.

Hey! Your PR title prompt caching issue ifx doesn't follow conventional commit format.

Please update it to start with one of:

feat: or feat(scope): new feature
fix: or fix(scope): bug fix
docs: or docs(scope): documentation changes
chore: or chore(scope): maintenance tasks
refactor: or refactor(scope): code refactoring
test: or test(scope): adding or updating tests

Where scope is the package name (e.g., app, desktop, opencode).

See CONTRIBUTING.md for details.

The following comment was made by an LLM, it may be inaccurate:

Based on my search, I found one potentially related PR:

PR fix: add prompt caching to session compaction - Issue #10342 #10380: "fix: add prompt caching to session compaction - Issue /compact doesn't utilize prompt caching #10342"
- fix: add prompt caching to session compaction - Issue #10342 #10380
- This PR also addresses prompt caching functionality, though it focuses on session compaction specifically (Issue /compact doesn't utilize prompt caching #10342), while PR fix: split system prompt into 2 messages for proper Anthropic prompt caching #14203 closes Issue Cache hit rate is low / zero #14065.

The searches suggest these are related to different issues, but both deal with prompt caching. Since PR #14203 specifically closes issue #14065, they appear to be addressing separate prompt caching concerns rather than being exact duplicates.

Pull request overview

Adjusts how system prompts are assembled for LLM.stream to improve provider prompt-caching effectiveness (aiming to address low/zero cache hit rate reported in #14065).

Changes:

Split the system prompt into a “static” portion (agent/provider prompt) and a “dynamic” portion (per-call and per-user system strings).
Keep post-plugin system prompt normalization that attempts to preserve a 2-part structure for caching.
Minor cleanup of inline comments.

Comments suppressed due to low confidence (1)

packages/opencode/src/session/llm.ts:100

header is captured from system[0] before plugin transforms, but system can be empty when both staticPrompt and dynamicPrompt are empty. If a plugin then adds 3+ system entries, the system.length > 2 && system[0] === header branch can run with header as undefined, causing an undefined system message to be pushed (and potentially breaking downstream serialization / provider requests). Consider making header a let, and if it’s initially missing, derive it from system[0] after the plugin hook (and guard the rejoin logic so it never pushes a missing header).

    const header = system[0]
    const original = clone(system)
    await Plugin.trigger(
      "experimental.chat.system.transform",
      { sessionID: input.sessionID, model: input.model },
      { system },
    )
    if (system.length === 0) {
      system.push(...original)
    }
    if (system.length > 2 && system[0] === header) {
      const rest = system.slice(1)
      system.length = 0
      system.push(header, rest.join("\n"))
    }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

This change introduces new behavior that splits system prompts into static vs dynamic parts to improve prompt caching, but there’s no test asserting the resulting messages structure (e.g., that the first system message stays constant when input.system / user.system changes, and that dynamic content is isolated in a later system message). Since this file already has request-shape tests for LLM.stream, adding an assertion against the captured request payload would prevent regressions in caching behavior.

Thanks for updating your PR! It now meets our contributing guidelines. 👍

… prefix

…mpt caching (upstream PR anomalyco#14203)

Closing this pull request because it has had no updates for more than 60 days. If you plan to continue working on it, feel free to reopen or open a new PR.

surapuramakhil added 2 commits February 18, 2026 16:25

Merge branch 'anomalyco:dev' into dev

4f402b9

Copilot AI review requested due to automatic review settings February 18, 2026 22:29

github-actions Bot added needs:title needs:compliance This means the issue will auto-close after 2 hours. labels Feb 18, 2026

Copilot started reviewing on behalf of surapuramakhil February 18, 2026 22:29 View session

Merge branch 'dev' into dev

6f7f110

Copilot AI reviewed Feb 18, 2026

View reviewed changes

surapuramakhil changed the title prompt caching issue ifx fix: prompt caching Feb 18, 2026

github-actions Bot removed the needs:title label Feb 18, 2026

surapuramakhil changed the title fix: prompt caching fix: split system prompt into 2 messages for proper Anthropic prompt caching Feb 18, 2026

github-actions Bot removed the needs:compliance This means the issue will auto-close after 2 hours. label Feb 18, 2026

surapuramakhil added 3 commits February 18, 2026 16:50

fix: restore @ts-expect-error comment for ProviderTransform.message

10bf756

Merge branch 'dev' into dev

53084e3

fix: cache instruction file contents within session for stable prompt…

5b2861f

… prefix

bhagirathsinh-vaghela mentioned this pull request Feb 23, 2026

fix(cache): improve Anthropic prompt cache hit rate with system split and tool stability #14743

Open

3 tasks

github-actions Bot mentioned this pull request Mar 14, 2026

feat(provider): add Google Vertex/AI context caching annotations #17485

Closed

henry701 mentioned this pull request Mar 23, 2026

fix(session): stabilize single-message prompt prefix for llama cache reuse #18710

Closed

6 tasks

This was referenced Mar 24, 2026

feat(bedrock): Add token caching for any amazon-bedrock provider #18959

Merged

fix: join system messages into single message #19464

Closed

fix(opencode): keep dynamic user.system separate for prompt caching #20109

Closed

sendhil-anduril added a commit to sendhil/opencode that referenced this pull request Apr 2, 2026

fix(cache): split system prompt into static/dynamic for Anthropic pro…

ec73000

…mpt caching (upstream PR anomalyco#14203)

opencode-agent Bot force-pushed the dev branch from 2f2ed11 to 9f201d6 Compare April 16, 2026 21:55

github-actions Bot closed this Apr 20, 2026

This was referenced May 12, 2026

feat(cache): add cache token audit logging behind OPENCODE_EXPERIMENTAL_CACHE_AUDIT #27007

Closed

feat(cache): split system prompt into stable/dynamic blocks for independent caching #27377

Closed

martinffx mentioned this pull request May 13, 2026

fix(cache): stabilize system prefix behind OPENCODE_EXPERIMENTAL_CACHE_STABILIZATION #27378

Closed

3 tasks

+                  const system: string[] = []
+                  const staticPrompt = [
+                    ...(input.agent.prompt ? [input.agent.prompt] : isCodex ? [] : SystemPrompt.provider(input.model)),
+                  ]
+                    .filter((x) => x)
+                    .join("\n")
+                  if (staticPrompt) system.push(staticPrompt)
+                  const dynamicPrompt = [
+                    ...input.system,
+                    ...(input.user.system ? [input.user.system] : []),
+                  ]
+                    .filter((x) => x)
+                    .join("\n")
+                  if (dynamicPrompt) system.push(dynamicPrompt)

fix: split system prompt into 2 messages for proper Anthropic prompt caching by surapuramakhil · Pull Request #14203 · anomalyco/opencode · GitHub

Uh oh!

fix: split system prompt into 2 messages for proper Anthropic prompt caching - #14203

fix: split system prompt into 2 messages for proper Anthropic prompt caching#14203
surapuramakhil wants to merge 6 commits into
anomalyco:devfrom
surapuramakhil:dev

surapuramakhil commented Feb 18, 2026 •
edited
Loading

Uh oh!

Uh oh!

github-actions Bot commented Feb 18, 2026

Uh oh!

github-actions Bot commented Feb 18, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 18, 2026

Uh oh!

github-actions Bot commented Feb 18, 2026

Uh oh!

github-actions Bot commented Apr 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fix: split system prompt into 2 messages for proper Anthropic prompt caching by surapuramakhil · Pull Request #14203 · anomalyco/opencode · GitHub

Uh oh!

fix: split system prompt into 2 messages for proper Anthropic prompt caching - #14203

fix: split system prompt into 2 messages for proper Anthropic prompt caching#14203surapuramakhil wants to merge 6 commits intoanomalyco:devanomalyco/opencode:devfrom surapuramakhil:devsurapuramakhil/opencode:devCopy head branch name to clipboard

Conversation

surapuramakhil commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue for this PR

Type of change

What does this PR do?

How did you verify your code works?

Screenshots / recordings

Checklist

Uh oh!

github-actions Bot commented Feb 18, 2026

Uh oh!

github-actions Bot commented Feb 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Feb 18, 2026

Uh oh!

github-actions Bot commented Apr 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fix: split system prompt into 2 messages for proper Anthropic prompt caching#14203
surapuramakhil wants to merge 6 commits into
anomalyco:devfrom
surapuramakhil:dev

surapuramakhil commented Feb 18, 2026 •
edited
Loading