Cutting LLM Costs by 90%: The 3-Tier Prompt Cache Architecture

The Problem With Naïve Agent Prompts

Most agent frameworks rebuild the full prompt from scratch on every round. The user message changes. The tool results change. The conversation history grows. Everything gets concatenated into one long string and posted to the LLM API.

This is architecturally correct — it's the simplest approach. But it's economically broken at scale.

Consider a typical multi-round coding session:

In a naïve implementation, all 43,000 tokens on round 10 are billed as input tokens — including the 8,000 token system prompt that has been identical since round 1.

With Anthropic's prefix caching (and OpenAI's equivalent), tokens that appear at the start of the prompt and match a previous call are cached at 90% discount. The trick: the prefix must be byte-for-byte identical and appear at the same position in the prompt. Any mutation upstream invalidates the cache for everything downstream.

The 3-Tier Architecture

The Myelix Agents planner prompt is split into three tiers based on exactly one property: how frequently the content changes.

Tier 1: Stable

This tier never changes between rounds — and ideally never changes between sessions for a given user. It contains:

• Role definition — who the agent is, what it does
• Core operating protocol — the reasoning loop, tool usage rules
• Schema digest — compressed representation of available tool contracts
• Static knowledge — language-specific idioms, security policies, style rules

Size: ~6,000–10,000 tokens. Cache hit rate: >99%. This tier is never rebuilt unless there is a deliberate release of new agent instructions.

The Stable tier is the most valuable cache target. It is always first in the prompt. It never moves.

Tier 2: Context

This tier changes once per session, or slowly across rounds. It contains:

• Intent anchors — the distilled summary of what this session is trying to accomplish
• Evidence ledger — facts established so far (file paths, decisions made, constraints discovered)
• Clarification log — questions asked of the user and their answers
• Memory context — learnings from previous sessions relevant to this task
• Execution contract — what the agent has committed to deliver

Size: ~3,000–8,000 tokens. Cache hit rate: ~85–95%, depending on session length. The evidence ledger grows incrementally, but the prefix up to the last unchanged line remains cached.

Tier 3: Volatile

This tier is rebuilt every single round. It contains:

• Current user task — the active message or instruction
• Prior tool results — raw output from the last tool call batch
• Gate violations — any failed checks from the satisfaction evaluator
• Scratchpad — the agent's working notes from the current round

Size: ~2,000–12,000 tokens. Cache hit rate: 0% — this content is intentionally different every round and is never expected to cache.

Why the Order Matters

Prefix caching works on exact byte-sequence matches from the beginning of the prompt. If Tier 1 appears first and is identical to the previous call, the provider caches it. If Tier 2 follows and its prefix is also identical, that portion is also cached. Tier 3 at the end is always new — and the provider never has to process the tokens before it again.

The key insight: by putting the most stable content first, we maximise the length of the cacheable prefix. Putting the volatile content first would invalidate the entire cache on every round.

The Numbers

For a typical 10-round session with 40,000 total input tokens per round:

At Anthropic's cache read pricing ($0.30/MTok vs $3.00/MTok for uncached input), that 72% token reduction compounds with a 90% per-token cost reduction on cached tokens. Real-world observed cost reductions in production sessions: 85–91%.

Implementation in Elixir

The three tiers are assembled in loop.ex via the planner_user_prompt/2 function, which takes the current session state and builds the context tier from live state. The stable tier is a compiled Markdown file that ships with the release and never changes at runtime.

defp planner_user_prompt(session, round_state) do
  %{
    # Context tier — refreshed from state each round
    intent_anchors:     format_intent_anchors(session.intent),
    evidence_ledger:    format_evidence(session.evidence),
    clarification_log:  format_clarifications(session.clarifications),
    memory_context:     MemoryContextProvider.fetch(session),
    execution_contract: format_contract(session.contract),

    # Volatile tier — always fresh
    current_task:       round_state.user_message,
    prior_tool_results: format_tool_results(round_state.tool_results),
    gate_violations:    format_violations(round_state.gate_violations),
    scratchpad:         round_state.scratchpad
  }
end

The stable tier is passed as the system parameter to the LLM client, which maps directly to the system prompt cache breakpoint in both Anthropic and OpenAI APIs.

Context Pruning Under Budget Pressure

The system maintains a token budget — by default 1,000,000 tokens. When the context tier approaches this limit, a pruning strategy activates:

1. Oldest journal entries are dropped first — the least recently relevant facts
2. Tool result payloads are replaced with structured metadata — keeping the shape but not the raw content
3. Protected slots are never pruned — active_context, intent_anchors, gate_violations, steer_directives

This pruning is designed to preserve the prefix of the context tier — so that content at the start of Tier 2 remains stable even as later entries are dropped. This maximises cache hits even during aggressive pruning.

Generalising the Pattern

The 3-tier pattern is not specific to Myelix Agents. Any multi-round LLM application benefits from it:

• Customer support bots: Tier 1 = product knowledge base. Tier 2 = customer context. Tier 3 = current message.
• Code review pipelines: Tier 1 = style guide + rules. Tier 2 = PR context + file summaries. Tier 3 = current diff chunk.
• RAG systems: Tier 1 = instructions. Tier 2 = retrieved document set. Tier 3 = query.

The principle is always the same: sort content by mutation frequency, put the slowest-changing content first, and let the provider's caching layer do the rest.

What This Means for Multi-Agent Economics

The cost advantage compounds with concurrency. When Myelix Agents spawns 20 parallel sub-agents, each running its own planner loop, each agent's Stable tier is independently cached. The 10,000-token system prompt is cached once per agent — but since Anthropic's cache is keyed per API key and prefix hash, all 20 agents reading the same stable tier can share the same cache entry.

In practice: the marginal cost of spawning an additional agent in an existing session approaches zero for the stable tier, and is only the volatile tier for subsequent rounds.

This is the economic foundation that makes running 50 concurrent agents financially viable — not just technically possible.