Domain 5: Context Management & Reliability

Domain 5 is 15% of the CCA-Foundations exam. It is the domain that separates a demo from a production system. A single prompt that works once tells you nothing about whether a multi-step agent stays correct after fifty tool calls, three summarization passes, and a context reset. This domain tests how you engineer the context window, propagate errors honestly, annotate provenance and coverage, persist state across boundaries, and control token cost while keeping a human in the loop where it matters.

Blueprint at a glance

The objectives cluster into three areas, mirroring the course modules:

1. Context engineering — fighting lost-in-the-middle, summarizing without losing facts, and budgeting the window.
2. Reliability & provenance — structured error propagation, coverage/citation annotations, and durable scratchpads.
3. Quality assurance & cost — stratified human review, blocking vs post-hoc gates, and prompt caching.

Context engineering

Lost-in-the-middle

Recall across a long context is U-shaped: a model attends most strongly to the beginning and end of its input and weakest to the middle. A 200K-token window does not mean 200K tokens of reliable recall. The architectural responses are:

• Place the most critical instructions and highest-relevance context at the ends of the prompt.
• Restate key constraints near the end, just before the model must act.
• Reduce middle bloat by trimming and by delegating verbose discovery to subagents — do not rely on a big window to "remember everything."

A bigger context window is a capacity, not a guarantee. The exam consistently rewards answers that engineer placement over answers that just add more tokens.

Progressive summarization

When a conversation outgrows the window, you compact older turns into a summary. This is necessary but lossy — every pass discards detail and errors compound across repeated summarization. Best practice:

• Keep durable facts on disk (a scratchpad or JSON snapshot), and only summarize prose.
• Trim removable tool noise first, then summarize what remains.
• Treat summarization as a last resort behind lossless trimming and subagent delegation, never as a way to recover lost detail.

Token budget

In an agent, the dominant cost is usually tool output re-sent every turn, not the system prompt. Budget the window into regions (instructions, recent turns, retrieved context, tool output) with ceilings, and evict to protect instructions and recent turns. The optimization order is:

1. Trim at the source (lossless) — strip null fields, truncate logs, keep only needed columns.
2. Delegate verbose discovery to a subagent to isolate the firehose from the parent window.
3. Cache the stable prefix (see below).
4. Summarize only as a final step.

Reliability & provenance

Structured error propagation

The single most important reliability rule in this domain: never conflate an access failure with a valid empty result. "I could not read the file" and "the file is empty" are different facts and must be different outcomes. In the Messages API, a tool returns a failure with is_error: true and a structured, typed payload:

Classify errors (transient / validation / permission / not_found) so the model can react correctly — retry a transient timeout, escalate a permission error. Propagate failures up the stack; do not swallow them or silently substitute a default. Fire escalation triggers — a non-retryable error, an exhausted retry budget, a high-stakes or irreversible action, or non-convergence — to hand off to a human instead of guessing.

Provenance and coverage annotations

Trustworthy agent output carries provenance (source, tool, line, timestamp — ideally real citations) and coverage (how much was examined vs. the total, what was skipped, and a confidence signal). Require these in the output schema, not as optional prose the model may forget. Coverage is what makes an empty result believable: it lets you distinguish "verified absent" from "not examined," the same honesty principle as the access-failure-vs-valid-empty distinction.

Scratchpads and durable state

The context window is ephemeral — compacted, reset, and not shared across subagent boundaries. Durable state belongs in files, not the window:

• A scratchpad (markdown TODO / decisions) for human-readable working memory.
• A JSON state snapshot (cursor, phase, processed IDs) for resumable machine state.

Write state after each unit of progress and make resume idempotent so a restart re-reads the snapshot instead of redoing work. Files are also the clean subagent handoff channel, isolating verbose discovery from the parent's window.

Quality assurance and cost

Stratified human review

You cannot human-review everything in a high-volume pipeline, and random sampling under-represents rare, high-risk cases. Use stratified sampling to over-sample risky strata. Choose the gate by stakes:

• Block (synchronously) on high-stakes / irreversible actions — a human must approve before execution.
• Post-hoc sample for quality and drift monitoring on low-stakes output.

Auto-route every escalation and is_error case to review, and run the non-blocking audit through the Message Batches API — it is cheaper and asynchronous. Never use a batch for the blocking gate, because batches are not real-time.

Prompt caching

Prompt caching is the top token-budget lever for long, stable prefixes. The cacheable prefix follows the order tools → system → messages, and the cache matches the longest exact prefix — so anything that changes invalidates that block and everything after it. The rule: stable content first, volatile content (timestamps, IDs, the new turn) last. Add cache_control: { type: "ephemeral" } to the last block of each stable section (up to 4 breakpoints). Reads cost ~0.1x input (a 90% discount); writes ~1.25x (5-min TTL) or ~2x (1-hour TTL). Below the minimum length it silently no-ops — verify with cache_read_input_tokens in the usage object.

How it ties together

Domain 5 is a single coherent idea: a long-running system must stay honest about what it knows and degrade gracefully when it doesn't. Engineer placement against lost-in-the-middle, prefer lossless trimming over lossy summarization, propagate typed errors instead of guessing, annotate provenance and coverage so outputs are auditable, persist durable state in files, sample reviews by risk, and cache the stable prefix to make all of it affordable.