Each turn of the interactive panel re-sends the same large prefix — the megaprompt instructions plus each persona's profile (a room has a per-room cast; the room is scene-less, so there are no briefs, and the running transcript is the only memory — no journals). Caching lets you pay for that once and read it cheap on every turn after. Here's the mechanic, then the two ways to handle the deliberation. This is the mechanic; the cost trade-offs (1h vs 5-min, write multiples, break-even) live in the economics companion, Cache TTL economics.
Everything you send is one stream, read front to back. Caching looks for the longest run that is identical to a previous request, starting at the very front.
The system block is the bulk of the tokens and never changes. That's the prize: re-reading it for free every turn.
Byte-identical isn't enough. A block also has to sit in the same place with everything before it unchanged. Change something in the middle and the cache stops there — even if later blocks are identical, they come after the change, so they're re-read.
C is a miss even though it didn't change, because the hit ended at B. Order and position matter as much as content.
Put the things that never change at the front, and the things that change every turn at the end. Anything volatile placed early pushes the "first differing byte" forward and throws away the cache for everything after it.
After the panel deliberates backstage, you choose what to store in the history for next turn: the whole confer, or only the public responses. Watch the same blocks turn gold (new) once, then green (cached) forever after. The two options differ only in how big the cached block is — never in hit-vs-miss.
Bigger cached block — the panel "remembers" how it argued. All cheap reads.
confer₁ → side panel, not stored in history
Smaller cached block — leaner, less memory of the argument. Still all cheap reads.
Both columns: system + every prior turn is a cache hit (green); only the new user line and the new generation are full price. They commit once to what each turn looks like and only ever append — so the prefix stays identical going forward. The choice is memory vs. leanness, not cheap vs. expensive.
Turn 1 pays a one-time write for the materials (~1.25× input). Every turn after reads them at ~0.1×, and only the new user line + new generation are full price. So per-turn cost stays roughly flat as the chat grows — and the longer the conversation, the more the one-time write is amortized.
Changing the cast is bust #3 — but only the naïve way. The fix is §4 and bust #2 again: freeze the system block at room creation and never touch it; let everything new ride in at the tail, after the last turn. The dialog (§5) is already a cached, append-only prefix — so the one cache-safe place for a new persona is after the last <turn>, right where the next user line goes. This is the product's live-cast feature — shipped: a room's cast is per-room, and voices can be added or retired mid-room exactly this way.
The whole dialog is re-sent every turn (it's the panel's only memory) — but as a cached prefix it re-reads nearly free. A joiner's profile slots in after that block, not into the system:
System + every prior turn stay green. Only Feynman's sheet is written, once. A join costs one sheet — no matter how long the conversation.
The tempting wrong move: tuck Feynman into the system block with the others — even at its very end.
The system sits before the dialog, so touching it — even appending to its end — moves the first-differing byte ahead of the conversation and re-reads all of it. The §3 miss, self-inflicted. (That's the catch in "just append the profile": append it to the sequence, after the last turn — not to the system block.)
Remove → "stop voicing X" at the tail, don't delete. Same rule. Deleting a sheet from the system shifts the prefix and re-reads the dialog. Instead leave the sheet where it is — dormant, still a cheap read — and append a "stop voicing X" line after the last turn.
Lovelace's tokens stay cached but inert — you keep paying the cheap read, and avoid a re-write entirely. A retire is essentially free.
Append-only (§8) is the steady-state mode — cheap, cache-friendly. But it's a log, and a log of cast changes goes wrong when the same voice churns in and out. Picture Feynman toggling all session:
+Feynman / −Feynman. Cheap in tokens, but the model must now reconstruct "is Feynman in right now?" from a pile of contradictory tags — and at high churn it mis-tracks. A correctness failure, not a cost one.Compaction is the fix: a one-time full rewrite that collapses the log into a clean snapshot of the current state — the same log→snapshot move as an LSM-tree or event-sourcing.
The ledger is gone, the current cast is explicit again. Feynman's past lines stay (real memory) — only his sheet and the tags drop. Append-only then resumes against the fresh snapshot.
Code can compact it precisely — no model needed. The prompt is a projection of the canonical event log + the current cast; the server never parses its own prompt, it re-renders from source:
Exact and lossless for the dialogue — this is cast / control compaction. (Summarising the dialogue itself to save tokens is a different, lossy job that would need the model — not this.)
And code can tell when to compact — thresholds on counters the server already keeps:
Two triggers: opportunistic (a cold cache past TTL, or an add — the write is forced anyway, so the garbage-collect rides free) and forced (churn or dead weight crosses a line — correctness now outranks the one-time write, so you compact even on a warm cache). You can't compute the model's exact confusion point — and you don't need to: set the threshold conservative and compact a little early. One cheap write beats a mis-voiced turn.
How cruft_pct is computed — the dead-weight share of the cached prefix, tracked at no extra cost:
The denominator falls out of the usage you already log; the numerator is a running counter the server bumps on each cast change (retire → move that sheet's tokens live→dead, re-add → back, each tag → +dead). A rough count (len(text)/≈4) is fine — it only feeds a threshold. E.g. two dormant ~2k sheets + ~30 tags ≈ 4.4k dead against a ~28k prefix → ~16%, over the 15% line → compact on the next write.
“Cast is now: Confucius, Lovelace.” rather than “stop voicing Feynman.” The model then reads the latest roster instead of replaying the ledger, which pushes the confusion trigger far out — leaving compaction to handle mostly token weight. Append-only stays clean longer; compaction still resets it eventually.