One model call per turn now does many jobs at once: the hosts confer, several speak, one may hand over a document or a diagram. The more the model carries, the more ways its output can arrive slightly off-spec — and anything downstream that trusts the format breaks. The harness is the whole chain that catches it: the server pipeline (lib/run_room.py) that turns one completion into correctly split, typed, attributed bubbles and cards, and the client render that has to resolve identities and lay them out. This page records the 2026-06-12/13 hardening and what production traffic has since taught us — including one assumption it overturned.
The failure modes are not abstract — they are the things a guest actually sees go wrong:
Each traces to the same root: the model's output is an in-band channel — structure and content share one stream — and a parser that treats a near-miss as fatal turns a cosmetic slip into a lost turn.
The model is the only non-deterministic actor. Everything after it is plain code whose job is to never lose what the model produced — and the client never parses model output at all; it renders pre-split JSON pushed over SSE.
one call → a panel_turn tool-call (typed JSON) or <speak>/<confer> prose: the backstage confer, the on-stage speeches, any artifacts
tool JSON through a repair ladder (bad escapes · unescaped inner quotes · truncation) before it's called dead; or the tolerant <speak> tag scan — any quote style, or none
↺ the tool's args written as plain content (a bare replies array) → decoded like the tool, per host; empty / un-parseable → retry, then salvage as the lead host — never a blank turn
confer paired by a tag scan (unclosed → stays in the drawer — unless it swallowed the whole turn, when it's promoted to the stage rather than leave a blank one); a missing </speak> healed; the non-dialogue "negative space" captured as artifacts
artifact kind by structure (prose-guarded, so "pie/timeline" speech stays speech); who resolved deterministically; the cast gate drops or quarantines a line from a speaker who isn't seated
one pass (extract_panel_blocks) lifts every action tag out of the reply text — rolls · reacts · notes · reveals · boards · gates · seals · votes · clocks — after the canonical history is frozen, so the panel's memory keeps the grammar it used while no client ever sees a tag
↺ a tag the model wrote outside its <speak> is salvaged (extract_outside_blocks) rather than dropped; an unusable tag still strips and is counted (clock_bad · vote_bad) — a leaked tag in a bubble is the one failure worse than a no-op; a reply emptied by its tags drops, because a pure action needs no bubble
re-serialized into one canonical <speak> form → history (the panel's memory) + the event log (with the verbatim model output + a parse report) + cost
per-persona bubbles · wildcard cards · Mermaid SVG — the browser renders the already-split JSON, so it can never re-introduce a parsing bug
The structured panel_turn tool is the durable fix for in-band tags — the turn arrives as validated JSON, so content can't be mistaken for framing. DeepSeek rejects a forced tool_choice in thinking mode, so the tool is offered "auto" and the model chooses. Pre-deploy sims said Pro takes it almost always (26/26) — so we expected the raw-<speak> parser to matter mainly for Flash. Production overturned that. Measured over the first instrumented turns on the live box:
| Model | Tool path | <speak> text path | Share on text path |
|---|---|---|---|
| DeepSeek V4 Pro | 8 | 31 | ~78% |
| DeepSeek V4 Flash | 1 | 13 | ~93% |
| All turns | 9 | 44 | ~81% |
<speak> prose (the SP documents it as a valid format) and declines the tool outright: every text-path turn carried no tool call, not a tool call that fell back. So the raw-<speak> parser isn't a rare safety net — it is the path ~81% of production turns travel, for both models. That makes P1's tolerant tag parser the load-bearing work, not an edge case. The reassuring corollary: because P1 made that path lossless, the low tool-use rate is fine — we hardened the path the model actually prefers, so there's no need to fight DeepSeek for more tool calls (which thinking-mode won't allow anyway).parse report — which path ran, what was repaired, healed, coerced, dropped, or quarantined. Process totals feed the admin Status → Parse health row. Before this, a parse incident left no trace; now it's a number.<speak> scanner (a who='x' / who=x / fullwidth-quote slip — or a drifted identity keyword, name=/speaker=/persona= — no longer drops the speech); stack-based confer containment; the tool-JSON repair ladder; field-type coercion instead of a crash; and quarantine-not-silence — a turn the gate would empty becomes a visible "Unattributed reply" note, kept out of the model's memory.mermaid fence lifts to its own card (not just the first); ~~~ and {mermaid} fences are accepted; ambiguous heads (pie/journey/timeline) need real structure before they count, so prose isn't retyped as a diagram; an unterminated fence stops at the first line of prose.resolve_who is tiered (slug → exact name → unique word → unique substring, with a minimum-signal rule). A name that fits two seated hosts — two Steves, two 李 — is marked ambiguous and quarantined, never bound to whoever sits first. And a host quoting <speak> inside a code fence no longer fragments the reply.@-mention stuck as "@someone". The cause wasn't the parser — the server stored and resolved it correctly — it was a render race: a room paints its turns synchronously, but the members list loads async, so a mention painted before that list arrived fell to the muted "someone" placeholder and was never re-touched. The fix is the harness invariant applied to rendering: degrade, then RECOVER — when the members land, a recovery pass re-resolves every mention span in place, so a transient miss heals the moment the data arrives instead of sticking forever. Diagnosed by reading the live room's stored events, not by guessing.Each failure class was pinned by a probe, then re-measured against the shipped code and a fresh 30-call live batch through the production pipeline.
| Failure class | Before | After | Evidence |
|---|---|---|---|
| Bad tool-JSON (escape · inner quote · truncation) | whole turn lost | recovered in place | ladder · fuzz 1833/0 |
who= quote variant / absent | speech dropped | kept | probes S1–S7 |
Unclosed <confer> | leaked on stage | held in drawer (promoted if it swallowed the turn) | probes C1, C3 |
| Stage speech before the confer | discarded | kept | probe C2 |
| Malformed tool-arg shape | parser crash | coerced | probes T1–T5 |
| Second diagram fence | stranded in bubble | own card | probe M1 |
| "pie / timeline …" prose | retyped as diagram | stays speech | probes M5, M6 |
| Shared first name | bound by seat order | quarantined | probes R1–R3 |
| All-ghost turn | silent blank | note card | probe G1 |
The deepest lesson came from the benchmark: a fresh batch surfaced unescaped inner quotes (…"绝对不要借钱"。) as the most common tool-JSON corruption — a class the ladder didn't yet handle. It was closed and re-shipped before any user hit it. That is exactly the loop P0 exists to enable: measure live, find the next class, close it.
Those before/after numbers were synthetic sims — the honest pre-deploy check. The page now reports the real thing: the durable per-turn parse records reconstructed from the live box (the in-RAM admin counter resets on every restart, so the truth lives on disk). Over the first instrumented production turns:
| Signal | Production | Reading |
|---|---|---|
| parser crashes | 0 | no turn lost to a crash |
json_repair (ladder rescued) | 3 | all the inner-quotes rung, firing on real Chinese rooms — turns that would have been blank pre-P0 |
ghost_lines dropped | 5 | cast gate catching contamination, as designed |
salvage (tag-less → lead host) | 1 (~2%) | rare — so P4 stays low priority |
| quarantined turns | 0 | no turn went unattributed |
quotes fix is confirmed working on the exact traffic that motivated it.replies array as plain content — neither a tool-call nor <speak> prose — so the tag-less fallback salvaged the entire blob into the lead host's one bubble, collapsing every voice. The fix is the same invariant: a strictly-gated decode (path=json-content) reads that content like the tool, restoring each host to its own bubble; a JSON deliverable a guest actually asked for is left untouched. Pinned by probes G4/G5 and the JSONCONTENT selftest — surfaced by replaying the one real production turn that broke, exactly the loop P0 exists to enable: measure live, find the next class, close it.The harness is exercised by a reusable probe suite (run after any change to it). It lives outside the deployed tree — runtime scratch, never committed — so it's recorded here and in project memory rather than discoverable from the repo alone.
| Probe | What it proves |
|---|---|
probe_parser.py | 33 adversarial cases (speak tags · confer · mermaid · tool-args · resolve_who · room-level). Exits non-zero on any regression — the tolerance spec. |
probe_live.py | ~30 real DeepSeek calls with the exact production request assembly; captures every raw response for offline replay. |
probe_analyze.py | runs the captures through the production pipeline; per-scenario metrics (tool rate, repairs, ghosts, leftover fences, who-form). |
probe_replay.py | differential old-vs-new outcome counts — proves a change recovers turns without regressing the rest. |
probe_fuzz.py | ~1833-trial fuzz of the JSON repair ladder (random truncations / injected escapes / garbage) — asserts zero crashes. |
probe_e2e.py | live Room.say() end-to-end on Pro and Flash; checks the parse report + raw capture land on the event. |
mermaid_check.html | renders every captured diagram in the vendored Mermaid bundle — ground truth, not a guess (28/28). |
parse_health_pull.py | the production diagnosis. Read-only pull of the durable parse records off the live box, a verbatim corpus saved for keeps, and --replay through the current parser — real traffic, not sims. |
Plus the always-on guards: run_room.py --selftest gained named blocks (TOLERANT · CONFER · REPAIR · COERCE · QUARANTINE · DIAGRAM · RESOLVE · SHIELD · JSONCONTENT), and lib/smoketest.py stays green.
The admin Status → Parse health row is a live, at-a-glance counter — but it lives in RAM and zeroes on every restart (the nightly 05:00 timer that guards memory creep, and every deploy). It answers "is anything wrong right now," not "how has it trended."
The longitudinal record is durable on disk: every turn's parse report plus the model's verbatim output is written to each room's event log under /var/lib/mad/, outside the repo, untouched by restarts. That's the real benchmark — and it doubles as a growing corpus of real model output to replay through any future parser change. One read-only command reconstructs the cumulative and saves the corpus; the diagnosis is repeatable, on real traffic, whenever there's more data.
Retention note: a delete-forever room takes its captures with it (the box's 7-day disk snapshot is the backstop), and a future §9 history-compaction must preserve the parse fields before trimming.
P0–P3 plus the benchmark fix are deployed to production (commits 46a873e → d4e691b, live since 2026-06-13). The client @-mention recovery pass (relabelMentions()) is live too. No prompt wording changed, so no cached prefix was invalidated. Two phases are deliberately held back:
<speak> fallback path — now known to be the majority path — history stores the healed raw rather than the re-serialized parse, so the model's memory can differ slightly from what the guest saw. Narrowed by P1/P3, not closed; production shows the divergence signals (salvage ~2%, quarantine 0) staying low, so it can wait. Touches durable history → its own deploy-and-observe cycle.smoketest.py so the tolerance spec gates every commit. Pure test infrastructure; no runtime surface.Known residuals (by design): a salvaged tag-less turn still assigns the lead host (flagged path=salvage in telemetry, not silent); a bare non-mermaid fence in speech is cosmetic (0 in production).
source records: lib/run_room.py (the pipeline) · lib/room-ui.html (the client recovery pass) · rooms-dev/_probe/ (the battery + parse_health_pull.py, gitignored) · project memory harness-hardening