One model call per turn now does many jobs at once: the hosts confer, several speak, one may hand over a document or a diagram. The more the model carries, the more ways its output can arrive slightly off-spec — and anything downstream that trusts the format breaks. The harness is the whole chain that catches it: the server pipeline (lib/run_room.py) that turns one completion into correctly split, typed, attributed bubbles and cards, and the client render that has to resolve identities and lay them out. This page records the 2026-06-12/13 hardening and what production traffic has since taught us — including one assumption it overturned.
The failure modes are not abstract — they are the things a guest actually sees go wrong:
Each traces to the same root: the model's output is an in-band channel — structure and content share one stream — and a parser that treats a near-miss as fatal turns a cosmetic slip into a lost turn.
The model is the only non-deterministic actor. Everything after it is plain code whose job is to never lose what the model produced — and the client never parses model output at all; it renders pre-split JSON pushed over SSE.
one call → a panel_turn tool-call (typed JSON) or <speak>/<confer> prose: the backstage confer, the on-stage speeches, any artifacts
tool JSON through a repair ladder (bad escapes · unescaped inner quotes · truncation) before it's called dead; or the tolerant <speak> tag scan — any quote style, or none
↺ the tool's args written as plain content (a bare replies array) → decoded like the tool, per host; empty / un-parseable → retry, then salvage as the lead host — never a blank turn
↺ the tool-call COSTUME (2026-08-24, on a local seat): the model describes a tool call instead of making one — a markdown heading, a literal <panel_turn> wrapper, a ```json fence, the function-call form panel_turn({…}), or Mistral’s raw [TOOL_CALLS] envelope. _peel_tool_dressing strips only the dressing (one layer per pass, any nesting order) and hands the core to the same strict who-keyed gate, which still decides — so prose that merely opens with a heading peels to prose, fails the gate, and salvages exactly as before. tool_mimicry joins the parse report. ⚠ A fourth dress needs DIGGING, not peeling (2026-08-25, Cydonia): a markdown report about the call — heading, prose, the arguments fenced in the middle, more prose after — stops the layer-peel at its first prose line, so _dig_tool_json pulls candidates out of the body instead. Every candidate, peeled or dug, faces the same gate at _args_if_panel_shaped
confer paired by a tag scan (unclosed → stays in the drawer — unless it swallowed the whole turn, when it's promoted to the stage rather than leave a blank one); a missing </speak> healed; the non-dialogue "negative space" captured as artifacts
↺ DSML residue (2026-08-23, with the vision id): an experimental DeepSeek model can print its own tool-call markup as prose（</｜｜DSML｜｜parameter>…）and the text path carried it straight into the bubble. The tags are stripped and the words between them kept — degrade, never delete — before the <speak> heal runs, and dsml_stripped joins the parse report. The strip is the catch, not the cure: the standing fix is one layer out — a seeing mind is never offered the panel_turn schema at all, so it has no tool markup to spill
artifact kind by structure (prose-guarded, so "pie/timeline" speech stays speech); who resolved deterministically; the cast gate drops or quarantines a line from a speaker who isn't seated
one pass (extract_panel_blocks) lifts every action tag out of the reply text — rolls · reacts · notes · reveals · boards · pads · gates · seals · votes · clocks · acts · rules · composes · entries — after the canonical history is frozen, so the panel's memory keeps the grammar it used while no client ever sees a tag. acts is the lifecycle verb (<act op="close|reveal|away" target="vote#1"/>), added with the address layer: the rows that put a new card down are named separately as PANEL_ARM_BLOCKS, which is the whole of the AI-side tools-off door. Two widenings since: G8's chip ticks ride the same tag as ops rather than a fourth grammar (arm · inc · dec · set beside close · reveal · away, with an optional n=), and the whole table is an address too — target="table" (or 桌面) is the game-over sweep, which the applier had always understood and the reader had always thrown away: both the attribute regex and _norm_handle matched only word#digits, so a real sweep tag arrived with an empty target and was refused as a card by that name (2026-08-02 — a door whose only test bypasses the door is a door that stays shut; the smoketest now goes through the parser). rules is the game device's ruling moment (D3): <rule pick="是"/>, a deliberate sibling of <act> — same shape, same stripping, same「a tag never renders」guarantee — because a row written from prose is a row written by a parser guessing at a sentence; the cartridge declares which answers are legal and anything outside that list is refused and re-asked. Two more siblings joined that family with the fourth cartridge (2026-08-03): composes is the authored round (<compose q="…" opts="…|…|…" key="2"/> — the host writes the question, the world keeps the key), and entries is the collect window's strict form (<entry who="Dan" count="3" face="4" mode="zhai"/>) — the host reads a player's natural line（「三个四斋」）into the machine's shape and the world rules on it against the window's named validator, so the reading is the persona's and the verdict never is. ⚠ The attribute set was one grammar's, and the second collect window found it (2026-08-06, 谁是卧底's describing round): the reader demanded a count, so a described line's tag was thrown away before anyone could rule on it — the panel could not land a single description and only the flash path worked. kind now sits beside count · face · mode and the row is kept if it carries either（<entry who="Dan" kind="desc"/>）, because the second grammar reads a line into a class（desc · copy · empty）rather than into numbers — a description has no strict form to transcribe into, and「is this a copy」is a judgment, so the mind hands over a closed kind and the arithmetic stays the world's. One raw tag becomes its grammar's strict form at a single reader (_game_entry_read), and an unrecognised kind fails OPEN to desc — a void the player did not earn is worse than a slack line getting through, because the table blames the player and the machine wrote the sentence. Both follow the family's law exactly: one self-closing tag, stripped everywhere, never rendered — and a reply that was only the data line drops, because an interpreting beat needs no bubble. Two more joined the family on 2026-08-04, each with its own pass rather than a widening of extract_panel_blocks, because neither puts a card down — both hand the room a door: <kit name="大话骰"/> (extract_panel_kits) is the host's mount move — it raises an offer card and a human's tap does the mounting, so no keyword and no tag ever seats a game by itself — and <dial name="ones" pick="always_wild"/> (extract_panel_dials) relays the room's rule ask, which the world then validates by option id and enforces from the next round. Same stripping, same never-rendered guarantee, and the same drop-the-empty-reply rule; what makes them worth their own readers is that the reading is the persona's and the authority never is
↺ a tag the model wrote outside its <speak> is salvaged (extract_outside_blocks) rather than dropped; an unusable tag still strips and is counted — clock_bad · vote_bad · roll_bad · seal_bad · board_bad · deal_bad · deck_bad · act_bad (a malformed deck refuses whole, never a partial deal; an <act> whose handle names nothing on the table is refused rather than guessed at) — a leaked tag in a bubble is the one failure worse than a no-op; a reply emptied by its tags drops, because a pure action needs no bubble. The same counters are the console's refusal-by-reason ledger: toolbox_health() reads PARSE_STATS directly rather than keeping a second copy, because a check that owns its own copy of a rule is a second rule. Four counters joined the ledger from outside the *_bad family, and they measure a different thing — not a malformed tag, but a well-formed one the world declined to honour. reveal_mismatch: a <reveal target="…"/> naming a card the table cannot match leaves the card sealed and says so on the outcome channel the next turn already reads — a named target the host cannot match is not a typo to absorb, it is the host telling you it has lost track of its own hand (⚠ a bare <reveal/> names nothing, so there is nothing to disagree with and it stays the documented door for「open the oldest」). tools_off_dropped: every arming row a persona wrote while the room's tools are off, counted as it is dropped — a dropped arm is never silent (the v555 law), because「no AI arms anything」has to be a guarantee rather than a request, and the number is how you know the door is closing on nothing. split_arm_walled: the same accounting for a cartridge room, where the server deals, counts and eliminates and the persona is asked only for moments — every arming row but the one move that is genuinely the host's (putting the game away: target="table") is walled off and counted. secret_midline: a hidden tag written inside a sentence rather than on its own line. ⚠ Counted, never repaired — the three secret tags are deleted body and all, which is exactly what keeps them secret and what leaves a hole in any sentence built around one（「我出<answer>剪刀</answer>。」reaches the room as「我出。」）; <reveal> could be cured by putting its words back because a reveal makes them public, and that cure is the one these three cannot have. So the cure is the manual's (put it on its own line) and this is only the measurement — a rewriter would have to find a sentence's edges in whatever language the room is speaking, which is guesswork that damages good turns to tidy bad ones
re-serialized into one canonical <speak> form → history (the panel's memory) + the event log (with the verbatim model output + a parse report) + cost
per-persona bubbles · wildcard cards · Mermaid SVG — the browser renders the already-split JSON, so it can never re-introduce a parsing bug
The structured panel_turn tool is the durable fix for in-band tags — the turn arrives as validated JSON, so content can't be mistaken for framing. DeepSeek rejects a forced tool_choice in thinking mode, so the tool is offered "auto" and the model chooses. Pre-deploy sims said Pro takes it almost always (26/26) — so we expected the raw-<speak> parser to matter mainly for Flash. Production overturned that. Measured over the first instrumented turns on the live box:
| Model | Tool path | <speak> text path | Share on text path |
|---|---|---|---|
| DeepSeek V4 Pro | 8 | 31 | ~78% |
| DeepSeek V4 Flash | 1 | 13 | ~93% |
| All turns | 9 | 44 | ~81% |
<speak> prose (the SP documents it as a valid format) and declines the tool outright: every text-path turn carried no tool call, not a tool call that fell back. So the raw-<speak> parser isn't a rare safety net — it is the path ~81% of production turns travel, for both models. That makes P1's tolerant tag parser the load-bearing work, not an edge case. The reassuring corollary: because P1 made that path lossless, the low tool-use rate is fine — we hardened the path the model actually prefers, so there's no need to fight DeepSeek for more tool calls (which thinking-mode won't allow anyway).parse report — which path ran, what was repaired, healed, coerced, dropped, or quarantined. Process totals feed the admin Status → Parse health row. Before this, a parse incident left no trace; now it's a number.<speak> scanner (a who='x' / who=x / fullwidth-quote slip — or a drifted identity keyword, name=/speaker=/persona= — no longer drops the speech); stack-based confer containment; the tool-JSON repair ladder; field-type coercion instead of a crash; and quarantine-not-silence — a turn the gate would empty becomes a visible "Unattributed reply" note, kept out of the model's memory.mermaid fence lifts to its own card (not just the first); ~~~ and {mermaid} fences are accepted; ambiguous heads (pie/journey/timeline) need real structure before they count, so prose isn't retyped as a diagram; an unterminated fence stops at the first line of prose.resolve_who is tiered (slug → exact name → unique word → unique substring, with a minimum-signal rule). A name that fits two seated hosts — two Steves, two 李 — is marked ambiguous and quarantined, never bound to whoever sits first. And a host quoting <speak> inside a code fence no longer fragments the reply.@-mention stuck as "@someone". The cause wasn't the parser — the server stored and resolved it correctly — it was a render race: a room paints its turns synchronously, but the members list loads async, so a mention painted before that list arrived fell to the muted "someone" placeholder and was never re-touched. The fix is the harness invariant applied to rendering: degrade, then RECOVER — when the members land, a recovery pass re-resolves every mention span in place, so a transient miss heals the moment the data arrives instead of sticking forever. Diagnosed by reading the live room's stored events, not by guessing.@Name becomes the entity it meant. A chip only ever renders from a marker (@{u:5}) — the client leaves a bare @foo alone, on purpose — so a panel line that wrote the literal「@mary」highlighted nobody. And it used to: counted over this repo's own dev rooms, panel lines went from an 89% marker share to 0% in two days with no change to the clause that teaches the grammar. What changed was everything around it — the kit brief, the room's record, the pad, the board, the staging note — until the roster sat at character 0 of a 3.5k block, five louder blocks from the generation point. That is this codebase's positional-dilution law landing a third time, and the answer is not a fourth prompt edit: a soft「you may」will always be probabilistic and a chip is binary, so the model is left to write what reads naturally and bind_bare_mentions() does the binding deterministically, once, at emit, storing the id. ⚠ It does not break the entity law, which forbids re-matching a name at render time (that breaks the moment somebody is renamed) — this binds against the live roster exactly as the composer does when a human picks from the popup, and every later render still resolves an id. It refuses rather than guesses, in four places, because a wrong chip is worse than a word: only a current member of this room; never a name two people share, and never one a seated persona also answers to; never after an ASCII word character (dan@mary.com is an address — ASCII-only deliberately, since Chinese runs straight into the @ with no space); and never inside a fence or over an existing marker. Longest name first, so a room holding both 小明 and 小明明 binds the person who was actually named. It runs after lint_mentions on purpose — that lint's ok:bad ratio is the evidence that the model writes well-formed markers, and folding our own bindings into it would flatter it — and mention_bound joins the parse report.Each failure class was pinned by a probe, then re-measured against the shipped code and a fresh 30-call live batch through the production pipeline.
| Failure class | Before | After | Evidence |
|---|---|---|---|
| Bad tool-JSON (escape · inner quote · truncation) | whole turn lost | recovered in place | ladder · fuzz 1833/0 |
who= quote variant / absent | speech dropped | kept | probes S1–S7 |
Unclosed <confer> | leaked on stage | held in drawer (promoted if it swallowed the turn) | probes C1, C3 |
| Stage speech before the confer | discarded | kept | probe C2 |
| Malformed tool-arg shape | parser crash | coerced | probes T1–T5 |
| Second diagram fence | stranded in bubble | own card | probe M1 |
| "pie / timeline …" prose | retyped as diagram | stays speech | probes M5, M6 |
| Shared first name | bound by seat order | quarantined | probes R1–R3 |
| All-ghost turn | silent blank | note card | probe G1 |
The deepest lesson came from the benchmark: a fresh batch surfaced unescaped inner quotes (…"绝对不要借钱"。) as the most common tool-JSON corruption — a class the ladder didn't yet handle. It was closed and re-shipped before any user hit it. That is exactly the loop P0 exists to enable: measure live, find the next class, close it.
Those before/after numbers were synthetic sims — the honest pre-deploy check. The page now reports the real thing: the durable per-turn parse records reconstructed from the live box (the in-RAM admin counter resets on every restart, so the truth lives on disk). Over the first instrumented production turns:
| Signal | Production | Reading |
|---|---|---|
| parser crashes | 0 | no turn lost to a crash |
json_repair (ladder rescued) | 3 | all the inner-quotes rung, firing on real Chinese rooms — turns that would have been blank pre-P0 |
ghost_lines dropped | 5 | cast gate catching contamination, as designed |
salvage (tag-less → lead host) | 1 (~2%) | rare — so P4 stays low priority |
| quarantined turns | 0 | no turn went unattributed |
quotes fix is confirmed working on the exact traffic that motivated it.replies array as plain content — neither a tool-call nor <speak> prose — so the tag-less fallback salvaged the entire blob into the lead host's one bubble, collapsing every voice. The fix is the same invariant: a strictly-gated decode (path=json-content) reads that content like the tool, restoring each host to its own bubble; a JSON deliverable a guest actually asked for is left untouched. Pinned by probes G4/G5 and the JSONCONTENT selftest — surfaced by replaying the one real production turn that broke, exactly the loop P0 exists to enable: measure live, find the next class, close it.<panel_turn> tag with the arguments inside — and the whole costume landed in one bubble as the lead host’s literal speech. Live probing found two more dresses within the hour (the function-call form panel_turn({…}), and nested wrappers <panel_turn><replies>…), plus Mistral’s [TOOL_CALLS] envelope — which starts with [ and so was skipped by the old starts-with-JSON pre-check, the reason the peel now runs unconditionally (no peel pattern can match bare JSON). ⚠ The tag rule is deliberately generic — one XML-ish layer per pass, whatever the model invents — and that is safe only because the strict who-keyed gate still decides: a real <speak> turn peels to prose, fails the JSON gate, and the caller falls back to the speak parser with the original raw untouched, pinned by a real-<speak>-never-hijacked probe. Six selftest costumes and two guards; the offending room’s six stored blobs were repaired in place (events · transcript · history · md, backup kept) so the model stops re-reading its own costume, with CACHE_SCHEMA bumped so a client’s saved copy of the old text cannot outlive the repair.# Replies heading, a **confer:** None… line, the arguments in a ```json fence, a rule, then a closing italic (2026-08-25). _peel_tool_dressing strips layers from the outside in, so it stopped dead at the first prose line and the whole report landed as the lead host’s speech. The answer is a second, narrower reader beside the peel: _dig_tool_json returns the blobs buried in prose, best first — every fenced block whose body opens as {/[, then any brace region that announces "replies" within its first 60 characters, cut out string-aware by _balanced_slice so a brace inside a quoted value never closes the region (and an unclosed one returns nothing, leaving the truncation ladder its job). ⚠ Digging is safe only because nothing here decides: the peel’s core and every dug candidate go through the same _args_if_panel_shaped gate, and who handles are matched against this room’s cast downstream — so a JSON table a guest actually asked for is not a panel turn and never becomes one. The MIMICRY block now runs nine rooms: five decode paths (heading + tag · call form · [TOOL_CALLS] envelope · the report dig · nested wrappers) and four guards (plain prose · a real <speak> · a sentence merely ending in a parenthesis · a fenced deliverable).panel_turn schema, it answered with the schema, two kickoffs in three (2026-08-26). No decode reaches that, because there is no turn buried inside it to dig for — the model is not attempting the call, it is reciting the manual. So the answer sits upstream of the harness: model_tools_ok(model) lets a MODELS row carry "tools": False, and the panel call then never offers the schema to that seat; its <speak> turns, flawless throughout, become its only path. ⚠ The rule this records: a costume earns a decode when the model is trying to make the call and getting the shape wrong. When it cannot make the call at all, take the manual away — the fallback path is already the one ~81% of turns travel, so a seat that lives on it exclusively costs the harness nothing.The harness is exercised by a reusable probe suite (run after any change to it). It lives outside the deployed tree — runtime scratch, never committed — so it's recorded here and in project memory rather than discoverable from the repo alone.
| Probe | What it proves |
|---|---|
probe_parser.py | 33 adversarial cases (speak tags · confer · mermaid · tool-args · resolve_who · room-level). Exits non-zero on any regression — the tolerance spec. |
probe_live.py | ~30 real DeepSeek calls with the exact production request assembly; captures every raw response for offline replay. |
probe_analyze.py | runs the captures through the production pipeline; per-scenario metrics (tool rate, repairs, ghosts, leftover fences, who-form). |
probe_replay.py | differential old-vs-new outcome counts — proves a change recovers turns without regressing the rest. |
probe_fuzz.py | ~1833-trial fuzz of the JSON repair ladder (random truncations / injected escapes / garbage) — asserts zero crashes. |
probe_e2e.py | live Room.say() end-to-end on Pro and Flash; checks the parse report + raw capture land on the event. |
mermaid_check.html | renders every captured diagram in the vendored Mermaid bundle — ground truth, not a guess (28/28). |
parse_health_pull.py | the production diagnosis. Read-only pull of the durable parse records off the live box, a verbatim corpus saved for keeps, and --replay through the current parser — real traffic, not sims. |
lib/smoketest.py on every commit — the same「a probe must fail on a known failure before its 0 counts」rule this page's fuzz and probe suites were built on.Plus the always-on guards: run_room.py --selftest gained named blocks (TOLERANT · CONFER · REPAIR · COERCE · QUARANTINE · DIAGRAM · RESOLVE · SHIELD · JSONCONTENT · MIMICRY), and lib/smoketest.py stays green.
The admin Status → Parse health row is a live, at-a-glance counter — but it lives in RAM and zeroes on every restart (the nightly 05:00 timer that guards memory creep, and every deploy). It answers "is anything wrong right now," not "how has it trended."
The longitudinal record is durable on disk: every turn's parse report plus the model's verbatim output is written to each room's event log under /var/lib/mad/, outside the repo, untouched by restarts. That's the real benchmark — and it doubles as a growing corpus of real model output to replay through any future parser change. One read-only command reconstructs the cumulative and saves the corpus; the diagnosis is repeatable, on real traffic, whenever there's more data.
Retention note: a delete-forever room takes its captures with it (the box's 7-day disk snapshot is the backstop), and a future §9 history-compaction must preserve the parse fields before trimming.
P0–P3 plus the benchmark fix are deployed to production (commits 46a873e → d4e691b, live since 2026-06-13). The client @-mention recovery pass (relabelMentions()) is live too. No prompt wording changed, so no cached prefix was invalidated. Two phases are deliberately held back:
<speak> fallback path — now known to be the majority path — history stores the healed raw rather than the re-serialized parse, so the model's memory can differ slightly from what the guest saw. Narrowed by P1/P3, not closed; production shows the divergence signals (salvage ~2%, quarantine 0) staying low, so it can wait. Touches durable history → its own deploy-and-observe cycle.smoketest.py so the tolerance spec gates every commit. Pure test infrastructure; no runtime surface.Known residuals (by design): a salvaged tag-less turn still assigns the lead host (flagged path=salvage in telemetry, not silent); a bare non-mermaid fence in speech is cosmetic (0 in production).
source records: lib/run_room.py (the pipeline) · lib/room-ui.html (the client recovery pass) · rooms-dev/_probe/ (the battery + parse_health_pull.py, gitignored) · project memory harness-hardening