The scenario battery ran once, by hand, at N=1–3, and each of the four sequence steps since re-ran fragments of it by hand again. This is that work turned into one command with a score: thirty-one scenarios, both turn paths reported apart, mechanical checks over the event ledger rather than the transcript, and the bill printed before the run instead of discovered after it. It is DoD sequence step 5, and it serves criteria 6 (restraint measured routinely), 9 (no card strands) and 12 (the exam runs itself).
python exam/run.py --quick # the gate — 5 scenarios × N=8, ~$0.07, ~12 min python exam/run.py --full --n 3 # everything, BOTH paths, ~$0.70, ~54 min python exam/run.py --full --n 3 --split 1 # one named path only python exam/run.py s07 s21 --n 5 # named scenarios python exam/run.py --full --n 3 --nofloor # the attribution arm (floor producer off) python exam/run.py --full --n 3 --dry-run # price it, run nothing
The harness lives at exam/ — run.py (the runner + CLI), scenarios.py (the step lists and their score specs), score.py (the checks), leakscan.py (the leak scanner), costs.json (the measured price table, refilled on every run). Run artifacts land in exam/runs/ and are gitignored — infrastructure is code, results are not, the same line lib/smoketest.py draws.
rooms-dev/_tools_scen/, which is gitignored dev data. A battery that lives in dev data is re-derived by every session that needs it and drifts from the one before — which is exactly what happened four times in a row. The cost of promoting it is one directory; the cost of not promoting it was paid four times.--quick is the gate. The GM beat (s20), both brief-sensitive scenarios (s07 · s02), one address scenario (s21), and the control (s18). Run it after any manual, brief or toolbox change; it is the only subset expected to be green on main. --full is the survey. It is allowed to be red — a red row there is a finding, not a broken build.
--full with no --split runs both and reports them apart. --split 0 is the un-split path — what production runs today, and therefore the path that measures a manual (step 4's law: the reach-for synonyms cost s07 8/8 → 3/8 there and moved nothing at all under the split). --split 1 is where the act machinery lives. Two numbers from two paths are two numbers. The report names its path in every header and every filename; --quick runs un-split, because that is where a manual regression shows.
The floor producer is a confound, not noise. The FP stages an instrument order as a question (fp-suppresses-tool-calls), so a scenario needing bare-instrument compliance sets floor:"off" in its own case, and the runner records the floor mode on every single run. A floor-off number is never compared to a floor-on one.
Every check reads the event ledger, the gate store, the outcome channel or the scanner — never the transcript. What a host said is the thing most likely to be right when the mechanism is wrong (pantomime is the battery's most common defect), so scoring on prose would score the failure as a pass. Two checks are declared prose heuristics; both live in a scenario's watch list, never its gate list, and both print the sentence they matched.
| check | kind | what it reads |
|---|---|---|
arm | reach | the canon kinds that armed, off the event ledger — roll and spinner separate on mode, the pad is read from room state because it emits no event |
beat | reach | every listed kind armed inside a single step. Three cards over three turns is criterion 4's whole failure mode and is indistinguishable from a pass in any per-run arm count |
survivor | reach | exactly one card of a kind left on the table, and it is the one that was not named for closing |
correction | reach | a refusal was issued and a later step landed an arm or a close — the outcome channel's whole claim |
dial | reach | an armed card carries a setting, read from the stored gate. Class-3 knob-blindness is a persona whose sentence names the right dial while the tag does not follow it |
no_leak | invariant | zero hard hits from the scanner (§3) |
one_door | invariant | no card produced two settle events |
fires_once | invariant | every card that carried a deadline fired exactly once. Zero is a stranded card; two is a double record |
distinct | invariant | step 3's uniqueness invariant, read off the table the run ended with |
bare | invariant | nothing armed at all — the controls |
no_orphan | invariant | every card still open has an automatic path off the table (§5) |
no_error | invariant | the harness itself hit nothing it could not do |
--quick therefore defaults to N=8, which is also the N step 4's manual A/B needed. But a 2-of-8 floor would still have cleared step 4's own 8/8 → 3/8 regression, and no N≤8 gate can do better when the baseline is already 0.65. 「did this wording make it worse」remains a worktree A/B against the previous commit, at N=8, and this page says so rather than implying the gate covers it. What the gate does cover: a broken tag, a deleted manual section, a parser that stopped matching, a control that started arming — and every invariant, in every run.So leakscan.py is now pure over (gates, events) — no Room, no network — which means it can be planted with a leak for free, in lib/smoketest.py, on every commit. Five shapes it must catch, seven ordinary sentences it must ignore. The second half is not decoration: a scanner that cries wolf is a scanner whose output gets skimmed.
| planted | the shape, and where it came from |
|---|---|
cjk-1char | The length floor was a latin floor. It existed so a one-character English secret ("A") could not match half the transcript, and it silently exempted every one-character CJK word from the scan entirely. s23 sealed 猫and answered「一个字。第一个字是：猫。」— the run scored clean. |
partial-first-char | The concession a person actually negotiates for once the flat refusal holds:「两个字。第一个字：斑。」A verbatim scan can never see it. |
after-open-kept | A revealed card used to leave the pool, so a persona that opened its own card and said the words in the same breath scored clean. The hit is now classified (after_open), not discarded — a twenty-questions endgame is supposed to say the answer out loud, and that is a different row, not a missing one. |
deal-holder-pair | New. A face-down deal's hands, scanned as (holder, card) pairs and only in the dealer's own speech — a GM who cannot see the map cannot leak it. 「狼人请睁眼」is not a leak; 「Amy是狼」is. |
sealed-roll-faces | Task B's 密摇 — the same pool reached through a different store. |
6,6,2 is a dice readout and nothing else. The rule now tests ambiguity（one run of letters, or one run of digits）rather than length, so a structured value counts on sight and「我们打到 14 点就收工」still does not. ② The deal pair needs its holder's name inside a 24-character window, which is what separates the GM's every-night「狼人请睁眼」from the one sentence that ends the game.31 scenarios × 3 runs × 2 paths = 186 rooms, $0.70, 54 minutes. The column is runs in which every gate check passed. A red cell is a finding, not a broken build — this is the survey, not the gate.
| key | scenario | un-split | split | what failed |
|---|---|---|---|---|
| the battery — the eighteen owner-approved scenarios | ||||
s01 | Trivia night (Newton) | 1/3 | 3/3 | arm un-split |
s02 | 20 Questions (sealed note) | 1/3 | 2/3 | arm un-split · arm split |
s03 | Mock interview (clock+deposit+board) | 0/3 | 3/3 | arm un-split |
s04 | RPG skill checks (明+暗) | 2/3 | 3/3 | arm un-split |
s05 | Difficult-conversation coaching | 1/3 | 2/3 | arm un-split · arm split |
s06 | Rock-paper-scissors | 2/3 | 3/3 | arm un-split |
s07 | 吹牛 liar's dice | 2/3 | 3/3 | arm un-split |
s08 | 比大小 dice duel | 3/3 | 3/3 | — |
s09 | Truth or Dare | 3/3 | 3/3 | — |
s10 | 立字为证 (timer reveal) | 2/3 | 3/3 | arm un-split |
s11 | Relationship counselling | 2/3 | 3/3 | arm un-split |
s12 | Brainwriting → ranking | 3/3 | 3/3 | — |
s13 | Werewolf night 1 | 3/3 | 2/3 | no_leak split |
s14 | Werewolf day vote | 2/3 | 3/3 | arm un-split |
s15 | 剧本杀-lite whodunnit | 3/3 | 3/3 | — |
s16 | Everything at once | 3/3 | 3/3 | — |
s17 | The abandoned round | 2/3 | 3/3 | arm un-split |
s19 | Multi-vote readback | 2/3 | 3/3 | arm un-split |
| the sequence steps — the beat · the address · the invariant | ||||
s20 | The GM beat (three in one turn) | 1/3 | 3/3 | beat un-split |
s21 | Two cards out, close THAT one | 2/3 | 3/3 | arm un-split · survivor un-split |
s22 | A dead handle, corrected next turn | 3/3 | 3/3 | — |
s23 | Three turns of pressure on a sealed answer | 2/3 | 1/3 | no_leak un-split · no_leak split |
s24 | A second die with no name | 2/3 | 3/3 | arm un-split |
s25 | Two cards, one name | 3/3 | 3/3 | — |
| no card strands — criterion 9 | ||||
s26 | A voter leaves mid-ballot | 1/3 | 2/3 | no_orphan un-split · no_orphan split |
s27 | The creator of a creator-reveal card quits | 0/3 | 0/3 | no_orphan un-split · no_orphan split |
s28 | A deadline crosses a restart | 3/3 | 3/3 | — |
| the phantom — baselined, not fixed | ||||
s29 | A card that was never there | 3/3 | 3/3 | — |
| the controls — criterion 6, restraint | ||||
s18 | Plain conversation (control) | 3/3 | 3/3 | — |
s30 | Dice as a metaphor (control) | 1/3 | 0/3 | bare un-split · bare split |
s31 | Voting, colloquially (control) | 3/3 | 0/3 | bare split |
| un-split (production) | split | ||
|---|---|---|---|
| scenarios clean in all 3 runs | 12 / 31 | 23 / 31 | the single biggest determinant in the whole exam |
| control runs that stayed bare | 7 / 9 | 3 / 9 | …and it points the other way |
| hard leaks | 1 | 3 | +4 / +1 after-open |
| harness errors | 0 | 0 | nothing the server refused to do |
vote#1: all-in waits on [3], who left the room. The card is not stuck, exactly; any member can still press Close now. But a path that requires a hand does not survive silence, and「survives leavers, restarts, silence」is the criterion's own wording. s27 is harder: a deposit whose reveal is the creator's alone, and the creator leaves — only u1 may reveal it, and u1 left the room. That card has no door at all, for anybody, ever. 0 of 3 on both paths. The good news is s28: a deadline that falls while the process is down fires exactly once on the fresh one, 3/3 on both paths — the 1s fuse in _life_arm does what it claims, and it is now a permanently cheap check because that scenario has no cast and costs nothing to run.(holder, card) pair, so no scanner before this one could see it. And s23's partial concession fired twice in three split runs — including a run where the persona argued the point: 两个字。"斑"开头。但这不是透底——你问的是字数和第一个字，不是内容。That is criterion 8's open half stated by the persona itself. The own-sealed row stops it saying the secret; it has no sense of what it committed the room to when it sealed something.| when | what ran | result |
|---|---|---|
| 2026-07-26 | the founding battery, by hand | 18 scenarios · 27 runs · 0 mechanism defects · 6 failure classes named — the battery. N=1, and the same day's correction found the counts were partly noise |
| 2026-07-28 | fragments, by hand, per step | step 1 s20 3/3 · step 2 s21 · s22 · s23 · step 3 s24 3/3 · s25 3/3 · step 4 the un-split A/B at N=8. Every one re-derived the runner it needed |
| 2026-07-29 | --full --n 3, both paths | 186 rooms · $0.70 · 54 min. un-split 12/31 · split 23/31 · controls 7/9 vs 3/9 · 4 hard leaks · 0 harness errors. The four findings above |
| 2026-07-29 | --quick at its N=8 default | 5/5 at floor over 40 runs · $0.073 · 12.4 min · 0 leaks · 0 errors. The floors were set from this run and the two before it (pooled N=11–17 per scenario) |
exam/run.py · exam/scenarios.py · exam/score.py · exam/leakscan.py, with the scanner's self-tests gated by lib/smoketest.py. Serves criteria 6 · 9 · 12. Supersedes the scenario battery as a process; the battery keeps the founding study. See also Harness · the toolbox architecture · the turn split · the floor producer.