Conversation quality — a production diagnosis

The empirical half of roadmap §1 (the floor producer). §1 asserted that the megaprompt "averages the room." This is the measurement: all 80 production rooms pulled from the live box on 2026-06-23, the 39 real multi-host conversations scored on a rubric built from actual user feedback — what users said is good (voice, fidelity — not re-scored here) and what they said is bad. Every finding is triangulated three ways: a mechanical pass (pure arithmetic), an LLM judge reading every transcript, and the users' own in-room complaints. The diagnosis points directly at what the floor producer must fix.

How it was measured

The rubric was cut down to only the four things users complained about. Users said voice distinctness and character fidelity are already good, so those are not scored here — every judge token goes at a real pain. The instrument never sees the live model; it reads the recorded event logs. Human turns are anonymized (Human-A/B) before anything reads them; raw transcripts stay off the repo.

The rubric — four pains, in users' words

Finding 1 — length is the pervasive tax

The single most consistent defect across all 39 rooms. The median host reply is ~314 words (p90 482); per-room medians run from 83 up to 714 words. A comfortable chat bubble is one to three sentences — call it ~40 words. Hosts routinely answer at 7–18× that. And the response gate's "concur in one short line" tier is essentially dead: the short-line rate is ~1%. Nobody nods; everyone gives a speech.

The judge confirms it isn't just long but padded: 36% of all host lines are rated ≥40% cuttable with no loss of substance (preamble, restatement, hedging). 14 of 39 rooms are above 40% padded. And it is not a big-cast artifact — two of the worst-padded rooms are 2-host coaching rooms (padded 0.57 and 0.51), where a small panel simply monologues. Users felt it and said so in the room:

Finding 2 — redundancy by convergence

When more than one host speaks in a turn, 40% of the later contributions add no new angle — they paraphrase or re-affirm what was already said. The signature move the judge saw again and again: "X said it well — let me add…" followed by the same point in new words. 23 of 39 rooms sit at or above 40% redundancy.

It scales with cast size and "everyone speaks" turns, but it is not only a chorus effect: high-all-speak rooms average 0.42 redundancy vs 0.37 for the rest — a real but modest gap. The other half is cross-turn self-redundancy: a single host re-running their own signature point every turn (one host re-cited the same "40% survey" rule five times; another re-derived the same pricing formula turn after turn). The good news is that it is clearly solvable — the best rooms scored 0.00–0.19 because each host held a genuinely distinct lens:

Finding 3 — the conflict dial barely works the sharpest finding

This is the user's most specific complaint, and the data is blunt. The extreme temperament promises a sharp, sustained clash (level 3). Across 17 production rooms it delivered a perceived conflict of 1.0 — "polite divergence." 5 of those 17 rooms scored a flat 0 — total mutual agreement, at the maximum setting. The natural dial, by contrast, is well-calibrated (gap 0.33).

An honest correction: an earlier pass on a handful of dev rooms suggested the dials made rooms more choral. The full production sample (17 extreme rooms vs 3 in dev) overturns that — on real data the dial story is under-delivered conflict, not changed participation. Pulling everything was the right call. (probing and combative each have only n=2 in production — too few to bucket; the natural-vs-extreme contrast carries the finding.)

Finding 4 — a few signature gestures, worn to death

Pain 4 is real but local. Most rooms are clean; gesture-repetition concentrates in a handful of personas — and where it appears, it is egregious. The worst single room is a textbook case the user named almost exactly:

This is different in kind from the first three. It is not a staging problem the room can fix per-turn — it is a persona-authoring problem: a signature gesture licensed in profile.md gets over-deployed at runtime. The fix lives in the persona kit (single-source / cap the gesture, the way the authoring kit already single-sources a signature line), not in the floor producer.

The numbers, by dial

* probing / combative n=2 each — shown for completeness, too few to read into. Overall (39 rooms): redundancy 0.40 · padded 0.36 · perceived conflict 0.80/3 · median reply 314 w · p90 482 w · short-line rate ~0.01.

What this means for the floor producer

Three of the four pains are the same root problem from different sides: one mind voices the whole panel, and that mind always has something for everyone to say, at length, in agreement. It cannot self-correct, because the writer and the staging-decider are the same context. That is precisely the case §1 makes for an exogenous per-turn floor producer — and this diagnosis tells it exactly which levers to pull.

• Silence is the headline lever. 40% of non-lead bubbles should not exist. The producer's most valuable word is "sit this one out" — which §1 already predicted, and the redundancy number now sizes.
• A length budget per turn. The writer defaults to ~300-word speeches and never uses the one-line concur. An exogenous "one line each" / "let Plato run, the rest hold" is the only thing that has reliably moved length in testing — the static instruction drifts back.
• Contention must be assignable, not just dialed. The new finding: extreme fails because the writer routes heat at the human, not between hosts. The producer needs to name who disagrees with whom this turn — staging the split, never scripting the position (the dumb-pipe contract holds).
• Gesture repetition is out of scope for the producer — it goes back to the persona kit as a single-source cap.

What to trust, and what not to

• One judge per room, no human calibration. By directive we skipped a calibration batch. Mitigation: the judge was bias-hardened against length/agreeableness, and its three headline numbers agree with the mechanical metrics (length, all-speak, gesture counts are pure arithmetic) and with users' own in-room complaints. Where all three point the same way — length, redundancy, the dial — confidence is high. The judge's finer per-line verdicts are one model's read.
• Observational production data. Rooms differ in topic, cast, and persona choice. The dial finding is the most robust because it is judged within setting (each extreme room against extreme's own promise), not across confounded groups.
• The sample is what users actually ran — heavy on extreme (17 rooms) and Chinese-language sessions, light on probing/combative. That is a feature: it reflects real usage, not a lab.
• Voice and fidelity were not scored — users said they're good, so this says nothing about them. It is a staging diagnosis only.