Feedback widget — the v1 instrument

How we measure the floor producer when there's no user base to A/B and the real conversations are full of personal information. The answer: collect the signal in-product, from the user, about their own turns — so nothing private ever leaves the room. This is the measurement loop for a 3-tester world, and the first thing to build. A working mock-up — click the thumbs and tags.

In the room

The rating rides inside each reply's bubble: three bare icons — 👍 👎 ⌄ — in one quiet right-aligned row under the text, and nothing below the bubble until it's needed (the fold-out panel is born folded and costs zero height, so the stream keeps its constant 15px rhythm). A clicked thumb becomes a neutral pill — deliberately not green/red, which drown against the bubble's own background — read per theme: a quiet translucent ink-wash in dark mode, a lifted translucent-white keycap (with the idle rail faded further back) in light mode, where an ink wash looked dirty on cream — the same material both ways, a wash over the bubble rather than a solid chip. 👍 is one tap, no expand; 👎 lights and opens the tag panel below the bubble, where the lever-tags sit in fixed columns so the user learns where each one lives. The chevron folds and unfolds. Opening the panel scrolls it fully into view — clear of the frosted top bar and composer, under which the chat scrollport runs — and Add a note focuses the box with the caret at the end, then nudges it above the composer / mobile keyboard once the shell's keyboard resize settles. While the note box has focus the composer hides — never two text areas on screen at once — and returns the moment focus leaves the note. In a bubble that carries a link chip (a DOC / diagram hand-over) the rail sits below the chip: the thumbs are always the bubble's last row. An open panel auto-folds when the next message lands (releasing the note's focus with it) — so even in a busy panel the cards never pile up open. (This rail-in-bubble form is CW's ui/sandbox design, adopted v266 with two changes: one button size across iOS/Android/PC, and the ink-wash pressed state in place of her green/red icon tint.)

Two replies: Buffett's rail idle (three bare icons in the bubble's corner, nothing below), Jensen's panel open — what ⌄ or 👎 reveals: lever-tags in fixed columns, LENGTH · TONE · CONTENT. (Try it — thumbs and chips toggle; the chevron folds.)

The widget, up close

A thumb is the cheap signal; the tags are the actionable one — each maps to a lever the producer controls, so a complaint points straight at what to change. Here it is mid-use:

The two CONTENT tags are independent — a reply can be both off-ask and a rehash. Each opposed pair — too long / too short and too soft / too harsh — is mutually exclusive: tapping one clears the other. The old PANEL group (no-debate / repeated-each-other / wrong-person) was dropped when the card went per-bubble: those are turn-level judgements about how the hosts played off one another, and they don't fit a rating attached to a single reply.

The panel opens on demand. Every reply is born with just its in-bubble rail — 👍 👎 ⌄ — and zero rows below the bubble. A clicked thumb reads as the ink-wash pill; 👎 also unfolds the tags below; the chevron toggles. And the moment the next message lands, any open panel auto-folds — so a panel of three never stacks three open bodies down the screen:

The folded default: two replies, two rails, nothing below either bubble (the first already 👍'd — the ink-wash pill). Tap ⌄ or 👎 to open one — the rest stay out of the way.

The check-in survey

Per-turn taps catch the local read; they miss the cumulative one — "this whole room is wearing me out," the thing the machine judge is blindest to. So every N turns — a cadence the admin sets in console → System → Feedback (0 = off) — the same vocabulary appears in long form, and the user can summon it any time via Take Survey in any turn's card (or the room menu). The counter is server-truth: it counts rendered panel rounds, so in a multi-user room it reaches every present person (a lurker who never types is asked too) and it survives a reload. A qualified submit resets that user's clock — answer the survey and the next one is N rounds out, not at the next fixed multiple. Pushed when we want the read; pullable when they have more to say:

The long form. Same vocabulary as the per-turn tags, asked once about the whole session.

The arms — no FP v0 v1

The widget is arm-agnostic, so it doubles as the coarse comparison you can't get from a real-user A/B. Each room is assigned one arm at creation — blind to the user (no badge, FP-bubble off) — and its feedback accumulates under that arm:

• no FP the raw panel, no producer at all — the honest floor. Tells you whether any staging beats doing nothing.
• v0 the deterministic floor (live today). The bar v1 has to clear.
• v1 the smart producer. The thing we're tuning.

What each tap logs

The point of per-reply is attribution: a tag is stored against the exact reply that drew it, with the staging that produced that turn + the signals the producer saw — so "too long" isn't a vague gripe, it's a labelled example of a specific decision. That same record is the seed of a trained policy later (a 👍/👎 pair is exactly what DPO trains on).

Why it's built this way

• Per reply, not per turn. Each host's reply is rated on its own, so a multi-host turn yields one label per voice instead of one blended rating for the whole batch — a finer-grained signal exactly where the producer's per-seat decisions land. The old objection — a card under every bubble is naggy — is answered by the in-bubble rail (three bare icons in the reply's own corner, zero rows below the bubble) plus panels that auto-fold when the next message arrives: a panel of three doesn't fill the screen with open bodies. The honest trade-off: the turn-level panel-dynamics read (no-debate / wrong-person / hosts-echoing-each-other) is no longer collected here — those judgements have no single reply to attach to.
• Tags are levers. Each chip is a thing the producer can turn — so feedback is directly actionable, not a sentiment score. Lean on the tags; the bare thumb skews positive and sparse.
• The producer only owns staging. Filter content gripes ("that's wrong") out of the tuning signal — keep the staging tags (too-long, too-harsh, echoes-others). The note + thumb stay for the user; the tags drive f.
• PII-safe by construction. The user rates their own turns inside their own room; no transcript is ever handed to a reviewer. That's why this replaces offline replay.
• It's a feature, not just telemetry. Letting people steer the room ("too long," "too soft") gives them control and signals we're listening — it earns its place on screen.
• Pushed + pullable. The check-in auto-fires for the cumulative read, but Take Survey in any turn's card lets a user open it whenever they've got more to say (with a persistent entry in the room menu too). Feedback shouldn't only happen on our schedule.