Owner (09-04):「Design a self-improving loop. Aim at the quality of the major media as the target; tweak the brew engine — the process, the prompts — until what we produce is close to the targets. English first. To quantify quality we need an objective rubric and judge. The iterations are a mini gradient descent, except the weights are the process and the prompts. Estimate the cost of training per iteration. No art, no translation. Do we need a human in the loop for alignment?」This page is the plan. The short answers: the targets are named in §1; the loss is a pairwise win rate against the majors, judged blind by two model families and anchored by code (§2); the「weights」are a versioned brief and a handful of process switches (§3); one step costs about $3–7 and half an hour (§4); and yes, a human stays in the loop — to set the target and to audit the judge, never to grade the pieces (§5).
Gradient descent needs a differentiable loss over continuous weights. Ours are a brief in English and a handful of switches (how many exemplars, which model, whether the writer outlines first), and the loss is a judge's verdict. So the honest name is coordinate ascent over prompt variants with a paired evaluation — the method the prompt-optimisation literature converged on (OPRO, DSPy's MIPRO, TextGrad's「textual gradients」): propose one edit, measure it on a fixed set against the incumbent, keep it if it wins, and let a model read the failures to propose the next edit. The parts that make it work are the same parts that make gradient descent work — a fixed loss, a fixed evaluation set, one change at a time, a step only when the signal beats the noise — which is why the owner's framing is the right one even though the arithmetic differs. The rest of this page is those four parts, in order.
What「the quality of the major media」means has to be a corpus, not a feeling: a frozen set of professional pieces the judge reads beside ours. It is chosen by format class, because a fable is not judged against a leader. English only for now. The pieces are stored on the shelf (<ink_dir>/standard/, outside the repo, never republished — calibration material, the same status as a persona's corpus), with the outlet, the date, the format class and the word count.
| Format class | Outlets (the standard) | Access | Pieces |
|---|---|---|---|
| Essay · Notes (the argued piece) | NYT Opinion guest essays · the Economist's leaders and named columns (Bagehot, Lexington, Charlemagne, Bartleby) · the Guardian's comment page · the Atlantic's Ideas · the FT's columnists · Aeon (the long form) · Project Syndicate | NYT, Economist, FT paywalled — one week's subscription collects a year's worth; the rest free | 40 |
| Notes (numbered, the ledger) | Matt Levine's Money Stuff (free by mail) · Axios's items · the Economist's Espresso · Kottke | free | 15 |
| Review · Scorecard | NYT Book Review · the New Yorker's critics · the Verge · Tetlock-style public post-mortems (Zvi, Scott Alexander's yearly predictions) | metered / free | 15 |
| Letter · Advice (voiced) | Buffett's shareholder letters (free) · Dear Sugar (the Rumpus archive) · Slate's Dear Prudence · Ask Polly · Rilke, Seneca — the public-domain letter of counsel | free / metered | 15 |
| Dispatch | the New Yorker's Letter from… · NYT dispatches · Dan Wang's annual letters · Granta's reportage | metered / free | 10 |
| Light (Shouts · Overheard · Kicker · Fable) | the New Yorker's Shouts & Murmurs · McSweeney's Internet Tendency · the Onion's op-eds · Thurber's fables (public domain) · Axios's「1 fun thing」 | free | 25 |
About 120 pieces, collected once and refreshed a quarter at a time (a fifth of the set swapped, so the standard moves with the field but the loss stays comparable across a run). The subscription question: yes, one week of NYT and the Economist is worth it — they are the two titles every persona reader would name, and the collection is a one-time harvest (the calibration shelf already fetches the free outlets; §5). The FT can wait. What the first cut of the calibration already said about these pieces: E 0.65–0.95, S 0.3–0.9, 1,400–4,300 words, and a lint that had to be length-scaled to let them through (the quality score §5).
The Pulitzer opinion criteria the quality page already adapted, restated for a pair and with two lines the field's judges add: clarity of style · sound reasoning · originality (a point the reader did not have) · specificity (reported, not asserted) · structure (one claim carried; the ending earns) · voice (a person wrote it — no machine tells) · carry (one thing to repeat tomorrow). The judge answers each line with A / B / even and gives one overall verdict; the overall is the loss, the lines are the diagnosis the optimizer reads. Rules the literature insists on and the harness enforces: bylines, outlets and dates stripped; both orders (A-B and B-A — a verdict that flips with position is「even」); length-matched pairs (±30%) so length cannot be the tell; never the writer's family as judge (the writer is DeepSeek; the judges are Gemini 3.7 Flash and, as the second family, Claude Sonnet — used sparingly, as a judge only); the two judges' disagreement is reported, not averaged away.
The standard corpus gives reference distributions for what code can measure without a model call: words per piece · sentence-length variance · paragraph-length distribution · the opening sentence's length · dashes per thousand words · the AI-vocabulary rate · the antithesis rate · specifics per hundred words · type-token ratio · quotation rate. A piece's distance is its mean absolute z-score against the standard of its class. It is cheap, deterministic, and its direction never lies — which makes it the guard against the loop learning to please the judge with something the majors do not do (§5).
Before the loop trusts the judge, the judge passes an exam: a professional piece against a deliberately damaged copy of itself (padded by a fifth, the antithesis turn inserted twice, a paragraph of consultant vocabulary) must be preferred ≥ 90% of the time, in both orders; and on the owner's blind pairs (§5) the judge must agree with the owner at least as often as two humans agree with each other (~75–80%). A judge that fails either is swapped, and that is recorded as an instrument change, so no run's numbers are read across it.
The brew is a pipeline of prompts and switches. The loop changes one at a time, and every change is a versioned artefact (lib/prompts/ink-brief.vN.md, the same stamped-variant discipline as the v1|v2 manner): every piece records the brief version that wrote it, so an edition can always be read back to its weights.
| Weight | Today | The range the loop may search | Why it might move the loss |
|---|---|---|---|
| The writer's brief — the house rules, the commission's wording | BOUNDARY + the commission block | free text, one edit a step (a rule added, reworded, removed) | the main weight; where the「textual gradient」lands |
| The process shape | one call: profile + brief → piece | outline-then-write · draft → self-critique against the rubric → revise · two drafts, the writer picks · the reader's-objection pass | the largest single lever in every published result on long-form generation; costs 2–3× the writer call |
| Exemplars (the persona's own paragraphs) | 3, nearest the subject | 0 · 3 · 6 · 10; nearest vs most characteristic | voice (V) — but too many crowd the argument |
| The notebook | the peg + 3 searches, 2 pages | depth 1–4 pages; a second research call on the persona × subject | specificity (S), the axis the majors beat us on most |
| Length floors | the field's (§3i of the brew) | ±20%; the extension pass on/off; floor as hard rule vs guidance | the field's pieces are longer; the writer undershoots |
| The editor | 3 rival-model passes on a lint fail | 0–3 passes; on every piece vs on failures; the voice pass with more exemplars | removes tells; can flatten voice (the calibration's「sterile」finding) |
| Model · thinking · temperature | V4 Pro, thinking on, 0.9 | Pro / Flash / Sonnet as writer; thinking off; 0.6–1.0 | the expensive weight — a model change is a new baseline, not a step |
| The slate (who writes what) | the editor's budget meeting | persona × beat affinities, the format mix | out of scope for the loop: it changes the commissions, so the dev set could not stay fixed |
| Cost of one iteration | Flash judges only | + Sonnet as the second family | Basis |
|---|---|---|---|
| Writing 48 pieces (V4 Pro, thinking; the extension pass where it fires) | $1.00 | $1.00 | ~2¢ a piece measured on the 09-07 brew |
| The editor on the pieces that fail the lint (~40%) | $0.20 | $0.20 | ~1¢ a failing piece |
| Paired A–B judging, 96 calls | $0.40 | $2.30 | Flash ~0.4¢ a call on two 1,200-word texts; Sonnet ~4¢ |
| Against the majors, 96 calls | $0.40 | $0.40 | Flash |
| The optimizer's proposal (one call over the verdicts) | $0.05 | $0.30 | Flash / Sonnet |
| Per iteration | ≈ $2 | ≈ $4–5 | plus ~$0.60 once for the dev set's notebooks; 25–35 minutes off-peak |
| A run of 20 iterations | ≈ $40 | ≈ $90 | doubling the pairs for finer steps doubles it |
The real cost is not the API bill; it is the reading a human owes the judge (§5): about an hour a month. Off-peak (DeepSeek's evening rates) is the rule for every run, as for the brew.
What the human is guarding against — Goodhart. A loop that optimises a judge learns the judge, not the standard: pieces that grow longer because the judge likes thoroughness, that name-drop numbers because the rubric says specificity, that adopt the judge's own family's cadence. The guards, in the order they fire: the code distance (§2) refuses a step that moves any measured statistic away from the majors even if the judge liked it; the rotating dev set exposes a brief tuned to twenty-four pegs; two judge families cannot both be pleased by one family's cadence; the known-answer test is re-run after every accepted step; the production readers — thumbs, completion (recorded since v1070), the doors opened — are the out-of-sample check no offline judge can fake, read weekly against the loop's claimed progress; and the weekly diff of the brief is read by a human, because a rule that says「mention three figures in the first paragraph」is visible to a person and invisible to a metric.
| Phase | What is built | Done when | Cost |
|---|---|---|---|
| 0 · The standard | the harvest of §1's outlets into <ink_dir>/standard/ — the free ones with the calibration shelf as it is, NYT + the Economist over one subscribed week; each piece frozen with its class and length | ~120 pieces on the shelf, 40 in the essay class; the reference distributions computed | $15–20 subscriptions · $1 fetching |
| 1 · The judge | lib/ink_judge.py: the pairwise rubric, both orders, two families; the known-answer test; the code distance; the first reading — today's brew's win rate against the majors, by class — the baseline number this page will be read against | the judge passes its exam; the baseline is on the ledger page | $5 |
| 2 · The harness | lib/ink_opt.py: the frozen dev set (24 commissions + notebooks), the versioned brief (lib/prompts/ink-brief.vN.md, read by the writer — the brief leaves the code), the variant runner, paired judging, the sign test, the ledger (docs/ink-loop-ledger.html, generated like the brew log) | one iteration runs end to end on a no-op edit and reports 50% ± noise | $3 |
| 3 · Manual steps | the edits humans already suspect, one at a time: outline-then-write · self-critique against the rubric before submitting · exemplars 3 → 6 · the notebook two pages deeper · the floor as a hard rule · the editor's voice pass on every piece | 6–10 iterations; the accepted brief is the brew's; the owner's first audit sample read | $20–50 |
| 4 · The optimizer proposes | the OPRO step: a model reads the losing pieces' verdicts and writes the next edit; the human reads the weekly diff and the monthly audit sample; the loop runs nightly off-peak, one step a night, on the dev set — never on the live edition until a step is accepted | the win rate vs the majors climbs from the baseline (expected 25–35%) toward 45%+ over 20 steps, and the production readers' numbers move the same way | ≈ $2–5 a night |
What is deliberately out of scope: art and renditions (owner); the slate (it changes the commissions, §3); the persona profiles themselves (a persona is product content with its own pipeline — the loop tunes how the house asks, not who the writers are); and the model as a weight (a model change resets the baseline and is a decision, not a step).