Prose Quality — benchmark & decision (A4 vs. Sudowrite Muse)

Status: decision doc / investigation (2026-07-02). Workstream A4 of COMPETITIVE_LANDSCAPE_STORY_SOFTWARE.md. Question: Sudowrite's Muse is the acknowledged sentence-level prose leader. Do we need a "best-first-draft" mode, or does our critique-loop strategy close the gap? This doc characterizes the gap, gives a repeatable benchmark methodology, and makes a ranked recommendation.

1. The two bets (why this isn't apples-to-apples)

The honest read: on a single passage's sentence-level craft, a fiction-tuned model has a structural advantage we can't match by swapping prompts. On a whole manuscript's consistency, we win — which is the market's #1 pain (["why your AI novel falls apart after chapter 5"]). So the goal is not to out-Muse Muse on one scene — it's to be close enough on craft that our consistency + pipeline win the writer.

2. What makes Muse lead (2026 research)

• Fiction-only training lets it reach a craft ceiling general models don't (sudowrite.com/muse).
• Pacing — "can sit with tension, create suspense, unfold a mystery over time."
• Cliché avoidance measured + trained out.
• Style Examples + Creativity slider — teach it your voice; control liberty.
• Independent 2026 reviews still call Muse the best sentence-level creative prose, with the caveat that long-context consistency is where novel-focused tools beat it (MyLifeNote 2026).

Evaluation research: the differentiator top evals reward is sentence-level rhythm — varied cadence, deliberate fragments, a long sentence turning on one comma then a two-word beat (stacc 2026 benchmarks). LLM-as-judge on multi-criteria rubrics (EQ-Bench's 22 criteria; a 7-dimension novel rubric) reliably separates polished from careless prose but struggles with deep literary mechanics — good enough to rank our own options, not to certify literary greatness.

3. Where we already compete — and where the gap is real

Already strong (our prompt already encodes a lot of craft, see season-novel-chapter.ts callChapterPromptGPT): show-don't-tell, "sentences breathe / periods over commas," no em-dashes, "trust the reader," no rephrasing, bible voice/tone/do-not-use rules, plus the 13-critic loop (tension, dialogue, repetition, subtext, specifics, cliché, first-reader, continuity, delivery).

Real gaps vs. Muse:

1. No voice-exemplar path. We author voice_rules; Muse ingests the writer's actual prose. Exemplars beat rules for voice-matching.
2. No explicit sentence-rhythm lever. Our critics cover tension/cliché/specifics but nothing targets cadence variation — the exact thing 2026 evals say separates top prose.
3. Draft model may be under-powered. Default draft is Haiku (cost tier); Opus only on promote. Muse's first draft is its strength. We may want a stronger draft model for the writer-facing editor (quality-first) vs. the audio pipeline (cost-first).
4. No objective measurement. There is no prose-quality eval/rubric/judge in the codebase today — every quality call is vibes.

4. The benchmark methodology (ready to run)

Goal: rank our own configurations (and sanity-check vs. a Muse sample) so the model/mode decisions are data-driven. LLM-judge is fit for ranking, not certifying — use it accordingly.

• Scenario set (fixed, ~8–12): golden scene briefs spanning the hard cases — a dialogue-heavy confrontation, a quiet interior beat, an action climax, an exposition-heavy setup, an emotional turn. Reuse real bibles (e.g. The Turing Logs) so voice grounding is realistic. Commit them as fixtures.
• Configurations under test (the model/strategy matrix):
  • haiku-1shot (current default draft)
  • opus-1shot (current promote quality, no critique)
  • haiku + critique loop (1 pass) (our process bet)
  • sonnet-1shot (mid tier)
  • optional: a pasted Muse sample for the same brief (manual, from a Sudowrite trial) as the external anchor.
• Rubric (0–10 each; the judge scores blind): sentence-rhythm/cadence · voice consistency · sensory specificity (show-don't-tell) · dialogue authenticity · pacing/tension · cliché-freeness · overall readability. (Mirrors EQ-Bench / the 7-dim novel rubric.)
• Judge: a strong model (Opus/Sonnet) as LLM-judge, blind (config identity stripped, order shuffled), scoring the rubric with a 1–2 sentence justification per dimension. Run each sample past ≥3 judge invocations (vary the judge prompt seed via ordering) and average — mirrors the "3 blind reviewers, averaged" protocol.
• Output: a scorecard per config across scenarios + the aggregate. The decision bar: does haiku + 1 critique pass reach within ~1.0 of opus-1shot? Does any config get within ~1.0 of the Muse anchor on rhythm/voice?
• Harness shape: a script that, per (scenario × config), calls the existing generator, then the judge, and writes a scorecard. Reuse the complete() shim + feature_key: prose_eval.* for spend attribution, and the critic-runner pattern for the judge. Cost is real (dozens of LLM calls) — run on demand, not in CI.

5. Recommendations (ranked)

1. R1 — Ship "Style Examples" (voice exemplar). ✅ SHIPPED. story_seasons.style_example (migration 445); a "🎙 Voice" editor in the prose editor (StyleExampleModal.tsx, ~1,000-word guide) writes it via GET/PATCH /api/story-seasons/:seasonId/style-example (ownership-gated); injected as a voice-exemplar block into both chapter generation (callChapterPromptGPT) and inline prose-assist. Muse's single most-loved feature — now we have an analog.

2. R2 — Add a sentence_rhythm critic. ✅ SHIPPED. "Rhythm Reader" in script-critics.ts — flags flat cadence (uniform sentence length, no deliberate fragments, repetitive openers, comma-splice/metronomic monotony) and suggests specific re-cadenced rewrites without changing content. Added to DEFAULT_CRITIC_KEYS so it runs on every take; auto-surfaces in the critic-chip UI via /api/story-seasons/critics. Targets Muse's headline strength within our process bet.

3. R3 — Build the eval harness (§4). ✅ SHIPPED. Pure core apps/backend/src/services/prose-eval.ts (rubric, blind-letter assignment, judge-prompt builder, score parsing, aggregation) — 12 golden tests. Runner apps/backend/scripts/prose-quality-eval.ts (run on demand): cd apps/backend && npx tsx scripts/prose-quality-eval.ts [--configs a,b,c --judge <model> --runs N]. Generates each scenario × config, judges blind (config hidden, order rotated, ≥3 runs averaged), prints a per-config scorecard + the gap-vs-default-draft line for the R4 call. Costs live LLM calls — not in CI. feature_key prose_eval.generate|judge for spend attribution. Paste a Muse sample into MUSE_SAMPLES to include an external anchor.

4. R4 — Split draft-model policy by surface. ✅ DECIDED (from R3 data, 2026-07-02). The R3 harness ran (6 scenarios × 3 blind judge runs, judge = Opus):

   config	rhythm	voice	specificity	dialogue	pacing	cliché-free	readability	overall	n
   claude-opus-4-7	8.7	8.8	9.0	7.7	8.6	8.1	9.1	8.55	18
   claude-sonnet-4-6	8.4	8.5	8.7	7.2	8.6	7.9	8.9	8.31	18
   claude-haiku-4-5	6.6	6.7	6.5	6.4	6.5	5.2	7.4	6.47	18

   The default Haiku draft trails by 2.08 overall (well past the ~1.0 bar; weakest on cliché-freeness 5.2 + rhythm 6.6). Sonnet is the sweet spot — within 0.24 of Opus at ~1/5 the cost. Decision: writer-facing generative surfaces default to Sonnet (WRITER_QUALITY_MODEL_KEY); the audio pipeline keeps Haiku (cost-first) with Opus on promote. Shipped: the editor's inline prose-assist now runs on Sonnet.

5. R5 — Keep the critique loop as the strategy. It is our structural answer to Muse's long-context weakness and works with any base model (including a future one stronger than Muse). Don't trade the process bet away; augment it.

6. Decision

Do NOT chase a "beat Muse on one paragraph" rewrite of the engine. That's a losing bet against a fine-tuned model. Instead: (a) close the two cheapest, highest-visibility gaps now — Style Examples (R1) and a rhythm critic (R2) — which are the features writers actually feel; (b) stand up the eval harness (R3) so the draft-model tier (R4) is chosen from data; (c) keep leaning on the consistency + production-pipeline moat, which is where we beat Muse and where the market hurts most.

Net: our answer to Muse is not a better base model — it's your voice (exemplars) + our consistency (bible) + our process (critics) + the pipeline (audio/screenplay/scene). Muse writes a beautiful orphan paragraph; we write a consistent, produceable book in your voice.

Automation — model-change-gated cron (prose_quality_eval)

The eval re-runs itself as new models come out, cost-safely:

• Cron prose_quality_eval (wired in index.ts, toggle in cron-toggles.ts, label in system-health.ts) ticks daily but only runs the eval when the available-model registry fingerprint changes (a model added/removed) OR it's been >90 days since the last run. Every other tick is a $0 skip (one DB read + a fingerprint compare).
• When it runs, it benchmarks all available models, persists the scorecard to prose_eval_runs (migration 446), and sets a tier_review_recommended flag — surfaced as a partial heartbeat on /dashboard/admin/system-health — when a model beats the current WRITER_QUALITY_MODEL_KEY by >1.0 overall (the decision bar). A pricier model winning by a hair (Opus over Sonnet by 0.24) does not flag.
• OFF by default (PROSE_QUALITY_EVAL_ENABLED env + the super-admin toggle at /dashboard/admin/crons) — it spends LLM, so it stays dark until you turn it on. Kill switch in ~60s from the crons dashboard.
• Shared logic lives in prose-eval-runner.ts (used by both the cron and the manual scripts/prose-quality-eval.ts --save), so automated + ad-hoc runs benchmark the identical scenario set. The gating + tier-review math is pure + golden-tested (prose-eval.ts, 21 tests).

To turn it on: set PROSE_QUALITY_EVAL_ENABLED=true (or flip the toggle) after migration 446 is applied. It'll benchmark on the next new-model deploy.

Sources

• Sudowrite Muse · Which model / prose modes · Best AI fiction 2026 (MyLifeNote) · AI writing benchmarks 2026 (stacc) · Best AI models for novel writing 2026 (Inkfluence) </content>