Quality signals for the writer engine

Status: shipped through Phase 5 (v1.10–v1.14) as of 2026-05-14.

This document records the architecture and reasoning behind the five-phase quality-signals stack added to the writer engine in May 2026. The original implementation plan lived at ~/.claude/plans/compiled-purring-aho.md; this is the durable copy.

Why this exists

Two specific quality gaps prompted the project:

1. Cliché-prone titles. "Echoes / reflections / whispers" landed as defaults in season generation. The pre-existing BANNED_TITLE_NOUN_REGEX retry validator (in plan-persona-season.ts) was reactive — it blocked known bad patterns without promoting good ones.
2. Generic stakes / pacing. The Architect's Mirror generated stakes like "risks losing a vital connection to his mentor's legacy" — accurate but flat. The engine had no empirical sense of what concrete stakes look like or what chapter-length distributions read as well-paced.

The obvious move — train/RAG on Kindle bestsellers — was rejected:

• Legal exposure. The open Authors Guild + NYT v OpenAI cases make training or in-context use of copyrighted prose a real liability for a commercial product.
• Voice homogenization. Training on bestseller prose tends to pull AI personas toward bestseller default voice, which collapses the distinctness of each persona.

The agreed safer mix: structural signals only, no prose ingestion, drawn from four independent sources:

1. Craft principles — Save the Cat (Brody/Snyder), McKee's substance, Vogler's Hero's Journey. Our own prose summarizing public principles.
2. Internal engagement — top-quartile patterns from the platform's own published seasons (reads + follows + tips weighted).
3. Public-domain corpus — structural stats from ~50 curated Project Gutenberg works (Austen, Dickens, Twain, Wharton, Conrad, Doyle, Wells, etc.).
4. Bestseller market norms — curated structural facts (chapter count, page count, series position) from ~80 modern bestsellers. Scrapable Wikipedia / publisher-listing metadata, not the books themselves.

Every signal feeds the writer engine as prompt context, heuristic priors, or critic-chain hints. No model fine-tuning. No prose persisted to the database.

Architecture

Signal flow

┌─────────────────────────────────────────────────────────┐
│                    Season Generation                     │
└─────────────────────────────────────────────────────────┘
                            │
        ┌───────────────────┼───────────────────┐
        ▼                   ▼                   ▼
plan-persona-season.ts  chapter-stakes.ts  chapter-dramatic-roles.ts
        │                   │                   │
        │ injects:          │ injects:          │ injects:
        │  • craft beats    │  • craft beats    │  • craft beats
        │    per chapter    │    per chapter    │    per chapter
        │  • top-quartile   │  • corpus norm    │
        │    patterns by    │    per chapter    │
        │    genre          │  • top-quartile   │
        │  • bestseller     │    intensity      │
        │    norms          │    curve          │
        │  • example titles │
        │  • precedence     │
        │    note           │
        ▼                   ▼                   ▼
       LLM call            LLM call         (deterministic
        ↓                    ↓               heuristic)
   season plan        chapter stakes        role assignment

Data sources

Wire-in points

Signal precedence

When the four signals disagree (and they will), the planner prompt instructs the LLM:

1. Voice + premise authenticity always wins. If a persona's voice calls for a 12-chapter arc, write 12 — even if bestsellers in the genre run 30.
2. Platform top-quartile beats bestseller market norms. We know what works HERE before we know what works elsewhere.
3. Craft beats are tonal targets, not numeric targets. They shape pressure profile, not chapter length or count.

Phases

Phase 1 (v1.10) — Craft principles

Encodes Save the Cat's 15 beats, McKee's 5 substance movements, and Vogler's 8-stage Hero's Journey as TypeScript constants with position fractions and our-own-prose craft summaries. getCraftBeatsForChapter(chapterNumber, totalChapters) returns the nearest beat in each framework. Wired into all three writer-engine entry points.

Cost: zero data, zero infra. Pure prompt-engineering layer.

Phase 2 (v1.11) — Internal engagement signal

Materialized view scores every published novel-mode season by reads_30d × 1 + follows_30d × 5 + tip_count_30d × 10, NTILE(4)-buckets per genre. Top quartile feeds the planner as a soft prior. Admin surface at /dashboard/admin/quality-signals with manual refresh. Weekly cron + heartbeat tracking.

Caveat: only ~24 published novel-mode seasons today — quartiles are noisy. getTopQuartilePatterns() returns null when sample is below 8 seasons in the genre, and the planner falls back to its built-in defaults.

Phase 3 (v1.12) — Gutenberg corpus stats

Two tables (literary_corpus_works + literary_corpus_chapter_stats) hold per-chapter statistics only from ~50 curated Project Gutenberg works. Ingest script reads each chapter, computes word_count / sentence_count / avg_sentence_length / dialogue_density / opening + closing line word counts, writes the numbers, discards the prose. Stats cached for 1h. Wired into chapter-stakes prompts as per-position corpus norms.

Legal stance: only structural numbers persisted. No quotations, no sentences.

Phase 4 (v1.13) — Bestseller market norms

Curated seed of ~80 modern (2005–2023) bestsellers across literary / thriller / romance / mystery / fantasy / sci-fi / YA / historical / horror. Each entry is structural facts only (title, author, year, genre, chapter_count, page_count, series_position) — public-knowledge metadata. getGenreNorms(genre) returns P25/P50/P75 chapter counts + median page count.

Programmatic scraping deferred — manual curation initially. Re-evaluate once we have enough generations to know if the data shifts quality.

Phase 5 (v1.14) — Integration polish

Three additions tighten how the four signal sources stack:

1. Signal-precedence note in the planner prompt (above).
2. Concrete title-shape examples per genre — getExampleTitlesPerGenre(3) pulls 3 recent bestseller titles per genre and presents them as grounding examples.
3. Top-quartile intensity curve injected into chapter-stakes prompts when the season has a genre with enough top-quartile data.

No new data, no new tables — pure prompt-engineering layer.

Operations

One-time setup per environment

1. Apply migrations 251, 252, 253 via Supabase SQL editor.
2. Run npx tsx scripts/ingest-gutenberg-corpus.ts (Phase 3 data — downloads ~50 works, 2–5 minutes).
3. Run npx tsx scripts/seed-bestseller-signals.ts (Phase 4 data — instant, ~80 rows).
4. Visit /dashboard/admin/quality-signals and click ↻ Refresh now (Phase 2 materialized view).

Ongoing

• Phase 2 auto-refreshes weekly via engagement_quartiles_refresh cron (heartbeat-tracked).
• Phase 3 is static unless we add works — re-run the ingest with --ids 12345,67890 to add new ones; idempotent.
• Phase 4 is static unless we curate more entries.

Inspecting what's live

Admin page /dashboard/admin/quality-signals shows:

• Per-genre top-quartile patterns (Phase 2)
• Literary corpus summary by genre + era (Phase 3)
• Bestseller market norms per genre (Phase 4)

What we explicitly chose not to do

• No prose ingestion. Not into training data, not into RAG context. Every signal is structural.
• No fine-tuning. All four signals layer as prompt context.
• No web-scraping for bestseller metadata. Curated seed only — defer programmatic scraping unless the seed proves valuable.
• No copying craft-book prose. Save the Cat / McKee / Campbell summaries in story-craft-principles.ts are our own writing. Beat NAMES are common-vocabulary industry terms.

Open questions for future iteration

• Small internal sample. Quartiles over 24 seasons are noisy. Revisit Phase 2 cutoffs once we have ≥50 published seasons per genre.
• Genre normalization. story_seasons.genre is free-text. Phase 4 + Phase 2 both normalize to lowercase, but a genre_canonical enum would make signal lookup more reliable.
• Prompt budget. Stacking all four signals into chapter prompts pushes token counts up — track avg prompt tokens per generation; if it grows >2× from v1.9, consider compacting.
• Critic chain integration. Currently only the WRITER side of the engine consumes these signals. The critic chain (9 critics in v1.14) could read them too — e.g., "this chapter is 1500w, corpus P25 is 3000w; flag as thin?"