How we test

The bug was never broken code. It was a green check over a wrong artifact.

Type-checks pass, tests pass, every route returns 200 — and the page still ships wrong. That’s the failure we kept hitting, and it’s the one a test suite designed to check what you told it to check will never catch. So the disciplines below aren’t a methodology we adopted. Each one is a specific bug that got past us, and the rule we wrote so it couldn’t happen the same way twice.

Verify the artifact that shipped, not the code’s story about itself

The most expensive misses share a shape: the source is correct and the received output is wrong, so nothing that reads the source ever sees it. The only instrument that catches this is fetching the served bytes and reading them.

We ran that sweep across every public page. Five surfaces were fetching their own content in an effect and shipping an empty shell to first paint; we fixed them. And the pages that mattered most for search — the comparison and roundup content — were clean. That’s worth stating plainly: a sweep that finds nothing where it would hurt most is a result, not a wasted afternoon. Silence is only reassuring if you actually ran the check.

A machine’s finding is a claim until a human agrees

We run browsing personas — backstoried, cost-capped agents that visit a public page with a first-time-visitor’s goal (“order dinner near me tonight”) and report what they actually hit: dead ends, confusing copy, a page that renders nothing. They catch the class Playwright can’t, because Playwright asserts what you told it to; a persona notices what you didn’t.

The load-bearing part isn’t the agent — it’s the discipline around trusting it:

• Every finding files as triage, never open — a machine’s observation is a claim until a human confirms it. The status is hard-coded on the receiving side, so a payload that tries to file itself as confirmed work still lands in triage. One bad run must not be able to flood the real queue; that’s how a tool earns being left on.
• “I couldn’t find X” and “I hit X” are different evidence. An absence the agent searched for and didn’t find is weaker than friction it ran into directly — because “didn’t find” is indistinguishable from “didn’t look.” We tag them separately and cap the weak one, so a confident-sounding absence can’t outrank a thing that actually happened.
• The label travels with the finding. Every report carries a fixed note — “AI persona observation, behaves as a real person would, not a human tester” — rendered verbatim, never stripped. A finding sold as human testing is a lie the first bad session exposes.

And it isn’t a script we keep to ourselves — it’s a product, AI Personas, run on our own surfaces before anyone else’s. Each one is a real backstory the agent inhabits — the point of the backstory is the blind spot it removes. A few of the roster (representative examples):

And here is where their runs land — the operations side, for a human to weigh, at triage, never as confirmed work:

Generated output is checked by a second instrument, never trusted by one

When a model reads a screenshot or writes a draft, “the code is correct” tells you nothing about the output — it’s the one artifact where the two are unrelated. So generated output gets a human confirm before it counts as real, and format gets its own separate check from content.

Same rule, one product over: when a vision model parses a calendar screenshot into events, the parse is shown for review before anything saves. A calendar that silently invents a meeting is worse than one you confirm — so the confirm isn’t optional.

Check your own guarantee before someone else repeats it

A sibling product wanted to tell its users, in its own voice, that our engine stores nothing they paste. Before we let that sentence stand, we read our own system end to end — every route, every write. The only persistence in the whole path is a usage log, and it records metadata only: which feature, token counts, cost. No pasted text, no résumé, no answer. We could confirm the claim as a fact instead of a hope, and the contract between the two products states it that way — with the exact edit it must never receive noted in the code, because “never sent anywhere” is one careless pass from “never stored.”

Cost is a test constraint, not an afterthought

Agentic tests and generative steps spend real money, so the budget is part of the test design: a hard step cap and a per-run cost cap, a kill switch, and a cache that never re-synthesizes a line it has already produced. An unbounded test loop is a test you can’t afford to run often enough to trust — so bounding it is what makes it a gate rather than a demo.

The through-line

None of this is rigor for its own sake. Every rule here is the scar tissue of a specific green-but-wrong ship, and the cheapest instrument that would have caught it: a curl, a second reader with no prior commitment to what the page was supposed to be, a re-import with real data present, a file over the bytes. Test the output your users receive, not the story your code tells about itself — because the gap between those two is exactly where the expensive bugs live.

These are HiveJournal’s disciplines, drawn from real incidents across a small mesh of products. Written to be useful, not to impress — if a rule here reads as obvious, it’s only obvious after the bug that taught it.