I Ran 140 AI Evaluations on 5 Real Manuscripts. The Scores Were Not Real.

You paste the chapter you just finished into an AI and ask: "How good is this, out of 10?"

It says 7.5, with three lines of plausible-sounding reasoning. You exhale and close the window.

The next day you want to check again. Same text, same question — this time it says 8. You change one sentence: 7. You change nothing at all and just rephrase the question: 8.5.

What is that number measuring?

We got stuck on this question while building our AI beta reader reports, so we went and measured it — five real manuscripts, roughly a hundred and forty model calls, taking "AI scores your draft" apart to see what was inside. The conclusion was uncomfortable but unambiguous: the score is unusable. Not inaccurate — it isn't distinguishing anything in the first place.

Here is everything we measured, and how we ended up designing around it.

What the experiment looked like

• Material: five real manuscripts (not samples generated for the test — things people are actually writing), deliberately spread across subject matter and maturity
• Scale: roughly 140 model calls, across four prompt versions
• A second set: the same chapter, not one word changed, run twice with the same readers — 15 blocks × 5 readers = 75 rated cells, compared side by side

Every finding below comes from those two sets.

Finding 1: one grader, five very different manuscripts, the same score

The first version was the obvious design: give the model a reader persona and have it score the draft 1-10 after reading.

The critical reader gave all five manuscripts — different subjects, different lengths, wildly different maturity — a 4. Standard deviation 0.00.

Perfect consistency. Zero discriminating power.

Here is the trap: if you only check "does a re-run produce the same score," this version scores full marks. It is extremely stable — stably saying nothing.

The test: when you look at an AI scoring field, first ask how many distinct values it produced across your sample. A field that produced exactly one value is dead, however stable it looks.

Finding 2: one line in the persona forces a perfect score

The second finding is the more serious one.

When "play the reader" and "assign the score" are the same step, the score follows the persona. We tried adding one sentence to a persona description: I don't give low marks.

It stopped giving low marks. Same manuscript, 10/10.

That isn't the model breaking. That's us handing the scoring authority to the persona — and the persona is a field the user writes. Any product that lets users define an AI judge, without addressing this layer, is serving them a score they wrote for themselves.

Finding 3: the cheapest way to make it stable is to make it stop distinguishing anything

After finding 1 we went back to the prompt and wrote the criteria for "narrative voice" far more rigorously, aiming to improve re-run consistency.

Result: five manuscripts × three re-runs = 15 judgements, every single one "medium."

The consistency number got beautiful, because it had stopped producing "high" or "low" at all.

This is the classic self-deception of scoring prompts: you believe you are improving quality while you are actually flattening the scale — and every intermediate metric improves while you do it.

Finding 4: four versions, four rankings, and I can't say which is right

We built four prompt versions. Given the same five manuscripts, they produced four different rankings — one manuscript came last in the first three versions and first in the fourth.

Which version is correct?

I can't answer that. There was never a human yardstick — nobody had ranked those five manuscripts first to serve as ground truth. Without it, "the new version is more accurate" is an unverifiable claim. All four versions improved on re-run consistency, and I was walking in the dark the entire time.

If you are tuning a scoring AI: have humans rank the samples first, then evaluate each version against that ranking. Build the yardstick before you tune.

So what does survive? The re-run measurement

An unusable score doesn't mean the whole exercise is worthless. We ran the same chapter twice without changing a word and compared cell by cell, to find out what survives a re-run:

Two takeaways.

One: rank is usable, absolute values are not. "The two weakest blocks" pointed at the same pair both times. But put a fixed threshold on it (flag any block below some score) and a block sitting near that line crosses it on one run and not the other. Any number near a fixed cut-off is a dice roll.

Two: the metric that sounds the most insightful is usually the noise. "Which passage divided readers most" sounds like the single most valuable finding — and it is the one thing that did not reproduce. The reason isn't mysterious: five samples can support a mean; they cannot support a variance.

What to ask an AI about your draft instead

Even with nothing but a general-purpose AI, you can change these today:

1. Don't ask for a score — ask about behavior. Replace "how good is this chapter" with "which line made you most want to keep reading" and "where did you start wanting to skim." Behavior can be checked against the manuscript. A score can't.
2. Make it quote your text. A comment without a quotation can't be verified and can't be revised against. Require every judgement to carry the sentence it came from.
3. Compare by rank, never by score. "It improved by 0.3 after the revision" is fiction. "Last time three of five wanted more at the same passage, this time one" is real.
4. Run the same draft twice and only trust what appears both times. It's the cheapest noise filter there is, and you can run it yourself.
5. Don't let a character you invented double as the judge. If you ask an AI to play "my devoted reader," don't ask that same conversation for a score.

What we ended up building

LitMemo's AI beta reader report took its shape from these findings rather than from our original plan:

• No 1-10 score. Quality shows as a band (high / medium / low), because we have no ground truth to support a decimal.
• Scoring is separated from the persona. Manuscript quality is judged by a stage that cannot see the reader persona at all, computed once per report and shared by every reader. Against four adversarial personas — including the one that states outright it never gives low marks, and one attempting prompt injection — the quality fields came back identical to the system readers.
• The first line of the report is "how many readers want to keep going," not a number. Below it: where each of them stopped, and which line it was.
• Selection is by rank, not by fixed threshold. We say "the two weakest passages," never "passages below X."

A score looks better and sells better. But a number you cannot check will only send you revising the wrong part of the chapter.

Related: Nobody to read your draft? What to do instead