I Hid 22 Contradictions in a Novel and Asked AI to Find Them. The First Version Found 6.
In chapter two, the protagonist slices his right palm open on broken glass. Five stitches; the doctor tells him not to put weight on it for a week. In chapter three, the next day, he braces his right hand on the gunwale and hauls himself up onto the deck, "his palm flat against the rough wood, and it didn't hurt at all."
You won't notice while you're writing. The two sentences sit a chapter apart and were written a night apart, and each reads fine on its own. Readers notice, usually in the comments.
This kind of mismatch is exactly what AI ought to be good at: it doesn't get tired, doesn't skip lines, and can read every chapter word by word. So we ran a test to see how many it actually finds.
The test novel
We wrote a 15-chapter novel of a little over 30,000 Chinese characters, The December Light, set in a seaside town with a lighthouse, a typhoon and an old fisherman who has gone missing. While writing it we deliberately planted two kinds of things:
- 22 contradictions: the grandmother is seventy-eight in chapter one, and in chapter thirteen says she has lived eighty-five years; October 3 is a Friday, yet October 9 is written as a Wednesday; the boat called the Fair Tide becomes the Fair Wind in chapter fourteen; the dog that died last winter is lying at its owner's feet again in chapter twelve.
- 10 traps: places that look inconsistent but that the story explains. A character tells a lie and later admits it; a stranger on the docks turns out, a chapter later, to be someone in disguise; a sprained ankle "rested for a week" finally lets someone run.
A good check finds the first kind and lets the second go. A trap reported as a contradiction is a false positive.
Why we rebuilt it
The original consistency check first condensed each chapter into a summary of 300 to 500 characters, then gave the summaries, characters and world settings to the model to look for contradictions. A detail like "five stitches in his right palm" doesn't survive a summary. One chapter's summary says "the protagonist is injured at a bar," the next says "the protagonist goes out to sea," and side by side those two lines contradict nothing. The detail is lost in the condensing, and no amount of comparison afterwards can see it.
So the new version reads the original text from the start. This test novel is how we measured each change to it.
Version 1: reading the original text chapter by chapter, 6 found
The first version read the original text one chapter at a time, listed the facts that chapter established, each with its quote, and then compared the facts across the whole book.
It found 6 of the 22, with zero false positives on the 10 traps.
The problem was the extraction step. The model pulled out only about 10 facts per chapter, and of the 22 contradictions, only 6 had both sides extracted. If one side never gets extracted, the comparison can't see it.
Three rounds of changes
Version 2: a must-extract checklist. Times, weekdays, numbers, names exactly as spelled in the text, where an injury is and whether it's left or right, where each person is at the end of the chapter, who is holding what, who can swim, how people met. Sweeping item by item, it extracted 25 to 40 facts per chapter. Contradictions with both sides extracted rose to 16, yet only 9 were found. The bottleneck had moved to the next step: with every fact in the book compared in one pass, the model could only pick out a dozen or so suspects.
Version 3: compare in groups. Facts about the same character in one group, times in another, places in another, compared group by group. It found 15.
Version 4: each chapter checks itself first, and every suspect goes back to the text. Some contradictions happen inside a single chapter: in the morning "I never drink coffee," in the evening "made himself his usual cup of black coffee." So each chapter is checked against itself right after extraction. Each suspect the comparison picks out is then reread against the original text around both passages, and listed only if nothing in between explains it. This step is what lets the traps through: the character who lied admits it later, the recheck reads that line, and the item is dropped.
Version 4 found 20, with 1 false positive on the 10 traps.
Before launch: the same book, seven runs
Lab numbers and the real product are two different things, so before launch we ran the book seven times through the production pipeline, with the same model:
| Result | |
|---|---|
| Contradictions found, out of 22 | 17, 19, 19, 17, 17, 18, 16; average 17.6 |
| False positives on the 10 traps | 1 to 2 per run |
| Share of quoted passages that can be found in the manuscript | 100% |
Across seven runs on the same book, the count moved between 16 and 19. Only two contradictions were missed in all seven.
The two missed every time
The stitched hand. The one from the opening. In chapter three, "braces his right hand on the gunwale" was never extracted as "the state of his right hand" in any of the seven runs. It reads as an action, not as a fact about the character.
The full moon that never comes. In chapter five the grandmother says, "Wait for the next full moon and you'll understand," and no full moon appears anywhere in the rest of the book. This is a different kind of problem: no two sentences clash. What's missing is something that should have appeared. The check compares what was written against what was written; what was never written is invisible to it. Whether foreshadowing pays off is a job for a different tool.
These two misses mark out what it can do. Facts written on the page, it matches in fine detail. Inferring a physical state from an action, or noticing that something "should have shown up and didn't," still falls to you.
What you can borrow for your own checks
These work without any AI:
- Make a must-extract checklist. Times, weekdays, ages, headcounts, how names are spelled, which side an injury is on, where everyone is at the end of each chapter. Sweep against the list when you revise; it catches more than rereading whole chapters.
- Check in groups. Read every scene with one character in a single pass, then every date in a single pass. Looking for one kind of thing at a time keeps your eyes from sliding over it.
- When you suspect a contradiction, look for an explanation in between. Many "contradictions" are turns you wrote yourself and forgot because they were too far apart.
- Keep a separate list of foreshadowing. Note each one when you plant it, cross it off when it pays off. Comparison can't catch this.
LitMemo's consistency check
AI consistency checking is version 4: it reads the original text chapter by chapter, extracts facts against the checklist, compares them in groups and takes every suspect back to the text. Each finding shows the passages from both places, and a click jumps to that spot in the manuscript. Lies a character states outright are marked "possibly intentional," and anything you mark as intentional isn't listed again. On later checks it rereads only new or edited chapters, still comparing against the whole book.
Related: Your Character Doesn't Feel Like Themselves Anymore? 3 Ways to Catch (and Fix) OOC / Character Drift
