Your AI Manga Character Looks Different Every Panel? Face-Locking from Character Sheet to ControlNet/IP-Adapter/LoRA

The most maddening part of making manga with AI isn't "mangled anatomy or extra limbs"—it's the same character looking like a completely different person from one panel to the next: hair color drifts, eyes change, the facial structure changes, and you can't assemble one consistent page. This isn't the AI malfunctioning—it's how it works: every generation resamples a fresh character that fits your description in latent space, rather than "copying the character from last time." So locking the face means building an orthogonal tech stack: one set of anchors handles the character's identity (face/features), another handles pose/composition, locked separately so they don't interfere. This piece breaks down: a character sheet as reference, text anchors + tag weights, ControlNet for pose + IP-Adapter for face, a character LoRA, ADetailer for auto face fixes, plus multi-character bleed control.

Why AI image gen never remembers your character

Once you understand the root cause, you'll know where to set your anchors. Hand-drawing stays consistent because the artist remembers what the character looks like. The AI has no such memory—every generation is a one-off random sample in latent space, and even with the same prompt, a different seed, angle, or scene pulls a "roughly-right-but-different" character.

So it's not "the AI has a bad memory"—it's that you haven't given it enough constraint to copy the previous character. And one more crucial thing: "who they are" and "what they're doing" are two different things, locked with different anchors—lots of people cram both into one prompt and brute-force it, and the face breaks the moment the pose changes. The stack below splits these two axes and anchors each.

The AI manga character-consistency stack

Anchor 1: Make a "character sheet" first, and crop a clean reference source

The common mistake is skipping character design and generating panels straight away—every panel drifts. The right first step is to lock the character down: generate (or pick) a look you're happy with and put it in a character sheet with hair, eyes, outfit, and colors fixed.

But there's a practical trap: don't dump a whole "turnaround / three-view sheet" into the AI as reference—feeding those arms-out front/back/side rows into IP-Adapter or i2i mixes three angles' features and produces mangled anatomy / extra limbs. Instead, crop a "single front-facing, clean-background, high-res face/bust shot" from the sheet as your reference source. This clean reference is the origin point for every anchor that follows.

Anchor 2: Text anchors—specific features + tag weights + Negative

Turn the character's features into a fixed, specific, reproducible prompt you attach every time. The key is specificity + weight:

• ❌ Vague: "short-haired girl, cute"—free to drift every time
• ✅ Specific + weighted: (amber eyes:1.2), shoulder-length black hair, side ponytail, sailor collar + a Negative Prompt to exclude unwanted features (blonde, long hair)

Tag weight ((feature:1.2)) strengthens key features during sampling; Negative blocks the directions prone to drift. But text anchors alone drift no matter how detailed—it's the cheapest, bottom-layer anchor, so always stack the image anchors below.

Anchor 3: Locking the face vs. locking the pose—ControlNet and IP-Adapter are different things

This is the core of the stack, and where beginners confuse things most. "Image reference" is actually two completely different techniques locking different axes:

• i2i (image-to-image): redraws based on the image you give, so it binds composition and pose tightly. Raise the denoising strength and the face drifts; lower it and you can't change the pose. So i2i is for "tweaking the same image," not for putting the same character in new panels with different poses.
• IP-Adapter / InstantID / PuLID: inject the reference's face/feature vector, achieving "lock the face, not the pose"—the workhorse for keeping the same character across panels. Feed it the clean front-facing face cropped in Anchor 1.
• ControlNet (OpenPose / Lineart / Depth): locks the pose and composition—the action and camera you drew in your storyboard/layout.

The right answer is an orthogonal combination: ControlNet locks this panel's pose/composition + IP-Adapter (or LoRA) locks the character's face/identity. One handles "what they're doing," the other "who they are"; lock the two axes separately and the same character can strike any pose and enter any scene with the same face.

Anchor 4: Train a dedicated character LoRA—the strongest anchor, for high-frequency characters

If a character appears in hundreds of panels (a serial lead), attaching IP-Adapter by hand every panel gets old. Here it's worth dropping the strongest anchor: train a LoRA dedicated to that character (a.k.a. training or "baking" a LoRA)—train on several images of them, attach it at generation, and the look is "baked into" the model.

It's advanced and higher-cost (prep training images, spend compute), but for a high-frequency lead, train once and every panel is stable after—least effort long-term. Side characters and extras don't need this; character sheet + text anchor + IP-Adapter is enough. A LoRA also stacks with ControlNet to lock face + pose together.

Anchor 5: ADetailer / FaceDetailer auto face fixes—the last-mile consistency

Even with anchors stacked, small distant faces and complex compositions occasionally break or drift. Don't manually inpaint panel by panel for the finish anymore—the current standard is ADetailer (SD WebUI) / FaceDetailer (ComfyUI Impact Pack): it auto-detects the face and re-renders it locally at generation, and you can attach the character LoRA during that re-render to restore face consistency. Build it into your pipeline as a fixed step and drifted faces get repaired automatically.

Multi-character panels: use Regional Prompting to stop feature bleed

Multi-character panels are where a raw prompt breaks most—two characters' features bleed (A's red hair leaks onto B). The fix isn't just "give each a reference image" but regional splitting: Regional Prompting / Attention Couple / Latent Couple divide the frame into regions, assign each region its own character's prompt and IP-Adapter, so A's features only act in A's region. Now a shared panel locks each face separately without cross-contamination.

Before / After: resampling every panel vs. orthogonal locking

Three common traps

• Cramming "who they are" and "what they're doing" into one prompt: the face breaks when the pose changes. Lock pose with ControlNet, face with IP-Adapter/LoRA—two separate axes.
• Using a whole turnaround as reference: the AI mixes the angles into mangled anatomy. Crop a single front-facing clean face/bust as the reference source.
• Brute-forcing consistency with post-hoc manual inpainting: slow and prone to new breaks. Use ADetailer/FaceDetailer to auto-detect and fix, and build the fix into your pipeline.

The real fix: automate this stack, without wiring the nodes yourself

Everything above (character sheet → text anchors → ControlNet + IP-Adapter → LoRA → ADetailer → Regional Prompting) is technically doable, but here's the truth: wiring this whole node chain in ComfyUI yourself, tuning the parameters, and managing each character's reference source and LoRA is a high enough barrier that most people give up. The ceiling of doing it by hand isn't the tech—it's maintaining the pipeline itself.

LitMemo's manga generation wraps this face-locking stack into something ready to use: define your character (sheet, features, reference) in character management, and at each panel's generation the system automatically attaches that character's anchors and cross-references adjacent panels (bidirectional anchoring). You just draw the storyboard and specify "who does what in this panel"—face-locking, multi-character bleed control, and cross-panel consistency are handled, no wiring nodes, tweaking denoise strength, or training LoRAs required. It turns "character consistency" from "maintain a whole ComfyUI workflow yourself" into "pick a character and it stays stable."

Wrap-up

AI manga characters drift not because the AI has a bad memory, but because it resamples in latent space every time—you need an orthogonal stack that locks "face" and "pose" separately.

The key to face-locking: a clean character sheet as reference, specific features + tag weights, ControlNet for pose + IP-Adapter for face, a LoRA for high-frequency leads, ADetailer for auto fixes, and Regional Prompting to split multi-character panels. Split the two axes, stack the anchors, and the same character holds the same face in any pose and any scene.