5 methods to generate consistent face with Stable Diffusion

Are you looking for ways to generate consistent faces across multiple images with Stable Diffusion? You may be working on illustrations of a storybook or a comic strip. In this post, you will find 3 methods to generate consistent faces.

1. Multiple celebrity names
2. Face swap with Reactor
3. Dreambooth
4. LoRA
5. ControlNet IP adapter face

Software

We will use AUTOMATIC1111 Stable Diffusion GUI. You can use this GUI on Windows, Mac, or Google Colab.

Check out the Quick Start Guide if you are new to Stable Diffusion.

Method 1: Multiple celebrity names

Using celebrity names is a sure way to generate consistent faces. Let’s study the following base prompt, which generates a generic face.

Base prompt:

photo of young woman, highlight hair, sitting outside restaurant, wearing dress, rim lighting, studio lighting, looking at the camera, dslr, ultra quality, sharp focus, tack sharp, dof, film grain, Fujifilm XT3, crystal clear, 8K UHD, highly detailed glossy eyes, high detailed skin, skin pores

We will use the same negative prompt for the rest of this article.

disfigured, ugly, bad, immature, cartoon, anime, 3d, painting, b&w

They are nice faces, but they are different. There are occasions when you want to generate the same face across multiple images.

As we have studied in the prompt guide, celebrity name is a powerful effect. Using them is a proven way to generate consistent faces.

Let’s add a strong name in Stable Diffusion — Emma Waston, to the prompt.

Emma Watson, photo of young woman, highlight hair, sitting outside restaurant, wearing dress, rim lighting, studio lighting, looking at the camera, dslr, ultra quality, sharp focus, tack sharp, dof, film grain, Fujifilm XT3, crystal clear, 8K UHD, highly detailed glossy eyes, high detailed skin, skin pores

We get Emma in all images.

But what if you don’t want images of any recognizable face? You just want a generic face in multiple images. There’s a technique for that. You can use multiple celebrity names to blend their faces into a single, consistent face.

Let’s use these three names: Emma Watson, Tara Reid, and Ana de Armas. Stable Diffusion will take all 3 faces and blend them together to form a new face.

Emma Watson, Tara Reid, Ana de Armas, photo of young woman, highlight hair, sitting outside restaurant, wearing dress, rim lighting, studio lighting, looking at the camera, dslr, ultra quality, sharp focus, tack sharp, dof, film grain, Fujifilm XT3, crystal clear, 8K UHD, highly detailed glossy eyes, high detailed skin, skin pores

That’s good. The face is consistent across these images. But why do they look so… Emma? The reason is Emma Watson is a very strong keyword in Stable Diffusion. You have to dial her down using a keyword weight. In AUTOMATIC1111, you use the syntax (keyword: weight) to apply a weight to a keyword.

Adjusting the weights of each name allows you to dial in the facial feature. We arrive at the prompt:

(Emma Watson:0.5), (Tara Reid:0.9), (Ana de Armas:1.2), photo of young woman, highlight hair, sitting outside restaurant, wearing dress, rim lighting, studio lighting, looking at the camera, dslr, ultra quality, sharp focus, tack sharp, dof, film grain, Fujifilm XT3, crystal clear, 8K UHD, highly detailed glossy eyes, high detailed skin, skin pores

See this face repeating across the images!

Use multiple celebrity names and keyword weights to carefully tune the facial feature you want. You can also use celebrity names in the negative prompt to avoid facial features you DON’T want.

Experiment with multiple celebrity LoRAs to achieve the same.

Method 2: Face Swap with ReActor

AUTOMATIC1111’s ReActor extension, a fork of the Roop extension, lets you copy a face from a reference photo to images generated with Stable Diffusion.

Installing the ReActor extension

Google Colab

Installing the ReActor extension on our Stable Diffusion Colab notebook is easy. All you need to do is to select the Reactor extension.

Windows or Mac

Follow these steps to install the ReActor extension in AUTOMATIC1111.

1. Start AUTOMATIC1111 Web-UI normally.

2. Navigate to the Extension Page.

3. Click the Install from URL tab.

4. Enter the following URL in the URL for extension’s git repository field.

https://github.com/Gourieff/sd-webui-reactor

5. Wait for the confirmation message that the installation is complete.

6. Restart AUTOMATIC1111.

Generating new images with ReActor

We will use text-to-image to generate new images.

photo of a 25 year old woman, new york city standing, skirt, looking at the camera, dslr, ultra quality, sharp focus, tack sharp, dof, film grain, Fujifilm XT3, crystal clear, 8K UHD, highly detailed glossy eyes, high detailed skin, skin pores

disfigured, ugly, bad, immature, cartoon, anime, 3d, painting, b&w

Model: dreamshaperXL_turboDpmppSDE

To enable face swap with ReActor, scroll down to the ReActor section.

Enable: Yes

Upload an image with the face you want to the Single Source Image.

Here are the results. The faces are swapped. Now you get the same face from all images!

Tips for using ReActor

Use a clear, high-resolution image for the reference face.

Method 3: Dreambooth

Perhaps the most reliable way to generate the same face is to use Dreambooth to create your own Stable Diffusion model.

Dreambooth is a technique to create a new Stable Diffusion checkpoint model with your own subject or style. In this case, the subject would be the person with your desired face.

Follow this link to find a step-by-step tutorial. You will need a few images of the person.

Gathering the training images could be a challenge. Here are a few options.

1. Ask someone you know for permission to use his/her photos.
2. Take some selfies.
3. Use the multiple celebrity name method above to generate training images.
4. Use the ReActor method above to generate training images.

We will use ReActor to generate the training images.

Step 1: Generate training images with ReActor

Follow the instructions from the previous section to generate 8 to 15 images with the same face using ReActor. Below are two examples of the training images. It’s fine to use blurry images.

Step 2: Train a new checkpoint model with Dreambooth

Follow the Dreambooth tutorial and download the Dreambooth training Colab notebook.

Since we want to train a model with a realistic style, we will use Realistic Vision v2.

MODEL_NAME:

SG161222/Realistic_Vision_V2.0

BRANCH:

main

Your new girl will be called zwx, a rare but existing token in Stable Diffusion. Since zwx is a woman, the instance prompt is

photo of zwx woman

The class is the category zwx belongs to, which is woman. So the class prompt is

photo of woman

By defining the class prompt correctly, you take advantage of all the prior attributes of women in the model and apply them to your girl.

Press the Play button to start training.

Upload the training image when prompted.

It would take some time. If everything goes well, the new model file will be saved to the designated output file name.

Step 3: Using the model

You can conveniently test your new model using the AUTOMATIC1111 Colab notebook. The dreambooth model is available to load if you don’t change the default paths of both notebooks.

Select your new dreambooth model in AUTOMATIC1111’s checkpoint dropdown menu.

Now test with a prompt with your girl’s name zwx:

photo of young zwx woman, highlight hair, sitting outside restaurant, wearing dress, rim lighting, studio lighting, looking at the camera, dslr, ultra quality, sharp focus, tack sharp, dof, film grain, Fujifilm XT3, crystal clear, 8K UHD, highly detailed glossy eyes, high detailed skin, skin pores

Now you get a consistent and sharp face every time you use the keyword zwx!

You can also generate this person in a different style.

oil painting of zwx young woman, highlight hair

disfigured, ugly, bad, immature, b&w, frame

Method 4: LoRA

A LoRA model is a small patch file for modifying a checkpoint model. You can train a LoRA model in a way similar to Dreambooth. See this tutorial for a step-by-step guide and the Colab notebook.

Method 5: ControlNet IP-adapter face

ControlNet is a neural network model designed to use with a Stable Diffusion model to influence image generation. Users typically use ControlNet to copy the composition or a human pose from a reference image. But do you know there’s a ControlNet for copying faces? It’s called the IP-adapter plus face model.