How to use Dreambooth to put anything in Stable Diffusion (Colab notebook)

Dreambooth is a way to put anything — your loved one, your dog, your favorite toy — into a Stable Diffusion model. We will introduce what Dreambooth is, how it works, and how to perform the training.

This tutorial is aimed at people who have used Stable Diffusion but have not used Dreambooth before.

You will follow the step-by-step guide to prepare your training images and use our easy 1-click Colab notebook for dreambooth training — No coding required!

Do you know many custom models are trained using Dreambooth? After completing this tutorial, you will know how to make your own.

You will first learn about what Dreambooth is and how it works. But You can skip to the step-by-step guide if you are only interested in the training.

What is Dreambooth?

Published in 2022 by Google research team, Dreambooth is a technique to fine-tune diffusion models (like Stable Diffusion) by injecting a custom subject to the model.

Why does it call Dreambooth? According to the Google research team,

It’s like a photo booth, but once the subject is captured, it can be synthesized wherever your dreams take you.

Sounds great! But how well does it work? Below is an example in the research article. Using just 3 images of a particular dog (Let’s call her Devora) as input, the dreamboothed model can generate images of Devora in different context.

How does Dreambooth work?

You may ask, why can’t you simply train the model with additional steps with those images? The issue is that doing so is known to cause catastrophic failure due to overfitting (since the dataset is quite small) and language drift.

Dreambooth resolves these problems by

1. Using a rare word for the new subject (Notice I used a rare name Devora for the dog) so that it does not have a lot of meaning in the model in the first place.
2. Prior preservation on class: In order to preserve the meaning of the class (dog in the above case), the model is fine-tuned in a way that the subject (Devora) is injected while the image generation of the class (dog) is preserved.

There’s another similar technique called textual inversion. The difference is that Dreambooth fine-tunes the whole model, while textual inversion injects a new word, instead of reusing a rare one, and fine-tunes only the text embedding part of the model.

What you need to train Dreambooth

You will need three things

1. A few custom images
2. An unique identifier
3. A class name

In the above example. The unique identifier is Devora. The class name is dog.

Then you will need to construct your instance prompt:

a photo of [unique identifier] [class name]

And a class prompt:

a photo of [class name]

In the above example, instance prompt is

a photo of Devora dog

Since Devora is a dog, the class prompt is

a photo of a dog

Now you understand what you need, let’s dive into the training!

Step-by-step guide

Step 1: Prepare training images

As in any machine learning tasks, high-quality training data is the single most important factor to your success.

Take 3-10 picture of your custom subject. The picture should be taken from different angles.

The subject should also be in a variety of background so that the model can differentiate the subject against the background.

I will use this toy in the tutorial.

Step 2: Resize your images to 512×512

In order to use the images in training, you will first need to resize them to 512×512 pixels for training with v1 models.

BIRME is a convenient site for resizing images.

1. Drop your images to the BIRME page.
2. Adjust the canvas of each image so that it shows the subject adequately.
3. Make sure the width and height are both 512 px.
4. Press SAVE FILES to save the resized images to your computer.

Alternatively, you can download my resized images if you just want to go through the tutorial.

Step 3: Training

I recommend using Google Colab for training because it saves you the trouble of setting up. The following notebook is modified from Shivam Shrirao’s repository but is made more user-friendly. Follow the repository’s instructions if you prefer other setups.

The whole training takes about 30 minutes. If you don’t use Google Colab much, you can probably complete the training without getting disconnected. Purchase some compute credits to avoid the frustration of getting disconnected.

The notebook will save the model to your Google Drive. Make sure you have at least 2GB if you choose fp16 (recommended) and 4GB if you don’t.

1. Get this Dreambooth Guide and open the Colab notebook.
2. Enter the MODEL_NAME. The default name is for Stable Diffusion v1.5 model. You can find more models on HuggingFace here. The model name should be in the format user/model.
3. Enter the BRANCH name. See screenshot below on how to find the model name and the branch name.

4. Put in instance prompt and class prompt. For my images, I name my toy rabbit zwx so my instance prompt is “photo of zwx toy” and my class prompt is “photo of a toy”.

5. Click the Play button ( ▶️ ) on the left of the cell to start processing.

6. Grant permission to access Google Drive. Currently, there’s no easy way to download the model file except by saving it to Google Drive.

7. Press Choose Files to upload the resized images.

8. It should take about 30 minutes to complete the training. When it is done, you should see a few sample images generated from the new model.

8. Your custom model will be saved in your Google Drive, under the folder Dreambooth_model. Download the model checkpoint file and install it in your favorite GUI.

That’s it!

Step 4: Testing the model (optional)

You can also use the second cell of the notebook to test using the model.

Using the prompt

oil painting of zwx in style of van gogh

with my newly trained model, I am happy with what I got:

Note that you have to run this cell right after the training is complete. Otherwise your notebook may be disconnected.

Using the model

You can use the model checkpoint file in AUTOMATIC1111 GUI. It is a free and full-featured GUI you can install on Windows, and Mac, or run on Google Colab.

Using the model with the Stable Diffusion Colab notebook is easy. Your new model is saved in the folder AI_PICS/models in your Google Drive. It is available to load without any moving around.

If you have not used the GUI and the model file has been saved in your Google Drive, the easiest way is the Google Colab option. All you need to do is to put the path to the model in Google Drive to use it. See the step-by-step tutorial for more details.

How to train from a different model

You will need to change the MODEL_NAME and BRANCH.

Currently, the notebook only supports training half-precision v1 and v2 models. You can tell by looking at the model size. It should be about 2GB for v1 models.

You can find the model name and the branch name below on a Huggingface page. The page shown below is here.

Example: a realistic person

Realistic Vision v2 is a good model for training a new model with a realistic person. Use the following settings for a woman.

MODEL_NAME:

SG161222/Realistic_Vision_V2.0

BRANCH:

main

Instance prompt:

photo of zwx woman

Class prompt:

photo of woman

Here are a few images from the new model. You can find the training images in the Dreambooth guide.

Further readings

I recommend the following articles if you want to dive deeper into Dreambooth.

• Training Stable Diffusion with Dreambooth using Diffusers – Huggingface blog
• Dreambooth training guide – nitrosocke
• BlueFaux’s dreamBooth guide
• The research paper