How to use Dreambooth to put anything in Stable Diffusion

Updated Categorized as Tutorial Tagged 49 Comments on How to use Dreambooth to put anything in Stable Diffusion
Dreambooth a subject to a stable diffusion model

Dreambooth is a way to put anything — your loved one, your dog, your favorite toy — into a Stable Diffusion model. We will introduce what Dreambooth is, how does it work, and how to perform the training.

This tutorial is aim for people who have used Stable Diffusion but have not used Dreambooth before.

Do you know many custom models are trained using Dreambooth? After completing this tutorial, you will know how to make your own.

You can skip to the training part if you are only interested in the training.

What is Dreambooth?

Published in 2022 by Google research team, Dreambooth is a technique to fine-tune diffusion models (like Stable Diffusion) by injecting a custom subject to the model.

Why does it call Dreambooth? According to the Google research team,

It’s like a photo booth, but once the subject is captured, it can be synthesized wherever your dreams take you.

Sounds great! But how well does it work? Below is an example in the research article. Using just 3 images of a particular dog (Let’s call her Devora) as input, the dreamboothed model can generate images of Devora in different context.

dreambooth examples from the dreambooth research article
With as few as 3 training images, Dreambooth injects a custom subject to a diffusion model seamlessly.

How does Dreambooth work?

You may ask, why can’t you simply train the model with additional steps with those images? The issue is that doing so is known to cause catastrophic failure due to overfitting (since the dataset is quite small) and language drift.

Dreambooth resolves these problems by

  1. Using a rare word for the new subject (Notice I used a rare name Devora for the dog) so that it does not have a lot of meaning in the model in the first place.
  2. Prior preservation on class: In order to preserve the meaning of the class (dog in the above case), the model is fine-tuned in a way that the subject (Devora) is injected while the image generation of the class (dog) is preserved.

There’s another similar technique called textual inversion. The difference is that Dreambooth fine-tunes the whole model, while textual inversion injects a new word, instead of reusing a rare one, and fine-tunes only the text embedding part of the model.

What you need to train Dreambooth

You will need three things

  1. A few custom images
  2. An unique identifier
  3. A class name

In the above example. The unique identifier is Devora. The class name is dog.

Then you will need to construct your instance prompt:

a photo of [unique identifier] [class name]

And a class prompt:

a photo of [class name]

In the above example, instance prompt is

a photo of Devora dog

Since Devora is a dog, the class prompt is

a photo of a dog

Now you understand what you need, let’s dive into the training!

Step-by-step guide

Get training images

As in any machine learning tasks, high-quality training data is the single most important factor to your success.

Take 3-10 picture of your custom subject. The picture should be taken from different angles.

The subject should also be in a variety of background so that the model can differentiate the subject against the background.

I will use this toy in the tutorial.

Resize your images

In order to use the images in training, you will first need to resize them to 512×512 pixels for training with v1 models.

BIRME is a convenient site for resizing images.

  1. Drop your images to the BIRME page.
  2. Adjust the canvas of each image so that it shows the subject adequately.
  3. Make sure the width and height are both 512 px.
  4. Press SAVE FILES to save the resized images to your computer.

Alternatively, you can download my resized images if you just want to go through the tutorial.

Training

I recommend using Google Colab for training because it saves you trouble of setting up. The following notebook is modified from Shivam Shrirao‘s repository but is made more user-friendly. Follow the repository’s instruction if you prefer other setups.

The whole training takes about 30 minutes. If you don’t use Google Colab much, you probably can complete the training without getting disconnected. Purchase some compute credits to avoid the frustration of getting disconnected. As of Dec 2022, $10 USD will get you 50 hours so its not much of a cost.

The notebook will save the model to your Google Drive. Make sure you have at least 2GB if you choose fp16 (recommended) and 4GB if you don’t.

  1. Get this Dreambooth Guide and open the Colab notebook.
  2. You don’t need to change MODEL_NAME if you want to train from Stable Diffusion v1.5 model (Recommended).
  3. Put in instance prompt and class prompt. For my images, I name my toy rabbit zwx so my instance prompt is “photo of zwx toy” and my class prompt is “photo of a toy”.

4. Click the Play button ( ▶️ ) on the left of the cell to start processing.

5. Grant permission to access Google Drive. Currently there’s no easy way to download the model file except through saving to Google Drive.

6. Press Choose Files to upload the resized images.

7. It should take about 30 minutes to complete the training. When it is done, you should see a few sample images generated from the new model.

8. Your custom model will be saved in your Google Drive, under the folder Dreambooth_model. Download the model checkpoint file and install it in your favorite GUI.

That’s it!

Testing the model

You can also use the second cell of the notebook to test using the model.

Using the prompt

oil painting of zwx in style of van gogh

with my newly trained model, I am happy with what I got:

Using the model

You can use the model checkpoint file in AUTOMATIC1111 GUI. It is a free and full-feature GUI you can install on Windows, Mac or running on Google Colab.

If you have not used the GUI and the model file has been saved in your Google Drive, the easiest way is the Google Colab option. All you need to do is to put in the path to the model in Google Drive to use it. See the step-by-step tutorial for more details.

How to train from a different model

You will need to change the MODEL_NAME and BRANCH.

Currently the notebook only supports training half precision v1 models. You can tell by looking at the model size. It should be about 2GB.

You can find the model name and the branch name like below on a Huggingface page. The page shown below is here.

Further readings

I recommend the following articles if you want to dive deeper into Dreambooth.


Buy Me A Coffee

49 comments

  1. Thank you very much, I will follow that instruction.

    Also, I tried the AUTOMATIC1111 to solve the issue.
    When I put “zwx” it generates the rabbit, but putting “oil painting of zwx in style of Van Gogh” generates just a painting of Van Gogh (without the rabbit) .

    Is there any setting that I have to do before using dreambooth on AUTOMATIC1111??

    1. No, the notebook converted the model to a format that A1111 can read. I just tested the dreambooth model in A1111. The prompt “oil painting of zwx in style of Van Gogh” works as expected. I used 25 steps and CFG 7.

  2. Thank you for writing this article. But it seem that I have a problem here.

    I have run the second cell but I keep on getting results from the original model (not the one that I’ve trained).

    I’ve changed the model_path to the new model that’s in GoogleDrive but it won’t still function.
    It seems that this for statement hasn’t been run.
     if ‘pipe’ not in locals():

    So I deleted that line but got this error
     OSError: It looks like the config file at ‘/content/drive/MyDrive/Dreambooth_model/model1.ckpt’ is not a valid JSON file.

    1. Hi, model_path should point to a model in Hugging Face, not your own google drive. The second cell uses the trained model. It should be run right after the training.

      I suggest you use my example images first to ensure your setup is ok. You should get results similar to mine in the article. Then you can proceed with your own training images.

    1. Hi I tested the notebook and it is working correctly with my examples.
      There’s likely an error that happened during the training, causing the weight files not written. You take a look to see what the actual error is.

  3. Hello,

    I found a model which I’d like to use that has the 16fp pruned version. Does that work for the training? The other non-pruned files are not 16fp. Could I use those in training? I read your notebook only supports fp16. What is the best way forward? Thanks for the guide and collab!

  4. Thoughts on replacing the unique instance token name e.g. ohwx with the name of a celebrity that looks similar (using a tool like starmyface.com) to your subject when training people, as per JoePenna’s github: “Best practice is to change the token to a celebrity name (note: token, not class — so your prompt would be something like: Chris Evans person)”

  5. Hello!
    I am getting this error while testing image generation from model:

    —————————————————————————
    OutOfMemoryError Traceback (most recent call last)
    in
    30
    31 with autocast(“cuda”), torch.inference_mode():
    —> 32 images = pipe(
    33 prompt,
    34 height=height,

    13 frames
    /usr/local/lib/python3.8/dist-packages/diffusers/models/cross_attention.py in get_attention_scores(self, query, key, attention_mask)
    206
    207 if attention_mask is None:
    –> 208 baddbmm_input = torch.empty(
    209 query.shape[0], query.shape[1], key.shape[1], dtype=query.dtype, device=query.device
    210 )

    OutOfMemoryError: CUDA out of memory. Tried to allocate 4.50 GiB (GPU 0; 14.76 GiB total capacity; 11.97 GiB already allocated; 1.50 GiB free; 11.99 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

    What should I do?

      1. Are you running on colab or your own hardware? Colab should work fine. You may reduce ram usage with settings but that’s out of scope of this tutorial.

  6. Whenever I try and use a model path from Huggingface I keep getting errors. Suggestions?

    File “/usr/local/lib/python3.8/dist-packages/requests/models.py”, line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
    requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/darkstorm2150/Protogen_x3.4_Official_Release/resolve/fp16/model_index.json

    Revision Not Found for url: https://huggingface.co/darkstorm2150/Protogen_x3.4_Official_Release/resolve/fp16/model_index.json.
    Invalid rev id: fp16

    OSError: fp16 is not a valid git identifier (branch name, tag name or commit id) that exists for this model name. Check the model page at ‘https://huggingface.co/darkstorm2150/Protogen_x3.4_Official_Release’ for available revisions.
    Traceback (most recent call last):

    IndexError Traceback (most recent call last)
    in
    119 from glob import glob
    120 import os
    –> 121 WEIGHTS_DIR = natsorted(glob(OUTPUT_DIR + os.sep + “*”))[-1]
    122
    123

    IndexError: list index out of range

  7. Thanks a lot for the tutorial! Just a question on the last step, as I am not able to find the .ckpt file in my google drive (the Dreambooth_model folder is there though). Am I missing any step?

    I get the following message.

    Dreambooth completed successfully. It took 19.5 minutes.
    cp: cannot stat ‘/content/stable_diffusion_weights/output/800/model.ckpt’: No such file or directory
    Model saved to /content/drive/MyDrive/Dreambooth_model/model.ckpt

  8. Hi Andrew, I’m trying to load a model from another source other than huggingface. I have several in my stable diffusion model folder, but when I put the copied path from my gdrive, I get errors saying that it couldn’t connect to huggingface.co, and there doesn’t appear to be a model_index.json file. What am I doing wrong, and could you give me an example of a proper format to put into that field. Thanks!

  9. How do i upload the completed downloaded model to a GUI for usage? Where can I find said GUI’s, and how do install the models onto them?

  10. Hi Andrew,
    Thanks so much for all these tutorials, they’ve been really helpful.

    I couldn’t get the Dreambooth tutorial to work at all, I’m really lost…
    I am on a laptop and using the free version of colab.

    I run Dreambooth and upload your photos of the toy, it does export a .ckpt file, but it also gives lots of warnings along the way.

    When I tested the model, it just output similar photos of the toy, despite having the Van Gogh prompt, doesn’t look like a Van Gogh painting.

    The output model appears in my Google Drive, just called “model.ckpt”
    I then moved that file into “stable-diffusion-webui/models/Stable-diffusion/”
    Then I close the Dreambooth session and open Automatic1111. If I try a prompt with “photo of zxw” the toy does not show up in the picture. Should it? is that how it works?

    In Automatic1111, the zxw toy model, called “model” is not listed in the drop down, if I refresh the models, should it show up there? Do I need to tick something when I run the colab where it says “only check the models you are going to use”?

    I don’t know whether the .ckpt I’m creating in Dreambooth is broken as it doesn’t seem to work in the Dreambooth testing, or if I’m not understanding how to use it?

    1. Hi Steve,

      Are you able to test the trained model with the 2nd cell of dreambooth notebook? It has to be done right after training.

      If I understand correctly, you are using the Google Colab notebook provided in Quick Start Guide to run automatic1111. You will need to first rename the model file to something unique (e.g. model_toy.ckpt). The upload the file to Colab under /content/stable-diffusion-webui/models/Stable-diffusion. You can access the folder by clicking the “File” icon on the right side panel. Open the folders: stable-diffusion-webui → models → Stable-diffusion. Drag and drop the model file from your local computer to the Stable-diffusion folder.

      You should see your model file name after refreshing the checkpoint drop down menu.

      1. Hi Andrew,
        Thanks for replying. Yes that’s correct, I’m using the Automatic1111 colab recommended in the Quick Start Guide.
        When I test Dreambooth in cell 2, immediately after training and using the same prompt in the example, I do get 2 photos of the toy, but not in the style of Van Gogh.

        I tried to upload the .ckpt file to colab but the file is 2GB and uploads very slowly, it will take about 1-2 hours and I got disconnected from colab before it finished. If I do manage to get it to work, would I have to re-upload any .ckpt files I want to use every single time I use colab and then they get deleted when the session closes? Or can I save a version of that whole notebook to my GDrive, and the toy model will then be available next time I run it?

        Thanks for your help and sorry for all the basic noob questions
        Steve

        1. Hi Steve,

          There’s a bug in the dreambooth notebook causing incorrect training. I have updated the notebook and now the training should be working.

          I also updated the Colab Notebook in Quick Start Guide to have the ability to load a custom model in your Google Drive. Try it out.

          1. Wow! thank you so much Andrew, it works perfectly now. I really appreciate your help.
            One last thing (I promise!) – is there a way to load more than one custom .ckpt model from the Google Drive? If I put a comma, or a colon or slash between the files or something, so that I could load in a custom object AND a custom Style, or two objects, etc.?
            Thanks,
            Steve

  11. Hi Andrew,

    I also am getting an IndexError warning you can see below:

    IndexError Traceback (most recent call last)
    in
    120 from glob import glob
    121 import os
    –> 122 WEIGHTS_DIR = natsorted(glob(OUTPUT_DIR + os.sep + “*”))[-1]
    123
    124

    IndexError: list index out of range

    any guidance would be appreciated! thanks

  12. Hi, I received this error and I am at a loss at what to do.

    [*] WEIGHTS_DIR=/content/stable_diffusion_weights/output/0
    Dreambooth completed successfully. It took 89.7 minutes.
    —————————————————————————
    IndexError Traceback (most recent call last)
    in
    146
    147 row = len(folders)
    –> 148 col = len(os.listdir(os.path.join(weights_folder, folders[0], “samples”)))
    149 scale = 4
    150 fig, axes = plt.subplots(row, col, figsize=(col*scale, row*scale), gridspec_kw={‘hspace’: 0, ‘wspace’: 0})

    IndexError: list index out of range

      1. Thank you, that works now. New issue, Test image generation from model is not working.

        —————————————————————————
        ModuleNotFoundError Traceback (most recent call last)
        in
        13 import torch
        14 from torch import autocast
        —> 15 from diffusers import StableDiffusionPipeline, DDIMScheduler
        16 from IPython.display import display
        17

        ModuleNotFoundError: No module named ‘diffusers’

        —————————————————————————
        NOTE: If your import is failing due to a missing package, you can
        manually install dependencies using either !pip or !apt.

        To view examples of installing some common dependencies, click the
        “Open Examples” button below.
        —————————————————————————

        1. I just tested and it was working. By any chance your session has timed out? diffuser was imported in the first cell. You will need to run the test cell right after you completed first.

          1. Sorry to keep bugging. another error. Trying so hard to get this to work.

            [*] WEIGHTS_DIR=/content/stable_diffusion_weights/output/800
            Dreambooth completed successfully. It took 1.7 minutes.
            —————————————————————————
            FileNotFoundError Traceback (most recent call last)
            in
            146
            147 row = len(folders)
            –> 148 col = len(os.listdir(os.path.join(weights_folder, folders[0], “samples”)))
            149 scale = 4
            150 fig, axes = plt.subplots(row, col, figsize=(col*scale, row*scale), gridspec_kw={‘hspace’: 0, ‘wspace’: 0})

            FileNotFoundError: [Errno 2] No such file or directory: ‘/content/stable_diffusion_weights/output/800/samples’

          2. The training is way too short (1.7mins). There should be an error before that. You can disable clear log to see what it is.

  13. Hello Andrew,
    Why I always get the same images whatever I write as a prompt? Apparently everything worked fine.

  14. Hi Andrew,
    I wonder how can I use another model. The notebook has a field MODEL_NAME where the runwayml 1.5 is set by default. But if I try to give it an URL, a huggingface address or a PATH to my gdrive, it says that i cannot find the file. What format should I use, or what can I do?

    Thank you

    BTW, thank you for your amazing guides

      1. Thank you for your answer.

        Could it be possible to use a direct link to a ckpt file, or to a path on the drive?. There are models that are in civitai or in other places, that are not in huggingface.
        Also, how many steps do you recommend per image?

        Thank you

        1. Yes, you should be able to use a path in colab. You can copy the path from the context menu of colab’s file explorer sidebar. 800 steps are enough for most cases.

          1. I manged to make ti work with some HF models by removing this from the training cell:

            –pretrained_vae_name_or_path=”stabilityai/sd-vae-ft-mse” \
            –revision=”fp16″ \

            So it kinda works now. Got really good results for a face.
            However my new question (dorry to bother you), is how can I use this to train a style? How many images, steps and settings?

            Thanks again for your help

  15. Hi Andrew,

    Most “training” tutorials focus on applying popular art styles to images or training to understand objects.

    But I’m having a hard time finding any tutorials/guides that explain how to import a custom/personal art style.

    For example, I’m a PhotoShop artist. It takes me an unnecessarily long time to create this unique neon aesthetic via brushing and filters. It’s not a popular or public style…so I can’t really use text prompts to imitate it.

    It’s a very distinctive style, so I think Stable Diffusion could understand it via “training” – then I’d be able to apply it to my desired images. I’ve got so many past images for it to train on.

    But I have two problems…

    1) As per the intro, I can’t find a tutorial that explains how to train to a style (rather than object/person).

    2) I’m confused as to whether it’s possible to train in a simple manner on a budget graphics card like my Nvidia 1660 TI card. Like if I’m only trying to train a “style” (that involves colors, lines, and brightness)…do I still need to use complex “Google Colab” solutions?

    Hoping you could shed some light or point me in the right direction! I learn so much from your site – and absolutely love what you do!

    1. Hi Misha, you should be able to train your personal style with dreambooth. I haven’t written a tutorial for style but it can be done in the same way. prompts for each training image will need to be set accordingly.

      Another direction is to use textual inversion. The result is a embedding file you can use with a model.

      Dreambooth would need at least 10GB VRAM. If you don’t have a graphic card like this colab is a convenient option.

    1. The images are uploaded to the temporary Google Colab environment only you have access to. When you are disconnected from the runtime, the whole thing got deleted, including the images.

      So they cannot be accessed by another person.

Leave a Reply