Dreambooth is a way to put anything — your loved one, your dog, your favorite toy — into a Stable Diffusion model. We will introduce what Dreambooth is, how it works, and how to perform the training.
This tutorial is aimed at people who have used Stable Diffusion but have not used Dreambooth before.
You will follow the step-by-step guide to prepare your training images and use our easy 1-click Colab notebook for dreambooth training — No coding required!
Do you know many custom models are trained using Dreambooth? After completing this tutorial, you will know how to make your own.
You will first learn about what Dreambooth is and how it works. But You can skip to the step-by-step guide if you are only interested in the training.
Contents
What is Dreambooth?
Published in 2022 by Google research team, Dreambooth is a technique to fine-tune diffusion models (like Stable Diffusion) by injecting a custom subject to the model.
Why does it call Dreambooth? According to the Google research team,
It’s like a photo booth, but once the subject is captured, it can be synthesized wherever your dreams take you.
Sounds great! But how well does it work? Below is an example in the research article. Using just 3 images of a particular dog (Let’s call her Devora) as input, the dreamboothed model can generate images of Devora in different context.
How does Dreambooth work?
You may ask, why can’t you simply train the model with additional steps with those images? The issue is that doing so is known to cause catastrophic failure due to overfitting (since the dataset is quite small) and language drift.
Dreambooth resolves these problems by
- Using a rare word for the new subject (Notice I used a rare name Devora for the dog) so that it does not have a lot of meaning in the model in the first place.
- Prior preservation on class: In order to preserve the meaning of the class (dog in the above case), the model is fine-tuned in a way that the subject (Devora) is injected while the image generation of the class (dog) is preserved.
There’s another similar technique called textual inversion. The difference is that Dreambooth fine-tunes the whole model, while textual inversion injects a new word, instead of reusing a rare one, and fine-tunes only the text embedding part of the model.
What you need to train Dreambooth
You will need three things
- A few custom images
- An unique identifier
- A class name
In the above example. The unique identifier is Devora. The class name is dog.
Then you will need to construct your instance prompt:
a photo of [unique identifier] [class name]
And a class prompt:
a photo of [class name]
In the above example, instance prompt is
a photo of Devora dog
Since Devora is a dog, the class prompt is
a photo of a dog
Now you understand what you need, let’s dive into the training!
Step-by-step guide
Step 1: Prepare training images
As in any machine learning tasks, high-quality training data is the single most important factor to your success.
Take 3-10 picture of your custom subject. The picture should be taken from different angles.
The subject should also be in a variety of background so that the model can differentiate the subject against the background.
I will use this toy in the tutorial.
Step 2: Resize your images to 512×512
In order to use the images in training, you will first need to resize them to 512×512 pixels for training with v1 models.
BIRME is a convenient site for resizing images.
- Drop your images to the BIRME page.
- Adjust the canvas of each image so that it shows the subject adequately.
- Make sure the width and height are both 512 px.
- Press SAVE FILES to save the resized images to your computer.
Alternatively, you can download my resized images if you just want to go through the tutorial.
Step 3: Training
I recommend using Google Colab for training because it saves you the trouble of setting up. The following notebook is modified from Shivam Shrirao’s repository but is made more user-friendly. Follow the repository’s instructions if you prefer other setups.
The whole training takes about 30 minutes. If you don’t use Google Colab much, you can probably complete the training without getting disconnected. Purchase some compute credits to avoid the frustration of getting disconnected.
The notebook will save the model to your Google Drive. Make sure you have at least 2GB if you choose fp16 (recommended) and 4GB if you don’t.
- Get this Dreambooth Guide and open the Colab notebook.
- Enter the MODEL_NAME. The default name is for Stable Diffusion v1.5 model. You can find more models on HuggingFace here. The model name should be in the format
user/model. - Enter the BRANCH name. See screenshot below on how to find the model name and the branch name.
4. Put in instance prompt and class prompt. For my images, I name my toy rabbit zwx so my instance prompt is “photo of zwx toy” and my class prompt is “photo of a toy”.
5. Click the Play button ( ▶️ ) on the left of the cell to start processing.
6. Grant permission to access Google Drive. Currently, there’s no easy way to download the model file except by saving it to Google Drive.
7. Press Choose Files to upload the resized images.
8. It should take about 30 minutes to complete the training. When it is done, you should see a few sample images generated from the new model.
8. Your custom model will be saved in your Google Drive, under the folder Dreambooth_model. Download the model checkpoint file and install it in your favorite GUI.
That’s it!
Step 4: Testing the model (optional)
You can also use the second cell of the notebook to test using the model.
Using the prompt
oil painting of zwx in style of van gogh
with my newly trained model, I am happy with what I got:
Note that you have to run this cell right after the training is complete. Otherwise your notebook may be disconnected.
Using the model
You can use the model checkpoint file in AUTOMATIC1111 GUI. It is a free and full-featured GUI you can install on Windows, and Mac, or run on Google Colab.
Using the model with the Stable Diffusion Colab notebook is easy. Your new model is saved in the folder AI_PICS/models in your Google Drive. It is available to load without any moving around.
If you have not used the GUI and the model file has been saved in your Google Drive, the easiest way is the Google Colab option. All you need to do is to put the path to the model in Google Drive to use it. See the step-by-step tutorial for more details.
How to train from a different model
You will need to change the MODEL_NAME and BRANCH.
Currently, the notebook only supports training half-precision v1 and v2 models. You can tell by looking at the model size. It should be about 2GB for v1 models.
You can find the model name and the branch name below on a Huggingface page. The page shown below is here.
Example: a realistic person
Realistic Vision v2 is a good model for training a new model with a realistic person. Use the following settings for a woman.
MODEL_NAME:
SG161222/Realistic_Vision_V2.0BRANCH:
mainInstance prompt:
photo of zwx woman
Class prompt:
photo of woman
Here are a few images from the new model. You can find the training images in the Dreambooth guide.
Further readings
I recommend the following articles if you want to dive deeper into Dreambooth.
- Training Stable Diffusion with Dreambooth using Diffusers – Huggingface blog
- Dreambooth training guide – nitrosocke
- BlueFaux’s dreamBooth guide
- The research paper
Hi… I need to train a model using difussers v0.14 or less… this is because later a want to extract the lora from the model using supermerger extension in automatic 1111 and this extension only works if the model use 0.14 difussers or less.
Is there any way to use this google colab in that way. I mean add or change something in the code to force that version of difussers. Somebody told me that is possible using dreambooth extension in automatic 1111 locally but I don’t have enough resources in my PC. Thanx in advance and sorry for my English.
Is there a way to train an object *with scale*? I’m a blacksmith. Getting an anvil out of any model is an exercise in sci-fi frustration. (Or a hammer, for that matter.) I tried textual inversion and an anvil-faced man is a sight that can’t be unseen. Would Dreambooth help with that? I don’t want a micro-anvil or a building-sized one, hence my question.
Haha, yes. dreambooth is a lot of powerful than textual inversion. It’s worth a try.
This is a really great article! Thank you a lot! Can we also train based on XL model? I tried with “stabilityai/stable-diffusion-xl-base-1.0” and it didn’t work. Could you please update the Colab and your article to show that? 🙂
SDXL is not supported with this notebook. I’m working on a lora training notebook. hope to get sdxl to work there.
Thank you for a great tutorial. I am trying to inject a custom model (natural person) into the realistic anime characters.
Would you please recommend a model to use and a step-by-step guideline process for creating an AI image of a 3D-realistic anime character pose and outfit with a real person. Thank you.
Anything v3 is a good base model for further finetuning. Find more here: https://stable-diffusion-art.com/anime-models-and-prompts/
You can experiment with using lora to make it look 3D. For example the following but there are many.
https://civitai.com/models/10679/stylized-3d-model-lora
Currently i’m using Automatic111 local on my machine with a custom model to create a real person from civitai.
I’d like to train a dreambooth model face using the colab notebook like this post. Is it possible to select that custom model that i’m using from civitai? or only the huggingface models are available for training? If so, how to train the custom model from civitai? Or it is best to train a lora?
Only huggingface is supported right now. There’s a way to but needs some setup.
Any SDXL dreambooth?
coming soon!
Hello. I appreciate the detailed tutorial however, it didn’t work for me. As far as I can tell, I have followed the instructions to a T. The behavior I saw was that the top portion finished and I was correctly shown the sample images but then I tried to generate the Van Gogh images using the identifier I used in the top section did not work. I realized that perhaps that identifier wasn’t specific enough so I tried again. It was an identifier that couldn’t be a common term yet I saw the same behavior. I’m not sure what other information you’d need to help but I am willing to try it a third time to see where it’s failing. Thanks!
I just tested the notebook and it is working as expected using the toy example. You can try that first to make sure your setup is correct.
You can also try not changing the dreambooth keyword to see if that works for you.
AttributeError: ‘NoneType’ object has no attribute ‘cuDeviceGetCount’
—————————————————————————
KeyboardInterrupt Traceback (most recent call last)
in ()
121 weightdirs = natsorted(glob(OUTPUT_DIR + os.sep + “*”))
122 if len(weightdirs) == 0:
–> 123 raise KeyboardInterrupt(“No training weights directory found”)
124 WEIGHTS_DIR = weightdirs[-1]
125
KeyboardInterrupt: No training weights directory found
Hi, I did a test run and it is working correctly. Use the image examples in the article to test your setup.
I still have the same error. How did you fix it?
Try running the example images in the tutorial.
This is the error:
HFValidationError: Repo id must be in the form ‘repo_name’ or ‘namespace/repo_name’: ‘/content/drive/MyDrive/Dreambooth_model/model1.ckpt’. Use `repo_type` argument if needed.
Okay, I have a model.ckpt saved in my drive. Now how do I use it directly? When I paste the path of the model in the 2nd cell, it is giving an error.
You can download and use it in Stable Diffusion as a custom model like those downloaded from civitai.
So apparently I was using an older version, it’s working now, thanks 😉
Hi, I’m getting the same error :
KeyboardInterrupt Traceback (most recent call last)
in ()
121 weightdirs = natsorted(glob(OUTPUT_DIR + os.sep + “*”))
122 if len(weightdirs) == 0:
–> 123 raise KeyboardInterrupt(“No training weights directory found”)
124 WEIGHTS_DIR = weightdirs[-1]
125
KeyboardInterrupt: No training weights directory found
Can you try with the latest updated notebook and with the example images in the article? All errors should have been fixed.
I’m getting (“No training weights directory found”) as of this afternoon (worked last night)
Hi! I´ve followed the guide step by step but I got the error
ImportError: accelerate>=0.20.3 is required for a normal functioning of this module, but found accelerate==0.19.0.
Try: pip install transformers -U or pip install -e ‘.[dev]’ if you’re working with git main
—————————————————————————
KeyboardInterrupt Traceback (most recent call last)
in ()
121 weightdirs = natsorted(glob(OUTPUT_DIR + os.sep + “*”))
122 if len(weightdirs) == 0:
–> 123 raise KeyboardInterrupt(“No training weights directory found”)
124 WEIGHTS_DIR = weightdirs[-1]
125
KeyboardInterrupt: No training weights directory found
I tried to restart the process as other user suggested but It did nothing, I´ve also tried with the sample images and nothing 🙁 Any help?
Hi, thanks for reporting. The issue is fixed. Please pull the latest notebook (7/19).
I am getting this error, does accelerate need to be updated again?
ImportError: accelerate>=0.20.3 is required for a normal functioning of this module, but found accelerate==0.19.0.
Try: pip install transformers -U or pip install -e ‘.[dev]’ if you’re working with git main
—————————————————————————
KeyboardInterrupt Traceback (most recent call last)
in ()
121 weightdirs = natsorted(glob(OUTPUT_DIR + os.sep + “*”))
122 if len(weightdirs) == 0:
–> 123 raise KeyboardInterrupt(“No training weights directory found”)
124 WEIGHTS_DIR = weightdirs[-1]
125
KeyboardInterrupt: No training weights directory found
I’m getting the same error. The notebook was working fine until yesterday. None of the fixes mentioned here work anymore. Maybe it’s some new bug
Hi, thanks for reporting. The issue is fixed. Please pull the latest notebook (7/19).
I tired again a few hours later and everything worked fine – ¯\_(ツ)_/¯
Hey, I have the same problem – there’s a warning about not being able to find TensorRT
You can safely ignore the warning.
Went through guide and keep coming up with Message Error after attempting to upload my png pics. Would appreciate some guidance. Specifically it says
MessageError Traceback (most recent call last)
in ()
75 for c in concepts_list:
76 print(f”Uploading instance images for `{c[‘instance_prompt’]}`”)
—> 77 uploaded = files.upload()
78 for filename in uploaded.keys():
79 dst_path = os.path.join(c[‘instance_data_dir’], filename)
3 frames
/usr/local/lib/python3.10/dist-packages/google/colab/_message.py in read_reply_from_input(message_id, timeout_sec)
101 ):
102 if ‘error’ in reply:
–> 103 raise MessageError(reply[‘error’])
104 return reply.get(‘data’, None)
105
MessageError: RangeError: Maximum call stack size exceeded.
Hi, I just tested the notebook and it is working properly. Please run with the example images in the tutorial to make sure your setup is correct.
It seems to exceed some kind of limit. Do you have too many images?
Still doesn’t work. 🙁 ‘No training weights directory found’. Can you help me?
Hi, are you able to run through the example in the post?
I paid the $5 for access but I get an error saying selected gpu is not available, I tried changing the hardware accelerator to none but it doesn’t work. Do I have to pay another $13 to get access??
Sorry, had to refresh the execution to get it working. Hope this helps other people.
It works. I just trained a model.
This is definately one of the HUGE problems for users like me and you when it comes to Colab services, not sure about the others but definately Google seems to upgrade their asses at every chance they get. Ive been at this since christmas and not sure how many times my notebooks crashes for some unknown reason… Heck – learned Python development aswell since then… 😀
Im struggling with the excact same error at the moment, but maby we should just wait since even Colab notebooks from Huggingface crash. Im referring to the official ones.
Hello! Using the colab from the quick start guide ran into this error. Interestingly, it worked maybe two days ago. Calling the realistic visions 2.0 model on branch main from huggingface.
Unsure if the true issue is a problem with the directory as per the traceback, or the TF-TRT warning about not being able to find TensorRT. Look forward to any advice!
===============================
Install xformers
2023-06-07 16:46:56.928864: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /content/train_dreambooth.py:869 in │
│ │
│ 866 │
│ 867 if __name__ == “__main__”: │
│ 868 │ args = parse_args() │
│ ❱ 869 │ main(args) │
│ 870 │
│ │
│ /content/train_dreambooth.py:425 in main │
│ │
│ 422 def main(args): │
│ 423 │ logging_dir = Path(args.output_dir, “0”, args.logging_dir) │
│ 424 │ │
│ ❱ 425 │ accelerator = Accelerator( │
│ 426 │ │ gradient_accumulation_steps=args.gradient_accumulation_steps, │
│ 427 │ │ mixed_precision=args.mixed_precision, │
│ 428 │ │ log_with=”tensorboard”, │
╰──────────────────────────────────────────────────────────────────────────────╯
TypeError: Accelerator.__init__() got an unexpected keyword argument
‘logging_dir’
—————————————————————————
KeyboardInterrupt Traceback (most recent call last)
in ()
121 weightdirs = natsorted(glob(OUTPUT_DIR + os.sep + “*”))
122 if len(weightdirs) == 0:
–> 123 raise KeyboardInterrupt(“No training weights directory found”)
124 WEIGHTS_DIR = weightdirs[-1]
125
================================
I solved it, the latest Accellerator library. So before training add another cell and add this code and run it:
!yes | pip uninstall accelerate
!pip install accelerate==0.19.0
Then everything should work.
Thanks for the pointer! The notebook is now fixed.
Hi, it should be fixed now.
so what I did in the end is instead of saving ckpt as the current notebook suggest, I saved the JSON file into google doc, so that the pre-trained model can be used for future runs.
good to know! adding to my list to change that.
How do I use StableDiffusionPipeline.from_pretraine to load previously trained ckpt file?
In the code, it says to use previously trained model saved in gdrive, replace model_path with the full path of model in gdrive. However, after I did that, StableDiffusionPipeline.from_pretrained is still looking for the json file instead of the ckpt file that we saved on the gdrive. How do I ask StableDiffusionPipeline.from_pretrained to read ckpt?
It is currently not supported. Someone was able to upload a custom checkpoint to HuggingFace and use it.
How can I find the colab notebook?
Hi, get the dreambooth notebook here: https://stable-diffusion-art.com/dreambooth-notebook-guide
Great stuff! Thanks for this. I just wanted to mention that there is a bug and it is fixed by increasing the torch version to torch==2.0.1
Yes, it’s also fixed now.
After that comes:
RuntimeError: Detected that PyTorch and torchvision were compiled with different
CUDA versions. PyTorch has CUDA Version=11.7 and torchvision has CUDA
Version=11.8. Please reinstall the torchvision that matches your PyTorch
install.
—————————————————————————
KeyboardInterrupt Traceback (most recent call last)
in ()
121 weightdirs = natsorted(glob(OUTPUT_DIR + os.sep + “*”))
122 if len(weightdirs) == 0:
–> 123 raise KeyboardInterrupt(“No training weights directory found”)
124 WEIGHTS_DIR = weightdirs[-1]
125
And
RuntimeError: Detected that PyTorch and torchvision were compiled with different
CUDA versions. PyTorch has CUDA Version=11.7 and torchvision has CUDA
Version=11.8. Please reinstall the torchvision that matches your PyTorch
install.
Hey Man just bought a bunch of your stuff – thanks for all the help! I’ve been trying to create a model for 2 days now and getting very frustrated. My current issue is this, RuntimeError: Detected that PyTorch and torchvision were compiled with different
CUDA versions. PyTorch has CUDA Version=11.7 and torchvision has CUDA
Version=11.8. Please reinstall the torchvision that matches your PyTorch
install. | Could you help out with this or advise? btw I have 0 experience in coding.
Appreciate your support!
The bug is now fixed. Please refresh the notebook (May 19 update).
It seems that this exact issue with the PyTorch and CUDA versions is persisting past May 19th as I cannot get it to work for these reasons. Thank you for any help in fixing the colab, I am in the midst of training models for animating a movie and your setup was very helpful for the first one I trained a week ago.
Hi, the notebook is fixed now.
Glad to help!
I get the manual and the notebook from the gumroad link. But the noteook has torch errors.
Building wheel for ffmpy (setup.py) … done
ERROR: pip’s dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchaudio 2.0.1+cu118 requires torch==2.0.0, but you have torch 2.0.1 which is incompatible.
torchvision 0.15.1+cu118 requires torch==2.0.0, but you have torch 2.0.1 which is incompatible.
Install xformers
2023-05-24 11:00:56.220492
Hi, the error is fixed. Please pull the May 24th updated copy.
I managed to train successfully, and the model works fine if I don’t try to do add too many prompt. But the moment I try to change lighting, background, etc in the prompt, the generated image would look nothing like the person I used to train. Is there anything specific that I need to do to the training images?
Awesome tuto, thanks for that.
It runs fine on 1.5 but I tried 2.1 (stabilityai/stable-diffusion-2-1-base), it created the model.
But then when I try to run it on my Mac M1 chip, I get this: Torch not compiled with CUDA enabled
Do you have any idea?
And I just want to emphasize that I use stabilityai/stable-diffusion-2-1-base on my mac without issue (when not fined tuned with dreambooth)
Hi, the error message suggests a problem with your setup because it should not call cuda on mac. But it should run if you can run the 2.1 base model. Do you have a config file for your fine-tuned model? copy the yaml file from 2.1 model and rename to match your new model.
Brilliant, I completely forgot about the yaml, thanks!
Hi, I asked a version of this question before but I think I understand better now what/how to ask: so I want to “sequentially” train a model and currently I have a handful of custom trained models for various friends and family but my goal would be to have a single custom model where I could prompt for anyone/anything I’ve trained it on, and in whatever combinations I want (including all at once like say for a group photo).
So I’m wondering what exactly it is pulling for files etc when you designate a model in the:
“Name/path of the initial model” model name field… I understand it pulls the model from huggingface, but is there a way to modify this collab notebook to have it use a cpkt file that you upload? Or somehow point it to the saved file(s) in Gdrive?
My novice thinking was to perhaps upload the file to my huggingface account so that it would have the same kind of pathway so I could enter that into the model name field, but the only file I even have is the ckpt file… Is that enough? Or are there additional files I would need to upload? And if so, where/what are they and how do I find them? Thanks!
Hi, I will need to see how hard it is to modify to load a model file in G Drive.
Adding to my to-do list…
Thanks Andrew. So I managed to download all the files from the collab notebook in the folder that is at the pathway: “stable diffusion weights—>output—>1800” (1800 I think refers to the steps I used) and then I uploaded all of these to a new model on my own Huggingface account page. I successfully used this now custom trained (on images of my friend) model as the “initial model” with your collab notebook and tried training it a second time with photos of myself. It actually DID work, however, instead of being able to prompt for both unique tokens (myself and my friend), it seemed to combine us both into one … If i used my token prompt word for me, it looked normal, but if I used the one I used for my friend, it spit out a hybrid version of us (but mostly looking like me and not him, but still unmistakably him).
Do you think this is because I used the same class name for both? (person). And is there a way to avoid this going forward? Perhaps only bringing specific json files from the newly trained model and instead using various files from the original base model page I trained from? (In other words, a combination of the new ckpt model and the other files? And if so, which ones? vae? unet? etc? ). Thanks and apologies for my noobishness.
Yes you can try a different class label, man, woman, dude… to try separating you two out.
Another possibility is try to describe TWO persons in the prompt. If you say xyz1 and xyz2 (new dreambooth tokens), SD tends to interpret it as one person but blending the facial features. Say “two men”, “A man and a woman”, etc. to get two people.
Another trick is to use regional prompt to specify your token in one region and your friend’s token in another region.
https://stable-diffusion-art.com/regional-prompter/
Finally, you can always train two models separately, generate two people with the first one, and inpaint the face with the other one.
Let me know how it goes.