How to Use DreamBooth to Fine-Tune Stable Diffusion (Colab)

Screenshot of generating using fine-tune Stable Diffusion
Prompt: borderlands style portrait of Sandman2022, intricate, highly detailed, digital painting, arstation, concept art, smooth, sharp focus, illustration

With Stable Diffusion DreamBooth, you can now create AI art generation images using your own trained images.

For example, you can generate images with yourself or a loved one as a popular video game character, as a fantastical creature, or just about anything you can think of – you can generate a sketch or a painting of your pet as a dragon or as the Emperor of Mankind.

You can also train your own styles and aesthetics like aetherpunk/magicpunk, or maybe people’s facial expressions like Zoolander’s Magnum (I haven’t tried this yet).

In this tutorial, we’ll cover the basics of fine-tuning Stable Diffusion with DreamBooth to generate your own customized images using Google Colab for free. After we’ve tuned Stable Diffusion, we’ll also test it out using Stable Diffusion WebUI built into the same Google Colab notebook.

Stable Diffusion is one of the best AI art generators, which has a free and open-source version that we’ll be using in our tutorial.

Google Colab is a cloud service offered by Google, and it has a generous free tier. That’s what we’ll be using to fine-tune Stable Diffusion, so you don’t need a powerful GPU or any strong hardware for this tutorial.

[powerkit_alert type=”info” dismissible=”false” multiline=”false”]
In machine learning, fine-tuning means adjusting a model that was trained on one dataset to work with a new, related dataset. This can make your model work better on the new dataset or help it work better in a new situation. A dataset, in our case, is a bunch of pictures and some words that tell a machine what it should be looking for in order to generate new images.
[/powerkit_alert]

Quick Video Demos

Training Multiple Subjects on Stable Diffusion 1.5

This is a quick video of me fine-tuning Stable Diffusion with DreamBooth from start to finish. In this example, I’m fine-tuning it using 10 images of the Sandman from The Sandman (TV Series) and 10 images of Aemond Targaryen from House of the Dragon.

The whole process took about 40 minutes on Google Colab with 2000 training steps because I trained 2 subjects (Aemond and Sandman) and changed some settings. If you’re training a face for the first time, then I recommend leaving default settings, which should take you about 15-20 minutes.

The following are two more video demos training a single subject on a model rather than multiple subjects on the same model. This is because when training multiple similar subjects on the same model, like faces of the same gender, they might get blended together.

Training Aemond Targaryen on Stable Diffusion 1.5

Training Sandman on Stable Diffusion 2.1

tl;dr AI News Section Preview
tl;dr AI News Section Preview

Sidenote: AI art tools are developing so fast it’s hard to keep up.

We set up a newsletter called tl;dr AI News.

In this newsletter we distill the information that’s most valuable to you into a quick read to save you time. We cover the latest news and tutorials in the AI art world on a daily basis, so that you can stay up-to-date with the latest developments.

Check tl;dr AI News

Use DreamBooth to Fine-Tune Stable Diffusion in Google Colab

Prepare Images

Choosing Images

When choosing images, it’s recommended to keep the following in mind to get the best results:

  1. Upload a variety of images of your subject. If you’re uploading images of a person, try something like 70% close-ups, 20% from the chest up, 10% full body, so Stable Diffusion also gets some idea of the rest of the subject and not only the face.
  2. Try to change things up as much as possible in each picture. This means:
    • Varying the body pose
    • Taking pictures on different days, in different lighting conditions, and with different backgrounds
    • Showing a variety of expressions and emotions
  3. When generating new images, whatever you capture will be over-represented. For example, if you take multiple pictures with the same green field behind you, it’s likely that the generated images of you will also contain the green field, even if you want a dystopic background. This can apply to anything, like jewelry, clothes, or even people in the background. If you want to avoid seeing that element in your generated image, make sure not to repeat it in every shot. On the other hand, if you want it in the generated images, make sure it’s in your pictures more often.
  4. It’s recommended that you provide 10 images of what you’d like to train Stable Diffusion on to get great results when training on faces.
  5. A note on training multiple subjects:
    • Training multiple subjects of the same gender on the same model is very likely to lead to blending between them. You may notice Sandman having one eye a bit different, which he “inherits” from Aemond’s eyepatch.
    • To mitigate the blending of multiple subjects, the author of the notebook (TheLastBen) recommended using UNet_Learning_Rate: 2e-6 instead of the default 5e-6. However, he recommends training a subject on a separate model to get the best results.

Resize & Crop to 512 x 512px

Once you’ve chosen your images, you should prepare them.

First, we need to resize and crop our images to be 512 x 512px. We can easily do this using the website https://birme.net.

To do this, just:

  1. Visit the website
  2. Upload your images
  3. Set your dimensions to 512 x 512px
  4. Adjust the cropping area to center your subject
  5. Click on Save as Zip to download the archive.
  6. You can then unzip it on your computer, and we’ll use them a bit later.
Birme.net - Resize Images
Resizing Images using Birme.net

Renaming Your Images

We’ll also want to rename our images to contain the subject’s name:

  1. Firstly, the subject name should be one unique/random/unknown keyword. This is because Stable Diffusion also has some knowledge of The Sandman from other sources other than the one played by Tom Sturridge, and we don’t want it to get confused and make a combination of interpretations of The Sandman. As such, I’ll call it Sandman2022 to make sure it’s unique.
  2. Renaming images to subject (1), subject (2) .. subject (30). This is because, using this method, you can train multiple subjects at once. If you want to fine-tune Stable Diffusion with Sandman, your friend Kevin, and your cat, you can give it prepared images for each of them. For the Sandman, you’d have Sandman2022 (1), Sandman2022 (2) … Sandman (30), for Kevin, you’d have KevinKevinson2022 (1)KevinKevinson2022 (2) … KevinKevinson (30), and for your cat you’d have DexterTheCat (1), DexterTheCat (2) … DexterTheCat(30).

Here’s me renaming my images for Sandman2022 in bulk on Windows. Just select them all, right-click one of them and click Rename and give it what name you want, and click anywhere to finish the renaming. Everything else will be renamed as well.

Bulk Rename Windows - Sandman2022
Bulk Renaming Images on Windows

When it’s time to upload my images to DreamBooth, I’ll want to train it for Sandman2022 and AemondHoD, and this is how my images will look like:

Ready to Train Sandman and Aemond

Open Fast Stable Diffusion DreamBooth Notebook in Google Colab

Next, we’ll open the Fast Stable Diffusion DreamBooth Colab notebook: https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast-DreamBooth.ipynb

[powerkit_alert type=”info” dismissible=”false” multiline=”false”]
The notebook we’re using is an implementation of DreamBooth by TheLastBen. The author optimized multiple implementations of Stable Diffusion. You can check the GitHub repository here.
[/powerkit_alert]

Fast DreamBooth Notebook Preview
Fast DreamBooth Notebook Preview

Enable GPU

Before running the notebook, we’ll first have to make sure Google Colab is using a GPU. This is because GPUs can process much more data than CPUs and allows you to train our machine learning models faster.

To do this:

  1. In the menu, go to Runtime > Change runtime type.

    Runtime > Change runtime type
    Runtime > Change runtime type
  2. A small popup will appear. Under the Hardware accelerator, make sure you have selected GPU. Click Save when you’re done.

    Hardware Accelerator > GPU
    Hardware Accelerator > GPU

Run First Cell to Connect Google Drive

By running the first cell, we’ll start connecting our notebook to Google Drive so we can save all of our files in it – this includes the Stable Diffusion DreamBooth files, our fine-tuned models, and our generated images.

After running the first cell, we’ll see a popup asking us if we really want to Connect to Google Drive.

After we click it, we’ll see another popup where we can select the account we want to connect with and then allow Google Colab some permissions to our Google Drive.

Mount Google Drive
Mount Google Drive

Run the Second Cell to Install Dependencies

Next, just run the second cell. There’s nothing for us to do there except wait for it to finish.

Install Dependencies
Install Dependencies

Run the Third Cell to Download Stable Diffusion

Next, we’ll want to download our Stable Diffusion model. This is the base model that we’ll train. You’ll notice that there are 3 default models available when clicking the Model_Version dropdown:

  • 1.5: This is still the most popular model. It knows many artists and styles and is the easiest to play with. Most custom models you’ll find are still trained on 1.5 at the time of writing.
  • V2.1-512px: V2.1 is the latest base model available. It excels at photorealism, but it’s more difficult to use, and it’s had a lot of artists and NSFW content taken out. If you know how to use it, you can get some excellent results. 512px means that it can generate 512px images, which you can then upscale to a higher resolution. I recommend checking out this Reddit thread discussing 2.1 vs 1.5 to get a better idea of the differences between them.
  • V2.1-768px: This is similar to the V2.1-512px, however, it can generate 768px images. This means that they’ll have more detail than 512px ones. This will require more RAM, however. (This model hasn’t worked for me on Google Colab free)

If you just want the base model, just select one of the versions from the dropdown and run the cell.

If you want a custom model, then you’ll need to choose the custom model version and provide the path to it.

Model Download
Model Download

Custom Models:

    • Custom_Model_Version: Custom models are also trained from base models. For the notebook to know the algorithm to use when training the model later on, it will need the model version it’s trained on (1.5, V2.1-512px, V2.1-768px). Make sure to get this right.
    • Path_to_HuggingFace: If you want to load and train over a different model from Hugging Face than the default one, you can provide the path to it. For example, if you want to train Stable Diffusion to generate pictures of your face but in Elden Ring style, you could get this already fine-tuned model https://huggingface.co/nitrosocke/elden-ring-diffusion. The path you should provide is what comes after huggingface.co. In our case, that’s nitrosocke/elden-ring-diffusion
    • CKPT_Path or CKPT_Link: If you already have an existing Stable Diffusion model that you’d like to fine-tune, you can provide the path to it in  CKPT_Path instead of the HuggingFace Token. Alternatively, if you have a Stable Diffusion model, whether it’s a link to any online .ckpt file or if it’s a shareable Google Drive Link, you can input it in the CKPT_Link field.

Setting Up Dreambooth

We can now get to setting up DreamBooth.

Here, we’ll input our Session_Name. This will be the name of the trained model that we’ll save. This is where you’ll input previous sessions to load them, should you want to fine-tune them further. It can be anything you want.

Important: Don’t use spaces in the session name. Instead, use _ or -.

Run the cell after you input the session name.

Create/Load Session
Create/Load Session

Notes:

  • Session_Name: This will be the name of your session and of your final model. You can name it anything. If you provide a name that doesn’t exist it will create a new session and if you use a name of a session that already exists in your Google Drive in My Drive > Fast-DreamBooth > Sessions then it will ask you whether you want to overwrite it or resume training it.
  • Session_Link_optional: Instead of providing the Session_Name you can provide the path to the session. For example, the path to mine will be /content/gdrive/MyDrive/Fast-Dreambooth/Sessions/Aemond_Sandman.
  • Model_Version: If you’re loading a previous session with a trained model, then select the Stable Diffusion version of that model.

Upload Your Instance Images

Next, you’ll see the Instance Images cell. This is where we upload our images.

If you run it Choose Files button will appear, allowing you to upload images.

Instance Images
Upload Instance Images

Additional Options:

  • Remove_existing_instance_images: If you already uploaded some images but want to remove them to run the cell and upload other images again, then that’s what Remove_existing_instance_images is for. If you want to keep the previously uploaded images, then uncheck that box.
  • IMAGES_FOLDER_OPTIONAL: If you have a folder on your Google Drive that already contains your images, then just provide the path to it and then run the cell, instead of uploading the images from your computer.
  • Crop_images: Check this if you haven’t already cropped them yourself. They’ll be cropped in squares, and you can set the crop size yourself. It’s left to 512 by default because 512 x 512 px is the usual image dimension used.

Start DreamBooth

Finally, we can run DreamBooth. We have a few configurations here.

The Training_Steps are what we care most about.

  • Training_Steps: The most important thing we can do here is set the training steps. We’ll want to set the total number of images we’ve uploaded multiplied by 100. I uploaded 10 images of Aemond and 10 images of the Sandman, so that’s a total of 20 * 100 = 2000. So that’s 2000 steps. If the model isn’t good enough, you can pick up where you left off and further train it.
Start DreamBooth Configurations
Start DreamBooth Configurations

You most likely won’t have to touch these options if this is your first time, but we’ll still explain them just in case:

  • Resume_Training: You’ll check this box if you want to continue training the model after you’ve tested it.
  • UNet_Training_Steps: These are the training steps (also referred to as iterations or epochs) that the model undergoes during the training process. Each step involves passing images through the model, computing the loss, and updating the model’s weights to improve its performance.
    • UNet is a popular neural network architecture that Stable Diffusion uses to identify the different components within an image. A neural network architecture is like the blueprint that tells the computer how to connect all its different parts so it can learn how to draw the lines correctly.
  • UNet_Learning_Rate: The learning rate is how quickly a model learns. More on learning rate below.
  • Text_Encoder_Training_Steps: The training steps for the text encoder. More on what the text encoder is.
  • Text_Encoder_Concept_Training_Steps: This is if you’re using concept images. Read about what concept images are below.
  • Text_Encoder_Learning_Rate: The learning rate for the text encoder. More on the learning rate below.
  • External_Captions: To use external captions for images just write a .txt file with the same name as the corresponding image, and upload the .txt files along with the images. Read here about captions here.
  • Style_Training: This is for when you’re training styles. I’m not sure what different it makes yet. Will update when I do.
  • Resolution: The resolution of the images you’re training the model on. I have only tested with 512px.
  • Save_Checkpoint_Every_n_Steps: This option enables you to save the fine-tuned model at different points during the training. This is useful if you suspect that the number of steps you’ve set is too many, and it will overtrain Stable Diffusion. Overtraining is where the model will end up generating almost exactly the pictures you gave it, which means it won’t generate original work.
    • Save_Checkpoint_Every: the model will save the model each X steps you set. If you set it to 500 it will save it at 500, then 1000, then 1500 and so on.
    • Start_saving_from_step: this is the minimum amount of steps from which DreamBooth will start to save the model.
  • Disconnect_after_training: You may want to leave it to train the model and forget about it. Google Colab free might not let you use a GPU for the rest of the day if you use it for too long, and Colab Pro uses up units. If you know you won’t be around when it’s done training, then best check this box.

Finally, you can run the cell when you’re done with the options.

This can take a while. In my case, 2000 steps took about 35 minutes with Google Colab free, using an Nvidia Tesla T4 GPU and decreased learning rate. If you’re training only one subject with default settings, then it should take you about 15-20 minutes.

Started Training
Start Training Dreambooth

Where Your New Model is Stored

When it’s done, you should find your model in your Google Drive. For example, here’s where Aemond_Sandman.ckpt was saved with default output folder settings. This should be My Drive > Fast-DreamBooth > Sessions > Your_Session_Name.

Default Save Directory in Google Drive
Default Save Directory in Google Drive

Test the Trained Model (with Stable Diffusion WebUI by AUTOMATIC1111)

After the training cell has finished running, we can test our new fine-tuned Stable Diffusion model.

Test The Trained Model
Test The Trained Model

This notebook comes with Stable Diffusion WebUI by AUTOMATIC1111, which is the most popular implementation of Stable Diffusion, and offers us a very convenient web user interface.

We have a few options:

  1. If you have just fine-tuned Stable Diffusion for the first time (this is us, most likely) and want to test your newly created model, then just run the Test the trained model cell. No need to fill out anything.
  2. Previous_Session: If you have previously fine-tuned a different Stable Diffusion model that you want to test, then insert the Previous_Session. Since I’m assuming this is our first time, you can leave this empty.
  3. Use_Custom_Path: If you have a model you want to load that’s in some folder in your Google Drive, then check Use_Custom_Path , and after you run the cell, you’ll see a field to provide the path to your model.
  4. You can leave Use_localtunnel unchecked. This is how our link will be generated for us to access the Stable Diffusion WebUI. When unchecked, it uses the servers of Gradio.app to generate a URL, as you see in the screenshot above. If it’s checked, then it uses a service called localtunnel. We have both these options available in case one of them doesn’t work.

When you run the cell, it will take about 5 minutes for Stable Diffusion WebUI to be ready to use. When it’s done, you’ll see some URLs like https://31767n39-6e5b-46fe.gradio.live when Use_localtunnel is left unchecked, and https://fancy-spies-punch-34-150-175-108.loca.lt when leaving it’s checked.

Click it, and then you can start using Stable Diffusion to generate our images. The user interface will open in a new tab, and you can start generating images right away.

Generate Images
Generate Images

Where Generated Images Are Stored

Images are stored by default in your Google Drive in My Drive > sd > stable-diffusion > outputs > txt2img-images.

Where Generated Images Are Stored
Where Generated Images Are Stored

Upload Your Trained Model to Hugging Face

You can also upload your trained model to Hugging Face, to the public library or just have it privately in your account. You can change it to public later on.

To do this, you’ll have to use a Hugging Face token that has WRITE role. Simply create a token and set the role to WRITE.

Hugging Face Token Write Permissions
Hugging Face Token Write Permissions

In the Upload The Trained Model to Hugging Face cell, you’ll have the following:

  1. name_of_your_concept: This will be the name of the model you upload, and it will show up in your URL as well. For example, https://huggingface.co/ByteXD/aemondhod-sandman2022.
  2. Save_concept_to: You can select between your profile (by default, it will be set to private) or the public library.
  3. hf_token_write: This is your Hugging Face token. Make sure it has its’ role set to WRITE.

Run the cell when you’re done. It will take ~10-15 minutes to finish uploading. When it’s done, it will output a link to where you can access your newly uploaded model.

Upload Model to Hugging Face 2
Finished Uploading the Model to Hugging Face

This is what it looks like: https://huggingface.co/ByteXD/aemondhod-sandman2022-feb2023. To make it publicly accessible, go in the model’s settings section and click on Make this model public.

Aemond-Sandman-Uploaded-HuggingFace
Aemond Sandman Model Page

FAQ

What is a .ckpt file?

The .ckpt file extension is commonly used for checkpoint files, and the file is also referred to as the weights of the model. Although not exactly accurate, we can think of it as the model file. Checkpoint files are used to save the state of a program or process at a particular point in time.

What is the learning rate, and how does it affect generated images?

The “learning rate” is a tuning parameter that determines how quickly a model learns from the data during training.

Imagine you are trying to learn a new skill, such as playing the piano. If you practice too slowly, you may not make much progress, and it may take a long time to learn. On the other hand, if you practice too quickly, you may make mistakes and not really learn the skill well. The learning rate in machine learning is similar – it controls how quickly the model learns from the data.

A high learning rate means that the model will learn more quickly from the data, but it may also make more mistakes and not learn the best representation of the data. Conversely, a low learning rate means that the model will learn more slowly but may be more accurate in the long run.

A lower learning rate is not always better. In some cases, it may be better to have a higher learning rate and more steps, for example.

A learning rate that’s too low, along with too many training steps, may lead to overfitting, and you may not generate creative images and generate images too similar to those that you trained Stable Diffusion on. And a learning rate that’s too high might lead to images that are not as accurate or realistic.

To find the best learning rate for UNet and text encoder, I recommend checking what others are saying from their experiments or experimenting yourself.

What is the text encoder in DreamBooth?

The text encoder translates (encodes) the prompt into a language the model understands, and fine-tuning it on specific images will make the model converge faster to their likeness. (quoting the TheLastBen)

What are concept images in DreamBooth?

Prior preservation loss proved as a weak method for regularizing stability’s model, I implemented concept images to replace class images, and they act as heavy regularization. They force the text encoder to widen its range of diversity after getting narrowed by the instance images.

Example: you use 10 instance pictures of a specific car in a garage using the identifier “crrrr“, without concept images. At inference, most of the output when you use the term “crrrr” or even car will be a car inside a garage, because the terms car and garage are overwritten by the embeddings from the instance images when training the text encoder.

Now you start over with the same instance images, but this time you add 200 concept images of different cars in different places, you set the text_encoder_concept_steps to 1500 steps, and now the text encoder receives more information about the token “car” and able to put it in different situations. At inference you will get relatively diverse situations for the car whilst keeping the same characteristics of the trained car.

TL;DR: Concept images are like class images but during training, they are treated as instance images without including the identifier and trained only to the text encoder to help with diversity and variety.

The filename doesn’t matter for concept images or the resolution. All that matters is the content of the image.

(quoting the TheLastBen)

Troubleshooting

In this section, we’ll address some common errors.

ModuleNotFoundError: No module named ‘modules.hypernetworks’

As of writing this, this error should be fixed. If you’re encountering it then we’ll want to do a clean run of the DreamBooth notebook. To do this:

  1. Delete your sd folder in your Google Drive, located at My Drive > sd.
    • Important – Back Up Previously Generated Images: If you previously generated images, back them up. They are located in My Drive > sd > stable-diffusion > outputs. They’ll be deleted if you don’t back them up.
  2. Then make sure you’re running the latest Colab notebook (in case you were running one saved to your Google Drive) https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast-DreamBooth.ipynb.

That’s it. Now it should work. We just had to do a fresh run of the DreamBooth notebook without the previously stored files.

Next, if you were interrupted before getting to train your model, you can continue with the instructions below.

ImportError: cannot import name ‘VectorQuantizer2’

To fix this:

  1. Run the following command in any cell:
    !pip install taming-transformers-rom1504
  2. Then restart the runtime by going in the menu to Runtime > Restart runtime.
  3. Now, if you run everything again, it should work.

Next, if you have already fine-tuned your model, to get back to testing it quickly, follow the instructions below.

If You Just Trained a Model but Didn’t Get to Test It Because of an Error

If you got an error in the last cell Test the trained model , fixed it, and now restarted the notebook, you don’t have to go through the training again.

  1. Just run the first two cells (Connect to Google Drive and Setting up the environment)
  2. After that’s done, go to the Test the trained model cell and just insert your INSTANCE_NAME from earlier (Sandman2022 in my case) or Use_Custom_Path (if you have it somewhere else other than My Drive), and run the cell and it should work. When you trained the model earlier, it got saved in your Google Drive, so now the notebook will just load your fine-tuned Stable Diffusion model.

Conclusion

In this tutorial, we covered how to fine-tune Stable Diffusion using DreamBooth via Google Colab for free to generate our own unique image styles. We hope this tutorial helped you break the ice in fine-tuning Stable Diffusion.

If you encounter any issues or have any questions, please feel free to leave a comment, and we’ll get back to you as soon as possible.

Very Useful Resources

  • A Discord Server for Stable Diffusion DreamBooth – This is a Discord community dedicated to experimenting with DreamBooth. It can be of great help in better understanding how to use DreamBooth to get desired results, such as how to better design prompts and other troubleshooting tips, so I highly recommend it! Also, fun fact: one of the moderators is Joe Penna (Mystery Guitar Man), who many might know from his very popular YouTube channel years ago.
  • Lexica.art, OpenArt.ai, Krea.ai – these are search engines for Stable Diffusion prompts, along with their resulting images. They are fantastic for inspiration, and you can easily get some great results by using them.
  • Civitai.com – this is a hub where you can find/share Stable Diffusion models.
  • The Wiki for AUTOMATIC1111’s Stable Diffusion WebUI – The web interface we’re using when generating images has many features, as you might have seen. This wiki is the documentation for all those features, and it gives us a great overview of how to use the interface. I highly recommend you check it out.
0 Shares:
Subscribe
Notify of
guest
Receive notifications when your comment receives a reply. (Optional)
Your username will link to your website. (Optional)

131 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Barry
Barry
1 year ago

Thank you very much for this

Dave
Dave
1 year ago

getting this error when running “test the trained model”. it was working yesterday

Traceback (most recent call last):
File “/content/gdrive/MyDrive/sd/stable-diffusion-webui/webui.py”, line 32, in <module>
import modules.hypernetworks.hypernetwork
ModuleNotFoundError: No module named ‘modules.hypernetworks’

Curious about AI
Curious about AI
1 year ago

I only have a 3080 10 GB can I run this? The other options don’t work for me because I have very little space on my C: drive and can’t install WSL and python and all these other choices. It doesn’t seem to say what the system requirements are here and seems to work magically from your example.

Skunk
Skunk
1 year ago

How can I continue training a ckpt file I already created?
Where would I put it or transform it into the files I need?
(In the collab)
thank you!

Steven
Steven
1 year ago

Hey there. At the end of the training, I get an error. It simply says “something went wrong’

Here’s the last few lines of code: Any idea what might have went wrong and how to fix it?

Steven
Steven
1 year ago

Hey there. At the end of the training, I get an error. It simply says “something went wrong’
Here’s the last few lines of code:File “/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py”, line 354, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command ‘[‘/usr/bin/python3’, ‘/content/diffusers/examples/dreambooth/train_dreambooth.py’, ‘–train_text_encoder’, ‘–pretrained_model_name_or_path=/content/stable-diffusion-v1-5’, ‘–instance_data_dir=/content/data/SteveJP’, ‘–class_data_dir=/content/data/Man’, ‘–output_dir=/content/models/SteveJP’, ‘–with_prior_preservation’, ‘–prior_loss_weight=1.0’, ‘–instance_prompt=photo of SteveJP Man’, ‘–class_prompt=a photo of a Man, ultra detailed’, ‘–seed=45576’, ‘–resolution=512’, ‘–mixed_precision=no’, ‘–train_batch_size=1’, ‘–gradient_accumulation_steps=1’, ‘–gradient_checkpointing’, ‘–use_8bit_adam’, ‘–learning_rate=2e-6’, ‘–lr_scheduler=constant’, ‘–lr_warmup_steps=0’, ‘–center_crop’, ‘–max_train_steps=1500’, ‘–num_class_images=200′]’ returned non-zero exit status 1.
Something went wrong

Javier
Javier
1 year ago
Reply to  EdXD

Hi, I have the same problem

Mihai
Mihai
1 year ago
Reply to  EdXD

Same problem here

Guadatonix
Guadatonix
1 year ago
Reply to  EdXD

Same problem here, it happens just after clicking “run”

Leo
Leo
1 year ago

Hi, I’m trying to test a previously trained model, but keep getting the error: ImportError: cannot import name ‘script_callbacks’ from ‘modules’ (unknown location)

Konstantin
Konstantin
1 year ago

Hi. Great article, thank you.
How to bypass the error “ImportError: cannot import name ‘script_callbacks’ from ‘modules’ (unknown location)” when testing the trained model?

Konstantin
Konstantin
1 year ago
Reply to  EdXD

This probably will work. Now I have the issue that I cannot connect to a GPU backend due to usage limitations.
I can connect without GPU, but I suppose it won’t work, correct?

Leo
Leo
1 year ago

figured it out, you have to check the box “update repo” to successfully run from drive

elmar8
elmar8
1 year ago

Hi Im a total newbe , I made my pictures, but where and how do I upload the images?

Mustaf
Mustaf
1 year ago

I’m getting the following error with a saving regimen of every 500 steps starting at step 500. I’ve successfully trained models previously, so I don’t know what I’ve done:

Steps: 2% 499/27200 [08:09<7:14:05, 1.03it/s, loss=0.0131, lr=1.97e-6] SAVING CHECKPOINT: /content/gdrive/MyDrive/mustaf_mv_session_1_step_500.ckpt
Traceback (most recent call last):
File “/content/diffusers/examples/dreambooth/train_dreambooth.py”, line 733, in <module>
main()
File “/content/diffusers/examples/dreambooth/train_dreambooth.py”, line 701, in main
if args.train_text_encoder and os.path.exists(frz_dir):
UnboundLocalError: local variable ‘frz_dir’ referenced before assignment
Steps: 2% 499/27200 [08:38<7:42:06, 1.04s/it, loss=0.0131, lr=1.97e-6]
Traceback (most recent call last):
File “/usr/local/bin/accelerate”, line 8, in <module>
sys.exit(main())
File “/usr/local/lib/python3.7/dist-packages/accelerate/commands/accelerate_cli.py”, line 43, in main
args.func(args)
File “/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py”, line 837, in launch_command
simple_launcher(args)
File “/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py”, line 354, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command ‘[‘/usr/bin/python3’, ‘/content/diffusers/examples/dreambooth/train_dreambooth.py’, ‘–image_captions_filename’, ‘–train_text_encoder’, ‘–save_starting_step=500’, ‘–stop_text_encoder_training=2720’, ‘–save_n_steps=500’, ‘–pretrained_model_name_or_path=/content/stable-diffusion-v1-5’, ‘–instance_data_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/mustaf_mv_session_1/instance_images’, ‘–output_dir=/content/models/mustaf_mv_session_1’, ‘–instance_prompt=’, ‘–seed=96576’, ‘–resolution=512’, ‘–mixed_precision=fp16’, ‘–train_batch_size=1’, ‘–gradient_accumulation_steps=1’, ‘–use_8bit_adam’, ‘–learning_rate=2e-6’, ‘–lr_scheduler=polynomial’, ‘–center_crop’, ‘–lr_warmup_steps=0’, ‘–max_train_steps=27200′]’ returned non-zero exit status 1.
Something went wrong

andre
andre
1 year ago

I have trained two models and want to use them in checkpoint merger. But only one of them is in the list. What should I do to make both models appear in the list? Unfortunately in the video I did not understand how to do it. Thank you very much for your work.

Edward Jonah
1 year ago

It works just great. But how do I name the files if I want to teach SD not a face, but an artist’s style, which is not in the dataset?

Steve
Steve
1 year ago

Hi

I’m getting this error when I try to test the model – everything up to this point seemed to work fine

Warning: Taming Transformers not found at path /content/gdrive/MyDrive/sd/stable-diffusion/src/taming-transformers/taming”

Help!!!

steve
steve
1 year ago
Reply to  EdXD

Thanks for your help, everything is working great now thanks, quick question, can you combine 2 different subjects in one image using this method? sandman and the other guy for example

steve
steve
1 year ago
Reply to  EdXD

that looks awesome thanks!! I was thinking more along the lines of having both subjects in the same image, it just combines them when i try

Steve
Steve
1 year ago
Reply to  EdXD

yeah same, its inpredictable, sometimes you get a great image but for every great one there are maybe 6 bad. I’ve tried tips from different forums but you cant bank on it. If you have hours to kill you’ll get some good ones, occasionally

Salva Robles
Salva Robles
1 year ago

Hi, great tutorial. I got an error whe testing this with EMA version of the model. How that could be solved. Thanks.

Qain
Qain
1 year ago

Hi there! Big question.

Once I have my model ready and trained (I’m currently setting up a new one with over 100 pictures for reference), can I use my local stable diffusion webUI client with my custom model? If so, how?

I’m asking because, if I can, I’d rather leave the free space for someone who wants to use Collab for themselves.

Konstantin
Konstantin
1 year ago
Reply to  EdXD

Hi,
I am able to download the Model (cpkt-file) from my G-Drive to my computer, but I do not have (or don’t find the Stable Diffusion folder on my computer (Mac).

And also, I don’t know how to get the local file in the StableDiffusion Web UI – the dropdown only contains a single entry and no way to link/open a local model…

Konstantin
Konstantin
1 year ago
Reply to  EdXD

This was probably a misunderstanding of mine. I thought you could use to local web-client also locally/without being connected to a runtime.
I have downloaded a Stable Diffusion Mac App from https://diffusionbee.com/download and it works quite nicely so far.

Kiroishi
Kiroishi
1 year ago

hi, thanks for the guide. everything works. but I have encountered such a problem. after some time, the resulting web UI throws a 404 error. How can I fix it?

Kiroishi
Kiroishi
1 year ago
Reply to  EdXD

I didn’t check the box. I’ll try with him (as I understand it). Thanks for the answer

infra
infra
1 year ago

Thanks for this detailed guide!

I seem to struggle to train a subject with the latest version that has ‘Enable_text_encoder_training’ checkbox and a percentage value below.

I have tried repeatedly with different values for ‘Train_text_encoder_for’ ranging from 10 to 100 and I keep getting bad results.

I can’t tell whether I should train the model further (already trained for 3000 steps with 16 images).

The results either don’t look like the original subject or it’s impossible to stylize. In no case it is accurate.

Worth noting that I have previously managed to train 3 different models/subjects with 10-14 images each and the results were stellar!

Evangelina
1 year ago

Thanks guys!

charles1212
charles1212
1 year ago

Hi I run the “Downloading the model” then have error, how to fix??

error.png
charles1212
charles1212
1 year ago
Reply to  EdXD

hey, It’s work! can continue the following step

charles1212
charles1212
1 year ago

one more question: when the section off, reconnect will resume to default, need run again those step, can save to google driver?

charles1212
charles1212
1 year ago
Reply to  charles1212

like this, I have save each 500 step, but it can’t find previous setting

error.png
charles1212
charles1212
1 year ago
Reply to  EdXD

after finish the training will creat a .ckpt file, later can i load the file again to another account or for later test?

charles1212
charles1212
1 year ago

Hi, I already trained model, success load seccion, but can’t creat WebUI link, how to fix it?

error.png
charles1212
charles1212
1 year ago
Reply to  EdXD

Hi, now can load the webui, but have some different show in the picture, don’t have the progress bar and just run back to google.colab.
is that the This share link expires in 72 hours?

error.png
Jonathan Vaneyck
Jonathan Vaneyck
1 year ago

Hi Thanks very much for this tuto!

I got a problem when in the image upload step: when I select the images and click the upload button, I got an error “MessageError: RangeError: Maximum call stack size exceeded.” .

Do you have an idea how I can fix it?

Thanks in advance.

Jonathan Vaneyck

charles1212
charles1212
1 year ago

Hi, you go the the link https://birme.net and resize picture to 512 x 512px?

Jonathan Vaneyck
Jonathan Vaneyck
1 year ago
Reply to  charles1212

Yes I have used this website to resize all my pictures to 512×512 px. In addition, I have formatted the name as described in the tuto.

Jonathan Vaneyck
Jonathan Vaneyck
1 year ago

I have found the issue… it is the weight of the image. If the image exceeds 100 kb, I get the error. So by decreasing the weight, the upload succeed.

Max
Max
1 year ago

Thanks! >50kb in my case

Ethan
Ethan
1 year ago

i keep getting this issue when trying to start the Training any idea how to fix?

image_2022-11-20_164312029.png
Huzaifa
Huzaifa
8 months ago
Reply to  EdXD

sir I got the same error How do i fix this?

Huzaifa
Huzaifa
1 year ago

Hi. This is a awesome tutorial. Can you please make a tutorial on SD 2.0 with Google Colab in Automatic1111?

EVE
EVE
1 year ago

Hi, thanks for the amazing tutorial!
I’m getting the following error when I try to run webui.py.
Any idea why?
I just run the entire notebook with no changes.
Can it be about the images?

Thanks!!

Traceback (most recent call last):
 File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/webui.py", line 13, in <module>
   from modules import shared, devices, sd_samplers, upscaler, extensions, localization, ui_tempdir
 File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/sd_samplers.py", line 11, in <module>
   from modules import prompt_parser, devices, processing, images
 File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/processing.py", line 15, in <module>
   import modules.sd_hijack
 File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modulesdafsd/sd_hijack.py", line 31, in <module>
   ldm.modules.attention.BasicTransformerBlock.ATTENTION_MODES["softmax-xformers"] = ldm.modules.attention.CrossAttention
AttributeError: type object 'BasicTransformerBlock' has no attribute 'ATTENTION_MODES'
Alan Benitez
Alan Benitez
1 year ago
Reply to  EdXD

I am having the same error. And after the update, it still does not work. Do you know can I fix it? Thanks in advance

Jonny
Jonny
1 year ago
Reply to  Alan Benitez

Me too!

Jonny
Jonny
1 year ago
Reply to  EdXD

When deleting sd (or disconnecting/reconnecting and refreshing the page) I’m now able to run the cell, but no option to run the server is presented. Instead I see the following:

LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 865.91 M params.
Downloading100% 3.94G/3.94G [01:37<00:00, 40.5MB/s]
^C

Last edited 1 year ago by EdXD
Jonny
Jonny
1 year ago
Reply to  Jonny

LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 865.91 M params.
Downloading100% 3.94G/3.94G [01:37<00:00, 40.5MB/s]
^C

Jonny
Jonny
1 year ago
Reply to  Jonny

I now get this error when creating new sessions. I get the feeling TheLastBen is making lots of changes right now for SD2 🙂 https://github.com/TheLastBen/fast-stable-diffusion/commits/main

Untitled 13.png
Jonny
Jonny
1 year ago
Reply to  EdXD

Yes SD 2.0. I tried 1.5 again to check, and got a different error (: Will try to recreate it so I can share here

Jonny
Jonny
1 year ago
Reply to  Jonny

I’m now following this to get to this colab notebook, but again when starting the gradio app I run the cell but nothing appears, see image below

Untitled 14.png
Jonny
Jonny
1 year ago
Reply to  Jonny

I think this is a TheLastBen issue as it seemed to work fine a couple of days ago, and there have been no commits merged in AUTOMATIC’s repo in this time. So I’ve created an issues here https://github.com/TheLastBen/fast-stable-diffusion/issues/824

Jonny
Jonny
1 year ago
Reply to  Jonny

Ah there is another issue posted for this here https://github.com/TheLastBen/fast-stable-diffusion/issues/794

Eduardo
Eduardo
1 year ago
Reply to  EdXD

Fixed deleting sd folder (or in my case renaming). Thanks!

Omidreza
1 year ago

Hi, Thank you very much for the excellent tutorial, I get an error when fine-tuning using SD 2 and trying to test the trained model as below:

File “<ipython-input-10-b1a5bc118eb8>”, line 187
NM==”True”:
^
SyntaxError: invalid syntax

any idea what the problem might be?

charles1212
charles1212
1 year ago

Hi, one question ask:
I create a training model run 80000 step and I save a checkpoint at 40000 step, can resume training at 40001? is choose the “Resume_Training:“?

error.png
charles1212
charles1212
1 year ago
Reply to  EdXD

I choose resume training, but it still form step 1 to train…..
(since I use 850 pictures, so at least X100)

charles1212
charles1212
1 year ago
Reply to  EdXD

just two people…..can reduce train step, for example 850×50
the photo show the GPU machine stop run 🙁

train.PNG
Henri
Henri
1 year ago

Hi ! Super usefull content 🙂 Thank you very much.
I’m trying to customize Shoes from this brand shoes53045.com.

I gave around 40 pictures, and the results are really impressive.

However, it’s hard to have very creative results. Prompt is almost no taken into account.

Is there something to set up before the TRAINING ?

Thank’s

Henri

Thyhack
Thyhack
1 year ago

[duplicate comment]

Last edited 1 year ago by EdXD
Thyhack
Thyhack
1 year ago

I created a python script to rename all files within a directory. Because renaming files manually sounds like hell.

import os
import re


#set the directory
os.chdir('/home/PATH/Pictures/PATH')
#list the files in the directory
files = os.listdir('/home/PATH/Pictures/PATH')


#loop through the files
for file in files:
    #find the number in the file name
    num = re.findall(r'\d+', file)
    #convert the number to an integer
    num = int(num[0])
    #increment the number by 1
    num = num + 1
    #convert the number back to a string
    num = str(num)
    #find the extension
    ext = re.findall(r'\.\w+', file)
    #create the new file name
    newfile = 'XFILENAMEX (' + num + ')' + ext[0]
    #rename the file
    os.rename(file, newfile)
olo
olo
1 year ago

Hello! I have trained 2 models some weeks ago and everything went fine, but now when I trying to train, I get errors (..json.decoder.JSONDecodeError: Expecting ‘,’ delimiter: line 12 column 3 (char 284)). the code on fast-DreamBooth has change from last time I used it.
Anybody know what i´m doing wrong?
thx

Screen Shot 2022-12-16 at 02.30.17.png
yqq
yqq
1 year ago
Reply to  olo

have you solved the problem?

yqq
yqq
1 year ago
Reply to  olo

met the same question

yqq
yqq
1 year ago
Reply to  olo

follow this:https://github.com/TheLastBen/fast-stable-diffusion/issues/997

Iliya
Iliya
1 year ago

Something went wrong. Could you explain the solution in more detail?

Ron
Ron
1 year ago

A Discord Server for Stable Diffusion DreamBooth – This is a Discord community dedicated to experimenting with DreamBooth. ”– this invitation link is invalid now, how can I join this Discord?

Oblidas
Oblidas
1 year ago
Reply to  EdXD

This link doesn’t work either.

Jhong
Jhong
1 year ago

Hey, wondering if there’s room for an update in the article for the few seeings that have changed since? Learning rate is now an option which is very important.

C.P
C.P
1 year ago

Hi EdXD,

Fab article and colab notebook, I was able to run it with ease. You also explain everything clearly which is great.

I can see that you said you’ll be updating the article soon as there are new features added. Can I ask you include something on captioning images when you update?

And one further question, when training would you recommend avoiding training on objects and styles together?

Thanks!

CP

PeteB
PeteB
1 year ago

Great tutorial thanks – just coming fresh to this & a newbie – all works until I try and upload image instances and it loads the first image and then the error below. I’ve tried smaller images but didn’t work – can you help please? Thanks

·  thomas.jpg(image/jpeg) – 58846 bytes, last modified: n/a – 100% done
·  thomas1.jpg(image/jpeg) – 68448 bytes, last modified: n/a – 0% done
—————————————————————————
MessageError                              Traceback (most recent call last)
<ipython-input-17-0bc75dc439cd> in <module>
     83 elif IMAGES_FOLDER_OPTIONAL ==””:
     84   up=””
—> 85   uploaded = files.upload()
     86   for filename in uploaded.keys():
     87     if filename.split(“.”)[-1]==”txt”:

3 frames

/usr/local/lib/python3.8/dist-packages/google/colab/_message.py in read_reply_from_input(message_id, timeout_sec)
    100         reply.get(‘colab_msg_id’) == message_id):
    101       if ‘error’ in reply:
–> 102         raise MessageError(reply[‘error’])
    103       return reply.get(‘data’, None)
    104

MessageError: RangeError: Maximum call stack size exceeded.

Pete
Pete
1 year ago
Reply to  PeteB

Hi, just to add, I managed to get it to work by adding the photos to Google Drive, and then applying the link rather than uploading them directly from my Mac, not sure if that helps anyone else?

laiba
laiba
1 year ago

Hi I run the “Downloading the model” then have error, how to fix??

IMG_20230205_144843.jpg
AI_Noob
AI_Noob
1 year ago
Reply to  laiba

I am getting the same error 🙁

John
John
1 year ago

Thank you bro, I thought it’s going to be really hard to do that with collab, but it’s super easy actually.

zeotres
zeotres
1 year ago

I think TheLastBen deleted the colab and created a new one, can you check please the link. Thank you

zeotres
zeotres
1 year ago
Reply to  EdXD

Thanks, I found it too

mobri
mobri
9 months ago

Thank you!