How to use Flux.1 Fill model for inpainting

Updated Categorized as Tutorial Tagged , , 7 Comments on How to use Flux.1 Fill model for inpainting

Using the standard Flux checkpoint for inpainting is not ideal. You must carefully adjust the denoising strength. Setting it too high causes inconsistency, and setting it too low does not change anything. The real bummer in this intermediate denoising strength is that the new content won’t deviate too much from the original color.

In this post, I will introduce a workflow using the Flux.1 Fill model which overcomes these limitations.

Software

We will use ComfyUI, an alternative to AUTOMATIC1111. You can use it on Windows, Mac, or Google Colab. If you prefer using a ComfyUI service, Think Diffusion offers our readers an extra 20% credit.

Read the ComfyUI beginner’s guide if you are new to ComfyUI. See the Quick Start Guide if you are new to AI images and videos.

Take the ComfyUI course to learn how to use ComfyUI step-by-step.

What is the Flux Fill model?

Model

The Flux.1 Fill model is an inpainting model. It is trained to fill in missing content while maintaining maximum consistency with the rest of the image.

The Flux.1 Dev Fill model is a fast version of the Flux.1 Pro Fill model. It is sped up using guidance distillation and denoises without classifier-free guidance (CFG). The result is a twofold speedup with a slight degradation in quality.

License

The Flux.1 Dev Fill model is under the same non-commercial Flux.1 Dev license. You can use the images generated by the model for commercial purposes.

VRAM requirement

This model is memory-hungry. You need an NVidia card with 24 GB VRAM to run this workflow.

Inpainting using Flux vs Flux Fill model

The benefits of using the Flux Fill model for inpainting are:

  • The maximum denoising strength (1) can be used while maintaining consistency with the image outside the inpaint mask.
  • The new content can be in different colors from the original content (because you use a denoising strength of 1).

Step-by-step guide

Step 0: Update ComfyUI

Before loading the workflow, make sure your ComfyUI is up-to-date. The easiest way to do this is to use ComfyUI Manager.

Click the Manager button on the top toolbar.

Select Update ComfyUI.

Restart ComfyUI.

Step 1: Download the fill diffusion model

Visit the Flux.1 Fill model page and click “Agree and access repository.”

Download the Flux.1 Dev fill model and save it to the ComfyUI > models > diffusion_models folder.

(If you use Google Colab: AI_PICS > models > diffusion_models)

Step 2: Download the CLIP models

Download the following two CLIP models, and put them in ComfyUI > models > clip.

(If you use Google Colab: AI_PICS > models > clip)

Step 3: Download the Flux VAE

Download the Flux VAE model file. Put it in ComfyUI > models > vae.

Step 4: Load the workflow

Download the workflow JSON file below and drop it to ComfyUI.

Step 5: Load image

Load the image you want to inpaint in the Load Image node.

You can download the test image below.

Step 6: Create an inpaint mask

Right-click the image to show the context menu. Select Open in MaskEditor.

Create a mask and click Save to node.

Now, you should see the mask in the node.

Step 7: Enter a prompt

Enter a prompt to describe the changed image. I want to change her to wear a top. So I write:

photo of woamn in a black top

Step 8: Generate the new image

Click Queue to start inpainting. Now, she wears a top.

Notes

If it doesn’t generate what you describe, try changing the seed in the KSampler node.

Avatar

By Andrew

Andrew is an experienced engineer with a specialization in Machine Learning and Artificial Intelligence. He is passionate about programming, art, photography, and education. He has a Ph.D. in engineering.

7 comments

  1. Hi do you know why quality is so bad when trying with furniture. It seams like it could only handle simple objects and not like more comple composition? Btw is there a way to get in contact with you?

  2. Hi Andrew, I have downloaded the model to diffusion_models folder on my google drive, but when I load the workflow, the Load diffusion model node doesn’t see it….. I get only “null” in the list…. any idea what I am doing wrong please?

Leave a comment

Your email address will not be published. Required fields are marked *