[Andrew]

Interesting. An option is to combine them – use flux to generate the initial image followed by SDXL with img2img at a low denoising strength.

[Andrew]

Hi, can you post a Flux image that looks plastic and one from Juggernaut that you think is good?

I need to see what you are looking for before giving advice.

[Andrew]

LoRA should be the go-to method for modifying models. A LoRA can also modify CLIP, but the main effect is in modifying the diffusion model.

I haven’t done a comparison, but sampling is a pretty standard process. I don’t think they would do anything different.

[Andrew]

Hi, we need three models to use a diffusion model like Stable Diffusion and Flux

• Diffusion model – for denoising during sampling.
• VAE – for converting the images between pixel and latent spaces.
• CLIP – for encoding to text prompt for conditioning during sampling.

Some checkpoints include all three in a single checkpoint file. Even if others don’t, the Load Checkpoint node uses the default VAE and CLIP models.

In addition to the original VAE, there are improved or finetuned versions, although they are rare. You can use the Load VAE node to specify the VAE you want. Typically, the difference is minimal.

Likewise, you can load the CLIP models directly using a node. Some models, like Flux, use two text encoders. Using the Dual CLIP loader allows you to put different text prompts to different encoders. Some people swear to see a difference between the two, but this remains an under-explored area.

 

 

 

[Andrew]

Hi, we need three models to use a diffusion model like Stable Diffusion and Flux

Diffusion model – for denoising during sampling.
VAE – for converting the images between pixel and latent spaces.
CLIP – for encoding to text prompt for conditioning during sampling.
Some checkpoints include all three in a single checkpoint file. Even if others don’t, the Load Checkpoint node uses the default VAE and CLIP models.

In addition to the original VAE, there are improved or finetuned versions, although they are rare. You can use the Load VAE node to specify the VAE you want. Typically, the difference is minimal.

Likewise, you can load the CLIP models directly using a node. Some models, like Flux, use two text encoders. Using the Dual CLIP loader allows you to put different text prompts to different encoders. Some people swear to see a difference between the two, but this remains an under-explored area.

 

 

 

[Andrew]

It’s great to see you figured it out. It’s a warning, so nothing to worry about.

[Andrew]

Mmm… The only things I can think of are:

• Use a T4 instance (that’s what I used)
• Remove or rename the custom_nodes folder in AI_PICS > ComfyUI

[Andrew]

Hi David, I just tested the workflow on Colab, and it works correctly. I used the dreamshaper model and the motion model in the workflow. Did you change any settings?

[Andrew]

Good find! I have fixed the issue.

[Andrew]

I haven’t used it but it uses Kohya in the backend, so it should work.

[Andrew]

Yes, I am in the process of writing the lessons. Should be up within a week.

[Andrew]

Thanks for the review!

I’m now refreshing the comfy course and will put more advanced techniques there.

[Andrew]

Yes: https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt/blob/main/svd_xt.safetensors

I also added it to the ComfyUI notebook.

[Andrew]

The SVD one is a bit tricky to fix. I will work on it, but you can use Comfyui in the meantime.

[Andrew]

Notebook fixed. Downgraded to Python 3.10. Using python3.11-venv is opening a can of worm 🙂

Follow the link on the tutorial page to access the new notebook. (updated 1/17)

• This reply was modified 8 months, 3 weeks ago by Andrew.
• This reply was modified 8 months, 3 weeks ago by Andrew.

[Author]

Posts