Animal dance video workflow (ComfyUI)

Updated November 23, 2024By AndrewCategorized as Workflow Tagged ComfyUI, Members only, vid2vid, Video 4 Comments

This workflow converts the subject of a dance video into an animal using Stable Diffusion. The following techniques will be used.

IP-adapter for consistent character.
Multiple ControlNets for consistent frame-to-frame motion.
AnimateDiff for frame-to-frame consistency.
LCM LoRA for speeding up video generation.

You will learn/get:

A downloadable ComfyUI workflow that generates this video.
Customization options.
Notes on building this workflow.

Become a member of this site to see this content

Become a member

Already a member? Log in here.

By Andrew

Andrew is an experienced engineer with a specialization in Machine Learning and Artificial Intelligence. He is passionate about programming, art, photography, and education. He has a Ph.D. in engineering.

View all of Andrew's posts.

4 comments

Tschai says:

October 8, 2024 at 4:41 am

Hi, I get this error message: Error(s) in loading state_dict for ImageProjModel: size mismatch for proj.weight: copying a param with shape torch.Size([3072, 1024]) from checkpoint, the shape in current model is torch.Size([3072, 1280]). What am I doing wrong?

Reply
1. Andrew says:
  
  October 8, 2024 at 7:14 am
  
  Did it say which model file? Which node was highlighted when it stopped?
  
  Reply
  1. Tschai says:
    
    October 8, 2024 at 8:43 am
    
    Thanks for the quick response! After restarting Stability Matrix and then ComfyUI, everything’s working perfectly now. Grazie mille
    
    Reply
Jacek says:

September 30, 2024 at 12:48 pm

<3

Reply

Become a member of this site to see this content

By Andrew

4 comments

Leave a comment Cancel reply