Animal dance video workflow (ComfyUI)

This workflow converts the subject of a dance video into an animal using Stable Diffusion. The following techniques will be used.

• IP-adapter for consistent character.
• Multiple ControlNets for consistent frame-to-frame motion.
• AnimateDiff for frame-to-frame consistency.
• LCM LoRA for speeding up video generation.

You will learn/get:

• A downloadable ComfyUI workflow that generates this video.
• Customization options.
• Notes on building this workflow.

Software

Stable Diffusion GUI

We will use ComfyUI, a node-based Stable Diffusion GUI. You can use ComfyUI on Window/Mac or Google Colab.

Check out Think Diffusion for a fully managed ComfyUI/A1111/Forge online service. They offer 20% extra credits to our readers. (and a small commission to support this site if you sign up)

See the beginner’s guide for ComfyUI if you haven’t used it.

Step-by-step guide

Step 1: Load the ComfyUI workflow

Download the workflow JSON file below.

Drag and drop it to ComfyUI to load.

Step 2: Go through the drill…

Every time you try to run a new workflow, you may need to do some or all of the following steps.

1. Install ComfyUI Manager
2. Install missing nodes
3. Update everything

Install ComfyUI Manager

Install ComfyUI manager if you haven’t done so already. It provides an easy way to update ComfyUI and install missing nodes.

To install this custom node, go to the custom nodes folder in the PowerShell (Windows) or Terminal (Mac) App:

cd ComfyUI/custom_nodes

Install ComfyUI by cloning the repository under the custom_nodes folder.

git clone https://github.com/ltdrdata/ComfyUI-Manager

Restart ComfyUI completely. You should see a new Manager button appearing on the menu.

If you don’t see the Manager button, check the terminal for error messages. One common issue is GIT not installed. Installing it and repeat the steps should resolve the issue.

Install missing custom nodes

To install the custom nodes that are used by the workflow but you don’t have:

1. Click Manager in the Menu.
2. Click Install Missing custom Nodes.
3. Restart ComfyUI completely.

Update everything

You can use ComfyUI manager to update custom nodes and ComfyUI itself.

1. Click Manager in the Menu.
2. Click Updates All. It may take a while to be done.
3. Restart the ComfyUI and refresh the ComfyUI page.

Step 3: Set input video

Upload an input video. You can find the video I used below.

Upload it in the Load Video (Upload) node.

The frame_load_cap sets the maximum number of frames to be used. Set it to 16 if you are testing settings.

The width and height setting needs to match the size of the video.

Step 4: Download models

Checkpoint model

This workflow uses an anime model.

Download the IMP v1.0 model. Put it in ComfyUI > models > checkpoints.

Refresh and select the model in Load Checkpoint.

VAE

This workflow uses Stability’s standard VAE.

Download the FT-MSE-840000 VAE model. Put it in ComfyUI > models > vae.

Refresh and select the model in Load VAE.

IP adapter

This workflow uses the IP-adapter to achieve a consistent face and clothing.

Download the IP adapter model. Put it in ComfyUI > models > ipadapter.

Refresh and select the model in Load IPAdapter Model.

Download the SD 1.5 CLIP vision model. Put it in ComfyUI > models > clip_vision. You may want to rename it to CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors to conform to the custom node’s naming convention.

Refresh and select the model in Load CLIP Vision Model.

Upload the reference image for the character in the Load Image (IP-adapter) node. You can download the image below.

LoRA

Download the add_detail LoRA. Put it in ComfyUI > models > loras.

This LoRA helps to improve details.

ControlNet

Download the Animal OpenPose ControlNet model. Put it in ComfyUI > models > controlnet.

Download the Soft Edge ControlNet model. Put it in ComfyUI > models > controlnet.

Refresh and select the models in the Load Advanced ControlNet Model nodes.

AnimateDiff

Download the Motion Model v0.1 module. Put it in custom_nodes > ComfyUI-AnimateDiff-Evolved > models.

Refresh and select the model.

LCM LoRA

This workflow uses the LCM LoRA to achieve the 3x speed up. Download the LCM LoRA. Put it in Put it in ComfyUI > models > loras. Rename it to lcm_lora_sd15.safetensors.

Step 5: Generate video

Press Queue Prompt to start generating the video.

If you see an out-of-memory error, you can add the extra argument --disable-smart-memory to run_nvidia_gpu.bat.

.python_embededpython.exe -s ComfyUImain.py --windows-standalone-build --disable-smart-memory

Customization

Seed

Changing the seed can sometimes fix an inconsistency.

Prompt

You can customize the animal by changing the prompt and the IP-adapter image. The best is to have them match each other.

The background cannot be changed too much since the Soft Edge ControlNet fixes it.

Notes

Consistent character

This workflow uses these components to generate a consistent character:

1. ControlNet Openpose
2. ControlNet Soft Edge
3. IP-adapter

The animal openpose generates animals following the openpose control image.

OpenPose alone cannot generate consistent human pose movement. ControlNet Soft Edge creates coarse outline.

The IP adapter fixes the animal’s appearance. Without it, the character changes too much.

Checkpoint model

The checkpoint model has a big impact on the quality of the animation. Some models, especially the anime ones, do not generate clear images.

LoRA model

Two LoRAs are used:

• Add_detail: Adding fine details.
• LCM LoRA: For speed up.

IP-adapter

The IP-adapter is critical in producing a consistent character. But the weight cannot be set to high. Otherwise, the video will be blurry.

Remove the background of the control image to avoid getting the background to the video,

ControlNet

This workflow uses ControlNet HED. Compared to the line art ControlNet used in the previous version, HED gives the workflow more freedom in painting the character since it is less fixed.

The weight and end percent are set to very low to avoid fixing the background too much.