How to use Hunyuan video LoRA to create consistent characters

Updated Categorized as Tutorial Tagged , , , 14 Comments on How to use Hunyuan video LoRA to create consistent characters

Low-Rank Adaptation (LoRA) has emerged as a game-changing technique for finetuning image models like Flux and Stable Diffusion. By focusing on adjusting a subset of model parameters, LoRA enables efficient finetuning without the need to retrain the entire model.

For the first time, LoRA has come to a large video model. Now, you can use LoRA to finetune Hunyuan Video.

In this tutorial, I will show you how to use LoRAs with Hunyuan Video. You can use multiple LoRAs to create a consistent character in scenes like the ones below.

Software

We will use ComfyUI, an alternative to AUTOMATIC1111. You can use it on Windows, Mac, or Google Colab. If you prefer using a ComfyUI service, Think Diffusion offers our readers an extra 20% credit.

Read the ComfyUI beginner’s guide if you are new to ComfyUI. See the Quick Start Guide if you are new to AI images and videos.

Take the ComfyUI course to learn how to use ComfyUI step-by-step.

How does Hunyuan LoRA work?

A Hunyuan LoRA works like LoRA in Stable Diffusion and Flux. It is a small model file that stores the changes to some parts of the Hunyuan base model. A LoRA file is typically ~100x smaller than the Hunyuan base model.

They are ideal for specific video tasks, such as generating specific objects, styles, and characters.

Use cases

Style transfer

You can add a LoRA to achieve a new style.

Here’s the video from the Hunyuan base model:

A 25yo blonde beautiful woman smiling in red leather jacket on a motorcycle in a busy new york city street.

Casting the video to an Anime style with the Makoto Shinkai Anime Style LoRA:

A 25yo blonde beautiful woman smiling in red leather jacket on a motorcycle in a busy new york city street. anime style.

Consistent character

You can train a LoRA to put a new face in the Hunyuan Video. If you don’t have the resources to train a LoRA, you can always blend two face LoRAs on CivitAI to create a new one.

For example, we can use the Emma Watson LoRA:

And mix with the Emily Bloom LoRA:

To create a new person Emma-ly…:

By carefully selecting the faces and adjusting the LoRA weights, you can create the consistent face you need.

One LoRA workflow

This workflow uses the ComfyUI-HunyuanVideoWrapper custom nodes with the FastVideo model.

Step 0: Update ComfyUI

Before loading the workflow, make sure your ComfyUI is up-to-date. The easiest way to do this is to use ComfyUI Manager.

Click the Manager button on the top toolbar.

Select Update ComfyUI.

Restart ComfyUI.

Step 1: Download the video model

Download the hunyuan_video_FastVideo_720_fp8_e4m3fn.safetensors and put it in ComfyUI > models > diffusion_models.

Google Colab

If you use my ComfyUI Colab notebook, put the model in AI_PICS > models > diffusion_models.

Step 2: Download VAE

Download hunyuan_video_vae_bf16.safetensors.

This VAE file differs from the one released by ComfyUI and is used in the text-to-video tutorial. Rename it to hunyuan_video_vae_bf16-kj.safetensors

Put it in ComfyUI > models > vae.

Google Colab

If you use my ComfyUI Colab notebook, put the model in AI_PICS > models > VAE.

Step 3: Download LoRA model

Download the Makoto Shinkai Anime Style LoRA model. Put it in ComfyUI > models > loras > Hunyuan.

Google Colab

If you use my ComfyUI Colab notebook, put the model in AI_PICS > models > Lora.

Step 4: Load workflow

Download the Hunyuan video workflow JSON file below.

Drop it to ComfyUI.

Step 5: Install missing nodes

If you see red blocks, you don’t have the custom node that this workflow needs.

Click Manager > Install missing custom nodes and install the missing nodes.

Restart ComfyUI.

Step 6: Run the workflow

The LoRA is loaded before the Hunyuan Video base model. It modifies the base model’s weights in memory.

You can optionally revise the prompt. This LoRA needs the trigger keywords “anime style” in the prompt.

Click the Queue button to run the workflow.

You should get this video:

Two-LoRA workflow

A two-LoRA workflow uses two LoRAs to modify the Hunyuan base model. It is an excellent choice for creating a consistent character by blending two faces.

The following steps assume you have completed the setup of the One-LoRA workflow above.

Step 1: Download the LoRA models

Download the Emma Watson Hunyuan video Lora and put it in ComfyUI > models > loras > Hunyuan.

Download the Emily Bloom Hunyuan video Lora and put it in ComfyUI > models > loras > Hunyuan.

Step 2: Run the workflow

Download the JSON workflow file below.

Drop it to ComfyUI.

Click the Queue button to run the workflow.

You should get this video:

Changing the prompt casts the character to a different scene.

Andrew

By Andrew

Andrew is an experienced software engineer with a specialization in Machine Learning and Artificial Intelligence. He is passionate about programming, art, and education. He has a doctorate degree in engineering.

14 comments

  1. I am struggling to get this to work on colab. Should I be uploading the model to my drive like the vae and lora?

  2. Thanks for the guides. I wanted to know the difference between this guide with LORAs and the other guide without Loras. As you mentioned this is using FastVideo FP8 and vae is different too. But is it possible to add Loras to the other guide or use bf16 model with this. The workflow nodes are quite different too.

    1. No, the other guide uses comfyui’s native nodes. This one uses custom nodes. I think this custom nodes support a bf16 model. You may need to follow their README doc and download it.

      To remove the LoRA, remove or disable the lora node.

  3. Thank you for this thorough writeup and for the json workflow – huge help and very satisfying to be able to get this to run on Google Colab.

    I have a Colab Pro plan and when attempting to generate a video that is 121 frames (24fps), 720×480, 1 lora, using the model + VAE you mentioned, it is taking approximately ~25 minutes per video on a A100 in Colab – is that to be expected? I would’ve thought the A100 would be churning these out in ~6 minutes or so. Any guidance on speeding this up would be appreciated if I perhaps have something configured incorrectly.

      1. Unfortunately, the L4 runs out of memory (has happened pretty much every time) — Even without a lora in the workflow.

  4. Hi, I’ve drop Emma Watson Hunyuan video JSON workflow file to my desktop ComfyUI, but why i can’t search any missing nodes in custom nodes manager, missing custom nodes and model manager, when i drop the JSON file.

  5. unfortunately I just get this error: HyVideoModelLoader.loadmodel() got an unexpected keyword argument ‘lora’

  6. Thanks for sharing this article on how to use LoRA with Hunyuan, can’t wait to get stuck into it once I’ve got free time later today!
    Are there any plans to share any tutorials or resources for creating LoRA that can be used as well?
    I’m liking what I’ve created so far, but would love to bring some of my characters that I’ve created for SDXL, Flux, Pony, SD3.5 into this as well so that I can continue to explore it all.

Leave a comment

Your email address will not be published. Required fields are marked *