How to use Hunyuan video LoRA to create consistent characters

Low-Rank Adaptation (LoRA) has emerged as a game-changing technique for finetuning image models like Flux and Stable Diffusion. By focusing on adjusting a subset of model parameters, LoRA enables efficient finetuning without the need to retrain the entire model.

For the first time, LoRA has come to a large video model. Now, you can use LoRA to finetune Hunyuan Video.

In this tutorial, I will show you how to use LoRAs with Hunyuan Video. You can use multiple LoRAs to create a consistent character in scenes like the ones below.

Software

How does Hunyuan LoRA work?

A Hunyuan LoRA works like LoRA in Stable Diffusion and Flux. It is a small model file that stores the changes to some parts of the Hunyuan base model. A LoRA file is typically ~100x smaller than the Hunyuan base model.

They are ideal for specific video tasks, such as generating specific objects, styles, and characters.

Use cases

Style transfer

You can add a LoRA to achieve a new style.

Here’s the video from the Hunyuan base model:

Casting the video to an Anime style with the Makoto Shinkai Anime Style LoRA:

Consistent character

You can train a LoRA to put a new face in the Hunyuan Video. If you don’t have the resources to train a LoRA, you can always blend two face LoRAs on CivitAI to create a new one.

For example, we can use the Emma Watson LoRA:

And mix with the Emily Bloom LoRA:

To create a new person Emma-ly…:

By carefully selecting the faces and adjusting the LoRA weights, you can create the consistent face you need.

One LoRA workflow

This workflow uses the ComfyUI-HunyuanVideoWrapper custom nodes with the FastVideo model.

Step 0: Update ComfyUI

Before loading the workflow, make sure your ComfyUI is up-to-date. The easiest way to do this is to use ComfyUI Manager.

Click the Manager button on the top toolbar.

Select Update ComfyUI.

Restart ComfyUI.

Step 1: Download the video model

Download the hunyuan_video_FastVideo_720_fp8_e4m3fn.safetensors and put it in ComfyUI > models > diffusion_models.

Step 2: Download VAE

Download hunyuan_video_vae_bf16.safetensors.

This VAE file differs from the one released by ComfyUI and is used in the text-to-video tutorial. Rename it to hunyuan_video_vae_bf16-kj.safetensors

Put it in ComfyUI > models > vae.

Step 3: Download LoRA model

Download the Makoto Shinkai Anime Style LoRA model. Put it in ComfyUI > models > loras > Hunyuan.

Step 4: Load workflow

Download the Hunyuan video workflow JSON file below.

Drop it to ComfyUI.

Step 5: Install missing nodes

If you see red blocks, you don’t have the custom node that this workflow needs.

Click Manager > Install missing custom nodes and install the missing nodes.

Restart ComfyUI.

Step 6: Run the workflow

The LoRA is loaded before the Hunyuan Video base model. It modifies the base model’s weights in memory.

You can optionally revise the prompt. This LoRA needs the trigger keywords “anime style” in the prompt.

Click the Queue button to run the workflow.

You should get this video:

Two-LoRA workflow

A two-LoRA workflow uses two LoRAs to modify the Hunyuan base model. It is an excellent choice for creating a consistent character by blending two faces.

The following steps assume you have completed the setup of the One-LoRA workflow above.

Step 1: Download the LoRA models

Download the Emma Watson Hunyuan video Lora and put it in ComfyUI > models > loras > Hunyuan.

Download the Emily Bloom Hunyuan video Lora and put it in ComfyUI > models > loras > Hunyuan.

Step 2: Run the workflow

Download the JSON workflow file below.

Drop it to ComfyUI.

Click the Queue button to run the workflow.

You should get this video:

Changing the prompt casts the character to a different scene.