Well, it’s more like a movie trailer for now. This ComfyUI workflow combines three Stable Video Diffusion (SVD) videos in a text-to-video workflow. All you need to do is put in each clip’s prompt.
You will learn:
- How to use a ComfyUI workflow (provided) to generate this video.
- Customization options.
You must be a member of this site to download this workflow.
Table of Contents
Software
Stable Diffusion GUI
We will use ComfyUI, a node-based Stable Diffusion GUI. See the installation and beginner’s guide for ComfyUI if you haven’t used it.
Step-by-step guide
Step 1: Load the ComfyUI workflow
Download the workflow JSON file below. Log in and download the workflow below.
Drag and drop it to ComfyUI to load.
You may need to
Step 2: Set the txt2img model
SDXL checkpoint
Download the Juggernaut XL model. Put it in ComfyUI > models > checkpoints.
Refresh the page and select the model in the Load Checkpoint dropdown menu.
SVD model
Download the SVD XT model. Put it in ComfyUI > models > checkpoints.
Refresh the page and select the model in the Image Only Checkpoint Loader (img2vid) dropdown menu.
Step 3: Generate video
Press Queue Prompt to generate the video. This is the MP4 video:
Customization
Prompts
You would probably change the prompts to make your own video.
To save time, you can work on one prompt at a time to generate an image you like without generating the video.
To disable video generation, select the first node of an image-to-video group. The first node should be SVD_img2vid_Conditioning.
Press Ctrl+M (Mute) to disable it.
Change the seed of the KSampler in the txt2img section to generate a different image.
When you get an image you like, re-enable the SVD_img2vid_Conditioning node. You can do that by selecting the node and press Ctrl+M.
Video
It is normal to see a subpar video even with a good initial image. Change the video’s seed to generate a new video.
Adding video clips
You can add more video clips by duplicating the text-to-image and the image-to-video blocks.
Is there a way to install comfyUi with Google collab? I have a mac (2019) so i think it will not work for me locally right?
Any way to use image to image (instead of text to Image)?
Yes you can change the workflow to use an input image. Just remove the text-to-image part. You will need to ensure the image size is exactly correct.