ComfyUI/docs/video.md

Video Workflows

ComfyUI LTS supports video workflows with AnimateDiff Evolved.

First, install this package using the Installation Instructions.

Then, install the custom nodes packages that support video creation workflows:

uv pip install git+https://github.com/AppMana/appmana-comfyui-nodes-video-frame-interpolation
uv pip install git+https://github.com/AppMana/appmana-comfyui-nodes-video-helper-suite
uv pip install git+https://github.com/AppMana/appmana-comfyui-nodes-animatediff-evolved
uv pip install git+https://github.com/AppMana/appmana-comfyui-nodes-controlnet-aux.git

Start creating an AnimateDiff workflow. When using these packages, the appropriate models will download automatically.

SageAttention

Improve the performance of your Mochi model video generation using Sage Attention:

Device	PyTorch 2.5.1	SageAttention	S.A. + TorchCompileModel
A5000	7.52s/it	5.81s/it	5.00s/it (but corrupted)

Use the default Mochi Workflow. This does not require any custom nodes or any change to your workflow.

Installation

On Windows, you will need the CUDA Toolkit and Visual Studio 2022. If you do not already have this, use chocolatey:

# install chocolatey
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
choco install -y visualstudio2022buildtools
# purposefully executed separately
choco install -y visualstudio2022-workload-vctools
choco install -y vcredist2010 vcredist2013 vcredist140

Then, visit NVIDIA.com's CUDA Toolkit Download Page and download and install the CUDA Toolkit. Verify it is correctly installed by running nvcc --version.

You are now ready to install Sage Attention 2.

Linux

uv pip install --no-build-isolation "sageattention@git+https://github.com/thu-ml/SageAttention.git"

Windows

Run this PowerShell script to install the correct version of Sage Attention for your installed PyTorch version:

$torch_version = (uv pip freeze | Select-String "torch==").ToString().Trim()
$cuda_version = $torch_version -replace ".*(cu\d+).*", "`$1"
if ($torch_version -match "\+cu") {
    $v = $torch_version -replace "torch==", ""
    $package_specifier = "sageattention==2.2.0+$($cuda_version)torch$v"
    uv pip install --find-links https://raw.githubusercontent.com/hiddenswitch/ComfyUI/main/pypi/sageattention_index.html $package_specifier
} else {
    Write-Host "Could not determine CUDA version from torch version: $torch_version"
}

To start ComfyUI with it:

uv run comfyui --use-sage-attention

With SageAttention

With PyTorch Attention

Flash Attention

Flash Attention 2 is supported on Linux only.

uv pip install --no-build-isolation flash_attn

To start ComfyUI with it:

uv run comfyui --use-flash-attention

With SageAttention

With PyTorch Attention

Cosmos Prompt Upsampling

The Cosmos prompt "upsampler," a fine tune of Mistral-Nemo-12b, correctly rewrites Cosmos prompts in the narrative style that NVIDIA's captioner used for the training data of Cosmos, improving generation results significantly.

Here is a comparison between a simple and "upsampled" prompt.

A dog is playing with a ball.

In a sun-drenched park, a playful golden retriever bounds joyfully across the lush green grass, its tail wagging with excitement. The dog, adorned with a vibrant red collar, is captivated by a bright yellow ball, which it cradles gently in its mouth. The camera captures the dog's animated expressions, from eager anticipation to sheer delight, as it trots and leaps, showcasing its agility and enthusiasm. The scene is bathed in warm, golden-hour light, enhancing the vibrant colors of the dog's fur and the ball. The background features a serene tree line, framing the playful interaction and creating a tranquil atmosphere. The static camera angle allows for an intimate focus on the dog's joyful antics, inviting viewers to share in this heartwarming moment of pure canine happiness.

To use the Cosmos upsampler, install the prerequisites:

uv pip install loguru pynvml
uv pip install --no-deps git+https://github.com/NVIDIA/Cosmos.git

Then, use the workflow embedded in the upsampled prompt by dragging and dropping the upsampled animation into your workspace.

The Cosmos upsampler ought to improve any text-to-image video generation pipeline. Use the Video2World upsampler nodes to download Pixtral-12b and upsample for an image to video workflow using NVIDIA's default prompt. Since Pixtral is not fine tuned, the improvement may not be significant over using another LLM.