mirror of
https://github.com/comfyanonymous/ComfyUI.git
synced 2026-07-07 15:10:50 +08:00
093d17c587
## Summary
Sampling a long SeedVR2 video in one pass runs out of VRAM: the DiT working set grows linearly with latent frames times pixel area, so a 100 frame clip at 4x upscale needs more memory than any consumer card has. This PR adds two workflow-level nodes that split the latent into overlapping temporal chunks and recombine the sampled chunks with a Hann crossfade. The executor's list handling runs the stock KSampler once per chunk, so the sampler itself is untouched.
- **Chunk SeedVR2 Latent** splits the latent on the temporal axis. `frames_per_chunk` is in pixel frames on the 4n+1 grid, `temporal_overlap` sets how many latent frames adjacent chunks share, and `chunking_mode=auto` solves the chunk size from measured free VRAM and the latent's own dimensions. The node outputs the effective overlap so the merge is wired, not typed.
- **Merge SeedVR2 Latent Chunks** recombines the sampled chunks in order, crossfading each shared region with a Hann window (flat shoulders on the outer thirds, fade across the middle third). Zero overlap is a plain concatenation, bit-identical to `torch.cat`.
## Changes
- Added 2 nodes and 1 crossfade helper to `comfy_extras/nodes_seedvr.py` (+201 lines).
- Added 4 chunk-law constants to `comfy/ldm/seedvr/constants.py` (+10 lines).
- Added pytest unit tests in `tests-unit/comfy_extras_test/test_seedvr2_temporal_chunk.py` (+66 lines): chunk geometry, 4n+1 and mode validation, overlap clamping, temporal noise mask slicing (5-D and 4-D), the auto law including batch scaling, crossfade weights and blend direction, round trip, and metadata handling.
Everything outside the two new nodes is byte-identical to the base branch. The new constants are read only by the chunk node, and workflows that do not use these nodes take no new code path.
## Auto mode calibration
The auto law is `max_latent_frames = (free_GiB - reserved - margin) / (0.30 * megapixels)`, calibrated on an RTX 5090 (32 GB) with the 3b fp16 model: a 17-cell resolution sweep plus temporal bisection located the activation wall and confirmed it is linear in latent frames times pixel area (the same total-voxel budget holds from 1.5:1 through 24:1 aspect ratios and under transposition). The margin is four standard deviations of the measured run-to-run spread, which costs about one latent frame of chunk size and makes an out-of-memory failure a lottery ticket rather than a coin flip. Manual mode bypasses the law entirely.
On a 32 GB card, a 640x480x100 input at 4x upscale sampled as a single chunk allocates past 31 GiB and dies; auto mode picks 49 frame chunks (three chunks with overlap) and the full pipeline completes in about 240 seconds. The same law stands down on small inputs: a 320x240x100 clip runs as a single chunk because it fits.
## Example workflow
Load a video, 4x upscale, auto chunking, temporal overlap 3. Expected output for a 640x480x100 input: 2560x1920, 100 frames, seams invisible at the default overlap.
<details>
<summary>API workflow JSON</summary>
```json
{
"14": {"inputs": {"vae_name": "ema_vae_fp16.safetensors"}, "class_type": "VAELoader"},
"17": {"inputs": {"video": ["24", 0]}, "class_type": "GetVideoComponents"},
"20": {"inputs": {"fps": ["17", 2], "images": ["30", 0], "audio": ["17", 1]}, "class_type": "CreateVideo"},
"21": {"inputs": {"tile_size": 192, "overlap": 64, "temporal_size": 64, "temporal_overlap": 8, "pixels": ["27", 0], "vae": ["14", 0]}, "class_type": "VAEEncodeTiled"},
"22": {"inputs": {"tile_size": 256, "overlap": 32, "temporal_size": 64, "temporal_overlap": 8, "samples": ["33", 0], "vae": ["14", 0]}, "class_type": "VAEDecodeTiled"},
"23": {"inputs": {"unet_name": "seedvr2_3b_fp16.safetensors", "weight_dtype": "default"}, "class_type": "UNETLoader"},
"24": {"inputs": {"file": "input.mp4", "video-preview": ""}, "class_type": "LoadVideo"},
"25": {"inputs": {"filename_prefix": "video/seedvr2_upscale", "format": "auto", "codec": "auto", "video": ["20", 0]}, "class_type": "SaveVideo"},
"26": {"inputs": {"resize_type": "scale by multiplier", "resize_type.multiplier": 4, "scale_method": "bicubic", "input": ["17", 0]}, "class_type": "ResizeImageMaskNode"},
"27": {"inputs": {"resized_images": ["26", 0]}, "class_type": "SeedVR2Preprocess"},
"28": {"inputs": {"model": ["23", 0], "vae_conditioning": ["32", 0]}, "class_type": "SeedVR2Conditioning"},
"29": {"inputs": {"seed": 5770521, "steps": 1, "cfg": 1, "sampler_name": "euler", "scheduler": "simple", "denoise": 1, "model": ["23", 0], "positive": ["28", 0], "negative": ["28", 1], "latent_image": ["32", 0]}, "class_type": "KSampler"},
"30": {"inputs": {"color_correction_method": "lab", "images": ["22", 0], "original_resized_images": ["26", 0]}, "class_type": "SeedVR2PostProcessing"},
"32": {"inputs": {"frames_per_chunk": 21, "temporal_overlap": 3, "chunking_mode": "auto", "latent": ["21", 0]}, "class_type": "SeedVR2TemporalChunk"},
"33": {"inputs": {"temporal_overlap": ["32", 1], "latent_chunks": ["29", 0]}, "class_type": "SeedVR2TemporalMerge"}
}
```
</details>
## Prior art
- Reference implementation: https://github.com/ByteDance-Seed/SeedVR
- Community precedent for temporal chunking with blended reassembly: https://github.com/numz/ComfyUI-SeedVR2_VideoUpscaler
Chunk boundaries are a mathematical compromise: the model attends within a chunk, so different chunkings produce different outputs. The overlap crossfade hides the seam; power users can widen or zero the overlap from the workflow.
|
||
|---|---|---|
| .. | ||
| audio_encoders | ||
| background_removal | ||
| cldm | ||
| comfy_types | ||
| extra_samplers | ||
| image_encoders | ||
| k_diffusion | ||
| ldm | ||
| sd1_tokenizer | ||
| t2i_adapter | ||
| taesd | ||
| text_encoders | ||
| weight_adapter | ||
| bg_removal_model.py | ||
| cli_args.py | ||
| clip_config_bigg.json | ||
| clip_model.py | ||
| clip_vision_config_g.json | ||
| clip_vision_config_h.json | ||
| clip_vision_config_vitl_336_llava.json | ||
| clip_vision_config_vitl_336.json | ||
| clip_vision_config_vitl.json | ||
| clip_vision_siglip2_base_naflex.json | ||
| clip_vision_siglip_384.json | ||
| clip_vision_siglip_512.json | ||
| clip_vision.py | ||
| conds.py | ||
| context_windows.py | ||
| controlnet.py | ||
| deploy_environment.py | ||
| diffusers_convert.py | ||
| diffusers_load.py | ||
| float.py | ||
| gligen.py | ||
| hooks.py | ||
| latent_formats.py | ||
| lora_convert.py | ||
| lora.py | ||
| memory_management.py | ||
| model_base.py | ||
| model_detection.py | ||
| model_management.py | ||
| model_patcher.py | ||
| model_prefetch.py | ||
| model_sampling.py | ||
| multigpu.py | ||
| nested_tensor.py | ||
| ops.py | ||
| options.py | ||
| patcher_extension.py | ||
| pinned_memory.py | ||
| pixel_space_convert.py | ||
| quant_ops.py | ||
| rmsnorm.py | ||
| sample.py | ||
| sampler_helpers.py | ||
| samplers.py | ||
| sd1_clip_config.json | ||
| sd1_clip.py | ||
| sd.py | ||
| sdxl_clip.py | ||
| supported_models_base.py | ||
| supported_models.py | ||
| utils.py | ||