mirror of
https://github.com/comfyanonymous/ComfyUI.git
synced 2026-07-07 15:10:50 +08:00
093d17c587
## Summary
Sampling a long SeedVR2 video in one pass runs out of VRAM: the DiT working set grows linearly with latent frames times pixel area, so a 100 frame clip at 4x upscale needs more memory than any consumer card has. This PR adds two workflow-level nodes that split the latent into overlapping temporal chunks and recombine the sampled chunks with a Hann crossfade. The executor's list handling runs the stock KSampler once per chunk, so the sampler itself is untouched.
- **Chunk SeedVR2 Latent** splits the latent on the temporal axis. `frames_per_chunk` is in pixel frames on the 4n+1 grid, `temporal_overlap` sets how many latent frames adjacent chunks share, and `chunking_mode=auto` solves the chunk size from measured free VRAM and the latent's own dimensions. The node outputs the effective overlap so the merge is wired, not typed.
- **Merge SeedVR2 Latent Chunks** recombines the sampled chunks in order, crossfading each shared region with a Hann window (flat shoulders on the outer thirds, fade across the middle third). Zero overlap is a plain concatenation, bit-identical to `torch.cat`.
## Changes
- Added 2 nodes and 1 crossfade helper to `comfy_extras/nodes_seedvr.py` (+201 lines).
- Added 4 chunk-law constants to `comfy/ldm/seedvr/constants.py` (+10 lines).
- Added pytest unit tests in `tests-unit/comfy_extras_test/test_seedvr2_temporal_chunk.py` (+66 lines): chunk geometry, 4n+1 and mode validation, overlap clamping, temporal noise mask slicing (5-D and 4-D), the auto law including batch scaling, crossfade weights and blend direction, round trip, and metadata handling.
Everything outside the two new nodes is byte-identical to the base branch. The new constants are read only by the chunk node, and workflows that do not use these nodes take no new code path.
## Auto mode calibration
The auto law is `max_latent_frames = (free_GiB - reserved - margin) / (0.30 * megapixels)`, calibrated on an RTX 5090 (32 GB) with the 3b fp16 model: a 17-cell resolution sweep plus temporal bisection located the activation wall and confirmed it is linear in latent frames times pixel area (the same total-voxel budget holds from 1.5:1 through 24:1 aspect ratios and under transposition). The margin is four standard deviations of the measured run-to-run spread, which costs about one latent frame of chunk size and makes an out-of-memory failure a lottery ticket rather than a coin flip. Manual mode bypasses the law entirely.
On a 32 GB card, a 640x480x100 input at 4x upscale sampled as a single chunk allocates past 31 GiB and dies; auto mode picks 49 frame chunks (three chunks with overlap) and the full pipeline completes in about 240 seconds. The same law stands down on small inputs: a 320x240x100 clip runs as a single chunk because it fits.
## Example workflow
Load a video, 4x upscale, auto chunking, temporal overlap 3. Expected output for a 640x480x100 input: 2560x1920, 100 frames, seams invisible at the default overlap.
<details>
<summary>API workflow JSON</summary>
```json
{
"14": {"inputs": {"vae_name": "ema_vae_fp16.safetensors"}, "class_type": "VAELoader"},
"17": {"inputs": {"video": ["24", 0]}, "class_type": "GetVideoComponents"},
"20": {"inputs": {"fps": ["17", 2], "images": ["30", 0], "audio": ["17", 1]}, "class_type": "CreateVideo"},
"21": {"inputs": {"tile_size": 192, "overlap": 64, "temporal_size": 64, "temporal_overlap": 8, "pixels": ["27", 0], "vae": ["14", 0]}, "class_type": "VAEEncodeTiled"},
"22": {"inputs": {"tile_size": 256, "overlap": 32, "temporal_size": 64, "temporal_overlap": 8, "samples": ["33", 0], "vae": ["14", 0]}, "class_type": "VAEDecodeTiled"},
"23": {"inputs": {"unet_name": "seedvr2_3b_fp16.safetensors", "weight_dtype": "default"}, "class_type": "UNETLoader"},
"24": {"inputs": {"file": "input.mp4", "video-preview": ""}, "class_type": "LoadVideo"},
"25": {"inputs": {"filename_prefix": "video/seedvr2_upscale", "format": "auto", "codec": "auto", "video": ["20", 0]}, "class_type": "SaveVideo"},
"26": {"inputs": {"resize_type": "scale by multiplier", "resize_type.multiplier": 4, "scale_method": "bicubic", "input": ["17", 0]}, "class_type": "ResizeImageMaskNode"},
"27": {"inputs": {"resized_images": ["26", 0]}, "class_type": "SeedVR2Preprocess"},
"28": {"inputs": {"model": ["23", 0], "vae_conditioning": ["32", 0]}, "class_type": "SeedVR2Conditioning"},
"29": {"inputs": {"seed": 5770521, "steps": 1, "cfg": 1, "sampler_name": "euler", "scheduler": "simple", "denoise": 1, "model": ["23", 0], "positive": ["28", 0], "negative": ["28", 1], "latent_image": ["32", 0]}, "class_type": "KSampler"},
"30": {"inputs": {"color_correction_method": "lab", "images": ["22", 0], "original_resized_images": ["26", 0]}, "class_type": "SeedVR2PostProcessing"},
"32": {"inputs": {"frames_per_chunk": 21, "temporal_overlap": 3, "chunking_mode": "auto", "latent": ["21", 0]}, "class_type": "SeedVR2TemporalChunk"},
"33": {"inputs": {"temporal_overlap": ["32", 1], "latent_chunks": ["29", 0]}, "class_type": "SeedVR2TemporalMerge"}
}
```
</details>
## Prior art
- Reference implementation: https://github.com/ByteDance-Seed/SeedVR
- Community precedent for temporal chunking with blended reassembly: https://github.com/numz/ComfyUI-SeedVR2_VideoUpscaler
Chunk boundaries are a mathematical compromise: the model attends within a chunk, so different chunkings produce different outputs. The overlap crossfade hides the seam; power users can widen or zero the overlap from the workflow.
|
||
|---|---|---|
| .. | ||
| chainner_models | ||
| frame_interpolation_models | ||
| mediapipe | ||
| color_util.py | ||
| nodes_ace.py | ||
| nodes_advanced_samplers.py | ||
| nodes_align_your_steps.py | ||
| nodes_apg.py | ||
| nodes_ar_video.py | ||
| nodes_attention_multiply.py | ||
| nodes_audio_encoder.py | ||
| nodes_audio.py | ||
| nodes_bernini.py | ||
| nodes_bg_removal.py | ||
| nodes_boogu.py | ||
| nodes_bounding_boxes.py | ||
| nodes_camera_trajectory.py | ||
| nodes_canny.py | ||
| nodes_cfg.py | ||
| nodes_chroma_radiance.py | ||
| nodes_clip_sdxl.py | ||
| nodes_color.py | ||
| nodes_compositing.py | ||
| nodes_cond.py | ||
| nodes_context_windows.py | ||
| nodes_controlnet.py | ||
| nodes_cosmos.py | ||
| nodes_curve.py | ||
| nodes_custom_sampler.py | ||
| nodes_dataset.py | ||
| nodes_depth_anything_3.py | ||
| nodes_differential_diffusion.py | ||
| nodes_easycache.py | ||
| nodes_edit_model.py | ||
| nodes_eps.py | ||
| nodes_flux.py | ||
| nodes_frame_interpolation.py | ||
| nodes_freelunch.py | ||
| nodes_fresca.py | ||
| nodes_gaussian_splat.py | ||
| nodes_gits.py | ||
| nodes_glsl.py | ||
| nodes_hidream_o1.py | ||
| nodes_hidream.py | ||
| nodes_hooks.py | ||
| nodes_hunyuan3d.py | ||
| nodes_hunyuan.py | ||
| nodes_hypernetwork.py | ||
| nodes_hypertile.py | ||
| nodes_ideogram4.py | ||
| nodes_image_compare.py | ||
| nodes_images.py | ||
| nodes_ip2p.py | ||
| nodes_json_prompt.py | ||
| nodes_kandinsky5.py | ||
| nodes_latent.py | ||
| nodes_load_3d.py | ||
| nodes_logic.py | ||
| nodes_lora_debug.py | ||
| nodes_lora_extract.py | ||
| nodes_lotus.py | ||
| nodes_lt_audio.py | ||
| nodes_lt_upsampler.py | ||
| nodes_lt.py | ||
| nodes_lumina2.py | ||
| nodes_mahiro.py | ||
| nodes_mask.py | ||
| nodes_math.py | ||
| nodes_mediapipe.py | ||
| nodes_mochi.py | ||
| nodes_model_advanced.py | ||
| nodes_model_downscale.py | ||
| nodes_model_merging_model_specific.py | ||
| nodes_model_merging.py | ||
| nodes_model_patch.py | ||
| nodes_moge.py | ||
| nodes_morphology.py | ||
| nodes_multigpu.py | ||
| nodes_nag.py | ||
| nodes_nop.py | ||
| nodes_number_convert.py | ||
| nodes_optimalsteps.py | ||
| nodes_pag.py | ||
| nodes_painter.py | ||
| nodes_perpneg.py | ||
| nodes_photomaker.py | ||
| nodes_pid.py | ||
| nodes_pixart.py | ||
| nodes_post_processing.py | ||
| nodes_preview_any.py | ||
| nodes_primitive.py | ||
| nodes_qwen.py | ||
| nodes_rebatch.py | ||
| nodes_replacements.py | ||
| nodes_resolution.py | ||
| nodes_rope.py | ||
| nodes_rtdetr.py | ||
| nodes_sag.py | ||
| nodes_sam3.py | ||
| nodes_save_3d.py | ||
| nodes_scail.py | ||
| nodes_sd3.py | ||
| nodes_sdpose.py | ||
| nodes_sdupscale.py | ||
| nodes_seed.py | ||
| nodes_seedvr.py | ||
| nodes_slg.py | ||
| nodes_stable3d.py | ||
| nodes_stable_cascade.py | ||
| nodes_string.py | ||
| nodes_tcfg.py | ||
| nodes_textgen.py | ||
| nodes_tomesd.py | ||
| nodes_toolkit.py | ||
| nodes_torch_compile.py | ||
| nodes_train.py | ||
| nodes_triposplat.py | ||
| nodes_upscale_model.py | ||
| nodes_video_model.py | ||
| nodes_video.py | ||
| nodes_void.py | ||
| nodes_wan.py | ||
| nodes_wandancer.py | ||
| nodes_wanmove.py | ||
| nodes_webcam.py | ||
| nodes_zimage.py | ||
| void_noise_warp.py | ||