wangbo/ComfyUI
1
0
Fork 0
mirror of https://github.com/comfyanonymous/ComfyUI.git synced 2026-07-04 05:31:03 +08:00
Code Issues Actions 19 Packages Projects Releases Wiki Activity
e96bd48e2d
ComfyUI/comfy_extras
History
huangfeice 5260e18cdf Add JoyImageEdit native model support 
JoyImageEdit is an image-edit diffusion transformer from JD (jd-opensource),
Apache 2.0. This adds native ComfyUI support so it loads and runs like other
edit models (load checkpoint -> TextEncode + ReferenceLatent -> KSampler ->
VAEDecode), with no diffusers dependency.

Architecture:
- Transformer (comfy/ldm/joyimage/model.py): dual-stream (img/txt) DiT with a
  Conv3d patch embed (patch_size [1,2,2]), Wan-style learnable modulation,
  and 3D RoPE (rope_dim_list [16,56,56]). All attention goes through
  comfy.ldm.modules.attention.optimized_attention.
- Text encoder (comfy/text_encoders/{qwen3_vl,joyimage}.py): a reusable
  Qwen3-VL multimodal stack (vision tower + LM) in qwen3_vl.py, plus a thin
  JoyImage-specific layer (prompt templates, drop_idx, tokenizer, te() factory)
  in joyimage.py that depends on it. text_dim 4096.
- VAE: reuses the existing Wan 2.1 latent format (AutoencoderKLWan), no new
  latent format.
- Edit conditioning: reuses the reference_latents mechanism. Reference and
  noise latents are stacked on a new n-slot dimension and rotated at the model
  boundary (model_base.JoyImage), so the transformer stays 5D-in/5D-out.
  Guidance-rescale is built into the CFG path.

Model wiring:
- model_base.JoyImage uses ModelType.FLOW with sampling_settings
  multiplier=1000 (the time embedding is trained on t in [0,1000]) and
  shift=1.5; FLOW's linear time_snr_shift matches the diffusers
  FlowMatchEuler sigma schedule.
- model_detection sniffs the transformer state-dict (double_blocks.*,
  condition_embedder.*, 5D img_in Conv3d) to route image_model="joyimage".
- supported_models.JoyImage and the CLIPLoader "joyimage" type register it.

User-facing node TextEncodeJoyImageEdit (comfy_extras/nodes_joyimage.py)
bucket-resizes the input image to the nearest 1024-base bucket, encodes the
prompt with the image, and emits both the conditioning and the bucketed image
so the same pixels feed VAEEncode and the negative encode (JoyImage requires
noise and reference latents to share spatial dims).
2026-06-17 18:53:36 +08:00
..
chainner_models Replace print with logging (#6138) 2024-12-20 16:24:55 -05:00
frame_interpolation_models fix: Proper memory estimation for frame interpolation when not using dynamic VRAM (#13698) 2026-05-04 20:20:40 +03:00
mediapipe Remove useless annotations imports. (#14105) 2026-05-25 19:23:29 -07:00
nodes_ace.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_advanced_samplers.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_align_your_steps.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_apg.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_ar_video.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_attention_multiply.py Update nodes categories and display names (CORE-89) (#13786) 2026-05-08 01:02:55 -04:00
nodes_audio_encoder.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_audio.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_bernini.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_bg_removal.py Move bg_removal_model input socket to first position for nicer display (#14353) 2026-06-09 23:36:56 +08:00
nodes_camera_trajectory.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_canny.py chore: Various QoL updates of nodes display names, descriptions and categories (CORE-190, CORE-191) (#13830) 2026-05-19 00:13:48 -04:00
nodes_cfg.py feat: Microsoft Lens support (CORE-248) (#14077) 2026-05-25 23:01:51 -07:00
nodes_chroma_radiance.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_clip_sdxl.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_color.py Add Color primitive (#14260) 2026-06-09 00:30:28 +08:00
nodes_compositing.py chore: Various QoL updates of nodes display names, descriptions and categories (CORE-190, CORE-191) (#13830) 2026-05-19 00:13:48 -04:00
nodes_cond.py Update nodes categories and display names (CORE-89) (#13786) 2026-05-08 01:02:55 -04:00
nodes_context_windows.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_controlnet.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_cosmos.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_curve.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_custom_sampler.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_dataset.py fix: Image grid bug fix (CORE-215) (#14100) 2026-06-03 16:03:32 -07:00
nodes_depth_anything_3.py Depth anything 3 (Core-135) (#13853) 2026-06-10 09:28:24 +08:00
nodes_differential_diffusion.py Update nodes categories and display names (CORE-89) (#13786) 2026-05-08 01:02:55 -04:00
nodes_easycache.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_edit_model.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_eps.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_flux.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_frame_interpolation.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_freelunch.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_fresca.py Update nodes categories and display names (CORE-89) (#13786) 2026-05-08 01:02:55 -04:00
nodes_gaussian_splat.py feat: add PreviewGaussianSplat + PreviewPointCloud nodes (#14194) 2026-06-05 12:30:58 -07:00
nodes_gits.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_glsl.py Add has_intermediate_output flag for nodes with interactive UI (#13048) 2026-03-27 21:06:38 -04:00
nodes_hidream_o1.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_hidream.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_hooks.py Disable dynamic_vram when weight hooks applied (#12653) 2026-02-28 16:50:18 -05:00
nodes_hunyuan3d.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_hunyuan.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_hypernetwork.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_hypertile.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_ideogram4.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_image_compare.py chore: Update display names and categories (CORE-151) (#13693) 2026-05-04 17:37:25 -07:00
nodes_images.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_ip2p.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_joyimage.py Add JoyImageEdit native model support 2026-06-17 18:53:36 +08:00
nodes_kandinsky5.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_latent.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_load_3d.py feat(3d): reorder Preview3DAdvanced / PreviewGaussianSplat / PreviewPointCloud inputs and outputs (#14308) 2026-06-05 19:01:57 -07:00
nodes_logic.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_lora_debug.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_lora_extract.py Update nodes categories and display names (CORE-89) (#13786) 2026-05-08 01:02:55 -04:00
nodes_lotus.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_lt_audio.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_lt_upsampler.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_lt.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_lumina2.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_mahiro.py Update nodes categories and display names (CORE-89) (#13786) 2026-05-08 01:02:55 -04:00
nodes_mask.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_math.py Fix uncaught OverflowError in Math Expression node for large int results (#14214) 2026-06-01 18:15:04 -07:00
nodes_mediapipe.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_mochi.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_model_advanced.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_model_downscale.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_model_merging_model_specific.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_model_merging.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_model_patch.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_moge.py Depth anything 3 (Core-135) (#13853) 2026-06-10 09:28:24 +08:00
nodes_morphology.py chore: Various QoL updates of nodes display names, descriptions and categories (CORE-190, CORE-191) (#13830) 2026-05-19 00:13:48 -04:00
nodes_multigpu.py multigpu: use unet_manual_cast for SelectModelDevice compute dtype (#14108) 2026-05-25 20:03:37 -07:00
nodes_nag.py Add category to Normalized Attention Guidance node (#12565) 2026-02-21 19:51:21 -05:00
nodes_nop.py chore: Various QoL updates of nodes display names, descriptions and categories (CORE-190, CORE-191) (#13830) 2026-05-19 00:13:48 -04:00
nodes_number_convert.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_optimalsteps.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_pag.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_painter.py Remove useless annotations imports. (#14105) 2026-05-25 19:23:29 -07:00
nodes_perpneg.py Update nodes categories and display names (CORE-89) (#13786) 2026-05-08 01:02:55 -04:00
nodes_photomaker.py Update nodes categories and display names (CORE-89) (#13786) 2026-05-08 01:02:55 -04:00
nodes_pid.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_pixart.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_post_processing.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_preview_any.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_primitive.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_qwen.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_rebatch.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_replacements.py Node Replacement API (#12014) 2026-02-15 02:12:30 -08:00
nodes_resolution.py Improve ResolutionSelector (#14309) 2026-06-09 01:05:10 +08:00
nodes_rope.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_rtdetr.py Update nodes titles (#14417) 2026-06-16 11:42:00 +08:00
nodes_sag.py Update nodes categories and display names (CORE-89) (#13786) 2026-05-08 01:02:55 -04:00
nodes_sam3.py Update nodes titles (#14417) 2026-06-16 11:42:00 +08:00
nodes_save_3d.py feat: add PreviewGaussianSplat + PreviewPointCloud nodes (#14194) 2026-06-05 12:30:58 -07:00
nodes_scail.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_sd3.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_sdpose.py Add the checkbox to disable head drawing in node SDPoseDrawKeypoints (#14446) 2026-06-16 16:21:04 +08:00
nodes_sdupscale.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_slg.py feat: mark 429 widgets as advanced for collapsible UI (#12197) 2026-02-19 19:20:02 -08:00
nodes_stable3d.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_stable_cascade.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_string.py Adding new StringFormat node (#13997) 2026-05-20 10:25:49 +08:00
nodes_tcfg.py convert nodes_tcfg.py to V3 schema (#9942) 2025-09-26 14:13:05 -07:00
nodes_textgen.py feat: Support text generation with Qwen3-VL (CORE-276) (#14298) 2026-06-17 08:12:44 +08:00
nodes_tomesd.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_toolkit.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_torch_compile.py Update nodes categories and display names (CORE-89) (#13786) 2026-05-08 01:02:55 -04:00
nodes_train.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_triposplat.py Do tripo dinov3 inference in fp32. (#14221) 2026-06-01 18:08:20 -07:00
nodes_upscale_model.py chore: Update nodes categories (#14145) 2026-05-27 20:43:33 -04:00
nodes_video_model.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_video.py Add 10-bit video support (#14452) 2026-06-13 16:05:25 +03:00
nodes_void.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_wan.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_wandancer.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_wanmove.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
nodes_webcam.py add search aliases to all nodes (#12035) 2026-01-22 18:36:58 -08:00
nodes_zimage.py chore: Update nodes categories (CORE-263) (#14460) 2026-06-17 08:33:09 +08:00
void_noise_warp.py Void model - pass 1 & 2 (CORE-38) (#13403) 2026-05-05 19:59:04 -07:00