wangbo/ComfyUI
1
0
Fork 0
mirror of https://github.com/comfyanonymous/ComfyUI.git synced 2026-07-18 20:38:15 +08:00
Code Issues Actions 27 Packages Projects Releases Wiki Activity

Merge branch 'master' into pixal3d

Browse Source 
# Conflicts:
#	comfy/sd.py
#	comfy_extras/nodes_save_3d.py
This commit is contained in:
kijai 2026-07-17 14:24:51 +03:00
commit 550af28f45
129 changed files with 11430 additions and 543 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

93
.github/workflows/cla.yml vendored Normal file
View File

@ -0,0 +1,93 @@
name: CLA Assistant
on:
issue_comment:
types: [created]
pull_request_target:
types: [opened, synchronize, closed]
permissions:
actions: write
contents: read # 'read' is enough because signatures live in a REMOTE repo
pull-requests: write
statuses: write
jobs:
cla-assistant:
runs-on: ubuntu-latest
steps:
# The CLA action normally requires every commit author in a PR to sign.
# We only want the PR author to sign, so we allowlist all other committers
# by computing them from the PR's commits and excluding the PR author.
- name: Build author-only allowlist
id: allowlist
if: >
github.event_name == 'pull_request_target' ||
(github.event_name == 'issue_comment' && github.event.issue.pull_request && (
github.event.comment.body == 'recheck' ||
github.event.comment.body == 'I have read and agree to the Contributor License Agreement'
))
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_NUMBER: ${{ github.event.pull_request.number || github.event.issue.number }}
PR_AUTHOR: ${{ github.event.pull_request.user.login || github.event.issue.user.login }}
BASE_ALLOWLIST: action@github.com,actions-user,ampagent,claude,comfy-pr-bot,GitHub Action,github-actions,github-actions[bot],Glary Bot,Glary-Bot,*[bot]
# For each commit emit the GitHub login when the author/committer email resolves to a GitHub account
# otherwise fall back to the raw git name.
run: |
others=$(gh api "repos/${{ github.repository }}/pulls/${PR_NUMBER}/commits" --paginate \
--jq '.[] | (.author.login // .commit.author.name // empty), (.committer.login // .commit.committer.name // empty)' \
| sort -u | grep -vix "${PR_AUTHOR}" | paste -sd, -)
if [ -n "$others" ]; then
echo "allowlist=${BASE_ALLOWLIST},${others}" >> "$GITHUB_OUTPUT"
else
echo "allowlist=${BASE_ALLOWLIST}" >> "$GITHUB_OUTPUT"
fi
- name: CLA Assistant
# Run on PR events, on "recheck" comment, or when someone posts the signing phrase.
# IMPORTANT: this phrase must match `custom-pr-sign-comment` below.
if: >
github.event_name == 'pull_request_target' ||
(github.event_name == 'issue_comment' && github.event.issue.pull_request && (
github.event.comment.body == 'recheck' ||
github.event.comment.body == 'I have read and agree to the Contributor License Agreement'
))
uses: contributor-assistant/github-action@ca4a40a7d1004f18d9960b404b97e5f30a505a08 # v2.6.1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# PAT required to write to the centralized signatures repo.
PERSONAL_ACCESS_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
with:
# Where the CLA document lives (shown to contributors)
path-to-document: https://github.com/Comfy-Org/comfy-cla/blob/main/comfyui_icla.md
# Centralized signature storage
remote-organization-name: comfy-org
remote-repository-name: comfy-cla
path-to-signatures: signatures/cla.json
branch: main
# Only the PR author must sign: bots plus every non-author committer
# are allowlisted via the "Build author-only allowlist" step above.
# *[bot] is a catch-all for any GitHub App bot account.
allowlist: ${{ steps.allowlist.outputs.allowlist }}
# Custom PR comment messages
custom-notsigned-prcomment: |
🎉 Thank you for your contribution, we really appreciate it! 🎉
Like many open source projects, we require contributors to sign our [Contributor License Agreement (CLA)](https://github.com/Comfy-Org/comfy-cla/blob/main/comfyui_icla.md). A CLA makes the ownership of contributions explicit, so contributors and the project share a clear understanding of how the code can be used. By signing, you:
- Confirm that you own your contribution.
- Keep the right to reuse your own code.
- Grant us a copyright license to include and share it within our projects.
CLAs are standard practice across major open source projects including those under the Apache Software Foundation and the Linux Foundation. Ours is based on the Apache Software Foundation's CLA. Most importantly, it would enable us to relicense the project under a more permissive license in the future, giving the project and its community greater flexibility.
✍ **To sign, please post a new comment on this PR with exactly the following text:** ✍
custom-pr-sign-comment: I have read and agree to the Contributor License Agreement
custom-allsigned-prcomment: |
✅ All contributors have signed the CLA. Thank you! This PR is ready to be merged.

43
AGENTS.md
View File

@ -19,6 +19,9 @@
better to remove a broken feature path than keep a complicated partial fix.
- Preserve existing APIs, node names, model-loading behavior, file layout, and
workflow compatibility unless the change is explicitly about replacing them.
- When compatibility is explicitly out of scope, remove compatibility-only
aliases, duplicate nodes, legacy entry points, and preset wrappers instead of
retaining parallel ways to perform the same operation.
- Code must look hand-written for this repository. Changes that read like
generic AI-generated code will be rejected automatically: unnecessary helper
layers, vague names, boilerplate comments, defensive branches without a real
@ -96,6 +99,13 @@
unless they are read by current code and change current behavior. Remove
pass-through or stored-but-unused values instead of preserving upstream or
deprecated API baggage.
- Do not add a model-specific option to a shared helper when only one caller
needs it. Keep one-off behavior at the model integration boundary, or extend
the shared helper only when the option is a coherent reusable capability.
- Implementations of shared model interfaces should accept the standard caller
contract without model-specific rejection branches for optional capabilities
they do not consume. Let supported behavior be determined by implementation
paths that actually use those inputs.
- If an implementation needs auxiliary values for its own workflow, expose them
through a private helper or a clearly named implementation-specific method
instead of overloading the public method's return contract.
@ -127,6 +137,8 @@
- Do not add unnecessary `try`/`except` blocks. Use them for optional dependency,
platform, or backend capability detection only when the program has a useful
fallback. Prefer specific exception types when changing new code.
- If a library version is pinned in `requirements.txt`, do not add code to
ComfyUI to handle older versions of that library.
- Remove any workarounds for PyTorch versions that ComfyUI no longer officially
supports. Deprecated workarounds include catching an exception and rerunning
the same op with the input cast to float. If a workaround does not have a
@ -152,6 +164,10 @@
`comfy-kitchen` helpers where they already solve the problem.
- Use optimized comfy-kitchen ops in places where they improve performance
without changing the expected dtype, device, memory, or interface behavior.
- Prefer ComfyUI's shared optimized kernels and backend dispatchers over
handwritten implementations of the same operation. Remove duplicate local
kernels and adapt inputs to the shared operation's documented layout while
preserving the model's original math and output contract.
- All models should use the optimized attention function selected by ComfyUI.
Treat optimized backend functions, dispatch helpers, and capability-selected
callables as opaque. Higher-level code must not inspect function identity,
@ -174,6 +190,12 @@
- Model detection code that inspects linear weight shapes should only use the
first dimension. The second dimension may be half the original size for
NVFP4 or other 4-bit quantized models.
- A model-detection signature must guard every state-dict key it dereferences.
Do not partially match a format and then raise an incidental `KeyError` while
extracting its configuration.
- Order model-detection checks from established or more-specific signatures to
newer or broader signatures. Put a broad new detector near the generic
fallback when giving it higher precedence could steal another model family.
- Avoid adding `einops` usage in core inference code. Use native torch tensor
ops such as `reshape`, `view`, `permute`, `transpose`, `flatten`, `unflatten`,
`unsqueeze`, and `squeeze` instead.
@ -190,11 +212,23 @@
methods for scalar or structural calculations.
- Avoid unnecessary casts and transfers. Preserve the intended compute dtype,
storage dtype, bias dtype, and original tensor shape metadata.
- Do not cast the result of an optimized backend operation back to its input
dtype unless that backend's documented result contract requires normalization.
In particular, trust the selected optimized-attention implementation to honor
its dtype contract.
- Keep model-native latent layout handling inside the model or latent-format
owner, not in helper nodes. Do not collapse, expand, pack, or unpack latent
dimensions in nodes or other caller-side adapters just to satisfy a model
forward; the model path should consume and return the native latent shape for
that model family.
- DiT models should accept latent dimensions that are not exact patch-size
multiples. Use `comfy.ldm.common_dit.pad_to_patch_size` on every patchified
target or reference input, then crop only the target output back to its
original dimensions.
- Avoid defensive shape and configuration checks that merely replace the clear
failure from the tensor operation immediately below them. Add explicit
validation only when it provides materially better context at a real boundary
or prevents silent incorrect output.
- Assume inputs to the main model forward are already in the compute dtype by
default, except integer inputs such as some model timestep tensors. Do not add
defensive or convenience casts in model code; it is better for invalid dtype
@ -258,6 +292,15 @@
- Model implementations should add the minimal number of ComfyUI nodes required
to run the model. Reuse existing nodes as much as possible; adapting the model
to work with existing nodes is strongly preferred over creating new nodes.
- Use `io.Autogrow` for a variable number of repeated inputs instead of a fixed
series of numbered optional sockets. Set its minimum to zero when the model
has a valid no-item path, and cap it only when the model has a real limit.
- Mark inputs optional when execution has a valid path that does not read them.
If one optional input is needed only to process another optional input, do not
force users on the path that supplies neither to connect it.
- Conditioning nodes should normally output conditioning only. Do not expose
input or intermediate images as convenience outputs for downstream sizing or
routing; use the existing image path or a dedicated image operation instead.
- Nodes should output only values they own. Do not add pass-through outputs for
workflow convenience unless the node is explicitly an output node. Existing
models, latents, conditioning, or other inputs should flow directly to the

1
CLAUDE.md
View File

@ -1 +0,0 @@
AGENTS.md

2
README.md
View File

@ -229,7 +229,7 @@ Python 3.14 works but some custom nodes may have issues. The free threaded varia
Python 3.13 is very well supported. If you have trouble with some custom node dependencies on 3.13 you can try 3.12
torch 2.4 and above is supported but some features and optimizations might only work on newer versions. We generally recommend using the latest major version of pytorch with the latest cuda version unless it is less than 2 weeks old.
torch 2.5 is minimally supported but using a newer version is extremely recommended. Some features and optimizations might only work on newer versions. We generally recommend using the latest major version of pytorch with the latest cuda version unless it is less than 2 weeks old. If your pytorch is more than 6 months old, please update it.
### Instructions:

107
alembic_db/versions/0005_allow_case_sensitive_tags.py Normal file
View File

@ -0,0 +1,107 @@
"""
Allow case-sensitive tag names.
Revision ID: 0005_allow_case_sensitive_tags
Revises: 0004_drop_tag_type
Create Date: 2026-06-16
"""
import sqlalchemy as sa
from alembic import op
revision = "0005_allow_case_sensitive_tags"
down_revision = "0004_drop_tag_type"
branch_labels = None
depends_on = None
def upgrade() -> None:
bind = op.get_bind()
if bind.dialect.name == "sqlite":
# SQLite cannot ALTER/DROP CHECK constraints. Recreate the small tag
# vocabulary table without the lowercase constraint while preserving
# existing tag names.
op.execute("PRAGMA foreign_keys=OFF")
try:
op.execute(
"CREATE TABLE tags_new ("
"name VARCHAR(512) NOT NULL, "
"CONSTRAINT pk_tags PRIMARY KEY (name)"
")"
)
op.execute("INSERT INTO tags_new(name) SELECT name FROM tags")
op.execute("DROP TABLE tags")
op.execute("ALTER TABLE tags_new RENAME TO tags")
finally:
op.execute("PRAGMA foreign_keys=ON")
return
op.drop_constraint("ck_tags_ck_tags_lowercase", "tags", type_="check")
def downgrade() -> None:
# Existing mixed-case tags cannot satisfy the old constraint. Lowercase them
# before restoring it, merging duplicate vocabulary/link rows that collide.
bind = op.get_bind()
tag_names = [row[0] for row in bind.execute(sa.text("SELECT name FROM tags"))]
existing_names = set(tag_names)
lowercase_names = sorted({name.lower() for name in tag_names})
missing_lowercase_rows = [
{"name": name} for name in lowercase_names if name not in existing_names
]
if missing_lowercase_rows:
bind.execute(sa.text("INSERT INTO tags(name) VALUES (:name)"), missing_lowercase_rows)
link_rows = bind.execute(
sa.text(
"SELECT asset_reference_id, tag_name, origin, added_at "
"FROM asset_reference_tags "
"ORDER BY asset_reference_id, tag_name"
)
).mappings()
deduped_links = {}
for row in link_rows:
key = (row["asset_reference_id"], row["tag_name"].lower())
deduped_links.setdefault(
key,
{
"asset_reference_id": row["asset_reference_id"],
"tag_name": row["tag_name"].lower(),
"origin": row["origin"],
"added_at": row["added_at"],
},
)
op.execute("DELETE FROM asset_reference_tags")
if deduped_links:
bind.execute(
sa.text(
"INSERT INTO asset_reference_tags "
"(asset_reference_id, tag_name, origin, added_at) "
"VALUES (:asset_reference_id, :tag_name, :origin, :added_at)"
),
list(deduped_links.values()),
)
op.execute("DELETE FROM tags WHERE name != lower(name)")
if bind.dialect.name == "sqlite":
op.execute("PRAGMA foreign_keys=OFF")
try:
op.execute(
"CREATE TABLE tags_new ("
"name VARCHAR(512) NOT NULL, "
"CONSTRAINT pk_tags PRIMARY KEY (name), "
"CONSTRAINT ck_tags_lowercase CHECK (name = lower(name))"
")"
)
op.execute("INSERT INTO tags_new(name) SELECT name FROM tags")
op.execute("DROP TABLE tags")
op.execute("ALTER TABLE tags_new RENAME TO tags")
finally:
op.execute("PRAGMA foreign_keys=ON")
return
op.create_check_constraint(
"ck_tags_ck_tags_lowercase", "tags", "name = lower(name)"
)

30
alembic_db/versions/0006_add_loader_path.py Normal file
View File

@ -0,0 +1,30 @@
"""
Add loader_path column to asset_references.
Stores the in-root loader path (path relative to the storage root with the
top-level model category dropped) derived from file_path at scan/ingest time,
so the assets API can return it without re-resolving against every registered
model-folder base on every request.
Revision ID: 0006_add_loader_path
Revises: 0005_allow_case_sensitive_tags
Create Date: 2026-07-02
"""
from alembic import op
import sqlalchemy as sa
revision = "0006_add_loader_path"
down_revision = "0005_allow_case_sensitive_tags"
branch_labels = None
depends_on = None
def upgrade() -> None:
with op.batch_alter_table("asset_references") as batch_op:
batch_op.add_column(sa.Column("loader_path", sa.Text(), nullable=True))
def downgrade() -> None:
with op.batch_alter_table("asset_references") as batch_op:
batch_op.drop_column("loader_path")

22
app/assets/api/routes.py
View File

@ -40,6 +40,7 @@ from app.assets.services import (
upload_from_temp_path,
)
from app.assets.services.cursor import InvalidCursorError
from app.assets.services.path_utils import compute_display_name
from app.assets.services.tagging import list_tag_histogram
ROUTES = web.RouteTableDef()
@ -161,11 +162,19 @@ def _build_asset_response(result: schemas.AssetDetailResult | schemas.UploadResu
preview_url = None
else:
preview_url = _build_preview_url_from_view(result.tags, result.ref.user_metadata)
if result.ref.file_path:
display_name = compute_display_name(result.ref.file_path)
# In-root loader path (model category dropped): what model loaders consume.
loader_path = result.ref.loader_path
else:
display_name, loader_path = None, None
asset_content_hash = result.asset.hash if result.asset else None
return schemas_out.Asset(
id=result.ref.id,
name=result.ref.name,
hash=asset_content_hash,
loader_path=loader_path,
display_name=display_name,
asset_hash=asset_content_hash,
size=int(result.asset.size_bytes) if result.asset else None,
mime_type=result.asset.mime_type if result.asset else None,
@ -419,17 +428,6 @@ async def upload_asset(request: web.Request) -> web.Response:
400, "INVALID_BODY", f"Validation failed: {ve.json()}"
)
if spec.tags and spec.tags[0] == "models":
if (
len(spec.tags) < 2
or spec.tags[1] not in folder_paths.folder_names_and_paths
):
delete_temp_file_if_exists(parsed.tmp_path)
category = spec.tags[1] if len(spec.tags) >= 2 else ""
return _build_error_response(
400, "INVALID_BODY", f"unknown models category '{category}'"
)
try:
# Fast path: hash exists, create AssetReference without writing anything
if spec.hash and parsed.provided_hash_exists is True:
@ -473,7 +471,7 @@ async def upload_asset(request: web.Request) -> web.Response:
return _build_error_response(400, e.code, str(e))
except ValueError as e:
delete_temp_file_if_exists(parsed.tmp_path)
return _build_error_response(400, "BAD_REQUEST", str(e))
return _build_error_response(400, "INVALID_BODY", str(e))
except HashMismatchError as e:
delete_temp_file_if_exists(parsed.tmp_path)
return _build_error_response(400, "HASH_MISMATCH", str(e))

24
app/assets/api/schemas_in.py
View File

@ -140,7 +140,7 @@ class CreateFromHashBody(BaseModel):
if v is None:
return []
if isinstance(v, list):
out = [str(t).strip().lower() for t in v if str(t).strip()]
out = [str(t).strip() for t in v if str(t).strip()]
seen = set()
dedup = []
for t in out:
@ -149,7 +149,7 @@ class CreateFromHashBody(BaseModel):
dedup.append(t)
return dedup
if isinstance(v, str):
return [t.strip().lower() for t in v.split(",") if t.strip()]
return list(dict.fromkeys(t.strip() for t in v.split(",") if t.strip()))
return []
@ -206,7 +206,7 @@ class TagsListQuery(BaseModel):
if v is None:
return v
v = v.strip()
return v.lower() or None
return v or None
class TagsAdd(BaseModel):
@ -220,7 +220,7 @@ class TagsAdd(BaseModel):
for t in v:
if not isinstance(t, str):
raise TypeError("tags must be strings")
tnorm = t.strip().lower()
tnorm = t.strip()
if tnorm:
out.append(tnorm)
seen = set()
@ -239,8 +239,8 @@ class TagsRemove(TagsAdd):
class UploadAssetSpec(BaseModel):
"""Upload Asset operation.
- tags: optional list; if provided, first is root ('models'|'input'|'output');
if root == 'models', second must be a valid category
- tags: labels plus one destination role ('models'|'input'|'output') for new bytes;
if role == 'models', exactly one model_type:<folder_name> tag is required
- name: display name
- user_metadata: arbitrary JSON object (optional)
- hash: optional canonical 'blake3:<hex>' for validation / fast-path
@ -309,7 +309,7 @@ class UploadAssetSpec(BaseModel):
norm = []
seen = set()
for t in items:
tnorm = str(t).strip().lower()
tnorm = str(t).strip()
if tnorm and tnorm not in seen:
seen.add(tnorm)
norm.append(tnorm)
@ -335,14 +335,4 @@ class UploadAssetSpec(BaseModel):
@model_validator(mode="after")
def _validate_order(self):
if not self.tags:
raise ValueError("at least one tag is required for uploads")
root = self.tags[0]
if root not in {"models", "input", "output"}:
raise ValueError("first tag must be one of: models, input, output")
if root == "models":
if len(self.tags) < 2:
raise ValueError(
"models uploads require a category tag as the second tag"
)
return self

14
app/assets/api/schemas_out.py
View File

@ -9,8 +9,20 @@ class Asset(BaseModel):
``id`` here is the AssetReference id, not the content-addressed Asset id."""
id: str
name: str
name: str = Field(
...,
deprecated=True,
description="Reference label, often caller-provided or derived from the filename. Deprecated for storage path/display semantics; use `loader_path` and `display_name` when present.",
)
hash: str | None = None
loader_path: str | None = Field(
default=None,
description="The value a loader consumes to load this asset. `None` when no loader can resolve the file.",
)
display_name: str | None = Field(
default=None,
description="Human-facing label for the asset. Not unique.",
)
asset_hash: str | None = None
size: int | None = None
mime_type: str | None = None

1
app/assets/api/upload.py
View File

@ -140,7 +140,6 @@ async def parse_multipart_upload(
provided_mime_type = ((await field.text()) or "").strip() or None
elif fname == "preview_id":
provided_preview_id = ((await field.text()) or "").strip() or None
if not file_present and not (provided_hash and provided_hash_exists):
raise UploadError(
400, "MISSING_FILE", "Form must include a 'file' part or a known 'hash'."

2
app/assets/database/models.py
View File

@ -76,6 +76,8 @@ class AssetReference(Base):
# Cache state fields (from former AssetCacheState)
file_path: Mapped[str | None] = mapped_column(Text, nullable=True)
# In-root loader path derived from file_path at scan/ingest time.
loader_path: Mapped[str | None] = mapped_column(Text, nullable=True)
mtime_ns: Mapped[int | None] = mapped_column(BigInteger, nullable=True)
needs_verify: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
is_missing: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)

7
app/assets/database/queries/asset_reference.py
View File

@ -650,6 +650,7 @@ def upsert_reference(
name: str,
mtime_ns: int,
owner_id: str = "",
loader_path: str | None = None,
) -> tuple[bool, bool]:
"""Upsert a reference by file_path. Returns (created, updated).
@ -659,6 +660,7 @@ def upsert_reference(
vals = {
"asset_id": asset_id,
"file_path": file_path,
"loader_path": loader_path,
"name": name,
"owner_id": owner_id,
"mtime_ns": int(mtime_ns),
@ -686,13 +688,14 @@ def upsert_reference(
AssetReference.asset_id != asset_id,
AssetReference.mtime_ns.is_(None),
AssetReference.mtime_ns != int(mtime_ns),
AssetReference.loader_path.is_distinct_from(loader_path),
AssetReference.is_missing == True, # noqa: E712
AssetReference.deleted_at.isnot(None),
)
)
.values(
asset_id=asset_id, mtime_ns=int(mtime_ns), is_missing=False,
deleted_at=None, updated_at=now,
asset_id=asset_id, mtime_ns=int(mtime_ns), loader_path=loader_path,
is_missing=False, deleted_at=None, updated_at=now,
)
)
res2 = session.execute(upd)

12
app/assets/database/queries/tags.py
View File

@ -265,6 +265,8 @@ def list_tags_with_usage(
order: str = "count_desc",
owner_id: str = "",
) -> tuple[list[tuple[str, str, int]], int]:
prefix_filter = prefix.strip() if prefix else ""
counts_sq = (
select(
AssetReferenceTag.tag_name.label("tag_name"),
@ -293,9 +295,8 @@ def list_tags_with_usage(
.join(counts_sq, counts_sq.c.tag_name == Tag.name, isouter=True)
)
if prefix:
escaped, esc = escape_sql_like_string(prefix.strip().lower())
q = q.where(Tag.name.like(escaped + "%", escape=esc))
if prefix_filter:
q = q.where(func.substr(Tag.name, 1, len(prefix_filter)) == prefix_filter)
if not include_zero:
q = q.where(func.coalesce(counts_sq.c.cnt, 0) > 0)
@ -306,9 +307,8 @@ def list_tags_with_usage(
q = q.order_by(func.coalesce(counts_sq.c.cnt, 0).desc(), Tag.name.asc())
total_q = select(func.count()).select_from(Tag)
if prefix:
escaped, esc = escape_sql_like_string(prefix.strip().lower())
total_q = total_q.where(Tag.name.like(escaped + "%", escape=esc))
if prefix_filter:
total_q = total_q.where(func.substr(Tag.name, 1, len(prefix_filter)) == prefix_filter)
if not include_zero:
visible_tags_sq = (
select(AssetReferenceTag.tag_name)

6
app/assets/helpers.py
View File

@ -41,10 +41,10 @@ def get_utc_now() -> datetime:
def normalize_tags(tags: list[str] | None) -> list[str]:
"""
Normalize a list of tags by:
- Stripping whitespace and converting to lowercase.
- Removing duplicates.
- Stripping whitespace.
- Removing exact duplicates while preserving order and case.
"""
return list(dict.fromkeys(t.strip().lower() for t in (tags or []) if (t or "").strip()))
return list(dict.fromkeys(t.strip() for t in (tags or []) if (t or "").strip()))
def validate_blake3_hash(s: str) -> str:

10
app/assets/scanner.py
View File

@ -36,7 +36,7 @@ from app.assets.services.hashing import HashCheckpoint, compute_blake3_hash
from app.assets.services.image_dimensions import extract_image_dimensions
from app.assets.services.metadata_extract import extract_file_metadata
from app.assets.services.path_utils import (
compute_relative_filename,
compute_loader_path,
get_comfy_models_folders,
get_name_and_tags_from_asset_path,
)
@ -63,7 +63,7 @@ RootType = Literal["models", "input", "output"]
def get_prefixes_for_root(root: RootType) -> list[str]:
if root == "models":
bases: list[str] = []
for _bucket, paths in get_comfy_models_folders():
for _bucket, paths, _exts in get_comfy_models_folders():
bases.extend(paths)
return [os.path.abspath(p) for p in bases]
if root == "input":
@ -81,7 +81,7 @@ def get_all_known_prefixes() -> list[str]:
def collect_models_files() -> list[str]:
out: list[str] = []
for folder_name, bases in get_comfy_models_folders():
for folder_name, bases, _exts in get_comfy_models_folders():
rel_files = folder_paths.get_filename_list(folder_name) or []
for rel_path in rel_files:
if not all(is_visible(part) for part in Path(rel_path).parts):
@ -308,7 +308,7 @@ def build_asset_specs(
if not stat_p.st_size:
continue
name, tags = get_name_and_tags_from_asset_path(abs_p)
rel_fname = compute_relative_filename(abs_p)
rel_fname = compute_loader_path(abs_p)
# Extract metadata (tier 1: filesystem, tier 2: safetensors header)
metadata = None
@ -430,7 +430,7 @@ def enrich_asset(
return new_level
initial_mtime_ns = get_mtime_ns(stat_p)
rel_fname = compute_relative_filename(file_path)
rel_fname = compute_loader_path(file_path)
mime_type: str | None = None
metadata = None

4
app/assets/services/asset_management.py
View File

@ -38,7 +38,7 @@ from app.assets.database.queries import (
update_reference_updated_at,
)
from app.assets.helpers import select_best_live_path
from app.assets.services.path_utils import compute_relative_filename
from app.assets.services.path_utils import compute_loader_path
from app.assets.services.schemas import (
AssetData,
AssetDetailResult,
@ -91,7 +91,7 @@ def update_asset_metadata(
update_reference_name(session, reference_id=reference_id, name=name)
touched = True
computed_filename = compute_relative_filename(ref.file_path) if ref.file_path else None
computed_filename = compute_loader_path(ref.file_path) if ref.file_path else None
new_meta: dict | None = None
if user_metadata is not None:

11
app/assets/services/bulk_ingest.py
View File

@ -56,6 +56,7 @@ class ReferenceRow(TypedDict):
id: str
asset_id: str
file_path: str
loader_path: str | None
mtime_ns: int
owner_id: str
name: str
@ -134,6 +135,14 @@ def batch_insert_seed_assets(
for spec in specs:
absolute_path = os.path.abspath(spec["abs_path"])
existing_asset_id = path_to_asset_id.get(absolute_path)
if existing_asset_id is not None:
existing_tags = asset_id_to_ref_data[existing_asset_id]["tags"]
asset_id_to_ref_data[existing_asset_id]["tags"] = list(
dict.fromkeys([*existing_tags, *spec["tags"]])
)
continue
asset_id = str(uuid.uuid4())
reference_id = str(uuid.uuid4())
absolute_path_list.append(absolute_path)
@ -164,6 +173,8 @@ def batch_insert_seed_assets(
"id": reference_id,
"asset_id": asset_id,
"file_path": absolute_path,
# spec["fname"] is compute_loader_path(abs_path) from build_asset_specs.
"loader_path": spec["fname"],
"mtime_ns": spec["mtime_ns"],
"owner_id": owner_id,
"name": spec["info_name"],

61
app/assets/services/ingest.py
View File

@ -33,8 +33,9 @@ from app.assets.services.bulk_ingest import batch_insert_seed_assets
from app.assets.services.file_utils import get_size_and_mtime_ns
from app.assets.services.image_dimensions import extract_image_dimensions
from app.assets.services.path_utils import (
compute_relative_filename,
compute_loader_path,
get_name_and_tags_from_asset_path,
get_path_derived_tags_from_path,
resolve_destination_from_tags,
validate_path_within_base,
)
@ -91,6 +92,7 @@ def _ingest_file_from_path(
name=info_name or os.path.basename(locator),
mtime_ns=mtime_ns,
owner_id=owner_id,
loader_path=compute_loader_path(locator),
)
# Get the reference we just created/updated
@ -101,17 +103,32 @@ def _ingest_file_from_path(
if preview_id and ref.preview_id != preview_id:
ref.preview_id = preview_id
norm = normalize_tags(list(tags))
if norm:
try:
backend_tags = get_path_derived_tags_from_path(locator)
except ValueError:
backend_tags = []
caller_tags = normalize_tags(tags)
backend_tags = normalize_tags(backend_tags)
all_tags = normalize_tags([*caller_tags, *backend_tags])
if all_tags:
if require_existing_tags:
validate_tags_exist(session, norm)
add_tags_to_reference(
session,
reference_id=reference_id,
tags=norm,
origin=tag_origin,
create_if_missing=not require_existing_tags,
)
validate_tags_exist(session, all_tags)
if backend_tags:
add_tags_to_reference(
session,
reference_id=reference_id,
tags=backend_tags,
origin="automatic",
create_if_missing=not require_existing_tags,
)
if caller_tags:
add_tags_to_reference(
session,
reference_id=reference_id,
tags=caller_tags,
origin=tag_origin,
create_if_missing=not require_existing_tags,
)
_update_metadata_with_filename(
session,
@ -228,7 +245,7 @@ def ingest_existing_file(
"mtime_ns": mtime_ns,
"info_name": name,
"tags": tags,
"fname": os.path.basename(abs_path),
"fname": compute_loader_path(abs_path),
"metadata": None,
"hash": None,
"mime_type": mime_type,
@ -288,7 +305,7 @@ def _register_existing_asset(
return result
new_meta = dict(user_metadata)
computed_filename = compute_relative_filename(ref.file_path) if ref.file_path else None
computed_filename = compute_loader_path(ref.file_path) if ref.file_path else None
if computed_filename:
new_meta["filename"] = computed_filename
@ -335,7 +352,7 @@ def _update_metadata_with_filename(
current_metadata: dict | None,
user_metadata: dict[str, Any],
) -> None:
computed_filename = compute_relative_filename(file_path) if file_path else None
computed_filename = compute_loader_path(file_path) if file_path else None
current_meta = current_metadata or {}
new_meta = dict(current_meta)
@ -474,6 +491,10 @@ def upload_from_temp_path(
existing = get_asset_by_hash(session, asset_hash=asset_hash)
if existing is not None:
# Once content is already known, duplicate byte uploads are treated as
# reference-only creation. Request tags are labels only here: do not
# require upload destination tags, do not move bytes, and do not
# synthesize path-derived classification or uploaded provenance.
with contextlib.suppress(Exception):
if temp_path and os.path.exists(temp_path):
os.remove(temp_path)
@ -535,7 +556,7 @@ def upload_from_temp_path(
owner_id=owner_id,
preview_id=preview_id,
user_metadata=user_metadata or {},
tags=tags,
tags=[*(tags or []), "uploaded"],
tag_origin="manual",
require_existing_tags=False,
)
@ -569,15 +590,19 @@ def register_file_in_place(
) -> UploadResult:
"""Register an already-saved file in the asset database without moving it.
Tags are derived from the filesystem path (root category + subfolder names),
merged with any caller-provided tags, matching the behavior of the scanner.
This helper is used by upload paths that have already written bytes before
registering the file, so it records the same ``uploaded`` tag as the
multipart byte-upload path.
Tags are derived from trusted filesystem classification and merged with any
caller-provided tags, matching the behavior of the scanner.
If the path is not under a known root, only the caller-provided tags are used.
"""
try:
_, path_tags = get_name_and_tags_from_asset_path(abs_path)
except ValueError:
path_tags = []
merged_tags = normalize_tags([*path_tags, *tags])
merged_tags = normalize_tags([*path_tags, *tags, "uploaded"])
try:
digest, _ = hashing.compute_blake3_hash(abs_path)

249
app/assets/services/path_utils.py
View File

@ -3,59 +3,66 @@ from pathlib import Path
from typing import Literal
import folder_paths
from app.assets.helpers import normalize_tags
_NON_MODEL_FOLDER_NAMES = frozenset({"custom_nodes"})
_NON_MODEL_FOLDER_NAMES = frozenset({"configs", "custom_nodes"})
_KNOWN_SUBFOLDER_TAGS = frozenset({"3d", "pasted", "painter", "threed", "webcam"})
def get_comfy_models_folders() -> list[tuple[str, list[str]]]:
"""Build list of (folder_name, base_paths[]) for all model locations.
def get_comfy_models_folders() -> list[tuple[str, list[str], set[str]]]:
"""Build list of (folder_name, base_paths[], extensions) for all model locations.
Includes every category registered in folder_names_and_paths,
regardless of whether its paths are under the main models_dir,
but excludes non-model entries like custom_nodes.
but excludes non-model entries like configs and custom_nodes.
An empty extensions set means the category accepts any extension,
matching folder_paths.filter_files_extensions semantics.
"""
targets: list[tuple[str, list[str]]] = []
targets: list[tuple[str, list[str], set[str]]] = []
for name, values in folder_paths.folder_names_and_paths.items():
if name in _NON_MODEL_FOLDER_NAMES:
continue
paths, _exts = values[0], values[1]
paths, exts = values[0], values[1]
if paths:
targets.append((name, paths))
targets.append((name, paths, set(exts)))
return targets
def resolve_destination_from_tags(tags: list[str]) -> tuple[str, list[str]]:
"""Validates and maps tags -> (base_dir, subdirs_for_fs)"""
if not tags:
raise ValueError("tags must not be empty")
root = tags[0].lower()
"""Validates and maps upload routing tags -> (base_dir, subdirs_for_fs).
The request tags are only used to choose the write destination. Extra tags
remain labels; they do not become path components or trusted classification.
"""
destination_roles = [t for t in tags if t in {"input", "models", "output"}]
if len(destination_roles) != 1:
raise ValueError("uploads require exactly one destination role: input, models, or output")
root = destination_roles[0]
if root == "models":
if len(tags) < 2:
raise ValueError("at least two tags required for model asset")
model_type_tags = [t for t in tags if t.startswith("model_type:")]
if len(model_type_tags) != 1:
raise ValueError("models uploads require exactly one model_type:<folder_name> tag")
folder_name = model_type_tags[0].split(":", 1)[1]
if not folder_name:
raise ValueError("models uploads require exactly one model_type:<folder_name> tag")
model_folder_paths = {
name: paths for name, paths, _exts in get_comfy_models_folders()
}
try:
bases = folder_paths.folder_names_and_paths[tags[1]][0]
bases = model_folder_paths[folder_name]
except KeyError:
raise ValueError(f"unknown model category '{tags[1]}'")
raise ValueError(f"unknown model category '{folder_name}'")
if not bases:
raise ValueError(f"no base path configured for category '{tags[1]}'")
raise ValueError(f"no base path configured for category '{folder_name}'")
base_dir = os.path.abspath(bases[0])
raw_subdirs = tags[2:]
elif root == "input":
base_dir = os.path.abspath(folder_paths.get_input_directory())
raw_subdirs = tags[1:]
elif root == "output":
base_dir = os.path.abspath(folder_paths.get_output_directory())
raw_subdirs = tags[1:]
else:
raise ValueError(f"unknown root tag '{tags[0]}'; expected 'models', 'input', or 'output'")
_sep_chars = frozenset(("/", "\\", os.sep))
for i in raw_subdirs:
if i in (".", "..") or _sep_chars & set(i):
raise ValueError("invalid path component in tags")
base_dir = os.path.abspath(folder_paths.get_output_directory())
return base_dir, raw_subdirs if raw_subdirs else []
return base_dir, []
def validate_path_within_base(candidate: str, base: str) -> None:
@ -65,14 +72,79 @@ def validate_path_within_base(candidate: str, base: str) -> None:
raise ValueError("destination escapes base directory")
def compute_relative_filename(file_path: str) -> str | None:
def _compute_relative_path(child: str, parent: str) -> str:
rel = os.path.relpath(os.path.abspath(child), os.path.abspath(parent))
if rel == ".":
return ""
return rel.replace(os.sep, "/")
def _is_relative_to(child: str, parent: str) -> bool:
return Path(os.path.abspath(child)).is_relative_to(os.path.abspath(parent))
def compute_asset_response_paths(file_path: str) -> tuple[str, str | None] | None:
"""Return (logical_path, display_name) for a file path.
``logical_path`` is the internal namespaced storage locator (e.g.
``models/checkpoints/foo/bar.safetensors``); ``display_name`` is the
human-facing label below that namespace, served on Asset responses. These
are storage locators, not model-loader namespaces. Registered model-folder
membership is represented by backend tags such as
``model_type:<folder_name>``; these paths only use known storage roots.
"""
Return the model's path relative to the last well-known folder (the model category),
using forward slashes, eg:
fp_abs = os.path.abspath(file_path)
candidates: list[tuple[int, int, str, str]] = []
for order, (namespace, base) in enumerate(
(
("input", folder_paths.get_input_directory()),
("output", folder_paths.get_output_directory()),
("temp", folder_paths.get_temp_directory()),
("models", getattr(folder_paths, "models_dir", "")),
)
):
if not base:
continue
base_abs = os.path.abspath(base)
if _is_relative_to(fp_abs, base_abs):
candidates.append((len(base_abs), -order, namespace, base_abs))
if not candidates:
return None
_base_len, _order, namespace, base = max(candidates)
rel = _compute_relative_path(fp_abs, base)
public_path = f"{namespace}/{rel}" if rel else namespace
return public_path, rel or None
def compute_display_name(file_path: str) -> str | None:
"""Return the asset's `display_name`, or None for unknown paths."""
result = compute_asset_response_paths(file_path)
return result[1] if result else None
def compute_logical_path(file_path: str) -> str | None:
"""Return the internal namespaced storage locator, or None for unknown paths."""
result = compute_asset_response_paths(file_path)
return result[0] if result else None
def compute_loader_path(file_path: str) -> str | None:
"""
Return the asset's in-root loader path: the path relative to the last
well-known folder (the model category), using forward slashes, eg:
/.../models/checkpoints/flux/123/flux.safetensors -> "flux/123/flux.safetensors"
/.../models/text_encoders/clip_g.safetensors -> "clip_g.safetensors"
For non-model paths, returns None.
This is the value model loaders consume (the model category is dropped). It
is persisted as ``AssetReference.loader_path`` and served as the public
Asset response `loader_path` field. The human-facing `display_name` comes
from compute_asset_response_paths().
For input/output/temp paths the full path relative to that root is returned.
For paths outside any known root, returns None.
"""
try:
root_category, rel_path = get_asset_category_and_relative_path(file_path)
@ -116,9 +188,10 @@ def get_asset_category_and_relative_path(
def _compute_relative(child: str, parent: str) -> str:
# Normalize relative path, stripping any leading ".." components
# by anchoring to root (os.sep) then computing relpath back from it.
return os.path.relpath(
rel = os.path.relpath(
os.path.join(os.sep, os.path.relpath(child, parent)), os.sep
)
return "" if rel == "." else rel.replace(os.sep, "/")
# 1) input
input_base = os.path.abspath(folder_paths.get_input_directory())
@ -136,8 +209,14 @@ def get_asset_category_and_relative_path(
return "temp", _compute_relative(fp_abs, temp_base)
# 4) models (check deepest matching base to avoid ambiguity)
ext = os.path.splitext(fp_abs)[1].lower()
best: tuple[int, str, str] | None = None # (base_len, bucket, rel_inside_bucket)
for bucket, bases in get_comfy_models_folders():
for bucket, bases, extensions in get_comfy_models_folders():
# A bucket only lists files within its extension set (empty set
# accepts any extension), so a bucket that cannot load the file
# must not contribute a loader path.
if extensions and ext not in extensions:
continue
for b in bases:
base_abs = os.path.abspath(b)
if not _check_is_within(fp_abs, base_abs):
@ -149,25 +228,111 @@ def get_asset_category_and_relative_path(
if best is not None:
_, bucket, rel_inside = best
combined = os.path.join(bucket, rel_inside)
return "models", os.path.relpath(os.path.join(os.sep, combined), os.sep)
normalized = os.path.relpath(os.path.join(os.sep, combined), os.sep)
return "models", normalized.replace(os.sep, "/")
raise ValueError(
f"Path is not within input, output, temp, or configured model bases: {file_path}"
)
def get_backend_system_tags_from_path(path: str) -> list[str]:
"""Return trusted backend tags derived from current filesystem facts.
The returned tags are only the backend-generated system tags: ``models``,
``model_type:<folder_name>``, ``input``, ``output``, and ``temp``. Model
type tags are based on registered folder names, not path components.
A ``model_type:<folder_name>`` tag is only emitted when the file's
extension is accepted by that folder's registered extension set, so
categories sharing a base directory tag only the files they can
actually load. Files under a model base whose extension matches no
category still get the ``models`` tag.
"""
fp_abs = os.path.abspath(path)
fp_path = Path(fp_abs)
tags: list[str] = []
def _add(tag: str) -> None:
if tag not in tags:
tags.append(tag)
for role, base in (
("input", folder_paths.get_input_directory()),
("output", folder_paths.get_output_directory()),
("temp", folder_paths.get_temp_directory()),
):
if fp_path.is_relative_to(os.path.abspath(base)):
_add(role)
ext = os.path.splitext(fp_abs)[1].lower()
model_types: list[str] = []
under_models_base = False
for folder_name, bases, extensions in get_comfy_models_folders():
for base in bases:
if fp_path.is_relative_to(os.path.abspath(base)):
under_models_base = True
# Empty set accepts any extension, matching
# folder_paths.filter_files_extensions semantics.
if not extensions or ext in extensions:
model_types.append(folder_name)
break
if under_models_base:
_add("models")
for folder_name in model_types:
_add(f"model_type:{folder_name}")
if not tags:
raise ValueError(
f"Path is not within input, output, temp, or configured model bases: {path}"
)
return tags
def get_known_subfolder_tags(subfolder: str | None) -> list[str]:
"""Return tags for known UI/input subfolder names."""
if subfolder in _KNOWN_SUBFOLDER_TAGS:
return [subfolder]
return []
def get_known_input_subfolder_tags_from_path(path: str) -> list[str]:
"""Return known input-layout tags for files in canonical input subfolders.
These are compatibility tags for current UI-origin input directories such as
``pasted`` and ``webcam``. They are intentionally narrow: only files directly
inside a known top-level input directory receive the matching tag.
"""
fp_abs = os.path.abspath(path)
input_base = os.path.abspath(folder_paths.get_input_directory())
if not Path(fp_abs).is_relative_to(input_base):
return []
rel = os.path.relpath(fp_abs, input_base)
parts = Path(rel).parts
if len(parts) == 2:
return get_known_subfolder_tags(parts[0])
return []
def get_path_derived_tags_from_path(path: str) -> list[str]:
"""Return all backend-derived tags for an asset path."""
tags = get_backend_system_tags_from_path(path)
for tag in get_known_input_subfolder_tags_from_path(path):
if tag not in tags:
tags.append(tag)
return tags
def get_name_and_tags_from_asset_path(file_path: str) -> tuple[str, list[str]]:
"""Return (name, tags) derived from a filesystem path.
- name: base filename with extension
- tags: [root_category] + parent folder names in order
- tags: backend-derived tags from root/model classification and known input
subfolder layout conventions
Raises:
ValueError: path does not belong to any known root.
"""
root_category, some_path = get_asset_category_and_relative_path(file_path)
p = Path(some_path)
parent_parts = [
part for part in p.parent.parts if part not in (".", "..", p.anchor)
]
return p.name, list(dict.fromkeys(normalize_tags([root_category, *parent_parts])))
return Path(file_path).name, get_path_derived_tags_from_path(file_path)

2
app/assets/services/schemas.py
View File

@ -25,6 +25,7 @@ class ReferenceData:
preview_id: str | None
created_at: datetime
updated_at: datetime
loader_path: str | None = None
system_metadata: dict[str, Any] | None = None
job_id: str | None = None
last_access_time: datetime | None = None
@ -93,6 +94,7 @@ def extract_reference_data(ref: AssetReference) -> ReferenceData:
id=ref.id,
name=ref.name,
file_path=ref.file_path,
loader_path=ref.loader_path,
user_metadata=ref.user_metadata,
preview_id=ref.preview_id,
system_metadata=ref.system_metadata,

6
app/model_manager.py
View File

@ -35,7 +35,11 @@ class ModelFileManager:
for folder in model_types:
if folder in folder_black_list:
continue
output_folders.append({"name": folder, "folders": folder_paths.get_folder_paths(folder)})
output_folders.append({
"name": folder,
"folders": folder_paths.get_folder_paths(folder),
"extensions": sorted(folder_paths.folder_names_and_paths[folder][1]),
})
return web.json_response(output_folders)
# NOTE: This is an experiment to replace `/models/{folder}`

2
comfy/cli_args.py
View File

@ -92,6 +92,7 @@ parser.add_argument("--directml", type=int, nargs="?", metavar="DIRECTML_DEVICE"
parser.add_argument("--oneapi-device-selector", type=str, default=None, metavar="SELECTOR_STRING", help="Sets the oneAPI device(s) this instance will use.")
parser.add_argument("--supports-fp8-compute", action="store_true", help="ComfyUI will act like if the device supports fp8 compute.")
parser.add_argument("--enable-triton-backend", action="store_true", help="ComfyUI will enable the use of Triton backend in comfy-kitchen. Is disabled at launch by default.")
parser.add_argument("--disable-triton-backend", action="store_true", help="Force-disable the comfy-kitchen Triton backend, overriding the automatic ROCm/AMD default and --enable-triton-backend.")
class LatentPreviewMethod(enum.Enum):
NoPreviews = "none"
@ -225,6 +226,7 @@ parser.add_argument(
)
parser.add_argument("--user-directory", type=is_valid_directory, default=None, help="Set the ComfyUI user directory with an absolute path. Overrides --base-directory.")
parser.add_argument("--models-directory", type=is_valid_directory, default=None, help="Set the ComfyUI models directory. Overrides the models folder in --base-directory.")
parser.add_argument("--enable-compress-response-body", action="store_true", help="Enable compressing response body.")

46
comfy/comfy_api_env.py Normal file
View File

@ -0,0 +1,46 @@
"""Runtime config the frontend reads from /features to follow --comfy-api-base.
For a non-prod comfy.org backend (staging or an ephemeral preview env), "/features" exposes the api and
platform base so the frontend talks to it without a rebuild, plus the Firebase environment it should use.
Prod bases are left alone and keep their build-time defaults.
"""
from typing import Any
from urllib.parse import urlparse
from comfy.cli_args import args
_STAGING_API_HOST = "stagingapi.comfy.org"
_TESTENV_HOST_SUFFIX = ".testenvs.comfy.org"
_STAGING_PLATFORM_BASE_URL = "https://stagingplatform.comfy.org"
def _is_staging_tier(host: str) -> bool:
return host == _STAGING_API_HOST or host.endswith(_TESTENV_HOST_SUFFIX)
def normalize_comfy_api_base(url: str) -> str:
"""Rewrite a testenv's friendly main host to its comfy-api '-registry' sibling."""
parsed = urlparse(url)
host = parsed.hostname or ""
if not host.endswith(_TESTENV_HOST_SUFFIX):
return url
label = host[: -len(_TESTENV_HOST_SUFFIX)]
if label.endswith("-registry"):
return url
return f"{parsed.scheme or 'https'}://{label}-registry{_TESTENV_HOST_SUFFIX}"
def environment_overrides_for_base(base_url: str) -> dict[str, Any] | None:
"""The /features overrides for a staging-tier base, or None for prod."""
if not _is_staging_tier(urlparse(base_url).hostname or ""):
return None
return {
"comfy_api_base_url": normalize_comfy_api_base(base_url).rstrip("/"),
"comfy_platform_base_url": _STAGING_PLATFORM_BASE_URL,
"firebase_env": "dev",
}
def get_environment_overrides() -> dict[str, Any] | None:
return environment_overrides_for_base(getattr(args, "comfy_api_base", "") or "")

4
comfy/latent_formats.py
View File

@ -826,6 +826,10 @@ class ACEAudio(LatentFormat):
latent_channels = 8
latent_dimensions = 2
class SeedVR2(LatentFormat):
latent_channels = 16
latent_dimensions = 3
class ACEAudio15(LatentFormat):
latent_channels = 64
latent_dimensions = 1

7
comfy/ldm/ace/ace_step15.py
View File

@ -217,10 +217,7 @@ class AceStepAttention(nn.Module):
cos, sin = position_embeddings
query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
n_rep = self.num_heads // self.num_kv_heads
if n_rep > 1:
key_states = key_states.repeat_interleave(n_rep, dim=1)
value_states = value_states.repeat_interleave(n_rep, dim=1)
gqa_kwargs = {"enable_gqa": True} if self.num_heads != self.num_kv_heads else {}
attn_bias = None
if self.sliding_window is not None and not self.is_cross_attention:
@ -244,7 +241,7 @@ class AceStepAttention(nn.Module):
else:
attn_bias = window_bias
attn_output = optimized_attention(query_states, key_states, value_states, self.num_heads, attn_bias, skip_reshape=True, low_precision_attention=False)
attn_output = optimized_attention(query_states, key_states, value_states, self.num_heads, attn_bias, skip_reshape=True, low_precision_attention=False, **gqa_kwargs)
attn_output = self.o_proj(attn_output)
return attn_output

11
comfy/ldm/audio/dit.py
View File

@ -425,19 +425,16 @@ class Attention(nn.Module):
if n == 1 and causal:
causal = False
if h != kv_h:
# Repeat interleave kv_heads to match q_heads
heads_per_kv_head = h // kv_h
k, v = map(lambda t: t.repeat_interleave(heads_per_kv_head, dim = 1), (k, v))
gqa_kwargs = {"enable_gqa": True} if h != kv_h else {}
if self.differential:
q, q_diff = q.unbind(dim=1)
k, k_diff = k.unbind(dim=1)
out = optimized_attention(q, k, v, h, skip_reshape=True, low_precision_attention=False, transformer_options=transformer_options)
out_diff = optimized_attention(q_diff, k_diff, v, h, skip_reshape=True, low_precision_attention=False, transformer_options=transformer_options)
out = optimized_attention(q, k, v, h, skip_reshape=True, low_precision_attention=False, transformer_options=transformer_options, **gqa_kwargs)
out_diff = optimized_attention(q_diff, k_diff, v, h, skip_reshape=True, low_precision_attention=False, transformer_options=transformer_options, **gqa_kwargs)
out = out - out_diff
else:
out = optimized_attention(q, k, v, h, skip_reshape=True, low_precision_attention=False, transformer_options=transformer_options)
out = optimized_attention(q, k, v, h, skip_reshape=True, low_precision_attention=False, transformer_options=transformer_options, **gqa_kwargs)
out = self.to_out(out)

7
comfy/ldm/boogu/model.py
View File

@ -74,11 +74,8 @@ class BooguDoubleStreamProcessor(nn.Module):
key = key.transpose(1, 2)
value = value.transpose(1, 2)
if attn.kv_heads < attn.heads:
key = key.repeat_interleave(attn.heads // attn.kv_heads, dim=1)
value = value.repeat_interleave(attn.heads // attn.kv_heads, dim=1)
hidden_states = optimized_attention_masked(query, key, value, attn.heads, attention_mask, skip_reshape=True, transformer_options=transformer_options)
gqa_kwargs = {"enable_gqa": True} if attn.kv_heads < attn.heads else {}
hidden_states = optimized_attention_masked(query, key, value, attn.heads, attention_mask, skip_reshape=True, transformer_options=transformer_options, **gqa_kwargs)
# Split back to instruction/image, apply per-stream output projections, recombine.
instruct_hidden_states = self.instruct_out(hidden_states[:, :L_instruct])

12
comfy/ldm/cosmos/predict2.py
View File

@ -14,6 +14,7 @@ from torchvision import transforms
import comfy.patcher_extension
from comfy.ldm.modules.attention import optimized_attention
import comfy.ldm.common_dit
import comfy.ops
import comfy.quant_ops
@ -161,11 +162,16 @@ class Attention(nn.Module):
def apply_norm_and_rotary_pos_emb(
q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, rope_emb: Optional[torch.Tensor]
) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
q = self.q_norm(q)
k = self.k_norm(k)
v = self.v_norm(v)
if self.is_selfattn and rope_emb is not None: # only apply to self-attention!
q, k = comfy.quant_ops.ck.apply_rope_split_half(q, k, rope_emb)
q_scale, _, q_offload_stream = comfy.ops.cast_bias_weight(self.q_norm, q, offloadable=True)
k_scale, _, k_offload_stream = comfy.ops.cast_bias_weight(self.k_norm, k, offloadable=True)
q, k = comfy.quant_ops.ck.rms_rope_split_half(q, k, rope_emb, q_scale, k_scale, self.q_norm.eps)
comfy.ops.uncast_bias_weight(self.q_norm, q_scale, None, q_offload_stream)
comfy.ops.uncast_bias_weight(self.k_norm, k_scale, None, k_offload_stream)
else:
q = self.q_norm(q)
k = self.k_norm(k)
return q, k, v
q, k, v = apply_norm_and_rotary_pos_emb(q, k, v, rope_emb)

12
comfy/ldm/hidream_o1/attention.py
View File

@ -15,24 +15,24 @@ def make_two_pass_attention(ar_len: int, transformer_options=None):
The AR pass goes through SDPA directand bypasses wrappers, it is only ~1% of T at typical edit sizes.
"""
def two_pass_attention(q, k, v, heads, **kwargs):
def two_pass_attention(q, k, v, heads, enable_gqa=False, **kwargs):
B, H, T, D = q.shape
if T < k.shape[2]: # KV-cache hot path: Q is shorter than K/V (cached AR prefix is in K/V only), all fresh Q positions are in the gen region, single full-attention call
out = optimized_attention(q, k, v, heads, mask=None, skip_reshape=True, skip_output_reshape=True, transformer_options=transformer_options)
out = optimized_attention(q, k, v, heads, mask=None, skip_reshape=True, skip_output_reshape=True, transformer_options=transformer_options, enable_gqa=enable_gqa)
elif ar_len >= T:
out = comfy.ops.scaled_dot_product_attention(q, k, v, attn_mask=None, dropout_p=0.0, is_causal=True)
out = comfy.ops.scaled_dot_product_attention(q, k, v, attn_mask=None, dropout_p=0.0, is_causal=True, enable_gqa=enable_gqa)
elif ar_len <= 0:
out = optimized_attention(q, k, v, heads, mask=None, skip_reshape=True, skip_output_reshape=True, transformer_options=transformer_options)
out = optimized_attention(q, k, v, heads, mask=None, skip_reshape=True, skip_output_reshape=True, transformer_options=transformer_options, enable_gqa=enable_gqa)
else:
out_ar = comfy.ops.scaled_dot_product_attention(
q[:, :, :ar_len], k[:, :, :ar_len], v[:, :, :ar_len],
attn_mask=None, dropout_p=0.0, is_causal=True,
attn_mask=None, dropout_p=0.0, is_causal=True, enable_gqa=enable_gqa,
)
out_gen = optimized_attention(
q[:, :, ar_len:], k, v, heads,
mask=None, skip_reshape=True, skip_output_reshape=True,
transformer_options=transformer_options,
transformer_options=transformer_options, enable_gqa=enable_gqa,
)
out = torch.cat([out_ar, out_gen], dim=2)

445
comfy/ldm/joyimage/model.py Normal file
View File

@ -0,0 +1,445 @@
# https://github.com/jdopensource/JoyAI-Image-Edit (Apache 2.0)
import math
from typing import Optional, Tuple
import comfy_kitchen
import torch
import torch.nn as nn
import comfy.ldm.common_dit
import comfy.ops
import comfy.patcher_extension
from comfy.ldm.lightricks.model import GELU_approx, PixArtAlphaTextProjection, TimestepEmbedding, Timesteps
from comfy.ldm.modules.attention import optimized_attention
class JoyImageModulate(nn.Module):
def __init__(self, hidden_size: int, factor: int, dtype=None, device=None):
super().__init__()
self.factor = factor
self.modulate_table = nn.Parameter(
torch.empty(1, factor, hidden_size, dtype=dtype, device=device)
)
def forward(self, x: torch.Tensor) -> list:
if x.ndim != 3:
x = x.unsqueeze(1)
table = comfy.ops.cast_to_input(self.modulate_table, x)
return [o.squeeze(1) for o in (table + x).chunk(self.factor, dim=1)]
class JoyImageFeedForward(nn.Module):
def __init__(
self,
dim: int,
inner_dim: int,
dtype=None,
device=None,
operations=None,
):
super().__init__()
self.net = nn.ModuleList([
GELU_approx(dim, inner_dim, dtype=dtype, device=device, operations=operations),
nn.Identity(),
operations.Linear(inner_dim, dim, bias=True, dtype=dtype, device=device),
])
def forward(self, x: torch.Tensor) -> torch.Tensor:
for module in self.net:
x = module(x)
return x
class JoyImageAttention(nn.Module):
def __init__(
self,
dim: int,
num_attention_heads: int,
attention_head_dim: int,
eps: float = 1e-6,
dtype=None,
device=None,
operations=None,
):
super().__init__()
self.num_attention_heads = num_attention_heads
inner_dim = num_attention_heads * attention_head_dim
self.img_attn_qkv = operations.Linear(dim, inner_dim * 3, bias=True, dtype=dtype, device=device)
self.img_attn_q_norm = operations.RMSNorm(attention_head_dim, eps=eps, dtype=dtype, device=device)
self.img_attn_k_norm = operations.RMSNorm(attention_head_dim, eps=eps, dtype=dtype, device=device)
self.img_attn_proj = operations.Linear(inner_dim, dim, bias=True, dtype=dtype, device=device)
self.txt_attn_qkv = operations.Linear(dim, inner_dim * 3, bias=True, dtype=dtype, device=device)
self.txt_attn_q_norm = operations.RMSNorm(attention_head_dim, eps=eps, dtype=dtype, device=device)
self.txt_attn_k_norm = operations.RMSNorm(attention_head_dim, eps=eps, dtype=dtype, device=device)
self.txt_attn_proj = operations.Linear(inner_dim, dim, bias=True, dtype=dtype, device=device)
def forward(
self,
img: torch.Tensor,
txt: torch.Tensor,
image_rotary_emb: torch.Tensor,
transformer_options=None,
) -> Tuple[torch.Tensor, torch.Tensor]:
heads = self.num_attention_heads
img_q, img_k, img_v = self.img_attn_qkv(img).chunk(3, dim=-1)
txt_q, txt_k, txt_v = self.txt_attn_qkv(txt).chunk(3, dim=-1)
img_q = img_q.unflatten(-1, (heads, -1))
img_k = img_k.unflatten(-1, (heads, -1))
img_v = img_v.unflatten(-1, (heads, -1))
txt_q = txt_q.unflatten(-1, (heads, -1))
txt_k = txt_k.unflatten(-1, (heads, -1))
txt_v = txt_v.unflatten(-1, (heads, -1))
img_q = self.img_attn_q_norm(img_q)
img_k = self.img_attn_k_norm(img_k)
txt_q = self.txt_attn_q_norm(txt_q)
txt_k = self.txt_attn_k_norm(txt_k)
img_q, img_k = comfy_kitchen.apply_rope(img_q, img_k, image_rotary_emb)
joint_q = torch.cat([img_q, txt_q], dim=1)
joint_k = torch.cat([img_k, txt_k], dim=1)
joint_v = torch.cat([img_v, txt_v], dim=1)
joint_q = joint_q.flatten(2, 3)
joint_k = joint_k.flatten(2, 3)
joint_v = joint_v.flatten(2, 3)
joint_out = optimized_attention(joint_q, joint_k, joint_v, heads=heads, transformer_options=transformer_options)
seq_img = img.shape[1]
img_out = joint_out[:, :seq_img, :]
txt_out = joint_out[:, seq_img:, :]
img_out = self.img_attn_proj(img_out)
txt_out = self.txt_attn_proj(txt_out)
return img_out, txt_out
class JoyImageTransformerBlock(nn.Module):
def __init__(
self,
dim: int,
num_attention_heads: int,
attention_head_dim: int,
mlp_width_ratio: float = 4.0,
eps: float = 1e-6,
dtype=None,
device=None,
operations=None,
):
super().__init__()
mlp_hidden_dim = int(dim * mlp_width_ratio)
self.img_mod = JoyImageModulate(dim, factor=6, dtype=dtype, device=device)
self.img_norm1 = operations.LayerNorm(dim, elementwise_affine=False, eps=eps, dtype=dtype, device=device)
self.img_norm2 = operations.LayerNorm(dim, elementwise_affine=False, eps=eps, dtype=dtype, device=device)
self.img_mlp = JoyImageFeedForward(dim, inner_dim=mlp_hidden_dim, dtype=dtype, device=device, operations=operations)
self.txt_mod = JoyImageModulate(dim, factor=6, dtype=dtype, device=device)
self.txt_norm1 = operations.LayerNorm(dim, elementwise_affine=False, eps=eps, dtype=dtype, device=device)
self.txt_norm2 = operations.LayerNorm(dim, elementwise_affine=False, eps=eps, dtype=dtype, device=device)
self.txt_mlp = JoyImageFeedForward(dim, inner_dim=mlp_hidden_dim, dtype=dtype, device=device, operations=operations)
self.attn = JoyImageAttention(
dim=dim,
num_attention_heads=num_attention_heads,
attention_head_dim=attention_head_dim,
eps=eps,
dtype=dtype,
device=device,
operations=operations,
)
def forward(
self,
hidden_states: torch.Tensor,
encoder_hidden_states: torch.Tensor,
temb: torch.Tensor,
image_rotary_emb: torch.Tensor,
transformer_options=None,
) -> Tuple[torch.Tensor, torch.Tensor]:
(
img_mod1_shift,
img_mod1_scale,
img_mod1_gate,
img_mod2_shift,
img_mod2_scale,
img_mod2_gate,
) = self.img_mod(temb)
(
txt_mod1_shift,
txt_mod1_scale,
txt_mod1_gate,
txt_mod2_shift,
txt_mod2_scale,
txt_mod2_gate,
) = self.txt_mod(temb)
img_normed = self.img_norm1(hidden_states)
txt_normed = self.txt_norm1(encoder_hidden_states)
img_modulated = img_normed * (1 + img_mod1_scale.unsqueeze(1)) + img_mod1_shift.unsqueeze(1)
txt_modulated = txt_normed * (1 + txt_mod1_scale.unsqueeze(1)) + txt_mod1_shift.unsqueeze(1)
img_attn, txt_attn = self.attn(img_modulated, txt_modulated, image_rotary_emb, transformer_options=transformer_options)
hidden_states = hidden_states + img_attn * img_mod1_gate.unsqueeze(1)
encoder_hidden_states = encoder_hidden_states + txt_attn * txt_mod1_gate.unsqueeze(1)
img_ffn_normed = self.img_norm2(hidden_states)
txt_ffn_normed = self.txt_norm2(encoder_hidden_states)
img_ffn_input = img_ffn_normed * (1 + img_mod2_scale.unsqueeze(1)) + img_mod2_shift.unsqueeze(1)
txt_ffn_input = txt_ffn_normed * (1 + txt_mod2_scale.unsqueeze(1)) + txt_mod2_shift.unsqueeze(1)
hidden_states = hidden_states + self.img_mlp(img_ffn_input) * img_mod2_gate.unsqueeze(1)
encoder_hidden_states = encoder_hidden_states + self.txt_mlp(txt_ffn_input) * txt_mod2_gate.unsqueeze(1)
return hidden_states, encoder_hidden_states
class JoyImageTimeTextImageEmbedding(nn.Module):
def __init__(
self,
dim: int,
time_freq_dim: int,
time_proj_dim: int,
text_embed_dim: int,
dtype=None,
device=None,
operations=None,
):
super().__init__()
self.timesteps_proj = Timesteps(num_channels=time_freq_dim, flip_sin_to_cos=True, downscale_freq_shift=0)
self.time_embedder = TimestepEmbedding(
in_channels=time_freq_dim,
time_embed_dim=dim,
dtype=dtype,
device=device,
operations=operations,
)
self.act_fn = nn.SiLU()
self.time_proj = operations.Linear(dim, time_proj_dim, bias=True, dtype=dtype, device=device)
self.text_embedder = PixArtAlphaTextProjection(
text_embed_dim, dim, act_fn="gelu_tanh", dtype=dtype, device=device, operations=operations,
)
def forward(self, timestep: torch.Tensor, encoder_hidden_states: torch.Tensor):
timestep = self.timesteps_proj(timestep)
temb = self.time_embedder(timestep.to(dtype=encoder_hidden_states.dtype)).type_as(encoder_hidden_states)
timestep_proj = self.time_proj(self.act_fn(temb))
encoder_hidden_states = self.text_embedder(encoder_hidden_states)
return temb, timestep_proj, encoder_hidden_states
class JoyImageTransformer3DModel(nn.Module):
def __init__(
self,
patch_size: list = [1, 2, 2],
in_channels: int = 16,
out_channels: Optional[int] = None,
hidden_size: int = 3072,
num_attention_heads: int = 24,
text_dim: int = 4096,
mlp_width_ratio: float = 4.0,
num_layers: int = 20,
rope_dim_list: list = [16, 56, 56],
theta: int = 256,
image_model=None,
dtype=None,
device=None,
operations=None,
):
super().__init__()
self.dtype = dtype
self.out_channels = out_channels or in_channels
self.patch_size = list(patch_size)
self.rope_dim_list = list(rope_dim_list)
self.theta = theta
attention_head_dim = hidden_size // num_attention_heads
self.img_in = operations.Conv3d(
in_channels,
hidden_size,
kernel_size=tuple(self.patch_size),
stride=tuple(self.patch_size),
dtype=dtype,
device=device,
)
self.condition_embedder = JoyImageTimeTextImageEmbedding(
dim=hidden_size,
time_freq_dim=256,
time_proj_dim=hidden_size * 6,
text_embed_dim=text_dim,
dtype=dtype,
device=device,
operations=operations,
)
self.double_blocks = nn.ModuleList([
JoyImageTransformerBlock(
dim=hidden_size,
num_attention_heads=num_attention_heads,
attention_head_dim=attention_head_dim,
mlp_width_ratio=mlp_width_ratio,
dtype=dtype,
device=device,
operations=operations,
)
for _ in range(num_layers)
])
self.norm_out = operations.LayerNorm(hidden_size, elementwise_affine=False, eps=1e-6, dtype=dtype, device=device)
self.proj_out = operations.Linear(
hidden_size,
self.out_channels * math.prod(self.patch_size),
bias=True,
dtype=dtype,
device=device,
)
def _get_rotary_pos_embed_for_range(
self,
start: Tuple[int, int, int],
stop: Tuple[int, int, int],
device=None,
) -> torch.Tensor:
# 3D RoPE for the patch grid range [start, stop) over (t, h, w). Token order after
# reshape(-1) is (t, h, w), matching the img_in Conv3d flatten.
rope_dim_list = self.rope_dim_list
grids = [torch.arange(start[i], stop[i], dtype=torch.float32, device=device) for i in range(3)]
mesh = torch.stack(torch.meshgrid(*grids, indexing="ij"), dim=0)
angles_parts = []
for i, dim in enumerate(rope_dim_list):
pos = mesh[i].reshape(-1)
freqs = 1.0 / (self.theta ** (torch.arange(0, dim, 2, dtype=torch.float32, device=device)[: (dim // 2)] / dim))
angles_parts.append(torch.outer(pos, freqs))
angles = torch.cat(angles_parts, dim=1)
cos = angles.cos()
sin = angles.sin()
return torch.stack((cos, -sin, sin, cos), dim=-1).unflatten(-1, (2, 2))
def get_rotary_pos_embed_for_components(
self,
component_sizes,
device=None,
) -> torch.Tensor:
# Per-component 3D RoPE. component_sizes is a list of (t, h, w) patch grid sizes in
# sequence order [target, ref0, ref1, ...]; h/w restart at 0 for each component while t
# continues from the running offset, giving every image its own temporal position band.
freqs_parts = []
t_offset = 0
for (t, h, w) in component_sizes:
freqs = self._get_rotary_pos_embed_for_range(
start=(t_offset, 0, 0),
stop=(t_offset + t, h, w),
device=device,
)
freqs_parts.append(freqs)
t_offset += t
return torch.cat(freqs_parts, dim=0).unsqueeze(0).unsqueeze(2)
def unpatchify(self, x: torch.Tensor, t: int, h: int, w: int) -> torch.Tensor:
c = self.out_channels
pt, ph, pw = self.patch_size
x = x.reshape(x.shape[0], t, h, w, pt, ph, pw, c)
x = x.permute(0, 7, 1, 4, 2, 5, 3, 6)
return x.reshape(x.shape[0], c, t * pt, h * ph, w * pw)
def forward(
self,
hidden_states: torch.Tensor,
timestep: torch.Tensor,
context: torch.Tensor = None,
ref_latents=None,
control=None,
transformer_options=None,
**kwargs,
) -> torch.Tensor:
transformer_options = {} if transformer_options is None else transformer_options.copy()
return comfy.patcher_extension.WrapperExecutor.new_class_executor(
self._forward,
self,
comfy.patcher_extension.get_all_wrappers(comfy.patcher_extension.WrappersMP.DIFFUSION_MODEL, transformer_options)
).execute(hidden_states, timestep, context, ref_latents, transformer_options, **kwargs)
def _forward(
self,
hidden_states: torch.Tensor,
timestep: torch.Tensor,
context: torch.Tensor,
ref_latents=None,
transformer_options=None,
**kwargs,
) -> torch.Tensor:
pt, ph, pw = self.patch_size
_, _, ot, oh, ow = hidden_states.shape
components = [hidden_states, *(ref_latents or [])]
component_sizes = []
img_tokens = []
for comp in components:
comp = comfy.ldm.common_dit.pad_to_patch_size(comp, self.patch_size)
_, _, ct, ch, cw = comp.shape
component_sizes.append((ct // pt, ch // ph, cw // pw))
tokens = self.img_in(comp).flatten(2).transpose(1, 2) # (B, n_i, D)
img_tokens.append(tokens)
img = torch.cat(img_tokens, dim=1)
_, vec, txt = self.condition_embedder(timestep, context)
vec = vec.unflatten(1, (6, -1))
image_rotary_emb = self.get_rotary_pos_embed_for_components(
component_sizes,
device=hidden_states.device,
)
patches_replace = transformer_options.get("patches_replace", {})
blocks_replace = patches_replace.get("dit", {})
transformer_options["total_blocks"] = len(self.double_blocks)
transformer_options["block_type"] = "double"
for i, block in enumerate(self.double_blocks):
transformer_options["block_index"] = i
if ("double_block", i) in blocks_replace:
def block_wrap(args):
out = {}
out["img"], out["txt"] = block(
hidden_states=args["img"],
encoder_hidden_states=args["txt"],
temb=args["vec"],
image_rotary_emb=args["pe"],
transformer_options=args.get("transformer_options"),
)
return out
out = blocks_replace[("double_block", i)]({"img": img,
"txt": txt,
"vec": vec,
"pe": image_rotary_emb,
"transformer_options": transformer_options},
{"original_block": block_wrap})
txt = out["txt"]
img = out["img"]
else:
img, txt = block(
hidden_states=img,
encoder_hidden_states=txt,
temb=vec,
image_rotary_emb=image_rotary_emb,
transformer_options=transformer_options,
)
tt, th, tw = component_sizes[0]
target_tokens = tt * th * tw
img = img[:, :target_tokens, :]
img = self.proj_out(self.norm_out(img))
img = self.unpatchify(img, tt, th, tw)
return img[:, :, :ot, :oh, :ow]

165
comfy/ldm/modules/attention.py
View File

@ -1,5 +1,6 @@
import math
import sys
import inspect
import torch
import torch.nn.functional as F
@ -14,16 +15,16 @@ from .sub_quadratic_attention import efficient_dot_product_attention
from comfy import model_management
TORCH_HAS_GQA = model_management.torch_version_numeric >= (2, 5)
if model_management.xformers_enabled():
import xformers
import xformers.ops
SAGE_ATTENTION_IS_AVAILABLE = False
SAGE_ATTENTION_SUPPORTS_MASK = False
try:
from sageattention import sageattn
SAGE_ATTENTION_IS_AVAILABLE = True
SAGE_ATTENTION_SUPPORTS_MASK = "attn_mask" in inspect.signature(sageattn).parameters
except ImportError as e:
if model_management.sage_attention_enabled():
if e.name == "sageattention":
@ -89,6 +90,44 @@ def default(val, d):
return val
return d
def _gqa_repeat_factor(query_heads, key_heads, value_heads):
if key_heads != value_heads:
raise ValueError(f"Key/value head count mismatch for GQA: {key_heads} != {value_heads}")
if query_heads == key_heads:
return 1
if query_heads % key_heads != 0:
raise ValueError(f"Query heads must be divisible by key/value heads for GQA: {query_heads} vs {key_heads}")
return query_heads // key_heads
def _repeat_kv_for_gqa(k, v, query_heads, head_dim):
n_rep = _gqa_repeat_factor(query_heads, k.shape[head_dim], v.shape[head_dim])
if n_rep > 1:
k = k.repeat_interleave(n_rep, dim=head_dim)
v = v.repeat_interleave(n_rep, dim=head_dim)
return k, v
def _heads_from_dim(tensor, dim_head, name):
inner_dim = tensor.shape[-1]
if inner_dim % dim_head != 0:
raise ValueError(f"{name} inner dimension {inner_dim} is not divisible by head dimension {dim_head}")
return inner_dim // dim_head
def _reshape_qkv_to_heads(q, k, v, b, heads, dim_head, enable_gqa=False, expand_kv=True):
q = q.unsqueeze(3).reshape(b, -1, heads, dim_head)
if enable_gqa:
key_heads = _heads_from_dim(k, dim_head, "Key")
value_heads = _heads_from_dim(v, dim_head, "Value")
else:
key_heads = heads
value_heads = heads
k = k.unsqueeze(3).reshape(b, -1, key_heads, dim_head)
v = v.unsqueeze(3).reshape(b, -1, value_heads, dim_head)
if enable_gqa:
_gqa_repeat_factor(heads, key_heads, value_heads)
if expand_kv:
k, v = _repeat_kv_for_gqa(k, v, heads, -2)
return q, k, v
# feedforward
class GEGLU(nn.Module):
@ -152,28 +191,19 @@ def attention_basic(q, k, v, heads, mask=None, attn_precision=None, skip_reshape
b, _, dim_head = q.shape
dim_head //= heads
if kwargs.get("enable_gqa", False) and q.shape[-3] != k.shape[-3]:
n_rep = q.shape[-3] // k.shape[-3]
k = k.repeat_interleave(n_rep, dim=-3)
v = v.repeat_interleave(n_rep, dim=-3)
scale = kwargs.get("scale", dim_head ** -0.5)
h = heads
if skip_reshape:
q, k, v = map(
if kwargs.get("enable_gqa", False):
k, v = _repeat_kv_for_gqa(k, v, q.shape[-3], -3)
q, k, v = map(
lambda t: t.reshape(b * heads, -1, dim_head),
(q, k, v),
)
else:
q, k, v = map(
lambda t: t.unsqueeze(3)
.reshape(b, -1, heads, dim_head)
.permute(0, 2, 1, 3)
.reshape(b * heads, -1, dim_head)
.contiguous(),
(q, k, v),
)
q, k, v = _reshape_qkv_to_heads(q, k, v, b, heads, dim_head, kwargs.get("enable_gqa", False))
q, k, v = map(lambda t: t.permute(0, 2, 1, 3).reshape(b * heads, -1, dim_head).contiguous(), (q, k, v))
# force cast to fp32 to avoid overflowing
if attn_precision == torch.float32:
@ -231,13 +261,16 @@ def attention_sub_quad(query, key, value, heads, mask=None, attn_precision=None,
query = query * (kwargs["scale"] * dim_head ** 0.5)
if skip_reshape:
if kwargs.get("enable_gqa", False):
key, value = _repeat_kv_for_gqa(key, value, query.shape[-3], -3)
query = query.reshape(b * heads, -1, dim_head)
value = value.reshape(b * heads, -1, dim_head)
key = key.reshape(b * heads, -1, dim_head).movedim(1, 2)
else:
query = query.unsqueeze(3).reshape(b, -1, heads, dim_head).permute(0, 2, 1, 3).reshape(b * heads, -1, dim_head)
value = value.unsqueeze(3).reshape(b, -1, heads, dim_head).permute(0, 2, 1, 3).reshape(b * heads, -1, dim_head)
key = key.unsqueeze(3).reshape(b, -1, heads, dim_head).permute(0, 2, 3, 1).reshape(b * heads, dim_head, -1)
query, key, value = _reshape_qkv_to_heads(query, key, value, b, heads, dim_head, kwargs.get("enable_gqa", False))
query = query.permute(0, 2, 1, 3).reshape(b * heads, -1, dim_head)
value = value.permute(0, 2, 1, 3).reshape(b * heads, -1, dim_head)
key = key.permute(0, 2, 3, 1).reshape(b * heads, dim_head, -1)
dtype = query.dtype
@ -304,19 +337,15 @@ def attention_split(q, k, v, heads, mask=None, attn_precision=None, skip_reshape
scale = kwargs.get("scale", dim_head ** -0.5)
if skip_reshape:
q, k, v = map(
if kwargs.get("enable_gqa", False):
k, v = _repeat_kv_for_gqa(k, v, q.shape[-3], -3)
q, k, v = map(
lambda t: t.reshape(b * heads, -1, dim_head),
(q, k, v),
)
else:
q, k, v = map(
lambda t: t.unsqueeze(3)
.reshape(b, -1, heads, dim_head)
.permute(0, 2, 1, 3)
.reshape(b * heads, -1, dim_head)
.contiguous(),
(q, k, v),
)
q, k, v = _reshape_qkv_to_heads(q, k, v, b, heads, dim_head, kwargs.get("enable_gqa", False))
q, k, v = map(lambda t: t.permute(0, 2, 1, 3).reshape(b * heads, -1, dim_head).contiguous(), (q, k, v))
r1 = torch.zeros(q.shape[0], q.shape[1], v.shape[2], device=q.device, dtype=q.dtype)
@ -438,7 +467,7 @@ def attention_xformers(q, k, v, heads, mask=None, attn_precision=None, skip_resh
disabled_xformers = True
if disabled_xformers:
return attention_pytorch(q, k, v, heads, mask, skip_reshape=skip_reshape, **kwargs)
return attention_pytorch(q, k, v, heads, mask, skip_reshape=skip_reshape, skip_output_reshape=skip_output_reshape, **kwargs)
if skip_reshape:
# b h k d -> b k h d
@ -446,13 +475,12 @@ def attention_xformers(q, k, v, heads, mask=None, attn_precision=None, skip_resh
lambda t: t.permute(0, 2, 1, 3),
(q, k, v),
)
if kwargs.get("enable_gqa", False):
k, v = _repeat_kv_for_gqa(k, v, q.shape[-2], -2)
# actually do the reshaping
else:
dim_head //= heads
q, k, v = map(
lambda t: t.reshape(b, -1, heads, dim_head),
(q, k, v),
)
q, k, v = _reshape_qkv_to_heads(q, k, v, b, heads, dim_head, kwargs.get("enable_gqa", False))
if mask is not None:
# add a singleton batch dimension
@ -474,7 +502,7 @@ def attention_xformers(q, k, v, heads, mask=None, attn_precision=None, skip_resh
mask = mask_out[..., :mask.shape[-1]]
mask = mask.expand(b, heads, -1, -1)
out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=mask)
out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=mask, scale=kwargs.get("scale", None))
if skip_output_reshape:
out = out.permute(0, 2, 1, 3)
@ -498,10 +526,8 @@ def attention_pytorch(q, k, v, heads, mask=None, attn_precision=None, skip_resha
else:
b, _, dim_head = q.shape
dim_head //= heads
q, k, v = map(
lambda t: t.view(b, -1, heads, dim_head).transpose(1, 2),
(q, k, v),
)
q, k, v = _reshape_qkv_to_heads(q, k, v, b, heads, dim_head, kwargs.get("enable_gqa", False), expand_kv=False)
q, k, v = map(lambda t: t.transpose(1, 2), (q, k, v))
if mask is not None:
# add a batch dimension if there isn't already one
@ -511,9 +537,7 @@ def attention_pytorch(q, k, v, heads, mask=None, attn_precision=None, skip_resha
if mask.ndim == 3:
mask = mask.unsqueeze(1)
# Pass through extra SDPA kwargs (scale, enable_gqa) if provided
# enable_gqa requires PyTorch 2.5+; older versions use manual KV expansion above
sdpa_keys = ("scale", "enable_gqa") if TORCH_HAS_GQA else ("scale",)
sdpa_keys = ("scale", "enable_gqa")
sdpa_extra = {k: v for k, v in kwargs.items() if k in sdpa_keys}
if SDP_BATCH_LIMIT >= b:
@ -541,20 +565,19 @@ def attention_pytorch(q, k, v, heads, mask=None, attn_precision=None, skip_resha
@wrap_attn
def attention_sage(q, k, v, heads, mask=None, attn_precision=None, skip_reshape=False, skip_output_reshape=False, **kwargs):
if kwargs.get("low_precision_attention", True) is False:
if kwargs.get("low_precision_attention", True) is False or (mask is not None and not SAGE_ATTENTION_SUPPORTS_MASK):
return attention_pytorch(q, k, v, heads, mask=mask, skip_reshape=skip_reshape, skip_output_reshape=skip_output_reshape, **kwargs)
exception_fallback = False
if skip_reshape:
b, _, _, dim_head = q.shape
tensor_layout = "HND"
if kwargs.get("enable_gqa", False):
k, v = _repeat_kv_for_gqa(k, v, q.shape[-3], -3)
else:
b, _, dim_head = q.shape
dim_head //= heads
q, k, v = map(
lambda t: t.view(b, -1, heads, dim_head),
(q, k, v),
)
q, k, v = _reshape_qkv_to_heads(q, k, v, b, heads, dim_head, kwargs.get("enable_gqa", False))
tensor_layout = "NHD"
if mask is not None:
@ -565,8 +588,12 @@ def attention_sage(q, k, v, heads, mask=None, attn_precision=None, skip_reshape=
if mask.ndim == 3:
mask = mask.unsqueeze(1)
sage_kwargs = {"is_causal": False, "tensor_layout": tensor_layout, "sm_scale": kwargs.get("scale", None), "smooth_k": False}
if mask is not None:
sage_kwargs["attn_mask"] = mask
try:
out = sageattn(q, k, v, attn_mask=mask, is_causal=False, tensor_layout=tensor_layout)
out = sageattn(q, k, v, **sage_kwargs)
except Exception as e:
logging.error("Error running sage attention: {}, using pytorch attention instead.".format(e))
exception_fallback = True
@ -616,7 +643,6 @@ def attention3_sage(q, k, v, heads, mask=None, attn_precision=None, skip_reshape
skip_output_reshape=skip_output_reshape,
**kwargs
)
q_s, k_s, v_s = q, k, v
N = q.shape[2]
dim_head = D
else:
@ -642,11 +668,15 @@ def attention3_sage(q, k, v, heads, mask=None, attn_precision=None, skip_reshape
**kwargs
)
if not skip_reshape:
q_s, k_s, v_s = map(
lambda t: t.view(B, -1, heads, dim_head).permute(0, 2, 1, 3).contiguous(),
(q, k, v),
)
if skip_reshape:
q_s = q
if kwargs.get("enable_gqa", False):
k_s, v_s = _repeat_kv_for_gqa(k, v, H, -3)
else:
k_s, v_s = k, v
else:
q_s, k_s, v_s = _reshape_qkv_to_heads(q, k, v, B, heads, dim_head, kwargs.get("enable_gqa", False))
q_s, k_s, v_s = map(lambda t: t.permute(0, 2, 1, 3).contiguous(), (q_s, k_s, v_s))
B, H, L, D = q_s.shape
try:
@ -662,7 +692,7 @@ def attention3_sage(q, k, v, heads, mask=None, attn_precision=None, skip_reshape
q, k, v, heads,
mask=mask,
attn_precision=attn_precision,
skip_reshape=False,
skip_reshape=skip_reshape,
skip_output_reshape=skip_output_reshape,
**kwargs
)
@ -679,21 +709,22 @@ def attention3_sage(q, k, v, heads, mask=None, attn_precision=None, skip_reshape
return out
try:
@torch.library.custom_op("flash_attention::flash_attn", mutates_args=())
@torch.library.custom_op("comfy::flash_attn", mutates_args=())
def flash_attn_wrapper(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor,
dropout_p: float = 0.0, causal: bool = False) -> torch.Tensor:
return flash_attn_func(q, k, v, dropout_p=dropout_p, causal=causal)
dropout_p: float = 0.0, causal: bool = False, softmax_scale: float = -1.0) -> torch.Tensor:
softmax_scale_arg = None if softmax_scale == -1.0 else softmax_scale
return flash_attn_func(q, k, v, dropout_p=dropout_p, causal=causal, softmax_scale=softmax_scale_arg)
@flash_attn_wrapper.register_fake
def flash_attn_fake(q, k, v, dropout_p=0.0, causal=False):
def flash_attn_fake(q, k, v, dropout_p=0.0, causal=False, softmax_scale=-1.0):
# Output shape is the same as q
return q.new_empty(q.shape)
except AttributeError as error:
FLASH_ATTN_ERROR = error
def flash_attn_wrapper(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor,
dropout_p: float = 0.0, causal: bool = False) -> torch.Tensor:
dropout_p: float = 0.0, causal: bool = False, softmax_scale: float = -1.0) -> torch.Tensor:
assert False, f"Could not define flash_attn_wrapper: {FLASH_ATTN_ERROR}"
@wrap_attn
@ -703,10 +734,8 @@ def attention_flash(q, k, v, heads, mask=None, attn_precision=None, skip_reshape
else:
b, _, dim_head = q.shape
dim_head //= heads
q, k, v = map(
lambda t: t.view(b, -1, heads, dim_head).transpose(1, 2),
(q, k, v),
)
q, k, v = _reshape_qkv_to_heads(q, k, v, b, heads, dim_head, kwargs.get("enable_gqa", False), expand_kv=False)
q, k, v = map(lambda t: t.transpose(1, 2), (q, k, v))
if mask is not None:
# add a batch dimension if there isn't already one
@ -725,10 +754,16 @@ def attention_flash(q, k, v, heads, mask=None, attn_precision=None, skip_reshape
v.transpose(1, 2),
dropout_p=0.0,
causal=False,
softmax_scale=kwargs.get("scale", -1.0),
).transpose(1, 2)
except Exception as e:
logging.warning(f"Flash Attention failed, using default SDPA: {e}")
out = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=mask, dropout_p=0.0, is_causal=False)
sdpa_extra = {}
if kwargs.get("enable_gqa", False):
sdpa_extra["enable_gqa"] = True
if "scale" in kwargs:
sdpa_extra["scale"] = kwargs["scale"]
out = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=mask, dropout_p=0.0, is_causal=False, **sdpa_extra)
if not skip_output_reshape:
out = (
out.transpose(1, 2).reshape(b, -1, heads * dim_head)
@ -1209,5 +1244,3 @@ class SpatialVideoTransformer(SpatialTransformer):
x = self.proj_out(x)
out = x + x_in
return out

6
comfy/ldm/modules/diffusionmodules/model.py
View File

@ -22,7 +22,7 @@ def torch_cat_if_needed(xl, dim):
else:
return None
def get_timestep_embedding(timesteps, embedding_dim):
def get_timestep_embedding(timesteps, embedding_dim, flip_sin_to_cos=False, downscale_freq_shift=1):
"""
This matches the implementation in Denoising Diffusion Probabilistic Models:
From Fairseq.
@ -33,11 +33,13 @@ def get_timestep_embedding(timesteps, embedding_dim):
assert len(timesteps.shape) == 1
half_dim = embedding_dim // 2
emb = math.log(10000) / (half_dim - 1)
emb = math.log(10000) / (half_dim - downscale_freq_shift)
emb = torch.exp(torch.arange(half_dim, dtype=torch.float32) * -emb)
emb = emb.to(device=timesteps.device)
emb = timesteps.float()[:, None] * emb[None, :]
emb = torch.cat([torch.sin(emb), torch.cos(emb)], dim=1)
if flip_sin_to_cos:
emb = torch.cat([emb[:, half_dim:], emb[:, :half_dim]], dim=-1)
if embedding_dim % 2 == 1: # zero pad
emb = torch.nn.functional.pad(emb, (0,1,0,0))
return emb

7
comfy/ldm/omnigen/omnigen2.py
View File

@ -141,11 +141,8 @@ class Attention(nn.Module):
key = key.transpose(1, 2)
value = value.transpose(1, 2)
if self.kv_heads < self.heads:
key = key.repeat_interleave(self.heads // self.kv_heads, dim=1)
value = value.repeat_interleave(self.heads // self.kv_heads, dim=1)
hidden_states = optimized_attention_masked(query, key, value, self.heads, attention_mask, skip_reshape=True, transformer_options=transformer_options)
gqa_kwargs = {"enable_gqa": True} if self.kv_heads < self.heads else {}
hidden_states = optimized_attention_masked(query, key, value, self.heads, attention_mask, skip_reshape=True, transformer_options=transformer_options, **gqa_kwargs)
hidden_states = self.to_out[0](hidden_states)
return hidden_states

4
comfy/ldm/pixeldit/model.py
View File

@ -197,6 +197,9 @@ class PixDiT_T2I(nn.Module):
"""Hook for subclasses to inject per-block state into the patch stream (e.g. PiD's LQ gate)."""
return s
def _pre_pixel_blocks(self, s, **kwargs):
return s
def _forward(self, x, timesteps, context=None, attention_mask=None, transformer_options={}, **kwargs):
H_orig, W_orig = x.shape[2], x.shape[3]
x = comfy.ldm.common_dit.pad_to_patch_size(x, (self.patch_size, self.patch_size))
@ -226,6 +229,7 @@ class PixDiT_T2I(nn.Module):
s, y_emb = blk(s, y_emb, condition, pos_img, pos_txt, None, transformer_options=transformer_options)
s = F.silu(t_emb + s)
s = self._pre_pixel_blocks(s, **kwargs)
s_cond = s.view(B * L, self.hidden_size)
x_pixels = self.pixel_embedder(x, patch_size=self.patch_size)
for blk in self.pixel_blocks:

64
comfy/ldm/pixeldit/pid.py
View File

@ -13,15 +13,15 @@ from .model import PixDiT_T2I
from .modules import precompute_freqs_cis_2d
class SigmaAwareGatePerTokenPerDim(nn.Module):
class SigmaAwareGate(nn.Module):
"""gate = sigmoid(content_proj(cat[x, lq]) - exp(log_alpha) * sigma); out = x + gate * lq.
Trained init gives ~0.88 gate at sigma=0, ~0.05 at sigma=1.
"""
def __init__(self, dim: int, dtype=None, device=None, operations=None):
def __init__(self, dim: int, per_token: bool = False, dtype=None, device=None, operations=None):
super().__init__()
self.content_proj = operations.Linear(dim * 2, dim, dtype=dtype, device=device)
self.content_proj = operations.Linear(dim * 2, 1 if per_token else dim, dtype=dtype, device=device)
self.log_alpha = nn.Parameter(torch.empty((), dtype=dtype, device=device))
def forward(self, x: torch.Tensor, lq: torch.Tensor, sigma: torch.Tensor) -> torch.Tensor:
@ -36,15 +36,15 @@ class SigmaAwareGatePerTokenPerDim(nn.Module):
class ResBlock(nn.Module):
"""Pre-activation ResNet block: GN -> SiLU -> Conv -> GN -> SiLU -> Conv + skip."""
def __init__(self, channels: int, num_groups: int = 4, dtype=None, device=None, operations=None):
def __init__(self, channels: int, num_groups: int = 4, conv_padding_mode: str = "zeros", dtype=None, device=None, operations=None):
super().__init__()
self.block = nn.Sequential(
operations.GroupNorm(num_groups, channels, dtype=dtype, device=device),
nn.SiLU(),
operations.Conv2d(channels, channels, kernel_size=3, padding=1, dtype=dtype, device=device),
operations.Conv2d(channels, channels, kernel_size=3, padding=1, padding_mode=conv_padding_mode, dtype=dtype, device=device),
operations.GroupNorm(num_groups, channels, dtype=dtype, device=device),
nn.SiLU(),
operations.Conv2d(channels, channels, kernel_size=3, padding=1, dtype=dtype, device=device),
operations.Conv2d(channels, channels, kernel_size=3, padding=1, padding_mode=conv_padding_mode, dtype=dtype, device=device),
)
def forward(self, x: torch.Tensor) -> torch.Tensor:
@ -62,9 +62,13 @@ class LQProjection2D(nn.Module):
patch_size: int = 16,
sr_scale: int = 4,
latent_spatial_down_factor: int = 8,
latent_unpatchify_factor: int = 1,
num_res_blocks: int = 4,
num_outputs: int = 7,
interval: int = 2,
conv_padding_mode: str = "zeros",
gate_per_token: bool = False,
pit_output: bool = False,
dtype=None, device=None, operations=None,
):
super().__init__()
@ -74,34 +78,38 @@ class LQProjection2D(nn.Module):
self.patch_size = patch_size
self.sr_scale = sr_scale
self.latent_spatial_down_factor = latent_spatial_down_factor
self.latent_unpatchify_factor = latent_unpatchify_factor
self.num_outputs = num_outputs
self.interval = interval
z_to_patch_ratio = (sr_scale * latent_spatial_down_factor) / patch_size
effective_latent_channels = latent_channels // (latent_unpatchify_factor * latent_unpatchify_factor)
effective_spatial_down_factor = latent_spatial_down_factor // latent_unpatchify_factor
z_to_patch_ratio = (sr_scale * effective_spatial_down_factor) / patch_size
self.z_to_patch_ratio = z_to_patch_ratio
if z_to_patch_ratio >= 1:
self.latent_fold_factor = 0
latent_proj_in_ch = latent_channels
latent_proj_in_ch = effective_latent_channels
else:
fold_factor = int(1 / z_to_patch_ratio)
assert fold_factor * z_to_patch_ratio == 1.0
self.latent_fold_factor = fold_factor
latent_proj_in_ch = latent_channels * fold_factor * fold_factor
latent_proj_in_ch = effective_latent_channels * fold_factor * fold_factor
layers = [
operations.Conv2d(latent_proj_in_ch, hidden_dim, kernel_size=3, padding=1, dtype=dtype, device=device),
operations.Conv2d(latent_proj_in_ch, hidden_dim, kernel_size=3, padding=1, padding_mode=conv_padding_mode, dtype=dtype, device=device),
nn.SiLU(),
operations.Conv2d(hidden_dim, hidden_dim, kernel_size=3, padding=1, dtype=dtype, device=device),
operations.Conv2d(hidden_dim, hidden_dim, kernel_size=3, padding=1, padding_mode=conv_padding_mode, dtype=dtype, device=device),
]
for _ in range(num_res_blocks):
layers.append(ResBlock(hidden_dim, dtype=dtype, device=device, operations=operations))
layers.append(ResBlock(hidden_dim, conv_padding_mode=conv_padding_mode, dtype=dtype, device=device, operations=operations))
self.latent_proj = nn.Sequential(*layers)
self.output_heads = nn.ModuleList(
[operations.Linear(hidden_dim, out_dim, dtype=dtype, device=device) for _ in range(num_outputs)]
)
self.pit_head = operations.Linear(hidden_dim, out_dim, dtype=dtype, device=device) if pit_output else None
self.gate_modules = nn.ModuleList(
[SigmaAwareGatePerTokenPerDim(out_dim, dtype=dtype, device=device, operations=operations)
[SigmaAwareGate(out_dim, per_token=gate_per_token, dtype=dtype, device=device, operations=operations)
for _ in range(num_outputs)]
)
@ -115,6 +123,11 @@ class LQProjection2D(nn.Module):
return self.gate_modules[out_idx](x, lq_feature, sigma)
def _align_latent_to_patch_grid(self, lq_latent: torch.Tensor, pH: int, pW: int) -> torch.Tensor:
f = self.latent_unpatchify_factor
if f > 1:
B, C, H, W = lq_latent.shape
lq_latent = lq_latent.reshape(B, C // (f * f), f, f, H, W)
lq_latent = lq_latent.permute(0, 1, 4, 2, 5, 3).reshape(B, C // (f * f), H * f, W * f)
B, z_dim = lq_latent.shape[:2]
if self.z_to_patch_ratio >= 1:
if lq_latent.shape[2] != pH or lq_latent.shape[3] != pW:
@ -134,7 +147,10 @@ class LQProjection2D(nn.Module):
feat = self._align_latent_to_patch_grid(lq_latent, target_pH, target_pW)
B, C, H, W = feat.shape
tokens = feat.permute(0, 2, 3, 1).contiguous().view(B, H * W, C)
return [head(tokens) for head in self.output_heads]
outputs = [head(tokens) for head in self.output_heads]
if self.pit_head is not None:
outputs.append(self.pit_head(tokens))
return outputs
class PidNet(PixDiT_T2I):
@ -148,6 +164,10 @@ class PidNet(PixDiT_T2I):
lq_interval: int = 2,
sr_scale: int = 4,
latent_spatial_down_factor: int = 8,
lq_latent_unpatchify_factor: int = 1,
lq_conv_padding_mode: str = "zeros",
lq_gate_per_token: bool = False,
pit_lq_inject: bool = False,
rope_ref_h: int = 1024, # NTK ref resolution in PIXEL units: 1024px / patch=16 -> grid_ref=64.
rope_ref_w: int = 1024,
image_model=None,
@ -165,6 +185,8 @@ class PidNet(PixDiT_T2I):
for blk in self.pixel_blocks:
blk._rope_fn = _pit_rope_fn
self.pit_lq_inject = pit_lq_inject
num_lq_outputs = (self.patch_depth + lq_interval - 1) // lq_interval
self.lq_proj = LQProjection2D(
latent_channels=lq_latent_channels,
@ -173,13 +195,20 @@ class PidNet(PixDiT_T2I):
patch_size=self.patch_size,
sr_scale=sr_scale,
latent_spatial_down_factor=latent_spatial_down_factor,
latent_unpatchify_factor=lq_latent_unpatchify_factor,
num_res_blocks=lq_num_res_blocks,
num_outputs=num_lq_outputs,
interval=lq_interval,
conv_padding_mode=lq_conv_padding_mode,
gate_per_token=lq_gate_per_token,
pit_output=pit_lq_inject,
dtype=dtype,
device=device,
operations=operations,
)
self.pit_lq_gate = SigmaAwareGate(
self.hidden_size, per_token=lq_gate_per_token, dtype=dtype, device=device, operations=operations
) if pit_lq_inject else None
def _fetch_patch_pos(self, height, width, device, dtype, **rope_opts):
return precompute_freqs_cis_2d(
@ -197,6 +226,11 @@ class PidNet(PixDiT_T2I):
return s
return self.lq_proj.gate(s, pid_lq_features[out_idx], pid_degrade_sigma, out_idx)
def _pre_pixel_blocks(self, s, pid_pit_lq_feature=None, pid_degrade_sigma=None, **kwargs):
if pid_pit_lq_feature is None:
return s
return self.pit_lq_gate(s, pid_pit_lq_feature, pid_degrade_sigma)
def _forward(self, x, timesteps, context=None, attention_mask=None, transformer_options={}, lq_latent=None, degrade_sigma=None, **kwargs):
if lq_latent is None:
raise ValueError("PidNet requires lq_latent — attach via PiDConditioning")
@ -216,12 +250,14 @@ class PidNet(PixDiT_T2I):
degrade_sigma = degrade_sigma.expand(B).contiguous()
lq_features = self.lq_proj(lq_latent=lq_latent.to(x), target_pH=Hs, target_pW=Ws)
pit_lq_feature = lq_features.pop() if self.pit_lq_inject else None
return super()._forward(
x, timesteps,
context=context, attention_mask=attention_mask,
transformer_options=transformer_options,
pid_lq_features=lq_features,
pid_pit_lq_feature=pit_lq_feature,
pid_degrade_sigma=degrade_sigma,
**kwargs,
)

51
comfy/ldm/seedvr/attention.py Normal file
View File

@ -0,0 +1,51 @@
import torch
from comfy.ldm.modules import attention as _attention
def _var_attention_qkv(q, k, v, heads, skip_reshape):
if skip_reshape:
return q, k, v, q.shape[-1]
total_tokens, embed_dim = q.shape
head_dim = embed_dim // heads
return (
q.view(total_tokens, heads, head_dim),
k.view(k.shape[0], heads, head_dim),
v.view(v.shape[0], heads, head_dim),
head_dim,
)
def _var_attention_output(out, heads, head_dim, skip_output_reshape):
if skip_output_reshape:
return out
return out.reshape(-1, heads * head_dim)
def var_attention_optimized_split(q, k, v, heads, cu_seqlens_q, cu_seqlens_k, *args, skip_reshape=False, skip_output_reshape=False, **kwargs):
q, k, v, head_dim = _var_attention_qkv(q, k, v, heads, skip_reshape)
q_split_indices = cu_seqlens_q[1:-1]
k_split_indices = cu_seqlens_k[1:-1]
if k.shape[0] != v.shape[0]:
raise ValueError("cu_seqlens_k does not match v token count")
q_splits = torch.tensor_split(q, q_split_indices, dim=0)
k_splits = torch.tensor_split(k, k_split_indices, dim=0)
v_splits = torch.tensor_split(v, k_split_indices, dim=0)
if len(q_splits) != len(k_splits) or len(q_splits) != len(v_splits):
raise ValueError("cu_seqlens_q and cu_seqlens_k must describe the same sequence count")
out = []
for q_i, k_i, v_i in zip(q_splits, k_splits, v_splits):
q_i = q_i.permute(1, 0, 2).unsqueeze(0)
k_i = k_i.permute(1, 0, 2).unsqueeze(0)
v_i = v_i.permute(1, 0, 2).unsqueeze(0)
out_i = _attention.optimized_attention(q_i, k_i, v_i, heads, skip_reshape=True, skip_output_reshape=True)
out.append(out_i.squeeze(0).permute(1, 0, 2))
out = torch.cat(out, dim=0)
return _var_attention_output(out, heads, head_dim, skip_output_reshape)
optimized_var_attention = var_attention_optimized_split

301
comfy/ldm/seedvr/color_fix.py Normal file
View File

@ -0,0 +1,301 @@
import torch
import torch.nn.functional as F
from torch import Tensor
from comfy.ldm.seedvr.constants import (
CIELAB_DELTA,
CIELAB_KAPPA,
D65_WHITE_X,
D65_WHITE_Z,
WAVELET_DECOMP_LEVELS,
)
def wavelet_blur(image: Tensor, radius):
max_safe_radius = max(1, min(image.shape[-2:]) // 8)
if radius > max_safe_radius:
radius = max_safe_radius
num_channels = image.shape[1]
kernel_vals = [
[0.0625, 0.125, 0.0625],
[0.125, 0.25, 0.125],
[0.0625, 0.125, 0.0625],
]
kernel = torch.tensor(kernel_vals, dtype=image.dtype, device=image.device)
kernel = kernel[None, None].repeat(num_channels, 1, 1, 1)
image = F.pad(image, (radius, radius, radius, radius), mode='replicate')
output = F.conv2d(image, kernel, groups=num_channels, dilation=radius)
return output
def wavelet_decomposition(image: Tensor, levels: int = WAVELET_DECOMP_LEVELS):
high_freq = torch.zeros_like(image)
for i in range(levels):
radius = 2 ** i
low_freq = wavelet_blur(image, radius)
high_freq.add_(image).sub_(low_freq)
image = low_freq
return high_freq, low_freq
def wavelet_reconstruction(content_feat: Tensor, style_feat: Tensor) -> Tensor:
if content_feat.shape != style_feat.shape:
if len(content_feat.shape) >= 3:
style_feat = F.interpolate(
style_feat,
size=content_feat.shape[-2:],
mode='bilinear',
align_corners=False
)
content_high_freq, content_low_freq = wavelet_decomposition(content_feat)
del content_low_freq
style_high_freq, style_low_freq = wavelet_decomposition(style_feat)
del style_high_freq
if content_high_freq.shape != style_low_freq.shape:
style_low_freq = F.interpolate(
style_low_freq,
size=content_high_freq.shape[-2:],
mode='bilinear',
align_corners=False
)
content_high_freq.add_(style_low_freq)
return content_high_freq.clamp_(-1.0, 1.0)
def _histogram_matching_channel(source: Tensor, reference: Tensor) -> Tensor:
original_shape = source.shape
source_flat = source.flatten()
reference_flat = reference.flatten()
source_sorted, source_indices = torch.sort(source_flat)
reference_sorted, _ = torch.sort(reference_flat)
del reference_flat
n_source = len(source_sorted)
n_reference = len(reference_sorted)
if n_source == n_reference:
matched_sorted = reference_sorted
else:
source_quantiles = torch.linspace(0, 1, n_source, device=source.device)
ref_indices = (source_quantiles * (n_reference - 1)).long()
ref_indices.clamp_(0, n_reference - 1)
matched_sorted = reference_sorted[ref_indices]
del source_quantiles, ref_indices, reference_sorted
del source_sorted, source_flat
inverse_indices = torch.argsort(source_indices)
del source_indices
matched_flat = matched_sorted[inverse_indices]
del matched_sorted, inverse_indices
return matched_flat.reshape(original_shape)
def _lab_to_rgb_batch(lab: Tensor, matrix_inv: Tensor, epsilon: float, kappa: float) -> Tensor:
L, a, b = lab[:, 0], lab[:, 1], lab[:, 2]
fy = (L + 16.0) / 116.0
fx = a.div(500.0).add_(fy)
fz = fy - b / 200.0
del L, a, b
x = torch.where(
fx > epsilon,
torch.pow(fx, 3.0),
fx.mul(116.0).sub_(16.0).div_(kappa)
)
y = torch.where(
fy > epsilon,
torch.pow(fy, 3.0),
fy.mul(116.0).sub_(16.0).div_(kappa)
)
z = torch.where(
fz > epsilon,
torch.pow(fz, 3.0),
fz.mul(116.0).sub_(16.0).div_(kappa)
)
del fx, fy, fz
x.mul_(D65_WHITE_X)
z.mul_(D65_WHITE_Z)
xyz = torch.stack([x, y, z], dim=1)
del x, y, z
B, _, H, W = xyz.shape
xyz_flat = xyz.permute(0, 2, 3, 1).reshape(-1, 3)
del xyz
xyz_flat = xyz_flat.to(dtype=matrix_inv.dtype)
rgb_linear_flat = torch.matmul(xyz_flat, matrix_inv.T)
del xyz_flat
rgb_linear = rgb_linear_flat.reshape(B, H, W, 3).permute(0, 3, 1, 2)
del rgb_linear_flat
mask = rgb_linear > 0.0031308
rgb = torch.where(
mask,
torch.pow(torch.clamp(rgb_linear, min=0.0), 1.0 / 2.4).mul_(1.055).sub_(0.055),
rgb_linear * 12.92
)
del mask, rgb_linear
return torch.clamp(rgb, 0.0, 1.0)
def _rgb_to_lab_batch(rgb: Tensor, matrix: Tensor, epsilon: float, kappa: float) -> Tensor:
mask = rgb > 0.04045
rgb_linear = torch.where(
mask,
torch.pow((rgb + 0.055) / 1.055, 2.4),
rgb / 12.92
)
del mask
B, _, H, W = rgb_linear.shape
rgb_flat = rgb_linear.permute(0, 2, 3, 1).reshape(-1, 3)
del rgb_linear
rgb_flat = rgb_flat.to(dtype=matrix.dtype)
xyz_flat = torch.matmul(rgb_flat, matrix.T)
del rgb_flat
xyz = xyz_flat.reshape(B, H, W, 3).permute(0, 3, 1, 2)
del xyz_flat
xyz[:, 0].div_(D65_WHITE_X)
xyz[:, 2].div_(D65_WHITE_Z)
epsilon_cubed = epsilon ** 3
mask = xyz > epsilon_cubed
f_xyz = torch.where(
mask,
torch.pow(xyz, 1.0 / 3.0),
xyz.mul(kappa).add_(16.0).div_(116.0)
)
del xyz, mask
L = f_xyz[:, 1].mul(116.0).sub_(16.0)
a = (f_xyz[:, 0] - f_xyz[:, 1]).mul_(500.0)
b = (f_xyz[:, 1] - f_xyz[:, 2]).mul_(200.0)
del f_xyz
return torch.stack([L, a, b], dim=1)
def lab_color_transfer(
content_feat: Tensor,
style_feat: Tensor,
luminance_weight: float = 0.8
) -> Tensor:
content_feat = wavelet_reconstruction(content_feat, style_feat)
if content_feat.shape != style_feat.shape:
style_feat = F.interpolate(
style_feat,
size=content_feat.shape[-2:],
mode='bilinear',
align_corners=False
)
device = content_feat.device
original_dtype = content_feat.dtype
content_feat = content_feat.float()
style_feat = style_feat.float()
rgb_to_xyz_matrix = torch.tensor([
[0.4124564, 0.3575761, 0.1804375],
[0.2126729, 0.7151522, 0.0721750],
[0.0193339, 0.1191920, 0.9503041]
], dtype=torch.float32, device=device)
xyz_to_rgb_matrix = torch.tensor([
[ 3.2404542, -1.5371385, -0.4985314],
[-0.9692660, 1.8760108, 0.0415560],
[ 0.0556434, -0.2040259, 1.0572252]
], dtype=torch.float32, device=device)
epsilon = CIELAB_DELTA
kappa = CIELAB_KAPPA
content_feat.add_(1.0).mul_(0.5).clamp_(0.0, 1.0)
style_feat.add_(1.0).mul_(0.5).clamp_(0.0, 1.0)
content_lab = _rgb_to_lab_batch(content_feat, rgb_to_xyz_matrix, epsilon, kappa)
del content_feat
style_lab = _rgb_to_lab_batch(style_feat, rgb_to_xyz_matrix, epsilon, kappa)
del style_feat, rgb_to_xyz_matrix
matched_a = _histogram_matching_channel(content_lab[:, 1], style_lab[:, 1])
matched_b = _histogram_matching_channel(content_lab[:, 2], style_lab[:, 2])
if luminance_weight < 1.0:
matched_L = _histogram_matching_channel(content_lab[:, 0], style_lab[:, 0])
result_L = content_lab[:, 0].mul(luminance_weight).add_(matched_L.mul(1.0 - luminance_weight))
del matched_L
else:
result_L = content_lab[:, 0]
del content_lab, style_lab
result_lab = torch.stack([result_L, matched_a, matched_b], dim=1)
del result_L, matched_a, matched_b
result_rgb = _lab_to_rgb_batch(result_lab, xyz_to_rgb_matrix, epsilon, kappa)
del result_lab, xyz_to_rgb_matrix
result = result_rgb.mul_(2.0).sub_(1.0)
del result_rgb
result = result.to(original_dtype)
return result
def wavelet_color_transfer(content_feat: Tensor, style_feat: Tensor) -> Tensor:
return wavelet_reconstruction(content_feat, style_feat)
def adain_color_transfer(content_feat: Tensor, style_feat: Tensor, eps: float = 1e-5) -> Tensor:
if content_feat.shape != style_feat.shape:
style_feat = F.interpolate(
style_feat,
size=content_feat.shape[-2:],
mode='bilinear',
align_corners=False,
)
original_dtype = content_feat.dtype
content_feat = content_feat.float()
style_feat = style_feat.float()
b, c = content_feat.shape[:2]
content_flat = content_feat.reshape(b, c, -1)
style_flat = style_feat.reshape(b, c, -1)
content_mean = content_flat.mean(dim=2).reshape(b, c, 1, 1)
content_std = (content_flat.var(dim=2, correction=0) + eps).sqrt().reshape(b, c, 1, 1)
style_mean = style_flat.mean(dim=2).reshape(b, c, 1, 1)
style_std = (style_flat.var(dim=2, correction=0) + eps).sqrt().reshape(b, c, 1, 1)
del content_flat, style_flat
normalized = (content_feat - content_mean) / content_std
del content_mean, content_std
result = normalized * style_std + style_mean
del normalized, style_mean, style_std
result = result.clamp_(-1.0, 1.0)
if result.dtype != original_dtype:
result = result.to(original_dtype)
return result

48
comfy/ldm/seedvr/constants.py Normal file
View File

@ -0,0 +1,48 @@
"""SeedVR2 constants."""
# Temporal chunk-size law: the sampler's activation wall is linear in
# T_latent * pixel area (17-cell resolution sweep + T bisection, RTX 5090, 3b fp16):
# max_latent_frames = (free_GiB - RESERVED - K*SIGMA) / (GIB_PER_MPX_FRAME * megapixels)
# RESERVED covers model staging plus fixed CUDA/torch overhead; SIGMA is the measured
# run-to-run spread of the wall; K=4 trades ~10% smaller chunks for ~1e-5 OOM odds.
SEEDVR2_CHUNK_GIB_PER_MPX_FRAME = 0.55
SEEDVR2_CHUNK_RESERVED_GIB = 8.5
SEEDVR2_CHUNK_SIGMA_GIB = 0.55
SEEDVR2_CHUNK_SIGMA_K = 4
SEEDVR2_7B_VID_DIM = 3072
SEEDVR2_OOM_BACKOFF_DIVISOR = 2
SEEDVR2_DTYPE_BYTES_FLOOR = 4
SEEDVR2_7B_MLP_CHUNK = 8192
SEEDVR2_ROPE_PARTIAL_CHUNK_TOKENS = 4096 # partial-RoPE application token-chunk.
SEEDVR2_LATENT_CHANNELS = 16
SEEDVR2_COLOR_MEM_HEADROOM = 0.75
SEEDVR2_LAB_SCALE_MULTIPLIER = 13
SEEDVR2_WAVELET_SCALE_MULTIPLIER = 10 # per-frame byte multiplier, wavelet path.
SEEDVR2_ADAIN_SCALE_MULTIPLIER = 6
BYTEDANCE_VAE_SCALING_FACTOR = 0.9152 # configs_3b/main.yaml:57.
BYTEDANCE_VAE_SHIFTING_FACTOR = 0.0
BYTEDANCE_VAE_CONV_MEM_GIB = 0.5
BYTEDANCE_VAE_NORM_MEM_GIB = 0.5
BYTEDANCE_LOGVAR_CLAMP_MIN = -30.0 # video_vae_v3/modules/types.py:28.
BYTEDANCE_LOGVAR_CLAMP_MAX = 20.0 # video_vae_v3/modules/types.py:28.
BYTEDANCE_GN_CHUNKS_FP16 = 4 # causal_inflation_lib.py:351 (GroupNorm chunk count, fp16).
BYTEDANCE_GN_CHUNKS_FP32 = 2 # causal_inflation_lib.py:351 (GroupNorm chunk count, fp32).
BYTEDANCE_BLOCK_OUT_CHANNELS = (128, 256, 512, 512) # s8_c16_t4_inflation_sd3.yaml:7-11.
BYTEDANCE_SLICING_SAMPLE_MIN = 4 # s8_c16_t4_inflation_sd3.yaml:22 (slicing_sample_min_size).
BYTEDANCE_VAE_TEMPORAL_DOWNSAMPLE = 4 # infer.py:230 (temporal_downsample_factor); the 4n+1 factor.
BYTEDANCE_VAE_SPATIAL_DOWNSAMPLE = 8 # infer.py:231 (spatial_downsample_factor).
BYTEDANCE_720P_REF_AREA = 45 * 80 # dit_v2/window.py:32 (720p reference area for window scaling).
BYTEDANCE_MAX_TEMPORAL_WINDOW = 30 # dit_v2/window.py:35 (max temporal window frames).
BYTEDANCE_ROPE_MAX_FREQ = 256 # dit_v2/rope.py:31 (pixel-RoPE max frequency).
BYTEDANCE_SINUSOIDAL_DIM = 256 # dit_3b/nadit.py:120 (timestep sinusoidal embed dim).
ROPE_THETA = 10000 # RoPE base; Su et al., "RoFormer", arXiv:2104.09864.
CIELAB_DELTA = 6.0 / 29.0 # CIE 15 (delta).
CIELAB_KAPPA = (29.0 / 3.0) ** 3 # CIE 15 (kappa).
D65_WHITE_X = 0.95047 # CIE D65 standard illuminant Xn (Yn = 1).
D65_WHITE_Z = 1.08883 # CIE D65 standard illuminant Zn.
WAVELET_DECOMP_LEVELS = 5 # wavelet color-fix decomposition depth (GIMP/Krita; StableSR).

1361
comfy/ldm/seedvr/model.py Normal file
View File

File diff suppressed because it is too large Load Diff

1610
comfy/ldm/seedvr/vae.py Normal file
View File

File diff suppressed because it is too large Load Diff

35
comfy/model_base.py
View File

@ -55,8 +55,10 @@ import comfy.ldm.pixeldit.model
import comfy.ldm.pixeldit.pid
import comfy.ldm.ace.model
import comfy.ldm.omnigen.omnigen2
import comfy.ldm.seedvr.model
import comfy.ldm.boogu.model
import comfy.ldm.qwen_image.model
import comfy.ldm.joyimage.model
import comfy.ldm.ideogram4.model
import comfy.ldm.krea2.model
import comfy.ldm.kandinsky5.model
@ -933,6 +935,17 @@ class HunyuanDiT(BaseModel):
out['image_meta_size'] = comfy.conds.CONDRegular(torch.FloatTensor([[height, width, target_height, target_width, 0, 0]]))
return out
class SeedVR2(BaseModel):
def __init__(self, model_config, model_type=ModelType.FLOW, device=None):
super().__init__(model_config, model_type, device=device, unet_model=comfy.ldm.seedvr.model.NaDiT)
def extra_conds(self, **kwargs):
out = super().extra_conds(**kwargs)
condition = kwargs.get("condition", None)
if condition is not None:
out["condition"] = comfy.conds.CONDRegular(condition)
return out
class PixArt(BaseModel):
def __init__(self, model_config, model_type=ModelType.EPS, device=None):
super().__init__(model_config, model_type, device=device, unet_model=comfy.ldm.pixart.pixartms.PixArtMS)
@ -2286,6 +2299,28 @@ class QwenImage(BaseModel):
out['ref_latents'] = list([1, 16, sum(map(lambda a: math.prod(a.size()), ref_latents)) // 16])
return out
class JoyImage(BaseModel):
def __init__(self, model_config, model_type=ModelType.FLOW, device=None):
super().__init__(model_config, model_type, device=device, unet_model=comfy.ldm.joyimage.model.JoyImageTransformer3DModel)
self.memory_usage_factor_conds = ("ref_latents",)
def extra_conds(self, **kwargs):
out = super().extra_conds(**kwargs)
cross_attn = kwargs.get("cross_attn", None)
if cross_attn is not None:
out['c_crossattn'] = comfy.conds.CONDRegular(cross_attn)
ref_latents = kwargs.get("reference_latents", None)
if ref_latents is not None:
out['ref_latents'] = comfy.conds.CONDList([self.process_latent_in(lat) for lat in ref_latents])
return out
def extra_conds_shapes(self, **kwargs):
out = {}
ref_latents = kwargs.get("reference_latents", None)
if ref_latents is not None:
out['ref_latents'] = list([1, 16, sum(map(lambda a: math.prod(a.size()), ref_latents)) // 16])
return out
class Ideogram4(BaseModel):
def __init__(self, model_config, model_type=ModelType.FLOW, device=None):
super().__init__(model_config, model_type, device=device, unet_model=comfy.ldm.ideogram4.model.Ideogram4Transformer2DModel)

101
comfy/model_detection.py
View File

@ -489,15 +489,46 @@ def detect_unet_config(state_dict, key_prefix, metadata=None):
# PiD (Pixel Diffusion Decoder). Must check BEFORE plain PixelDiT_T2I.
_lq_w_key = '{}lq_proj.latent_proj.0.weight'.format(key_prefix)
if _lq_w_key in state_dict_keys:
in_ch = int(state_dict[_lq_w_key].shape[1])
latent_proj_in_channels = int(state_dict[_lq_w_key].shape[1])
hidden_dim = int(state_dict[_lq_w_key].shape[0])
_gate_prefix = '{}lq_proj.gate_modules.'.format(key_prefix)
num_gates = len({k[len(_gate_prefix):].split('.')[0]
for k in state_dict_keys if k.startswith(_gate_prefix)})
pid_v1_5 = '{}lq_proj.pit_head.weight'.format(key_prefix) in state_dict_keys
dit_config = {"image_model": "pid",
"lq_latent_channels": in_ch,
"latent_spatial_down_factor": 16 if in_ch >= 64 else 8}
"lq_hidden_dim": hidden_dim}
if num_gates > 0:
dit_config["lq_interval"] = (14 + num_gates - 1) // num_gates
if pid_v1_5:
pid_v1_5_variants = {
16: { # Flux and QwenImage
"lq_latent_channels": 16,
"latent_spatial_down_factor": 8,
"lq_latent_unpatchify_factor": 1,
},
32: { # Flux2 after 2x latent unpatchify
"lq_latent_channels": 128,
"latent_spatial_down_factor": 16,
"lq_latent_unpatchify_factor": 2,
},
}
variant = pid_v1_5_variants.get(latent_proj_in_channels)
if variant is None:
raise ValueError(f"Unsupported PiD v1.5 latent projection with {latent_proj_in_channels} input channels")
gate_weight = state_dict['{}lq_proj.gate_modules.0.content_proj.weight'.format(key_prefix)]
dit_config.update(variant)
dit_config.update({
"lq_conv_padding_mode": "replicate",
"lq_gate_per_token": gate_weight.shape[0] == 1,
"pit_lq_inject": True,
"rope_ref_h": 2048,
"rope_ref_w": 2048,
})
else:
dit_config.update({
"lq_latent_channels": latent_proj_in_channels,
"latent_spatial_down_factor": 16 if latent_proj_in_channels >= 64 else 8,
})
return dit_config
if '{}core.pixel_embedder.proj.weight'.format(key_prefix) in state_dict_keys: # PixelDiT T2I
@ -617,6 +648,44 @@ def detect_unet_config(state_dict, key_prefix, metadata=None):
return dit_config
seedvr2_7b_separate_key = "{}blocks.35.mlp.vid.proj_out.weight".format(key_prefix)
if seedvr2_7b_separate_key in state_dict_keys and state_dict[seedvr2_7b_separate_key].shape[0] == 3072: # seedvr2 7b
dit_config = {}
dit_config["image_model"] = "seedvr2"
dit_config["vid_dim"] = 3072
dit_config["heads"] = 24
dit_config["num_layers"] = 36
# This checkpoint uses separate vid/txt MMModule keys in every block.
dit_config["mm_layers"] = 36
dit_config["norm_eps"] = 1e-5
dit_config["rope_type"] = "rope3d"
dit_config["rope_dim"] = 64
dit_config["mlp_type"] = "normal"
return dit_config
if "{}blocks.35.mlp.all.proj_in_gate.weight".format(key_prefix) in state_dict_keys: # seedvr2 7b
dit_config = {}
dit_config["image_model"] = "seedvr2"
dit_config["vid_dim"] = 3072
dit_config["heads"] = 24
dit_config["num_layers"] = 36
# This checkpoint uses shared all.* MMModule keys after the initial blocks.
dit_config["mm_layers"] = 10
dit_config["norm_eps"] = 1e-5
dit_config["rope_type"] = "rope3d"
dit_config["rope_dim"] = 64
dit_config["mlp_type"] = "swiglu"
return dit_config
if "{}blocks.31.mlp.all.proj_in_gate.weight".format(key_prefix) in state_dict_keys: # seedvr2 3b
dit_config = {}
dit_config["image_model"] = "seedvr2"
dit_config["vid_dim"] = 2560
dit_config["heads"] = 20
dit_config["num_layers"] = 32
dit_config["norm_eps"] = 1.0e-05
dit_config["mlp_type"] = "swiglu"
dit_config["vid_out_norm"] = True
return dit_config
if '{}head.modulation'.format(key_prefix) in state_dict_keys: # Wan 2.1
dit_config = {}
dit_config["image_model"] = "wan2.1"
@ -1008,6 +1077,25 @@ def detect_unet_config(state_dict, key_prefix, metadata=None):
dit_config["image_model"] = "SAM31"
return dit_config
if (
'{}double_blocks.0.attn.img_attn_qkv.weight'.format(key_prefix) in state_dict_keys
and '{}double_blocks.0.attn.img_attn_q_norm.weight'.format(key_prefix) in state_dict_keys
and '{}condition_embedder.time_embedder.linear_1.weight'.format(key_prefix) in state_dict_keys
and '{}img_in.weight'.format(key_prefix) in state_dict_keys
and len(state_dict['{}img_in.weight'.format(key_prefix)].shape) == 5
):
img_in = state_dict['{}img_in.weight'.format(key_prefix)]
head_dim = state_dict['{}double_blocks.0.attn.img_attn_q_norm.weight'.format(key_prefix)].shape[0]
return {
"image_model": "joyimage",
"in_channels": img_in.shape[1],
"hidden_size": img_in.shape[0],
"patch_size": list(img_in.shape[2:]),
"num_layers": count_blocks(state_dict_keys, '{}double_blocks.'.format(key_prefix) + '{}.'),
"num_attention_heads": img_in.shape[0] // head_dim,
"text_dim": 4096,
}
if '{}input_blocks.0.0.weight'.format(key_prefix) not in state_dict_keys:
return None
@ -1138,9 +1226,10 @@ def detect_unet_config(state_dict, key_prefix, metadata=None):
return unet_config
def model_config_from_unet_config(unet_config, state_dict=None):
def model_config_from_unet_config(unet_config, state_dict=None, unet_key_prefix=""):
for model_config in comfy.supported_models.models:
if model_config.matches(unet_config, state_dict):
if model_config.matches(unet_config, state_dict, unet_key_prefix=unet_key_prefix):
return model_config(unet_config)
logging.error("no match {}".format(unet_config))
@ -1150,7 +1239,7 @@ def model_config_from_unet(state_dict, unet_key_prefix, use_base_if_no_match=Fal
unet_config = detect_unet_config(state_dict, unet_key_prefix, metadata=metadata)
if unet_config is None:
return None
model_config = model_config_from_unet_config(unet_config, state_dict)
model_config = model_config_from_unet_config(unet_config, state_dict, unet_key_prefix)
if model_config is None and use_base_if_no_match:
model_config = comfy.supported_models_base.BASE(unet_config)

11
comfy/model_management.py
View File

@ -616,6 +616,8 @@ PIN_PRESSURE_HYSTERESIS = 256 * 1024 * 1024
#Freeing registerables on pressure does imply a GPU sync, so go big on
#the hysteresis so each expensive sync gives us back a good chunk.
REGISTERABLE_PIN_HYSTERESIS = 2048 * 1024 * 1024
WINDOWS_PIN_EVICTION_SWAP_PERCENT = 5.0
WINDOWS_PIN_EVICTION_EMERGENCY_AVAILABLE = 512 * 1024 ** 2
def module_size(module):
module_mem = 0
@ -642,6 +644,15 @@ def free_pins(size, evict_active=False):
size -= freed
return freed_total
def should_free_pins_for_ram_pressure(shortfall):
if shortfall <= 0:
return False
if not WINDOWS:
return True
if psutil.virtual_memory().available < WINDOWS_PIN_EVICTION_EMERGENCY_AVAILABLE:
return True
return psutil.swap_memory().percent >= WINDOWS_PIN_EVICTION_SWAP_PERCENT
def ensure_pin_budget(size, evict_active=False):
if args.high_ram:
return True

42
comfy/ops.py
View File

@ -174,6 +174,8 @@ def cast_modules_with_vbar(comfy_modules, dtype, device, bias_dtype, non_blockin
elif xfer_dest2 is not None:
xfer_source.prepare(xfer_dest2, stream, copy=True, commit=False)
return
else:
return
comfy.model_management.cast_to_gathered(xfer_source, xfer_dest, non_blocking=non_blocking, stream=stream, r2=xfer_dest2)
def handle_pin(m, pin, source, dest, subset="weights", size=None):
@ -1102,6 +1104,21 @@ def _load_quantized_module(module, super_load, state_dict, prefix, local_metadat
scales["convrot_groupsize"] = int(
layer_conf.get("convrot_groupsize", params_conf.get("convrot_groupsize", 256))
)
elif module.quant_format == "convrot_w4a4":
scale = pop_scale("weight_scale")
if scale is None:
raise ValueError(f"Missing ConvRot W4A4 weight scale for layer {layer_name}")
params_conf = layer_conf.get("params", {})
if not isinstance(params_conf, dict):
params_conf = {}
scales = {
"scale": scale,
"convrot_groupsize": int(
layer_conf.get("convrot_groupsize", params_conf.get("convrot_groupsize", 256))
),
"quant_group_size": 64,
"linear_dtype": layer_conf.get("linear_dtype", params_conf.get("linear_dtype", "int4")),
}
else:
raise ValueError(f"Unsupported quantization format: {module.quant_format}")
@ -1148,6 +1165,11 @@ def _quantized_weight_state_dict(module, sd, prefix, extra_quant_conf=None, extr
if module.quant_format == "int8_tensorwise" and getattr(params, "convrot", False):
quant_conf["convrot"] = True
quant_conf["convrot_groupsize"] = getattr(params, "convrot_groupsize", 256)
elif module.quant_format == "convrot_w4a4":
quant_conf["convrot_groupsize"] = getattr(params, "convrot_groupsize", 256)
linear_dtype = getattr(params, "linear_dtype", "int4")
if linear_dtype != "int4":
quant_conf["linear_dtype"] = linear_dtype
if extra_quant_conf:
quant_conf.update(extra_quant_conf)
sd[f"{prefix}comfy_quant"] = torch.tensor(list(json.dumps(quant_conf).encode("utf-8")), dtype=torch.uint8)
@ -1235,7 +1257,7 @@ def mixed_precision_ops(quant_config={}, compute_dtype=torch.bfloat16, full_prec
run_every_op()
input_shape = input.shape
reshaped_3d = False
reshaped_nd = False
#If cast needs to apply lora, it should be done in the compute dtype
compute_dtype = input.dtype
@ -1272,12 +1294,12 @@ def mixed_precision_ops(quant_config={}, compute_dtype=torch.bfloat16, full_prec
# Inference path (unchanged)
if _use_quantized and quantize_input:
# Reshape 3D tensors to 2D for quantization (needed for NVFP4 and others)
input_reshaped = input.reshape(-1, input_shape[2]) if input.ndim == 3 else input
# Reshape >=3D tensors to 2D for quantization (needed for NVFP4 and others)
input_reshaped = input.reshape(-1, input_shape[-1]) if input.ndim >= 3 else input
# Fall back to non-quantized for non-2D tensors
if input_reshaped.ndim == 2:
reshaped_3d = input.ndim == 3
reshaped_nd = input.ndim >= 3
# dtype is now implicit in the layout class
scale = getattr(self, 'input_scale', None)
if scale is not None:
@ -1292,9 +1314,9 @@ def mixed_precision_ops(quant_config={}, compute_dtype=torch.bfloat16, full_prec
weight_only_quant=weight_only_quant,
)
# Reshape output back to 3D if input was 3D
if reshaped_3d:
output = output.reshape((input_shape[0], input_shape[1], self.weight.shape[0]))
# Reshape output back to original rank if input was >2D
if reshaped_nd:
output = output.reshape((*input_shape[:-1], self.weight.shape[0]))
return output
@ -1428,6 +1450,12 @@ def mixed_precision_ops(quant_config={}, compute_dtype=torch.bfloat16, full_prec
}
if hasattr(params, "block_scale"): # NVFP4
kwargs["block_scale"] = params.block_scale[i]
if hasattr(params, "quant_group_size"):
kwargs["quant_group_size"] = params.quant_group_size
if hasattr(params, "convrot_groupsize"):
kwargs["convrot_groupsize"] = params.convrot_groupsize
if hasattr(params, "linear_dtype"):
kwargs["linear_dtype"] = params.linear_dtype
return QuantizedTensor(weight._qdata[i], weight._layout_cls, type(params)(**kwargs))
def state_dict(self, *args, destination=None, prefix="", **kwargs):

46
comfy/quant_ops.py
View File

@ -3,6 +3,22 @@ import logging
from comfy.cli_args import args
def _rocm_kitchen_arch_supported():
"""comfy-kitchen's INT8 Triton kernels compile tl.dot to matrix-core instructions.
RDNA3/3.5/4 (gfx11xx/gfx12xx) have WMMA and CDNA (gfx9xx) has MFMA; RDNA1/RDNA2
(gfx10xx) have neither, so the INT8 path hangs the GPU there. Gates the automatic
ROCm default so those cards stay on the eager fallback (an explicit
--enable-triton-backend still forces it on any arch)."""
try:
arch = torch.cuda.get_device_properties(torch.cuda.current_device()).gcnArchName.split(":")[0]
except Exception:
return False
if arch.startswith(("gfx11", "gfx12")):
return True
return arch in ("gfx908", "gfx90a", "gfx940", "gfx941", "gfx942", "gfx950")
try:
import comfy_kitchen as ck
from comfy_kitchen.tensor import (
@ -10,6 +26,7 @@ try:
QuantizedLayout,
TensorCoreFP8Layout as _CKFp8Layout,
TensorCoreNVFP4Layout as _CKNvfp4Layout,
TensorCoreConvRotW4A4Layout as _CKTensorCoreConvRotW4A4Layout,
TensorWiseINT8Layout as _CKTensorWiseINT8Layout,
register_layout_op,
register_layout_class,
@ -24,10 +41,22 @@ try:
ck.registry.disable("cuda")
logging.warning("WARNING: You need pytorch with cu130 or higher to use optimized CUDA operations.")
if args.enable_triton_backend:
# On ROCm/AMD the CUDA backend is unavailable, so Triton is the only accelerated
# comfy-kitchen backend. Enable it by default there, but only on Triton >= 3.7 AND a
# matrix-core GPU (RDNA3+ WMMA gfx11xx/gfx12xx, CDNA MFMA gfx9xx). RDNA1/RDNA2
# (gfx10xx) have no WMMA -> the INT8 tl.dot path hangs the GPU, so they stay eager.
# older Triton lacks libdevice.rint on the HIP backend and hard-crashes the INT8 path.
if args.disable_triton_backend:
ck.registry.disable("triton")
elif args.enable_triton_backend: # or (torch.version.hip is not None and _rocm_kitchen_arch_supported()):
try:
import triton
logging.info("Found triton %s. Enabling comfy-kitchen triton backend.", triton.__version__)
triton_version = tuple(int(v) for v in triton.__version__.split(".")[:2])
if args.enable_triton_backend or triton_version >= (3, 7):
logging.info("Found triton %s. Enabling comfy-kitchen triton backend.", triton.__version__)
else:
logging.info("Triton %s is too old for the ROCm INT8 path (needs >= 3.7); comfy-kitchen triton backend disabled.", triton.__version__)
ck.registry.disable("triton")
except ImportError as e:
logging.error(f"Failed to import triton, Error: {e}, the comfy-kitchen triton backend will not be available.")
ck.registry.disable("triton")
@ -51,6 +80,9 @@ except ImportError as e:
class _CKTensorWiseINT8Layout:
pass
class _CKTensorCoreConvRotW4A4Layout:
pass
def register_layout_class(name, cls):
pass
@ -179,6 +211,7 @@ class TensorCoreFP8E5M2Layout(_TensorCoreFP8LayoutBase):
# Backward compatibility alias - default to E4M3
TensorCoreFP8Layout = TensorCoreFP8E4M3Layout
TensorWiseINT8Layout = _CKTensorWiseINT8Layout
TensorCoreConvRotW4A4Layout = _CKTensorCoreConvRotW4A4Layout
# ==============================================================================
@ -190,6 +223,7 @@ register_layout_class("TensorCoreFP8E4M3Layout", TensorCoreFP8E4M3Layout)
register_layout_class("TensorCoreFP8E5M2Layout", TensorCoreFP8E5M2Layout)
register_layout_class("TensorCoreNVFP4Layout", TensorCoreNVFP4Layout)
register_layout_class("TensorWiseINT8Layout", _CKTensorWiseINT8Layout)
register_layout_class("TensorCoreConvRotW4A4Layout", _CKTensorCoreConvRotW4A4Layout)
if _CK_MXFP8_AVAILABLE:
register_layout_class("TensorCoreMXFP8Layout", TensorCoreMXFP8Layout)
@ -227,6 +261,13 @@ QUANT_ALGOS["int8_tensorwise"] = {
"quantize_input": False,
}
QUANT_ALGOS["convrot_w4a4"] = {
"storage_t": torch.int8,
"parameters": {"weight_scale"},
"comfy_tensor_layout": "TensorCoreConvRotW4A4Layout",
"quantize_input": False,
}
# ==============================================================================
# Re-exports for backward compatibility
@ -239,6 +280,7 @@ __all__ = [
"TensorCoreFP8E4M3Layout",
"TensorCoreFP8E5M2Layout",
"TensorCoreNVFP4Layout",
"TensorCoreConvRotW4A4Layout",
"TensorWiseINT8Layout",
"QUANT_ALGOS",
"register_layout_op",

116
comfy/sd.py
View File

@ -17,6 +17,7 @@ import comfy.ldm.wan.vae
import comfy.ldm.trellis2.vae
import comfy.ldm.wan.vae2_2
import comfy.ldm.hunyuan3d.vae
import comfy.ldm.seedvr.vae
import comfy.ldm.triposplat.vae
import comfy.ldm.ace.vae.music_dcae_pipeline
import comfy.ldm.cogvideo.vae
@ -76,6 +77,7 @@ import comfy.text_encoders.gemma4
import comfy.text_encoders.cogvideo
import comfy.text_encoders.sa3
import comfy.text_encoders.gpt_oss
import comfy.text_encoders.joyimage
import comfy.model_patcher
import comfy.lora
@ -469,9 +471,13 @@ class CLIP:
def decode(self, token_ids, skip_special_tokens=True):
return self.tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
def is_dynamic(self):
return self.patcher.is_dynamic()
class VAE:
def __init__(self, sd=None, device=None, config=None, dtype=None, metadata=None):
if 'decoder.up_blocks.0.resnets.0.norm1.weight' in sd.keys(): #diffusers format
is_seedvr2_vae = "decoder.up_blocks.2.upsamplers.0.upscale_conv.weight" in sd
if not is_seedvr2_vae and 'decoder.up_blocks.0.resnets.0.norm1.weight' in sd.keys(): #diffusers format
sd = diffusers_convert.convert_vae_state_dict(sd)
if model_management.is_amd():
@ -498,6 +504,8 @@ class VAE:
self.upscale_index_formula = None
self.extra_1d_channel = None
self.crop_input = True
self.handles_tiling = False
self.format_encoded = None
self.audio_sample_rate = 44100
@ -554,6 +562,22 @@ class VAE:
self.memory_used_decode = lambda shape, dtype: (2500 * shape[2] * shape[3]) * model_management.dtype_size(dtype)
self.memory_used_encode = lambda shape, dtype: (2500 * shape[2] * shape[3]) * model_management.dtype_size(dtype)
self.first_stage_model = comfy.ldm.trellis2.vae.TextureVae()
elif "decoder.up_blocks.2.upsamplers.0.upscale_conv.weight" in sd: # seedvr2
self.first_stage_model = comfy.ldm.seedvr.vae.VideoAutoencoderKLWrapper()
self.latent_channels = comfy.ldm.seedvr.vae.SEEDVR2_LATENT_CHANNELS
self.latent_dim = 3
self.disable_offload = True
self.memory_used_decode = lambda shape, dtype: self.first_stage_model.comfy_memory_used_decode(shape)
self.memory_used_encode = lambda shape, dtype: (max(shape[2], 5) * shape[3] * shape[4] * 64) * model_management.dtype_size(dtype)
self.working_dtypes = [torch.float16, torch.bfloat16, torch.float32]
self.handles_tiling = True
self.format_encoded = self.first_stage_model.comfy_format_encoded
self.downscale_ratio = (lambda a: max(0, math.floor((a + 3) / 4)), 8, 8)
self.downscale_index_formula = (4, 8, 8)
self.upscale_ratio = (lambda a: max(0, a * 4 - 3), 8, 8)
self.upscale_index_formula = (4, 8, 8)
self.process_input = lambda image: image * 2.0 - 1.0
self.crop_input = False
elif "decoder.conv_in.weight" in sd:
if sd['decoder.conv_in.weight'].shape[1] == 64:
ddconfig = {"block_out_channels": [128, 256, 512, 512, 1024, 1024], "in_channels": 3, "out_channels": 3, "num_res_blocks": 2, "ffactor_spatial": 32, "downsample_match_channel": True, "upsample_match_channel": True}
@ -1020,6 +1044,10 @@ class VAE:
decode_fn = lambda a: self.first_stage_model.decode(a.to(self.vae_dtype).to(self.device)).to(dtype=self.vae_output_dtype())
return self.process_output(comfy.utils.tiled_scale_multidim(samples, decode_fn, tile=(tile_t, tile_x, tile_y), overlap=overlap, upscale_amount=self.upscale_ratio, out_channels=self.output_channels, index_formulas=self.upscale_index_formula, output_device=self.output_device))
def _decode_tiled_owned(self, samples, **kwargs):
out = self.first_stage_model.decode_tiled(samples.to(self.vae_dtype).to(self.device), **kwargs)
return self.process_output(out.to(device=self.output_device, dtype=self.vae_output_dtype(), copy=True))
def encode_tiled_(self, pixel_samples, tile_x=512, tile_y=512, overlap = 64):
steps = pixel_samples.shape[0] * comfy.utils.get_tiled_scale_steps(pixel_samples.shape[3], pixel_samples.shape[2], tile_x, tile_y, overlap)
steps += pixel_samples.shape[0] * comfy.utils.get_tiled_scale_steps(pixel_samples.shape[3], pixel_samples.shape[2], tile_x // 2, tile_y * 2, overlap)
@ -1056,6 +1084,25 @@ class VAE:
encode_fn = lambda a: self.first_stage_model.encode((self.process_input(a)).to(self.vae_dtype).to(self.device)).to(dtype=self.vae_output_dtype())
return comfy.utils.tiled_scale_multidim(samples, encode_fn, tile=(tile_t, tile_x, tile_y), overlap=overlap, upscale_amount=self.downscale_ratio, out_channels=self.latent_channels, downscale=True, index_formulas=self.downscale_index_formula, output_device=self.output_device)
def _encode_tiled_owned(self, pixel_samples, **kwargs):
x = self.process_input(pixel_samples).to(self.vae_dtype).to(self.device)
out = self.first_stage_model.encode_tiled(x, **kwargs)
return out.to(device=self.output_device, dtype=self.vae_output_dtype())
def _owned_tiled_args(self, tile_x=None, tile_y=None, overlap=None, tile_t=None, overlap_t=None):
args = {}
if tile_x is not None:
args["tile_x"] = tile_x
if tile_y is not None:
args["tile_y"] = tile_y
if overlap is not None:
args["overlap"] = overlap
if tile_t is not None:
args["tile_t"] = tile_t
if overlap_t is not None:
args["overlap_t"] = overlap_t
return args
def decode(self, samples_in, vae_options={}):
self.throw_exception_if_invalid()
pixel_samples = None
@ -1103,11 +1150,19 @@ class VAE:
if dims == 1 or self.extra_1d_channel is not None:
pixel_samples = self.decode_tiled_1d(samples_in)
elif dims == 2:
pixel_samples = self.decode_tiled_(samples_in)
if self.handles_tiling:
tile = 256 // self.spacial_compression_decode()
overlap = tile // 4
pixel_samples = self._decode_tiled_owned(samples_in, tile_x=tile, tile_y=tile, overlap=overlap)
else:
pixel_samples = self.decode_tiled_(samples_in)
elif dims == 3:
tile = 256 // self.spacial_compression_decode()
overlap = tile // 4
pixel_samples = self.decode_tiled_3d(samples_in, tile_x=tile, tile_y=tile, overlap=(1, overlap, overlap))
if self.handles_tiling:
pixel_samples = self._decode_tiled_owned(samples_in, tile_x=tile, tile_y=tile, overlap=overlap)
else:
pixel_samples = self.decode_tiled_3d(samples_in, tile_x=tile, tile_y=tile, overlap=(1, overlap, overlap))
pixel_samples = pixel_samples.to(self.output_device).movedim(1,-1)
return pixel_samples
@ -1135,7 +1190,9 @@ class VAE:
args["overlap"] = overlap
with model_management.cuda_device_context(self.device):
if dims == 1 or self.extra_1d_channel is not None:
if self.handles_tiling and dims in (2, 3):
output = self._decode_tiled_owned(samples, **self._owned_tiled_args(tile_x, tile_y, overlap, tile_t, overlap_t))
elif dims == 1 or self.extra_1d_channel is not None:
args.pop("tile_y")
output = self.decode_tiled_1d(samples, **args)
elif dims == 2:
@ -1196,12 +1253,17 @@ class VAE:
if self.latent_dim == 3:
tile = 256
overlap = tile // 4
samples = self.encode_tiled_3d(pixel_samples, tile_x=tile, tile_y=tile, overlap=(1, overlap, overlap))
if self.handles_tiling:
samples = self._encode_tiled_owned(pixel_samples, tile_x=tile, tile_y=tile, overlap=overlap)
else:
samples = self.encode_tiled_3d(pixel_samples, tile_x=tile, tile_y=tile, overlap=(1, overlap, overlap))
elif self.latent_dim == 1 or self.extra_1d_channel is not None:
samples = self.encode_tiled_1d(pixel_samples)
else:
samples = self.encode_tiled_(pixel_samples)
if self.format_encoded is not None:
samples = self.format_encoded(samples)
return samples
def encode_tiled(self, pixel_samples, tile_x=None, tile_y=None, overlap=None, tile_t=None, overlap_t=None):
@ -1209,7 +1271,7 @@ class VAE:
pixel_samples = self.vae_encode_crop_pixels(pixel_samples)
dims = self.latent_dim
pixel_samples = pixel_samples.movedim(-1, 1)
if dims == 3:
if dims == 3 and pixel_samples.ndim < 5:
if not self.not_video:
pixel_samples = pixel_samples.movedim(1, 0).unsqueeze(0)
else:
@ -1233,21 +1295,27 @@ class VAE:
elif dims == 2:
samples = self.encode_tiled_(pixel_samples, **args)
elif dims == 3:
if tile_t is not None:
tile_t_latent = max(2, self.downscale_ratio[0](tile_t))
if self.handles_tiling:
samples = self._encode_tiled_owned(pixel_samples, **self._owned_tiled_args(tile_x, tile_y, overlap, tile_t, overlap_t))
else:
tile_t_latent = 9999
args["tile_t"] = self.upscale_ratio[0](tile_t_latent)
if tile_t is not None:
tile_t_latent = max(2, self.downscale_ratio[0](tile_t))
else:
tile_t_latent = 9999
args["tile_t"] = self.upscale_ratio[0](tile_t_latent)
if overlap_t is None:
args["overlap"] = (1, overlap, overlap)
else:
args["overlap"] = (self.upscale_ratio[0](max(1, min(tile_t_latent // 2, self.downscale_ratio[0](overlap_t)))), overlap, overlap)
maximum = pixel_samples.shape[2]
maximum = self.upscale_ratio[0](self.downscale_ratio[0](maximum))
spatial_overlap = overlap if overlap is not None else 64
if overlap_t is None:
args["overlap"] = (1, spatial_overlap, spatial_overlap)
else:
args["overlap"] = (self.upscale_ratio[0](max(1, min(tile_t_latent // 2, self.downscale_ratio[0](overlap_t)))), spatial_overlap, spatial_overlap)
maximum = pixel_samples.shape[2]
maximum = self.upscale_ratio[0](self.downscale_ratio[0](maximum))
samples = self.encode_tiled_3d(pixel_samples[:,:,:maximum], **args)
samples = self.encode_tiled_3d(pixel_samples[:,:,:maximum], **args)
if self.format_encoded is not None:
samples = self.format_encoded(samples)
return samples
def get_sd(self):
@ -1271,6 +1339,11 @@ class VAE:
except:
return None
def is_dynamic(self):
# A VAE built from a state dict with no detectable VAE weights returns early
# from __init__ ("No VAE weights detected") before self.patcher is assigned.
patcher = getattr(self, "patcher", None)
return patcher is not None and patcher.is_dynamic()
class StyleModel:
def __init__(self, model, device="cpu"):
@ -1325,6 +1398,7 @@ class CLIPType(Enum):
IDEOGRAM4 = 30
BOOGU = 31
KREA2 = 32
JOYIMAGE = 33
@ -1654,6 +1728,10 @@ def load_text_encoder_state_dicts(state_dicts=[], embedding_directory=None, clip
clip_data[0] = comfy.utils.state_dict_prefix_replace(clip_data[0], {"model.language_model.": "model.", "model.visual.": "visual.", "lm_head.": "model.lm_head."})
clip_target.clip = comfy.text_encoders.krea2.te(**llama_detect(clip_data))
clip_target.tokenizer = comfy.text_encoders.krea2.Krea2Tokenizer
elif clip_type == CLIPType.JOYIMAGE and te_model == TEModel.QWEN3VL_8B: # JoyImageEdit: full Qwen3-VL-8B, edit-conditioning template + drop_idx.
clip_data[0] = comfy.utils.state_dict_prefix_replace(clip_data[0], {"model.language_model.": "model.", "model.visual.": "visual.", "lm_head.": "model.lm_head."})
clip_target.clip = comfy.text_encoders.joyimage.te(**llama_detect(clip_data))
clip_target.tokenizer = comfy.text_encoders.joyimage.JoyImageTokenizer
elif clip_type in (CLIPType.FLUX, CLIPType.FLUX2): # Flux2 Klein reuses the Qwen3-VL LM (3-layer tap -> 12288); visual unused.
klein_model_type = "qwen3_8b" if te_model == TEModel.QWEN3VL_8B else "qwen3_4b"
clip_target.clip = comfy.text_encoders.flux.klein_te(**llama_detect(clip_data), model_type=klein_model_type)
@ -1910,7 +1988,7 @@ def load_state_dict_guess_config(sd, output_vae=True, output_clip=True, output_c
manual_cast_dtype = model_management.unet_manual_cast(None, load_device, model_config.supported_inference_dtypes)
else:
manual_cast_dtype = model_management.unet_manual_cast(unet_dtype, load_device, model_config.supported_inference_dtypes)
model_config.set_inference_dtype(unet_dtype, manual_cast_dtype)
model_config.set_inference_dtype(unet_dtype, manual_cast_dtype, device=load_device)
if model_config.clip_vision_prefix is not None:
if output_clipvision:
@ -2051,7 +2129,7 @@ def load_diffusion_model_state_dict(sd, model_options={}, metadata=None, disable
manual_cast_dtype = model_management.unet_manual_cast(None, load_device, model_config.supported_inference_dtypes)
else:
manual_cast_dtype = model_management.unet_manual_cast(unet_dtype, load_device, model_config.supported_inference_dtypes)
model_config.set_inference_dtype(unet_dtype, manual_cast_dtype)
model_config.set_inference_dtype(unet_dtype, manual_cast_dtype, device=load_device)
if custom_operations is not None:
model_config.custom_operations = custom_operations

69
comfy/supported_models.py
View File

@ -27,6 +27,7 @@ import comfy.text_encoders.z_image
import comfy.text_encoders.ideogram4
import comfy.text_encoders.boogu
import comfy.text_encoders.krea2
import comfy.text_encoders.joyimage
import comfy.text_encoders.anima
import comfy.text_encoders.ace15
import comfy.text_encoders.longcat_image
@ -1710,6 +1711,40 @@ class Chroma(supported_models_base.BASE):
t5_detect = comfy.text_encoders.sd3_clip.t5_xxl_detect(state_dict, "{}t5xxl.transformer.".format(pref))
return supported_models_base.ClipTarget(comfy.text_encoders.pixart_t5.PixArtTokenizer, comfy.text_encoders.pixart_t5.pixart_te(**t5_detect))
class SeedVR2(supported_models_base.BASE):
unet_config = {
"image_model": "seedvr2"
}
unet_extra_config = {}
required_keys = {
"{}positive_conditioning",
"{}negative_conditioning",
}
latent_format = comfy.latent_formats.SeedVR2
vae_key_prefix = ["vae."]
text_encoder_key_prefix = ["text_encoders."]
supported_inference_dtypes = [torch.bfloat16, torch.float16, torch.float32]
sampling_settings = {
"shift": 1.0,
}
def set_inference_dtype(self, dtype, manual_cast_dtype, device=None):
if (
dtype == torch.float16
and manual_cast_dtype is None
and comfy.model_management.should_use_bf16(device)
):
manual_cast_dtype = torch.bfloat16
super().set_inference_dtype(dtype, manual_cast_dtype, device=device)
def get_model(self, state_dict, prefix="", device=None):
out = model_base.SeedVR2(self, device=device)
return out
def clip_target(self, state_dict={}):
return None
class ChromaRadiance(Chroma):
unet_config = {
"image_model": "chroma_radiance",
@ -1902,6 +1937,38 @@ class QwenImage(supported_models_base.BASE):
hunyuan_detect = comfy.text_encoders.hunyuan_video.llama_detect(state_dict, "{}qwen25_7b.transformer.".format(pref))
return supported_models_base.ClipTarget(comfy.text_encoders.qwen_image.QwenImageTokenizer, comfy.text_encoders.qwen_image.te(**hunyuan_detect))
class JoyImage(supported_models_base.BASE):
unet_config = {
"image_model": "joyimage",
}
sampling_settings = {
"multiplier": 1000,
"shift": 1.5,
}
memory_usage_factor = 1.8
unet_extra_config = {
"theta": 10000,
"rope_dim_list": [16, 56, 56],
}
latent_format = latent_formats.Wan21
supported_inference_dtypes = [torch.bfloat16, torch.float32]
vae_key_prefix = ["vae."]
text_encoder_key_prefix = ["text_encoders."]
def get_model(self, state_dict, prefix="", device=None):
return model_base.JoyImage(self, device=device)
def clip_target(self, state_dict={}):
pref = self.text_encoder_key_prefix[0]
qwen3vl_detect = comfy.text_encoders.hunyuan_video.llama_detect(state_dict, "{}qwen3vl.transformer.".format(pref))
return supported_models_base.ClipTarget(comfy.text_encoders.joyimage.JoyImageTokenizer, comfy.text_encoders.joyimage.te(**qwen3vl_detect))
class HunyuanImage21(HunyuanVideo):
unet_config = {
"image_model": "hunyuan_video",
@ -2373,12 +2440,14 @@ models = [
HiDream,
HiDreamO1,
Chroma,
SeedVR2,
ChromaRadiance,
ACEStep,
ACEStep15,
Omnigen2,
Boogu,
QwenImage,
JoyImage,
Ideogram4,
Krea2,
Flux2,

6
comfy/supported_models_base.py
View File

@ -54,13 +54,13 @@ class BASE:
optimizations = {"fp8": False}
@classmethod
def matches(s, unet_config, state_dict=None):
def matches(s, unet_config, state_dict=None, unet_key_prefix=""):
for k in s.unet_config:
if k not in unet_config or s.unet_config[k] != unet_config[k]:
return False
if state_dict is not None:
for k in s.required_keys:
if k not in state_dict:
if k.format(unet_key_prefix) not in state_dict:
return False
return True
@ -115,7 +115,7 @@ class BASE:
replace_prefix = {"": self.vae_key_prefix[0]}
return utils.state_dict_prefix_replace(state_dict, replace_prefix)
def set_inference_dtype(self, dtype, manual_cast_dtype):
def set_inference_dtype(self, dtype, manual_cast_dtype, device=None):
self.unet_config['dtype'] = dtype
self.manual_cast_dtype = manual_cast_dtype

2
comfy/text_encoders/gemma4.py
View File

@ -1088,7 +1088,7 @@ class Gemma4_Tokenizer():
h, w = samples.shape[2], samples.shape[3]
patch_size = 16
pooling_k = 3
max_soft_tokens = 70 if is_video else 280 # video uses smaller token budget per frame
max_soft_tokens = kwargs.get("max_soft_tokens", 70 if is_video else 280)
max_patches = max_soft_tokens * pooling_k * pooling_k
target_px = max_patches * patch_size * patch_size
factor = (target_px / (h * w)) ** 0.5

6
comfy/text_encoders/gpt_oss.py
View File

@ -12,7 +12,7 @@ import torch.nn.functional as F
import comfy.ops
from comfy import sd1_clip
from comfy.ldm.modules.attention import TORCH_HAS_GQA, optimized_attention_for_device
from comfy.ldm.modules.attention import optimized_attention_for_device
from comfy.text_encoders.llama import RMSNorm, apply_rope
@ -110,10 +110,6 @@ def _attention_with_sinks(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, sin
putting the sink logit in the mask at that column.
"""
if num_kv_groups > 1 and not TORCH_HAS_GQA:
k = k.repeat_interleave(num_kv_groups, dim=1)
v = v.repeat_interleave(num_kv_groups, dim=1)
B, _, S_q, D = q.shape
H_kv = k.shape[1]
S_kv = k.shape[-2]

97
comfy/text_encoders/joyimage.py Normal file
View File

@ -0,0 +1,97 @@
import torch
from comfy import sd1_clip
import comfy.text_encoders.qwen_vl
from comfy.text_encoders.qwen3vl import Qwen3VL, Qwen3VLTokenizer
JOYIMAGE_VISION_BLOCK = "<|vision_start|><|image_pad|><|vision_end|>"
JOYIMAGE_TEMPLATE_TEXT = (
"<|im_start|>system\n \\nDescribe the image by detailing the color, shape, size, texture, "
"quantity, text, spatial relationships of the objects and background:<|im_end|>\n"
"<|im_start|>user\n{}<|im_end|>\n<|im_start|>assistant\n"
)
JOYIMAGE_TEMPLATE_IMAGE = (
"<|im_start|>system\n \\nDescribe the image by detailing the color, shape, size, texture, "
"quantity, text, spatial relationships of the objects and background:<|im_end|>\n"
f"<|im_start|>user\n{JOYIMAGE_VISION_BLOCK}{{}}<|im_end|>\n<|im_start|>assistant\n"
)
# The DiT was trained without the leading system-prompt tokens.
JOYIMAGE_DROP_IDX = 34
PAD_TOKEN = 151643
class Qwen3VL8B_JoyImage(Qwen3VL):
model_type = "qwen3vl_8b"
def preprocess_embed(self, embed, device):
if embed["type"] == "image":
image, grid = comfy.text_encoders.qwen_vl.process_qwen2vl_images(
embed["data"], min_pixels=65536, max_pixels=16777216, patch_size=16,
image_mean=[0.5, 0.5, 0.5], image_std=[0.5, 0.5, 0.5],
interpolation="bicubic",
)
merged, deepstack = self.visual(image.to(device, dtype=torch.float32), grid)
return merged, {"grid": grid, "deepstack": deepstack}
return None, None
class JoyImageTokenizer(Qwen3VLTokenizer):
def __init__(self, embedding_directory=None, tokenizer_data={}):
super().__init__(
embedding_directory=embedding_directory, tokenizer_data=tokenizer_data,
model_type="qwen3vl_8b",
)
self.llama_template = JOYIMAGE_TEMPLATE_TEXT
self.llama_template_images = JOYIMAGE_TEMPLATE_IMAGE
def tokenize_with_weights(self, text, return_word_ids=False, llama_template=None, images=None, **kwargs):
kwargs.pop("thinking", None)
return super().tokenize_with_weights(
text, return_word_ids=return_word_ids, llama_template=llama_template,
images=images or [], thinking=True, **kwargs,
)
class _JoyImageClipModel(sd1_clip.SDClipModel):
def __init__(self, device="cpu", layer="hidden", layer_idx=-1, dtype=None,
attention_mask=True, model_options={}):
super().__init__(
device=device, layer=layer, layer_idx=layer_idx, textmodel_json_config={},
# JoyImage conditions on the pre-final-norm output of the last decoder layer.
dtype=dtype, special_tokens={"pad": PAD_TOKEN}, layer_norm_hidden_state=False,
model_class=Qwen3VL8B_JoyImage, enable_attention_masks=attention_mask,
return_attention_masks=attention_mask, model_options=model_options,
)
class JoyImageTEModel(sd1_clip.SD1ClipModel):
def __init__(self, device="cpu", dtype=None, model_options={}):
super().__init__(
device=device, dtype=dtype, name="qwen3vl_8b",
clip_model=_JoyImageClipModel, model_options=model_options,
)
def encode_token_weights(self, token_weight_pairs):
out, pooled, extra = super().encode_token_weights(token_weight_pairs)
if out.shape[1] <= JOYIMAGE_DROP_IDX:
raise ValueError(
f"JoyImageTEModel: encoded sequence length {out.shape[1]} is shorter "
f"than drop_idx={JOYIMAGE_DROP_IDX}; the prompt did not include the "
f"template prefix."
)
out = out[:, JOYIMAGE_DROP_IDX:]
if "attention_mask" in extra:
extra["attention_mask"] = extra["attention_mask"][:, JOYIMAGE_DROP_IDX:]
return out, pooled, extra
def te(dtype_llama=None, llama_quantization_metadata=None):
class JoyImageTEModel_(JoyImageTEModel):
def __init__(self, device="cpu", dtype=None, model_options={}):
if llama_quantization_metadata is not None:
model_options = model_options.copy()
model_options["quantization_metadata"] = llama_quantization_metadata
if dtype_llama is not None:
dtype = dtype_llama
super().__init__(device=device, dtype=dtype, model_options=model_options)
return JoyImageTEModel_

47
comfy/text_encoders/llama.py
View File

@ -550,10 +550,8 @@ class Attention(nn.Module):
xv = xv[:, :, -sliding_window:]
attention_mask = attention_mask[..., -sliding_window:] if attention_mask is not None else None
xk = xk.repeat_interleave(self.num_heads // self.num_kv_heads, dim=1)
xv = xv.repeat_interleave(self.num_heads // self.num_kv_heads, dim=1)
output = optimized_attention(xq, xk, xv, self.num_heads, mask=attention_mask, skip_reshape=True)
gqa_kwargs = {"enable_gqa": True} if self.num_heads != self.num_kv_heads else {}
output = optimized_attention(xq, xk, xv, self.num_heads, mask=attention_mask, skip_reshape=True, **gqa_kwargs)
return self.o_proj(output), present_key_value
class MLP(nn.Module):
@ -937,22 +935,41 @@ class BaseGenerate:
return torch.argmax(logits, dim=-1, keepdim=True)
# Sampling mode
if repetition_penalty != 1.0:
for i in range(logits.shape[0]):
for token_id in set(token_history):
logits[i, token_id] *= repetition_penalty if logits[i, token_id] < 0 else 1/repetition_penalty
if presence_penalty is not None and presence_penalty != 0.0:
for i in range(logits.shape[0]):
for token_id in set(token_history):
logits[i, token_id] -= presence_penalty
if len(token_history) > 0 and (repetition_penalty != 1.0 or (presence_penalty is not None and presence_penalty != 0.0)):
token_ids = torch.tensor(list(set(token_history)), device=logits.device)
token_logits = logits[:, token_ids]
if repetition_penalty != 1.0:
token_logits = torch.where(token_logits < 0, token_logits * repetition_penalty, token_logits / repetition_penalty)
if presence_penalty is not None and presence_penalty != 0.0:
token_logits = token_logits - presence_penalty
logits[:, token_ids] = token_logits
if temperature != 1.0:
logits = logits / temperature
if top_k > 0:
indices_to_remove = logits < torch.topk(logits, top_k)[0][..., -1, None]
logits[indices_to_remove] = torch.finfo(logits.dtype).min
top_k = min(top_k, logits.shape[-1])
logits, top_indices = torch.topk(logits, top_k)
if min_p > 0.0:
probs_before_filter = torch.nn.functional.softmax(logits, dim=-1)
top_probs, _ = probs_before_filter.max(dim=-1, keepdim=True)
min_threshold = min_p * top_probs
indices_to_remove = probs_before_filter < min_threshold
logits[indices_to_remove] = torch.finfo(logits.dtype).min
if top_p < 1.0:
sorted_logits, sorted_indices = torch.sort(logits, descending=True)
cumulative_probs = torch.cumsum(torch.nn.functional.softmax(sorted_logits, dim=-1), dim=-1)
sorted_indices_to_remove = cumulative_probs > top_p
sorted_indices_to_remove[..., 0] = False
indices_to_remove = torch.zeros_like(logits, dtype=torch.bool)
indices_to_remove.scatter_(1, sorted_indices, sorted_indices_to_remove)
logits[indices_to_remove] = torch.finfo(logits.dtype).min
probs = torch.nn.functional.softmax(logits, dim=-1)
next_token = torch.multinomial(probs, num_samples=1, generator=generator)
return top_indices.gather(1, next_token)
if min_p > 0.0:
probs_before_filter = torch.nn.functional.softmax(logits, dim=-1)

8
comfy/text_encoders/qwen35.py
View File

@ -366,12 +366,8 @@ class GatedAttention(nn.Module):
xv = torch.cat((past_value[:, :, :index], xv), dim=2)
present_key_value = (xk, xv, index + num_tokens)
# Expand KV heads for GQA
if self.num_heads != self.num_kv_heads:
xk = xk.repeat_interleave(self.num_heads // self.num_kv_heads, dim=1)
xv = xv.repeat_interleave(self.num_heads // self.num_kv_heads, dim=1)
output = optimized_attention(xq, xk, xv, self.num_heads, mask=attention_mask, skip_reshape=True)
gqa_kwargs = {"enable_gqa": True} if self.num_heads != self.num_kv_heads else {}
output = optimized_attention(xq, xk, xv, self.num_heads, mask=attention_mask, skip_reshape=True, **gqa_kwargs)
output = output * gate.sigmoid()
return self.o_proj(output), present_key_value

21
comfy/text_encoders/qwen3vl.py
View File

@ -90,6 +90,27 @@ class Qwen3VL(BaseLlama, BaseQwen3, BaseGenerate, torch.nn.Module):
deepstack = [torch.cat([deepstack[i], ds[i]], dim=0) for i in range(len(ds))]
return position_ids, visual_pos_masks, deepstack
def forward(self, input_ids, attention_mask=None, embeds=None, num_tokens=None, intermediate_output=None, final_layer_norm_intermediate=True, dtype=None, embeds_info=[], **kwargs):
position_ids = kwargs.pop("position_ids", None)
visual_pos_masks = kwargs.pop("visual_pos_masks", None)
deepstack_embeds = kwargs.pop("deepstack_embeds", None)
if embeds is not None and position_ids is None:
position_ids, visual_pos_masks, deepstack_embeds = self.build_image_inputs(embeds, embeds_info)
return self.model(
input_ids,
attention_mask=attention_mask,
embeds=embeds,
num_tokens=num_tokens,
intermediate_output=intermediate_output,
final_layer_norm_intermediate=final_layer_norm_intermediate,
dtype=dtype,
position_ids=position_ids,
embeds_info=embeds_info,
visual_pos_masks=visual_pos_masks,
deepstack_embeds=deepstack_embeds,
**kwargs,
)
def _make_qwen3vl_model(model_type):
class Qwen3VL_(Qwen3VL):

4
comfy/text_encoders/qwen_vl.py
View File

@ -15,6 +15,7 @@ def process_qwen2vl_images(
merge_size: int = 2,
image_mean: list = None,
image_std: list = None,
interpolation: str = "bilinear",
):
if image_mean is None:
image_mean = [0.48145466, 0.4578275, 0.40821073]
@ -47,10 +48,9 @@ def process_qwen2vl_images(
img_resized = F.interpolate(
img.unsqueeze(0),
size=(h_bar, w_bar),
mode='bilinear',
mode=interpolation,
align_corners=False
).squeeze(0)
normalized = img_resized.clone()
for c in range(3):
normalized[c] = (img_resized[c] - image_mean[c]) / image_std[c]

1
comfy_api/feature_flags.py
View File

@ -100,6 +100,7 @@ def _parse_cli_feature_flags() -> dict[str, Any]:
# Default server capabilities
_CORE_FEATURE_FLAGS: dict[str, Any] = {
"supports_preview_metadata": True,
"supports_model_type_tags": True,
"max_upload_size": args.max_upload_size * 1024 * 1024, # Convert MB to bytes
"extension": {"manager": {"supports_v4": True}},
"node_replacements": True,

394
comfy_api/latest/_input_impl/video_types.py
View File

@ -1,5 +1,6 @@
from av.container import InputContainer
from av.subtitles.stream import SubtitleStream
from av.video.reformatter import ColorRange
from fractions import Fraction
from typing import Optional
from .._input import AudioInput, VideoInput
@ -9,6 +10,7 @@ import itertools
import json
import numpy as np
import math
import os
import torch
from .._util import VideoContainer, VideoCodec, VideoComponents
import logging
@ -58,6 +60,57 @@ def video_stream_bit_depth(stream) -> int:
return max(component.bits for component in stream.format.components)
def last_decodable_audio_stream(container: InputContainer):
"""Streams FFmpeg has no decoder for have no codec context, and decoding their
packets crashes the process (e.g. APAC spatial-audio track in iPhone)."""
stream = next(
(s for s in reversed(container.streams.audio) if s.codec_context is not None),
None,
)
if stream is None and len(container.streams.audio):
logging.warning("No decodable audio stream found in video; ignoring audio.")
return stream
def probe_audio_params(container: InputContainer, audio_stream, max_packets: int = 200):
"""Containers probed only up to a window (mpegts) leave audio codec parameters unset when
audio starts beyond it; learn them by decoding ahead. The caller must seek back afterwards.
Returns (sample_rate, channels), zeros when the stream never yields a decodable frame."""
for i, packet in enumerate(container.demux(audio_stream)):
try:
frames = packet.decode()
except av.error.FFmpegError:
frames = ()
if frames:
return frames[0].sample_rate, frames[0].layout.nb_channels
if i >= max_packets:
break
return 0, 0
def write_output_metadata(container: InputContainer, output, metadata: dict | None):
"""Copy the source container's metadata, then overlay the caller's tags."""
for key, value in container.metadata.items():
if metadata is None or key not in metadata:
output.metadata[key] = value
if metadata is not None:
for key, value in metadata.items():
output.metadata[key] = value if isinstance(value, str) else json.dumps(value)
def mp4_output_open_kwargs(path: str | io.BytesIO, format: VideoContainer, codec: VideoCodec) -> dict:
if format != VideoContainer.AUTO and format != VideoContainer.MP4:
raise ValueError("Only MP4 format is supported for now")
if codec != VideoCodec.AUTO and codec != VideoCodec.H264:
raise ValueError("Only H264 codec is supported for now")
open_kwargs = {"mode": "w", "options": {"movflags": "use_metadata_tags"}}
if isinstance(format, VideoContainer) and format != VideoContainer.AUTO:
open_kwargs["format"] = format.value
elif isinstance(path, io.BytesIO):
open_kwargs["format"] = "mp4" # no file extension to infer the format from
return open_kwargs
class VideoFromFile(VideoInput):
"""
Class representing video input from a file.
@ -192,13 +245,10 @@ class VideoFromFile(VideoInput):
return estimated_frames
# 3. Last resort: decode frames and count them (streaming)
if self.__start_time < 0:
start_time = max(self._get_raw_duration() + self.__start_time, 0)
else:
start_time = self.__start_time
start_time, duration = self.get_active_trim_window()
frame_count = 1
start_pts = int(start_time / video_stream.time_base)
end_pts = int((start_time + self.__duration) / video_stream.time_base)
end_pts = int((start_time + duration) / video_stream.time_base)
container.seek(start_pts, stream=video_stream)
frame_iterator = (
container.decode(video_stream)
@ -253,17 +303,14 @@ class VideoFromFile(VideoInput):
def get_components_internal(self, container: InputContainer) -> VideoComponents:
video_stream = self._get_first_video_stream(container)
if self.__start_time < 0:
start_time = max(self._get_raw_duration() + self.__start_time, 0)
else:
start_time = self.__start_time
start_time, duration = self.get_active_trim_window()
# Get video frames
frames = []
audio_frames = []
alphas = None
start_pts = int(start_time / video_stream.time_base)
end_pts = int((start_time + self.__duration) / video_stream.time_base)
end_pts = int((start_time + duration) / video_stream.time_base)
if start_pts != 0:
container.seek(start_pts, stream=video_stream)
@ -281,8 +328,8 @@ class VideoFromFile(VideoInput):
video_done = False
audio_done = True
if len(container.streams.audio):
audio_stream = container.streams.audio[-1]
audio_stream = last_decodable_audio_stream(container)
if audio_stream is not None:
streams += [audio_stream]
resampler = av.audio.resampler.AudioResampler(format='fltp')
audio_done = False
@ -298,7 +345,7 @@ class VideoFromFile(VideoInput):
for frame in packet.decode():
if frame.pts < start_pts:
continue
if self.__duration and frame.pts >= end_pts:
if duration and frame.pts >= end_pts:
video_done = True
break
@ -365,7 +412,7 @@ class VideoFromFile(VideoInput):
map(resampler.resample, packet.decode())
)
for frame in aframes:
if self.__duration and frame.time > start_time + self.__duration:
if duration and frame.time > start_time + duration:
audio_done = True
break
@ -387,8 +434,8 @@ class VideoFromFile(VideoInput):
if len(audio_frames) > 0:
audio_data = np.concatenate(audio_frames, axis=1) # shape: (channels, total_samples)
if self.__duration:
audio_data = audio_data[..., :int(self.__duration * audio_stream.sample_rate)]
if duration:
audio_data = audio_data[..., :int(duration * audio_stream.sample_rate)]
audio_tensor = torch.from_numpy(audio_data).unsqueeze(0) # shape: (1, channels, total_samples)
audio = AudioInput({
@ -434,33 +481,22 @@ class VideoFromFile(VideoInput):
if not reuse_streams:
if bit_depth is None:
bit_depth = source_bit_depth
components = self.get_components_internal(container)
video = VideoFromComponents(components)
return video.save_to(
path, format=format, codec=codec, metadata=metadata, bit_depth=bit_depth,
)
return self._save_transcoded(container, path, format=format, codec=codec, metadata=metadata, bit_depth=bit_depth)
streams = container.streams
open_kwargs = get_open_write_kwargs(path, container_format, format)
with av.open(path, **open_kwargs) as output_container:
# Copy over the original metadata
for key, value in container.metadata.items():
if metadata is None or key not in metadata:
output_container.metadata[key] = value
# Add metadata before writing any streams
write_output_metadata(container, output_container, metadata)
# Add our new metadata
if metadata is not None:
for key, value in metadata.items():
if isinstance(value, str):
output_container.metadata[key] = value
else:
output_container.metadata[key] = json.dumps(value)
# Add streams to the new container
# Add streams to the new container. Streams with no codec context cannot be used as an output template.
stream_map = {}
for stream in streams:
if isinstance(stream, (av.VideoStream, av.AudioStream, SubtitleStream)):
if stream.codec_context is None:
logging.warning("Skipping %s stream %d with unsupported codec", stream.type, stream.index)
continue
out_stream = output_container.add_stream_from_template(template=stream, opaque=True)
stream_map[stream] = out_stream
@ -470,6 +506,282 @@ class VideoFromFile(VideoInput):
packet.stream = stream_map[packet.stream]
output_container.mux(packet)
def _save_transcoded(
self,
container: InputContainer,
path: str | io.BytesIO,
format: VideoContainer,
codec: VideoCodec,
metadata: dict | None,
bit_depth: int,
):
"""Re-encode to H.264/AAC one frame at a time; peak memory does not scale with video length."""
open_kwargs = mp4_output_open_kwargs(path, format, codec)
video_stream = self._get_first_video_stream(container)
start_time, duration = self.get_active_trim_window()
start_pts = int(start_time / video_stream.time_base)
end_pts = int((start_time + duration) / video_stream.time_base) if duration else None
stream_end_pts = None
if video_stream.duration is not None:
stream_end_pts = (video_stream.start_time or 0) + video_stream.duration
output_end_pts = end_pts
if stream_end_pts is not None and (output_end_pts is None or stream_end_pts < output_end_pts):
output_end_pts = stream_end_pts
if start_pts != 0:
container.seek(start_pts, stream=video_stream)
audio_stream = last_decodable_audio_stream(container)
pix_fmt = "yuv420p10le" if bit_depth >= 10 else "yuv420p"
rate = Fraction(video_stream.average_rate) if video_stream.average_rate else Fraction(1)
resampler = None
sample_rate = 0
audio_time_base = None
duration_cap = None
if audio_stream is not None:
sample_rate = audio_stream.codec_context.sample_rate
channels = audio_stream.codec_context.channels
if not sample_rate:
sample_rate, channels = probe_audio_params(container, audio_stream)
container.seek(start_pts, stream=video_stream)
if sample_rate:
audio_stream.codec_context.flush_buffers()
else:
logging.warning("Audio stream parameters could not be determined; ignoring audio.")
audio_stream = None
if audio_stream is not None:
audio_time_base = Fraction(1, sample_rate)
layout = {1: "mono", 2: "stereo", 6: "5.1"}.get(channels, "stereo")
resampler = av.audio.resampler.AudioResampler(format="fltp", layout=layout, rate=sample_rate)
if duration:
duration_cap = math.ceil(duration * sample_rate)
streams = [video_stream] if audio_stream is None else [video_stream, audio_stream]
pts_step = max(1, int(round((1 / rate) / video_stream.time_base)))
video_done = False
audio_done = audio_stream is None
video_pts_offset = None
last_video_pts = None
last_video_end = None
# rebased pts -> true display duration: the mp4 muxer pads the last sample with 1/rate otherwise
video_frame_durations = {}
source_size = None
rotation_k = 0
rotation_filter = None
audio_started = False
samples_written = 0
pending_audio = []
# The output opens lazily on the first kept frame: it decides the geometry (90/270 rotation swaps dims),
# and never seeking back keeps webm/mkv leading audio intact.
output = None
out_video = None
out_audio = None
def audio_frame_from_ndarray(nd_planar):
frame = av.AudioFrame.from_ndarray(np.ascontiguousarray(nd_planar), format="fltp", layout=layout)
frame.sample_rate = sample_rate
return frame
def drain_audio(final=False):
# Audio may cover the pts span of the video written so far, capped by the requested duration
nonlocal samples_written, audio_done
if last_video_end is None:
cap = 0
else:
cap = math.ceil(last_video_end * video_stream.time_base * sample_rate)
if duration_cap is not None:
cap = min(cap, duration_cap)
while pending_audio and not audio_done:
frame = pending_audio[0]
if samples_written + frame.samples <= cap:
frame.pts = samples_written
frame.time_base = audio_time_base
output.mux(out_audio.encode(frame))
samples_written += frame.samples
pending_audio.pop(0)
continue
if final:
keep = frame.to_ndarray()[..., :cap - samples_written]
if keep.shape[-1] > 0:
tail = audio_frame_from_ndarray(keep)
tail.pts = samples_written
tail.time_base = audio_time_base
output.mux(out_audio.encode(tail))
samples_written += keep.shape[-1]
pending_audio.clear()
break
if duration_cap is not None and samples_written >= duration_cap:
audio_done = True
return cap
try:
for packet in container.demux(*streams):
if video_done and audio_done:
break
if packet.stream == video_stream and not video_done:
try:
frames = packet.decode()
except av.error.InvalidDataError:
logging.info("pyav decode error")
continue
for frame in frames:
if frame.pts is not None and frame.pts < start_pts:
continue
if end_pts is not None and frame.pts is not None and frame.pts >= end_pts:
video_done = True
if last_video_pts is not None:
# the source continues past the window: hold the last kept frame to the window end
end_offset = video_pts_offset if video_pts_offset is not None else start_pts
last_video_end = max(last_video_end, end_pts - end_offset)
break
# the source's true display duration of this frame; average_rate is not a
# frame duration (sparse/VFR sources), so it is only the fallback
frame_duration = frame.duration if frame.duration else pts_step
if end_pts is not None and frame.pts is not None:
frame_duration = min(frame_duration, end_pts - frame.pts)
if output is None:
rotation_k = int(round(frame.rotation // 90)) % 4 if frame.rotation else 0
if rotation_k % 2:
out_width, out_height = frame.height, frame.width
else:
out_width, out_height = frame.width, frame.height
if out_width % 2 or out_height % 2:
raise ValueError(f"H.264 output requires even dimensions, got {out_width}x{out_height}")
source_size = (frame.width, frame.height)
output = av.open(path, **open_kwargs)
# Add metadata before writing any streams
write_output_metadata(container, output, metadata)
out_video = output.add_stream("h264", rate=rate)
# no B-frames: reordering makes mp4 sample durations follow decode order,
# so irregular-VFR spans and trim windows land wrong
out_video.codec_context.max_b_frames = 0
out_video.width = out_width
out_video.height = out_height
out_video.pix_fmt = pix_fmt
# source pts pass through (rebased to 0), so variable frame rate survives
out_video.codec_context.time_base = video_stream.time_base
if audio_stream is not None:
out_audio = output.add_stream("aac", rate=sample_rate, layout=layout)
if (frame.width, frame.height) != source_size:
# encoding would silently rescale the new geometry into the old one
raise ValueError(
f"Video resolution changes mid-stream "
f"({source_size[0]}x{source_size[1]} -> {frame.width}x{frame.height}); cannot transcode"
)
if rotation_k:
if rotation_filter is None:
g = av.filter.Graph()
g_src = g.add_buffer(width=frame.width, height=frame.height,
format=frame.format.name, time_base=video_stream.time_base)
tail = g_src
for filter_name, filter_args in {1: [("transpose", "cclock")],
2: [("hflip", None), ("vflip", None)],
3: [("transpose", "clock")]}[rotation_k]:
step = g.add(filter_name, filter_args)
tail.link_to(step)
tail = step
g_sink = g.add("buffersink")
tail.link_to(g_sink)
g.configure()
rotation_filter = (g_src, g_sink)
rotation_filter[0].push(frame)
frame = rotation_filter[1].pull()
if frame.color_range == ColorRange.JPEG:
# compress full-range sources (yuvj/MJPEG) to limited range
frame = frame.reformat(format=pix_fmt, src_color_range="JPEG", dst_color_range="MPEG")
else:
frame = frame.reformat(format=pix_fmt)
frame_output_end = None
if frame.pts is not None:
if video_pts_offset is None:
video_pts_offset = frame.pts
frame.pts -= video_pts_offset
if output_end_pts is not None:
frame_output_end = output_end_pts - video_pts_offset
if frame.pts + frame_duration > frame_output_end:
clamped_pts = frame_output_end - frame_duration
if clamped_pts >= 0 and (last_video_pts is None or clamped_pts > last_video_pts):
frame.pts = min(frame.pts, clamped_pts)
elif frame.pts < frame_output_end:
frame_duration = frame_output_end - frame.pts
else:
continue
if frame.pts is None or (last_video_pts is not None and frame.pts <= last_video_pts):
# broken sources emit missing/backward timestamps mid-stream, which the
# muxer rejects; nudge them forward by one nominal frame interval
frame.pts = 0 if last_video_pts is None else last_video_pts + pts_step
if frame_output_end is not None and frame.pts + frame_duration > frame_output_end:
if frame.pts >= frame_output_end:
continue
frame_duration = frame_output_end - frame.pts
last_video_pts = frame.pts
last_video_end = frame.pts + frame_duration
video_frame_durations[frame.pts] = frame_duration
# the decoded pict_type would force x264's frame types (intra-only
# sources like MJPEG/ProRes would come out all-keyframe)
frame.pict_type = 0
for out_packet in out_video.encode(frame):
out_packet.duration = video_frame_durations.pop(out_packet.pts, 0)
output.mux(out_packet)
drain_audio()
elif packet.stream == audio_stream and not audio_done:
for resampled in itertools.chain.from_iterable(map(resampler.resample, packet.decode())):
frame_start = None
if resampled.pts is not None:
# passthrough frames keep the source stream's time base
tb = resampled.time_base if resampled.time_base else audio_time_base
frame_start = float(resampled.pts * tb)
if duration and not audio_started and frame_start >= start_time + duration:
audio_done = True
break
if not audio_started:
if frame_start is None:
frame_start = 0.0
to_skip = max(0, int((start_time - frame_start) * sample_rate))
if to_skip >= resampled.samples:
continue
audio_started = True
if duration and frame_start > start_time:
duration_cap = min(duration_cap, math.ceil((start_time + duration - frame_start) * sample_rate))
if to_skip:
pending_audio.append(audio_frame_from_ndarray(resampled.to_ndarray()[..., to_skip:]))
continue
pending_audio.append(resampled)
if video_done:
# the video window is complete so the cap is final, but containers
# that interleave audio behind video (fragmented mp4) still owe most
# of it: stop only once the demuxed audio covers the cap
cap = drain_audio()
if pending_audio or samples_written >= cap:
drain_audio(final=True)
audio_done = True
break
if output is None:
raise ValueError(f"No decodable video frames found in file '{self.__file}'")
if out_audio is not None and not audio_done:
drain_audio(final=True)
window_fill = last_video_end - last_video_pts if video_done and last_video_pts is not None else 0
for out_packet in out_video.encode(None):
duration = video_frame_durations.pop(out_packet.pts, 0)
if out_packet.pts == last_video_pts:
duration = max(duration, window_fill)
out_packet.duration = duration
output.mux(out_packet)
if out_audio is not None:
output.mux(out_audio.encode(None))
except BaseException:
if output is not None:
output.close()
if isinstance(path, (str, os.PathLike)) and os.path.exists(path):
os.remove(path)
raise
else:
if output is not None:
output.close()
def _get_first_video_stream(self, container: InputContainer):
if len(container.streams.video):
return container.streams.video[0]
@ -517,22 +829,12 @@ class VideoFromComponents(VideoInput):
bit_depth: int | None = None,
):
"""Save the video to a file path or BytesIO buffer."""
if format != VideoContainer.AUTO and format != VideoContainer.MP4:
raise ValueError("Only MP4 format is supported for now")
if codec != VideoCodec.AUTO and codec != VideoCodec.H264:
raise ValueError("Only H264 codec is supported for now")
open_kwargs = mp4_output_open_kwargs(path, format, codec)
# None means "use the depth this video was created with" (CreateVideo's choice).
if bit_depth is None:
bit_depth = self.__bit_depth
is_10bit = bit_depth >= 10
extra_kwargs = {}
if isinstance(format, VideoContainer) and format != VideoContainer.AUTO:
extra_kwargs["format"] = format.value
elif isinstance(path, io.BytesIO):
# BytesIO has no file extension, so av.open can't infer the format.
# Default to mp4 since that's the only supported format anyway.
extra_kwargs["format"] = "mp4"
with av.open(path, mode='w', options={'movflags': 'use_metadata_tags'}, **extra_kwargs) as output:
with av.open(path, **open_kwargs) as output:
# Add metadata before writing any streams
if metadata is not None:
for key, value in metadata.items():

22
comfy_api_nodes/apis/bytedance.py
View File

@ -17,6 +17,10 @@ class Seedream4Options(BaseModel):
max_images: int = Field(15)
class Seedream5OptimizePromptOptions(BaseModel):
thinking: Literal["auto", "enabled", "disabled"] = Field(...)
class Seedream4TaskCreationRequest(BaseModel):
model: str = Field(...)
prompt: str = Field(...)
@ -24,10 +28,11 @@ class Seedream4TaskCreationRequest(BaseModel):
image: list[str] | None = Field(None, description="Image URLs")
size: str = Field(...)
seed: int = Field(..., ge=0, le=2147483647)
sequential_image_generation: str = Field("disabled")
sequential_image_generation_options: Seedream4Options = Field(Seedream4Options(max_images=15))
sequential_image_generation: str | None = Field("disabled")
sequential_image_generation_options: Seedream4Options | None = Field(Seedream4Options(max_images=15))
watermark: bool = Field(False)
output_format: str | None = None
optimize_prompt_options: Seedream5OptimizePromptOptions | None = None
class ImageTaskCreationResponse(BaseModel):
@ -261,6 +266,19 @@ _PRESETS_SEEDREAM_4K = [
_CUSTOM_PRESET = [("Custom", None, None)]
_PRESETS_SEEDREAM_2K_PRO = [
("(2K) 2048x2048 (1:1)", 2048, 2048),
("(2K) 1728x2304 (3:4)", 1728, 2304),
("(2K) 2304x1728 (4:3)", 2304, 1728),
# ("(2K) 2848x1600 (16:9)", 2848, 1600), # 4,556,800 px - temporarily unavailable
# ("(2K) 1600x2848 (9:16)", 1600, 2848), # 4,556,800 px - temporarily unavailable
("(2K) 1664x2496 (2:3)", 1664, 2496),
("(2K) 2496x1664 (3:2)", 2496, 1664),
# ("(2K) 3136x1344 (21:9)", 3136, 1344), # 4,214,784 px - temporarily unavailable
]
RECOMMENDED_PRESETS_SEEDREAM_5_PRO = (
_PRESETS_SEEDREAM_1K + _PRESETS_SEEDREAM_2K_PRO + _CUSTOM_PRESET
)
RECOMMENDED_PRESETS_SEEDREAM_5_LITE = (
_PRESETS_SEEDREAM_2K + _PRESETS_SEEDREAM_3K + _PRESETS_SEEDREAM_4K + _CUSTOM_PRESET
)

1
comfy_api_nodes/apis/hunyuan3d.py
View File

@ -77,6 +77,7 @@ class To3DUVTaskRequest(BaseModel):
class To3DPartTaskRequest(BaseModel):
File: TaskFile3DInput = Field(...)
EnableStagedGeneration: bool | None = Field(None)
class TextureEditImageInfo(BaseModel):

2
comfy_api_nodes/apis/openai.py
View File

@ -128,7 +128,7 @@ class OpenAIResponse(ModelResponseProperties, ResponseProperties):
parallel_tool_calls: bool | None = Field(True)
status: str | None = Field(
None,
description="One of `completed`, `failed`, `in_progress`, or `incomplete`.",
description="One of `completed`, `failed`, `in_progress`, `incomplete`, `queued`, or `cancelled`.",
)
usage: ResponseUsage | None = Field(None)

49
comfy_api_nodes/apis/sync_so.py Normal file
View File

@ -0,0 +1,49 @@
from pydantic import BaseModel, Field
class SyncInputItem(BaseModel):
type: str = Field(..., description="Input kind: 'video', 'image' or 'audio'.")
url: str = Field(...)
class SyncActiveSpeakerDetection(BaseModel):
auto_detect: bool | None = Field(
None, description="Detect the active speaker automatically. Video input only; rejected for images."
)
frame_number: int | None = Field(
None, description="Frame used for manual speaker selection. Must be 0 for image inputs."
)
coordinates: list[int] | None = Field(
None, description="Pixel [x, y] of the speaker's face in the frame selected by frame_number."
)
class SyncGenerationOptions(BaseModel):
sync_mode: str | None = Field(
None,
description="How to resolve an audio/video duration mismatch: "
"cut_off, bounce, loop, silence or remap. Ignored for image inputs.",
)
i2v_prompt: str | None = Field(
None, description="Motion prompt for image-to-video generation. Image input only."
)
active_speaker_detection: SyncActiveSpeakerDetection | None = Field(None)
class SyncGenerationRequest(BaseModel):
model: str = Field(..., description="Generation model, e.g. 'sync-3'.")
input: list[SyncInputItem] = Field(
..., description="Exactly one visual input (video or image) plus one audio input."
)
options: SyncGenerationOptions | None = Field(None)
class SyncGeneration(BaseModel):
"""Subset of the Generation object returned by POST /v2/generate and GET /v2/generate/{id}."""
id: str = Field(...)
status: str = Field(..., description="PENDING | PROCESSING | COMPLETED | FAILED | REJECTED")
outputUrl: str | None = Field(None)
outputDuration: float | None = Field(None)
error: str | None = Field(None, description="Human-readable failure message.")
errorCode: str | None = Field(None, description="Stable machine-readable code from the GET /v2/errors catalog.")

161
comfy_api_nodes/nodes_bytedance.py
View File

@ -16,6 +16,7 @@ from comfy_api_nodes.apis.bytedance import (
RECOMMENDED_PRESETS_SEEDREAM_4_0,
RECOMMENDED_PRESETS_SEEDREAM_4_5,
RECOMMENDED_PRESETS_SEEDREAM_5_LITE,
RECOMMENDED_PRESETS_SEEDREAM_5_PRO,
SEEDANCE2_REF_VIDEO_PIXEL_LIMITS,
VIDEO_TASKS_EXECUTION_TIME,
GetAssetResponse,
@ -33,6 +34,7 @@ from comfy_api_nodes.apis.bytedance import (
SeedanceVirtualLibraryCreateAssetRequest,
Seedream4Options,
Seedream4TaskCreationRequest,
Seedream5OptimizePromptOptions,
TaskAudioContent,
TaskAudioContentUrl,
TaskCreationResponse,
@ -80,12 +82,14 @@ _VERIFICATION_POLL_TIMEOUT_SEC = 120
_VERIFICATION_POLL_INTERVAL_SEC = 3
SEEDREAM_MODELS = {
"seedream 5.0 pro": "seedream-5-0-pro-260628",
"seedream 5.0 lite": "seedream-5-0-260128",
"seedream-4-5-251128": "seedream-4-5-251128",
"seedream-4-0-250828": "seedream-4-0-250828",
}
SEEDREAM_PRESETS = {
"seedream-5-0-pro-260628": RECOMMENDED_PRESETS_SEEDREAM_5_PRO,
"seedream-5-0-260128": RECOMMENDED_PRESETS_SEEDREAM_5_LITE,
"seedream-4-5-251128": RECOMMENDED_PRESETS_SEEDREAM_4_5,
"seedream-4-0-250828": RECOMMENDED_PRESETS_SEEDREAM_4_0,
@ -743,8 +747,15 @@ class ByteDanceSeedreamNode(IO.ComfyNode):
return IO.NodeOutput(torch.cat([await download_url_to_image_tensor(i) for i in urls]))
def _seedream_model_inputs(*, max_ref_images: int, presets: list):
return [
def _seedream_model_inputs(
*,
max_ref_images: int,
presets: list,
max_width: int = 6240,
max_height: int = 4992,
supports_batch: bool = True,
):
inputs = [
IO.Combo.Input(
"size_preset",
options=[label for label, _, _ in presets],
@ -754,7 +765,7 @@ def _seedream_model_inputs(*, max_ref_images: int, presets: list):
"width",
default=2048,
min=1024,
max=6240,
max=max_width,
step=2,
tooltip="Custom width for image. Value is working only if `size_preset` is set to `Custom`",
),
@ -762,22 +773,27 @@ def _seedream_model_inputs(*, max_ref_images: int, presets: list):
"height",
default=2048,
min=1024,
max=4992,
max=max_height,
step=2,
tooltip="Custom height for image. Value is working only if `size_preset` is set to `Custom`",
),
IO.Int.Input(
"max_images",
default=1,
min=1,
max=max_ref_images,
step=1,
display_mode=IO.NumberDisplay.number,
tooltip="Maximum number of images to generate. With 1, exactly one image is produced. "
"With >1, the model generates between 1 and max_images related images "
"(e.g., story scenes, character variations). "
"Total images (input + generated) cannot exceed 15.",
),
]
if supports_batch:
inputs.append(
IO.Int.Input(
"max_images",
default=1,
min=1,
max=max_ref_images,
step=1,
display_mode=IO.NumberDisplay.number,
tooltip="Maximum number of images to generate. With 1, exactly one image is produced. "
"With >1, the model generates between 1 and max_images related images "
"(e.g., story scenes, character variations). "
"Total images (input + generated) cannot exceed 15.",
)
)
inputs.append(
IO.Autogrow.Input(
"images",
template=IO.Autogrow.TemplateNames(
@ -787,14 +803,18 @@ def _seedream_model_inputs(*, max_ref_images: int, presets: list):
),
tooltip=f"Optional reference image(s) for image-to-image or multi-reference generation. "
f"Up to {max_ref_images} images.",
),
IO.Boolean.Input(
"fail_on_partial",
default=False,
tooltip="If enabled, abort execution if any requested images are missing or return an error.",
advanced=True,
),
]
)
)
if supports_batch:
inputs.append(
IO.Boolean.Input(
"fail_on_partial",
default=False,
tooltip="If enabled, abort execution if any requested images are missing or return an error.",
advanced=True,
)
)
return inputs
class ByteDanceSeedreamNodeV2(IO.ComfyNode):
@ -816,6 +836,16 @@ class ByteDanceSeedreamNodeV2(IO.ComfyNode):
IO.DynamicCombo.Input(
"model",
options=[
IO.DynamicCombo.Option(
"seedream 5.0 pro",
_seedream_model_inputs(
max_ref_images=10,
presets=RECOMMENDED_PRESETS_SEEDREAM_5_PRO,
max_width=3136,
max_height=2496,
supports_batch=False,
),
),
IO.DynamicCombo.Option(
"seedream 5.0 lite",
_seedream_model_inputs(max_ref_images=14, presets=RECOMMENDED_PRESETS_SEEDREAM_5_LITE),
@ -846,6 +876,17 @@ class ByteDanceSeedreamNodeV2(IO.ComfyNode):
tooltip='Whether to add an "AI generated" watermark to the image.',
advanced=True,
),
IO.Boolean.Input(
"thinking",
default=True,
tooltip=(
"Enable the model's prompt-optimization reasoning ('thinking') for better adherence. "
"Can substantially increase generation time — notably on Seedream 5.0 Pro. "
"Can only be disabled for text-to-image (not when reference images are provided)."
),
optional=True,
advanced=True,
),
],
outputs=[
IO.Image.Output(),
@ -857,15 +898,27 @@ class ByteDanceSeedreamNodeV2(IO.ComfyNode):
],
is_api_node=True,
price_badge=IO.PriceBadge(
depends_on=IO.PriceBadgeDepends(widgets=["model"]),
depends_on=IO.PriceBadgeDepends(
widgets=["model", "model.size_preset", "model.width", "model.height"]
),
expr="""
(
$price := $contains(widgets.model, "5.0 lite") ? 0.035 :
$contains(widgets.model, "4-5") ? 0.04 : 0.03;
$sp := $lookup(widgets, "model.size_preset");
$px := $lookup(widgets, "model.width") * $lookup(widgets, "model.height");
$isPro := $contains(widgets.model, "5.0 pro");
$price := $isPro
? (
$contains($sp, "custom")
? ($px <= 2360000 ? 0.045 : 0.09)
: ($contains($sp, "1k") ? 0.045 : 0.09)
)
: $contains(widgets.model, "5.0 lite") ? 0.035
: $contains(widgets.model, "4-5") ? 0.04
: 0.03;
{
"type":"usd",
"type": "usd",
"usd": $price,
"format": { "suffix":" x images/Run", "approximate": true }
"format": { "suffix": $isPro ? "/Image" : " x images/Run", "approximate": true }
}
)
""",
@ -879,10 +932,12 @@ class ByteDanceSeedreamNodeV2(IO.ComfyNode):
model: dict,
seed: int = 0,
watermark: bool = False,
thinking: bool = True,
) -> IO.NodeOutput:
validate_string(prompt, strip_whitespace=True, min_length=1)
model_id = SEEDREAM_MODELS[model["model"]]
presets = SEEDREAM_PRESETS[model_id]
is_pro = "seedream-5-0-pro" in model_id
size_preset = model.get("size_preset", presets[0][0])
width = model.get("width", 2048)
@ -902,19 +957,29 @@ class ByteDanceSeedreamNodeV2(IO.ComfyNode):
out_num_pixels = w * h
mp_provided = out_num_pixels / 1_000_000.0
if ("seedream-4-5" in model_id or "seedream-5-0" in model_id) and out_num_pixels < 3686400:
raise ValueError(
f"Minimum image resolution for the selected model is 3.68MP, but {mp_provided:.2f}MP provided."
)
if "seedream-4-0" in model_id and out_num_pixels < 921600:
raise ValueError(
f"Minimum image resolution that the selected model can generate is 0.92MP, "
f"but {mp_provided:.2f}MP provided."
)
if out_num_pixels > 16_777_216:
raise ValueError(
f"Maximum image resolution for the selected model is 16.78MP, but {mp_provided:.2f}MP provided."
)
if is_pro:
if out_num_pixels < 921_600:
raise ValueError(
f"Minimum image resolution for the selected model is 0.92MP, but {mp_provided:.2f}MP provided."
)
if out_num_pixels > 4_194_304:
raise ValueError(
f"Maximum image resolution for the selected model is 4.19MP, but {mp_provided:.2f}MP provided."
)
else:
if ("seedream-4-5" in model_id or "seedream-5-0" in model_id) and out_num_pixels < 3_686_400:
raise ValueError(
f"Minimum image resolution for the selected model is 3.68MP, but {mp_provided:.2f}MP provided."
)
if "seedream-4-0" in model_id and out_num_pixels < 921_600:
raise ValueError(
f"Minimum image resolution that the selected model can generate is 0.92MP, "
f"but {mp_provided:.2f}MP provided."
)
if out_num_pixels > 16_777_216:
raise ValueError(
f"Maximum image resolution for the selected model is 16.78MP, but {mp_provided:.2f}MP provided."
)
image_tensors: list[Input.Image] = [t for t in images_dict.values() if t is not None]
n_input_images = sum(get_number_of_images(t) for t in image_tensors)
@ -927,6 +992,10 @@ class ByteDanceSeedreamNodeV2(IO.ComfyNode):
raise ValueError(
"The maximum number of generated images plus the number of reference images cannot exceed 15."
)
if not thinking and n_input_images > 0:
raise ValueError(
"'thinking' can only be disabled for text-to-image; enable it when using reference images."
)
reference_images_urls: list[str] = []
if image_tensors:
@ -940,6 +1009,9 @@ class ByteDanceSeedreamNodeV2(IO.ComfyNode):
wait_label="Uploading reference images",
)
optimize_prompt_options = None
if n_input_images == 0:
optimize_prompt_options = Seedream5OptimizePromptOptions(thinking="enabled" if thinking else "disabled")
response = await sync_op(
cls,
ApiEndpoint(path=BYTEPLUS_IMAGE_ENDPOINT, method="POST"),
@ -950,9 +1022,10 @@ class ByteDanceSeedreamNodeV2(IO.ComfyNode):
image=reference_images_urls,
size=f"{w}x{h}",
seed=seed,
sequential_image_generation=sequential_image_generation,
sequential_image_generation_options=Seedream4Options(max_images=max_images),
sequential_image_generation=None if is_pro else sequential_image_generation,
sequential_image_generation_options=None if is_pro else Seedream4Options(max_images=max_images),
watermark=watermark,
optimize_prompt_options=optimize_prompt_options,
),
)
if len(response.data) == 1:

6
comfy_api_nodes/nodes_gemini.py
View File

@ -1133,7 +1133,9 @@ class GeminiImage2(IO.ComfyNode):
) -> IO.NodeOutput:
validate_string(prompt, strip_whitespace=True, min_length=1)
if model == "Nano Banana 2 (Gemini 3.1 Flash Image)":
model = "gemini-3.1-flash-image-preview"
model = "gemini-3.1-flash-image"
elif model == "gemini-3-pro-image-preview":
model = "gemini-3-pro-image"
parts: list[GeminiPart] = [GeminiPart(text=prompt)]
if images is not None:
@ -1507,7 +1509,7 @@ class GeminiNanoBanana2V2(IO.ComfyNode):
validate_string(prompt, strip_whitespace=True, min_length=1)
model_choice = model["model"]
if model_choice == "Nano Banana 2 (Gemini 3.1 Flash Image)":
model_id = "gemini-3.1-flash-image-preview"
model_id = "gemini-3.1-flash-image"
elif model_choice == "Nano Banana 2 Lite":
model_id = "gemini-3.1-flash-lite-image"
else:

1
comfy_api_nodes/nodes_hunyuan3d.py
View File

@ -642,6 +642,7 @@ class Tencent3DPartNode(IO.ComfyNode):
response_model=To3DProTaskCreateResponse,
data=To3DPartTaskRequest(
File=TaskFile3DInput(Type=file_format.upper(), Url=model_url),
EnableStagedGeneration=True,
),
is_rate_limited=_is_tencent_rate_limited,
)

18
comfy_api_nodes/nodes_openai.py
View File

@ -41,6 +41,9 @@ STARTING_POINT_ID_PATTERN = r"<starting_point_id:(.*)>"
class SupportedOpenAIModel(str, Enum):
gpt_5_6_sol = "gpt-5.6-sol"
gpt_5_6_terra = "gpt-5.6-terra"
gpt_5_6_luna = "gpt-5.6-luna"
gpt_5_5_pro = "gpt-5.5-pro"
gpt_5_5 = "gpt-5.5"
gpt_5 = "gpt-5"
@ -1063,6 +1066,21 @@ class OpenAIChatNode(IO.ComfyNode):
"usd": [0.002, 0.008],
"format": { "approximate": true, "separator": "-", "suffix": " per 1K tokens" }
}
: $contains($m, "gpt-5.6-terra") ? {
"type": "list_usd",
"usd": [0.0025, 0.015],
"format": { "approximate": true, "separator": "-", "suffix": " per 1K tokens" }
}
: $contains($m, "gpt-5.6-luna") ? {
"type": "list_usd",
"usd": [0.001, 0.006],
"format": { "approximate": true, "separator": "-", "suffix": " per 1K tokens" }
}
: $contains($m, "gpt-5.6") ? {
"type": "list_usd",
"usd": [0.005, 0.03],
"format": { "approximate": true, "separator": "-", "suffix": " per 1K tokens" }
}
: $contains($m, "gpt-5.5-pro") ? {
"type": "list_usd",
"usd": [0.03, 0.18],

391
comfy_api_nodes/nodes_sync_so.py Normal file
View File

@ -0,0 +1,391 @@
from typing_extensions import override
from comfy_api.latest import IO, ComfyExtension, Input
from comfy_api_nodes.apis.sync_so import (
SyncActiveSpeakerDetection,
SyncGeneration,
SyncGenerationOptions,
SyncGenerationRequest,
SyncInputItem,
)
from comfy_api_nodes.util import (
ApiEndpoint,
download_url_to_video_output,
downscale_image_tensor,
downscale_image_tensor_by_max_side,
get_image_dimensions,
get_number_of_images,
poll_op,
sync_op,
upload_audio_to_comfyapi,
upload_image_to_comfyapi,
upload_video_to_comfyapi,
validate_audio_duration,
)
class SyncLipSyncNode(IO.ComfyNode):
@classmethod
def define_schema(cls) -> IO.Schema:
return IO.Schema(
node_id="SyncLipSyncNode",
display_name="sync.so Lip Sync",
category="partner/video/sync.so",
description=(
"Re-sync mouth movement in a video to new speech audio using sync.so. "
"Handles close-ups, profiles and obstructions automatically while preserving "
"the speaker's expression. Cost scales with output duration."
),
inputs=[
IO.Video.Input(
"video",
tooltip="Footage of the speaker to re-sync. Up to 4K (4096x2160); "
"a constant frame rate of 24/25/30 fps works best.",
),
IO.Audio.Input(
"audio",
tooltip="Speech audio to sync the mouth to.",
),
IO.Int.Input(
"seed",
default=42,
min=0,
max=2147483647,
control_after_generate=True,
tooltip="Seed controls whether the node should re-run; "
"results are non-deterministic regardless of seed.",
),
IO.DynamicCombo.Input(
"model",
options=[
IO.DynamicCombo.Option(
"sync-3",
[
IO.Combo.Input(
"sync_mode",
options=["bounce", "cut_off", "loop", "silence", "remap"],
default="bounce",
tooltip=(
"How to handle a duration mismatch between video and audio; "
"this also sets the output length. "
"bounce: video plays forward then backward until the audio ends "
"(output = audio length). "
"loop: video restarts until the audio ends (output = audio length). "
"remap: video is time-stretched to match the audio (output = audio length). "
"cut_off: the longer track is trimmed (output = shorter length). "
"silence: nothing is trimmed; the shorter track is padded "
"(output = longer length)."
),
),
IO.Combo.Input(
"speaker_selection",
options=["default", "auto-detect", "coordinates"],
default="default",
tooltip=(
"Which face to lipsync when several people are visible. "
"default: let the model decide. "
"auto-detect: detect and follow the active speaker. "
"coordinates: target the face at pixel (speaker_x, speaker_y) "
"in the frame chosen by speaker_frame."
),
),
IO.Int.Input(
"speaker_frame",
default=0,
min=0,
max=1_000_000,
advanced=True,
tooltip="Video frame used to locate the speaker. "
"Only used when speaker_selection is 'coordinates'.",
),
IO.Int.Input(
"speaker_x",
default=0,
min=0,
max=4096,
advanced=True,
tooltip="X pixel coordinate of the speaker's face. "
"Only used when speaker_selection is 'coordinates'.",
),
IO.Int.Input(
"speaker_y",
default=0,
min=0,
max=4096,
advanced=True,
tooltip="Y pixel coordinate of the speaker's face. "
"Only used when speaker_selection is 'coordinates'.",
),
],
)
],
tooltip="sync.so generation model.",
),
],
outputs=[IO.Video.Output()],
hidden=[
IO.Hidden.auth_token_comfy_org,
IO.Hidden.api_key_comfy_org,
IO.Hidden.unique_id,
],
is_api_node=True,
price_badge=IO.PriceBadge(
expr="""{"type":"usd","usd":0.19019,"format":{"approximate":true,"suffix":"/second"}}""",
),
)
@classmethod
async def execute(
cls,
video: Input.Video,
audio: Input.Audio,
seed: int,
model: dict,
) -> IO.NodeOutput:
try:
width, height = video.get_dimensions()
except Exception:
width = height = None
if width and height and (max(width, height) > 4096 or width * height > 4096 * 2160):
raise ValueError(
f"sync.so rejects videos above 4K (4096x2160); got {width}x{height}. Downscale the video first."
)
validate_audio_duration(audio, max_duration=600)
if model["speaker_selection"] == "auto-detect":
speaker_detection = SyncActiveSpeakerDetection(auto_detect=True)
elif model["speaker_selection"] == "coordinates":
speaker_detection = SyncActiveSpeakerDetection(
frame_number=model["speaker_frame"],
coordinates=[model["speaker_x"], model["speaker_y"]],
)
else:
speaker_detection = None
video_url = await upload_video_to_comfyapi(cls, video, max_duration=600)
audio_url = await upload_audio_to_comfyapi(cls, audio)
generation = await sync_op(
cls,
ApiEndpoint(path="/proxy/synclabs/v2/generate", method="POST"),
response_model=SyncGeneration,
data=SyncGenerationRequest(
model=model["model"],
input=[
SyncInputItem(type="video", url=video_url),
SyncInputItem(type="audio", url=audio_url),
],
options=SyncGenerationOptions(
sync_mode=model["sync_mode"],
active_speaker_detection=speaker_detection,
),
),
)
generation = await poll_op(
cls,
ApiEndpoint(path=f"/proxy/synclabs/v2/generate/{generation.id}"),
response_model=SyncGeneration,
status_extractor=lambda g: g.status,
completed_statuses=["COMPLETED", "FAILED", "REJECTED"],
failed_statuses=[],
queued_statuses=["PENDING"],
poll_interval=10.0,
)
if generation.status != "COMPLETED":
code = f" [{generation.errorCode}]" if generation.errorCode else ""
raise ValueError(
f"sync.so generation {generation.status.lower()}{code}: "
f"{generation.error or 'no error details provided'}"
)
if not generation.outputUrl:
raise ValueError("sync.so generation completed but no output URL was returned.")
return IO.NodeOutput(await download_url_to_video_output(generation.outputUrl))
class SyncTalkingImageNode(IO.ComfyNode):
@classmethod
def define_schema(cls) -> IO.Schema:
return IO.Schema(
node_id="SyncTalkingImageNode",
display_name="sync.so Talking Image",
category="partner/video/sync.so",
description=(
"Animate a still portrait into a talking video driven by speech audio, "
"using sync.so's sync-3 model. The output duration matches the audio. "
"Cost scales with output duration."
),
inputs=[
IO.Image.Input(
"image",
tooltip="A single image with a clearly visible face, up to 4K (4096x2160).",
),
IO.Audio.Input(
"audio",
tooltip="Speech audio driving the talking video; the output duration matches it. "
"Chain any TTS node here to drive the animation from text.",
),
IO.String.Input(
"prompt",
multiline=True,
default="",
tooltip="Optional guidance for how the portrait comes to life, e.g. "
"'make the subject smile and look at the camera'. "
"Leave empty for natural talking motion.",
),
IO.Int.Input(
"seed",
default=0,
min=0,
max=2147483647,
control_after_generate=True,
tooltip="Seed controls whether the node should re-run; "
"results are non-deterministic regardless of seed.",
),
IO.DynamicCombo.Input(
"model",
options=[
IO.DynamicCombo.Option(
"sync-3",
[
IO.Combo.Input(
"speaker_selection",
options=["default", "coordinates"],
default="default",
tooltip=(
"Which face to animate when several people are visible. "
"default: let the model decide. "
"coordinates: target the face at pixel (speaker_x, speaker_y) "
"in the image. Auto-detection is not supported for images."
),
),
IO.Int.Input(
"speaker_x",
default=0,
min=0,
max=4096,
advanced=True,
tooltip="X pixel coordinate of the speaker's face. "
"Only used when speaker_selection is 'coordinates'.",
),
IO.Int.Input(
"speaker_y",
default=0,
min=0,
max=4096,
advanced=True,
tooltip="Y pixel coordinate of the speaker's face. "
"Only used when speaker_selection is 'coordinates'.",
),
IO.Boolean.Input(
"auto_downscale",
default=True,
advanced=True,
tooltip="Automatically downscale the image if it exceeds the 4K "
"(4096x2160) input limit; speaker coordinates are scaled to match. "
"When disabled, an oversized image raises an error instead.",
),
],
)
],
tooltip="sync.so generation model. Image input is exclusive to sync-3.",
),
],
outputs=[IO.Video.Output()],
hidden=[
IO.Hidden.auth_token_comfy_org,
IO.Hidden.api_key_comfy_org,
IO.Hidden.unique_id,
],
is_api_node=True,
price_badge=IO.PriceBadge(
expr="""{"type":"usd","usd":0.19019,"format":{"approximate":true,"suffix":"/second"}}""",
),
)
@classmethod
async def execute(
cls,
image: Input.Image,
audio: Input.Audio,
prompt: str,
seed: int,
model: dict,
) -> IO.NodeOutput:
if get_number_of_images(image) != 1:
raise ValueError("Exactly one image is required; got a batch. Pick one frame first.")
validate_audio_duration(audio, max_duration=600)
height, width = get_image_dimensions(image)
speaker_x, speaker_y = model["speaker_x"], model["speaker_y"]
if max(width, height) > 4096 or width * height > 4096 * 2160:
if not model["auto_downscale"]:
raise ValueError(
f"sync.so rejects images above 4K (4096x2160); got {width}x{height}. "
"Downscale the image first or enable auto_downscale."
)
image = downscale_image_tensor(image, total_pixels=4096 * 2160)
image = downscale_image_tensor_by_max_side(image, max_side=4096)
new_height, new_width = get_image_dimensions(image)
# speaker coordinates are given in the original image's pixel space
speaker_x = min(new_width - 1, round(speaker_x * new_width / width))
speaker_y = min(new_height - 1, round(speaker_y * new_height / height))
if model["speaker_selection"] == "coordinates":
speaker_detection = SyncActiveSpeakerDetection(
frame_number=0, # images have a single frame; auto_detect is rejected by the API
coordinates=[speaker_x, speaker_y],
)
else:
speaker_detection = None
image_url = await upload_image_to_comfyapi(cls, image, mime_type="image/png", total_pixels=None)
audio_url = await upload_audio_to_comfyapi(cls, audio)
generation = await sync_op(
cls,
ApiEndpoint(path="/proxy/synclabs/v2/generate", method="POST"),
response_model=SyncGeneration,
data=SyncGenerationRequest(
model=model["model"],
input=[
SyncInputItem(type="image", url=image_url),
SyncInputItem(type="audio", url=audio_url),
],
options=SyncGenerationOptions(
i2v_prompt=prompt.strip() or None,
active_speaker_detection=speaker_detection,
),
),
)
generation = await poll_op(
cls,
ApiEndpoint(path=f"/proxy/synclabs/v2/generate/{generation.id}"),
response_model=SyncGeneration,
status_extractor=lambda g: g.status,
completed_statuses=["COMPLETED", "FAILED", "REJECTED"],
failed_statuses=[],
queued_statuses=["PENDING"],
poll_interval=10.0,
)
if generation.status != "COMPLETED":
code = f" [{generation.errorCode}]" if generation.errorCode else ""
raise ValueError(
f"sync.so generation {generation.status.lower()}{code}: "
f"{generation.error or 'no error details provided'}"
)
if not generation.outputUrl:
raise ValueError("sync.so generation completed but no output URL was returned.")
return IO.NodeOutput(await download_url_to_video_output(generation.outputUrl))
class SyncExtension(ComfyExtension):
@override
async def get_node_list(self) -> list[type[IO.ComfyNode]]:
return [
SyncLipSyncNode,
SyncTalkingImageNode,
]
async def comfy_entrypoint() -> SyncExtension:
return SyncExtension()

12
comfy_api_nodes/util/_helpers.py
View File

@ -11,9 +11,12 @@ from io import BytesIO
from yarl import URL
from comfy.cli_args import args
from comfy.comfy_api_env import normalize_comfy_api_base
from comfy.deploy_environment import get_deploy_environment
from comfy.model_management import processing_interrupted
from comfy_api.latest import IO
from comfy_execution.utils import get_executing_context
from comfyui_version import __version__ as comfyui_version
from .common_exceptions import ProcessingInterrupted
@ -55,15 +58,20 @@ def get_comfy_api_headers(node_cls: type[IO.ComfyNode]) -> dict[str, str]:
relative/cloud URLs resolved against ``default_base_url()``; because the result
includes auth, callers must not attach it to arbitrary absolute/presigned URLs.
"""
return {
headers = {
**get_auth_header(node_cls),
"Comfy-Env": get_deploy_environment(),
"Comfy-Usage-Source": get_usage_source(node_cls),
"Comfy-Core-Version": comfyui_version,
}
ctx = get_executing_context()
if ctx is not None:
headers["Comfy-Job-Id"] = ctx.prompt_id
return headers
def default_base_url() -> str:
return getattr(args, "comfy_api_base", "https://api.comfy.org")
return normalize_comfy_api_base(getattr(args, "comfy_api_base", "https://api.comfy.org"))
async def sleep_with_interrupt(

7
comfy_api_nodes/util/request_logger.py
View File

@ -9,6 +9,7 @@ from typing import Any
import folder_paths
logger = logging.getLogger(__name__)
_SENSITIVE_HEADERS = {"authorization", "x-api-key"}
def get_log_directory():
@ -73,6 +74,10 @@ def _format_data_for_logging(data: Any) -> str:
return str(data)
def _redact_headers(headers: dict) -> dict:
return {k: ("***" if k.lower() in _SENSITIVE_HEADERS else v) for k, v in headers.items()}
def log_request_response(
operation_id: str,
request_method: str,
@ -101,7 +106,7 @@ def log_request_response(
log_content.append(f"Method: {request_method}")
log_content.append(f"URL: {request_url}")
if request_headers:
log_content.append(f"Headers:\n{_format_data_for_logging(request_headers)}")
log_content.append(f"Headers:\n{_format_data_for_logging(_redact_headers(request_headers))}")
if request_params:
log_content.append(f"Params:\n{_format_data_for_logging(request_params)}")
if request_data is not None:

9
comfy_api_nodes/util/upload_helpers.py
View File

@ -158,7 +158,14 @@ async def upload_video_to_comfyapi(
# Convert VideoInput to BytesIO using specified container/codec
video_bytes_io = BytesIO()
video.save_to(video_bytes_io, format=container, codec=codec)
try:
video.save_to(video_bytes_io, format=container, codec=codec)
except Exception as e:
raise ValueError(
f"Could not convert the input video to {container.value.upper()} for upload; "
f"the file may be corrupted or use an unsupported codec. "
f"Try re-exporting it as MP4 (H.264). Original error: {e}"
) from e
video_bytes_io.seek(0)
return await upload_file_to_comfyapi(cls, video_bytes_io, filename, upload_mime_type, wait_label)

44
comfy_execution/caching.py
View File

@ -503,6 +503,22 @@ RAM_CACHE_DEFAULT_RAM_USAGE = 0.05
RAM_CACHE_OLD_WORKFLOW_OOM_MULTIPLIER = 1.3
RAM_CACHE_LARGE_INTERMEDIATE = 512 * 1024 ** 2
def all_outputs_dynamic(outputs):
if outputs is None:
return False
for output in outputs:
if isinstance(output, (list, tuple)):
if not all_outputs_dynamic(output):
return False
elif not hasattr(output, "is_dynamic") or not output.is_dynamic():
return False
return True
class RAMPressureCache(LRUCache):
def __init__(self, key_class, enable_providers=False):
@ -524,20 +540,25 @@ class RAMPressureCache(LRUCache):
self.timestamps[self.cache_key_set.get_data_key(node_id)] = time.time()
super().set_local(node_id, value)
def ram_release(self, target, free_active=False):
def ram_release(self, target, free_active=False, min_entry_size=0):
if psutil.virtual_memory().available >= target:
return
return 0
clean_list = []
for key, cache_entry in self.cache.items():
if not free_active and self.used_generation[key] == self.generation:
continue
oom_score = RAM_CACHE_OLD_WORKFLOW_OOM_MULTIPLIER ** (self.generation - self.used_generation[key])
if all_outputs_dynamic(cache_entry.outputs) and self.used_generation[key] == self.generation:
continue
oom_score = RAM_CACHE_OLD_WORKFLOW_OOM_MULTIPLIER ** (self.generation - self.used_generation[key])
ram_usage = RAM_CACHE_DEFAULT_RAM_USAGE
oom_ram_usage = ram_usage
def scan_list_for_ram_usage(outputs):
nonlocal ram_usage
nonlocal ram_usage, oom_ram_usage
if outputs is None:
return
for output in outputs:
@ -545,19 +566,26 @@ class RAMPressureCache(LRUCache):
scan_list_for_ram_usage(output)
elif isinstance(output, torch.Tensor) and output.device.type == 'cpu':
ram_usage += output.numel() * output.element_size()
oom_ram_usage += output.numel() * output.element_size()
elif isinstance(output, ModelPatcher) and self.used_generation[key] != self.generation:
#old ModelPatchers are the first to go
ram_usage = 1e30
oom_ram_usage = 1e30
scan_list_for_ram_usage(cache_entry.outputs)
oom_score *= ram_usage
if ram_usage < min_entry_size:
continue
oom_score *= oom_ram_usage
#In the case where we have no information on the node ram usage at all,
#break OOM score ties on the last touch timestamp (pure LRU)
bisect.insort(clean_list, (oom_score, self.timestamps[key], key))
bisect.insort(clean_list, (oom_score, self.timestamps[key], key, ram_usage))
freed = 0
while psutil.virtual_memory().available < target and clean_list:
_, _, key = clean_list.pop()
_, _, key, ram_usage = clean_list.pop()
del self.cache[key]
self.used_generation.pop(key, None)
self.timestamps.pop(key, None)
self.children.pop(key, None)
freed += ram_usage
return freed

15
comfy_execution/jobs.py
View File

@ -56,6 +56,9 @@ PREVIEWABLE_MEDIA_TYPES = frozenset({'images', 'video', 'audio', '3d', 'text'})
# 3D file extensions for preview fallback (no dedicated media_type exists)
THREE_D_EXTENSIONS = frozenset({'.obj', '.fbx', '.gltf', '.glb', '.usdz'})
# Text file extensions for preview fallback (the formats SaveText can produce)
TEXT_EXTENSIONS = frozenset({'.txt', '.md', '.json'})
def has_3d_extension(filename: str) -> bool:
lower = filename.lower()
@ -143,9 +146,10 @@ def is_previewable(media_type: str, item: dict) -> bool:
Maintains backwards compatibility with existing logic.
Priority:
1. media_type is 'images', 'video', 'audio', or '3d'
1. media_type is 'images', 'video', 'audio', '3d', or 'text'
2. format field starts with 'video/' or 'audio/'
3. filename has a 3D extension (.obj, .fbx, .gltf, .glb, .usdz)
4. filename has a text extension (.txt, .md, .json, ...)
"""
if media_type in PREVIEWABLE_MEDIA_TYPES:
return True
@ -156,10 +160,12 @@ def is_previewable(media_type: str, item: dict) -> bool:
if fmt and (fmt.startswith('video/') or fmt.startswith('audio/')):
return True
# Check for 3D files by extension
# Check for 3D and text files by extension
filename = item.get('filename', '').lower()
if any(filename.endswith(ext) for ext in THREE_D_EXTENSIONS):
return True
if any(filename.endswith(ext) for ext in TEXT_EXTENSIONS):
return True
return False
@ -255,6 +261,10 @@ def get_outputs_summary(outputs: dict) -> tuple[int, Optional[dict]]:
Preview priority (matching frontend):
1. type="output" with previewable media
2. Any previewable media
Text content entries (strings under 'text') are preview-only metadata,
matching the frontend's METADATA_KEYS: they can serve as the fallback
preview but are not counted as outputs.
"""
count = 0
preview_output = None
@ -275,7 +285,6 @@ def get_outputs_summary(outputs: dict) -> tuple[int, Optional[dict]]:
if normalized is None:
# Not a 3D file string — check for text preview
if media_type == 'text':
count += 1
if preview_output is None:
if isinstance(item, tuple):
text_value = item[0] if item else ''

1
comfy_extras/nodes_audio.py
View File

@ -298,6 +298,7 @@ class PreviewAudio(IO.ComfyNode):
search_aliases=["play audio"],
display_name="Preview Audio",
category="audio",
description="Preview the audio without saving it to the ComfyUI output directory.",
inputs=[
IO.Audio.Input("audio"),
],

132
comfy_extras/nodes_bounding_boxes.py
View File

@ -1,3 +1,5 @@
import json
import numpy as np
import torch
from PIL import Image, ImageDraw, ImageEnhance, ImageFont
@ -166,6 +168,111 @@ def boxes_to_regions(boxes, width: int, height: int) -> list:
return regions
def normalize_incoming_boxes(bboxes) -> list:
if isinstance(bboxes, dict):
frame = [bboxes]
elif not isinstance(bboxes, list) or not bboxes:
frame = []
elif isinstance(bboxes[0], dict):
frame = bboxes
else:
frame = bboxes[0] if isinstance(bboxes[0], list) else []
boxes = []
for box in frame:
if not isinstance(box, dict):
continue
norm = {
"x": box.get("x", 0),
"y": box.get("y", 0),
"width": box.get("width", 0),
"height": box.get("height", 0),
}
meta = box.get("metadata")
if isinstance(meta, dict):
norm["metadata"] = meta
boxes.append(norm)
return boxes
def _looks_like_element(box: dict) -> bool:
bbox = box.get("bbox")
return isinstance(bbox, (list, tuple)) and len(bbox) == 4
def _looks_like_bbox(box: dict) -> bool:
return all(key in box for key in ("x", "y", "width", "height"))
def elements_to_boxes(elements: list, width: int, height: int) -> list:
boxes = []
for element in elements:
if not isinstance(element, dict):
continue
bbox = element.get("bbox")
if not (isinstance(bbox, (list, tuple)) and len(bbox) == 4):
raise ValueError("bboxes element is missing a valid 'bbox' [ymin, xmin, ymax, xmax]")
try:
ymin, xmin, ymax, xmax = (float(v) / 1000.0 for v in bbox)
except (TypeError, ValueError):
raise ValueError("bboxes element 'bbox' must contain four numbers")
etype = "text" if element.get("type") == "text" else "obj"
boxes.append({
"x": round(min(xmin, xmax) * width),
"y": round(min(ymin, ymax) * height),
"width": round(abs(xmax - xmin) * width),
"height": round(abs(ymax - ymin) * height),
"metadata": {
"type": etype,
"text": element.get("text", "") if etype == "text" else "",
"desc": element.get("desc", ""),
"palette": element.get("color_palette", []) or [],
},
})
return boxes
def boxes_from_input(data, width: int, height: int) -> list:
if data is None:
return []
if isinstance(data, str):
text = data.strip()
if not text:
return []
try:
data = json.loads(text)
except (ValueError, TypeError) as exc:
raise ValueError(f"bboxes string input is not valid JSON: {exc}") from exc
if isinstance(data, dict):
if _looks_like_element(data):
return elements_to_boxes([data], width, height)
if _looks_like_bbox(data):
return normalize_incoming_boxes(data)
raise ValueError(
"bboxes dict must be a bounding box (x, y, width, height) or an element (with a 'bbox')"
)
if not isinstance(data, list):
raise ValueError(
"bboxes input must be bounding boxes, elements, or a JSON string, "
f"got {type(data).__name__}"
)
if not data:
return []
first = data[0]
if isinstance(first, list):
return normalize_incoming_boxes(data)
if isinstance(first, dict):
if _looks_like_element(first):
return elements_to_boxes(data, width, height)
if _looks_like_bbox(first):
return normalize_incoming_boxes(data)
raise ValueError(
"bboxes items must be bounding boxes (x, y, width, height) or elements (with a 'bbox')"
)
raise ValueError(
f"bboxes list must contain bounding boxes or elements, got {type(first).__name__}"
)
def _norm_bbox(region: dict) -> list[int]:
def grid(value: float) -> int:
return max(0, min(1000, round(value * 1000)))
@ -217,29 +324,48 @@ class CreateBoundingBoxes(io.ComfyNode):
optional=True,
tooltip="Optional image used as background in the canvas and preview.",
),
io.MultiType.Input(
"bboxes",
[io.BoundingBox, io.Array, io.String],
optional=True,
tooltip="Bounding boxes, elements, or a JSON string to initialize the canvas. A new upstream value initializes the canvas; edits made on the canvas take priority and are kept until the upstream value changes again.",
),
io.Int.Input("width", default=1024, min=64, max=16384, step=16,
tooltip="Width of the canvas and the pixel grid for the bounding boxes."),
io.Int.Input("height", default=1024, min=64, max=16384, step=16,
tooltip="Height of the canvas and the pixel grid for the bounding boxes."),
editor_state,
io.BoundingBoxes.Input(
"last_incoming",
optional=True,
tooltip="Internal state managed by the canvas: the upstream bboxes value that last initialized it. Leave empty to re-initialize the canvas from the bboxes input on the next run.",
),
],
outputs=[
io.Image.Output(display_name="preview"),
io.BoundingBox.Output(display_name="bboxes"),
io.Array.Output(display_name="elements"),
],
is_output_node=True,
is_experimental=True,
)
@classmethod
def execute(cls, width, height, editor_state=None, background=None) -> io.NodeOutput:
regions = boxes_to_regions(editor_state, width, height)
def execute(cls, width, height, editor_state=None, last_incoming=None, background=None, bboxes=None) -> io.NodeOutput:
incoming = boxes_from_input(bboxes, width, height)
applied = last_incoming if isinstance(last_incoming, list) else []
upstream_changed = bool(incoming) and incoming != applied
source = incoming if upstream_changed else (editor_state or [])
regions = boxes_to_regions(source, width, height)
preview = render_preview(regions, width, height, _bg_from_image(background))
ui = {"dims": [width, height]}
if incoming:
ui["input_bboxes"] = incoming
return io.NodeOutput(
preview,
fractions_to_bbox_frame(regions, width, height),
build_elements(regions),
ui={"dims": [width, height]},
ui=ui,
)

37
comfy_extras/nodes_images.py
View File

@ -988,15 +988,18 @@ class ImageMergeTileList(IO.ComfyNode):
# Format specifications
# ---------------------------------------------------------------------------
# Maps (file_format, bit_depth, has_alpha) -> (numpy dtype scale, av pixel format,
# stream pix_fmt). Keeps the encode path declarative instead of branchy.
# Maps (file_format, bit_depth, num_channels) -> (quantization scale, numpy dtype,
# av frame pix_fmt, stream pix_fmt). Keeps the encode path declarative instead of branchy.
_FORMAT_SPECS = {
("png", "8-bit", False): {"scale": 255.0, "dtype": np.uint8, "frame_fmt": "rgb24", "stream_fmt": "rgb24"},
("png", "8-bit", True): {"scale": 255.0, "dtype": np.uint8, "frame_fmt": "rgba", "stream_fmt": "rgba"},
("png", "16-bit", False): {"scale": 65535.0, "dtype": np.uint16, "frame_fmt": "rgb48le", "stream_fmt": "rgb48be"},
("png", "16-bit", True): {"scale": 65535.0, "dtype": np.uint16, "frame_fmt": "rgba64le", "stream_fmt": "rgba64be"},
("exr", "32-bit float", False): {"scale": 1.0, "dtype": np.float32, "frame_fmt": "gbrpf32le", "stream_fmt": "gbrpf32le"},
("exr", "32-bit float", True): {"scale": 1.0, "dtype": np.float32, "frame_fmt": "gbrapf32le", "stream_fmt": "gbrapf32le"},
("png", "8-bit", 1): {"scale": 255.0, "dtype": np.uint8, "frame_fmt": "gray", "stream_fmt": "gray"},
("png", "8-bit", 3): {"scale": 255.0, "dtype": np.uint8, "frame_fmt": "rgb24", "stream_fmt": "rgb24"},
("png", "8-bit", 4): {"scale": 255.0, "dtype": np.uint8, "frame_fmt": "rgba", "stream_fmt": "rgba"},
("png", "16-bit", 1): {"scale": 65535.0, "dtype": np.uint16, "frame_fmt": "gray16le", "stream_fmt": "gray16be"},
("png", "16-bit", 3): {"scale": 65535.0, "dtype": np.uint16, "frame_fmt": "rgb48le", "stream_fmt": "rgb48be"},
("png", "16-bit", 4): {"scale": 65535.0, "dtype": np.uint16, "frame_fmt": "rgba64le", "stream_fmt": "rgba64be"},
("exr", "32-bit float", 1): {"scale": 1.0, "dtype": np.float32, "frame_fmt": "grayf32le", "stream_fmt": "grayf32le"},
("exr", "32-bit float", 3): {"scale": 1.0, "dtype": np.float32, "frame_fmt": "gbrpf32le", "stream_fmt": "gbrpf32le"},
("exr", "32-bit float", 4): {"scale": 1.0, "dtype": np.float32, "frame_fmt": "gbrapf32le", "stream_fmt": "gbrapf32le"},
}
@ -1035,10 +1038,11 @@ def hlg_to_linear(t: torch.Tensor) -> torch.Tensor:
return torch.cat([hlg_to_linear(rgb), alpha], dim=-1)
# Piecewise: sqrt branch below 0.5, log branch above.
# Clamp inside the log branch so negative / out-of-range values don't blow up;
# Clamp the log branch at the 0.5 branch point (not above it) so the
# unselected lane stays finite in exp() without altering selected values;
# values above 1.0 are allowed and extrapolate naturally.
low = (t ** 2) / 3.0
high = (torch.exp((t.clamp(min=_HLG_C) - _HLG_C) / _HLG_A) + _HLG_B) / 12.0
high = (torch.exp((t.clamp(min=0.5) - _HLG_C) / _HLG_A) + _HLG_B) / 12.0
return torch.where(t <= 0.5, low, high)
@ -1231,7 +1235,8 @@ def _encode_image(
bit_depth: str,
colorspace: str,
) -> bytes:
"""Encode a single HxWxC tensor to PNG or EXR bytes in memory.
"""Encode a single HxWxC (or channel-less HxW grayscale) tensor to PNG or
EXR bytes in memory. Grayscale is written as single-channel PNG / Y-only EXR.
For EXR the input is interpreted according to `colorspace` and converted
to scene-linear (EXR's convention) before writing:
@ -1245,10 +1250,16 @@ def _encode_image(
For PNG, colorspace selection does not modify pixels — PNG is delivered
sRGB-encoded and there is no PNG path for wide-gamut HDR in this node.
"""
if img_tensor.ndim == 2:
img_tensor = img_tensor.unsqueeze(-1) # Some nodes emit grayscale as (H, W) with no channel dim, mask-style.
height, width, num_channels = img_tensor.shape
has_alpha = num_channels == 4
spec = _FORMAT_SPECS[(file_format, bit_depth, has_alpha)]
spec = _FORMAT_SPECS.get((file_format, bit_depth, num_channels))
if spec is None:
raise ValueError(
f"No {file_format}/{bit_depth} encoder for {num_channels}-channel images: "
"supported channel counts are 1 (grayscale), 3 (RGB) and 4 (RGBA)."
)
if spec["dtype"] == np.float32:
# EXR path: preserve full range, no clamp.

102
comfy_extras/nodes_joyimage.py Normal file
View File

@ -0,0 +1,102 @@
from typing_extensions import override
import comfy.utils
import node_helpers
from comfy_api.latest import ComfyExtension, io
# fmt: off
BUCKETS_1024 = [
(512, 1792), (512, 1856), (512, 1920), (512, 1984), (512, 2048),
(576, 1600), (576, 1664), (576, 1728), (576, 1792),
(640, 1472), (640, 1536), (640, 1600),
(704, 1344), (704, 1408), (704, 1472),
(768, 1216), (768, 1280), (768, 1344),
(832, 1152), (832, 1216),
(896, 1088), (896, 1152),
(960, 1024), (960, 1088),
(1024, 960), (1024, 1024),
(1088, 896), (1088, 960),
(1152, 832), (1152, 896),
(1216, 768), (1216, 832),
(1280, 768),
(1344, 704), (1344, 768),
(1408, 704),
(1472, 640), (1472, 704),
(1536, 640),
(1600, 576), (1600, 640),
(1664, 576),
(1728, 576),
(1792, 512), (1792, 576),
(1856, 512),
(1920, 512),
(1984, 512),
(2048, 512),
]
# fmt: on
def _find_best_bucket(height: int, width: int) -> tuple[int, int]:
target_ratio = height / width
return min(BUCKETS_1024, key=lambda hw: abs(hw[0] / hw[1] - target_ratio))
def _resize_reference(image):
if image.shape[0] != 1:
raise ValueError("JoyImage reference inputs must contain one image each")
samples = image.movedim(-1, 1)
bucket_h, bucket_w = _find_best_bucket(samples.shape[2], samples.shape[3])
resized = comfy.utils.common_upscale(samples, bucket_w, bucket_h, "bilinear", "center")
return resized.movedim(1, -1)[:, :, :, :3]
def _encode(clip, prompt, vae, images):
resized_images = [_resize_reference(image) for image in images]
conditioning = clip.encode_from_tokens_scheduled(clip.tokenize(prompt, images=resized_images))
if vae is not None and resized_images:
ref_latents = [vae.encode(image) for image in resized_images]
conditioning = node_helpers.conditioning_set_values(
conditioning, {"reference_latents": ref_latents}, append=True,
)
return conditioning
class TextEncodeJoyImageEdit(io.ComfyNode):
@classmethod
def define_schema(cls):
image_template = io.Autogrow.TemplatePrefix(
io.Image.Input("image"),
prefix="image",
min=0,
max=6,
)
return io.Schema(
node_id="TextEncodeJoyImageEdit",
category="model/conditioning/joyimage",
inputs=[
io.Clip.Input("clip"),
io.String.Input("prompt", multiline=True, dynamic_prompts=True),
io.Vae.Input("vae", optional=True),
io.Autogrow.Input("images", template=image_template, optional=True),
],
outputs=[
io.Conditioning.Output(),
],
)
@classmethod
def execute(cls, clip, prompt, vae=None, images: io.Autogrow.Type = None) -> io.NodeOutput:
images = images or {}
return io.NodeOutput(_encode(clip, prompt, vae, list(images.values())))
class JoyImageExtension(ComfyExtension):
@override
async def get_node_list(self) -> list[type[io.ComfyNode]]:
return [
TextEncodeJoyImageEdit,
]
async def comfy_entrypoint() -> JoyImageExtension:
return JoyImageExtension()

26
comfy_extras/nodes_load_3d.py
View File

@ -61,14 +61,10 @@ class Load3D(IO.ComfyNode):
@classmethod
def execute(cls, model_file, image, **kwargs) -> IO.NodeOutput:
image_path = folder_paths.get_annotated_filepath(image['image'])
mask_path = folder_paths.get_annotated_filepath(image['mask'])
normal_path = folder_paths.get_annotated_filepath(image['normal'])
load_image_node = nodes.LoadImage()
output_image, ignore_mask = load_image_node.load_image(image=image_path)
ignore_image, output_mask = load_image_node.load_image(image=mask_path)
normal_image, ignore_mask2 = load_image_node.load_image(image=normal_path)
output_image, ignore_mask = load_image_node.load_image(image=image['image'])
ignore_image, output_mask = load_image_node.load_image(image=image['mask'])
normal_image, ignore_mask2 = load_image_node.load_image(image=image['normal'])
video = None
@ -96,6 +92,7 @@ class Preview3D(IO.ComfyNode):
search_aliases=["view mesh", "3d viewer"],
display_name="Preview 3D & Animation",
category="3d",
description="Preview a 3D model file without saving it to the ComfyUI output directory.",
is_experimental=True,
is_output_node=True,
inputs=[
@ -140,6 +137,7 @@ class Preview3DAdvanced(IO.ComfyNode):
display_name="Preview 3D (Advanced)",
search_aliases=["preview 3d", "3d viewer", "view mesh", "frame 3d", "3d camera output"],
category="3d",
description="Preview a 3D model file without saving it to the ComfyUI output directory.",
is_experimental=True,
is_output_node=True,
inputs=[
@ -176,8 +174,9 @@ class Preview3DAdvanced(IO.ComfyNode):
filename = f"preview3d_advanced_{uuid.uuid4().hex}.{model_3d.format}"
model_3d.save_to(os.path.join(folder_paths.get_temp_directory(), filename))
viewport_state = viewport_state if isinstance(viewport_state, dict) else {}
camera_info_input = kwargs.get("camera_info", None)
camera_info = camera_info_input if camera_info_input is not None else viewport_state['camera_info']
camera_info = camera_info_input if camera_info_input is not None else viewport_state.get('camera_info')
model_3d_info_input = kwargs.get("model_3d_info", None)
model_3d_info = model_3d_info_input if model_3d_info_input is not None else viewport_state.get('model_3d_info', [])
return IO.NodeOutput(
@ -197,6 +196,7 @@ class PreviewGaussianSplat(IO.ComfyNode):
node_id="PreviewGaussianSplat",
display_name="Preview Splat",
category="3d",
description="Preview a gaussian splat 3D file without saving it to the ComfyUI output directory.",
is_experimental=True,
is_output_node=True,
search_aliases=[
@ -244,8 +244,9 @@ class PreviewGaussianSplat(IO.ComfyNode):
filename = f"preview_splat_{uuid.uuid4().hex}.{model_3d.format}"
model_3d.save_to(os.path.join(folder_paths.get_temp_directory(), filename))
viewport_state = viewport_state if isinstance(viewport_state, dict) else {}
camera_info_input = kwargs.get("camera_info", None)
camera_info = camera_info_input if camera_info_input is not None else viewport_state['camera_info']
camera_info = camera_info_input if camera_info_input is not None else viewport_state.get('camera_info')
model_3d_info_input = kwargs.get("model_3d_info", None)
model_3d_info = model_3d_info_input if model_3d_info_input is not None else viewport_state.get('model_3d_info', [])
return IO.NodeOutput(
@ -265,6 +266,7 @@ class PreviewPointCloud(IO.ComfyNode):
node_id="PreviewPointCloud",
display_name="Preview Point Cloud",
category="3d",
description="Preview a point cloud 3D file without saving it to the ComfyUI output directory.",
is_experimental=True,
is_output_node=True,
search_aliases=[
@ -303,8 +305,9 @@ class PreviewPointCloud(IO.ComfyNode):
filename = f"preview_pointcloud_{uuid.uuid4().hex}.{model_3d.format}"
model_3d.save_to(os.path.join(folder_paths.get_temp_directory(), filename))
viewport_state = viewport_state if isinstance(viewport_state, dict) else {}
camera_info_input = kwargs.get("camera_info", None)
camera_info = camera_info_input if camera_info_input is not None else viewport_state['camera_info']
camera_info = camera_info_input if camera_info_input is not None else viewport_state.get('camera_info')
model_3d_info_input = kwargs.get("model_3d_info", None)
model_3d_info = model_3d_info_input if model_3d_info_input is not None else viewport_state.get('model_3d_info', [])
return IO.NodeOutput(
@ -375,8 +378,9 @@ class Load3DAdvanced(IO.ComfyNode):
file_3d = None
if model_file and model_file != "none":
file_3d = Types.File3D(folder_paths.get_annotated_filepath(model_file))
viewport_state = viewport_state if isinstance(viewport_state, dict) else {}
model_3d_info = viewport_state.get('model_3d_info', [])
return IO.NodeOutput(file_3d, model_3d_info, viewport_state['camera_info'], width, height)
return IO.NodeOutput(file_3d, model_3d_info, viewport_state.get('camera_info'), width, height)
class Load3DExtension(ComfyExtension):

5
comfy_extras/nodes_mask.py
View File

@ -419,17 +419,18 @@ class MaskPreview(IO.ComfyNode):
search_aliases=["show mask", "view mask", "inspect mask", "debug mask"],
display_name="Preview Mask",
category="image/mask",
description="Saves the input images to your ComfyUI output directory.",
description="Preview the masks without saving them to the ComfyUI output directory.",
inputs=[
IO.Mask.Input("mask"),
],
hidden=[IO.Hidden.prompt, IO.Hidden.extra_pnginfo],
is_output_node=True,
outputs=[IO.Mask.Output(display_name="mask")]
)
@classmethod
def execute(cls, mask, filename_prefix="ComfyUI") -> IO.NodeOutput:
return IO.NodeOutput(ui=UI.PreviewMask(mask))
return IO.NodeOutput(mask, ui=UI.PreviewMask(mask))
class MaskExtension(ComfyExtension):

1
comfy_extras/nodes_preview_any.py
View File

@ -18,6 +18,7 @@ class PreviewAny():
CATEGORY = "utilities"
SEARCH_ALIASES = ["show output", "inspect", "debug", "print value", "show text"]
DESCRIPTION = "Preview any input value as text."
def main(self, source=None):
torch.set_printoptions(edgeitems=6)

7
comfy_extras/nodes_primitive.py
View File

@ -10,11 +10,10 @@ class String(io.ComfyNode):
return io.Schema(
node_id="PrimitiveString",
search_aliases=["text", "string", "text box", "prompt"],
display_name="Text String (DEPRECATED)",
display_name="Text",
category="utilities/primitive",
inputs=[io.String.Input("value")],
outputs=[io.String.Output()],
is_deprecated=True
outputs=[io.String.Output()]
)
@classmethod
@ -28,7 +27,7 @@ class StringMultiline(io.ComfyNode):
return io.Schema(
node_id="PrimitiveStringMultiline",
search_aliases=["text", "string", "text multiline", "string multiline", "text box", "prompt"],
display_name="Input Text",
display_name="Text (Multiline)",
category="utilities/primitive",
essentials_category="Basics",
inputs=[io.String.Input("value", multiline=True)],

157
comfy_extras/nodes_save_3d.py
View File

@ -868,10 +868,165 @@ class GetMeshInfo(IO.ComfyNode):
return IO.NodeOutput(mesh, info, ui=UI.PreviewText(info))
def _save_file3d_to_output(model_3d: Types.File3D, filename_prefix: str) -> str:
full_output_folder, filename, counter, subfolder, filename_prefix = folder_paths.get_save_image_path(
filename_prefix, folder_paths.get_output_directory()
)
ext = model_3d.format or "glb"
saved_filename = f"{filename}_{counter:05}.{ext}"
model_3d.save_to(os.path.join(full_output_folder, saved_filename))
return f"{subfolder}/{saved_filename}" if subfolder else saved_filename
def execute_save_3d_advanced(model_3d, viewport_state, width, height, filename_prefix, kwargs) -> IO.NodeOutput:
model_file = _save_file3d_to_output(model_3d, filename_prefix)
viewport_state = viewport_state if isinstance(viewport_state, dict) else {}
camera_info_input = kwargs.get("camera_info", None)
camera_info = camera_info_input if camera_info_input is not None else viewport_state.get('camera_info')
model_3d_info_input = kwargs.get("model_3d_info", None)
model_3d_info = model_3d_info_input if model_3d_info_input is not None else viewport_state.get('model_3d_info', [])
return IO.NodeOutput(
model_3d,
model_3d_info,
camera_info,
width,
height,
ui=UI.PreviewUI3DAdvanced(model_file, camera_info, model_3d_info),
)
class Save3DAdvanced(IO.ComfyNode):
@classmethod
def define_schema(cls):
return IO.Schema(
node_id="Save3DAdvanced",
display_name="Save 3D (Advanced)",
search_aliases=["save 3d", "export 3d model", "save mesh advanced"],
category="3d",
is_experimental=True,
is_output_node=True,
inputs=[
IO.MultiType.Input(
"model_3d",
types=[
IO.File3DGLB,
IO.File3DGLTF,
IO.File3DFBX,
IO.File3DOBJ,
IO.File3DSTL,
IO.File3DUSDZ,
IO.File3DAny,
],
tooltip="3D model file from an upstream 3D node.",
),
IO.String.Input("filename_prefix", default="3d/ComfyUI"),
IO.Load3D.Input("viewport_state"),
IO.Load3DModelInfo.Input("model_3d_info", optional=True, advanced=True),
IO.Load3DCamera.Input("camera_info", optional=True, advanced=True),
IO.Int.Input("width", default=1024, min=1, max=4096, step=1),
IO.Int.Input("height", default=1024, min=1, max=4096, step=1),
],
outputs=[
IO.File3DAny.Output(display_name="model_3d"),
IO.Load3DModelInfo.Output(display_name="model_3d_info"),
IO.Load3DCamera.Output(display_name="camera_info"),
IO.Int.Output(display_name="width"),
IO.Int.Output(display_name="height"),
],
)
@classmethod
def execute(cls, model_3d: Types.File3D, viewport_state, width: int, height: int, filename_prefix: str, **kwargs) -> IO.NodeOutput:
return execute_save_3d_advanced(model_3d, viewport_state, width, height, filename_prefix, kwargs)
class SaveGaussianSplat(IO.ComfyNode):
@classmethod
def define_schema(cls):
return IO.Schema(
node_id="SaveGaussianSplat",
display_name="Save Splat",
search_aliases=["save splat", "save gaussian splat", "export gaussian", "export splat"],
category="3d",
is_experimental=True,
is_output_node=True,
inputs=[
IO.MultiType.Input(
"model_3d",
types=[
IO.File3DSplatAny,
IO.File3DPLY,
IO.File3DSPLAT,
IO.File3DSPZ,
IO.File3DKSPLAT,
],
tooltip="A gaussian splat 3D file.",
),
IO.String.Input("filename_prefix", default="3d/ComfyUI"),
IO.Load3D.Input("viewport_state"),
IO.Load3DModelInfo.Input("model_3d_info", optional=True, advanced=True),
IO.Load3DCamera.Input("camera_info", optional=True, advanced=True),
IO.Int.Input("width", default=1024, min=1, max=4096, step=1),
IO.Int.Input("height", default=1024, min=1, max=4096, step=1),
],
outputs=[
IO.File3DSplatAny.Output(display_name="model_3d"),
IO.Load3DModelInfo.Output(display_name="model_3d_info"),
IO.Load3DCamera.Output(display_name="camera_info"),
IO.Int.Output(display_name="width"),
IO.Int.Output(display_name="height"),
],
)
@classmethod
def execute(cls, model_3d: Types.File3D, viewport_state, width: int, height: int, filename_prefix: str, **kwargs) -> IO.NodeOutput:
return execute_save_3d_advanced(model_3d, viewport_state, width, height, filename_prefix, kwargs)
class SavePointCloud(IO.ComfyNode):
@classmethod
def define_schema(cls):
return IO.Schema(
node_id="SavePointCloud",
display_name="Save Point Cloud",
search_aliases=["save point cloud", "save pointcloud", "export point cloud"],
category="3d",
is_experimental=True,
is_output_node=True,
inputs=[
IO.MultiType.Input(
"model_3d",
types=[
IO.File3DPointCloudAny,
IO.File3DPLY,
],
tooltip="Point cloud file (.ply)",
),
IO.String.Input("filename_prefix", default="3d/ComfyUI"),
IO.Load3D.Input("viewport_state"),
IO.Load3DModelInfo.Input("model_3d_info", optional=True, advanced=True),
IO.Load3DCamera.Input("camera_info", optional=True, advanced=True),
IO.Int.Input("width", default=1024, min=1, max=4096, step=1),
IO.Int.Input("height", default=1024, min=1, max=4096, step=1),
],
outputs=[
IO.File3DPointCloudAny.Output(display_name="model_3d"),
IO.Load3DModelInfo.Output(display_name="model_3d_info"),
IO.Load3DCamera.Output(display_name="camera_info"),
IO.Int.Output(display_name="width"),
IO.Int.Output(display_name="height"),
],
)
@classmethod
def execute(cls, model_3d: Types.File3D, viewport_state, width: int, height: int, filename_prefix: str, **kwargs) -> IO.NodeOutput:
return execute_save_3d_advanced(model_3d, viewport_state, width, height, filename_prefix, kwargs)
class Save3DExtension(ComfyExtension):
@override
async def get_node_list(self) -> list[type[IO.ComfyNode]]:
return [SaveGLB, MeshToFile3D, RotateMesh, MergeMeshes, GetMeshInfo]
return [SaveGLB, MeshToFile3D, RotateMesh, MergeMeshes, GetMeshInfo, Save3DAdvanced, SaveGaussianSplat, SavePointCloud]
async def comfy_entrypoint() -> Save3DExtension:

614
comfy_extras/nodes_seedvr.py Normal file
View File

@ -0,0 +1,614 @@
import logging
from typing_extensions import override
from comfy_api.latest import ComfyExtension, io
import torch
import comfy.model_management
from comfy.ldm.seedvr.color_fix import (
adain_color_transfer,
lab_color_transfer,
wavelet_color_transfer,
)
from comfy.ldm.seedvr.constants import (
BYTEDANCE_VAE_SPATIAL_DOWNSAMPLE,
SEEDVR2_ADAIN_SCALE_MULTIPLIER,
SEEDVR2_CHUNK_GIB_PER_MPX_FRAME,
SEEDVR2_CHUNK_RESERVED_GIB,
SEEDVR2_CHUNK_SIGMA_GIB,
SEEDVR2_CHUNK_SIGMA_K,
SEEDVR2_COLOR_MEM_HEADROOM,
SEEDVR2_DTYPE_BYTES_FLOOR,
SEEDVR2_LAB_SCALE_MULTIPLIER,
SEEDVR2_LATENT_CHANNELS,
SEEDVR2_OOM_BACKOFF_DIVISOR,
SEEDVR2_WAVELET_SCALE_MULTIPLIER,
)
from torchvision.transforms import functional as TVF
from torchvision.transforms.functional import InterpolationMode
_SEEDVR2_INVALID_MODEL_MSG_PREFIX = "SeedVR2Conditioning: model object does not match expected SeedVR2 structure"
_ATTR_MISSING = object()
def _resolve_seedvr2_diffusion_model(model):
inner = getattr(model, "model", _ATTR_MISSING)
if inner is _ATTR_MISSING:
raise RuntimeError(
f"{_SEEDVR2_INVALID_MODEL_MSG_PREFIX}: input has no 'model' attribute "
f"(got type {type(model).__name__})."
)
if inner is None:
raise RuntimeError(
f"{_SEEDVR2_INVALID_MODEL_MSG_PREFIX}: input.model is None "
f"(input type {type(model).__name__})."
)
diffusion_model = getattr(inner, "diffusion_model", _ATTR_MISSING)
if diffusion_model is _ATTR_MISSING:
raise RuntimeError(
f"{_SEEDVR2_INVALID_MODEL_MSG_PREFIX}: 'model.model' has no "
f"'diffusion_model' attribute (got type {type(inner).__name__})."
)
if diffusion_model is None:
raise RuntimeError(
f"{_SEEDVR2_INVALID_MODEL_MSG_PREFIX}: 'model.model.diffusion_model' "
f"is None (model.model type {type(inner).__name__})."
)
return diffusion_model
def div_pad(image, factor):
height_factor, width_factor = factor
height, width = image.shape[-2:]
pad_height = (height_factor - (height % height_factor)) % height_factor
pad_width = (width_factor - (width % width_factor)) % width_factor
if pad_height == 0 and pad_width == 0:
return image
padding = (0, pad_width, 0, pad_height)
return torch.nn.functional.pad(image, padding, mode='constant', value=0.0)
def cut_videos(videos):
t = videos.size(1)
if t < 1:
raise ValueError("SeedVR2Preprocess expected at least one frame.")
if t == 1:
return videos
if t <= 4:
padding = videos[:, -1:].repeat(1, 4 - t + 1, 1, 1, 1)
return torch.cat([videos, padding], dim=1)
if (t - 1) % 4 == 0:
return videos
padding = videos[:, -1:].repeat(1, 4 - ((t - 1) % 4), 1, 1, 1)
videos = torch.cat([videos, padding], dim=1)
if (videos.size(1) - 1) % 4 != 0:
raise ValueError(f"SeedVR2Preprocess failed to pad video length to 4n+1; got {videos.size(1)} frames.")
return videos
def _seedvr2_input_shorter_edge(images, node_name):
if images.dim() == 4:
return min(images.shape[1], images.shape[2])
if images.dim() == 5:
return min(images.shape[2], images.shape[3])
raise ValueError(
f"{node_name}: expected 4-D or 5-D IMAGE tensor, "
f"got shape {tuple(images.shape)}"
)
def _seedvr2_pad(images, upscaled_shorter_edge, node_name):
if upscaled_shorter_edge < 2:
raise ValueError(
f"{node_name}: input shorter edge must be at least 2 pixels; "
f"got {upscaled_shorter_edge}."
)
if images.shape[-1] > 3:
images = images[..., :3]
if images.dim() == 4:
# Comfy video components arrive as a 4-D IMAGE frame sequence:
# (frames, H, W, C). SeedVR2 consumes that as one video.
images = images.unsqueeze(0)
elif images.dim() != 5:
raise ValueError(
f"{node_name}: expected 4-D or 5-D IMAGE tensor, "
f"got shape {tuple(images.shape)}"
)
images = images.permute(0, 1, 4, 2, 3)
b, t, c, h, w = images.shape
images = images.reshape(b * t, c, h, w)
images = torch.clamp(images, 0.0, 1.0)
images = div_pad(images, (16, 16))
_, _, new_h, new_w = images.shape
images = images.reshape(b, t, c, new_h, new_w)
images = cut_videos(images)
images_bthwc = images.permute(0, 1, 3, 4, 2).contiguous()
return io.NodeOutput(images_bthwc)
class SeedVR2Preprocess(io.ComfyNode):
@classmethod
def define_schema(cls):
return io.Schema(
node_id="SeedVR2Preprocess",
display_name="Pre-Process SeedVR2 Input",
category="image/pre-processors",
description="Pad a resized image for SeedVR2 model. Alpha channel is dropped. The node Post-Process SeedVR2 Output re-applies it from the original resized image.",
search_aliases=["seedvr2", "upscale", "video upscale", "pad", "preprocess"],
inputs=[
io.Image.Input("resized_images", tooltip="The resized image to process."),
],
outputs=[
io.Image.Output("images", tooltip="The padded image for VAE encoding."),
]
)
@classmethod
def execute(cls, resized_images):
upscaled_shorter_edge = _seedvr2_input_shorter_edge(resized_images, "SeedVR2Preprocess")
return _seedvr2_pad(
resized_images, upscaled_shorter_edge, "SeedVR2Preprocess",
)
class SeedVR2PostProcessing(io.ComfyNode):
@classmethod
def define_schema(cls):
return io.Schema(
node_id="SeedVR2PostProcessing",
display_name="Post-Process SeedVR2 Output",
category="image/post-processors",
description="Align the generated image with the original resized image and apply color correction.",
search_aliases=["seedvr2", "upscale", "color correction", "color match", "postprocess"],
inputs=[
io.Image.Input("images", tooltip="The generated image to process."),
io.Image.Input("original_resized_images", tooltip="The original resized image before pre-processing, used as reference."),
io.Combo.Input("color_correction_method", options=["lab", "wavelet", "adain", "none"], default="lab", tooltip="Method to match the generated image colors to the original image. lab: transfer color in CIELAB space, preserving detail (most faithful). wavelet: transfer low-frequency color, keeping upscaled high-frequency detail. adain: match per-channel mean/std (fastest, global tint). none: skip color transfer (geometry alignment only)."),
],
outputs=[io.Image.Output(display_name="images", tooltip="The aligned, color-corrected image.")],
)
@classmethod
def execute(cls, images, original_resized_images, color_correction_method):
alpha_input = None
if original_resized_images.shape[-1] == 4:
alpha_input = original_resized_images[..., 3:4]
original_resized_images = original_resized_images[..., :3]
decoded_5d, decoded_was_4d = cls._as_bthwc(images)
reference_full, _ = cls._as_bthwc(original_resized_images)
decoded_5d = cls._restore_reference_batch_time(decoded_5d, reference_full)
b = min(decoded_5d.shape[0], reference_full.shape[0])
t = min(decoded_5d.shape[1], reference_full.shape[1])
reference_h = reference_full.shape[2]
reference_w = reference_full.shape[3]
decoded_5d = decoded_5d[:b, :t, :, :, :]
target_h = min(decoded_5d.shape[2], reference_h)
target_w = min(decoded_5d.shape[3], reference_w)
decoded_5d = decoded_5d[:, :, :target_h, :target_w, :]
if color_correction_method in ("lab", "wavelet", "adain"):
reference_5d = reference_full[:b, :t, :, :, :]
reference_5d = cls._resize_reference(reference_5d, target_h, target_w)
output_device = decoded_5d.device
decoded_raw = cls._to_seedvr2_raw(decoded_5d)
reference_raw = cls._to_seedvr2_raw(reference_5d)
decoded_flat = decoded_raw.permute(0, 1, 4, 2, 3).reshape(b * t, decoded_raw.shape[4], target_h, target_w)
reference_flat = reference_raw.permute(0, 1, 4, 2, 3).reshape(b * t, reference_raw.shape[4], target_h, target_w)
output = cls._color_transfer_chunked(
decoded_flat, reference_flat, output_device, color_correction_method,
)
output = output.reshape(b, t, output.shape[1], output.shape[2], output.shape[3]).permute(0, 1, 3, 4, 2)
output = output.add(1.0).div(2.0).clamp(0.0, 1.0)
elif color_correction_method == "none":
output = decoded_5d
else:
raise ValueError(f"SeedVR2PostProcessing: unknown color_correction_method {color_correction_method!r}")
if alpha_input is not None:
alpha_5d, _ = cls._as_bthwc(alpha_input)
alpha_5d = alpha_5d[:output.shape[0], :output.shape[1], :output.shape[2], :output.shape[3], :]
output = torch.cat([output, alpha_5d.to(dtype=output.dtype, device=output.device)], dim=-1)
h2 = output.shape[-3] - (output.shape[-3] % 2)
w2 = output.shape[-2] - (output.shape[-2] % 2)
output = output[:, :, :h2, :w2, :]
if decoded_was_4d:
output = output.reshape(-1, output.shape[-3], output.shape[-2], output.shape[-1])
return io.NodeOutput(output)
@staticmethod
def _as_bthwc(images):
if images.ndim == 4:
return images.unsqueeze(0), True
if images.ndim == 5:
return images, False
raise ValueError(
f"SeedVR2PostProcessing: expected 4-D or 5-D IMAGE tensor, got shape {tuple(images.shape)}"
)
@staticmethod
def _restore_reference_batch_time(decoded, reference):
if decoded.shape[0] != 1:
return decoded
ref_b, ref_t = reference.shape[:2]
if ref_b < 1 or decoded.shape[1] % ref_b != 0:
return decoded
decoded_t = decoded.shape[1] // ref_b
if decoded_t < ref_t:
return decoded
return decoded.reshape(ref_b, decoded_t, decoded.shape[2], decoded.shape[3], decoded.shape[4])
@staticmethod
def _to_seedvr2_raw(images):
return images.mul(2.0).sub(1.0)
@staticmethod
def _color_transfer_on_vae_device(decoded_flat, reference_flat, output_device, transfer_fn):
color_device = comfy.model_management.vae_device()
decoded_flat = decoded_flat.to(device=color_device)
reference_flat = reference_flat.to(device=color_device)
output = transfer_fn(decoded_flat, reference_flat)
return output.to(device=output_device)
@staticmethod
def _lab_color_transfer_on_vae_device(decoded_flat, reference_flat, output_device):
color_device = comfy.model_management.vae_device()
result = None
for start in range(decoded_flat.shape[0]):
decoded_frame = decoded_flat[start:start + 1].to(device=color_device).clone()
reference_frame = reference_flat[start:start + 1].to(device=color_device).clone()
output = lab_color_transfer(decoded_frame, reference_frame).to(device=output_device)
if result is None:
result = torch.empty(
(decoded_flat.shape[0],) + tuple(output.shape[1:]),
device=output_device,
dtype=output.dtype,
)
result[start:start + 1].copy_(output)
if result is None:
raise ValueError("SeedVR2PostProcessing: LAB color correction requires at least one frame.")
return result
@classmethod
def _color_transfer_chunked(cls, decoded_flat, reference_flat, output_device, color_correction_method):
chunk_size = cls._estimate_color_correction_chunk_size(decoded_flat, color_correction_method)
while True:
try:
return cls._run_color_transfer_chunks(
decoded_flat, reference_flat, output_device, color_correction_method, chunk_size,
)
except Exception as e:
comfy.model_management.raise_non_oom(e)
if chunk_size <= 1:
raise RuntimeError(
"SeedVR2PostProcessing: color correction OOM at one frame; "
f"color_correction_method={color_correction_method}, shape={tuple(decoded_flat.shape)}."
) from e
chunk_size = max(1, chunk_size // SEEDVR2_OOM_BACKOFF_DIVISOR)
@classmethod
def _run_color_transfer_chunks(cls, decoded_flat, reference_flat, output_device, color_correction_method, chunk_size):
result = None
for start in range(0, decoded_flat.shape[0], chunk_size):
end = min(start + chunk_size, decoded_flat.shape[0])
decoded_chunk = decoded_flat[start:end]
reference_chunk = reference_flat[start:end]
if color_correction_method == "lab":
output = cls._lab_color_transfer_on_vae_device(decoded_chunk, reference_chunk, output_device)
elif color_correction_method == "wavelet":
output = cls._color_transfer_on_vae_device(
decoded_chunk, reference_chunk, output_device, wavelet_color_transfer,
)
else:
output = cls._color_transfer_on_vae_device(
decoded_chunk, reference_chunk, output_device, adain_color_transfer,
)
if result is None:
result = torch.empty(
(decoded_flat.shape[0],) + tuple(output.shape[1:]),
device=output_device,
dtype=output.dtype,
)
result[start:end].copy_(output)
if result is None:
raise ValueError("SeedVR2PostProcessing: color correction requires at least one frame.")
return result
@classmethod
def _estimate_color_correction_chunk_size(cls, decoded_flat, color_correction_method):
multiplier = cls._color_correction_memory_multiplier(color_correction_method)
frames = decoded_flat.shape[0]
_, channels, height, width = decoded_flat.shape
dtype_bytes = max(decoded_flat.element_size(), SEEDVR2_DTYPE_BYTES_FLOOR)
bytes_per_frame = height * width * channels * dtype_bytes * multiplier
if bytes_per_frame <= 0:
return frames
color_device = comfy.model_management.vae_device()
free_memory = comfy.model_management.get_free_memory(color_device)
chunk_size = int((free_memory * SEEDVR2_COLOR_MEM_HEADROOM) // bytes_per_frame)
return max(1, min(frames, chunk_size))
@staticmethod
def _color_correction_memory_multiplier(color_correction_method):
if color_correction_method == "lab":
return SEEDVR2_LAB_SCALE_MULTIPLIER
if color_correction_method == "wavelet":
return SEEDVR2_WAVELET_SCALE_MULTIPLIER
if color_correction_method == "adain":
return SEEDVR2_ADAIN_SCALE_MULTIPLIER
raise ValueError(f"SeedVR2PostProcessing: unknown color_correction_method {color_correction_method!r}")
@staticmethod
def _resize_reference(reference, height, width):
if reference.shape[2] == height and reference.shape[3] == width:
return reference
b, t = reference.shape[:2]
reference_flat = reference.permute(0, 1, 4, 2, 3).reshape(b * t, reference.shape[4], reference.shape[2], reference.shape[3])
resized = TVF.resize(
reference_flat,
size=(height, width),
interpolation=InterpolationMode.BICUBIC,
antialias=not (isinstance(reference_flat, torch.Tensor) and reference_flat.device.type == "mps"),
)
return resized.reshape(b, t, resized.shape[1], height, width).permute(0, 1, 3, 4, 2)
class SeedVR2Conditioning(io.ComfyNode):
@classmethod
def define_schema(cls):
return io.Schema(
node_id="SeedVR2Conditioning",
display_name="Apply SeedVR2 Conditioning",
category="model/conditioning",
description="Build SeedVR2 positive/negative conditioning from a VAE latent.",
search_aliases=["seedvr2", "upscale", "conditioning"],
inputs=[
io.Model.Input("model", tooltip="The SeedVR2 model."),
io.Latent.Input("vae_conditioning", display_name="latent"),
],
outputs=[
io.Conditioning.Output(display_name="positive", tooltip="The positive conditioning for sampling."),
io.Conditioning.Output(display_name="negative", tooltip="The negative conditioning for sampling."),
],
)
@classmethod
def execute(cls, model, vae_conditioning) -> io.NodeOutput:
vae_conditioning = vae_conditioning["samples"]
if vae_conditioning.ndim != 5:
raise ValueError(
"SeedVR2Conditioning expects a 5-D VAE latent in Comfy "
f"channel-first layout; got shape {tuple(vae_conditioning.shape)}."
)
if vae_conditioning.shape[1] != SEEDVR2_LATENT_CHANNELS:
if vae_conditioning.shape[-1] == SEEDVR2_LATENT_CHANNELS:
raise ValueError(
"SeedVR2Conditioning expects SeedVR2 VAE latents in Comfy "
f"channel-first layout (B, {SEEDVR2_LATENT_CHANNELS}, T, H, W); "
f"got channel-last shape {tuple(vae_conditioning.shape)}."
)
raise ValueError(
"SeedVR2Conditioning expects SeedVR2 VAE latents with "
f"{SEEDVR2_LATENT_CHANNELS} channels; got shape {tuple(vae_conditioning.shape)}."
)
vae_conditioning = vae_conditioning.movedim(1, -1).contiguous()
model = _resolve_seedvr2_diffusion_model(model)
pos_cond = model.positive_conditioning
neg_cond = model.negative_conditioning
mask = vae_conditioning.new_ones(vae_conditioning.shape[:-1] + (1,))
condition = torch.cat((vae_conditioning, mask), dim=-1)
condition = condition.movedim(-1, 1)
negative = [[neg_cond.unsqueeze(0), {"condition": condition}]]
positive = [[pos_cond.unsqueeze(0), {"condition": condition}]]
return io.NodeOutput(positive, negative)
def _seedvr2_chunk_crossfade_weights(overlap, device, dtype):
"""Descending previous-chunk weights across the overlap (next chunk gets ``1 - w``): a Hann fade over the middle third, flat shoulders on the outer thirds."""
ramp = torch.linspace(0.0, 1.0, steps=overlap, device=device, dtype=dtype)
ramp = ((ramp - 1.0 / 3.0) / (1.0 / 3.0)).clamp(0.0, 1.0)
return 0.5 + 0.5 * torch.cos(torch.pi * ramp)
class SeedVR2TemporalChunk(io.ComfyNode):
@classmethod
def define_schema(cls):
return io.Schema(
node_id="SeedVR2TemporalChunk",
display_name="Split SeedVR2 Latent",
category="model/latent/batch",
description="Split a SeedVR2 video latent into overlapping temporal chunks small enough to sample one at a time within VRAM, wiring latents outputs to both Apply SeedVR2 Conditioning and the sampler latent input before recombining with Merge SeedVR2 Latents.",
search_aliases=["seedvr2", "split", "chunk", "temporal", "video upscale", "rebatch"],
inputs=[
io.Latent.Input("latent", tooltip="The VAE-encoded SeedVR2 latent to split."),
io.Int.Input("temporal_overlap", default=0, min=0, max=16384,
tooltip="Latent frames shared between adjacent chunks and crossfaded at merge; 0 = no overlap."),
io.DynamicCombo.Input("chunking_mode",
tooltip="manual = use frames_per_chunk exactly; auto = predict the largest chunk that fits free VRAM.",
options=[
io.DynamicCombo.Option("auto", []),
io.DynamicCombo.Option("manual", [
io.Int.Input("frames_per_chunk", default=21, min=1, max=16384, step=4,
tooltip="Pixel frames per temporal chunk (4n+1: 1, 5, 9, 13, ...)."),
]),
]),
],
outputs=[
io.Latent.Output(display_name="latents", is_output_list=True,
tooltip="The temporal chunks in sequence order."),
io.Int.Output(display_name="temporal_overlap",
tooltip="The effective latent-frame overlap between adjacent chunks, for Merge SeedVR2 Latents."),
],
)
@classmethod
def execute(cls, latent, temporal_overlap, chunking_mode) -> io.NodeOutput:
samples = latent["samples"]
if samples.ndim != 5:
raise ValueError(
f"SeedVR2TemporalChunk: expected a 5-D video latent (B, C, T, H, W); "
f"got shape {tuple(samples.shape)}."
)
if samples.shape[1] != SEEDVR2_LATENT_CHANNELS:
raise ValueError(
f"SeedVR2TemporalChunk: expected {SEEDVR2_LATENT_CHANNELS} latent channels; "
f"got shape {tuple(samples.shape)}."
)
if temporal_overlap < 0:
raise ValueError(
f"SeedVR2TemporalChunk: temporal_overlap must be >= 0; got {temporal_overlap}."
)
mode = chunking_mode["chunking_mode"]
if mode not in ("auto", "manual"):
raise ValueError(
f"SeedVR2TemporalChunk: chunking_mode must be 'auto' or 'manual'; "
f"got {mode!r}."
)
t_latent = samples.shape[2]
t_pixel = 4 * (t_latent - 1) + 1
if mode == "auto":
free_gb = comfy.model_management.get_free_memory(
comfy.model_management.get_torch_device()) / (1024 ** 3)
mpx_per_frame = (samples.shape[0] * samples.shape[3] * samples.shape[4]) * (BYTEDANCE_VAE_SPATIAL_DOWNSAMPLE ** 2) / 1e6
budget_gb = free_gb - SEEDVR2_CHUNK_RESERVED_GIB - SEEDVR2_CHUNK_SIGMA_K * SEEDVR2_CHUNK_SIGMA_GIB
chunk_latent_max = max(1, int(budget_gb / (SEEDVR2_CHUNK_GIB_PER_MPX_FRAME * mpx_per_frame)))
frames_per_chunk = min(4 * (chunk_latent_max - 1) + 1, t_pixel)
logging.info(
"SeedVR2TemporalChunk auto: free=%.2fGiB, %.2fMpx -> frames_per_chunk=%d (t_pixel=%d).",
free_gb, mpx_per_frame, frames_per_chunk, t_pixel,
)
else:
frames_per_chunk = chunking_mode["frames_per_chunk"]
if frames_per_chunk < 1 or (frames_per_chunk - 1) % 4 != 0:
raise ValueError(
f"SeedVR2TemporalChunk: frames_per_chunk must be a 4n+1 pixel-frame count "
f"(1, 5, 9, 13, 17, 21, ...); got {frames_per_chunk}."
)
if t_pixel <= frames_per_chunk:
return io.NodeOutput([latent], 0)
chunk_latent = (frames_per_chunk - 1) // 4 + 1
temporal_overlap = min(temporal_overlap, chunk_latent - 1)
step = chunk_latent - temporal_overlap
chunks = []
for start in range(0, t_latent, step):
end = min(start + chunk_latent, t_latent)
chunk = latent.copy()
chunk["samples"] = samples[:, :, start:end].contiguous()
chunks.append(chunk)
if end >= t_latent:
break
return io.NodeOutput(chunks, temporal_overlap)
class SeedVR2TemporalMerge(io.ComfyNode):
@classmethod
def define_schema(cls):
return io.Schema(
node_id="SeedVR2TemporalMerge",
display_name="Merge SeedVR2 Latents",
category="model/latent/batch",
is_input_list=True,
description="Recombine sampled SeedVR2 latent temporal chunks into one latent, crossfading each overlap with a Hann window sized by the temporal_overlap wired from Split SeedVR2 Latent.",
search_aliases=["seedvr2", "merge", "temporal", "hann", "crossfade"],
inputs=[
io.Latent.Input("latents", tooltip="The sampled temporal chunks in sequence order."),
io.Int.Input("temporal_overlap", default=0, min=0, max=16384, force_input=True,
tooltip="The temporal_overlap output of Split SeedVR2 Latent. 0 = plain concatenation."),
],
outputs=[
io.Latent.Output(display_name="latent", tooltip="The recombined full-length latent."),
],
)
@classmethod
def execute(cls, latents, temporal_overlap) -> io.NodeOutput:
temporal_overlap = temporal_overlap[0]
if temporal_overlap < 0:
raise ValueError(
f"SeedVR2TemporalMerge: temporal_overlap must be >= 0; got {temporal_overlap}."
)
chunks = [entry["samples"] for entry in latents]
first = chunks[0]
if first.ndim != 5:
raise ValueError(
f"SeedVR2TemporalMerge: expected 5-D video latents (B, C, T, H, W); "
f"chunk 0 has shape {tuple(first.shape)}."
)
for i, chunk in enumerate(chunks[1:], start=1):
if chunk.shape[:2] != first.shape[:2] or chunk.shape[3:] != first.shape[3:]:
raise ValueError(
f"SeedVR2TemporalMerge: chunk {i} shape {tuple(chunk.shape)} does not "
f"match chunk 0 shape {tuple(first.shape)} outside the temporal axis."
)
if i < len(chunks) - 1 and chunk.shape[2] != first.shape[2]:
raise ValueError(
f"SeedVR2TemporalMerge: chunk {i} has {chunk.shape[2]} latent frames but "
f"chunk 0 has {first.shape[2]}; only the final chunk may be shorter."
)
out = latents[0].copy()
out.pop("noise_mask", None)
if len(chunks) == 1:
out["samples"] = first
return io.NodeOutput(out)
if temporal_overlap == 0:
out["samples"] = torch.cat(chunks, dim=2)
return io.NodeOutput(out)
chunk_latent = first.shape[2]
step = chunk_latent - min(temporal_overlap, chunk_latent - 1)
t_total = step * (len(chunks) - 1) + chunks[-1].shape[2]
b, c, _, h, w = first.shape
merged = torch.empty((b, c, t_total, h, w), device=first.device, dtype=first.dtype)
merged[:, :, :chunk_latent] = first
filled = chunk_latent
for i, chunk in enumerate(chunks[1:], start=1):
start = i * step
end = start + chunk.shape[2]
# Crossfade width is bounded by the previous fill frontier and by a runt
# final chunk shorter than the configured overlap.
fade = min(filled - start, chunk.shape[2])
if fade > 0:
w_prev = _seedvr2_chunk_crossfade_weights(
fade, chunk.device, chunk.dtype).view(1, 1, fade, 1, 1)
merged[:, :, start:start + fade] = (
merged[:, :, start:start + fade] * w_prev + chunk[:, :, :fade] * (1.0 - w_prev)
)
merged[:, :, start + fade:end] = chunk[:, :, fade:]
else:
merged[:, :, start:end] = chunk
filled = end
out["samples"] = merged
return io.NodeOutput(out)
class SeedVRExtension(ComfyExtension):
@override
async def get_node_list(self) -> list[type[io.ComfyNode]]:
return [
SeedVR2Conditioning,
SeedVR2Preprocess,
SeedVR2PostProcessing,
SeedVR2TemporalChunk,
SeedVR2TemporalMerge,
]
async def comfy_entrypoint() -> SeedVRExtension:
return SeedVRExtension()

71
comfy_extras/nodes_text.py Normal file
View File

@ -0,0 +1,71 @@
import os
import json
from typing_extensions import override
from comfy_api.latest import io, ComfyExtension, ui
import folder_paths
class SaveTextNode(io.ComfyNode):
"""Save text content to .txt, .md, or .json."""
@classmethod
def define_schema(cls):
return io.Schema(
node_id="SaveText",
search_aliases=["save text", "write text", "export text"],
display_name="Save Text",
category="text",
description="Save text content to a file in the output directory.",
inputs=[
io.String.Input("text", force_input=True),
io.String.Input("filename_prefix", default="ComfyUI"),
io.Combo.Input("format", options=["txt", "md", "json"], default="txt"),
],
outputs=[io.String.Output(display_name="text")],
is_output_node=True,
)
@classmethod
def execute(cls, text, filename_prefix, format):
full_output_folder, filename, counter, subfolder, filename_prefix = folder_paths.get_save_image_path(
filename_prefix,
folder_paths.get_output_directory(),
1,
1,
)
file = f"{filename}_{counter:05}.{format}"
filepath = os.path.join(full_output_folder, file)
if format == "json":
# tries to pretty print otherwise saves normally
try:
data = json.loads(text)
with open(filepath, "w", encoding="utf-8") as f:
json.dump(data, f, indent=2, ensure_ascii=False)
except json.JSONDecodeError:
with open(filepath, "w", encoding="utf-8") as f:
f.write(text)
else:
with open(filepath, "w", encoding="utf-8") as f:
f.write(text)
return io.NodeOutput(
text,
ui={
"text": (text,),
"files": [
ui.SavedResult(file, subfolder, io.FolderType.output)
]
}
)
class TextExtension(ComfyExtension):
@override
async def get_node_list(self) -> list[type[io.ComfyNode]]:
return [
SaveTextNode
]
async def comfy_entrypoint() -> TextExtension:
return TextExtension()

150
comfy_extras/nodes_text_overlay.py Normal file
View File

@ -0,0 +1,150 @@
import numpy as np
import torch
from PIL import Image as PILImage, ImageColor, ImageDraw, ImageFont
from typing_extensions import override
from comfy_api.latest import ComfyExtension, IO
class TextOverlay(IO.ComfyNode):
@classmethod
def define_schema(cls):
return IO.Schema(
node_id="TextOverlay",
display_name="Draw Text Overlay",
category="text",
description="Draw text overlay on an image or batch of images.",
search_aliases=["text", "label", "caption", "subtitle", "watermark", "title", "addlabel", "overlay"],
inputs=[
IO.Image.Input("images"),
IO.String.Input("text", multiline=True, default=""),
IO.Float.Input("font_size", default=5.0, min=0.5, max=50.0, step=0.5, tooltip="Font size as a percentage of the image height."),
IO.Color.Input("color", default="#ffffff", tooltip="Color of the text."),
IO.Combo.Input("position", options=["top", "bottom"], default="top"),
IO.Combo.Input("align", options=["left", "center", "right"], default="left"),
IO.Boolean.Input("outline", default=True, tooltip="Draw a black outline around the text."),
],
outputs=[IO.Image.Output(display_name="images")],
)
@classmethod
def execute(cls, images, text, font_size, color, position, align, outline) -> IO.NodeOutput:
if text.strip() == "":
return IO.NodeOutput(images)
text = text.replace("\\n", "\n").replace("\\t", "\t")
text_rgba = cls.parse_color_to_rgba(color)
outline_rgba = (0, 0, 0, 255) if outline else (0, 0, 0, 0)
# Render the overlay once and composite it across all frames in the batch
height = images.shape[1]
width = images.shape[2]
overlay_rgb, overlay_alpha = cls.render_overlay_text(width, height, text, position, align, font_size, text_rgba, outline_rgba)
overlay_rgb = overlay_rgb.to(device=images.device, dtype=images.dtype)
overlay_alpha = overlay_alpha.to(device=images.device, dtype=images.dtype)
result = images * (1.0 - overlay_alpha) + overlay_rgb * overlay_alpha
return IO.NodeOutput(result)
@staticmethod
def parse_color_to_rgba(color_string):
parsed = ImageColor.getrgb(color_string)
if len(parsed) == 3:
return (*parsed, 255)
return parsed
@classmethod
def render_overlay_text(cls, width, height, text, position, align, font_size, text_rgba, outline_rgba):
line_spacing = 1.2
margin_percent = 1.0
min_font_percent = 2.0
min_font_pixels = 10
outline_thickness_factor = 0.04
# Draw onto a transparent layer so the result can be alpha-composited over any frame.
layer = PILImage.new("RGBA", (width, height), (0, 0, 0, 0))
draw = ImageDraw.Draw(layer)
margin = int(round(margin_percent / 100.0 * min(width, height)))
max_width = max(1, width - 2 * margin)
max_height = max(1, height - 2 * margin)
# Font scales with resolution, then shrinks to fit the height.
size = max(1, int(round(font_size / 100.0 * height)))
floor = min(size, max(min_font_pixels, int(round(min_font_percent / 100.0 * height))))
while True:
font = ImageFont.load_default(size=size)
stroke = max(1, int(round(size * outline_thickness_factor))) if outline_rgba[3] > 0 else 0
block = "\n".join(cls.wrap_text(text, font, max_width))
# convert line spacing to pixel spacing
single = draw.textbbox((0, 0), "Ay", font=font, stroke_width=stroke)
double = draw.multiline_textbbox((0, 0), "Ay\nAy", font=font, spacing=0, stroke_width=stroke)
natural_advance = (double[3] - double[1]) - (single[3] - single[1])
pixel_spacing = int(round(size * line_spacing - natural_advance))
box = draw.multiline_textbbox((0, 0), block, font=font, spacing=pixel_spacing, stroke_width=stroke)
block_height = box[3] - box[1]
if block_height <= max_height or size <= floor:
break
size = max(floor, int(size * 0.9))
anchor_h, x = {"left": ("l", margin), "center": ("m", width / 2), "right": ("r", width - margin)}[align]
# Offset y so the rendered text sits flush against the margin
if position == "bottom":
y = height - margin - box[3]
else:
y = margin - box[1]
draw.multiline_text((x, y), block, font=font, fill=text_rgba, anchor=anchor_h + "a",
align=align, spacing=pixel_spacing, stroke_width=stroke, stroke_fill=outline_rgba)
overlay = np.array(layer).astype(np.float32) / 255.0
overlay_rgb = torch.from_numpy(overlay[:, :, :3])
overlay_alpha = torch.from_numpy(overlay[:, :, 3:4])
return overlay_rgb, overlay_alpha
@staticmethod
def wrap_text(text, font, max_width):
lines = []
for raw_line in text.split("\n"):
words = raw_line.split()
if not words:
lines.append("")
continue
current = ""
# Break the line into words and split words that are too long
for word in words:
while font.getlength(word) > max_width and len(word) > 1:
cut = 1
while cut < len(word) and font.getlength(word[:cut + 1]) <= max_width:
cut += 1
if current:
lines.append(current)
current = ""
lines.append(word[:cut])
word = word[cut:]
candidate = word if not current else current + " " + word
if not current or font.getlength(candidate) <= max_width:
current = candidate
else:
lines.append(current)
current = word
if current:
lines.append(current)
return lines
class TextOverlayExtension(ComfyExtension):
@override
async def get_node_list(self) -> list[type[IO.ComfyNode]]:
return [TextOverlay]
async def comfy_entrypoint() -> TextOverlayExtension:
return TextOverlayExtension()

2
comfy_extras/nodes_video.py
View File

@ -81,7 +81,7 @@ class SaveVideo(io.ComfyNode):
display_name="Save Video",
category="video",
essentials_category="Basics",
description="Saves the input images to your ComfyUI output directory.",
description="Saves the input videos to your ComfyUI output directory.",
inputs=[
io.Video.Input("video", tooltip="The video to save."),
io.String.Input("filename_prefix", default="video/ComfyUI", tooltip="The prefix for the file to save. This may include formatting information such as %date:yyyy-MM-dd% or %Empty Latent Image.width% to include values from nodes."),

2
comfyui_version.py
View File

@ -1,3 +1,3 @@
# This file is automatically generated by the build process when version is
# updated in pyproject.toml.
__version__ = "0.27.0"
__version__ = "0.28.0"

21
execution.py
View File

@ -29,6 +29,7 @@ from comfy_execution.caching import (
HierarchicalCache,
LRUCache,
RAMPressureCache,
RAM_CACHE_LARGE_INTERMEDIATE,
)
from comfy_execution.graph import (
DynamicPrompt,
@ -425,12 +426,12 @@ def _is_intermediate_output(dynprompt, node_id):
def _send_cached_ui(server, node_id, display_node_id, cached, prompt_id, ui_outputs):
if cached.ui is not None:
ui_outputs[node_id] = cached.ui
if server.client_id is None:
return
cached_ui = cached.ui or {}
server.send_sync("executed", { "node": node_id, "display_node": display_node_id, "output": cached_ui.get("output", None), "prompt_id": prompt_id }, server.client_id)
if cached.ui is not None:
ui_outputs[node_id] = cached.ui
async def execute(server, dynprompt, caches, current_item, extra_data, executed, prompt_id, execution_list, pending_subgraph_results, pending_async_nodes, ui_outputs):
unique_id = current_item
@ -794,12 +795,16 @@ class PromptExecutor:
if self.cache_type == CacheType.RAM_PRESSURE:
ram_release_callback(ram_inactive_headroom)
ram_shortfall = ram_headroom - psutil.virtual_memory().available
freed = comfy.model_management.free_pins(ram_shortfall + 512 * (1024 ** 2))
if freed < ram_shortfall:
if freed > 64 * (1024 ** 2):
# AIMDO MEM_DECOMMIT can outrun psutil.available catching up.
time.sleep(0.05)
ram_release_callback(ram_headroom, free_active=True)
if ram_shortfall > 0:
freed = ram_release_callback(ram_headroom, free_active=True, min_entry_size=RAM_CACHE_LARGE_INTERMEDIATE)
ram_shortfall -= freed
if comfy.model_management.should_free_pins_for_ram_pressure(ram_shortfall):
freed = comfy.model_management.free_pins(ram_shortfall + 512 * (1024 ** 2))
if freed < ram_shortfall:
if freed > 64 * (1024 ** 2):
# AIMDO MEM_DECOMMIT can outrun psutil.available catching up.
time.sleep(0.05)
ram_release_callback(ram_headroom, free_active=True)
else:
# Only execute when the while-loop ends without break
# Send cached UI for intermediate output nodes that weren't executed

6
folder_paths.py
View File

@ -17,7 +17,11 @@ if args.base_directory:
else:
base_path = os.path.dirname(os.path.realpath(__file__))
models_dir = os.path.join(base_path, "models")
if args.models_directory:
models_dir = os.path.abspath(args.models_directory)
else:
models_dir = os.path.join(base_path, "models")
folder_names_and_paths["checkpoints"] = ([os.path.join(models_dir, "checkpoints")], supported_pt_extensions)
folder_names_and_paths["configs"] = ([os.path.join(models_dir, "configs")], [".yaml"])

4
main.py
View File

@ -131,6 +131,10 @@ def apply_custom_paths():
if args.base_directory:
logging.info(f"Setting base directory to: {folder_paths.base_path}")
# --models-directory
if args.models_directory:
logging.info(f"Setting models directory to: {folder_paths.models_dir}")
# --output-directory, --input-directory, --user-directory
if args.output_directory:
output_dir = os.path.abspath(args.output_directory)

9
nodes.py
View File

@ -992,7 +992,7 @@ class CLIPLoader:
@classmethod
def INPUT_TYPES(s):
return {"required": { "clip_name": (folder_paths.get_filename_list("text_encoders"), ),
"type": (["stable_diffusion", "stable_cascade", "sd3", "stable_audio", "mochi", "ltxv", "pixart", "cosmos", "lumina2", "wan", "hidream", "chroma", "ace", "omnigen2", "qwen_image", "hunyuan_image", "flux2", "ovis", "longcat_image", "cogvideox", "lens", "pixeldit", "ideogram4", "boogu", "krea2"], ),
"type": (["stable_diffusion", "stable_cascade", "sd3", "stable_audio", "mochi", "ltxv", "pixart", "cosmos", "lumina2", "wan", "hidream", "chroma", "ace", "omnigen2", "qwen_image", "hunyuan_image", "flux2", "ovis", "longcat_image", "cogvideox", "lens", "pixeldit", "ideogram4", "boogu", "krea2", "joyimage"], ),
},
"optional": {
"device": (["default", "cpu"], {"advanced": True}),
@ -1002,7 +1002,7 @@ class CLIPLoader:
CATEGORY = "model/loaders"
DESCRIPTION = "Recipes:\nsd: clip-l\nstable cascade: clip-g\nsd3: t5 xxl / clip-g / clip-l\nstable audio: t5 base\nmochi: t5 xxl\ncogvideox: t5 xxl (226-token padding)\ncosmos: old t5 xxl\nlumina2: gemma 2 2B\nwan: umt5 xxl\nhidream: llama-3.1 (Recommend) or t5\nomnigen2: qwen vl 2.5 3B\nlens: gpt-oss-20b\npixeldit: gemma 2 2B elm"
DESCRIPTION = "Recipes:\nsd: clip-l\nstable cascade: clip-g\nsd3: t5 xxl / clip-g / clip-l\nstable audio: t5 base\nmochi: t5 xxl\ncogvideox: t5 xxl (226-token padding)\ncosmos: old t5 xxl\nlumina2: gemma 2 2B\nwan: umt5 xxl\nhidream: llama-3.1 (Recommend) or t5\nomnigen2: qwen vl 2.5 3B\njoyimage: qwen3-vl 8B\nlens: gpt-oss-20b\npixeldit: gemma 2 2B elm"
def load_clip(self, clip_name, type="stable_diffusion", device="default"):
clip_type = getattr(comfy.sd.CLIPType, type.upper(), comfy.sd.CLIPType.STABLE_DIFFUSION)
@ -1709,6 +1709,7 @@ class PreviewImage(SaveImage):
self.compress_level = 1
SEARCH_ALIASES = ["preview", "preview image", "show image", "view image", "display image", "image viewer"]
DESCRIPTION = "Preview the images without saving them to the ComfyUI output directory."
@classmethod
def INPUT_TYPES(s):
@ -2458,8 +2459,10 @@ async def init_builtin_extra_nodes():
"nodes_camera_trajectory.py",
"nodes_edit_model.py",
"nodes_tcfg.py",
"nodes_seedvr.py",
"nodes_context_windows.py",
"nodes_qwen.py",
"nodes_joyimage.py",
"nodes_boogu.py",
"nodes_chroma_radiance.py",
"nodes_pid.py",
@ -2478,6 +2481,7 @@ async def init_builtin_extra_nodes():
"nodes_glsl.py",
"nodes_lora_debug.py",
"nodes_textgen.py",
"nodes_text_overlay.py",
"nodes_color.py",
"nodes_toolkit.py",
"nodes_replacements.py",
@ -2504,6 +2508,7 @@ async def init_builtin_extra_nodes():
"nodes_triposplat.py",
"nodes_depth_anything_3.py",
"nodes_seed.py",
"nodes_text.py",
]
import_failed = []

6
openapi.yaml
View File

@ -3297,6 +3297,12 @@ paths:
schema:
$ref: '#/components/schemas/ErrorResponse'
description: Invalid request parameters
"401":
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
description: Unauthorized - Authentication required
"500":
content:
application/json:

2
pyproject.toml
View File

@ -1,6 +1,6 @@
[project]
name = "ComfyUI"
version = "0.27.0"
version = "0.28.0"
readme = "README.md"
license = { file = "LICENSE" }
requires-python = ">=3.10"

8
requirements.txt
View File

@ -1,6 +1,6 @@
comfyui-frontend-package==1.45.20
comfyui-workflow-templates==0.11.2
comfyui-embedded-docs==0.5.6
comfyui-frontend-package==1.45.21
comfyui-workflow-templates==0.11.9
comfyui-embedded-docs==0.5.8
torch
torchsde
torchvision
@ -22,7 +22,7 @@ alembic
SQLAlchemy>=2.0.0
filelock
av>=16.0.0
comfy-kitchen==0.2.16
comfy-kitchen==0.2.21
comfy-aimdo==0.4.10
requests
simpleeval>=1.0.0

12
server.py
View File

@ -39,6 +39,7 @@ from comfy.deploy_environment import get_deploy_environment
import comfy.utils
import comfy.model_management
from comfy_api import feature_flags
from comfy.comfy_api_env import get_environment_overrides
import node_helpers
from comfyui_version import __version__
from app.frontend_management import FrontendManager, parse_version
@ -46,6 +47,7 @@ from comfy_api.internal import _ComfyNodeInternal
from app.assets.seeder import asset_seeder
from app.assets.api.routes import register_assets_routes
from app.assets.services.ingest import register_file_in_place
from app.assets.services.path_utils import get_known_subfolder_tags
from app.assets.services.asset_management import resolve_hash_to_path
from app.user_manager import UserManager
@ -441,7 +443,9 @@ class PromptServer():
if args.enable_assets:
try:
tag = image_upload_type if image_upload_type in ("input", "output") else "input"
result = register_file_in_place(abs_path=filepath, name=filename, tags=[tag])
tags = [tag]
tags.extend(get_known_subfolder_tags(subfolder))
result = register_file_in_place(abs_path=filepath, name=filename, tags=tags)
resp["asset"] = {
"id": result.ref.id,
"name": result.ref.name,
@ -724,7 +728,11 @@ class PromptServer():
@routes.get("/features")
async def get_features(request):
return web.json_response(feature_flags.get_server_features())
features = feature_flags.get_server_features()
overrides = get_environment_overrides()
if overrides:
features.update(overrides)
return web.json_response(features)
@routes.get("/prompt")
async def get_prompt(request):

22
tests-unit/app_test/model_manager_test.py
View File

@ -24,6 +24,28 @@ def app(model_manager):
app.add_routes(routes)
return app
async def test_get_model_folders_includes_registered_extensions(aiohttp_client, app, tmp_path):
"""Folders expose their registered extension set verbatim; an empty list
means match-all (filter_files_extensions semantics)."""
with patch('folder_paths.folder_names_and_paths', {
'test_checkpoints': ([str(tmp_path)], {'.safetensors', '.ckpt'}),
'test_configs': ([str(tmp_path)], ['.yaml']),
'test_match_all': ([str(tmp_path)], set()),
'configs': ([str(tmp_path)], ['.yaml']),
}):
client = await aiohttp_client(app)
response = await client.get('/experiment/models')
assert response.status == 200
folders = {f['name']: f for f in await response.json()}
assert 'configs' not in folders # blocklisted
assert folders['test_checkpoints']['folders'] == [str(tmp_path)]
assert folders['test_checkpoints']['extensions'] == ['.ckpt', '.safetensors']
assert folders['test_configs']['extensions'] == ['.yaml']
# Match-all registrations are exposed honestly, not substituted.
assert folders['test_match_all']['extensions'] == []
async def test_get_model_preview_safetensors(aiohttp_client, app, tmp_path):
img = Image.new('RGB', (100, 100), 'white')
img_byte_arr = BytesIO()

30
tests-unit/app_test/test_migrations.py
View File

@ -8,6 +8,7 @@ upgrade/downgrade for 0003+.
"""
import os
import sqlite3
import pytest
from alembic import command
@ -30,6 +31,12 @@ def _make_config(db_path: str) -> Config:
return cfg
def _sqlite_path(cfg: Config) -> str:
url = cfg.get_main_option("sqlalchemy.url")
assert url is not None and url.startswith("sqlite:///")
return url.removeprefix("sqlite:///")
@pytest.fixture
def migration_db(tmp_path):
"""Yield an alembic Config pre-upgraded to the baseline revision."""
@ -55,3 +62,26 @@ def test_upgrade_downgrade_cycle(migration_db):
command.upgrade(migration_db, "head")
command.downgrade(migration_db, _BASELINE)
command.upgrade(migration_db, "head")
def test_case_sensitive_tags_downgrade_normalizes_existing_tags(migration_db):
"""Downgrading 0005 folds mixed-case tag vocabulary before restoring CHECK."""
command.upgrade(migration_db, "0005_allow_case_sensitive_tags")
db_path = _sqlite_path(migration_db)
with sqlite3.connect(db_path) as conn:
conn.execute("INSERT INTO tags(name) VALUES (?)", ("NewTag",))
conn.execute("INSERT INTO tags(name) VALUES (?)", ("newtag",))
conn.execute("INSERT INTO tags(name) VALUES (?)", ("model_type:LLM",))
command.downgrade(migration_db, "0004_drop_tag_type")
with sqlite3.connect(db_path) as conn:
tags = {row[0] for row in conn.execute("SELECT name FROM tags")}
assert "newtag" in tags
assert "model_type:llm" in tags
assert "NewTag" not in tags
assert "model_type:LLM" not in tags
with pytest.raises(sqlite3.IntegrityError):
conn.execute("INSERT INTO tags(name) VALUES (?)", ("Upper",))

2
tests-unit/assets_test/conftest.py
View File

@ -234,7 +234,7 @@ def seeded_asset(request: pytest.FixtureRequest, http: requests.Session, api_bas
p = getattr(request, "param", {}) or {}
tags: Optional[list[str]] = p.get("tags")
if tags is None:
tags = ["models", "checkpoints", "unit-tests", "alpha"]
tags = ["models", "model_type:checkpoints", "unit-tests", "alpha"]
meta = {"purpose": "test", "epoch": 1, "flags": ["x", "y"], "nullable": None}
# Unique content per test so the seed always creates a fresh asset (201).
# Delete is now always a soft delete, so content from a prior test survives

60
tests-unit/assets_test/queries/test_asset_info.py
View File

@ -133,6 +133,66 @@ class TestListReferencesPage:
assert total == 1
assert refs[0].name == "tagged"
def test_include_tags_filter_ands_persisted_model_tags(self, session: Session):
asset = _make_asset(session, "hash-model-tags")
checkpoint = _make_reference(session, asset, name="checkpoint")
lora = _make_reference(session, asset, name="lora")
input_ref = _make_reference(session, asset, name="input")
ensure_tags_exist(
session,
["models", "model_type:checkpoints", "model_type:loras", "unit-tests"],
)
add_tags_to_reference(
session,
reference_id=checkpoint.id,
tags=["models", "model_type:checkpoints", "unit-tests"],
origin="automatic",
)
add_tags_to_reference(
session,
reference_id=lora.id,
tags=["models", "model_type:loras", "unit-tests"],
origin="automatic",
)
add_tags_to_reference(
session,
reference_id=input_ref.id,
tags=["unit-tests"],
)
session.commit()
refs, _, total = list_references_page(
session,
include_tags=["models", "model_type:checkpoints", "unit-tests"],
)
assert total == 1
assert refs[0].id == checkpoint.id
def test_include_tags_filter_preserves_model_type_case(self, session: Session):
asset = _make_asset(session, "hash-model-case")
ref = _make_reference(session, asset, name="llm")
ensure_tags_exist(session, ["models", "model_type:LLM"])
add_tags_to_reference(
session,
reference_id=ref.id,
tags=["models", "model_type:LLM"],
origin="automatic",
)
session.commit()
refs, _, total = list_references_page(
session, include_tags=["models", "model_type:LLM"]
)
refs_lower, _, total_lower = list_references_page(
session, include_tags=["models", "model_type:llm"]
)
assert total == 1
assert refs[0].id == ref.id
assert total_lower == 0
assert refs_lower == []
def test_exclude_tags_filter(self, session: Session):
asset = _make_asset(session, "hash1")
_make_reference(session, asset, name="keep")

33
tests-unit/assets_test/queries/test_cache_state.py
View File

@ -176,6 +176,39 @@ class TestUpsertReference:
ref = session.query(AssetReference).filter_by(file_path=file_path).one()
assert ref.mtime_ns == final_mtime
def test_upsert_refreshes_loader_path_on_existing_reference(self, session: Session):
"""Re-ingesting an existing reference writes the loader_path computed
by that ingest, healing NULL or stale values even when nothing else
about the row changed."""
asset = _make_asset(session, "hash1")
file_path = "/models/checkpoints/sub/model.safetensors"
upsert_reference(
session, asset_id=asset.id, file_path=file_path, name="model",
mtime_ns=100, loader_path=None,
)
session.commit()
created, updated = upsert_reference(
session, asset_id=asset.id, file_path=file_path, name="model",
mtime_ns=100, loader_path="sub/model.safetensors",
)
session.commit()
assert created is False
assert updated is True
ref = session.query(AssetReference).filter_by(file_path=file_path).one()
assert ref.loader_path == "sub/model.safetensors"
# Identical loader_path is a no-op, not a spurious update.
created, updated = upsert_reference(
session, asset_id=asset.id, file_path=file_path, name="model",
mtime_ns=100, loader_path="sub/model.safetensors",
)
session.commit()
assert created is False
assert updated is False
def test_upsert_restores_missing_reference(self, session: Session):
"""Upserting a reference that was marked missing should restore it."""
asset = _make_asset(session, "hash1")

12
tests-unit/assets_test/queries/test_tags.py
View File

@ -58,7 +58,7 @@ class TestEnsureTagsExist:
session.commit()
tags = session.query(Tag).all()
assert {t.name for t in tags} == {"alpha", "beta"}
assert {t.name for t in tags} == {"ALPHA", "Beta", "alpha"}
def test_empty_list_is_noop(self, session: Session):
ensure_tags_exist(session, [])
@ -258,6 +258,16 @@ class TestListTagsWithUsage:
tag_names = {name for name, _ in rows}
assert tag_names == {"alpha", "alphabet"}
def test_prefix_filter_is_case_sensitive(self, session: Session):
ensure_tags_exist(session, ["model_type:LLM", "model_type:llm"])
session.commit()
rows, total = list_tags_with_usage(session, prefix="model_type:L")
tag_names = {name for name, _ in rows}
assert tag_names == {"model_type:LLM"}
assert total == 1
def test_order_by_name(self, session: Session):
ensure_tags_exist(session, ["zebra", "alpha", "middle"])
session.commit()

83
tests-unit/assets_test/services/test_asset_response_loader_path.py Normal file
View File

@ -0,0 +1,83 @@
"""Tests for how _build_asset_response derives the response `loader_path`.
Guards the persist-and-read contract: the response reads the stored
`loader_path` verbatim, with no read-time recomputation. Like tags, the
value is a seed-time derivative healed by the scan lifecycle.
"""
from datetime import datetime
from pathlib import Path
from unittest.mock import patch
from app.assets.api.routes import _build_asset_response
from app.assets.services.schemas import AssetDetailResult, ReferenceData
_TS = datetime(2024, 1, 1, 0, 0, 0)
def _make_result(
*, file_path: str | None, loader_path: str | None
) -> AssetDetailResult:
ref = ReferenceData(
id="ref-1",
name="model.safetensors",
file_path=file_path,
loader_path=loader_path,
user_metadata=None,
preview_id=None,
created_at=_TS,
updated_at=_TS,
last_access_time=_TS,
)
return AssetDetailResult(ref=ref, asset=None, tags=[])
def test_uses_persisted_loader_path_without_recomputing():
"""A stored loader_path is returned verbatim, not re-derived from file_path.
The sentinel value could never be produced by compute_loader_path for this
file_path, so seeing it in the response proves the stored column is read.
"""
result = _make_result(
file_path="/unmatched/root/model.safetensors",
loader_path="SENTINEL/stored.safetensors",
)
resp = _build_asset_response(result)
assert resp.loader_path == "SENTINEL/stored.safetensors"
def test_null_stored_loader_path_is_served_as_null(tmp_path: Path):
"""No read-time recomputation: a NULL column is served as null even when
the path would resolve."""
models = tmp_path / "models"
ckpt = models / "checkpoints"
ckpt.mkdir(parents=True)
f = ckpt / "bar.safetensors"
f.touch()
with patch("app.assets.services.path_utils.folder_paths") as mock_fp, patch(
"app.assets.services.path_utils.get_comfy_models_folders",
return_value=[("checkpoints", [str(ckpt)], {".safetensors"})],
):
mock_fp.get_input_directory.return_value = str(tmp_path / "in")
mock_fp.get_output_directory.return_value = str(tmp_path / "out")
mock_fp.get_temp_directory.return_value = str(tmp_path / "tmp")
mock_fp.models_dir = str(models)
result = _make_result(file_path=str(f), loader_path=None)
resp = _build_asset_response(result)
assert resp.loader_path is None
assert resp.display_name == "checkpoints/bar.safetensors"
def test_all_path_fields_null_without_file_path():
"""API-created / hash-only references (no file_path) expose no paths."""
result = _make_result(file_path=None, loader_path=None)
resp = _build_asset_response(result)
assert resp.loader_path is None
assert resp.display_name is None

Some files were not shown because too many files have changed in this diff Show More