16 Commits

Author	SHA1	Message	Date
Matt Miller	2d21956ac7	fix(assets): expand standalone bucket tag for nested category paths ... Path-derived tags for nested model layouts (e.g. models/checkpoints/flux/foo.safetensors) emitted only the slash-joined shape `["models", "checkpoints/flux"]`, which broke the frontend combo-widget set-membership filter `include_tags=models,checkpoints` — the literal `checkpoints` token was no longer present in the asset's tag set. Add `expand_bucket_prefixes` at the tag-write layer. When a tag's first slash segment is a registered model category (or input/output/temp root), the bucket is inserted as a standalone token immediately after the slash-joined form. This preserves tag[1] as the slash-joined positional contract cloud emits while restoring the set-membership token the frontend filter requires. The expansion is bounded to known buckets so free-form user labels with slashes (`my-org/team-a`) pass through unchanged. The helper is applied uniformly in `set_reference_tags`, `add_tags_to_reference`, and `batch_insert_seed_assets` so HTTP uploads, user-tag mutations, and path-scanning ingest all converge on the same canonical shape. Also align the upload-route category validator with `resolve_destination_from_tags` by extracting the first slash segment of tag[1], so HTTP uploads matching cloud's slash-joined emission shape are no longer rejected as `unknown models category`.	2026-05-20 20:33:39 -07:00
Matt Miller	00940fb24e	fix(assets): preserve caller order in add_tags_to_reference + align response helper ... Smoke test through the real HTTP upload + tag-add path exposed two ordering bugs the unit-layer tests missed: 1. add_tags_to_reference did `to_add = sorted(want - current)` — an alphabetical pre-sort defeating the microsecond-stagger fix from the previous commit. The stagger was encoding alphabetical positions, not the caller's insertion order. Fix: build to_add by walking the already-normalized caller list and filtering against the current set, so the staggered added_at timestamps reflect what the caller actually requested. 2. get_reference_tags used .order_by(tag_name.asc()) — alphabetical. It's called by the upload response path; meanwhile list_references_page and fetch_reference_asset_and_tags were already updated to order by added_at. The mismatch meant POST /api/assets returned tags in alphabetical order but a subsequent GET returned them in insertion order. Fix: order get_reference_tags by added_at too, so all three response-path helpers agree. New tests-unit/assets_test/test_user_tag_http_smoke.py exercises the full HTTP layer: POST /api/assets to upload, POST /api/assets/{id}/tags to add a user tag (using tag names like "aaa-user-tag" that would jump to position 0 under alphabetical), GET /api/assets/{id} to verify ordering. Catches the bugs above in CI going forward. Full assets suite: 340 passed, 10 pre-existing skipped.	2026-05-19 21:10:53 -07:00
Matt Miller	7ff001d7c8	fix(assets): stagger added_at in set_reference_tags + add ordering tests ... Cursor-reviews follow-up on PR #13994: 1. set_reference_tags / add_tags_to_reference now apply the same microsecond stagger as batch_insert_seed_assets. Per-tag get_utc_now() calls can collide at microsecond resolution on fast machines, dropping retrieval to the tag_name alphabetical tiebreaker. Using a single base_ts + timedelta(microseconds=i) preserves insertion order for any batch. 2. Docstring on get_name_and_tags_from_asset_path corrected: only the subpath is lowercased in code; the root category is lowercase by construction in get_asset_category_and_relative_path. 3. resolve_destination_from_tags docstring now states explicitly that hybrid shapes (mix of legacy multi-tag + new slash-joined within a single call) are accepted and resolve to the same destination. 4. New TestTagRetrievalOrder class in test_asset_info.py exercises the public write paths (set_reference_tags, add_tags_to_reference, remove_tags_from_reference) and asserts the public read paths (list_references_page, fetch_reference_asset_and_tags) return tags in insertion order rather than alphabetical. Tag names are chosen to fail loudly under alphabetical regression — "checkpoints" sorts before "models", "aaa-user-tag" sorts before every path tag, etc. Full assets suite: 338 passed, 10 pre-existing skipped.	2026-05-19 21:05:54 -07:00
Matt Miller	3ffc49aa0e	fix(assets): lowercase subpath, parse slash-joined upload tags, stagger added_at ... Three bugs surfaced by an end-to-end smoke test of the read+write round-trip; all in this PR's scope. 1. FK violation on uppercase paths get_name_and_tags_from_asset_path was preserving case on the subpath (e.g. "diffusers/Kolors/text_encoder"). ensure_tags_exist lowercases via normalize_tags before inserting into the tags table, so the asset_reference_tags.tag_name FK to tags.name failed for any path containing uppercase letters — including the diffusers case the PR was designed to support. Fix: lowercase the slash-joined subpath in get_name_and_tags_from_asset_path to match the canonicalization ensure_tags_exist applies. Providers keyed on original-case subpaths need to normalize their lookup key to lowercase. 2. resolve_destination_from_tags rejected the new tag shape The inverse function only accepted the legacy one-tag-per-dir shape (["models", "diffusers", "Kolors", "text_encoder"]). An upload using the slash-joined shape returned by /api/assets raised "unknown model category" or "invalid path component". Fix: pre-split every entry after tags[0] on "/" so both shapes resolve identically. For models, the first expanded segment is the category and the rest are subdirs; for input/output the full expansion becomes the subdirs. 3. Within-batch tag order was lost bulk_ingest wrote every tag in a single batch with the same added_at = current_time. The retrieval ORDER BY added_at, tag_name then fell back to the tag_name tiebreaker, sorting the path-derived pair alphabetically — putting "checkpoints/..." ahead of "models" since "c" < "m". The tags[0] = root contract was lost on bulk- ingested rows. Fix: stagger added_at by microseconds per tag index within a reference so the retrieval order matches the input list order. Path-derived tags now consistently land in position-0 = root, position-1 = subpath. Tests - TestGetNameAndTagsFromAssetPath updated: subpath is now lowercase. - New TestResolveDestinationFromTags covers both tag shapes, the unknown-category case for slash-joined input, traversal rejection, and input/output paths. - Full suite: 333 passed, 10 pre-existing skipped.	2026-05-19 20:30:04 -07:00
Matt Miller	36f9a6fdef	feat(assets): preserve insertion order on tag retrieval ... The /api/assets response previously sorted tags alphabetically via .order_by(Tag.name.asc()). That breaks the structurally meaningful "root category first, then subpath" invariant the path-collapsing change relies on: alphabetical sort puts a custom user tag (or even the bare "models" root) at unpredictable positions, so positional access like tags[1] is not reliable on local. Cloud already preserves insertion order — its Ent WithTags() eager- load has no explicit ORDER BY, so Postgres returns rows in physical insertion order. Local's composite primary key on (asset_reference_id, tag_name) means SQLite walks the index in tag_name order even without an explicit ORDER BY, so just dropping the clause isn't enough. Switching to ORDER BY added_at ASC, tag_name ASC keeps the path tags inserted via set_reference_tags in their original order (microsecond-resolution timestamps disambiguate same-batch inserts; tag_name is a deterministic tiebreaker for the rare collision case). Custom tags added later via add_tags_to_reference land after the path tags in their own added_at bucket. Applies to both response-shaping queries: - list_references_page (GET /api/assets, tag_map join) - fetch_reference_asset_and_tags (GET /api/assets/{id}) Catalog/histogram queries in app/assets/database/queries/tags.py keep their alphabetical sort — those endpoints are listing all tags, not per-asset tags, and alphabetical is the right shape there.	2026-05-19 20:14:01 -07:00
Matt Miller	7ab346fc7b	chore(assets): drop unused normalize_tags import after subpath-collapse refactor ... normalize_tags lowercased every tag, which would have stripped case from the slash-joined subpath (e.g. "diffusers/Kolors/text_encoder" -> "diffusers/kolors/text_encoder") and broken consumer lookups keyed on the original-case path. The refactored implementation inlines a strip + dedup so the import is no longer needed.	2026-05-19 14:51:00 -07:00
Matt Miller	5b7288d700	feat(assets): collapse nested asset path into a single slash-joined tag ... The /api/assets response previously emitted one tag per parent directory between the root category and the filename. For nested categories like diffusers, this produced ["models", "diffusers", "Kolors", "text_encoder"] where consumers that look up a category via tags[1] would only see the top-level bucket name and miss the model-specific sub-path that uniquely identifies the component. This collapses the parent subpath into a single slash-joined tag so the result is ["models", "diffusers/Kolors/text_encoder"]. Consumers can now read tags[1] as a stable category identifier regardless of how deep the file lives in the bucket. Case is preserved on the subpath so providers keyed on the original-case path (e.g. "diffusers/Kolors/text_encoder") resolve correctly. Same shape applies uniformly: - input/foo.png -> ["input"] - output/00001.png -> ["output"] - models/checkpoints/flux.safetensors -> ["models", "checkpoints"] - models/diffusers/Kolors/text_encoder/m.sft -> ["models", "diffusers/Kolors/text_encoder"] - models/loras/my/custom/path/v1.safetensors -> ["models", "loras/my/custom/path"] Integration tests that filtered by individual subdirectory tags (`include_tags=unit-tests,scope`) updated to use the new slash-joined shape (`include_tags=unit-tests/scope`). Unit tests cover flat input, flat output, flat models, diffusers-style nested, and deep user-subpath cases.	2026-05-19 14:48:49 -07:00
Luke Mino-Altherr	3eba2dcf2d	fix(assets): recognize temp directory in asset category resolution (#13159)	2026-03-25 19:59:59 -07:00
Luke Mino-Altherr	7d5534d8e5	feat(assets): register output files as assets after prompt execution (#12812)	2026-03-24 20:48:55 -07:00
Luke Mino-Altherr	2bd4d82b4f	feat(assets): align local API with cloud spec (#12863) ... * feat(assets): align local API with cloud spec Unify response models, add missing fields, and align input schemas with the cloud OpenAPI spec at cloud.comfy.org/openapi. - Replace AssetSummary/AssetDetail/AssetUpdated with single Asset model - Add is_immutable, metadata (system_metadata), prompt_id fields - Support mime_type and preview_id in update endpoint - Make CreateFromHashBody.name optional, add mime_type, require >=1 tag - Add id/mime_type/preview_id to upload, relax tags to optional - Rename total_tags → tags in tag add/remove responses - Add GET /api/assets/tags/refine histogram endpoint - Add DB migration for system_metadata and prompt_id columns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix review issues: tags validation, size nullability, type annotation, hash mismatch check, and add tag histogram tests - Remove contradictory min_length=1 from CreateFromHashBody.tags default - Restore size field to int|None=None for proper null semantics - Add Union type annotation to _build_asset_response result param - Add hash mismatch validation on idempotent upload path (409 HASH_MISMATCH) - Add unit tests for list_tag_histogram service function Amp-Thread-ID: https://ampcode.com/threads/T-019cd993-f43c-704e-b3d7-6cfc3d4d4a80 Co-authored-by: Amp <amp@ampcode.com> * Add preview_url to /assets API response using /api/view endpoint For input and output assets, generate a preview_url pointing to the existing /api/view endpoint using the asset's filename and tag-derived type (input/output). Handles subdirectories via subfolder param and URL-encodes filenames with spaces, unicode, and special characters. This aligns the OSS backend response with the frontend AssetCard expectation for thumbnail rendering. Amp-Thread-ID: https://ampcode.com/threads/T-019cda3f-5c2c-751a-a906-ac6c9153ac5c Co-authored-by: Amp <amp@ampcode.com> * chore: remove unused imports from asset_reference queries Amp-Thread-ID: https://ampcode.com/threads/T-019cda7d-cb21-77b4-a51b-b965af60208c Co-authored-by: Amp <amp@ampcode.com> * feat: resolve blake3 hashes in /view endpoint via asset database Amp-Thread-ID: https://ampcode.com/threads/T-019cda7d-cb21-77b4-a51b-b965af60208c Co-authored-by: Amp <amp@ampcode.com> * Register uploaded images in asset database when --enable-assets is set Add register_file_in_place() service function to ingest module for registering already-saved files without moving them. Call it from the /upload/image endpoint to return asset metadata in the response. Amp-Thread-ID: https://ampcode.com/threads/T-019ce023-3384-7560-bacf-de40b0de0dd2 Co-authored-by: Amp <amp@ampcode.com> * Exclude None fields from asset API JSON responses Add exclude_none=True to model_dump() calls across asset routes to keep response payloads clean by omitting unset optional fields. Amp-Thread-ID: https://ampcode.com/threads/T-019ce023-3384-7560-bacf-de40b0de0dd2 Co-authored-by: Amp <amp@ampcode.com> * Add comment explaining why /view resolves blake3 hashes Amp-Thread-ID: https://ampcode.com/threads/T-019ce023-3384-7560-bacf-de40b0de0dd2 Co-authored-by: Amp <amp@ampcode.com> * Move blake3 hash resolution to asset_management service Extract resolve_hash_to_path() into asset_management.py and remove _resolve_blake3_to_path from server.py. Also revert loopback origin check to original logic. Amp-Thread-ID: https://ampcode.com/threads/T-019ce023-3384-7560-bacf-de40b0de0dd2 Co-authored-by: Amp <amp@ampcode.com> * Require at least one tag in UploadAssetSpec Enforce non-empty tags at the Pydantic validation layer so uploads with no tags are rejected with a 400 before reaching ingest. Adds test_upload_empty_tags_rejected to cover this case. Amp-Thread-ID: https://ampcode.com/threads/T-019ce377-8bde-7048-bc28-a9df063409f9 Co-authored-by: Amp <amp@ampcode.com> * Add owner_id check to resolve_hash_to_path Filter asset references by owner visibility so the /view endpoint only resolves hashes for assets the requesting user can access. Adds table-driven tests for owner visibility cases. Amp-Thread-ID: https://ampcode.com/threads/T-019ce377-8bde-7048-bc28-a9df063409f9 Co-authored-by: Amp <amp@ampcode.com> * Make ReferenceData.created_at and updated_at required Remove None defaults and type: ignore comments. Move fields before optional fields to satisfy dataclass ordering. Amp-Thread-ID: https://ampcode.com/threads/T-019ce377-8bde-7048-bc28-a9df063409f9 Co-authored-by: Amp <amp@ampcode.com> * Fix double commit in create_from_hash Move mime_type update into _register_existing_asset so it shares a single transaction with reference creation. Log a warning when the hash is not found instead of silently returning None. Amp-Thread-ID: https://ampcode.com/threads/T-019ce377-8bde-7048-bc28-a9df063409f9 Co-authored-by: Amp <amp@ampcode.com> * Add exclude_none=True to create/upload responses Align with get/update/list endpoints for consistent JSON output. Amp-Thread-ID: https://ampcode.com/threads/T-019ce377-8bde-7048-bc28-a9df063409f9 Co-authored-by: Amp <amp@ampcode.com> * Change preview_id to reference asset by reference ID, not content ID Clients receive preview_id in API responses but could not dereference it through public routes (which use reference IDs). Now preview_id is a self-referential FK to asset_references.id so the value is directly usable in the public API. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Filter soft-deleted and missing refs from visibility queries list_references_by_asset_id and list_tags_with_usage were not filtering out deleted_at/is_missing refs, allowing /view?filename=blake3:... to serve files through hidden references and inflating tag usage counts. Add list_all_file_paths_by_asset_id for orphan cleanup which intentionally needs unfiltered access. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Pass preview_id and mime_type through all asset creation fast paths The duplicate-content upload path and hash-based creation paths were silently dropping preview_id and mime_type. This wires both fields through _register_existing_asset, create_from_hash, and all route call sites so behavior is consistent regardless of whether the asset content already exists. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove unimplemented client-provided ID from upload API The `id` field on UploadAssetSpec was advertised for idempotent creation but never actually honored when creating new references. Remove it rather than implementing the feature. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Make asset mime_type immutable after first ingest Prevents cross-tenant metadata mutation when multiple references share the same content-addressed Asset row. mime_type can now only be set when NULL (first ingest); subsequent attempts to change it are silently ignored. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use resolved content_type from asset lookup in /view endpoint The /view endpoint was discarding the content_type computed by resolve_hash_to_path() and re-guessing from the filename, which produced wrong results for extensionless files or mismatched extensions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Merge system+user metadata into filter projection Extract rebuild_metadata_projection() to build AssetReferenceMeta rows from {**system_metadata, **user_metadata}, so system-generated metadata is queryable via metadata_filter and user keys override system keys. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Standardize tag ordering to alphabetical across all endpoints Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Derive subfolder tags from path in register_file_in_place Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Reject client-provided id, fix preview URLs, rename tags→total_tags - Reject 'id' field in multipart upload with 400 UNSUPPORTED_FIELD instead of silently ignoring it - Build preview URL from the preview asset's own metadata rather than the parent asset's - Rename 'tags' to 'total_tags' in TagsAdd/TagsRemove response schemas for clarity Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: SQLite migration 0003 FK drop fails on file-backed DBs (MB-2) Add naming_convention to Base.metadata so Alembic batch-mode reflection can match unnamed FK constraints created by migration 0002. Pass naming_convention and render_as_batch=True through env.py online config. Add migration roundtrip tests (upgrade/downgrade/cycle from baseline). Amp-Thread-ID: https://ampcode.com/threads/T-019ce466-1683-7471-b6e1-bb078223cda0 Co-authored-by: Amp <amp@ampcode.com> * Fix missing tag count for is_missing references and update test for total_tags field - Allow is_missing=True references to be counted in list_tags_with_usage when the tag is 'missing', so the missing tag count reflects all references that have been tagged as missing - Add update_is_missing_by_asset_id query helper for bulk updates by asset - Update test_add_and_remove_tags to use 'total_tags' matching the API schema Amp-Thread-ID: https://ampcode.com/threads/T-019ce482-05e7-7324-a1b0-a56a929cc7ef Co-authored-by: Amp <amp@ampcode.com> * Remove unused imports in scanner.py Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Rename prompt_id to job_id on asset_references Rename the column in the DB model, migration, and service schemas. The API response emits both job_id and prompt_id (deprecated alias) for backward compatibility with the cloud API. Amp-Thread-ID: https://ampcode.com/threads/T-019cef41-60b0-752a-aa3c-ed7f20fda2f7 Co-authored-by: Amp <amp@ampcode.com> * Add index on asset_references.preview_id for FK cascade performance Amp-Thread-ID: https://ampcode.com/threads/T-019cef45-a4d2-7548-86d2-d46bcd3db419 Co-authored-by: Amp <amp@ampcode.com> * Add clarifying comments for Asset/AssetReference naming and preview_id Amp-Thread-ID: https://ampcode.com/threads/T-019cef49-f94e-7348-bf23-9a19ebf65e0d Co-authored-by: Amp <amp@ampcode.com> * Disallow all-null meta rows: add CHECK constraint, skip null values on write - convert_metadata_to_rows returns [] for None values instead of an all-null row - Remove dead None branch from _scalar_to_row - Simplify null filter in common.py to just check for row absence - Add CHECK constraint ck_asset_reference_meta_has_value to model and migration 0003 Amp-Thread-ID: https://ampcode.com/threads/T-019cef4e-5240-7749-bb25-1f17fcf9c09c Co-authored-by: Amp <amp@ampcode.com> * Remove dead None guards on result.asset in upload handler register_file_in_place guarantees a non-None asset, so the 'if result.asset else None' checks were unreachable. Amp-Thread-ID: https://ampcode.com/threads/T-019cef5b-4cf8-723c-8a98-8fb8f333c133 Co-authored-by: Amp <amp@ampcode.com> * Remove mime_type from asset update API Clients can no longer modify mime_type after asset creation via the PUT /api/assets/{id} endpoint. This reduces the risk of mime_type spoofing. The internal update_asset_hash_and_mime function remains available for server-side use (e.g., enrichment). Amp-Thread-ID: https://ampcode.com/threads/T-019cef5d-8d61-75cc-a1c6-2841ac395648 Co-authored-by: Amp <amp@ampcode.com> * Fix migration constraint naming double-prefix and NULL in mixed metadata lists - Use fully-rendered constraint names in migration 0003 to avoid the naming convention doubling the ck_ prefix on batch operations. - Add table_args to downgrade so SQLite batch mode can find the CHECK constraint (not exposed by SQLite reflection). - Fix model CheckConstraint name to use bare 'has_value' (convention auto-prefixes). - Skip None items when converting metadata lists to rows, preventing all-NULL rows that violate the has_value check constraint. Amp-Thread-ID: https://ampcode.com/threads/T-019cef87-94f9-7172-a6af-c6282290ce4f Co-authored-by: Amp <amp@ampcode.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Amp <amp@ampcode.com>	2026-03-16 12:34:04 -07:00
comfyanonymous	e4b0bb8305	Import assets seeder later, print some package versions. (#12841)	2026-03-08 16:25:30 -04:00
Luke Mino-Altherr	29b24cb517	refactor(assets): modular architecture + async two-phase scanner & background seeder (#12621)	2026-03-07 20:37:25 -05:00
Jedrzej Kosinski	6ea8c128a3	Assets Part 2 - add more endpoints (#12125)	2026-01-31 02:22:05 -05:00
Jedrzej Kosinski	acd0e53653	Make bulk_ops not use .returning to be compatible with python 3.10 and 3.11 sqlalchemy (#11839)	2026-01-13 00:15:24 -05:00
Jedrzej Kosinski	ec0a832acb	Add workaround for hacky nodepack(s) that edit folder_names_and_paths to have values with tuples of more than 2. Other things could potentially break with those nodepack(s), so I will hunt for the guilty nodepack(s) now. (#11755)	2026-01-08 22:49:12 -08:00
Jedrzej Kosinski	1dc3da6314	Add most basic Asset support for models (#11315) ... * Brought over minimal elements from PR 10045 to reproduce seed_assets and register_assets_system without adding anything to the DB or server routes yet, for now making everything sync (can introduce async once everything is cleaned up and brought over) * Added db script to insert assets stuff, cleaned up some code; assets (models) now get added/rescanned * Added support for 5 http endpoints for assets * Replaced Optional with | None in schemas_in.py and schemas_out.py * Remove two routes that will not be relevant yet in this PR: HEAD /api/assets/hash/<hash> and PUT /api/assets/<id>/preview * Remove some functions the two deleted endpoints were using * Don't show assets scan message upon calling /object_info endpoint * removed unsued import to satisfy ruff * Simplified hashing function tpye hint and _hash_file_obj * Satisfied ruff	2026-01-08 22:21:51 -05:00