25 Commits

Author	SHA1	Message	Date
Luke Mino-Altherr	d8cb122dfb	chore: sort imports in assets package ... Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	2026-02-03 20:02:52 -08:00
Luke Mino-Altherr	0f75def5b5	refactor: move scanner.py out of services to top-level assets module ... Scanner is used externally by main.py and server.py for startup/maintenance, not as part of the regular service layer. Moving it to app/assets/scanner.py makes the public API clearer. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	2026-02-03 19:56:29 -08:00
Luke Mino-Altherr	bf7fbb6317	chore: remove unused get_utc_now import ... Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	2026-02-03 18:47:11 -08:00
Luke Mino-Altherr	5571508e61	refactor: use query functions instead of direct ORM modifications in service layer ... Add update_asset_info_name and update_asset_info_updated_at query functions and update asset_management.py to use them instead of modifying ORM objects directly. This ensures the service layer only uses explicit operations from the queries package. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	2026-02-03 18:44:23 -08:00
Luke Mino-Altherr	e3b8e512ca	refactor: use explicit dataclasses instead of ORM objects in service layer ... Replace dict/ORM object returns with explicit dataclasses to fix DetachedInstanceError when accessing ORM attributes after session closes. - Add app/assets/services/schemas.py with AssetData, AssetInfoData, AssetDetailResult, and RegisterAssetResult dataclasses - Update asset_management.py and ingest.py to return dataclasses - Update manager.py to use attribute access on dataclasses - Fix created_new to be False in create_asset_from_hash (content exists) - Add DependencyMissingError for better blake3 missing error handling - Update tests to use attribute access instead of dict subscripting Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	2026-02-03 18:39:07 -08:00
Luke Mino-Altherr	ccfc5dedd4	fix: handle missing blake3 module gracefully to prevent server crash ... Make blake3 an optional import that fails gracefully at import time, with a clear error message when hashing functions are actually called. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	2026-02-03 14:46:33 -08:00
Luke Mino-Altherr	e9ca190098	refactor: remove try-finally wrapper in seed_assets by extracting helpers ... Extract focused helper functions to eliminate the try-finally block that wrapped ~50 lines just for logging. The new helpers (_collect_paths_for_roots, _build_asset_specs, _insert_asset_specs) make seed_assets a simple linear flow. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	2026-02-03 14:33:54 -08:00
Luke Mino-Altherr	ed60e93696	refactor: flatten nested try blocks and if statements in assets package ... Extract helper functions to eliminate nested try-except blocks in scanner.py and remove duplicated type-checking logic in asset_info.py. Simplify nested conditionals in asset_management.py for clearer control flow. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	2026-02-03 14:28:16 -08:00
Luke Mino-Altherr	fef2f01671	refactor: improve function naming for clarity and consistency ... Rename functions to use clearer verb-based names: - pick_best_live_path → select_best_live_path - escape_like_prefix → escape_sql_like_string - list_tree → list_files_recursively - check_asset_file_fast → verify_asset_file_unchanged - _seed_from_paths_batch → _batch_insert_assets_from_paths - reconcile_cache_states_for_root → sync_cache_states_with_filesystem - touch_asset_info_by_id → update_asset_info_access_time - replace_asset_info_metadata_projection → set_asset_info_metadata - expand_metadata_to_rows → convert_metadata_to_rows - _rows_per_stmt → _calculate_rows_per_statement - ensure_within_base → validate_path_within_base - _cleanup_temp → _delete_temp_file_if_exists - validate_hash_format → normalize_and_validate_hash - get_relative_to_root_category_path_of_asset → get_asset_category_and_relative_path Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	2026-02-03 14:20:36 -08:00
Luke Mino-Altherr	481a2fa263	refactor: rename functions to verb-based naming convention ... Rename functions across app/assets/ to follow verb-based naming: - is_scalar → check_is_scalar - project_kv → expand_metadata_to_rows - _visible_owner_clause → _build_visible_owner_clause - _chunk_rows → _iter_row_chunks - _at_least_one → _validate_at_least_one_field - _tags_norm → _normalize_tags_field - _ser_dt → _serialize_datetime - _ser_updated → _serialize_updated_at - _error_response → _build_error_response - _validation_error_response → _build_validation_error_response - file_sender → stream_file_chunks - seed_assets_endpoint → seed_assets - utcnow → get_utc_now - _safe_sort_field → _validate_sort_field - _safe_filename → _sanitize_filename - fast_asset_file_check → check_asset_file_fast - prefixes_for_root → get_prefixes_for_root - blake3_hash → compute_blake3_hash - blake3_hash_async → compute_blake3_hash_async - _is_within → _check_is_within - _rel → _compute_relative Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	2026-02-03 13:58:14 -08:00
Luke Mino-Altherr	4e02245012	fix: ruff linting errors and add comprehensive test coverage for asset queries ... - Fix unused imports in routes.py, asset.py, manager.py, asset_management.py, ingest.py - Fix whitespace issues in upload.py, asset_info.py, ingest.py - Fix typo in manager.py (stray character after result["asset"]) - Fix broken import in test_metadata.py (project_kv moved to asset_info.py) - Add fixture override in queries/conftest.py for unit test isolation Add 48 new tests covering all previously untested query functions: - asset.py: upsert_asset, bulk_insert_assets - cache_state.py: upsert_cache_state, delete_cache_states_outside_prefixes, get_orphaned_seed_asset_ids, delete_assets_by_ids, get_cache_states_for_prefixes, bulk_set_needs_verify, delete_cache_states_by_ids, delete_orphaned_seed_asset, bulk_insert_cache_states_ignore_conflicts, get_cache_states_by_paths_and_asset_ids - asset_info.py: insert_asset_info, get_or_create_asset_info, update_asset_info_timestamps, replace_asset_info_metadata_projection, bulk_insert_asset_infos_ignore_conflicts, get_asset_info_ids_by_ids - tags.py: bulk_insert_tags_and_meta Total: 119 tests pass (up from 71) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	2026-02-03 13:21:12 -08:00
Luke Mino-Altherr	9f9db2c2c2	refactor: extract multipart upload parsing from routes ... - Add app/assets/api/upload.py with parse_multipart_upload() for HTTP parsing - Add ParsedUpload dataclass to schemas_in.py - Add domain exceptions (AssetValidationError, AssetNotFoundError, HashMismatchError) - Add manager.process_upload() with domain exceptions (no HTTP status codes) - Routes map domain exceptions to HTTP responses - Slim down upload_asset route to ~20 lines (was ~150) Amp-Thread-ID: https://ampcode.com/threads/T-019c2519-abe1-738a-ad2e-29ece17c0e42 Co-authored-by: Amp <amp@ampcode.com>	2026-02-03 13:08:04 -08:00
Luke Mino-Altherr	e987bd268f	Move get_comfy_models_folders to path_utils.py to avoid late import ... Amp-Thread-ID: https://ampcode.com/threads/T-019c2510-33fa-7199-ae4b-bc31102277a7 Co-authored-by: Amp <amp@ampcode.com>	2026-02-03 13:08:04 -08:00
Luke Mino-Altherr	2eb100adf9	Refactor helpers.py: move functions to their respective modules ... - Move scanner-only functions to scanner.py - Move query-only functions (is_scalar, project_kv) to asset_info.py - Move get_query_dict to routes.py - Create path_utils.py service for path-related functions - Reduce helpers.py to shared utilities only Amp-Thread-ID: https://ampcode.com/threads/T-019c2510-33fa-7199-ae4b-bc31102277a7 Co-authored-by: Amp <amp@ampcode.com>	2026-02-03 13:08:04 -08:00
Luke Mino-Altherr	a02f160e20	Move hashing.py to services directory ... Amp-Thread-ID: https://ampcode.com/threads/T-019c2510-33fa-7199-ae4b-bc31102277a7 Co-authored-by: Amp <amp@ampcode.com>	2026-02-03 13:08:04 -08:00
Luke Mino-Altherr	c3105b1174	refactor: move bulk_ops to queries and scanner service ... - Delete bulk_ops.py, moving logic to appropriate layers - Add bulk insert query functions: - queries/asset.bulk_insert_assets - queries/cache_state.bulk_insert_cache_states_ignore_conflicts - queries/cache_state.get_cache_states_by_paths_and_asset_ids - queries/asset_info.bulk_insert_asset_infos_ignore_conflicts - queries/asset_info.get_asset_info_ids_by_ids - queries/tags.bulk_insert_tags_and_meta - Move seed_from_paths_batch orchestration to scanner._seed_from_paths_batch Amp-Thread-ID: https://ampcode.com/threads/T-019c24fd-157d-776a-ad24-4f19cf5d3afe Co-authored-by: Amp <amp@ampcode.com>	2026-02-03 13:08:04 -08:00
Luke Mino-Altherr	64d2f51dfc	refactor: move scanner to services layer with pure query extraction ... - Move app/assets/scanner.py to app/assets/services/scanner.py - Extract pure queries from fast_db_consistency_pass: - get_cache_states_for_prefixes() - bulk_set_needs_verify() - delete_cache_states_by_ids() - delete_orphaned_seed_asset() - Split prune_orphaned_assets into pure queries: - delete_cache_states_outside_prefixes() - get_orphaned_seed_asset_ids() - delete_assets_by_ids() - Add reconcile_cache_states_for_root() service function - Add prune_orphaned_assets() service function - Remove function injection pattern - Update imports in main.py, server.py, routes.py Amp-Thread-ID: https://ampcode.com/threads/T-019c24f1-3385-701b-87e0-8b6bc87e841b Co-authored-by: Amp <amp@ampcode.com>	2026-02-03 13:08:04 -08:00
Luke Mino-Altherr	fba4570e49	refactor: move in-function imports to top-level and remove keyword-only argument pattern ... - Move imports from inside functions to module top-level in: - app/assets/database/queries/asset.py - app/assets/database/queries/asset_info.py - app/assets/database/queries/cache_state.py - app/assets/manager.py - app/assets/services/asset_management.py - app/assets/services/ingest.py - Remove keyword-only argument markers (*,) from app/assets/ to match codebase conventions Amp-Thread-ID: https://ampcode.com/threads/T-019c24eb-bfa2-727f-8212-8bc976048604 Co-authored-by: Amp <amp@ampcode.com>	2026-02-03 13:08:04 -08:00
Luke Mino-Altherr	15ee03f65c	Refactor asset database: separate business logic from queries ... Architecture changes: - API Routes -> manager.py (thin adapter) -> services/ (business logic) -> queries/ (atomic DB ops) - Services own session lifecycle via create_session() - Queries accept Session as parameter, do single-table atomic operations New app/assets/services/ layer: - __init__.py - exports all service functions - ingest.py - ingest_file_from_path(), register_existing_asset() - asset_management.py - get_asset_detail(), update_asset_metadata(), delete_asset_reference(), set_asset_preview() - tagging.py - apply_tags(), remove_tags(), list_tags() Removed from queries/asset_info.py: - ingest_fs_asset (moved to services/ingest.py as ingest_file_from_path) - update_asset_info_full (moved to services/asset_management.py as update_asset_metadata) - create_asset_info_for_existing_asset (moved to services/ingest.py as register_existing_asset) Updated manager.py: - Now a thin adapter that transforms API schemas to/from service calls - Delegates all business logic to services layer - No longer imports sqlalchemy.orm.Session or models directly Test updates: - Fixed test_cache_state.py import of pick_best_live_path (moved to helpers.py) - Added comprehensive service layer tests (41 new tests) - All 112 query + service tests pass Amp-Thread-ID: https://ampcode.com/threads/T-019c24e2-7ae4-707f-ad19-c775ed8b82b5 Co-authored-by: Amp <amp@ampcode.com>	2026-02-03 13:08:04 -08:00
Luke Mino-Altherr	70a600baf0	chore: remove unused Asset import from manager.py ... Amp-Thread-ID: https://ampcode.com/threads/T-019c24bb-475b-7442-9ff9-8288edea3345 Co-authored-by: Amp <amp@ampcode.com>	2026-02-03 13:08:04 -08:00
Luke Mino-Altherr	17ad7e393f	refactor(assets): split queries.py into modular query modules ... Split the ~1000 line app/assets/database/queries.py into focused modules: - queries/asset.py - Asset entity queries (asset_exists_by_hash, get_asset_by_hash) - queries/asset_info.py - AssetInfo queries (~15 functions) - queries/cache_state.py - AssetCacheState queries (list_cache_states_by_asset_id, pick_best_live_path, prune_orphaned_assets, fast_db_consistency_pass) - queries/tags.py - Tag queries (8 functions including ensure_tags_exist, add/remove tag functions, list_tags_with_usage) - queries/__init__.py - Re-exports all public functions for backward compatibility Also adds comprehensive unit tests using in-memory SQLite: - tests-unit/assets_test/queries/conftest.py - Session fixture - tests-unit/assets_test/queries/test_asset.py - 5 tests - tests-unit/assets_test/queries/test_asset_info.py - 23 tests - tests-unit/assets_test/queries/test_cache_state.py - 8 tests - tests-unit/assets_test/queries/test_metadata.py - 12 tests for _apply_metadata_filter - tests-unit/assets_test/queries/test_tags.py - 23 tests All 71 unit tests pass. Existing integration tests unaffected. Amp-Thread-ID: https://ampcode.com/threads/T-019c24bb-475b-7442-9ff9-8288edea3345 Co-authored-by: Amp <amp@ampcode.com>	2026-02-03 13:08:04 -08:00
Jedrzej Kosinski	6ea8c128a3	Assets Part 2 - add more endpoints (#12125)	2026-01-31 02:22:05 -05:00
Jedrzej Kosinski	acd0e53653	Make bulk_ops not use .returning to be compatible with python 3.10 and 3.11 sqlalchemy (#11839)	2026-01-13 00:15:24 -05:00
Jedrzej Kosinski	ec0a832acb	Add workaround for hacky nodepack(s) that edit folder_names_and_paths to have values with tuples of more than 2. Other things could potentially break with those nodepack(s), so I will hunt for the guilty nodepack(s) now. (#11755)	2026-01-08 22:49:12 -08:00
Jedrzej Kosinski	1dc3da6314	Add most basic Asset support for models (#11315) ... * Brought over minimal elements from PR 10045 to reproduce seed_assets and register_assets_system without adding anything to the DB or server routes yet, for now making everything sync (can introduce async once everything is cleaned up and brought over) * Added db script to insert assets stuff, cleaned up some code; assets (models) now get added/rescanned * Added support for 5 http endpoints for assets * Replaced Optional with | None in schemas_in.py and schemas_out.py * Remove two routes that will not be relevant yet in this PR: HEAD /api/assets/hash/<hash> and PUT /api/assets/<id>/preview * Remove some functions the two deleted endpoints were using * Don't show assets scan message upon calling /object_info endpoint * removed unsued import to satisfy ruff * Simplified hashing function tpye hint and _hash_file_obj * Satisfied ruff	2026-01-08 22:21:51 -05:00