# Background Asset Seeder Design Document

## Problem Statement

The `seed_assets` function in `app/assets/scanner.py` scans filesystem directories and imports assets into the database. Currently it runs synchronously, which causes two problems:

1. **Startup blocking**: When ComfyUI starts, `setup_database()` calls `seed_assets(["models"])` synchronously. If the models directory contains thousands of files, startup is delayed significantly before the UI becomes available.

2. **API request blocking**: The `POST /api/assets/seed` endpoint runs synchronously, blocking the HTTP request until scanning completes. For large directories, this causes request timeouts and a poor user experience.

## Goals

- Move asset scanning to a background thread so startup and API requests return immediately
- Provide visibility into scan progress via API and WebSocket events
- Support graceful cancellation of in-progress scans
- Maintain backward compatibility for tests that rely on synchronous behavior
- Ensure thread safety when accessing shared state

## Non-Goals

- Targeted/priority scanning of specific paths (addressed separately via synchronous `scan_paths()`)
- Parallel scanning across multiple threads
- Persistent scan state across restarts

---

## Architecture Overview

### Component: AssetSeeder Singleton

A new `AssetSeeder` class in `app/assets/seeder.py` manages background scanning with the following responsibilities:

- Owns a single background `threading.Thread` for scanning work
- Tracks scan state and progress in a thread-safe manner
- Provides cancellation support via `threading.Event`
- Emits WebSocket events for UI progress updates

### State Machine

```
IDLE ──start()──► RUNNING ──(completes)──► IDLE
                     │
                  cancel()
                     │y
                     ▼
                CANCELLING ──(thread exits)──► IDLE
```

### Integration Points

| Component | Change |
|-----------|--------|
| `main.py` | Call `asset_seeder.start()` (non-blocking) instead of `seed_assets()` |
| `main.py` | Call `asset_seeder.shutdown()` in `finally` block alongside `cleanup_temp()` |
| API routes | New endpoints for status and cancellation |
| WebSocket | Emit progress events during scanning |
| Test helper | Use `?wait=true` query param for synchronous behavior |

---

## Tasks

### Task 1: Create AssetSeeder Class

**Description**: Implement the core `AssetSeeder` singleton class with thread management, state tracking, and cancellation support.

**Acceptance Criteria**:
- [ ] `AssetSeeder` is a singleton accessible via module-level instance
- [ ] `State` enum with values: `IDLE`, `RUNNING`, `CANCELLING`
- [ ] `start(roots)` method spawns a daemon thread and returns immediately
- [ ] `start()` is idempotent—calling while already running is a no-op and returns `False`
- [ ] `cancel()` method signals the thread to stop gracefully
- [ ] `wait(timeout)` method blocks until thread completes or timeout expires
- [ ] `get_status()` returns current state and progress information
- [ ] All state access is protected by `threading.Lock`
- [ ] Thread creates its own database sessions (no session sharing across threads)
- [ ] Progress tuple tracks: `(scanned, total, created, skipped)`
- [ ] Errors during scanning are captured and available via `get_status()`

### Task 2: Add Cancellation Checkpoints

**Description**: Modify the scanning logic to check for cancellation between batches, allowing graceful early termination.

**Acceptance Criteria**:
- [ ] `threading.Event` is checked between batch operations
- [ ] When cancellation is requested, current batch completes before stopping
- [ ] Partial progress is committed (assets already scanned remain in database)
- [ ] State transitions to `IDLE` after cancellation completes
- [ ] Cancellation is logged with partial progress statistics

### Task 3: Update Startup Integration

**Description**: Modify `main.py` to use non-blocking asset seeding at startup.

**Acceptance Criteria**:
- [ ] `setup_database()` calls `asset_seeder.start(roots=["models"])` instead of `seed_assets()`
- [ ] Startup proceeds immediately without waiting for scan completion
- [ ] `asset_seeder.shutdown()` called in `finally` block alongside `cleanup_temp()`
- [ ] `--disable-assets-autoscan` flag continues to skip seeding entirely
- [ ] Startup logs indicate background scan was initiated (not completed)

### Task 4: Create API Endpoints

**Description**: Add REST endpoints for triggering, monitoring, and cancelling background scans.

**Endpoints**:

#### POST /api/assets/seed
Trigger a background scan for specified roots.

**Acceptance Criteria**:
- [ ] Accepts `{"roots": ["models", "input", "output"]}` in request body
- [ ] Returns `202 Accepted` with `{"status": "started"}` when scan begins
- [ ] Returns `409 Conflict` with `{"status": "already_running"}` if scan in progress
- [ ] Supports `?wait=true` query param for synchronous behavior (blocks until complete)
- [ ] With `?wait=true`, returns `200 OK` with final statistics on completion

#### GET /api/assets/seed/status
Get current scan status and progress.

**Acceptance Criteria**:
- [ ] Returns `{"state": "IDLE|RUNNING|CANCELLING", "progress": {...}, "errors": [...]}`
- [ ] Progress object includes: `scanned`, `total`, `created`, `skipped`
- [ ] When idle, progress reflects last completed scan (or null if never run)
- [ ] Errors array contains messages from any failures during last/current scan

#### POST /api/assets/seed/cancel
Request cancellation of in-progress scan.

**Acceptance Criteria**:
- [ ] Returns `200 OK` with `{"status": "cancelling"}` if scan was running
- [ ] Returns `200 OK` with `{"status": "idle"}` if no scan was running
- [ ] Cancellation is graceful—does not corrupt database state

### Task 5: Add WebSocket Progress Events

**Description**: Emit WebSocket events during scanning so the UI can display progress.

**Acceptance Criteria**:
- [ ] Event type: `assets.seed.started` with `{"roots": [...], "total": N}`
- [ ] Event type: `assets.seed.progress` with `{"scanned": N, "total": M, "created": C}`
- [ ] Event type: `assets.seed.completed` with final statistics
- [ ] Event type: `assets.seed.cancelled` if scan was cancelled
- [ ] Event type: `assets.seed.error` if scan failed with error message
- [ ] Progress events emitted at reasonable intervals (not every file, ~every 100 files or 1 second)

### Task 6: Update Test Helper

**Description**: Modify the test helper to use synchronous behavior via query parameter.

**Acceptance Criteria**:
- [ ] `trigger_sync_seed_assets()` uses `?wait=true` query parameter
- [ ] Tests continue to pass with synchronous blocking behavior
- [ ] Remove artificial `time.sleep(0.2)` delay (no longer needed with `wait=true`)

### Task 7: Unit Tests for AssetSeeder

**Description**: Add unit tests covering the seeder state machine and thread safety.

**Acceptance Criteria**:
- [ ] Test: `start()` transitions state from IDLE to RUNNING
- [ ] Test: `start()` while RUNNING returns False (idempotent)
- [ ] Test: `cancel()` transitions state from RUNNING to CANCELLING
- [ ] Test: `wait()` blocks until thread completes
- [ ] Test: `wait(timeout)` returns False if timeout expires
- [ ] Test: `get_status()` returns correct progress during scan
- [ ] Test: Concurrent `start()` calls are safe (only one thread spawned)
- [ ] Test: Scan commits partial progress on cancellation
- [ ] Test: Database errors are captured in status, don't crash thread

---

## Thread Safety Considerations

| Shared Resource | Protection Strategy |
|-----------------|---------------------|
| `_state` enum | Protected by `threading.Lock` |
| `_progress` tuple | Protected by `threading.Lock` |
| `_errors` list | Protected by `threading.Lock` |
| `_thread` reference | Protected by `threading.Lock` |
| `_cancel_event` | `threading.Event` (inherently thread-safe) |
| Database sessions | Created per-operation inside thread (no sharing) |

## Error Handling

- Database connection failures: Log error, set state to IDLE, populate errors list
- Individual file scan failures: Log warning, continue with next file, increment error count
- Thread crashes: Caught by outer try/except, state reset to IDLE, error captured

## Future Considerations

- **Priority queue**: If targeted scans need to be non-blocking in the future, the seeder could be extended with a priority queue
- **Persistent state**: Scan progress could be persisted to allow resume after restart
- **Parallel scanning**: Multiple threads could scan different roots concurrently (requires careful session management)
- **Throttling**: If scanning competes with generation (e.g., disk I/O contention when hashing large files), add configurable sleep between batches. Currently considered low risk since scanning is I/O-bound and generation is GPU-bound.