mirror of
https://github.com/Comfy-Org/ComfyUI-Manager.git
synced 2026-07-26 10:07:38 +08:00
fix(security): harden CSRF with Content-Type gate and expand E2E coverage (#2818)
Defense-in-depth over GET→POST alone: reject the three CORS-safelisted simple-form Content-Types (x-www-form-urlencoded, multipart/form-data, text/plain) on 16 no-body POST handlers (glob + legacy) to block <form method=POST> CSRF that bypasses method-only gating. Move comfyui_switch_version to a JSON body so the preflight requirement applies. Split db_mode/policy/update/channel_url_list into GET(read) + POST(write). Tighten do_fix (high → high+) and gate three previously-ungated config setters at middle. Resynchronize openapi.yaml (27 paths, 30 operations, ComfyUISwitchVersionParams as a shared $ref component). Add E2E harness variants, Playwright config, CSRF/secgate suites, 39-endpoint coverage, and a CHANGELOG. Breaking: legacy per-op POST routes (install/uninstall/fix/disable/update/ reinstall/abort_current) are removed; callers already use queue/batch. Legacy /manager/notice (v1) is removed; /v2/manager/notice is retained. Reported-by: XlabAI Team of Tencent Xuanwu Lab CVSS: 8.1 (AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:H/A:H)
This commit is contained in:
Dr.Lt.Data
@@ -0,0 +1,355 @@
|
||||
# API Coverage Matrix — pytest E2E + Playwright
|
||||
|
||||
**Date**: 2026-04-20
|
||||
**Worklist**: `wl-afbf982ffe41`
|
||||
**Checklist**: `cl-20260420-wl-afbf982ff`
|
||||
**Scope**: 39 unique (method, path) endpoints across glob and legacy managers.
|
||||
**Sources**: 4 member checklist YAMLs (gteam-teng 10 · gteam-reviewer 10 · gteam-dev 10 · gteam-dbg 9).
|
||||
|
||||
---
|
||||
|
||||
## 1. Coverage Summary
|
||||
|
||||
| Axis | Y | I | N | P | NA | Total |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| **pytest E2E** | 38 | 1 | 0 | — | — | 39 |
|
||||
| **Playwright** | 17 | — | 1 | 14 | 7 | 39 |
|
||||
|
||||
### Code legend
|
||||
|
||||
| Code | pytest meaning | Playwright meaning |
|
||||
|------|----------------|--------------------|
|
||||
| **Y** | Direct positive-path test exists | UI trigger exists AND a spec exercises it |
|
||||
| **I** | Indirect only (e.g. CSRF-reject test, no positive assertion) | — |
|
||||
| **N** | No coverage | Endpoint has no UI surface AND no spec covers it (1 case, wi-009 internal-only) |
|
||||
| **P** | — | UI trigger exists but NO spec exercises it (PENDING Playwright) |
|
||||
| **NA** | — | Endpoint has no UI surface at all (backend/CLI/gating only) |
|
||||
|
||||
### Effective pytest ceiling
|
||||
|
||||
Y (direct) + I (indirect) = **39 / 39 = 100%** — post-WI-UU every endpoint
|
||||
has automated pytest coverage. The 6 legacy-only GET endpoints
|
||||
(wi-031/032/033/034/035/036) that were `N` at matrix creation have been
|
||||
closed as `Y (WI-TT)` via direct positive-path tests in
|
||||
`tests/e2e/test_e2e_legacy_endpoints.py` (§22 of
|
||||
`e2e_verification_audit.md`). wi-039 (POST /v2/manager/queue/batch) was
|
||||
closed as `Y (WI-UU)` via `TestLegacyQueueBatch` in the same file — empty
|
||||
`{}` payload exercises request-parse → action loop → finalize →
|
||||
worker-lock release → JSON serialize. Only 1 I-rated row remains: wi-027
|
||||
POST /v2/snapshot/restore, which stays I by intentional design (the
|
||||
endpoint is destructive and is covered only behind a skip-by-default
|
||||
marker; upgrading it to Y would require a reversible snapshot fixture).
|
||||
|
||||
Count progression: matrix-creation baseline Y=29/I=4/N=6 → post-WI-YY
|
||||
Y=31/I=2/N=6 → post-WI-TT Y=37/I=2/N=0 → **post-WI-UU Y=38/I=1/N=0**.
|
||||
|
||||
### Playwright P = systemic gap
|
||||
|
||||
10 / 39 = **26%** of endpoints have a UI trigger that Playwright never
|
||||
exercises. Originally 18/39 = 46%; WI-VV closed 4 LOW-risk P items
|
||||
(wi-001 / wi-005 / wi-017 / wi-021) via real UI-click tests; WI-WW closed
|
||||
5 MED items via mock-based UI→API wiring tests; WI-WW.2 closed wi-015 via
|
||||
a 2-hop mock. **WI-YY** then replaced 2 of the WI-WW mocks (wi-020, wi-024)
|
||||
with real pytest E2E execution — the mocks were removed and the Playwright
|
||||
column reverted to P (UI-click path unexercised — pytest covers the
|
||||
backend contract). The remaining 10 P items are: wi-014/037/038
|
||||
(retained as WI-WW mocks — real execution requires `high+` security gate
|
||||
that fails at default `security_level=normal`, needs a permissive
|
||||
harness — scope for WI-YY.2), plus 7 other source-checklist-classified
|
||||
P items outside the WI-VV/WW/WW.2/YY scope.
|
||||
|
||||
**Honesty note on mock-based closures**: rows marked `Y (WI-WW-mock)`
|
||||
assert UI→API wiring only — request URL + method + payload shape.
|
||||
They do NOT assert backend handler behavior, which pytest covers via
|
||||
positive-path tests. A regression that kept the UI firing correctly
|
||||
but broke the backend would not be caught by the mock tests alone.
|
||||
|
||||
---
|
||||
|
||||
## 2. Tier Distribution
|
||||
|
||||
39 endpoints split across three registration tiers:
|
||||
|
||||
| Tier | Count | Definition |
|
||||
|------|------:|------------|
|
||||
| Shared | 29 | Registered in BOTH `glob/manager_server.py` AND `legacy/manager_server.py` |
|
||||
| glob-only | 1 | Registered only in `glob/manager_server.py` |
|
||||
| legacy-only | 9 | Registered only in `legacy/manager_server.py` |
|
||||
|
||||
> **Note on dispatch**: the WI-SS-E dispatch text cited `Shared 28, glob-only 2,
|
||||
> legacy-only 9` but source-code verification (grep of `@routes.(get\|post)` in
|
||||
> both managers) yields 29/1/9. Only `POST /v2/manager/queue/task` is confirmed
|
||||
> glob-only by the audit (`reports/e2e_verification_audit.md:299`). This
|
||||
> matrix reports the verified counts.
|
||||
|
||||
### Tier × coverage crosstab
|
||||
|
||||
| Tier | pytest Y | pytest I | pytest N | PW Y | PW P | PW NA | PW N |
|
||||
|------|---------:|---------:|---------:|-----:|-----:|------:|-----:|
|
||||
| Shared (29) | 28 | 1 | 0 | 15 | 7 | 6 | 1 |
|
||||
| glob-only (1) | 1 | 0 | 0 | 0 | 0 | 1 | 0 |
|
||||
| legacy-only (9) | 9 | 0 | 0 | 2 | 7 | 0 | 0 |
|
||||
| **Total** | **38** | **1** | **0** | **17** | **14** | **7** | **1** |
|
||||
|
||||
**Observations** (post-WI-UU):
|
||||
- Legacy-only (9) pytest coverage is now **fully direct**: 9 Y + 0 I +
|
||||
0 N. WI-TT closed 6 N → Y (wi-031/032/033/034/035/036). WI-YY-real
|
||||
closed 2 I → Y (wi-037/038 via the permissive harness). WI-UU closed
|
||||
the final I → Y (wi-039 via `TestLegacyQueueBatch`). The remaining
|
||||
weakness for this tier is Playwright — 7/9 are Playwright-P.
|
||||
- Shared (29) holds the sole remaining pytest-I (wi-027 snapshot/restore,
|
||||
intentional skip-by-default design). 0 pytest-N, balanced Y/P on
|
||||
Playwright.
|
||||
- glob-only has only 1 endpoint (queue/task) and it is Playwright-NA by
|
||||
design — the legacy UI uses `queue/batch` (wi-039) instead.
|
||||
|
||||
---
|
||||
|
||||
## 3. Full Matrix (39 rows, sorted by wi-id)
|
||||
|
||||
| wi | METHOD path | tier | pytest | Playwright | gap |
|
||||
|---|---|---|---|---|---|
|
||||
| wi-001 | GET /v2/comfyui_manager/comfyui_versions | shared | Y | Y (WI-VV) | 🟢 closed — legacy-ui-manager-menu.spec.ts asserts Switch ComfyUI click → GET returns non-empty versions list |
|
||||
| wi-002 | GET /v2/customnode/fetch_updates | shared | Y | NA | 🟢 none (deprecated 410, no JS trigger) |
|
||||
| wi-003 | GET /v2/customnode/getmappings | shared | Y | Y | 🟢 none (dual coverage) |
|
||||
| wi-004 | GET /v2/customnode/installed | shared | Y | Y | 🟢 none (dual coverage) |
|
||||
| wi-005 | GET /v2/manager/channel_url_list | shared | Y | Y (WI-VV) | 🟢 closed — legacy-ui-manager-menu.spec.ts polls channel combo for populated options via stable selector `select[title^="Configure the channel"]` |
|
||||
| wi-006 | GET /v2/manager/db_mode | shared | Y | Y | 🟢 none (dual coverage) |
|
||||
| wi-007 | GET /v2/manager/is_legacy_manager_ui | shared | Y | NA | 🟢 none (server-side gating flag) |
|
||||
| wi-008 | GET /v2/manager/policy/update | shared | Y | P | 🟢 LOW — add Playwright assertion (pytest fully covers contract) |
|
||||
| wi-009 | GET /v2/manager/queue/history | shared | Y | N | 🟢 none (no UI surface, internal API only) |
|
||||
| wi-010 | GET /v2/manager/queue/history_list | shared | Y | NA | 🟢 none (backend-only, not surfaced in UI) |
|
||||
| wi-011 | GET /v2/manager/queue/status | shared | Y | Y | 🟢 none (dual coverage) |
|
||||
| wi-012 | GET /v2/manager/version | shared | Y | P | 🟢 LOW — add version-string assertion to bootstrap spec |
|
||||
| wi-013 | GET /v2/snapshot/get_current | shared | Y | P | 🟢 none — 3rd-party share extensions only, out-of-scope for legacy-ui |
|
||||
| wi-014 | POST /v2/comfyui_manager/comfyui_switch_version | shared | Y (WI-YY-real) | P | 🟢 pytest closed — test_e2e_legacy_real_ops.py::TestSwitchComfyuiSelfSwitch executes real POST via permissive-harness (security_level=normal-) with a no-op self-switch (ver=<current>). Playwright mock REMOVED. |
|
||||
| wi-015 | POST /v2/customnode/import_fail_info | shared | Y (WI-YY-real) | P | 🟢 pytest closed — test_e2e_legacy_real_ops.py::TestImportFailInfoReal pre-seeds `ComfyUI-YoloWorld-EfficientSAM` via git clone (no pip install), scan captures ImportError in cm_global.error_dict, warmup via `/v2/customnode/import_fail_info_bulk` populates active_nodes, then POST single-endpoint with cnr_id=directory-basename returns the captured `{name, path, msg}` payload. Playwright mock REMOVED (spec file deleted). |
|
||||
| wi-016 | POST /v2/customnode/import_fail_info_bulk | shared | Y | NA | 🟢 none (server-internal/CLI-only) |
|
||||
| wi-017 | POST /v2/manager/channel_url_list | shared | Y | Y (WI-VV) | 🟢 closed — legacy-ui-manager-menu.spec.ts selects alternate option, intercepts POST → 200, restores original in finally |
|
||||
| wi-018 | POST /v2/manager/db_mode | shared | Y | Y | 🟢 none (dual coverage via UI close-reopen round-trip) |
|
||||
| wi-019 | POST /v2/manager/policy/update | shared | Y | Y | 🟢 none (dual coverage) |
|
||||
| wi-020 | POST /v2/manager/queue/install_model | shared | Y (WI-YY-real) | P | 🟢 pytest closed — test_e2e_legacy_real_ops.py::TestInstallModelRealDownload downloads the real TAEF1 Decoder (~4.7MB from github.com/madebyollin/taesd raw), polls for disk artifact, asserts size > 1MB, teardown deletes file. Playwright mock REMOVED in WI-YY; UI-click path still unexercised (P) — scope for WI-YY.2 if needed. |
|
||||
| wi-021 | POST /v2/manager/queue/reset | shared | Y | Y (WI-VV) | 🟢 closed — legacy-ui-manager-menu.spec.ts exercises endpoint via `page.request.post` (UI-click path unsafe at idle: restart_stop_button at idle triggers rebootAPI, not queue/reset; `.cn-manager-stop` / `.model-manager-stop` are display:none). Asserts 200 + queue/status still callable post-reset. |
|
||||
| wi-022 | POST /v2/manager/queue/start | shared | Y | NA | 🟢 none (server/test-only) |
|
||||
| wi-023 | POST /v2/manager/queue/update_all | shared | Y | NA | 🟢 LOW — UI uses queue/batch not this endpoint (possibly CLI-only; confirm intent) |
|
||||
| wi-024 | POST /v2/manager/queue/update_comfyui | shared | Y (WI-YY-real) | P | 🟢 pytest closed — test_e2e_legacy_real_ops.py::TestUpdateComfyuiQueued asserts direct endpoint returns 200 at default security_level (no gate — legacy/manager_server.py:1572-1576). Handler just queues an "update-comfyui" entry; triggering git pull would require a subsequent /queue/batch call (explicitly avoided to preserve test-env git state). COMFYUI_MANAGER_SKIP_MANAGER_REQUIREMENTS=1 safety belt exported at server startup covers pip install runaway risk. Playwright mock REMOVED in WI-YY. |
|
||||
| wi-025 | POST /v2/manager/reboot | shared | Y | P | 🟢 none — visibility checked; click omitted for safety (would kill test server) |
|
||||
| wi-026 | POST /v2/snapshot/remove | shared | Y | Y | 🟢 none (dual coverage) |
|
||||
| wi-027 | POST /v2/snapshot/restore | shared | I | P | 🟡 add pytest coverage behind skip-by-default (reversible via saved snapshot); Playwright restore also missing |
|
||||
| wi-028 | POST /v2/snapshot/save | shared | Y | Y | 🟢 none (strong dual coverage) |
|
||||
| wi-029 | GET /v2/snapshot/getlist | shared | Y | Y | 🟢 none (dual coverage) |
|
||||
| wi-030 | POST /v2/manager/queue/task | glob-only | Y | NA | 🟢 LOW — glob-UI Playwright harness needed to cover queue/task from UI tier |
|
||||
| wi-031 | GET /customnode/alternatives | legacy-only | Y (WI-TT) | P | 🟢 closed — test_e2e_legacy_endpoints.py (§22 of e2e_verification_audit) asserts positive-path GET with `mode=local` |
|
||||
| wi-032 | GET /v2/customnode/disabled_versions/{node_name} | legacy-only | Y (WI-TT) | P | 🟢 closed — test_e2e_legacy_endpoints.py (§22) asserts disabled-version list schema |
|
||||
| wi-033 | GET /v2/customnode/getlist | legacy-only | Y (WI-TT) | Y | 🟢 closed — test_e2e_legacy_endpoints.py (§22) asserts schema (`channel`/`node_packs`) + mode param variants |
|
||||
| wi-034 | GET /v2/customnode/versions/{node_name} | legacy-only | Y (WI-TT) | Y | 🟢 closed — test_e2e_legacy_endpoints.py (§22) asserts schema + 404 |
|
||||
| wi-035 | GET /v2/externalmodel/getlist | legacy-only | Y (WI-TT) | Y | 🟢 closed — test_e2e_legacy_endpoints.py (§22) asserts `?mode=local` schema + non-empty list |
|
||||
| wi-036 | GET /v2/manager/notice | legacy-only | Y (WI-TT) | P | 🟢 closed — test_e2e_legacy_endpoints.py (§22) asserts notice payload (200 + dict) |
|
||||
| wi-037 | POST /v2/customnode/install/git_url | legacy-only | Y (WI-YY-real) | P | 🟢 pytest closed — test_e2e_legacy_real_ops.py::TestInstallViaGitUrlRealClone executes real POST via permissive-harness cloning `nodepack-test1-do-not-install` (same test-fixture repo used by tests/cli/test_uv_compile.py); verifies custom_nodes/<dir>/.git exists; teardown rm -rf. Playwright mock REMOVED. |
|
||||
| wi-038 | POST /v2/customnode/install/pip | legacy-only | Y (WI-YY-real) | P | 🟢 pytest closed — test_e2e_legacy_real_ops.py::TestInstallPipRealExecute executes real POST via permissive-harness with trusted `text-unidecode` pkg; asserts install-scripts.txt lazy reservation (handler uses `#FORCE` prefix at manager_core.py:2370 → reserve_script schedules pip install for next startup, not synchronous). Playwright mock REMOVED. |
|
||||
| wi-039 | POST /v2/manager/queue/batch | legacy-only | Y (WI-UU) | Y | 🟢 closed via TestLegacyQueueBatch empty-`{}` payload positive-path (exercises request-parse → action loop → finalize → worker-lock release → JSON serialize pipeline); response shape `{failed: [...]}` status 200; landed in `tests/e2e/test_e2e_legacy_endpoints.py` (§22 of e2e_verification_audit) |
|
||||
|
||||
---
|
||||
|
||||
## 4. Priority Gap List
|
||||
|
||||
### 🔴 HIGH — ALL CLOSED (0 items)
|
||||
|
||||
_All 🔴 HIGH gaps have been closed. WI-TT closed the 6 pytest-N items
|
||||
(wi-031/032/033/034/035/036 — see LOW-closed-WI-TT). WI-YY-real promoted
|
||||
wi-037/038 from I to Y via the permissive harness (see
|
||||
LOW-closed-real-permissive). WI-UU closed the final high-fanout indirect
|
||||
item — wi-039 (POST /v2/manager/queue/batch) — via
|
||||
`TestLegacyQueueBatch`; see LOW-closed-WI-UU below._
|
||||
|
||||
### 🟡 MEDIUM — Playwright P with real UI surface
|
||||
|
||||
All 10 original MED items have been closed across WI-VV / WI-WW / WI-WW.2. No remaining 🟡 items at the legacy-UI surface.
|
||||
|
||||
### 🟢 LOW-closed-real (4 items via WI-VV — real UI-click)
|
||||
|
||||
- wi-001 GET comfyui_versions — 'Switch ComfyUI' button → legacy-ui-manager-menu.spec.ts
|
||||
- wi-005 GET channel_url_list — channel combo populate → legacy-ui-manager-menu.spec.ts
|
||||
- wi-017 POST channel_url_list — channel combo change event → legacy-ui-manager-menu.spec.ts
|
||||
- wi-021 POST queue/reset — idle POST via `page.request` (UI-click path unsafe at idle) → legacy-ui-manager-menu.spec.ts
|
||||
|
||||
### 🟢 LOW-closed-mock (removed in WI-YY)
|
||||
|
||||
Previously 3 items (wi-014/037/038) were covered by WI-WW mock tests.
|
||||
All three have been PROMOTED to real pytest E2E via the permissive
|
||||
harness (see LOW-closed-real-permissive below). The `high+` gate
|
||||
remains the production security contract — it's the supported
|
||||
operator-configured downgrade to `normal-` that enables these
|
||||
features in trusted environments, which is exactly what the harness
|
||||
reproduces.
|
||||
|
||||
### 🟢 LOW-closed-mock-2hop (retired — promoted to real E2E via WI-YY.3)
|
||||
|
||||
Originally wi-015 was covered via a 2-hop Playwright mock (WI-WW.2).
|
||||
WI-YY.3 replaced this with real pytest E2E using a pre-seeded broken
|
||||
pack (see LOW-closed-real-broken-pack-preseed below). The mock spec
|
||||
file `tests/playwright/legacy-ui-mock-install.spec.ts` has been
|
||||
DELETED — all 6 items it once covered now have real pytest E2E
|
||||
coverage (via default / permissive / broken-pack fixtures).
|
||||
|
||||
### 🟢 LOW-closed-real (2 items via WI-YY — REAL pytest E2E at default security)
|
||||
|
||||
- wi-020 POST install_model — TestInstallModelRealDownload (real 4.7MB TAEF1 download + disk-artifact verify + teardown) → test_e2e_legacy_real_ops.py
|
||||
- wi-024 POST update_comfyui — TestUpdateComfyuiQueued (direct endpoint POST returns 200; worker-trigger intentionally deferred to preserve test-env git state) → test_e2e_legacy_real_ops.py
|
||||
|
||||
### 🟢 LOW-closed-real-permissive (3 items via WI-YY — REAL pytest E2E at normal- security)
|
||||
|
||||
Permissive harness (`start_comfyui_permissive.sh`) patches config.ini
|
||||
`security_level = normal-` so `high+` gates pass. All inputs are
|
||||
HARDCODED TRUSTED constants — never derived from test input or env:
|
||||
|
||||
- wi-014 POST comfyui_switch_version — TestSwitchComfyuiSelfSwitch (self-switch no-op: GET current version → POST ver=<current>) → test_e2e_legacy_real_ops.py
|
||||
- wi-037 POST install/git_url — TestInstallViaGitUrlRealClone (real clone of `nodepack-test1-do-not-install` → verify .git dir → rm -rf teardown) → test_e2e_legacy_real_ops.py
|
||||
- wi-038 POST install/pip — TestInstallPipRealExecute (POST text-unidecode → verify install-scripts.txt reservation; lazy schedule per manager_core.py:2370 `#FORCE` prefix → reserve_script) → test_e2e_legacy_real_ops.py
|
||||
|
||||
### 🟢 LOW-closed-real-broken-pack-preseed (1 item via WI-YY.3 — REAL pytest E2E with state seeding)
|
||||
|
||||
Pre-seed fixture (`comfyui_with_broken_pack`) clones a known-broken
|
||||
pack into custom_nodes/ BEFORE server start (NO pip install of its
|
||||
deps — import must fail). Server scan captures the ImportError into
|
||||
cm_global.error_dict (prestartup_script.py:302-305). Test warms up
|
||||
state via `/v2/customnode/import_fail_info_bulk` (which calls
|
||||
reload + get_custom_nodes), then POSTs single-endpoint with the
|
||||
DIRECTORY-BASENAME cnr_id. Teardown rm -rf the seed.
|
||||
|
||||
- wi-015 POST import_fail_info — TestImportFailInfoReal::test_import_fail_info_returns_error (cnr_id=`ComfyUI-YoloWorld-EfficientSAM` → 200 + {name, path, msg} with real traceback) + test_import_fail_info_unknown_cnr_id_returns_400 (control) → test_e2e_legacy_real_ops.py
|
||||
|
||||
### 🟢 LOW-closed-WI-TT (6 items — pytest N→Y via direct positive-path)
|
||||
|
||||
All 6 legacy-only GET endpoints that were `pytest=N` at matrix creation
|
||||
have been closed via direct positive-path tests landed in
|
||||
`tests/e2e/test_e2e_legacy_endpoints.py` (Section 22 of
|
||||
`e2e_verification_audit.md`). This lifts pytest effective ceiling from
|
||||
33/39 = 85% to **39/39 = 100%**.
|
||||
|
||||
- wi-031 GET /customnode/alternatives — closed (mode=local schema + list)
|
||||
- wi-032 GET /v2/customnode/disabled_versions/{node_name} — closed (disabled-version list)
|
||||
- wi-033 GET /v2/customnode/getlist — closed (channel / node_packs + mode variants)
|
||||
- wi-034 GET /v2/customnode/versions/{node_name} — closed (schema + 404)
|
||||
- wi-035 GET /v2/externalmodel/getlist — closed (?mode=local schema + non-empty)
|
||||
- wi-036 GET /v2/manager/notice — closed (notice payload / 200 + dict)
|
||||
|
||||
### 🟢 LOW-closed-WI-UU (1 item — high-fanout pytest I→Y via direct positive-path)
|
||||
|
||||
The final 🔴 HIGH item (wi-039 POST /v2/manager/queue/batch,
|
||||
high-fanout over install_model / update_all / update_comfyui) has been
|
||||
closed via direct positive-path in
|
||||
`tests/e2e/test_e2e_legacy_endpoints.py` (§22 of
|
||||
`e2e_verification_audit.md`). This lifts pytest to **38 Y + 1 I + 0 N**;
|
||||
the last I is wi-027 snapshot/restore, retained by intentional design.
|
||||
|
||||
- wi-039 POST /v2/manager/queue/batch — closed (TestLegacyQueueBatch empty-`{}` payload; exercises request-parse → action loop → finalize → worker-lock release → JSON serialize; response shape `{failed: [...]}` status 200)
|
||||
|
||||
**Note**: wi-027 POST snapshot/restore is MED on Playwright (UI trigger at
|
||||
`snapshot.js:12`) and HIGH on pytest (intentionally untested as destructive;
|
||||
needs skip-by-default marker).
|
||||
|
||||
### 🟢 LOW / adequate-with-rationale
|
||||
|
||||
- wi-023 POST queue/update_all — UI uses `/queue/batch` not this endpoint (possibly CLI-only)
|
||||
- wi-025 POST reboot — click intentionally omitted; clicking would kill the test server mid-run
|
||||
- wi-022 POST queue/start, wi-010 history_list, wi-030 queue/task — server/test-only or glob-UI N/A
|
||||
- wi-016 POST import_fail_info_bulk — backend/CLI-only path
|
||||
- wi-002 GET fetch_updates — deprecated 410, no JS trigger
|
||||
- wi-009 GET queue/history — internal API only (no UI surface)
|
||||
- wi-013 GET snapshot/get_current — 3rd-party share extensions only (out-of-scope for legacy-ui)
|
||||
- wi-008 GET policy/update, wi-012 GET manager/version — pytest fully covers contract; Playwright add would be nice-to-have
|
||||
|
||||
---
|
||||
|
||||
## 5. Key Systemic Observations
|
||||
|
||||
1. **Playwright P = 14 / 39 = 36%** (post WI-VV+WW+WW.2+YY+YY.3; was 18/39=46%).
|
||||
Coverage evolution: WI-VV closed 4 LOW-risk items via real UI-click;
|
||||
WI-WW closed 5 MED items via mock-based UI→API wiring; WI-WW.2
|
||||
closed wi-015 via 2-hop mock. **WI-YY** then promoted 5 of the 6
|
||||
mocks (wi-014/020/024/037/038) to REAL pytest E2E — 2 run under
|
||||
the default-security fixture (wi-020 real TAEF1 download,
|
||||
wi-024 direct endpoint POST) and 3 run under a permissive-harness
|
||||
fixture (security_level=normal-) using HARDCODED TRUSTED inputs
|
||||
(wi-014 self-switch no-op, wi-037 nodepack-test1-do-not-install,
|
||||
wi-038 text-unidecode lazy-install). Playwright mocks for these 5
|
||||
items were REMOVED; the Playwright column reverted to P (UI-click
|
||||
not yet exercised in Playwright — backend contract now fully
|
||||
covered by pytest). wi-015 remains as a 2-hop mock (legitimate:
|
||||
pytest negative-path tests cover the 400 branch; UI→API wiring is
|
||||
asserted via mock). COMFYUI_MANAGER_SKIP_MANAGER_REQUIREMENTS=1 is
|
||||
exported as the safety belt in start_comfyui.sh for any
|
||||
install/update flow. Permissive harness security note: these
|
||||
endpoints exist to serve a supported feature — operators in a
|
||||
trusted environment lower security_level to `normal-`/`weak` to
|
||||
enable them. The 200 path IS the feature; testing it requires
|
||||
exactly this configuration, with TRUSTED fixed inputs (never user
|
||||
input).
|
||||
|
||||
2. **pytest effective ceiling = 39 / 39 = 100%** (Y=38 + I=1, N=0)
|
||||
post-WI-UU. WI-TT closed 6 N → Y
|
||||
(wi-031/032/033/034/035/036); WI-UU closed the final high-fanout
|
||||
I → Y (wi-039 via `TestLegacyQueueBatch`). The sole remaining I
|
||||
row is **wi-027 POST /v2/snapshot/restore** — intentional design,
|
||||
NOT a gap: the endpoint is destructive and sits behind a
|
||||
skip-by-default marker; upgrading it to Y requires a reversible
|
||||
snapshot fixture, scoped as an optional WI-XX.
|
||||
|
||||
3. **Legacy-only tier — pytest now fully direct**:
|
||||
- 0 pytest-N, 0 pytest-I, 9 pytest-Y = 9/9 direct coverage.
|
||||
- 7 / 9 legacy-only endpoints remain Playwright-P — the audit focus
|
||||
shifts from pytest coverage to UI-surface Playwright expansion.
|
||||
|
||||
4. **Shared tier is healthy**: 0 pytest-N, 27/29 pytest-Y, 11/29 Playwright-Y.
|
||||
The 11 Shared-tier Playwright-P items are all UI-exists-but-not-tested —
|
||||
never a protocol gap.
|
||||
|
||||
5. **glob-only is structurally Playwright-NA**: The single glob-only endpoint
|
||||
(`queue/task`) has no legacy UI surface by design — the legacy UI dispatches
|
||||
through `queue/batch` (wi-039). Closing this needs a glob-UI Playwright
|
||||
harness, which is an upstream-ComfyUI scope concern.
|
||||
|
||||
---
|
||||
|
||||
## 6. Recommended Follow-up WIs
|
||||
|
||||
| WI | Scope | Closes | Priority |
|
||||
|---|---|---|---|
|
||||
| **WI-TT** | Add 6 direct positive-path pytest tests for legacy-only GET endpoints — landed in `tests/e2e/test_e2e_legacy_endpoints.py` (§22 of `e2e_verification_audit.md`); closed 2026-04-21 | wi-031, 032, 033, 034, 035, 036 | 🟢 DONE |
|
||||
| **WI-UU** | Add pytest positive-path for `POST /v2/manager/queue/batch` (high-fanout) — landed `TestLegacyQueueBatch` in `tests/e2e/test_e2e_legacy_endpoints.py` (§22 of `e2e_verification_audit.md`); closed 2026-04-21 | wi-039 | 🟢 DONE |
|
||||
| **WI-VV** | Legacy-UI Playwright — 4 LOW-risk P closures via real UI-click (closed 2026-04-20) | wi-001, 005, 017, 021 | 🟢 DONE |
|
||||
| **WI-WW** | env var skip + 5 mock-based Playwright P closures (closed 2026-04-20) | wi-014, 020, 024, 037, 038 | 🟢 DONE |
|
||||
| **WI-WW.2** | Playwright P closure for wi-015 via 2-hop mock (getlist stub + POST intercept; closed 2026-04-21) | wi-015 | 🟢 DONE |
|
||||
| **WI-YY** | Replace 5 of 6 mocks with REAL pytest E2E — default-security (wi-020, wi-024) + permissive-harness with trusted fixed inputs (wi-014 self-switch, wi-037 nodepack-test1, wi-038 text-unidecode) + env var safety belt + start_comfyui_permissive.sh harness (closed 2026-04-21) | wi-014, wi-020, wi-024, wi-037, wi-038 | 🟢 DONE |
|
||||
| **WI-YY.3** | Replace remaining mock (wi-015) with REAL pytest E2E via pre-seeded broken pack (ComfyUI-YoloWorld-EfficientSAM cloned without pip deps; warmup via import_fail_info_bulk; cnr_id=directory-basename lookup) — deleted the legacy-ui-mock-install.spec.ts file (closed 2026-04-21) | wi-015 | 🟢 DONE |
|
||||
| **WI-WW** (optional) | pytest-I → pytest-Y for install endpoints (`install/git_url`, `install/pip`) — superseded by WI-YY-real (wi-037 via TestInstallViaGitUrlRealClone, wi-038 via TestInstallPipRealExecute in test_e2e_legacy_real_ops.py) | wi-037, 038 | 🟢 DONE (via WI-YY) |
|
||||
| **WI-XX** (optional) | Skip-by-default pytest for `POST /v2/snapshot/restore` | wi-027 | 🟡 MEDIUM |
|
||||
|
||||
Post-WI-UU pytest coverage is **38/39 Y + 1/39 I = 100% effective**
|
||||
(N = 0). 0 🔴 HIGH items remain. 5 matrix rows carry the
|
||||
`Y (WI-YY-real)` annotation — real-E2E execution replacing prior
|
||||
`I`-only markers on mock-covered endpoints. The sole remaining
|
||||
pytest-I row is wi-027 POST /v2/snapshot/restore, retained by
|
||||
intentional design (destructive endpoint behind a skip-by-default
|
||||
marker; scoped as optional WI-XX for future reversible-fixture
|
||||
upgrade). Playwright is **17/39 Y = 44%** post-WI-YY.3 (from
|
||||
13/39 = 33% at matrix creation; peaked at 18/39 pre-YY.3 before
|
||||
the wi-015 mock was removed in favor of real pytest E2E via a
|
||||
pre-seeded broken pack). 6 mocks removed in total (5 via WI-YY
|
||||
+ 1 via WI-YY.3) in favor of real pytest E2E — the trade-off is
|
||||
honest real-execution coverage via pytest (with a permissive
|
||||
harness for high+ gated items using trusted fixed inputs) instead
|
||||
of mock-only UI-wiring via Playwright.
|
||||
|
||||
---
|
||||
|
||||
## 7. Source YMLs
|
||||
|
||||
| Member | File | Items |
|
||||
|--------|------|------:|
|
||||
| gteam-teng | `.claude/pair-working/checklists/cl-20260420-wl-afbf982ff/gteam-teng.yml` | 10 |
|
||||
| gteam-reviewer | `.claude/pair-working/checklists/cl-20260420-wl-afbf982ff/gteam-reviewer.yml` | 10 |
|
||||
| gteam-dev | `.claude/pair-working/checklists/cl-20260420-wl-afbf982ff/gteam-dev.yml` | 10 |
|
||||
| gteam-dbg | `.claude/pair-working/checklists/cl-20260420-wl-afbf982ff/gteam-dbg.yml` | 9 |
|
||||
| | **Total** | **39** |
|
||||
@@ -0,0 +1,267 @@
|
||||
# Report Y — Cross-Artifact Consistency Audit
|
||||
|
||||
**Generated**: 2026-04-19
|
||||
**Auditor**: gteam-doc
|
||||
**Scope**: 9 markdown files in `reports/` directory (post-Wave3 PR preparation stage)
|
||||
**Mandate**: Audit-only — fixes deferred to follow-up WIs
|
||||
**Baseline gate**: `python scripts/verify_audit_counts.py` → **PASS** (94/0/0/15/109 matches between Summary Matrix and per-file rows)
|
||||
|
||||
---
|
||||
|
||||
## 📌 Status update (WI-Z, 2026-04-19)
|
||||
|
||||
All 13 drift items identified below have been **RESOLVED** via WI-Z patch application (consolidated Y1+Y2+Y3+Y4). Final verify: `(100, 0, 0, 15, 115)` — PASS.
|
||||
|
||||
> **WI-II note (2026-04-20)**: The `test_e2e_csrf.py` function/parametrization tally recorded below as `4 functions / 29 parametrized invocations (16+2+11)` — see L47, L75, L236 — reflects the pre-WI-HH state. WI-HH removed 3 dual-purpose endpoints (`db_mode`, `policy/update`, `channel_url_list`) from the reject-GET fixture (they legitimately answer GET on the read-path and are covered only in the allow-GET class), bringing the current count to `4 functions / 26 parametrized invocations (13+2+11)`. This audit's historical figures are preserved as a time-stamped snapshot of the 2026-04-19 state; the current state is recorded in `e2e_verification_audit.md` §18, `test_contract_audit.md` §1.5, `coverage_gaps.md` CSRF-Mitigation Layer Coverage, `e2e_test_coverage.md` test_e2e_csrf.py subsection, and `verification_design.md` §10.
|
||||
|
||||
| Patch | Scope | Items resolved | Net effect |
|
||||
|-------|-------|----------------|------------|
|
||||
| **Y1** | snapshot_lifecycle audit §7 add path-traversal row | B-3, B-4, D-1 | §7: 6→7 PASS; SR3 Key gap → RESOLVED |
|
||||
| **Y2** | e2e_test_coverage.md stale sync | A-1, A-2, A-3, A-4, B-1, B-2 | Summary 119→117; customnode header 9→11; snapshot rows swapped; navigation 4→2 |
|
||||
| **Y3** | config_api audit §4 add 5 uncounted rows | B-5 | §4: 10→15 PASS (junk_value ×3 + persists_to_config_ini ×2) |
|
||||
| **Y4** | do_fix security level middle→high | C-1, C-2, C-3 | 3 locations updated: endpoint_scenarios L18/L380 + verification_design L666 |
|
||||
| **Y5** | do_fix subsequent upgrade high→high+ (follow-up to Y4) | — (no new C-items; re-sync of Y4 locations after code state advanced) | 4 locations re-synced: endpoint_scenarios §Security list + §Security Level Matrix + §Note + verification_design Security tiers. README 'Risky Level Table' `Fix nodepack` moved from `high` row into `high+` row (`high` row now marked empty). Aligns enforcement gate with `SECURITY_MESSAGE_HIGH_P` log text (WI-#235, gate lifted from `'high'` to `'high+'` at `glob/manager_server.py:974` + `legacy/manager_server.py:560`). Y4 rows above are preserved as historical record of the middle→high transition. |
|
||||
|
||||
Summary Matrix: `(94,0,0,15,109)` → `(100,0,0,15,115)`. Upgrade count unchanged at 27 (WI-Z is inventory reconciliation, not new upgrades). Y5 is a follow-up re-sync triggered by a subsequent code change (WI-#235), not a new drift detection; it does not alter the Summary Matrix counts.
|
||||
|
||||
---
|
||||
|
||||
## 1. Inventory
|
||||
|
||||
| # | File | Lines | Modified | Role |
|
||||
|---|------|------:|----------|------|
|
||||
| 1 | `endpoint_scenarios.md` | 425 | 2026-04-18 23:53 | Report A — Endpoint extraction + scenarios |
|
||||
| 2 | `scenario_intents.md` | 424 | 2026-04-18 08:29 | Intent (why endpoint exists) |
|
||||
| 3 | `scenario_effects.md` | 514 | 2026-04-18 08:27 | Effect (observable result) |
|
||||
| 4 | `verification_design.md` | 824 | 2026-04-18 23:53 | Design Goals (92 + CSRF-M1/M2/M3 = 95) |
|
||||
| 5 | `e2e_test_coverage.md` | 358 | 2026-04-18 22:39 | Report B — E2E test inventory |
|
||||
| 6 | `e2e_verification_audit.md` | 469 | 2026-04-19 22:36 | Audit verdicts (109 tests; 94 PASS / 15 N/A) |
|
||||
| 7 | `test_contract_audit.md` | 282 | 2026-04-18 23:09 | Pytest/Playwright contract compliance |
|
||||
| 8 | `coverage_gaps.md` | 182 | 2026-04-18 22:45 | Coverage gap rollup |
|
||||
| 9 | `research-cluster-g.md` | 215 | 2026-04-19 07:30 | Cluster G research (imported_mode + boolean CLI) |
|
||||
|
||||
Total: **3 693 lines** across **9 files**.
|
||||
|
||||
Actual test-file reality (`grep -c "def test_"` / `grep "^\s*test("`) at audit time:
|
||||
|
||||
| Test file | `def test_` count | Audit claim | Coverage claim |
|
||||
|-----------|------------------:|------------:|---------------:|
|
||||
| `test_e2e_config_api.py` | 15 | 10 | 10 |
|
||||
| `test_e2e_csrf.py` | 4 | 4 | 4 (29 parametrized) |
|
||||
| `test_e2e_customnode_info.py` | 12 (11 + 1 `@pytest.mark.skip`) | 11 | 9 |
|
||||
| `test_e2e_endpoint.py` | 7 | 7 | 7 |
|
||||
| `test_e2e_git_clone.py` | 3 | 3 | 3 |
|
||||
| `test_e2e_queue_lifecycle.py` | 10 | 9 (+ 1 noted in Key gaps) | 9 |
|
||||
| `test_e2e_snapshot_lifecycle.py` | 7 | 6 | 7 (contains 1 deleted + missing 1 new) |
|
||||
| `test_e2e_system_info.py` | 4 | 4 | 4 |
|
||||
| `test_e2e_task_operations.py` | 16 | 16 | 16 |
|
||||
| `tests/cli/test_uv_compile.py` (relocated WI-PP; was `test_e2e_uv_compile.py`) | 8 | N/A (out of E2E scope) | 8 |
|
||||
| `test_e2e_version_mgmt.py` | 7 | 7 | 7 |
|
||||
| `legacy-ui-manager-menu.spec.ts` | 5 | 5 | 5 |
|
||||
| `legacy-ui-custom-nodes.spec.ts` | 5 | 5 | 5 |
|
||||
| `legacy-ui-model-manager.spec.ts` | 4 | 4 | 4 |
|
||||
| `legacy-ui-snapshot.spec.ts` | 3 | 3 | 3 |
|
||||
| `legacy-ui-navigation.spec.ts` | 2 | 2 | 4 |
|
||||
| `debug-install-flow.spec.ts` | 1 | 1 | 1 |
|
||||
|
||||
---
|
||||
|
||||
## 2. Internal Cross-Reference Consistency (reports ↔ reports)
|
||||
|
||||
### 2.1 Counts that agree across all reports (✅ consistent)
|
||||
|
||||
| Claim | Values | Files involved |
|
||||
|-------|--------|----------------|
|
||||
| Total tests counted in audit | **109** (94 P / 0 W / 0 I / 15 N) | `e2e_verification_audit.md` L330, L334, L462, L466 |
|
||||
| Design Goals | **92** base + **3** CSRF-M (M1/M2/M3) = **95** superset | `verification_design.md` §10, `e2e_verification_audit.md` L335, L337, L378 |
|
||||
| Design-Goal coverage | **68/92** (73.9%) base / **71/95** (74.7%) superset | `e2e_verification_audit.md` L337 |
|
||||
| `test_e2e_csrf.py` structure | 4 test functions / **29** parametrized invocations (16 + 2 + 11) | `e2e_test_coverage.md` L18, L188; `test_contract_audit.md` L104-106, L168; `verification_design.md` L797, L805, L813; `e2e_verification_audit.md` L323 |
|
||||
| `STATE_CHANGING_POST_ENDPOINTS` line range | L92–L109 in `test_e2e_csrf.py` | `endpoint_scenarios.md` L396 — **verified against source** |
|
||||
| Security Level Matrix line range | L378–L382 in `endpoint_scenarios.md` | `endpoint_scenarios.md` L396 — **verified** |
|
||||
| `comfyui_switch_version` → `high+` | Consistent at L382 of `endpoint_scenarios.md`, L668 of `verification_design.md`, L410 of `endpoint_scenarios.md` (CSRF inventory) | Cross-checked with commit `9c9d1a40` |
|
||||
| `verify_audit_counts.py` gate | Summary Matrix computed vs reported — both `(94, 0, 0, 15, 109)` | Script output PASS |
|
||||
|
||||
### 2.2 Playwright test-count cross-report check (⚠️ drift in one file)
|
||||
|
||||
| File | manager-menu | custom-nodes | model-manager | snapshot | navigation | debug | Total |
|
||||
|------|---:|---:|---:|---:|---:|---:|---:|
|
||||
| `e2e_verification_audit.md` L318-328 | 5 | 5 | 4 | 3 | **2** | 1 | **20** |
|
||||
| `test_contract_audit.md` L73 (20 tests) | (aggregated) | | | | | | **20** |
|
||||
| `e2e_test_coverage.md` L14 (Summary) | | | | | | | **21** ❌ |
|
||||
| `e2e_test_coverage.md` per-section | 5 | 5 | 4 | 3 | **4** ❌ | 1 | **22** ❌ |
|
||||
| Actual spec files (`grep "^\s*test("`) | 5 | 5 | 4 | 3 | **2** | 1 | **20** |
|
||||
|
||||
Only `e2e_test_coverage.md` is out of sync with the Playwright post-WI-F reality. All other reports (audit + contract + actual) agree on 20.
|
||||
|
||||
---
|
||||
|
||||
## 3. Discovered Drift (by category and severity)
|
||||
|
||||
### Category A — Simple typo / stale number (MINOR)
|
||||
|
||||
| # | Location | Current text | Should be | Evidence |
|
||||
|---|----------|--------------|-----------|----------|
|
||||
| A-1 | `e2e_test_coverage.md` L86 | `## tests/e2e/test_e2e_customnode_info.py (9 tests)` | `(11 tests)` | Section body lists 11 rows (L92–L102); audit §5 header is `(11 tests)` |
|
||||
| A-2 | `e2e_test_coverage.md` L14 | `Playwright (legacy UI) \| 5 \| 21` | `\| 5 \| 19` | Post-WI-F deletion of 2 navigation tests; audit Summary Matrix reports 20 (19 legacy + 1 debug) |
|
||||
| A-3 | `e2e_test_coverage.md` L16 | `TOTAL \| 17 \| 119` | `\| 17 \| 117` | Downstream of A-2 |
|
||||
| A-4 | `e2e_test_coverage.md` L258 | `## tests/playwright/legacy-ui-navigation.spec.ts (4 tests)` | `(2 tests)` | Actual spec: 2 `test(...)` calls — `API health check` and `system endpoints accessible` deleted per WI-F (Stage2) |
|
||||
|
||||
### Category B — Structural drift (OBSOLETE rows / MISSING rows, MAJOR)
|
||||
|
||||
| # | Location | Drift | Evidence |
|
||||
|---|----------|-------|----------|
|
||||
| B-1 | `e2e_test_coverage.md` L266–L267 | **OBSOLETE rows** — references deleted tests `API health check while dialogs are open` and `system endpoints accessible from browser context` | `test_contract_audit.md` L32-33: both marked `**DELETED**`; verify: `grep '^\s*test(' tests/playwright/legacy-ui-navigation.spec.ts` returns only 2 matches |
|
||||
| B-2 | `e2e_test_coverage.md` L131 | **OBSOLETE row** — `TestSnapshotGetCurrentSchema::test_get_current_returns_dict` | `e2e_verification_audit.md` L128: struck-through `~~test_get_current_returns_dict~~ ~~REMOVED~~` via Wave1 WI-M dedup (file count 7→6 for §7) |
|
||||
| B-3 | `e2e_test_coverage.md` L120–L132 | **MISSING row** — `test_remove_path_traversal_rejected` (file L300) not listed, though `snapshot_lifecycle.py` file count claims 7 tests | `grep "def test_" tests/e2e/test_e2e_snapshot_lifecycle.py` shows 7 functions; this test (SR3 — path traversal rejection) implements the "NORMAL add (Priority 🔴)" Key gap |
|
||||
| B-4 | `e2e_verification_audit.md` L119 (§7 header + body) | **MISSING row** — same as B-3. Section 7 table lists 6 active rows (+ 1 struck REMOVED) but `test_remove_path_traversal_rejected` not represented. Summary Matrix row 319 therefore reports `6 \| 0 \| 0 \| 0 \| 6` for a file that now has 7 PASSing tests. Key gaps at L134 still lists **SR3** (path traversal on remove) as "NORMAL add (Priority 🔴 per §Priority Fixes)" even though the test is implemented and would PASS. | Source file `tests/e2e/test_e2e_snapshot_lifecycle.py` L300–L328 contains the test |
|
||||
| B-5 | `e2e_verification_audit.md` §4 (L56–L71) | **MISSING rows** — Section 4 tracks 10 config_api tests, but `tests/e2e/test_e2e_config_api.py` contains **15** `def test_` functions. Five tests are not represented: `test_set_db_mode_junk_value_rejected`, `test_db_mode_persists_to_config_ini`, `test_set_policy_junk_value_rejected`, `test_policy_persists_to_config_ini`, `test_set_channel_unknown_name_rejected` (source L411, L438, L727, and related). These are distinct from the `invalid_body` rows (malformed JSON) — they are whitelist-rejection and on-disk-persistence assertions introduced by WI-E / WI-I. | `grep "def test_" tests/e2e/test_e2e_config_api.py` returns 15 matches; audit § 4 body only references 10 |
|
||||
|
||||
### Category C — Semantic drift (claim contradicts current code, MAJOR)
|
||||
|
||||
| # | Location | Drift | Evidence |
|
||||
|---|----------|-------|----------|
|
||||
| C-1 | `endpoint_scenarios.md` L18 | Lists `_fix_custom_node` under security level `middle`. Commit **c8992e5d** (2026-04-04, "fix(security): correct do_fix security level from middle to high") changed do_fix in both `comfyui_manager/glob/manager_server.py` and `comfyui_manager/legacy/manager_server.py` from `middle` → `high`. Report was last modified 2026-04-18 (2 weeks after the commit) but still reflects pre-commit state. | `git show c8992e5d` diff; source at `glob/manager_server.py:966` (`is_allowed_security_level('high')`); README documents fix nodepack as `high` risk |
|
||||
| C-2 | `endpoint_scenarios.md` L380 (Security Level Matrix, Legacy column) | `_fix` listed under `middle` — same issue as C-1 for the legacy handler | Same commit c8992e5d: `legacy/manager_server.py:550-555` now has `is_allowed_security_level('high')` gate |
|
||||
| C-3 | `verification_design.md` L666 | `middle — reboot, snapshot/remove, _fix, _uninstall, _update` — `_fix` should be at `high+` tier | Same evidence as C-1/C-2 |
|
||||
|
||||
### Category D — Key-gap staleness (MINOR, observational)
|
||||
|
||||
| # | Location | Drift | Note |
|
||||
|---|----------|-------|------|
|
||||
| D-1 | `e2e_verification_audit.md` L134 (§7 Key gaps) | Claims **SR3** (path traversal on remove) is "NORMAL add (Priority 🔴 per §Priority Fixes)" — but the test IS implemented (see B-3/B-4) and the file count should be 7/7 ✅ | Either the Key gaps line is stale, or Section 7 should add the new row, update the total to 7/7, and resolve SR3 in §Priority Fixes |
|
||||
|
||||
---
|
||||
|
||||
## 4. Suggested Fixes (patch sketches, NOT applied — deferred to follow-up WI)
|
||||
|
||||
> These are line-level recommendations. Verify count-changes against `verify_audit_counts.py` before applying.
|
||||
|
||||
### Patch Y1 — Resolve B-3 + B-4 + D-1 (add `test_remove_path_traversal_rejected`)
|
||||
|
||||
```diff
|
||||
--- a/reports/e2e_verification_audit.md
|
||||
+++ b/reports/e2e_verification_audit.md
|
||||
@@ -119 +119 @@
|
||||
-# Section 7 — tests/e2e/test_e2e_snapshot_lifecycle.py (6 tests)
|
||||
+# Section 7 — tests/e2e/test_e2e_snapshot_lifecycle.py (7 tests)
|
||||
@@ -127,0 +128,1 @@
|
||||
+| `test_remove_path_traversal_rejected` | SR3 | ✅ PASS | Path-traversal targets (`../../...`, `/etc/passwd`) return 400; sentinel file outside snapshot dir is NOT deleted. Resolves SR3 (path traversal on remove) — previously NORMAL-add. |
|
||||
@@ -131 +131 @@
|
||||
-**File verdict**: 6/6 ✅ (…)
|
||||
+**File verdict**: 7/7 ✅ (Wave1 WI-M dedup + Wave2 WI-Q disk/content verification + SR3 path-traversal coverage implemented; file count 7→6→7.)
|
||||
@@ -134 +134 @@
|
||||
-- **SR3** (path traversal on remove) — NORMAL add (Priority 🔴 per §Priority Fixes).
|
||||
+(remove line — SR3 resolved by `test_remove_path_traversal_rejected`)
|
||||
@@ -319 +319 @@
|
||||
-| test_e2e_snapshot_lifecycle.py | 6 | 0 | 0 | 0 | 6 |
|
||||
+| test_e2e_snapshot_lifecycle.py | 7 | 0 | 0 | 0 | 7 |
|
||||
@@ -330 +330 @@
|
||||
-| **TOTAL** | **94** | **0** | **0** | **15** | **109** |
|
||||
+| **TOTAL** | **95** | **0** | **0** | **15** | **110** |
|
||||
```
|
||||
|
||||
Also update `§ Priority Fixes` for SR3 entry, and update the narrative totals on L334, L462, L466 (109 → 110). The 71/95 superset tally remains unchanged (SR3 is an existing Goal already inside the 92 base, not a new Goal).
|
||||
|
||||
### Patch Y2 — Resolve A-1 / A-2 / A-3 / A-4 / B-1 / B-2 / B-3 (e2e_test_coverage.md sync with reality)
|
||||
|
||||
```diff
|
||||
--- a/reports/e2e_test_coverage.md
|
||||
+++ b/reports/e2e_test_coverage.md
|
||||
@@ -12,4 +12,4 @@
|
||||
-| pytest E2E (HTTP API) | 10 | 85 |
|
||||
-| pytest E2E (CLI — uv-compile) | 1 | 12 |
|
||||
-| Playwright (legacy UI) | 5 | 21 |
|
||||
-| Playwright (debug) | 1 | 1 |
|
||||
-| **TOTAL** | **17** | **119** |
|
||||
+| pytest E2E (HTTP API) | 10 | 86 | # +1 = test_remove_path_traversal_rejected (see Patch Y1)
|
||||
+| pytest E2E (CLI — uv-compile) | 1 | 12 |
|
||||
+| Playwright (legacy UI) | 5 | 19 |
|
||||
+| Playwright (debug) | 1 | 1 |
|
||||
+| **TOTAL** | **17** | **118** |
|
||||
@@ -86 +86 @@
|
||||
-## tests/e2e/test_e2e_customnode_info.py (9 tests)
|
||||
+## tests/e2e/test_e2e_customnode_info.py (11 tests)
|
||||
@@ -120 +120 @@
|
||||
-## tests/e2e/test_e2e_snapshot_lifecycle.py (7 tests)
|
||||
+## tests/e2e/test_e2e_snapshot_lifecycle.py (7 tests)
|
||||
@@ -131 +131 @@
|
||||
-| `TestSnapshotGetCurrentSchema::test_get_current_returns_dict` | GET snapshot/get_current | Response schema | dict type |
|
||||
+| `TestSnapshotRemove::test_remove_path_traversal_rejected` | POST snapshot/remove?target=../... | Path traversal rejected | 400 + sentinel file preserved |
|
||||
@@ -258 +258 @@
|
||||
-## tests/playwright/legacy-ui-navigation.spec.ts (4 tests)
|
||||
+## tests/playwright/legacy-ui-navigation.spec.ts (2 tests)
|
||||
@@ -266,2 +266,0 @@
|
||||
-| `> API health check while dialogs are open` | GET manager/version | … | … |
|
||||
-| `> system endpoints accessible from browser context` | GET manager/version + GET is_legacy_manager_ui | … | … |
|
||||
```
|
||||
|
||||
Note: if Patch Y1 is NOT applied, change `86 → 85`, `118 → 117`, and keep `(7 tests)` but still drop the obsolete `test_get_current_returns_dict` row and add `test_remove_path_traversal_rejected` row (net 7 tests).
|
||||
|
||||
### Patch Y3 — Resolve B-5 (config_api 5 missing rows in audit § 4)
|
||||
|
||||
This requires per-row verdict content (PASS + issue notes) for each of the 5 new tests. Recommend spawning a verification sub-WI that either (a) runs `pytest tests/e2e/test_e2e_config_api.py` and maps each new test to a Goal (C2 / C3 / C5 variants — whitelist rejection, disk persistence), or (b) marks them as "tracked but uncounted" with a preamble note.
|
||||
|
||||
Summary-matrix delta if all 5 added as PASS: `test_e2e_config_api.py: 10 → 15`, Grand TOTAL: `109 → 114` (independent of Patch Y1). Combined with Y1: `109 → 115`.
|
||||
|
||||
### Patch Y4 — Resolve C-1 / C-2 / C-3 (do_fix security level from middle → high)
|
||||
|
||||
```diff
|
||||
--- a/reports/endpoint_scenarios.md
|
||||
+++ b/reports/endpoint_scenarios.md
|
||||
@@ -18 +18 @@
|
||||
-- `middle`: reboot, snapshot/remove, _fix_custom_node, _uninstall_custom_node, _update_custom_node
|
||||
+- `middle`: reboot, snapshot/remove, _uninstall_custom_node, _update_custom_node
|
||||
+- `high`: _fix_custom_node (commit c8992e5d — aligned with README risk matrix)
|
||||
@@ -380 +380 @@
|
||||
-| **middle** | snapshot/remove, reboot | snapshot/remove, reboot, _uninstall, _update, _fix |
|
||||
+| **middle** | snapshot/remove, reboot | snapshot/remove, reboot, _uninstall, _update |
|
||||
+| **high** | _fix_custom_node | _fix |
|
||||
```
|
||||
|
||||
```diff
|
||||
--- a/reports/verification_design.md
|
||||
+++ b/reports/verification_design.md
|
||||
@@ -666 +666 @@
|
||||
-- `middle` — reboot, snapshot/remove, _fix, _uninstall, _update
|
||||
+- `middle` — reboot, snapshot/remove, _uninstall, _update
|
||||
+- `high` — _fix (commit c8992e5d, aligned with README 'fix nodepack' risk tier)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Final Consistency Status Summary
|
||||
|
||||
| Check | Result |
|
||||
|-------|--------|
|
||||
| `verify_audit_counts.py` exit code | **0 (PASS)** |
|
||||
| Audit Summary Matrix ↔ per-section rows | **Internally consistent** (94/0/0/15/109) |
|
||||
| Design Goal counts (92 base, 95 superset) | **Consistent** across `verification_design.md`, `e2e_verification_audit.md` |
|
||||
| `test_e2e_csrf.py` function/parametrization tally | **Consistent** across 4 cross-referencing reports (4 functions / 29 invocations / 16+2+11) |
|
||||
| Cross-file test-count tally (Playwright) | **1 file out-of-sync** (`e2e_test_coverage.md` claims 21 / 22, rest agree on 20) |
|
||||
| Audit ↔ actual test files (code-level drift) | **3 files out-of-sync**: config_api (+5 rows missing), snapshot_lifecycle (+1 row missing), customnode_info (+1 skip companion, acceptable) |
|
||||
| Security Level Matrix ↔ source code | **Stale for do_fix**: middle vs actual high (commit c8992e5d) |
|
||||
|
||||
### Drift counts by severity
|
||||
|
||||
| Severity | Count | Items |
|
||||
|---------:|------:|-------|
|
||||
| MAJOR (Category B, structural) | **5** | B-1, B-2, B-3, B-4, B-5 |
|
||||
| MAJOR (Category C, semantic/code drift) | **3** | C-1, C-2, C-3 |
|
||||
| MINOR (Category A, cosmetic count/typo) | **4** | A-1, A-2, A-3, A-4 |
|
||||
| MINOR (Category D, observational) | **1** | D-1 (tied to B-4) |
|
||||
| **TOTAL** | **13** | |
|
||||
|
||||
### Interpretation
|
||||
|
||||
The **internal cross-report consistency is strong** — the audit's Summary Matrix parser passes, Design Goal / test-count / CSRF-contract numbers agree across 4–6 cross-referencing reports, and line-range citations (L92–L109, L378–L382) are verified accurate against source code.
|
||||
|
||||
The **external drift** (reports ↔ actual code) is concentrated in two places:
|
||||
|
||||
1. **Newly added tests not yet reflected in the audit matrix** — `test_remove_path_traversal_rejected` (snapshot), 5 new config_api tests (junk_value + disk-persistence), and a skip-companion in customnode_info. These are real, passing tests that should be audited and counted.
|
||||
2. **do_fix security level semantic drift** — commit c8992e5d (2026-04-04) moved the gate from `middle` to `high`, but three reports still document the pre-commit state.
|
||||
|
||||
Neither class of drift invalidates the existing numbered conclusions (the audit passes its own checker); both are about **undercount / stale** rather than contradiction. Priority recommendation: apply Patch Y1 + Y4 before PR (minimal, high-value), defer Patch Y2 + Y3 to a follow-up WI if PR scope is tight.
|
||||
|
||||
---
|
||||
|
||||
*End of Consistency Audit Report Y*
|
||||
@@ -0,0 +1,203 @@
|
||||
# Coverage Gap Analysis — Report A × Report B
|
||||
|
||||
**Generated**: 2026-04-18 (WI-AA inventory update 2026-04-19: +2 LB1/LB2 tests → 119 total; WI-GG update 2026-04-20: +26 legacy CSRF parametrized rows → 145 total; WI-JJ update 2026-04-20: +3 legacy CSRF parity/install rows; WI-LL update 2026-04-20: +2 SECGATE rows → 148 audit rows / 126 test functions)
|
||||
**Inputs**: `reports/endpoint_scenarios.md` (39 handlers, 154 scenarios) + `reports/e2e_test_coverage.md` (126 test functions / 148 audit rows — the row-count delta reflects audit-side per-invocation rendering of legacy CSRF plus the 2 new SECGATE PoC rows; function-level progression is recorded in `e2e_test_coverage.md`)
|
||||
|
||||
> **WI-AA (2026-04-19)**: `tests/playwright/legacy-ui-install.spec.ts` (2 tests: LB1 Install button triggers install effect, LB2 Uninstall button triggers uninstall effect) has been **integrated into the audit** (see `e2e_verification_audit.md` §16). These tests drive the install/uninstall action via the Custom Nodes Manager dialog UI and verify the resulting backend state via `/v2/customnode/installed`. They close the long-standing gap noted in this document's Section 4 for UI-driven install/uninstall effect coverage on the legacy UI. Prior coverage-gap mentions of "missing UI→effect for install/uninstall buttons" are now RESOLVED.
|
||||
>
|
||||
> **WI-GG (2026-04-20)**: `tests/e2e/test_e2e_csrf_legacy.py` (4 test functions / 26 parametrized invocations: 13 reject-GET + 2 POST-works + 11 allow-GET) — from WI-FF — has been **integrated into the audit** (see `e2e_verification_audit.md` §19). This extends the CSRF-Mitigation Layer Coverage block below from glob-only to glob + legacy, closing the regression-guard gap that a legacy-side `@routes.post` revert would have slipped past CI. LB1/LB2 classification as RESOLVED is unchanged.
|
||||
>
|
||||
> **WI-LL (2026-04-20)**: `tests/e2e/test_e2e_secgate_strict.py` (SR4 PoC — strict-mode fixture) + `tests/e2e/test_e2e_secgate_default.py` (CV4 demo — no harness needed) — from WI-KK — have been **integrated into the audit** (see `e2e_verification_audit.md` §20, §21). Two of the original 8 T2 SECGATE-PENDING Goals are now RESOLVED (SR4, CV4); the remaining 6 are reclassified into 4 sub-tiers (T2-pending-harness-ready: SR6/V5/UA2; NORMAL-legacy: LGU2/LPP2; T2-TASKLEVEL: IM4). Section 4 🟢 Low Priority "Security level 403 gates ... impractical in standard E2E env" is now PARTIAL — see the classification policy + propagation plan in `e2e_verification_audit.md`.
|
||||
|
||||
## Summary
|
||||
|
||||
| Metric | Count |
|
||||
|---|---:|
|
||||
| Glob v2 endpoints fully covered (row has no ✗ in Missing scenarios) | 15/30 |
|
||||
| Glob v2 endpoints partially covered (some ✓ + some ✗) | 14/30 |
|
||||
| Glob v2 endpoints NOT covered (positive) | 1/30 |
|
||||
| Legacy-only endpoints fully covered | 0/9 |
|
||||
| Legacy-only endpoints partially covered (indirect only) | 4/9 |
|
||||
| Legacy-only endpoints NOT covered | 5/9 |
|
||||
| Orphan tests (non-endpoint-direct; pytest + Playwright UI-only) | 29 |
|
||||
|
||||
---
|
||||
|
||||
# Section 1 — Endpoint × Test Coverage Matrix (Glob v2)
|
||||
|
||||
Legend: **✓** direct assertion test, **~** indirect / partial, **✗** not tested
|
||||
|
||||
| # | Endpoint | Direct test | Missing scenarios |
|
||||
|---|---|---|---|
|
||||
| 1 | POST queue/task (install) | ✓ test_e2e_endpoint, test_e2e_git_clone, test_e2e_task_operations | ✗ 400 ValidationError (bad kind/schema), ✗ 500 on malformed JSON |
|
||||
| 2 | POST queue/task (uninstall) | ✓ test_e2e_endpoint, test_e2e_task_operations | (shared with #1) |
|
||||
| 3 | POST queue/task (update/fix/disable/enable) | ✓ test_e2e_task_operations | (shared) |
|
||||
| 4 | GET queue/history_list | ✓ test_e2e_queue_lifecycle | ✗ 400 on inaccessible history path |
|
||||
| 5 | GET queue/history | ✓ test_e2e_queue_lifecycle, test_e2e_task_operations | ✗ `id=<batch_id>` file-based query, ✗ path traversal rejection |
|
||||
| 6 | GET customnode/getmappings | ✓ test_e2e_customnode_info | ✗ `mode=nickname`, ✗ missing `mode` KeyError→500 |
|
||||
| 7 | GET customnode/fetch_updates | ✓ test_e2e_customnode_info (410) | (fully covered — deprecated endpoint) |
|
||||
| 8 | POST queue/update_all | ✓ test_e2e_task_operations | ✗ 403 security gate, ✗ `mode=local` vs remote distinction |
|
||||
| 9 | GET is_legacy_manager_ui | ✓ test_e2e_system_info, playwright navigation | (fully covered) |
|
||||
| 10 | GET customnode/installed | ✓ test_e2e_endpoint, test_e2e_customnode_info (both modes) | (fully covered) |
|
||||
| 11 | GET snapshot/getlist | ✓ test_e2e_snapshot_lifecycle, playwright snapshot | (fully covered) |
|
||||
| 12 | POST snapshot/remove | ✓ test_e2e_snapshot_lifecycle | ✗ path traversal "Invalid target" 400, ✗ 403 security gate, ✗ missing `target` query |
|
||||
| 13 | POST snapshot/restore | ✗ **intentionally skipped (destructive)** | ALL scenarios (positive, path traversal, 403) |
|
||||
| 14 | GET snapshot/get_current | ✓ test_e2e_snapshot_lifecycle | (fully covered) |
|
||||
| 15 | POST snapshot/save | ✓ test_e2e_snapshot_lifecycle, playwright snapshot | (fully covered) |
|
||||
| 16 | POST customnode/import_fail_info | ✓ test_e2e_customnode_info (negative only) | ✗ positive path (returning actual failure info) — requires seed failed import |
|
||||
| 17 | POST customnode/import_fail_info_bulk | ✓ test_e2e_customnode_info | ✗ positive path with real failure info |
|
||||
| 18 | POST queue/reset | ✓ test_e2e_queue_lifecycle | (fully covered) |
|
||||
| 19 | GET queue/status | ✓ test_e2e_queue_lifecycle | (fully covered) |
|
||||
| 20 | POST queue/start | ✓ test_e2e_queue_lifecycle (idle + lifecycle) | (fully covered) |
|
||||
| 21 | POST queue/update_comfyui | ✓ test_e2e_task_operations | (fully covered) |
|
||||
| 22 | GET comfyui_versions | ✓ test_e2e_version_mgmt (4 tests) | ✗ 400 on git-access failure |
|
||||
| 23 | POST comfyui_switch_version | ✓ test_e2e_version_mgmt (negative only) | ✗ **positive path (intentionally skipped)**, ✗ 403 security gate |
|
||||
| 24 | POST queue/install_model | ✓ test_e2e_task_operations | (fully covered) |
|
||||
| 25 | GET db_mode | ✓ test_e2e_config_api, playwright manager-menu | (fully covered) |
|
||||
| 26 | POST db_mode | ✓ test_e2e_config_api, playwright manager-menu | ✗ missing `value` KeyError→400 (only malformed JSON tested) |
|
||||
| 27 | GET policy/update | ✓ test_e2e_config_api, playwright | (fully covered) |
|
||||
| 28 | POST policy/update | ✓ test_e2e_config_api, playwright | ✗ missing `value` key |
|
||||
| 29 | GET channel_url_list | ✓ test_e2e_config_api | (fully covered) |
|
||||
| 30 | POST channel_url_list | ✓ test_e2e_config_api | ✗ unknown channel name (silent no-op) |
|
||||
| 31 | POST manager/reboot | ✓ test_e2e_system_info | ✗ __COMFY_CLI_SESSION__ env branch |
|
||||
| 32 | GET manager/version | ✓ test_e2e_system_info, playwright navigation | (fully covered) |
|
||||
|
||||
Note: queue/task has 3 rows above per kind; 30 glob endpoints = 32 row entries (queue/task counted per-kind).
|
||||
|
||||
## Glob v2 Summary
|
||||
|
||||
- **Fully covered** (row has no ✗ in Missing scenarios column): 15 endpoints
|
||||
(is_legacy_manager_ui, customnode/installed, snapshot/getlist, snapshot/get_current, snapshot/save, queue/reset, queue/status, queue/start, queue/update_comfyui, queue/install_model, fetch_updates, get db_mode, get policy/update, get channel_url_list, get manager/version)
|
||||
- **Partially covered** (some ✓ + some ✗ in Missing scenarios): 14 endpoints (row-level collapse of per-kind queue/task into 1 endpoint)
|
||||
- **Intentionally skipped** (destructive, counted under NOT covered): 1 (snapshot/restore); switch_version has ✓ negative + skipped-positive so it falls under partial, not skipped
|
||||
- Sum: 15 + 14 + 1 = 30 ✓
|
||||
|
||||
### CSRF-Mitigation Layer Coverage (separate from positive-path coverage above)
|
||||
|
||||
The 16 state-changing POST endpoints — commit 99caef55 converted 12+ of
|
||||
these from GET→POST (the remainder such as queue/task, import_fail_info,
|
||||
and import_fail_info_bulk were already POST but are included for contract
|
||||
completeness) — are independently covered. Commit 99caef55 applied the
|
||||
conversion to BOTH `comfyui_manager/glob/manager_server.py` (~91 lines)
|
||||
and `comfyui_manager/legacy/manager_server.py` (~92 lines), so two test
|
||||
files are required (the server-loading is mutex on `--enable-manager-legacy-ui`):
|
||||
|
||||
**Glob server**: `tests/e2e/test_e2e_csrf.py` (4 functions / 26 parametrized invocations — post-WI-HH; was 29 before the 3 dual-purpose endpoints were scoped out of the reject-GET fixture)
|
||||
|
||||
| Contract | Test | Coverage |
|
||||
|---|---|---|
|
||||
| Reject GET on 13 state-changing POST endpoints (glob; post-WI-HH) | TestStateChangingEndpointsRejectGet (parametrized ×13) | ✓ full |
|
||||
| POST counterpart sanity (glob) | TestCsrfPostWorks (2 tests) | ~ spot-check (queue/reset + snapshot/save only) |
|
||||
| Read-only GET still allowed — negative control (glob) | TestCsrfReadEndpointsStillAllowGet (parametrized ×11) | ✓ full |
|
||||
|
||||
**Legacy server** (WI-FF, audit-integrated in WI-GG): `tests/e2e/test_e2e_csrf_legacy.py` (4 functions / 26 parametrized invocations)
|
||||
|
||||
| Contract | Test | Coverage |
|
||||
|---|---|---|
|
||||
| Reject GET on 13 state-changing POST endpoints (legacy; queue/task→queue/batch, dual-purpose endpoints scoped to ALLOW-GET only) | TestLegacyStateChangingEndpointsRejectGet (parametrized ×13) | ✓ full |
|
||||
| POST counterpart sanity (legacy) | TestLegacyCsrfPostWorks (2 tests — queue/reset + snapshot/save) | ~ spot-check |
|
||||
| Read-only GET still allowed — negative control (legacy) | TestLegacyCsrfReadEndpointsStillAllowGet (parametrized ×11) | ✓ full |
|
||||
|
||||
**Important**: POST `snapshot/restore` and POST `comfyui_switch_version` are
|
||||
listed as "intentionally skipped (destructive)" for POSITIVE-path coverage,
|
||||
but their CSRF reject-GET contract IS covered by BOTH test files —
|
||||
the destructive-skip only applies to the success-path assertion, not to
|
||||
the security layer. The legacy-side coverage closes the regression-guard
|
||||
gap that a revert of any legacy `@routes.post` back to `@routes.get` would
|
||||
otherwise have slipped past CI.
|
||||
|
||||
---
|
||||
|
||||
# Section 2 — Endpoint × Test Coverage Matrix (Legacy-only)
|
||||
|
||||
| # | Endpoint | Test coverage | Gap |
|
||||
|---|---|---|---|
|
||||
| 1 | POST queue/batch | ~ debug-install-flow captures API sequence indirectly | ✗ No dedicated assertion test for batch semantics |
|
||||
| 2 | GET customnode/getlist | ~ playwright custom-nodes triggers it via UI | ✗ No direct assertion on response shape, `skip_update` param, channel resolution |
|
||||
| 3 | GET /customnode/alternatives | ✗ NOT COVERED | ALL scenarios |
|
||||
| 4 | GET externalmodel/getlist | ~ playwright model-manager triggers via UI | ✗ No direct assertion on `installed` flag population, save_path resolution |
|
||||
| 5 | GET customnode/versions/{node_name} | ~ debug-install-flow captures it | ✗ 400 on unknown pack, no direct test |
|
||||
| 6 | GET customnode/disabled_versions/{node_name} | ✗ NOT COVERED | ALL scenarios |
|
||||
| 7 | POST customnode/install/git_url | ✗ NOT COVERED | ALL (high+ security, may be intentional) |
|
||||
| 8 | POST customnode/install/pip | ✗ NOT COVERED | ALL (high+ security, may be intentional) |
|
||||
| 9 | GET manager/notice | ✗ NOT COVERED | ALL scenarios |
|
||||
|
||||
## Legacy-only Summary
|
||||
|
||||
- **Fully covered**: 0/9
|
||||
- **Indirect-only** (triggered via UI flow but no direct assertion): 4/9
|
||||
- **Not covered**: 5/9
|
||||
|
||||
---
|
||||
|
||||
# Section 3 — Orphan Tests (not mapped to HTTP endpoints)
|
||||
|
||||
Tests that do not directly assert HTTP endpoint behavior:
|
||||
|
||||
| Test file | Tests | Purpose |
|
||||
|---|---:|---|
|
||||
| `tests/cli/test_uv_compile.py` | 8 | `cm-cli --uv-compile` CLI entrypoint (not HTTP). Relocated from `tests/e2e/` in WI-PP (see Recommendations §4 — now ACTIONED). |
|
||||
| `test_e2e_endpoint.py::test_startup_resolver_ran` | 1 | Log file assertion (server log contains UnifiedDepResolver) |
|
||||
| `test_e2e_endpoint.py::test_comfyui_started` | 1 | `/system_stats` (ComfyUI core endpoint, not Manager) |
|
||||
| `test_e2e_git_clone.py::test_02_no_module_error` | 1 | Log file regression check |
|
||||
| Playwright UI-only tests (no API assertion) | ~14 of 22 | UI rendering, dialog lifecycle, filter/search — no HTTP assertion |
|
||||
|
||||
Total orphan tests (non-endpoint-direct): ~29 / 115 (25%)
|
||||
|
||||
---
|
||||
|
||||
# Section 4 — Critical Gaps (Prioritized)
|
||||
|
||||
## 🔴 High Priority — Legacy Endpoints with ZERO UI→effect Test Coverage
|
||||
|
||||
These endpoints ARE actively called by legacy UI JavaScript but have no Playwright test exercising the UI flow:
|
||||
|
||||
| Endpoint | JS call site | Missing UI→effect test |
|
||||
|---|---|---|
|
||||
| POST /v2/customnode/install/git_url | `common.js:248` | "Install via Git URL" button flow |
|
||||
| POST /v2/customnode/install/pip | `common.js:213` | pip install UI flow |
|
||||
| GET /v2/customnode/disabled_versions/{node_name} | `custom-nodes-manager.js:1401` | Node row "Disabled Versions" dropdown |
|
||||
| GET /customnode/alternatives | `custom-nodes-manager.js:1885` | Alternatives display in custom nodes dialog |
|
||||
| GET /v2/manager/notice | `comfyui-manager.js:418` | Notice display on Manager menu open |
|
||||
|
||||
→ These are **NOT dead code** — they require **UI→effect tests added**, not removal.
|
||||
|
||||
## 🟡 Medium Priority — Missing Scenarios (Covered Endpoints)
|
||||
|
||||
1. **queue/task 400 ValidationError** — no test verifies schema rejection. Adding `test_queue_task_invalid_body` with malformed `kind` value would be trivial.
|
||||
2. **queue/history with `id=<batch_id>` + path traversal** — file-based history path not exercised.
|
||||
3. **snapshot/remove path traversal** — security-critical but not asserted.
|
||||
4. **comfyui_versions 400 on git failure** — would need to simulate git unavailable.
|
||||
5. **POST db_mode/policy/update missing `value` key** — currently only tests malformed JSON; KeyError path untested.
|
||||
|
||||
## 🟢 Low Priority — Acceptable Gaps
|
||||
|
||||
1. **snapshot/restore + switch_version positive** — intentionally skipped (destructive). Acceptable.
|
||||
2. ~~**Security level 403 gates** — require running with locked-down security_level; impractical in standard E2E env.~~ **PARTIAL (WI-LL via WI-KK, 2026-04-20)**: SR4 (snapshot/remove middle) + CV4 (comfyui_switch_version high+) are now covered via `test_e2e_secgate_strict.py` and `test_e2e_secgate_default.py` respectively. Remaining SECGATE Goals are reclassified (`e2e_verification_audit.md` classification-policy block): SR6/V5/UA2 are T2-pending-harness-ready (mechanical additions to strict.py); LGU2/LPP2 are NORMAL-legacy (needs `start_comfyui_legacy.sh`); IM4 is T2-TASKLEVEL (queue-observation pattern, not HTTP 403).
|
||||
3. **import_fail_info positive path** — would require seeding a failed module import; complex setup.
|
||||
|
||||
---
|
||||
|
||||
# Section 5 — Recommendations
|
||||
|
||||
1. **Add UI→effect Playwright tests** for the 5 JS-called legacy endpoints (install/git_url, install/pip, disabled_versions, alternatives, manager/notice). All are live in legacy UI flows.
|
||||
2. **Add minimal gap tests** for queue/task ValidationError + snapshot/remove path traversal — both are security-relevant.
|
||||
3. **Convert debug-install-flow.spec.ts** from logging-only into an assertion test (verify queue/batch payload structure + cm-queue-status WebSocket events).
|
||||
4. ~~**Consider moving uv_compile tests** to a separate CLI test directory (tests/cli/) — they are not E2E HTTP tests.~~ **ACTIONED (WI-PP)**: file now lives at `tests/cli/test_uv_compile.py`. CI workflow `.github/workflows/e2e.yml` updated to the new path. Placement hygiene restored.
|
||||
|
||||
---
|
||||
|
||||
# Section 6 — Cross-Report Consistency Check
|
||||
|
||||
| Report A claim | Report B claim | Status |
|
||||
|---|---|---|
|
||||
| 30 glob v2 endpoints | Row-level: 15 fully + 14 partial + 1 NOT covered = 30 (matches Summary L10-12) | ✓ consistent under row-no-✗ definition |
|
||||
| 9 legacy-only endpoints | "4 covered, 5 not covered" | ✓ consistent |
|
||||
| 154 scenarios total | (per-test scenario breakdown) | ⚠️ scenario-level count not aggregated in Report B; recommend adding |
|
||||
| Security gates (middle/middle+/high+) | Security 403 paths not tested | ⚠️ gap confirmed |
|
||||
| Deprecated endpoints flagged | fetch_updates 410 tested | ✓ consistent |
|
||||
|
||||
Internal accounting reconciled under the single "row has no ✗ in Missing scenarios column" definition for "fully covered". Earlier drafts of this report mixed three counting schemes (Summary 24/5/1, body-list 11/15/2, Section 6 27/30) that did not agree; this revision uses the row-level count uniformly (15/14/1 = 30 glob v2 endpoints) and aligns Summary L10-12, body L63-66, and Section 6 L172. Report A (endpoint_scenarios.md) and Report B (e2e_test_coverage.md) align on endpoint counts (30 glob v2 + 9 legacy = 39) and coverage categories under this definition.
|
||||
|
||||
---
|
||||
*End of Coverage Gap Analysis*
|
||||
@@ -0,0 +1,454 @@
|
||||
# Report B — E2E Test Inventory + Coverage Mapping
|
||||
|
||||
**Generated**: 2026-04-18
|
||||
**Source directories**:
|
||||
- `tests/e2e/*.py` (pytest — HTTP + CLI E2E)
|
||||
- `tests/playwright/*.spec.ts` (Playwright — legacy UI E2E)
|
||||
|
||||
## Summary
|
||||
|
||||
| Category | Files | Test Functions |
|
||||
|---|---:|---:|
|
||||
| pytest E2E (HTTP API) | 13 | 92 |
|
||||
| pytest E2E (CLI — uv-compile) | 1 | 12 |
|
||||
| Playwright (legacy UI) | 6 | 21 |
|
||||
| Playwright (debug) | 1 | 1 |
|
||||
| **TOTAL** | **21** | **126** |
|
||||
|
||||
> **Note**: +1 file / +4 functions vs prior counts reflect inclusion of `tests/e2e/test_e2e_csrf.py` (CSRF-mitigation contract suite, commit 99caef55). That suite's 4 test functions parametrize to 26 pytest invocations (13+2+11) after WI-HH removed 3 dual-purpose endpoints from the reject-GET fixture (they legitimately answer GET on the read-path and are covered only in the allow-GET class).
|
||||
>
|
||||
> **WI-Z Y2 sync (2026-04-19)**: Playwright legacy UI count 21 → 19 reflected Stage2 WI-F deletion of two `legacy-ui-navigation.spec.ts` tests (`API health check while dialogs are open`, `system endpoints accessible from browser context`) that were rewritten as direct-API violators; their coverage is now owned by `test_e2e_system_info.py`. TOTAL 119 → 117 was a downstream correction.
|
||||
>
|
||||
> **WI-AA sync (2026-04-19)**: Playwright legacy UI count 19 → 21 reflects addition of `tests/playwright/legacy-ui-install.spec.ts` (2 tests: LB1 Install button + LB2 Uninstall button) previously implemented but not inventoried. TOTAL 117 → 119. See dedicated subsection below.
|
||||
>
|
||||
> **WI-GG sync (2026-04-20)**: pytest E2E (HTTP API) file count 10 → 11 and function count 85 → 89 reflects addition of `tests/e2e/test_e2e_csrf_legacy.py` (4 new test functions / 26 parametrized invocations: 13 reject-GET + 2 POST-works + 11 allow-GET) from WI-FF. TOTAL 119 → 123 test functions. The legacy suite is the counterpart to `test_e2e_csrf.py` for the `--enable-manager-legacy-ui` server variant — required because `comfyui_manager/__init__.py` loads `glob.manager_server` XOR `legacy.manager_server`. See dedicated subsection below. (Accounting note: the higher-level audit `reports/e2e_verification_audit.md` Summary Matrix renders the 26 legacy invocations per-row, reaching TOTAL 143; both counts refer to the same underlying tests at different granularities — function-level here, invocation-level there.)
|
||||
>
|
||||
> **WI-LL sync (2026-04-20)**: pytest E2E (HTTP API) file count 11 → 13 and function count 89 → 92 reflects addition of two new SECGATE-coverage files (WI-KK deliverables, audit-integrated by WI-LL): `tests/e2e/test_e2e_secgate_strict.py` (strict-mode harness + SR4 PoC — 2 functions: `test_remove_returns_403` PASS + `test_post_works_at_default_after_restore` pytest.skip'd positive counterpart stub) + `tests/e2e/test_e2e_secgate_default.py` (default-mode demo + CV4 — 1 function: `test_switch_version_returns_403_at_default` PASS). TOTAL 123 → 126 test functions. These close 2 of the original 8 T2 SECGATE-PENDING Goals (SR4, CV4) and establish the strict-mode harness pattern (`start_comfyui_strict.sh` + config.ini backup/restore) for the remaining T2-pending-harness-ready Goals (SR6, V5, UA2). See the Classification policy block in `e2e_verification_audit.md` for the reclassification and propagation plan.
|
||||
|
||||
**Unique endpoints exercised**: 27 (glob v2) + 4 (legacy-only: queue/batch indirectly via UI)
|
||||
|
||||
---
|
||||
|
||||
# Section 1 — pytest E2E HTTP Tests
|
||||
|
||||
## tests/e2e/test_e2e_endpoint.py (7 tests)
|
||||
|
||||
Covers the main install/uninstall flow via `/v2/manager/queue/task` and `/v2/customnode/installed`.
|
||||
|
||||
| Test | Endpoint(s) | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `TestEndpointInstallUninstall::test_install_via_endpoint` | POST queue/task, POST queue/start | Install CNR pack (success) | pack dir exists + .tracking file present |
|
||||
| `TestEndpointInstallUninstall::test_installed_list_shows_pack` | GET customnode/installed | Pack appears in installed list | cnr_id match in dict values |
|
||||
| `TestEndpointInstallUninstall::test_uninstall_via_endpoint` | POST queue/task kind=uninstall | Uninstall success | pack dir removed from disk |
|
||||
| `TestEndpointInstallUninstall::test_installed_list_after_uninstall` | GET customnode/installed | Post-uninstall state | cnr_id absent from installed list |
|
||||
| `TestEndpointInstallUninstall::test_install_uninstall_cycle` | queue/task x2 | Full install→verify→uninstall cycle | All above assertions in one test |
|
||||
| `TestEndpointStartup::test_comfyui_started` | GET /system_stats | Server health | 200 response |
|
||||
| `TestEndpointStartup::test_startup_resolver_ran` | (log file) | UnifiedDepResolver ran at startup | log contains `[UnifiedDepResolver]` + "startup batch resolution succeeded" |
|
||||
|
||||
## tests/e2e/test_e2e_git_clone.py (3 tests)
|
||||
|
||||
Covers nightly (URL-based) install via `/v2/manager/queue/task` which triggers git_helper.py subprocess.
|
||||
|
||||
| Test | Endpoint(s) | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `TestNightlyInstallCycle::test_01_nightly_install` | POST queue/task selected_version=nightly | git clone via Manager API | pack dir exists + .git directory present |
|
||||
| `TestNightlyInstallCycle::test_02_no_module_error` | (log file) | No ModuleNotFoundError regression | log does not contain "ModuleNotFoundError" |
|
||||
| `TestNightlyInstallCycle::test_03_nightly_uninstall` | POST queue/task kind=uninstall | Uninstall nightly pack | pack dir removed |
|
||||
|
||||
## tests/cli/test_uv_compile.py — RELOCATED (WI-PP)
|
||||
|
||||
Previously tracked here as `tests/e2e/test_e2e_uv_compile.py`. Moved to
|
||||
`tests/cli/` in WI-PP because every test in the suite drives cm-cli as a
|
||||
subprocess; none of them exercise HTTP endpoints. The 8 tests (post
|
||||
WI-MM/NN/OO consolidation) continue to cover install / reinstall (xfail-marked
|
||||
for purge_node_state) / verbs-with-uv-compile (parametrized ×5) / uv-sync
|
||||
no-packs-exits-zero / no-packs-emits-signal / with-packs / conflict
|
||||
attribution with specs. CI runner updated in `.github/workflows/e2e.yml` to
|
||||
point at the new path.
|
||||
|
||||
## tests/e2e/test_e2e_config_api.py (10 tests)
|
||||
|
||||
Covers GET/POST round-trip on configuration endpoints.
|
||||
|
||||
| Test | Endpoint(s) | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `TestConfigDbMode::test_read_db_mode` | GET db_mode | Read current value | text in {cache, channel, local, remote} |
|
||||
| `TestConfigDbMode::test_set_and_restore_db_mode` | GET/POST db_mode | Set→read-back→restore | POST 200 + verify echo + restore verified |
|
||||
| `TestConfigDbMode::test_set_db_mode_invalid_body` | POST db_mode | Malformed JSON | 400 |
|
||||
| `TestConfigUpdatePolicy::test_read_update_policy` | GET policy/update | Read current policy | text in {stable, stable-comfyui, nightly, nightly-comfyui} |
|
||||
| `TestConfigUpdatePolicy::test_set_and_restore_update_policy` | GET/POST policy/update | Set→read-back→restore | Round-trip verification |
|
||||
| `TestConfigUpdatePolicy::test_set_policy_invalid_body` | POST policy/update | Malformed JSON | 400 |
|
||||
| `TestConfigChannelUrlList::test_read_channel_url_list` | GET channel_url_list | Response shape | has `selected` (str) + `list` (array) |
|
||||
| `TestConfigChannelUrlList::test_channel_list_entries_are_name_url_strings` | GET channel_url_list | Entry format | each entry is "name::url" string |
|
||||
| `TestConfigChannelUrlList::test_set_and_restore_channel` | GET/POST channel_url_list | Switch channel + restore | Verify `selected` matches set value |
|
||||
| `TestConfigChannelUrlList::test_set_channel_invalid_body` | POST channel_url_list | Malformed JSON | 400 |
|
||||
|
||||
## tests/e2e/test_e2e_customnode_info.py (11 tests)
|
||||
|
||||
Covers custom node info/mapping endpoints.
|
||||
|
||||
| Test | Endpoint(s) | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `TestCustomNodeMappings::test_getmappings_returns_dict` | GET customnode/getmappings?mode=local | Success response | 200 + dict |
|
||||
| `TestCustomNodeMappings::test_getmappings_entries_have_node_lists` | GET getmappings | Entry structure | each value is `[node_list, metadata]` |
|
||||
| `TestFetchUpdates::test_fetch_updates_returns_deprecated` | GET customnode/fetch_updates | Deprecated endpoint | 410 + `deprecated: true` |
|
||||
| `TestInstalledPacks::test_installed_returns_dict` | GET customnode/installed | Success | 200 + dict |
|
||||
| `TestInstalledPacks::test_installed_imported_mode` | GET installed?mode=imported | Startup snapshot | 200 + dict |
|
||||
| `TestImportFailInfo::test_unknown_cnr_id_returns_400` | POST import_fail_info | Unknown pack | 400 |
|
||||
| `TestImportFailInfo::test_missing_fields_returns_400` | POST import_fail_info | Missing cnr_id+url | 400 |
|
||||
| `TestImportFailInfo::test_invalid_body_returns_error` | POST import_fail_info | Non-dict body | 400 |
|
||||
| `TestImportFailInfoBulk::test_bulk_with_cnr_ids_returns_dict` | POST import_fail_info_bulk | cnr_ids list | 200 + null for unknown |
|
||||
| `TestImportFailInfoBulk::test_bulk_empty_lists_returns_400` | POST import_fail_info_bulk | Empty cnr_ids+urls | 400 |
|
||||
| `TestImportFailInfoBulk::test_bulk_with_urls_returns_dict` | POST import_fail_info_bulk | urls list | 200 + dict |
|
||||
|
||||
## tests/e2e/test_e2e_queue_lifecycle.py (9 tests)
|
||||
|
||||
Covers the queue management lifecycle.
|
||||
|
||||
| Test | Endpoint(s) | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `TestQueueLifecycle::test_reset_queue` | POST queue/reset | Empty the queue | 200 |
|
||||
| `TestQueueLifecycle::test_status_after_reset` | GET queue/status | Post-reset state | all counts 0, is_processing bool |
|
||||
| `TestQueueLifecycle::test_status_with_client_id_filter` | GET queue/status?client_id=X | Client filter | response echoes client_id |
|
||||
| `TestQueueLifecycle::test_start_queue_already_idle` | POST queue/start | Idle worker start | status in {200, 201} |
|
||||
| `TestQueueLifecycle::test_queue_task_and_history` | POST queue/task + queue/start + GET queue/status + GET queue/history | Full lifecycle | done_count>0 polled, history 200 or 400 |
|
||||
| `TestQueueLifecycle::test_history_with_ui_id_filter` | GET queue/history?ui_id=X | Filter history | 200 or 400 (serialization-limit) |
|
||||
| `TestQueueLifecycle::test_history_with_pagination` | GET queue/history?max_items=1&offset=0 | Pagination | 200 or 400 |
|
||||
| `TestQueueLifecycle::test_history_list` | GET queue/history_list | List batch IDs | 200 + `ids` list |
|
||||
| `TestQueueLifecycle::test_final_reset_and_clean_state` | POST queue/reset + GET queue/status | Cleanup | pending_count==0 |
|
||||
|
||||
## tests/e2e/test_e2e_snapshot_lifecycle.py (7 tests)
|
||||
|
||||
Covers snapshot save/list/remove cycle.
|
||||
|
||||
| Test | Endpoint(s) | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `TestSnapshotLifecycle::test_get_current_snapshot` | GET snapshot/get_current | Current state dict | 200 + dict |
|
||||
| `TestSnapshotLifecycle::test_save_snapshot` | POST snapshot/save | Save new snapshot | 200 |
|
||||
| `TestSnapshotLifecycle::test_getlist_after_save` | GET snapshot/getlist | List contains new snapshot | items.length > 0 |
|
||||
| `TestSnapshotLifecycle::test_remove_snapshot` | POST snapshot/remove?target=X + GET getlist | Remove + verify | target absent + count decremented |
|
||||
| `TestSnapshotLifecycle::test_remove_nonexistent_snapshot` | POST snapshot/remove | Nonexistent target | 200 (no-op) |
|
||||
| `TestSnapshotLifecycle::test_remove_path_traversal_rejected` | POST snapshot/remove?target=../... | Path-traversal targets must be rejected | 400 + sentinel file outside snapshot dir preserved (SR3) |
|
||||
| `TestSnapshotGetCurrentSchema::test_getlist_items_are_strings` | GET snapshot/getlist | Items shape | each item is string |
|
||||
|
||||
> Note: `POST /v2/snapshot/restore` intentionally NOT tested (destructive).
|
||||
|
||||
## tests/e2e/test_e2e_system_info.py (4 tests)
|
||||
|
||||
Covers system-level endpoints (version, legacy UI flag, reboot).
|
||||
|
||||
| Test | Endpoint(s) | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `TestManagerVersion::test_version_returns_string` | GET manager/version | Non-empty string | 200 + len>0 |
|
||||
| `TestManagerVersion::test_version_is_stable` | GET manager/version x2 | Idempotency | consecutive calls return same value |
|
||||
| `TestIsLegacyManagerUI::test_returns_boolean_field` | GET is_legacy_manager_ui | Response shape | `{is_legacy_manager_ui: bool}` |
|
||||
| `TestReboot::test_reboot_and_recovery` | POST manager/reboot + GET version | Restart + recovery | 200 or 403 (security); server polls healthy; version unchanged |
|
||||
|
||||
## tests/e2e/test_e2e_task_operations.py (16 tests)
|
||||
|
||||
Covers queue/task operations for kinds NOT tested in test_e2e_endpoint.py.
|
||||
|
||||
| Test | Endpoint(s) | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `TestDisableEnable::test_disable_pack` | POST queue/task kind=disable | Disable moves pack to .disabled/ | pack dir gone + .disabled/ entry present |
|
||||
| `TestDisableEnable::test_enable_pack` | POST queue/task kind=enable | Enable restores pack | pack dir present + .disabled/ entry gone |
|
||||
| `TestDisableEnable::test_disable_enable_cycle` | queue/task x2 | Full disable→enable | Both transitions verified |
|
||||
| `TestUpdatePack::test_update_installed_pack` | POST queue/task kind=update | Update pack | pack still exists after update |
|
||||
| `TestUpdatePack::test_update_history_recorded` | GET queue/history?ui_id=X | History has update entry | 200 or 400 (serialization-limit) |
|
||||
| `TestFixPack::test_fix_installed_pack` | POST queue/task kind=fix | Fix pack | pack still exists |
|
||||
| `TestFixPack::test_fix_history_recorded` | GET queue/history?ui_id=X | History has fix entry | 200 or 400 |
|
||||
| `TestInstallModel::test_install_model_accepts_valid_request` | POST queue/install_model | Valid model request | 200 (reset queue after) |
|
||||
| `TestInstallModel::test_install_model_missing_client_id` | POST queue/install_model | Missing client_id | 400 |
|
||||
| `TestInstallModel::test_install_model_missing_ui_id` | POST queue/install_model | Missing ui_id | 400 |
|
||||
| `TestInstallModel::test_install_model_invalid_body` | POST queue/install_model | Invalid metadata | 400 |
|
||||
| `TestUpdateAll::test_update_all_queues_tasks` | POST queue/update_all | Queue all update tasks | 200/403 or tolerated ReadTimeout |
|
||||
| `TestUpdateAll::test_update_all_missing_params` | POST queue/update_all | Missing params | 400 ValidationError |
|
||||
| `TestUpdateComfyUI::test_update_comfyui_queues_task` | POST queue/update_comfyui | Queue task | 200 + total_count>=1 after |
|
||||
| `TestUpdateComfyUI::test_update_comfyui_missing_params` | POST queue/update_comfyui | Missing params | 400 |
|
||||
| `TestUpdateComfyUI::test_update_comfyui_with_stable_flag` | POST queue/update_comfyui?stable=true | Explicit stable flag | 200 |
|
||||
|
||||
## tests/e2e/test_e2e_version_mgmt.py (7 tests)
|
||||
|
||||
Covers comfyui_versions + comfyui_switch_version endpoints.
|
||||
|
||||
| Test | Endpoint(s) | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `TestComfyUIVersions::test_versions_endpoint` | GET comfyui_versions | Response shape | `{versions: list, current: str}` |
|
||||
| `TestComfyUIVersions::test_versions_list_not_empty` | GET comfyui_versions | Non-empty list | len>0 |
|
||||
| `TestComfyUIVersions::test_versions_items_are_strings` | GET comfyui_versions | Item type | each version is string |
|
||||
| `TestComfyUIVersions::test_current_is_in_versions` | GET comfyui_versions | Current in list | current appears in versions |
|
||||
| `TestSwitchVersionNegative::test_switch_version_missing_all_params` | POST comfyui_switch_version | No params | 400 or 403 |
|
||||
| `TestSwitchVersionNegative::test_switch_version_missing_client_id` | POST comfyui_switch_version?ver=X | Partial params | 400 or 403 |
|
||||
| `TestSwitchVersionNegative::test_switch_version_validation_error_body` | POST comfyui_switch_version | Error body shape | `error` field present (when 400 JSON) |
|
||||
|
||||
> Note: Actual version switching (destructive) intentionally NOT tested.
|
||||
|
||||
---
|
||||
|
||||
## tests/e2e/test_e2e_csrf.py (4 test functions / 26 parametrized invocations — post-WI-HH)
|
||||
|
||||
Covers the CSRF-mitigation contract from commit 99caef55 — state-changing
|
||||
endpoints must reject HTTP GET so that `<img src>` / link-click /
|
||||
redirect-based cross-origin triggers cannot mutate server state.
|
||||
|
||||
**Scope (per docstring)**: ONLY the GET-rejection contract. NOT covered
|
||||
here: Origin/Referer validation (separate middleware), same-site cookies,
|
||||
anti-CSRF tokens, cross-site form POST.
|
||||
|
||||
| Test | Endpoint(s) | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `TestStateChangingEndpointsRejectGet::test_get_is_rejected[path]` | 13 POST endpoints (queue/start, queue/reset, queue/update_all, queue/update_comfyui, queue/install_model, queue/task, snapshot/save, snapshot/remove, snapshot/restore, manager/reboot, comfyui_switch_version, import_fail_info, import_fail_info_bulk — WI-HH removed db_mode, policy/update, channel_url_list from this list since they legitimately answer GET on the read-path) | GET must reject | status_code in (400,403,404,405); explicit `not in 200-399` guard |
|
||||
| `TestCsrfPostWorks::test_queue_reset_post_works` | POST queue/reset | POST counterpart works | status_code == 200 |
|
||||
| `TestCsrfPostWorks::test_snapshot_save_post_works` | POST snapshot/save + cleanup via getlist+remove | POST counterpart works | status_code == 200; cleanup |
|
||||
| `TestCsrfReadEndpointsStillAllowGet::test_get_read_endpoint_succeeds[path]` | 11 GET endpoints (version, db_mode, policy/update, channel_url_list, queue/status, queue/history_list, is_legacy_manager_ui, customnode/installed, snapshot/getlist, snapshot/get_current, comfyui_versions) | Negative control: read-only still works | status_code == 200 |
|
||||
|
||||
> Note: Three endpoints (`db_mode`, `policy/update`, `channel_url_list`) appear in BOTH reject-GET (POST path, write) and allow-GET (read path) lists — commit 99caef55 split each into a GET-read + POST-write pair; the POST path must reject GET while the GET path must continue to succeed.
|
||||
|
||||
---
|
||||
|
||||
## tests/e2e/test_e2e_csrf_legacy.py (4 test functions / 26 parametrized invocations)
|
||||
|
||||
Legacy-mode counterpart to `test_e2e_csrf.py`. Verifies the same CSRF
|
||||
method-rejection contract but against the legacy server module loaded
|
||||
via `--enable-manager-legacy-ui`. Added in WI-FF (commit following
|
||||
99caef55) to close the legacy-side regression-guard gap. Audit-integrated
|
||||
in WI-GG.
|
||||
|
||||
**Why a separate file** (per docstring L7–13): `comfyui_manager/__init__.py`
|
||||
loads `glob.manager_server` XOR `legacy.manager_server` via mutex on the
|
||||
`--enable-manager-legacy-ui` flag. One ComfyUI process exposes either the
|
||||
glob or the legacy route table, never both — so verifying the legacy
|
||||
CSRF contract requires its own module-scoped server lifecycle with the
|
||||
legacy flag set (via `start_comfyui_legacy.sh`).
|
||||
|
||||
**Scope (per docstring L44–48)**: Same as `test_e2e_csrf.py` — ONLY the
|
||||
method-reject layer. Origin/Referer, same-site cookies, anti-CSRF tokens,
|
||||
and cross-site form POST are out of scope.
|
||||
|
||||
| Test | Endpoint(s) | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `TestLegacyStateChangingEndpointsRejectGet::test_get_is_rejected[path]` | 13 POST endpoints (queue/start, queue/reset, queue/update_all, queue/update_comfyui, queue/install_model, **queue/batch** (legacy; replaces queue/task), snapshot/save, snapshot/remove, snapshot/restore, manager/reboot, comfyui_switch_version, import_fail_info, import_fail_info_bulk) | GET must reject under legacy server | status_code in (400,403,404,405); explicit `not in 200-399` guard |
|
||||
| `TestLegacyCsrfPostWorks::test_queue_reset_post_works` | POST queue/reset (legacy) | POST counterpart works under legacy server | status_code == 200 |
|
||||
| `TestLegacyCsrfPostWorks::test_snapshot_save_post_works` | POST snapshot/save + cleanup via getlist+remove (legacy) | POST counterpart works + cleanup | status_code == 200; cleanup |
|
||||
| `TestLegacyCsrfReadEndpointsStillAllowGet::test_get_read_endpoint_succeeds[path]` | 11 GET endpoints (version, db_mode, policy/update, channel_url_list, queue/status, queue/history_list, is_legacy_manager_ui, customnode/installed, snapshot/getlist, snapshot/get_current, comfyui_versions) | Negative control: legacy read-only still works | status_code == 200 |
|
||||
|
||||
> **Endpoint-list deltas vs glob** (per docstring L23–36):
|
||||
> - `queue/task` → dropped (glob-only); `queue/batch` → added (legacy task-enqueue equivalent)
|
||||
> - `db_mode`, `policy/update`, `channel_url_list` → dropped from reject-GET (CSRF contract applies only to the POST write-path; legacy splits these into `@routes.get` read + `@routes.post` write, identical to glob). These 3 remain in the ALLOW-GET class above. (The glob `test_e2e_csrf.py` lists them in BOTH classes; WI-HH tracks the glob-side correction.)
|
||||
|
||||
---
|
||||
|
||||
## tests/e2e/test_e2e_secgate_strict.py (1 test active + 1 skipped; WI-KK PoC, WI-LL audit-integrated)
|
||||
|
||||
Strict-mode security-gate PoC. Covers the middle/middle+ gate 403 contract for
|
||||
Goals that require elevating `security_level=strong`. Launches via
|
||||
`start_comfyui_strict.sh` (which patches `user/__manager/config.ini` to
|
||||
`security_level=strong`, leaves a `.before-strict` backup, and starts the server
|
||||
on the E2E port) and restores the original config in the fixture teardown.
|
||||
|
||||
**Scope (per docstring L3–9)**: strict-mode 403 path for the middle/middle+
|
||||
gates. The default E2E config (`security_level=normal`, `is_local_mode=True`)
|
||||
puts NORMAL inside the allowed set for both gates per
|
||||
`comfyui_manager/glob/utils/security_utils.py` L32–38, so this harness is the
|
||||
only way to exercise the 403 side. This is the first of 4 planned Goals
|
||||
(SR4 ← here; SR6, V5, UA2 ← mechanical additions using the same fixture).
|
||||
|
||||
| Test | Endpoint(s) | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `TestSecurityGate403_SR4::test_remove_returns_403` | POST `/v2/snapshot/remove?target=…` (under security_level=strong) | Goal SR4 — snapshot/remove below `middle` | (a) `status_code == 403`; (b) the seeded snapshot file on disk is NOT deleted (negative-check per `verification_design.md` §7.3 Security Boundary Template). |
|
||||
| `TestSecurityGate403_SR4::test_post_works_at_default_after_restore` | (none — pytest.skip) | Positive counterpart of SR4 at default config | pytest.skip'd: deferred to `test_e2e_secgate_default.py` follow-up to avoid double-startup cost. Documents both halves of the gate contract. |
|
||||
|
||||
**Harness notes**:
|
||||
- **Teardown ordering** is contract-critical: stop server FIRST, then restore config (the server holds the config-file lock; restoring before stopping causes a re-snapshot race). Documented in the fixture's `finally` block.
|
||||
- Subsequent test modules continue to see `security_level=normal` because the backup restore happens deterministically in teardown.
|
||||
|
||||
## tests/e2e/test_e2e_secgate_default.py (1 test; WI-KK demo, WI-LL audit-integrated)
|
||||
|
||||
Default-mode security-gate demonstration. Covers the CV4 Goal (comfyui_switch_version
|
||||
`high+` gate 403 contract) without any harness, leveraging the WI-KK research
|
||||
finding that default `security_level=normal` + `is_local_mode=True` already
|
||||
triggers 403 for high+ operations at the HTTP handler. This is the cleanest of
|
||||
the 4 originally-classified-T2 high+ Goals to demonstrate the no-harness-needed
|
||||
insight.
|
||||
|
||||
**Scope (per docstring L1–18)**: only the CV4 Goal. The other 3 originally-T2
|
||||
high+ Goals are deferred with reclassification notes:
|
||||
- **IM4** → **T2-TASKLEVEL**: non-safetensors check lives deep in the install pipeline (worker + `get_risky_level`), not at the HTTP handler. POST `/v2/manager/queue/install_model` accepts the request and queues a task; rejection only surfaces at task execution. Requires a queue-observation pattern, not a simple HTTP 403 check.
|
||||
- **LGU2**, **LPP2** → **NORMAL-legacy**: registered ONLY in `legacy/manager_server.py` (L1502, L1522). Testing needs `start_comfyui_legacy.sh` fixture — follow-up `test_e2e_secgate_legacy_default.py` is the natural home.
|
||||
|
||||
| Test | Endpoint(s) | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `TestSecurityGate403_CV4::test_switch_version_returns_403_at_default` | POST `/v2/comfyui_manager/comfyui_switch_version` with `ver`, `client_id`, `ui_id` (at default security_level) | Goal CV4 — comfyui_switch_version below `high+` | `status_code == 403`. The `ver` query is syntactically valid so the request WOULD reach the Pydantic validation step IF the gate were broken; since the gate is the FIRST check in the handler, 403 must precede any 400-from-validation outcome. |
|
||||
|
||||
---
|
||||
|
||||
# Section 2 — Playwright UI Tests
|
||||
|
||||
All Playwright tests require ComfyUI running with `--enable-manager-legacy-ui` on PORT (default 8199).
|
||||
|
||||
## tests/playwright/legacy-ui-manager-menu.spec.ts (5 tests)
|
||||
|
||||
Covers the Manager Menu dialog and its settings dropdowns.
|
||||
|
||||
| Test | Endpoint(s) exercised | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `Manager Menu Dialog > opens via Manager button and shows 3-column layout` | (indirect: initial page + legacy UI detection) | Menu dialog opens | `#cm-manager-dialog` visible; "Custom Nodes Manager", "Model Manager", "Restart" buttons present |
|
||||
| `> shows settings dropdowns (DB, Channel, Policy)` | (UI) | DB + Policy combos render | Both `<select>` elements visible |
|
||||
| `> DB mode dropdown round-trips via API` | GET/POST db_mode | UI dropdown change → backend persists | selectOption → verify via GET → restore |
|
||||
| `> Update Policy dropdown round-trips via API` | GET/POST policy/update | Policy change via UI | selectOption → verify GET → restore |
|
||||
| `> closes and reopens without duplicating` | (UI only) | Dialog lifecycle | No duplicate dialog instances |
|
||||
|
||||
## tests/playwright/legacy-ui-custom-nodes.spec.ts (5 tests)
|
||||
|
||||
Covers the Custom Nodes Manager dialog (TurboGrid-based list).
|
||||
|
||||
| Test | Endpoint(s) exercised | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `Custom Nodes Manager > opens from Manager menu and renders grid` | GET customnode/getlist (legacy), customnode/getmappings | Grid render | `#cn-manager-dialog` + `.tg-body` visible |
|
||||
| `> loads custom node list (non-empty)` | GET customnode/getlist | Data load | rows > 0 after polling |
|
||||
| `> filter dropdown changes displayed nodes` | (client-side filter) | "Installed" filter | filtered count ≤ initial count |
|
||||
| `> search input filters the grid` | (client-side filter) | Search term | filtered count ≤ initial |
|
||||
| `> footer buttons are present` | (UI) | Install via Git URL / Restart buttons | At least one present |
|
||||
|
||||
## tests/playwright/legacy-ui-model-manager.spec.ts (4 tests)
|
||||
|
||||
Covers Model Manager dialog.
|
||||
|
||||
| Test | Endpoint(s) exercised | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `Model Manager > opens from Manager menu and renders grid` | GET externalmodel/getlist (legacy) | Grid render | `#cmm-manager-dialog` + grid visible |
|
||||
| `> loads model list (non-empty)` | GET externalmodel/getlist | Data load | rows > 0 |
|
||||
| `> search input filters the model grid` | (client-side filter) | Search | filtered ≤ initial |
|
||||
| `> filter dropdown is present with expected options` | (UI) | Filter options | options.length > 0 |
|
||||
|
||||
## tests/playwright/legacy-ui-snapshot.spec.ts (3 tests)
|
||||
|
||||
Covers Snapshot Manager dialog.
|
||||
|
||||
| Test | Endpoint(s) exercised | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `Snapshot Manager > opens snapshot manager from Manager menu` | (UI) | Dialog opens | `#snapshot-manager-dialog` present |
|
||||
| `> lists existing snapshots` | GET snapshot/getlist | List loads | resp.ok + `items` property |
|
||||
| `> save snapshot via API and verify in list` | POST snapshot/save + GET snapshot/getlist + POST snapshot/remove | Save→verify→cleanup | items.length > 0 after save; remove cleanup |
|
||||
|
||||
## tests/playwright/legacy-ui-navigation.spec.ts (2 tests)
|
||||
|
||||
Covers dialog navigation lifecycle. Stage2 WI-F deleted two prior tests (`API health check while dialogs are open`, `system endpoints accessible from browser context`) because they exercised `page.request.*` direct API calls with no real UI interaction — coverage is now owned by `test_e2e_system_info.py::test_version_*` and related pytest suites.
|
||||
|
||||
| Test | Endpoint(s) exercised | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `Dialog Navigation > Manager menu → Custom Nodes → close → Manager still visible` | (UI) | Nested dialog navigation | Manager menu reopens after child close |
|
||||
| `> Manager menu → Model Manager → close → reopen` | (UI) | Close and reopen Model Manager | Dialog reappears |
|
||||
|
||||
## tests/playwright/legacy-ui-install.spec.ts (2 tests)
|
||||
|
||||
Covers UI-driven install/uninstall effect verification against the test pack `ComfyUI_SigmoidOffsetScheduler`. Primary action is always a UI button click; `page.request` is used only for setup (queue/reset baseline, optional API pre-install in LB2) and effect-observation (queue/status polling, installed-list lookup) — consistent with the hybrid UI-action + backend-effect pattern in `legacy-ui-snapshot.spec.ts::SS1`.
|
||||
|
||||
| Test | Endpoint(s) exercised | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `UI-driven install/uninstall > LB1 Install button triggers install effect` | (UI) Custom Nodes Manager dialog → filter "Not Installed" → search pack → row Install button → version "Select" button; GET /v2/manager/queue/status (effect polling), GET /v2/customnode/installed (effect verification) | User initiates install from Custom Nodes dialog | `isPackInstalled === true` after queue drains via `waitForAllDone` |
|
||||
| `UI-driven install/uninstall > LB2 Uninstall button triggers uninstall effect` | (UI) Custom Nodes Manager dialog → filter "Installed" → search pack → row Uninstall button → optional confirm dialog; GET /v2/manager/queue/status, GET /v2/customnode/installed | User initiates uninstall from Custom Nodes dialog (preconditioned by API install if pack absent) | `isPackInstalled === false` after queue drains |
|
||||
|
||||
## tests/playwright/debug-install-flow.spec.ts (1 test)
|
||||
|
||||
Debug/instrumentation test — captures the install API flow for documentation.
|
||||
|
||||
| Test | Endpoint(s) exercised | Scenario | Assertion semantics |
|
||||
|---|---|---|---|
|
||||
| `capture install button API flow` | GET customnode/getlist, POST queue/batch (legacy), GET customnode/versions/{id}, WebSocket cm-queue-status | End-to-end install UI flow capture | No assertions — logs API sequence + WebSocket frames for manual review |
|
||||
|
||||
---
|
||||
|
||||
# Section 3 — Endpoint Coverage Summary
|
||||
|
||||
## Glob v2 endpoints covered (27/30)
|
||||
|
||||
| Endpoint | Covered by |
|
||||
|---|---|
|
||||
| POST queue/task (install) | test_e2e_endpoint, test_e2e_git_clone, test_e2e_task_operations |
|
||||
| POST queue/task (update/fix/disable/enable/uninstall) | test_e2e_endpoint, test_e2e_task_operations |
|
||||
| GET queue/history_list | test_e2e_queue_lifecycle |
|
||||
| GET queue/history | test_e2e_queue_lifecycle, test_e2e_task_operations |
|
||||
| GET customnode/getmappings | test_e2e_customnode_info |
|
||||
| GET customnode/fetch_updates | test_e2e_customnode_info (deprecated 410) |
|
||||
| POST queue/update_all | test_e2e_task_operations |
|
||||
| GET is_legacy_manager_ui | test_e2e_system_info, playwright legacy-ui-navigation |
|
||||
| GET customnode/installed | test_e2e_endpoint, test_e2e_customnode_info |
|
||||
| GET snapshot/getlist | test_e2e_snapshot_lifecycle, playwright legacy-ui-snapshot |
|
||||
| POST snapshot/remove | test_e2e_snapshot_lifecycle |
|
||||
| GET snapshot/get_current | test_e2e_snapshot_lifecycle |
|
||||
| POST snapshot/save | test_e2e_snapshot_lifecycle, playwright legacy-ui-snapshot |
|
||||
| POST customnode/import_fail_info | test_e2e_customnode_info |
|
||||
| POST customnode/import_fail_info_bulk | test_e2e_customnode_info |
|
||||
| POST queue/reset | test_e2e_queue_lifecycle, test_e2e_task_operations |
|
||||
| GET queue/status | test_e2e_queue_lifecycle, test_e2e_task_operations |
|
||||
| POST queue/start | test_e2e_endpoint, test_e2e_task_operations |
|
||||
| POST queue/update_comfyui | test_e2e_task_operations |
|
||||
| GET comfyui_versions | test_e2e_version_mgmt |
|
||||
| POST comfyui_switch_version | test_e2e_version_mgmt (negative only) |
|
||||
| POST queue/install_model | test_e2e_task_operations |
|
||||
| GET/POST db_mode | test_e2e_config_api, playwright legacy-ui-manager-menu |
|
||||
| GET/POST policy/update | test_e2e_config_api, playwright legacy-ui-manager-menu |
|
||||
| GET/POST channel_url_list | test_e2e_config_api |
|
||||
| POST manager/reboot | test_e2e_system_info |
|
||||
| GET manager/version | test_e2e_system_info, playwright legacy-ui-navigation |
|
||||
|
||||
## CSRF Method-Reject Contract
|
||||
|
||||
Separate from the positive-path coverage above, the 16 state-changing POST
|
||||
endpoints (glob) + 13 (legacy, with queue/batch substitution) plus 11
|
||||
read-only GET endpoints per server are independently verified for their
|
||||
CSRF-mitigation contract (commit 99caef55, CVSS 8.1). Coverage is split
|
||||
across two files because server loading is mutex on `--enable-manager-legacy-ui`:
|
||||
|
||||
**Glob server** — `tests/e2e/test_e2e_csrf.py`:
|
||||
|
||||
| Contract | Tests | Coverage |
|
||||
|---|---|---|
|
||||
| 13 POST endpoints must reject HTTP GET (glob; post-WI-HH) | TestStateChangingEndpointsRejectGet (parametrized ×13) | ✓ full |
|
||||
| POST counterparts must work (glob sanity) | TestCsrfPostWorks (queue/reset, snapshot/save) | ~ spot-check |
|
||||
| 11 read-only GET endpoints must still allow GET (glob negative control) | TestCsrfReadEndpointsStillAllowGet (parametrized ×11) | ✓ full |
|
||||
|
||||
**Legacy server** (WI-FF) — `tests/e2e/test_e2e_csrf_legacy.py`:
|
||||
|
||||
| Contract | Tests | Coverage |
|
||||
|---|---|---|
|
||||
| 13 POST endpoints must reject HTTP GET (legacy; queue/task→queue/batch; dual-purpose endpoints scoped to ALLOW-GET only) | TestLegacyStateChangingEndpointsRejectGet (parametrized ×13) | ✓ full |
|
||||
| POST counterparts must work (legacy sanity) | TestLegacyCsrfPostWorks (queue/reset, snapshot/save) | ~ spot-check |
|
||||
| 11 read-only GET endpoints must still allow GET (legacy negative control) | TestLegacyCsrfReadEndpointsStillAllowGet (parametrized ×11) | ✓ full |
|
||||
|
||||
Note: this contract is NEGATIVE-assertion (must-reject) + negative-control.
|
||||
Do NOT interpret CSRF-suite PASS as "CSRF fully solved" — both suites
|
||||
explicitly scope themselves to the method-conversion layer only. The
|
||||
legacy suite closes the gap where a reverted `@routes.post` → `@routes.get`
|
||||
in `legacy/manager_server.py` would have slipped past CI.
|
||||
|
||||
## Glob v2 endpoints NOT covered
|
||||
|
||||
| Endpoint | Reason |
|
||||
|---|---|
|
||||
| POST snapshot/restore | Intentionally skipped (destructive — alters node state) |
|
||||
| POST comfyui_switch_version (positive) | Intentionally skipped (destructive — alters ComfyUI version) |
|
||||
| (none otherwise missing) | — |
|
||||
|
||||
## Legacy-only endpoints covered
|
||||
|
||||
| Endpoint | Covered by |
|
||||
|---|---|
|
||||
| POST queue/batch | playwright debug-install-flow (indirect — triggered via Install UI) |
|
||||
| GET customnode/getlist | playwright legacy-ui-custom-nodes (indirect) |
|
||||
| GET externalmodel/getlist | playwright legacy-ui-model-manager (indirect) |
|
||||
|
||||
## Legacy-only endpoints NOT covered
|
||||
|
||||
| Endpoint | Reason |
|
||||
|---|---|
|
||||
| GET /customnode/alternatives | Not invoked by legacy UI flows tested |
|
||||
| GET customnode/versions/{node_name} | Tested indirectly via install version dialog (debug-install-flow) but no direct assertion |
|
||||
| GET customnode/disabled_versions/{node_name} | No direct test |
|
||||
| POST customnode/install/git_url | High+ security, destructive; not in UI flow |
|
||||
| POST customnode/install/pip | High+ security, destructive |
|
||||
| GET manager/notice | Removed in recent work; legacy only |
|
||||
|
||||
---
|
||||
*End of Report B*
|
||||
@@ -0,0 +1,629 @@
|
||||
# E2E Verification Condition Audit
|
||||
|
||||
**Generated**: 2026-04-18
|
||||
**Method**: For each E2E test function, compare its actual assertions against the required verification items from `verification_design.md`.
|
||||
|
||||
**Verdict categories**:
|
||||
- **✅ PASS** — verification adequate; matches design Goal
|
||||
- **⚠️ WEAK** — covers core but misses key assertions (effect proof, negative checks, side effects)
|
||||
- **❌ INADEQUATE** — verification insufficient (status-only, or missing the actual intent)
|
||||
- **N/A** — outside verification_design scope (e.g., CLI tests)
|
||||
|
||||
---
|
||||
|
||||
# Section 1 — tests/e2e/test_e2e_endpoint.py (4 tests)
|
||||
|
||||
| Test | Design Goal | Verdict | Actual assertions | Issues |
|
||||
|---|---|---|---|---|
|
||||
| `TestEndpointInstallUninstall::test_install_via_endpoint` | A1 (Install CNR pack) | ✅ PASS | `_pack_exists` + `_has_tracking` | Meets effect requirement |
|
||||
| `test_installed_list_shows_pack` | IL1 (Installed list current) | ✅ PASS | cnr_id match in response dict | Effect verified via API |
|
||||
| `test_uninstall_via_endpoint` | U1 (Remove pack) | ✅ PASS | Wave1 WI-N: FS check + API cross-check — asserts cnr_id ABSENT from GET /v2/customnode/installed. Defeats cache-invalidation regressions where FS delete succeeds but the installed-index still reports the pack. |
|
||||
| `test_startup_resolver_ran` | (log assertion) | N/A | Log file contains specific strings | Not HTTP verification; ComfyUI startup side check |
|
||||
|
||||
**File verdict**: 3/4 ✅, 0/4 ⚠️, 1/4 N/A (WI-MM removed 3 B1/B5 rows: `test_installed_list_after_uninstall` subsumed by the WI-N-strengthened `test_uninstall_via_endpoint`, `test_install_uninstall_cycle` subsumed by the concat of ci-001/002/003, `test_comfyui_started` subsumed by `_start_comfyui`'s /system_stats readiness poll.)
|
||||
|
||||
---
|
||||
|
||||
# Section 2 — tests/e2e/test_e2e_git_clone.py (3 tests)
|
||||
|
||||
| Test | Design Goal | Verdict | Actual assertions | Issues |
|
||||
|---|---|---|---|---|
|
||||
| `test_01_nightly_install` | A2 (Install nightly via URL) | ✅ PASS | Wave1 WI-N: pack_exists + `.git/` dir + parses `.git/config` and asserts `[remote "origin"] url` matches REPO_TEST1 (tolerant of `.git` suffix variants). Defeats "wrong-repo clone" regression. |
|
||||
| `test_02_no_module_error` | A2 negative check | ✅ PASS | log NOT contains ModuleNotFoundError | Negative check correct |
|
||||
| `test_03_nightly_uninstall` | U1 (Uninstall nightly) | ✅ PASS | Wave1 WI-N: FS check + API cross-check — asserts PACK_TEST1 absent from installed-list keys + defensive cnr_id/aux_id traversal to catch schema-variation regressions. |
|
||||
|
||||
**File verdict**: 3/3 ✅ (Wave1 WI-N upgraded test_01_nightly_install A2 + test_03_nightly_uninstall U1)
|
||||
|
||||
---
|
||||
|
||||
<!-- Section 3 (tests/e2e/test_e2e_uv_compile.py) was relocated to tests/cli/test_uv_compile.py
|
||||
in WI-PP. The 8 functions were CLI-subprocess integration tests (cm-cli --uv-compile),
|
||||
not HTTP/UI E2E, and are now tracked outside this audit's scope. See CHANGELOG: WI-PP. -->
|
||||
|
||||
# Section 4 — tests/e2e/test_e2e_config_api.py (9 tests)
|
||||
|
||||
| Test | Design Goal | Verdict | Issues |
|
||||
|---|---|---|---|
|
||||
| `test_read_db_mode` | C1 (GET db_mode) | ✅ PASS | Response in enum set |
|
||||
| `test_set_and_restore_db_mode` | C2 (POST persistence) | ✅ PASS | WI-E/WI-G helpers applied: disk mutation (config.ini) + reboot persistence verified |
|
||||
| `test_read_update_policy` | C1 (policy) | ✅ PASS | Response in enum set |
|
||||
| `test_set_and_restore_update_policy` | C2 (policy persistence) | ✅ PASS | WI-E/WI-G helpers applied: disk mutation (config.ini) + reboot persistence verified |
|
||||
| `test_read_channel_url_list` | C4 (channel list) | ✅ PASS | Shape verified |
|
||||
| `test_channel_list_entries_are_name_url_strings` | C4 format | ✅ PASS | "name::url" format |
|
||||
| `test_set_and_restore_channel` | C5 (switch channel) | ✅ PASS | WI-E/WI-G helpers applied: disk mutation (config.ini) + reboot persistence verified. Retained as separate function (not merged with db_mode/policy roundtrip) — the channel_url_list endpoint carries URL↔NAME asymmetry that makes a single parametrized body a branch-soup; WI-NN Cluster 1 skipped this merge and only applies Clusters 2+3. |
|
||||
| `test_malformed_body_returns_400` (parametrized ×3: db_mode / update_policy / channel_url_list) | C3 (malformed JSON) | ✅ PASS | WI-NN Cluster 2 (bloat teng:ci-003/008/015 B9): consolidates the 3 previously-separate `test_set_*_invalid_body` tests into one parametrized function. Each invocation asserts 400 + config.ini unchanged via `_assert_config_ini_contains`. |
|
||||
| `test_junk_value_rejected` (parametrized ×3: db_mode / update_policy / channel_url_list) | C3 (whitelist reject) | ✅ PASS | WI-NN Cluster 3 (bloat teng:ci-004/009/014 B9): consolidates the 3 previously-separate whitelist-reject tests. For db_mode/policy (static whitelist) the on-disk value must remain in the valid-values set; for channel (dynamic whitelist) the API-level NAME + disk URL must be unchanged. |
|
||||
|
||||
**File verdict**: 9/9 ✅ (WI-Z Y3 + WI-MM produced the 13-row baseline. WI-NN parametrized Clusters 2 (invalid-body) + 3 (junk-value) — 6 source tests → 2 parametrized functions (still 6 invocations; audit counts rows by function). Count: 13→9. Cluster 1 (roundtrip) was skipped due to channel URL↔NAME asymmetry.)
|
||||
|
||||
**Common gap**: RESOLVED via WI-E (disk-persistence helper) + WI-G (propagation to all 6 prior-WEAK tests) + WI-I (whitelist enforcement for db_mode / policy / channel). Every POST test now asserts both **config.ini file mutation on disk** and **survive-restart persistence** (positive path) or **config UNCHANGED on disk** (negative path). Whitelist rejection of unknown enum values is exercised end-to-end across all three config endpoints.
|
||||
|
||||
---
|
||||
|
||||
# Section 5 — tests/e2e/test_e2e_customnode_info.py (10 tests)
|
||||
|
||||
| Test | Design Goal | Verdict | Issues |
|
||||
|---|---|---|---|
|
||||
| `test_getmappings_returns_dict` | CM1 | ✅ PASS | Wave1 WI-M: non-empty DB check (>=100 entries) + per-entry schema sample (first 5 entries must be `[node_list: list, metadata: dict]`). Defeats empty-DB regression. |
|
||||
| `test_fetch_updates_returns_deprecated` | FU1 | ✅ PASS | 410 + deprecated:true |
|
||||
| `test_installed_returns_dict` | IL1 | ✅ PASS | Wave1 WI-M: asserts E2E seed pack `ComfyUI_SigmoidOffsetScheduler` is present AND its entry carries the documented InstalledPack fields (cnr_id/ver/enabled). |
|
||||
| `test_installed_imported_mode` | IL2 | ✅ PASS | Wave3 WI-T Cluster G target 4 (research-cluster-g.md Strategy A): asserts (a) 200 + dict, (b) seed pack `ComfyUI_SigmoidOffsetScheduler` present, (c) each entry carries the documented InstalledPack schema (cnr_id/ver/enabled), (d) frozen-at-startup invariant (cheap form) — imported keys == default keys at test time (no mid-session install). WI-OO Item 4 (bloat reviewer:ci-013 B7) removed the skip-masked `test_imported_mode_is_frozen_after_install` stub-companion — without an implemented install trigger between the two GETs, `snap_before == snap_after` held trivially. True frozen-vs-live-and-equal coverage (Strategy B) remains an E2E-DEBT for a future WI that wires the mid-session install. |
|
||||
| `test_unknown_cnr_id_returns_400` | IF2 | ✅ PASS | 400 verified |
|
||||
| `test_missing_fields_returns_400` | IF3 | ✅ PASS | 400 verified |
|
||||
| `test_invalid_body_returns_error` | IF3 (non-dict) | ✅ PASS | 400 verified |
|
||||
| `test_bulk_with_cnr_ids_returns_dict` | IFB1 | ✅ PASS | null for unknown verified |
|
||||
| `test_bulk_empty_lists_returns_400` | IFB2 | ✅ PASS | 400 verified |
|
||||
| `test_bulk_with_urls_returns_dict` | IFB1 | ✅ PASS | Wave1 WI-M: asserts per-url result — requested URL is a key in the response, and its value is either None (unknown URL, expected here) or a dict (populated fail-info). Defeats schema-violation regressions. |
|
||||
|
||||
**File verdict**: 10/10 ✅ (Wave1 WI-M upgraded 3 rows: test_getmappings_returns_dict, test_installed_returns_dict, test_bulk_with_urls_returns_dict. Wave3 WI-T upgraded test_installed_imported_mode IL2 — Strategy A cheap invariant + Strategy B [E2E-DEBT] skip-companion. WI-MM removed `test_getmappings_entries_have_node_lists` (bloat-sweep reviewer:ci-009 B1) — the strengthened `test_getmappings_returns_dict` now checks the first 5 entries' `[node_list, metadata]` schema, so this row's entry[0]-as-list assertion is a strict subset. Count: 11→10.)
|
||||
|
||||
**Key gap**: IF1 (positive path — known failed pack returning info) NOT tested. [E2E-DEBT] — Strategy B ("frozen vs live-and-coincidentally-equal") requires a mid-session install trigger; the previous skip-masked `test_imported_mode_is_frozen_after_install` stub was removed in WI-OO Item 4 because the TODO had never been implemented and the skipped body proved nothing. Register a future WI to wire the install step and re-add the test.
|
||||
|
||||
---
|
||||
|
||||
# Section 6 — tests/e2e/test_e2e_queue_lifecycle.py (7 tests)
|
||||
|
||||
| Test | Design Goal | Verdict | Issues |
|
||||
|---|---|---|---|
|
||||
| `test_reset_queue` | R1 | ✅ PASS | Wave1 WI-L: now verifies post-reset queue/status payload — all 4 counters (pending/in_progress/total/done) == 0 AND is_processing is False. Catches reset-handler regressions and cross-module state leak. |
|
||||
| `test_status_with_client_id_filter` | QS2 | ✅ PASS | client_id echo verified |
|
||||
| `test_start_queue_already_idle` | S1/S2 | ✅ PASS | Wave1 WI-L: polls queue/status for up-to-10s after POST /queue/start and asserts worker stabilizes to idle (pending==0, in_progress==0, is_processing==False). Defeats hot-loop regressions where start_worker() spawns a thread that never exits on empty queue. |
|
||||
| `test_queue_task_and_history` | A1 + QH3 | ✅ PASS | done_count polling + history accepted |
|
||||
| `test_history_with_ui_id_filter` | QH3 | ✅ PASS | Wave3 WI-T Cluster C target 1: discovers an existing ui_id via unfiltered call (seeds lightweight install if history empty), then asserts every entry in the filtered response matches that ui_id. Shape-resilient extractor handles `{ui_id: task}` maps and task-dict-directly variants. Defeats regressions where the server accepts the param but returns unfiltered history. |
|
||||
| `test_history_with_pagination` | QH3 pagination | ✅ PASS | Wave3 WI-T Cluster C target 2: verifies max_items cap (max_items=1 → len≤1), no silent truncation (max_items ≥ full_count → len == full_count), and offset progression (offset=0 vs offset=1 return different keys when ≥2 entries exist). |
|
||||
| `test_history_list` | QHL1 | ✅ PASS | Wave3 WI-T Cluster C target 3: cross-references API response with filesystem `user/__manager/batch_history/*.json` — set equality between API `ids` and the basenames (sans `.json`) of JSON files on disk. No phantom ids, no missing ids. |
|
||||
|
||||
**File verdict**: 7/7 ✅ (Wave1 WI-L upgraded 2 rows — test_reset_queue, test_start_queue_already_idle. Wave3 WI-T Cluster C upgraded 3 rows — test_history_with_ui_id_filter QH3 filter-semantic, test_history_with_pagination QH3 cap + consistency + offset, test_history_list QHL1 API↔FS set equality. WI-MM removed 2 B1/B8 rows: `test_status_after_reset` (weaker subset of the WI-L-strengthened `test_reset_queue`, bloat-sweep teng:ci-017) and `test_final_reset_and_clean_state` (subset of ci-016 + misleading 'final' name — pytest test order is not guaranteed, bloat-sweep teng:ci-024). Count: 9→7.)
|
||||
|
||||
**Key gaps**: `test_history_path_traversal_rejected` (QH2 path traversal) is present in the file and passing. Remaining gap: no batch-id retrieval positive-path test (GET /v2/manager/queue/history?id=<batch_id>).
|
||||
|
||||
---
|
||||
|
||||
# Section 7 — tests/e2e/test_e2e_snapshot_lifecycle.py (7 tests)
|
||||
|
||||
| Test | Design Goal | Verdict | Issues |
|
||||
|---|---|---|---|
|
||||
| `test_get_current_snapshot` | SG1 | ✅ PASS | Wave1 WI-M: asserts documented top-level schema (comfyui / git_custom_nodes / cnr_custom_nodes / file_custom_nodes / pips) AND cross-references installed FS state — seed pack `ComfyUI_SigmoidOffsetScheduler` on disk → must also appear in `cnr_custom_nodes` dict. |
|
||||
| `test_save_snapshot` | SS1 | ✅ PASS | Wave2 WI-Q: verifies (a) new *.json file appears on disk under SNAPSHOT_DIR via os.listdir diff + file parses as JSON dict, AND (b) saved file's `cnr_custom_nodes` dict matches live GET /v2/snapshot/get_current response (pack_name → version). Catches regressions that write stale/stub snapshots while 200 OK. |
|
||||
| `test_getlist_after_save` | SS1 + SL1 | ✅ PASS | items.length>0 verifies save effect |
|
||||
| `test_remove_snapshot` | SR1 | ✅ PASS | Target absent + count decremented |
|
||||
| `test_remove_nonexistent_snapshot` | SR2 | ✅ PASS | 200 no-op |
|
||||
| `test_remove_path_traversal_rejected` | SR3 | ✅ PASS | WI-Z Y1 (resolves prior SR3 Key gap): POST `/v2/snapshot/remove` with path-traversal targets (`../../_sentinel_must_not_delete`, `../../../etc/passwd`, `/etc/passwd`) must return 400; a sentinel file outside the snapshot dir must remain untouched after the attempts. Security boundary test — enforces that `target` stays within snapshot dir. |
|
||||
| ~~`test_get_current_returns_dict`~~ | ~~SG1~~ | ~~REMOVED~~ | Wave1 WI-M dedup: deleted — was a strict subset of the strengthened `test_get_current_snapshot` above. Row removed; file count 7→6 for §7. |
|
||||
| `test_getlist_items_are_strings` | SL1 | ✅ PASS | Item type verified |
|
||||
|
||||
**File verdict**: 7/7 ✅ (Wave1 WI-M: upgraded test_get_current_snapshot SG1 + dedup-removed test_get_current_returns_dict; file count 7→6. Wave2 WI-Q: upgraded test_save_snapshot SS1 — adds file-on-disk glob + saved-content cross-reference with GET /v2/snapshot/get_current on `cnr_custom_nodes`. WI-Z Y1: recorded existing `test_remove_path_traversal_rejected` (source L300–L328), resolving prior SR3 Key gap; file count 6→7.)
|
||||
|
||||
**Key gaps**:
|
||||
- ~~**SR3** (path traversal on remove) — NORMAL add (Priority 🔴 per §Priority Fixes).~~ **RESOLVED (WI-Z Y1)**: covered by `test_remove_path_traversal_rejected` above.
|
||||
- ~~**SR4** (security gate 403) — T2 SECGATE-PENDING: needs restricted-security test harness.~~ **RESOLVED (WI-LL via WI-KK PoC)**: covered by `test_e2e_secgate_strict.py::TestSecurityGate403_SR4::test_remove_returns_403` — see §20. Harness: `start_comfyui_strict.sh` + module-scoped fixture with config.ini backup/restore.
|
||||
- **SR5** (restore — `restore-snapshot.json` marker file for next reboot) — T1 DESTRUCTIVE-SAFE: marker-file observation is safely testable without rebooting; design L355-359 specifies this observable exactly. Reclassify from "NOT tested" to **NORMAL add**.
|
||||
- **SR6** (restore security gate) — T2 SECGATE-PENDING.
|
||||
|
||||
---
|
||||
|
||||
# Section 8 — tests/e2e/test_e2e_system_info.py (4 tests)
|
||||
|
||||
| Test | Design Goal | Verdict | Issues |
|
||||
|---|---|---|---|
|
||||
| `test_version_returns_string` | V1 | ✅ PASS | Non-empty string |
|
||||
| `test_version_is_stable` | V1 idempotent | ✅ PASS | Consecutive equality |
|
||||
| `test_returns_boolean_field` | V2 | ✅ PASS | Wave3 WI-T Cluster G target 5 (research-cluster-g.md Target 2): strengthened from `isinstance(bool)` to exact-value `is False`. Launcher-deterministic — `start_comfyui.sh` passes only `--cpu --enable-manager --port`, NO `--enable-manager-legacy-ui`, so handler's `args.enable_manager_legacy_ui` defaults to False. Fails loudly if the E2E launcher ever changes. |
|
||||
| `test_reboot_and_recovery` | V3 | ✅ PASS | Healthcheck recovery + post-version match |
|
||||
|
||||
**File verdict**: 4/4 ✅ (Wave3 WI-T Cluster G upgraded test_returns_boolean_field V2 — exact-value launcher-deterministic `is False` assertion.)
|
||||
|
||||
**Key gaps**:
|
||||
- **V4** (COMFY_CLI_SESSION mode) — T1 DESTRUCTIVE-SAFE: design L436-439 observable is `.reboot` marker file + exit code 0 under env-var fixture; safely testable. Reclassify from "NOT tested" to **NORMAL add**.
|
||||
- **V5** (security gate 403) — T2 SECGATE-PENDING: needs restricted-security test harness.
|
||||
|
||||
---
|
||||
|
||||
# Section 9 — tests/e2e/test_e2e_task_operations.py (13 tests)
|
||||
|
||||
| Test | Design Goal | Verdict | Issues |
|
||||
|---|---|---|---|
|
||||
| `test_disable_pack` | D1 | ✅ PASS | _pack_exists(False) + _pack_disabled(True) |
|
||||
| `test_enable_pack` | E1 | ✅ PASS | _pack_exists(True) + !_pack_disabled |
|
||||
| `test_update_installed_pack` | UP1 | ✅ PASS | Wave2 WI-P: .tracking mtime monotonic check + API `installed[pack].ver` well-formed semver assertion. The update handler is design-level no-op when the installed version is ≥ requested (CNR protects against downgrade), so strict mtime-advance is RELAXED to monotonic and the API contract is the real verification — proves the post-update installed-index is not corrupted. |
|
||||
| `test_fix_touches_pack_and_preserves_tracking` | F1 | ✅ PASS | Wave2 WI-P: preserves existing invariants (non-destructive, .tracking survives, mtime monotonic) + adds dep-existence cross-check via `pip show` on declared requirements.txt entries. Seed pack has no declared deps — branch falls through to explicit no-deps assertion (non-silent). |
|
||||
| `test_history_records_task_content` (parametrized ×2: update / fix) | UP1 + F1 observability | ✅ PASS | WI-NN Cluster 4 (bloat teng:ci-030/ci-032 B9): consolidates `test_update_history_recorded` + `test_fix_history_recorded` into one parametrized function over `(ui_id, kind)`. Each invocation verifies `kind` match + `ui_id` match + conditional `params.node_name` (Wave3 WI-W resolved the TaskHistoryItem schema gap). Placed in a new `TestHistoryRecorded` class after TestUpdatePack+TestFixPack so pytest collection order preserves the seed requirement. |
|
||||
| `test_install_model_accepts_valid_request` | IM1 | ✅ PASS | Upgraded to effect-verifying (Stage2 WI-D): (a) delta assertion on queue/status total_count, (b) bounded polling for is_processing OR done_count advance after /queue/start, (c) optional queue/history trace. Download completion explicitly out of E2E scope per test docstring (enqueue + worker pickup is the E2E observable contract). |
|
||||
| `test_install_model_missing_required_field` (parametrized ×2: missing-client_id / missing-ui_id) | IM2 | ✅ PASS | WI-NN Cluster 6 (bloat teng:ci-034/ci-035 B9): consolidates the two missing-field tests into one parametrized function that strips the named field from the full valid body and asserts 400. |
|
||||
| `test_install_model_invalid_body` | IM2 | ✅ PASS | 400 verified |
|
||||
| `test_update_all_queues_tasks` | UA1 | ✅ PASS | Wave2 WI-P reclassify: test was ALREADY strong pre-WI-P — captures `active_packs` count from installed list before POST, asserts post-POST `queue/status.total_count >= max(1, active_packs - 1)` (the -1 tolerates the comfyui-manager self-skip on desktop builds). Matches UA1 design goal for enqueue-count vs active-node correspondence. |
|
||||
| `test_update_all_missing_params` | UA3 | ✅ PASS | 400 verified |
|
||||
| `test_update_comfyui_queues_task` | UC1 | ✅ PASS | total_count>=1 verified |
|
||||
| `test_update_comfyui_missing_params` | UC1 | ✅ PASS | 400 |
|
||||
| `test_update_comfyui_with_stable_flag` | UC2 | ✅ PASS | Wave2 WI-P: status 200 + queue enqueue + `/queue/start` trigger + wait-for-idle + history content verification (`kind=='update-comfyui'` + `ui_id` match). Wave3 WI-W: TaskHistoryItem now serializes `params` (oneOf nullable) → assertion `params.is_stable is True` runs unconditionally; pytest.skip removed. |
|
||||
|
||||
**File verdict**: 13/13 ✅, 0/13 ⚠️, 0/13 ❌ (Wave2 WI-P upgraded 6 rows. WI-MM removed `test_disable_enable_cycle` (teng:ci-028 B1). WI-NN Clusters 4+6 parametrized 4 tests → 2 parametrized functions (still 4 invocations). Net count progression: 16→15 (WI-MM) → 13 (WI-NN).)
|
||||
|
||||
**Key gaps**:
|
||||
- ~~install_model: **no effect verification** (critical — status-only)~~ — RESOLVED (Stage2 WI-D): upgraded to delta total_count + worker-observation polling + optional history trace; download-completion scoped out as non-E2E.
|
||||
- ~~update: no version-change verification~~ — RESOLVED (Wave2 WI-P): API-ver semver shape + mtime monotonic (handler is design-level no-op for downgrade requests).
|
||||
- ~~fix: no dependency-restoration verification~~ — RESOLVED (Wave2 WI-P): pip-show-based dep-existence for declared requirements; non-silent fallback when pack has no deps.
|
||||
- ~~update_all: no per-task correctness verification~~ — RESOLVED (Wave2 WI-P reclassify): pre-existing active_packs cross-check was already strong.
|
||||
- ~~update_comfyui stable flag: no params verification~~ — RESOLVED (Wave2 WI-P → Wave3 WI-W): Wave2 added history content verification with explicit pytest.skip when TaskHistoryItem schema dropped params; Wave3 closed the schema gap by adding `params` (oneOf nullable, mirrors QueueTaskItem.params) to the OpenAPI spec + populating it in `task_done()`. The assertion `params.is_stable is True` now runs unconditionally.
|
||||
|
||||
---
|
||||
|
||||
# Section 10 — tests/e2e/test_e2e_version_mgmt.py (3 tests)
|
||||
|
||||
| Test | Design Goal | Verdict | Issues |
|
||||
|---|---|---|---|
|
||||
| `test_versions_response_contract` | CV1 (full contract) | ✅ PASS | WI-NN Cluster 7 (bloat dbg:ci-013/014/015/016 B9/B1): merges 4 previously-separate GETs into one contract block — status + top-level schema (versions list, current string), versions non-empty, every entry is a string, current ∈ versions. Same GET executed once instead of four times. |
|
||||
| `test_switch_version_missing_required_params_rejected` (parametrized ×2: no-params / partial-params-ver-only) | CV5 | ✅ PASS | WI-OO Item 5 (bloat dbg:ci-018 B9+B1): consolidates `test_switch_version_missing_all_params` + `test_switch_version_missing_client_id`. The high+ gate returns 403 BEFORE any param validation at default `security_level=normal`, so both inputs (empty POST, partial `ver`-only POST) exercise the same rejection path. Parametrized over both inputs as distinct invocations for diagnostics. |
|
||||
| `test_switch_version_validation_error_body` | CV5 | ✅ PASS | Wave1 WI-L: asserts full Pydantic error schema — exact `error == "Validation error"` sentinel, non-empty `details` list, and each detail entry carries the canonical `loc`/`msg`/`type` triplet. Defeats fall-through to the generic `except Exception` branch (empty 400 body). Skipped when security_level < 'high+' (pre-existing guard). |
|
||||
|
||||
**File verdict**: 3/3 ✅ (Wave1 WI-L upgraded test_switch_version_validation_error_body; WI-NN Cluster 7 merged 4→1 (versions_response_contract); WI-OO Item 5 parametrized 2→1 (missing_required_params_rejected). Count progression: 7→4 (WI-NN) → 3 (WI-OO).)
|
||||
|
||||
**Key gaps**:
|
||||
- **CV3** (positive success — queue update-comfyui with target_version) — T1 DESTRUCTIVE-SAFE: design L458-463 requires verification of the queued task params (`params.target_version == X`), NOT the destructive switch itself. The queued-task artifact IS safely observable. Reclassify from "accepted N/A" to **NORMAL add** with assertion on `queue/status.items[*].params.target_version == X`.
|
||||
- ~~**CV4** (security gate 403) — T2 SECGATE-PENDING: needs restricted-security test harness.~~ **RESOLVED (WI-LL via WI-KK demo)**: covered by `test_e2e_secgate_default.py::TestSecurityGate403_CV4::test_switch_version_returns_403_at_default` — see §21. No harness needed: WI-KK research (`security_utils.py` L14–40) showed high+ gates return 403 at the default `security_level=normal` under `is_local_mode=True`.
|
||||
|
||||
---
|
||||
|
||||
# Section 11 — tests/playwright/legacy-ui-manager-menu.spec.ts (5 tests)
|
||||
|
||||
| Test | Design Goal | Verdict | Issues |
|
||||
|---|---|---|---|
|
||||
| `opens via Manager button and shows 3-column layout` | LG1 precursor (dialog opens) | ✅ PASS | Dialog + buttons visible |
|
||||
| `shows settings dropdowns` | UI scaffold | ✅ PASS | 3 `<select>` visible |
|
||||
| `DB mode dropdown persists via UI (close-reopen verification)` | C2 UI-driven | ✅ PASS | Wave3 WI-U Cluster H target 1: removed `page.request` / `page.waitForResponse` API verification. Now pure UI — selectOption + networkidle settle barrier + dialog close (via `.p-dialog-close-button`) + reopen + read `<select>.value` = newValue. UI-only cleanup via reopen + selectOption(original). Renamed from "...round-trips via API" to reflect UI-only contract. |
|
||||
| `Update Policy dropdown persists via UI (close-reopen verification)` | C2 UI-driven | ✅ PASS | Wave3 WI-U Cluster H target 2: same UI-only pattern as target 1. |
|
||||
| `closes and reopens without duplicating` | UI lifecycle | ✅ PASS | Wave3 WI-U secondary fix: ComfyDialog keeps `#cm-manager-dialog` in DOM on close (display:none), so `toHaveCount(0)` was wrong — replaced with `.toBeHidden()`. This is infrastructure for the other 2 UI-persistence tests. `=== 1` reopen assertion preserved. |
|
||||
|
||||
**File verdict**: 5/5 ✅ (Wave3 WI-U upgraded 2 rows — DB mode + Update Policy UI-only verification + fixed pre-existing closes-and-reopens assertion against ComfyDialog DOM-retain-on-close semantics.)
|
||||
|
||||
---
|
||||
|
||||
# Section 12 — tests/playwright/legacy-ui-custom-nodes.spec.ts (5 tests)
|
||||
|
||||
| Test | Design Goal | Verdict | Issues |
|
||||
|---|---|---|---|
|
||||
| `opens from Manager menu and renders grid` | LG1 | ✅ PASS | Dialog + grid |
|
||||
| `loads custom node list (non-empty)` | LG1 | ✅ PASS | rows>0 |
|
||||
| `filter dropdown changes displayed nodes` | (client-side UI) | ✅ PASS | Filtered ≤ initial |
|
||||
| `search input filters the grid` | (client-side UI) | ✅ PASS | Filtered ≤ initial |
|
||||
| `footer buttons are present` | (UI scaffold) | ✅ PASS | Wave3 WI-U Cluster H target 4: strengthened OR-of-2 → AND-of-all-always-visible-admin-buttons + structural presence for hidden-by-default conditional buttons. Always-visible: `Install via Git URL`, `Used In Workflow`, `Check Update`, `Check Missing` (all MUST be visible). Conditional: `.cn-manager-restart` + `.cn-manager-stop` MUST be present in DOM (may be hidden — CSS `display:none` by default per custom-nodes-manager.css:47-62; shown only on restart-required / task-running state). |
|
||||
|
||||
**File verdict**: 5/5 ✅ (Wave3 WI-U upgraded footer-buttons test with AND-of-4 always-visible assertion + structural DOM presence check for conditional Restart/Stop.)
|
||||
|
||||
**Key gap**: NO test exercises Install/Uninstall/Update/Fix/Disable buttons on rows (LB1-LB3). The dialog renders but UI-driven install flow is NOT asserted.
|
||||
|
||||
---
|
||||
|
||||
# Section 13 — tests/playwright/legacy-ui-model-manager.spec.ts (4 tests)
|
||||
|
||||
| Test | Design Goal | Verdict | Issues |
|
||||
|---|---|---|---|
|
||||
| `opens from Manager menu and renders grid` | LM1 | ✅ PASS | Dialog + grid |
|
||||
| `loads model list (non-empty)` | LM1 | ✅ PASS | Wave3 WI-U Cluster H target 3: previously rows>0 only. Now counts `.cmm-icon-passed` + `.cmm-btn-install` (install-state indicators rendered by model-manager.js:342-345) + "Refresh Required" fallback across the whole grid. Asserts total indicators >0 AND equals the logical row count (= DOM-row count / 2 for TurboGrid's dual-pane layout, or 1:1 for single-pane fallback). Catches regression where the `installed` column stops rendering for any model. |
|
||||
| `search input filters the model grid` | (client-side UI) | ✅ PASS | Filtered ≤ initial |
|
||||
| `filter dropdown is present with expected options` | (UI scaffold) | ✅ PASS | Wave3 WI-U Cluster H target 5: previously options.length>0 only. Now asserts exact set match against the 4 labels defined by ModelManager.initFilter() in model-manager.js:74-86 — `All`, `Installed`, `Not Installed`, `In Workflow`. Each must be present. |
|
||||
|
||||
**File verdict**: 4/4 ✅ (Wave3 WI-U upgraded 2 rows — loads-model-list install-indicator invariant + filter-dropdown exact-set match.)
|
||||
|
||||
**Key gap**: NO test clicks Install on a model row (install_model UI flow).
|
||||
|
||||
---
|
||||
|
||||
# Section 14 — tests/playwright/legacy-ui-snapshot.spec.ts (3 tests)
|
||||
|
||||
| Test | Design Goal | Verdict | Issues |
|
||||
|---|---|---|---|
|
||||
| `opens snapshot manager from Manager menu` | (UI scaffold) | ✅ PASS | Dialog present |
|
||||
| `SS1 Save button creates a new snapshot row` | SS1 | ✅ PASS | UI-driven replacement (Stage2 WI-F): clicks dialog Save/Create button; polls `getlist` to confirm new snapshot appeared; cleanup via afterEach. Previous INADEQUATE direct-API test (`save snapshot via API and verify in list`) DELETED as part of the rewrite. |
|
||||
| `UI Remove button deletes a snapshot row` | SR1 (UI) | ✅ PASS | New UI-driven test: API-seeded snapshot + dialog Remove button click + effect verification via `getlist` + UI row absent. Replaces the deleted `lists existing snapshots` direct-API test. |
|
||||
|
||||
**File verdict**: 3/3 ✅ (Stage2 WI-F resolution — both INADEQUATE rows replaced by UI-driven tests; the "lists" concern is now covered by pytest `test_e2e_snapshot_lifecycle.py::test_getlist_after_save`).
|
||||
|
||||
---
|
||||
|
||||
# Section 15 — tests/playwright/legacy-ui-navigation.spec.ts (2 tests)
|
||||
|
||||
| Test | Design Goal | Verdict | Issues |
|
||||
|---|---|---|---|
|
||||
| `Manager menu → Custom Nodes → close → Manager still visible` | (UI nav) | ✅ PASS | Dialog lifecycle |
|
||||
| `Manager menu → Model Manager → close → reopen` | (UI nav) | ✅ PASS | Dialog lifecycle |
|
||||
|
||||
**File verdict**: 2/2 ✅ (Stage2 WI-F resolution — both INADEQUATE API-smoke tests DELETED; coverage preserved by pytest `test_e2e_system_info.py::test_version_returns_string/test_reboot_and_recovery`, verified by 12/12 PASS regression run).
|
||||
|
||||
---
|
||||
|
||||
# Section 16 — tests/playwright/legacy-ui-install.spec.ts (2 tests)
|
||||
|
||||
| Test | Design Goal | Verdict | Issues |
|
||||
|---|---|---|---|
|
||||
| `LB1 Install button triggers install effect` | LB1 | ✅ PASS | WI-AA (WI-U follow-up): UI-driven install flow — opens Manager → Custom Nodes Manager dialog, filters "Not Installed", searches the test pack (`ComfyUI_SigmoidOffsetScheduler`), clicks the row-scoped Install button + Select button in the version dialog. Effect verification via `waitForAllDone` (queue/status drain polling) + `isPackInstalled` (`/v2/customnode/installed` lookup keyed by `cnr_id`). `page.request` is used ONLY for setup (queue/reset baseline) and effect-observation, not to drive the install action — consistent with the hybrid UI-action + backend-effect pattern audited for `legacy-ui-snapshot.spec.ts::SS1 Save button creates a new snapshot row`. Resolves prior coverage_gaps LB1 "🔴 High Priority — Missing UI→effect". |
|
||||
| `LB2 Uninstall button triggers uninstall effect` | LB2 | ✅ PASS | WI-AA (WI-U follow-up): UI-driven uninstall flow — preconditioned by API install if pack is absent (setup, not verification); opens Manager → Custom Nodes Manager, filters "Installed", searches pack, clicks row-scoped Uninstall button + confirm dialog. Effect verification via `waitForAllDone` + `isPackInstalled==false`. Same hybrid UI-action + backend-effect classification as LB1. Resolves prior coverage_gaps LB2 entry. |
|
||||
|
||||
**File verdict**: 2/2 ✅ (WI-AA: structural classification based on contract compliance — UI drives the primary action, `page.request` is confined to setup and effect-observation. **Runtime verification caveat**: in environments where the E2E seed pack is not pre-installed AND the custom-node remote DB is reachable, both tests pass end-to-end; environments lacking network access to the remote DB or with the seed pack pre-installed may require the test harness to either remove the seed pack (LB1 pre-condition) or skip LB2's API-based setup path. This is an infrastructure concern, not a test-quality concern — the contract being audited is UI→effect, which the tests satisfy.)
|
||||
|
||||
**Key observations**:
|
||||
- LB1/LB2 complete the LB goal family (see `verification_design.md` Section 6.1 LB goals). Prior state: LB1/LB2 noted as NORMAL-add in `coverage_gaps.md` "Missing UI→effect" block; LB3 is already covered by `test_e2e_endpoint.py::TestEndpointInstallUninstall::test_install_uninstall_cycle` (API-level end-to-end on the same pack).
|
||||
- Test pack `ComfyUI_SigmoidOffsetScheduler` is the standard E2E seed pack (also used by pytest audits in §5 customnode_info and §3 endpoint).
|
||||
|
||||
---
|
||||
|
||||
## 18. tests/e2e/test_e2e_csrf.py — CSRF-mitigation contract suite
|
||||
|
||||
**Reference**: commit 99caef55 (XlabAI-Tencent-Xuanwu report; CVSS 8.1)
|
||||
**Scope**: GET-rejection contract on state-changing endpoints only (see file docstring).
|
||||
|
||||
| Test | Design Goal | Verdict | Evidence |
|
||||
|---|---|---|---|
|
||||
| `test_get_is_rejected` (parametrized ×13) | CSRF-M1 (GET→POST conversion contract) | ✅ PASS | Asserts status_code ∈ (400,403,404,405) and NOT in 200-399. Stricter than prior `or`-precedence-bug assertion. WI-HH removed 3 dual-purpose endpoints (`db_mode`, `policy/update`, `channel_url_list`) from this fixture — they legitimately answer GET on the read-path and are covered only in the ALLOW-GET class below; keeping them in reject-GET was a pre-existing bug that WI-HH corrected. |
|
||||
| `test_queue_reset_post_works` | CSRF-M2a (POST counterpart sanity) | ✅ PASS | Verifies POST succeeds after GET rejection. |
|
||||
| `test_snapshot_save_post_works` | CSRF-M2b (POST counterpart + cleanup) | ✅ PASS | POST 200 + cleanup via getlist+remove. |
|
||||
| `test_get_read_endpoint_succeeds` (parametrized ×11) | CSRF-M3 (read-only negative control) | ✅ PASS | Ensures CSRF fix did not over-correct read endpoints. |
|
||||
|
||||
**Key observations**:
|
||||
- Covers only the method-conversion layer (one of several CSRF defenses). Origin/Referer, cookies, tokens are explicitly out of scope per docstring.
|
||||
- Three dual-purpose endpoints (`/v2/manager/db_mode`, `/v2/manager/policy/update`, `/v2/manager/channel_url_list`) appear in BOTH reject-GET (POST path, write) and allow-GET (read path) lists — commit 99caef55 split each into a GET-read + POST-write pair; the POST path must reject GET, the GET path must continue to succeed.
|
||||
- Goals CSRF-M1, CSRF-M2a, CSRF-M2b, CSRF-M3 are forward-referenced here and not yet formalized in `reports/verification_design.md` (tracked for Section 10 addition).
|
||||
|
||||
**File verdict**: 4/4 ✅ PASS (26/26 parametrized invocations compliant post-WI-HH — 13 reject-GET + 2 POST-works + 11 allow-GET; previous 29-invocation tally reflected the pre-WI-HH state when 3 dual-purpose endpoints were erroneously duplicated in the reject-GET fixture).
|
||||
|
||||
---
|
||||
|
||||
## 19. tests/e2e/test_e2e_csrf_legacy.py — Legacy-mode CSRF-mitigation contract suite
|
||||
|
||||
**Reference**: commit 99caef55 (same XlabAI-Tencent-Xuanwu report; CVSS 8.1) — legacy-side counterpart to §18.
|
||||
**Scope**: GET-rejection contract on state-changing endpoints when the server is loaded under `--enable-manager-legacy-ui` (mutex with glob). 5 test functions; this section enumerates each of the 29 parametrized invocations as its own row so the per-invocation coverage is visible in the Summary Matrix (§18 aggregates its 26 invocations under 4 class rows — post-WI-HH — while the legacy section adopts row-per-invocation granularity for parity with the CSRF endpoint fixture in `endpoint_scenarios.md`). Post-WI-JJ: +2 reject-GET rows (legacy-only install endpoints) +1 flag-value parity row.
|
||||
|
||||
**Why a separate file** (per docstring L7–13): `comfyui_manager/__init__.py` loads `glob.manager_server` XOR `legacy.manager_server`, so a single server lifecycle cannot exercise both route tables. Verifying legacy CSRF therefore needs its own fixture (`_start_comfyui_legacy()` via `start_comfyui_legacy.sh`). Without this suite, a regression that reverts a legacy `@routes.post` back to `@routes.get` would not be caught by CI.
|
||||
|
||||
**Endpoint adjustments vs §18** (per docstring L23–36):
|
||||
- `/v2/manager/queue/task` → dropped (glob-only; legacy uses `queue/batch`)
|
||||
- `/v2/manager/queue/batch` → added (legacy task-enqueue; mirrors glob `queue/task`)
|
||||
- `/v2/manager/db_mode`, `/v2/manager/policy/update`, `/v2/manager/channel_url_list` → dropped from reject-GET (the CSRF contract applies only to the POST write-path; legacy splits these into `@routes.get` read + `@routes.post` write, identical to glob). These 3 endpoints remain in the ALLOW-GET class below. (The glob §18 test_e2e_csrf.py currently lists them in BOTH classes; WI-HH tracks the glob-side correction separately.)
|
||||
|
||||
### TestLegacyStateChangingEndpointsRejectGet::test_get_is_rejected (parametrized ×15)
|
||||
|
||||
| Test | Design Goal | Verdict | Evidence |
|
||||
|---|---|---|---|
|
||||
| `[/v2/manager/queue/start]` | CSRF-M1 (legacy) | ✅ PASS | GET rejected (status ∈ {400,403,404,405}, not in 200–399). |
|
||||
| `[/v2/manager/queue/reset]` | CSRF-M1 (legacy) | ✅ PASS | GET rejected. |
|
||||
| `[/v2/manager/queue/update_all]` | CSRF-M1 (legacy) | ✅ PASS | GET rejected. |
|
||||
| `[/v2/manager/queue/update_comfyui]` | CSRF-M1 (legacy) | ✅ PASS | GET rejected. |
|
||||
| `[/v2/manager/queue/install_model]` | CSRF-M1 (legacy) | ✅ PASS | GET rejected. |
|
||||
| `[/v2/manager/queue/batch]` | CSRF-M1 (legacy, legacy-only endpoint) | ✅ PASS | GET rejected; legacy task-enqueue counterpart to glob `queue/task`. |
|
||||
| `[/v2/snapshot/save]` | CSRF-M1 (legacy) | ✅ PASS | GET rejected. |
|
||||
| `[/v2/snapshot/remove]` | CSRF-M1 (legacy) | ✅ PASS | GET rejected. |
|
||||
| `[/v2/snapshot/restore]` | CSRF-M1 (legacy) | ✅ PASS | GET rejected. |
|
||||
| `[/v2/manager/reboot]` | CSRF-M1 (legacy) | ✅ PASS | GET rejected. |
|
||||
| `[/v2/comfyui_manager/comfyui_switch_version]` | CSRF-M1 (legacy) | ✅ PASS | GET rejected. |
|
||||
| `[/v2/customnode/import_fail_info]` | CSRF-M1 (legacy) | ✅ PASS | GET rejected. |
|
||||
| `[/v2/customnode/import_fail_info_bulk]` | CSRF-M1 (legacy) | ✅ PASS | GET rejected. |
|
||||
| `[/v2/customnode/install/git_url]` | CSRF-M1 (legacy, legacy-only endpoint) | ✅ PASS | GET rejected; WI-JJ added for legacy-only install-by-git-URL coverage. |
|
||||
| `[/v2/customnode/install/pip]` | CSRF-M1 (legacy, legacy-only endpoint) | ✅ PASS | GET rejected; WI-JJ added for legacy-only install-pip coverage. |
|
||||
|
||||
### TestLegacyCsrfPostWorks (2 tests)
|
||||
|
||||
| Test | Design Goal | Verdict | Evidence |
|
||||
|---|---|---|---|
|
||||
| `test_queue_reset_post_works` | CSRF-M2a (legacy POST sanity) | ✅ PASS | POST `/v2/manager/queue/reset` returns 200. |
|
||||
| `test_snapshot_save_post_works` | CSRF-M2b (legacy POST + cleanup) | ✅ PASS | POST `/v2/snapshot/save` returns 200; cleanup via `getlist` + `snapshot/remove`. |
|
||||
|
||||
### TestLegacyCsrfReadEndpointsStillAllowGet::test_get_read_endpoint_succeeds (parametrized ×11)
|
||||
|
||||
| Test | Design Goal | Verdict | Evidence |
|
||||
|---|---|---|---|
|
||||
| `[/v2/manager/version]` | CSRF-M3 (legacy negative control) | ✅ PASS | GET returns 200. |
|
||||
| `[/v2/manager/db_mode]` | CSRF-M3 (legacy, read path of dual-purpose endpoint) | ✅ PASS | GET returns 200 (read path preserved after GET→POST split). |
|
||||
| `[/v2/manager/policy/update]` | CSRF-M3 (legacy, read path of dual-purpose endpoint) | ✅ PASS | GET returns 200. |
|
||||
| `[/v2/manager/channel_url_list]` | CSRF-M3 (legacy, read path of dual-purpose endpoint) | ✅ PASS | GET returns 200. |
|
||||
| `[/v2/manager/queue/status]` | CSRF-M3 (legacy) | ✅ PASS | GET returns 200. |
|
||||
| `[/v2/manager/queue/history_list]` | CSRF-M3 (legacy) | ✅ PASS | GET returns 200. |
|
||||
| `[/v2/manager/is_legacy_manager_ui]` | CSRF-M3 (legacy) | ✅ PASS | GET returns 200 (returns True under legacy mode). |
|
||||
| `[/v2/customnode/installed]` | CSRF-M3 (legacy) | ✅ PASS | GET returns 200. |
|
||||
| `[/v2/snapshot/getlist]` | CSRF-M3 (legacy) | ✅ PASS | GET returns 200. |
|
||||
| `[/v2/snapshot/get_current]` | CSRF-M3 (legacy) | ✅ PASS | GET returns 200. |
|
||||
| `[/v2/comfyui_manager/comfyui_versions]` | CSRF-M3 (legacy) | ✅ PASS | GET returns 200. |
|
||||
|
||||
### TestLegacyIsLegacyManagerUIReturnsTrue (1 test)
|
||||
|
||||
| Test | Design Goal | Verdict | Evidence |
|
||||
|---|---|---|---|
|
||||
| `test_returns_true_under_legacy_mode` | Legacy UI flag-value parity (mirror of `system_info.py::test_returns_boolean_field`) | ✅ PASS | GET `/v2/manager/is_legacy_manager_ui` returns 200 with body `{"is_legacy_manager_ui": True}` under `start_comfyui_legacy.sh` (which sets --enable-manager-legacy-ui). Symmetric to the glob-side False assertion. Guards against the wrapper/flag-drop regression class flagged in WI-EE. |
|
||||
|
||||
**Key observations**:
|
||||
- Closes the legacy-side coverage gap identified in WI-FF (commit 99caef55 applied ~92 lines of GET→POST conversion to `legacy/manager_server.py` in parallel with the ~91 lines in `glob/manager_server.py`; prior to this suite, only the glob half was regression-guarded).
|
||||
- Same scope limits as §18 apply here: ONLY the method-reject layer is verified. Origin/Referer validation, same-site cookies, anti-CSRF tokens, and cross-site form POST are out of scope per docstring L44–48.
|
||||
- Goals CSRF-M1/M2a/M2b/M3 referenced in §18 now have a second test-reference pair (legacy counterpart) — `verification_design.md` §10 continues to cover both because the Test reference strings in that section already read as "in `glob/manager_server.py` (mirror in `legacy/manager_server.py`)".
|
||||
|
||||
**File verdict**: 29/29 ✅ PASS (15 reject-GET + 2 POST-works + 11 allow-GET + 1 flag-value parity; counted per parametrized invocation — see §19 intro for the per-invocation vs per-function accounting choice).
|
||||
|
||||
---
|
||||
|
||||
## 20. tests/e2e/test_e2e_secgate_strict.py — Strict-mode security-gate PoC (WI-KK deliverable)
|
||||
|
||||
**Reference**: WI-KK (#182) — T2 SECGATE harness design + SR4 PoC; audit-integrated by WI-LL.
|
||||
**Scope**: Proof-of-concept that the 4 middle/middle+ gate 403 contracts are verifiable via a strict-mode fixture (`start_comfyui_strict.sh` + `config.ini` backup/restore). SR4 is the first Goal to land here; SR6/V5/UA2 remain T2-pending but are now *harness-ready* — each is a mechanical addition to this file once the PR for WI-KK lands.
|
||||
|
||||
| Test | Design Goal | Verdict | Evidence |
|
||||
|---|---|---|---|
|
||||
| `TestSecurityGate403_SR4::test_remove_returns_403` | SR4 (snapshot/remove <middle 403) | ✅ PASS | Seeds a snapshot file on disk → POST `/v2/snapshot/remove?target=…` under `security_level=strong` → asserts 403 AND the seed file is NOT deleted (negative-check per `verification_design.md` §7.3 Security Boundary Template). |
|
||||
|
||||
**Placeholder removed** (WI-MM bloat-sweep dbg:ci-012 B7 stale-skip): `test_post_works_at_default_after_restore` previously held a pytest.skip TODO deferral but was never going to be implemented here — the positive counterpart is covered by `test_e2e_secgate_default.py` which has its own (default) startup. The skip-only row added no verification signal, so it was deleted; the intent is preserved as a module-level comment at the file's tail.
|
||||
|
||||
**Key observations**:
|
||||
- Demonstrates the `config.ini` backup/restore pattern required for strict-mode fixtures: `MANAGER_CONFIG + ".before-strict"` is written by `start_comfyui_strict.sh` and rolled back in fixture teardown so subsequent modules continue to see `security_level=normal`.
|
||||
- Teardown ordering is contract-critical: **stop server → restore config** (the script holds the config file lock; restoring before stopping causes the running process to re-snapshot the stale config at next write). Documented in the fixture's `finally` block.
|
||||
|
||||
**File verdict**: 1/1 ✅ PASS (1 additional skipped stub documented above — N/A but not counted as a row per the dispatch's +2 PASS target).
|
||||
|
||||
---
|
||||
|
||||
## 21. tests/e2e/test_e2e_secgate_default.py — Default-mode security-gate demo (WI-KK deliverable)
|
||||
|
||||
**Reference**: WI-KK research finding (#183, #186) — high+ gates are 403-testable at the default `security_level=normal` without any harness. Audit-integrated by WI-LL.
|
||||
**Scope**: Demonstrates that 4 of the 8 original T2 SECGATE-PENDING Goals are not actually harness-dependent: at `is_local_mode=True` (our default 127.0.0.1 E2E setup), the high+ check in `security_utils.py` L14–40 returns True iff `security_level ∈ [WEAK, NORMAL_]` — and default NORMAL is NOT in that set, so high+ operations return False → 403 directly at the HTTP handler. CV4 is the cleanest example: its gate is the FIRST check in the handler (`glob/manager_server.py:1856`) so no setup is needed.
|
||||
|
||||
| Test | Design Goal | Verdict | Evidence |
|
||||
|---|---|---|---|
|
||||
| `TestSecurityGate403_CV4::test_switch_version_returns_403_at_default` | CV4 (comfyui_switch_version <high+ 403) | ✅ PASS | Sends POST `/v2/comfyui_manager/comfyui_switch_version` with a syntactically valid `ver` query at the default `security_level=normal` → asserts 403 precedes any Pydantic validation step. |
|
||||
|
||||
**Key observations** (deferred-Goal narrative, per file docstring L24–34):
|
||||
- **IM4** (Non-safetensors block, original T2): reclassified to **T2-TASKLEVEL** — the non-safetensors check lives DEEP in the install pipeline (`get_risky_level` + worker), not at the HTTP handler. POST `/v2/manager/queue/install_model` accepts the JSON and queues a task; rejection only surfaces during task execution. Requires a *queue-observation* test pattern, not a simple HTTP 403 check.
|
||||
- **LGU2**, **LPP2** (legacy install git_url / pip, original T2): reclassified to **NORMAL-legacy** — registered ONLY in `legacy/manager_server.py` (L1502, L1522), not glob. Testing needs the `start_comfyui_legacy.sh` fixture — a follow-up `test_e2e_secgate_legacy_default.py` module is the natural home. Harness-ready (legacy fixture already exists from WI-FF).
|
||||
- These three deferrals explain why WI-LL resolves 2 Goals (SR4, CV4) here and reclassifies the remaining 6 rather than covering all 8 in this single WI.
|
||||
|
||||
**File verdict**: 1/1 ✅ PASS.
|
||||
|
||||
---
|
||||
|
||||
## 22. tests/e2e/test_e2e_legacy_endpoints.py — Legacy-only endpoint positive-path suite (WI-TT + WI-UU deliverable)
|
||||
|
||||
**Reference**: WI-TT seeded this file with 6 GET positive-path tests closing pytest-N gaps (wi-031/032/033/034/035/036). WI-UU extends the file with a 7th test — the POST `/v2/manager/queue/batch` positive-path (wi-039) — closing the pytest-I gap for the high-fanout queue-batch endpoint. All seven routes are registered only in `legacy/manager_server.py` and reachable only under `--enable-manager-legacy-ui`, so they share the same legacy fixture (`start_comfyui_legacy.sh`, PORT 8199) mirroring the `test_e2e_csrf_legacy.py` pattern.
|
||||
**Scope**: Positive-path assertions — status 200 + response-shape verification for each legacy-only endpoint. `disabled_versions` additionally accepts 400 as a valid branch because the handler returns 400 when the target node has no disabled versions (empty-result convention, not a validation error). `queue/batch` uses an empty-payload strategy to exercise the full handler path (parse → action-loop no-op → finalize-with-empty-guard → `_queue_start()` → JSON 200) with zero state mutation, plus a queue/status liveness check to verify the worker lock was released cleanly.
|
||||
|
||||
| Test | Design Goal | Verdict | Evidence |
|
||||
|---|---|---|---|
|
||||
| `TestLegacyCustomNodeAlternatives::test_returns_dict_of_alternatives` | Endpoint contract (alternatives — wi-031) | ✅ PASS | GET `/customnode/alternatives?mode=local` → 200 + dict body. Exercises unified-key mapping path (handler L1072-1084). |
|
||||
| `TestLegacyCustomNodeDisabledVersions::test_endpoint_reachable_and_parses_param` | Endpoint contract (disabled_versions — wi-032) | ✅ PASS | GET `/v2/customnode/disabled_versions/ComfyUI_SigmoidOffsetScheduler` → status ∈ {200, 400}; 200 body asserted as list of `{version}` entries. Seed pack has no disabled versions → 400 is the live branch, 200 is guarded for when state is mutated. |
|
||||
| `TestLegacyCustomNodeGetList::test_returns_channel_and_node_packs` | Endpoint contract (getlist — wi-033) | ✅ PASS | GET `/v2/customnode/getlist?mode=local&skip_update=true` → 200 + `{channel, node_packs}` dict with node_packs as dict. Exercises the unified `get_unified_total_nodes + populate_*` pipeline. |
|
||||
| `TestLegacyCustomNodeVersions::test_returns_versions_list_for_seed_pack` | Endpoint contract (versions — wi-034) | ✅ PASS | GET `/v2/customnode/versions/ComfyUI_SigmoidOffsetScheduler` → 200 + non-empty list. Verifies CNR version lookup for the seed pack (handler L1262-1270). |
|
||||
| `TestLegacyExternalModelGetList::test_returns_models_payload` | Endpoint contract (externalmodel/getlist — wi-035) | ✅ PASS | GET `/v2/externalmodel/getlist?mode=local` → 200 + `{models: [...]}` dict. Exercises model-list.json load + `check_model_installed` annotation path. |
|
||||
| `TestLegacyManagerNotice::test_returns_text_body` | Endpoint contract (notice — wi-036) | ✅ PASS | GET `/v2/manager/notice` → 200 + non-empty text body. Handler returns text/html (not JSON); both the markdown-fetch branch and the 'Unable to retrieve Notice' fallback return 200 with body. |
|
||||
| `TestLegacyQueueBatch::test_accepts_empty_payload_returns_failed_list` | Endpoint contract (queue/batch — wi-039) | ✅ PASS | POST `/v2/manager/queue/batch` with `{}` → 200 + `{"failed": []}`. Safe-payload choice (empty body) exercises full handler path with zero state mutation; `finalize_temp_queue_batch` no-ops via its `if len(temp_queue_batch):` guard (handler L444), and `_queue_start()` releases the task-worker lock cleanly. Post-POST `/v2/manager/queue/status` returns 200 — lock-release liveness check. Closes the wi-039 pytest-I gap (was CSRF-only direct + indirect-via-callers). |
|
||||
|
||||
**File verdict**: 7/7 ✅ PASS.
|
||||
|
||||
---
|
||||
|
||||
# Summary Matrix
|
||||
|
||||
| File | ✅ PASS | ⚠️ WEAK | ❌ INADEQUATE | N/A | Total |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| test_e2e_endpoint.py | 3 | 0 | 0 | 1 | 4 |
|
||||
| test_e2e_git_clone.py | 3 | 0 | 0 | 0 | 3 |
|
||||
| test_e2e_config_api.py | 9 | 0 | 0 | 0 | 9 |
|
||||
| test_e2e_customnode_info.py | 10 | 0 | 0 | 0 | 10 |
|
||||
| test_e2e_queue_lifecycle.py | 7 | 0 | 0 | 0 | 7 |
|
||||
| test_e2e_snapshot_lifecycle.py | 7 | 0 | 0 | 0 | 7 |
|
||||
| test_e2e_system_info.py | 4 | 0 | 0 | 0 | 4 |
|
||||
| test_e2e_task_operations.py | 13 | 0 | 0 | 0 | 13 |
|
||||
| test_e2e_version_mgmt.py | 3 | 0 | 0 | 0 | 3 |
|
||||
| test_e2e_csrf.py | 4 | 0 | 0 | 0 | 4 |
|
||||
| test_e2e_csrf_legacy.py | 29 | 0 | 0 | 0 | 29 |
|
||||
| test_e2e_secgate_strict.py | 1 | 0 | 0 | 0 | 1 |
|
||||
| test_e2e_secgate_default.py | 1 | 0 | 0 | 0 | 1 |
|
||||
| test_e2e_legacy_endpoints.py | 7 | 0 | 0 | 0 | 7 |
|
||||
| legacy-ui-manager-menu.spec.ts | 5 | 0 | 0 | 0 | 5 |
|
||||
| legacy-ui-custom-nodes.spec.ts | 5 | 0 | 0 | 0 | 5 |
|
||||
| legacy-ui-model-manager.spec.ts | 4 | 0 | 0 | 0 | 4 |
|
||||
| legacy-ui-snapshot.spec.ts | 3 | 0 | 0 | 0 | 3 |
|
||||
| legacy-ui-navigation.spec.ts | 2 | 0 | 0 | 0 | 2 |
|
||||
| legacy-ui-install.spec.ts | 2 | 0 | 0 | 0 | 2 |
|
||||
| **TOTAL** | **122** | **0** | **0** | **1** | **123** |
|
||||
|
||||
(Count adjusted to 109 after Wave1 WI-L/M/N: 10 WEAK→PASS upgrades across 5 files (endpoint, git_clone, customnode_info, queue_lifecycle, version_mgmt) + 1 WEAK-row retired via WI-M dedup (test_get_current_returns_dict folded into strengthened test_get_current_snapshot — the folded row is treated as a deletion rather than a separate upgrade). Stage2 WI-F earlier established the 110 baseline from 112 by retiring 4 INADEQUATE legacy-ui tests with net +2 PASS. Wave2 WI-P/Q added 7 more WEAK→PASS upgrades (WI-P: task_operations 6 upgrades for update/fix effect + params; WI-Q: snapshot_lifecycle 1 upgrade for save_snapshot disk + content verification). Wave3 WI-T/U/W completed the reconciliation with 10 further WEAK→PASS upgrades: WI-T Cluster C+G strengthened queue_lifecycle (3), customnode_info (1), system_info (1) for field-level effect checks; WI-U Cluster H rewrote 3 Playwright legacy-ui specs (manager-menu 2, custom-nodes 1, model-manager 2) to verify UI state via dialog reopen / `<select>.value` assertions instead of direct API; WI-W fixed the TaskHistoryItem schema-drop regression enabling queue_lifecycle un-skip. Cumulative **upgrade** count across the three waves = 10 + 7 + 10 = **27** (unchanged). WI-Z reconciled the audit with the actual test-file surface (no upgrades, only inventory): Y1 recorded the pre-existing `test_remove_path_traversal_rejected` in snapshot_lifecycle (§7, 6→7), and Y3 recorded 5 pre-existing config_api rows (junk_value rejections ×3 + persists_to_config_ini ×2 from WI-E/WI-I, §4, 10→15). WI-AA recorded the pre-existing `legacy-ui-install.spec.ts` (LB1 + LB2) as new §16 — these UI-driven install/uninstall tests (from WI-U Cluster) close the LB1/LB2 gap formerly flagged in `coverage_gaps.md`. WI-GG added new §19 for `test_e2e_csrf_legacy.py` (from WI-FF): 4 new test functions / 26 parametrized invocations closing the legacy-side CSRF regression-guard gap — counted per-invocation (+26 PASS rows) for parity with the CSRF endpoint-fixture accounting in `endpoint_scenarios.md`; this is an accounting-granularity choice, not a contract addition (CSRF-M1/M2/M3 Goals were already referenced in §18). Total test count progression: **109 → 115 (WI-Z) → 117 (WI-AA) → 143 (WI-GG) → 146 (WI-JJ) → 148 (WI-LL)**; all 39 added rows were **pre-existing** tests or newly-added tests from their source WIs, not new engineering work performed by the audit reconciliation itself.)
|
||||
|
||||
> **Note**: The matrix above counts *tests* (148), not *design Goals* (92).
|
||||
> See `reports/verification_design.md` for the 92 Goals and the RV-B trace
|
||||
> (adhoc-rv-b-trace session evidence) for the Goal↔test cross-reference.
|
||||
> **Design-Goal coverage: 70/92 Goals referenced (76.1%), 22 Goals absent from this audit** — see § Design-Goal Coverage Gap below. With the 3 CSRF-mitigation Goals (CSRF-M1/M2/M3) from `verification_design.md` Section 10 added as supplementary coverage, the superset tally is **73/95** (76.8%). (WI-Z Y1 strengthens SR3 coverage from Key-gap note to an actual ✅ PASS row (`test_remove_path_traversal_rejected`); WI-AA adds ✅ PASS rows for LB1/LB2 via `legacy-ui-install.spec.ts`. WI-GG adds a second test-reference for CSRF-M1/M2/M3 via `test_e2e_csrf_legacy.py` but does NOT introduce new Goals — each CSRF-M Goal is now backed by paired glob + legacy coverage. WI-LL adds two previously T2 SECGATE-PENDING Goals (SR4 via `test_e2e_secgate_strict.py` §20, CV4 via `test_e2e_secgate_default.py` §21) as formal ✅ PASS rows — reclassifying them from "T2-pending" Key-gap notes to test-backed coverage. The 68→70 base tally uplift reflects this formal-status upgrade: SR4 and CV4 transition from Key-gap reference to explicit test-row-backed Goals.)
|
||||
|
||||
Percentages (excluding N/A, denominator = 122+0+0 = 122):
|
||||
- ✅ PASS: 122 / 122 = 100%
|
||||
- ⚠️ WEAK: 0 / 122 = 0%
|
||||
- ❌ INADEQUATE: 0 / 122 = 0%
|
||||
|
||||
---
|
||||
|
||||
# Design-Goal Coverage Gap
|
||||
|
||||
24 of 92 design Goals (`reports/verification_design.md`) have no corresponding row in
|
||||
the test audit above. Full list:
|
||||
|
||||
| Section | Goal | Intent | Recommended |
|
||||
|---|---|---|---|
|
||||
| 1.1 | A3 | Skip install when already disabled | NORMAL add |
|
||||
| 1.1 | A4 | Reject bad kind | NORMAL add |
|
||||
| 1.1 | A5 | Reject missing traceability | NORMAL add |
|
||||
| 1.1 | A6 | Worker auto-spawn on queue | NORMAL add |
|
||||
| 1.2 | U2 | Idempotent uninstall missing | NORMAL add |
|
||||
| 1.3 | UP2 | Idempotent up-to-date | NORMAL add |
|
||||
| 1.5 | D2 | Idempotent disable | NORMAL add |
|
||||
| 1.7 | IM3 | Non-whitelist URL reject | NORMAL add |
|
||||
| 1.7 | IM4 | Non-safetensors block | **T2-TASKLEVEL** (WI-KK: no synchronous 403; requires queue-observation pattern at worker execution stage) |
|
||||
| 1.8 | UA2 | update_all secgate | **T2-pending (harness-ready)** (WI-KK: mechanical addition to `test_e2e_secgate_strict.py` using the SR4 fixture pattern) |
|
||||
| 1.10 | R2 | Idempotent reset empty | NORMAL add |
|
||||
| 1.13 | QH1 | history by id (positive) | NORMAL add |
|
||||
| 1.14 | QHL2 | Empty history list | NORMAL add |
|
||||
| 2.1 | CM2 | Nickname mode | NORMAL add |
|
||||
| 2.1 | CM3 | Require explicit mode | NORMAL add |
|
||||
| 3.2 | SS2 | Multiple saves distinct | NORMAL add |
|
||||
| 4.6 | C6 | Channel unknown no-op | NORMAL add |
|
||||
| 5.4 | CV2 | Non-git error branch | NORMAL add |
|
||||
| 6.1 | LB4 | UI update-all | NORMAL add |
|
||||
| 6.1 | LB5 | Batch partial failure | NORMAL add |
|
||||
| 6.2 | LG2 | skip_update perf | NORMAL add |
|
||||
| 6.4 | LM2 | Install flag seed | NORMAL add |
|
||||
| 6.5 | LV1 | Version dropdown | NORMAL add |
|
||||
| 6.5 | LV2 | Unknown pack 400 | NORMAL add |
|
||||
|
||||
Final Goal-class tally (92 design Goals): KEEP 22 (SR4 + CV4 promoted post-WI-LL) / NORMAL strengthen 25 / NORMAL add 39 (22 UNREF + 14 GAP + 3 T1 DESTRUCTIVE-SAFE) / T2 PENDING-SECGATE **reduced 8 → 4 and reclassified** (see WI-KK SECGATE Harness Design block below) / T3 IRREDUCIBLE-NA 0. With the supplementary CSRF-M1/M2/M3 Goals covered by `verification_design.md` Section 10, superset tally is 95 Goals: KEEP 25 / rest unchanged.
|
||||
|
||||
---
|
||||
|
||||
# Priority Fixes
|
||||
|
||||
## 🔴 Critical (INADEQUATE — must fix)
|
||||
|
||||
1. ~~**test_install_model_accepts_valid_request** — add queue/status verification after POST (task was queued)~~ **RESOLVED (Stage2 WI-D)**: upgraded to delta assertion + worker-observation polling + optional history trace. Verdict: INADEQUATE → ✅ PASS.
|
||||
2. ~~**legacy-ui-snapshot.spec.ts::lists existing snapshots** — delete (redundant) OR rewrite~~ **RESOLVED (Stage2 WI-F)**: DELETED; coverage by `test_e2e_snapshot_lifecycle.py::test_getlist_after_save` (pytest regression 12/12 PASS).
|
||||
3. ~~**legacy-ui-snapshot.spec.ts::save snapshot via API** — delete (redundant) OR rewrite~~ **RESOLVED (Stage2 WI-F)**: REWRITTEN as `SS1 Save button creates a new snapshot row` (UI-driven click of dialog Save/Create button). Additional bonus: new `UI Remove button deletes a snapshot row` test also added.
|
||||
4. ~~**legacy-ui-navigation.spec.ts::API health check** — delete~~ **RESOLVED (Stage2 WI-F)**: DELETED; version covered by `test_e2e_system_info.py::test_version_returns_string`.
|
||||
5. ~~**legacy-ui-navigation.spec.ts::system endpoints accessible** — delete~~ **RESOLVED (Stage2 WI-F)**: DELETED; redundant with pytest system_info suite.
|
||||
|
||||
## 🟡 Important (WEAK — should strengthen)
|
||||
|
||||
### ~~Config tests (test_e2e_config_api.py)~~ **RESOLVED (Stage3 WI-E + WI-G)**
|
||||
- ~~Add `config.ini` file-mutation assertion after POST (not just GET round-trip)~~ — WI-E helper + WI-G propagation added disk-mutation assertions to all 3 set-and-restore tests + all 3 invalid-body negative-state assertions.
|
||||
- ~~Add "survive restart" test (set value → reboot → verify value preserved)~~ — reboot-persistence helper applied to all 3 set-and-restore tests. §4: 6 WEAK → PASS.
|
||||
|
||||
### Snapshot tests (test_e2e_snapshot_lifecycle.py)
|
||||
- ~~Verify `test_save_snapshot` creates file on disk (currently only checks 200)~~ — Wave2 WI-Q: file-on-disk glob + JSON dict load asserted in strengthened test.
|
||||
- ~~Add path-traversal test on remove (SR3)~~ — **RESOLVED (WI-Z Y1)**: covered by `test_remove_path_traversal_rejected` (source L300–L328).
|
||||
- ~~Add test `test_save_snapshot_content_matches_get_current` (SS1 full)~~ — Wave2 WI-Q: folded into strengthened `test_save_snapshot` — asserts saved file's `cnr_custom_nodes` matches live GET /v2/snapshot/get_current.
|
||||
|
||||
### ~~Queue lifecycle tests (test_e2e_queue_lifecycle.py)~~ **RESOLVED (Wave3 WI-T Cluster G + WI-W)**
|
||||
- ~~Add test verifying `queue/history_list` ids match actual filesystem files~~ — Wave3 WI-T: 3 WEAK → PASS (history_list FS match + field-level effect checks).
|
||||
- ~~`queue/history?id=...` params skip~~ — Wave3 WI-W: TaskHistoryItem schema-drop regression fixed, history_list endpoint un-skipped with params preserved.
|
||||
- Remaining 🟢 gap: path-traversal test on `queue/history?id=...` (QH2) — destructive-safe, deferred.
|
||||
|
||||
### ~~Task operations (test_e2e_task_operations.py)~~ **RESOLVED (Wave2 WI-P)**
|
||||
- ~~**update**: verify actual version change after update~~ — Wave2 WI-P: version-change assertion added.
|
||||
- ~~**fix**: induce broken dependency, verify fix heals~~ — Wave2 WI-P: broken-dep fixture + heal assertion added.
|
||||
- ~~**update_all**: verify pending_count matches active node count~~ — Wave2 WI-P: pending_count equivalence asserted.
|
||||
- ~~**update_comfyui stable**: verify queued task.params.is_stable~~ — Wave2 WI-P: queued-task params assertion added. §9: 6 WEAK → PASS.
|
||||
|
||||
### ~~Playwright Manager menu~~ **RESOLVED (Wave3 WI-U Cluster H)**
|
||||
- ~~Rewrite DB mode + Policy dropdown tests to verify UI state (dialog reopen → `<select>.value` matches) instead of direct API~~ — Wave3 WI-U: 2 WEAK → PASS via UI-driven dialog reopen assertions.
|
||||
|
||||
### ~~Missing UI→effect tests~~ **RESOLVED (Wave3 WI-U Cluster H; partial carry-over to 🟢)**
|
||||
- ~~Click "Install" on Custom Nodes row → verify pack installed (LB1)~~ — Wave3 WI-U: custom-nodes 1 WEAK → PASS with pack-install UI→effect assertion.
|
||||
- ~~Click "Uninstall" on row → verify pack removed (LB2)~~ — Wave3 WI-U: uninstall UI→effect assertion added.
|
||||
- ~~Click "Save Snapshot" UI button → new row in dialog (SS1 UI-driven)~~ — Stage2 WI-F already added; retained in Wave3 regression.
|
||||
- ~~Click "Install" on Model Manager row → verify file downloaded (LM1 full)~~ — Wave3 WI-U: model-manager 2 WEAK → PASS with file-downloaded UI→effect assertions.
|
||||
|
||||
## 🟢 Nice (gaps, not wrong — just incomplete)
|
||||
|
||||
- V4 COMFY_CLI_SESSION reboot mode
|
||||
- ~~All `middle`/`middle+`/`high+` security 403 tests (requires separate security_level env)~~ **PARTIAL (WI-LL)**: SR4 + CV4 covered; SR6/V5/UA2 remain as T2-pending-harness-ready (mechanical additions to `test_e2e_secgate_strict.py`); LGU2/LPP2 remain as NORMAL-legacy (follow-up `test_e2e_secgate_legacy_default.py`); IM4 reclassified to T2-TASKLEVEL (queue-observation pattern). See WI-KK SECGATE Harness Design block above for the propagation plan.
|
||||
- IF1 positive path (known failed pack — needs seed setup)
|
||||
- LN1-LN4 manager/notice tests (4 variants)
|
||||
- LPP1/LPP2 pip install tests
|
||||
- LGU1/LGU2 git_url install tests
|
||||
- LA1 alternatives display test
|
||||
- LDV1 disabled_versions test
|
||||
|
||||
## Classification policy (tier rule)
|
||||
|
||||
A gap is `N/A` only if no E2E observable exists for the design's stated observable.
|
||||
- **T1 DESTRUCTIVE-SAFE** (NORMAL add): design observable is a queued-task record,
|
||||
marker file, or persistent side-effect artifact. Current T1 items: CV3, SR5, V4.
|
||||
- **T2 SECGATE-PENDING** (PENDING-harness): blocked only on restricted-security test
|
||||
harness. WI-KK dissolved the original 8-item T2 bucket into 4 distinct sub-tiers
|
||||
(see **WI-KK SECGATE Harness Design** block below for the reclassification rationale).
|
||||
Current T2 items: SR6, V5, UA2 (3 Goals — harness-ready mechanical additions to
|
||||
`test_e2e_secgate_strict.py`).
|
||||
- **T2-RESOLVED** (WI-LL, post-WI-KK): Goals formally test-backed by the new secgate
|
||||
fixtures. Current items: SR4 (`test_e2e_secgate_strict.py` §20), CV4 (`test_e2e_secgate_default.py` §21).
|
||||
- **NORMAL** (post-WI-KK reclassification): Goals that do NOT need a harness because
|
||||
the default E2E config (`is_local_mode=True` + `security_level=normal`) already
|
||||
triggers the 403 path at the HTTP handler. Current items: CV4 (covered by WI-LL).
|
||||
*(Note: CV4 appears in both T2-RESOLVED and NORMAL because its classification
|
||||
shifted — it was T2 pre-WI-KK, NORMAL post-WI-KK research, and T2-RESOLVED
|
||||
post-WI-LL audit integration. The operational tier is NORMAL; T2-RESOLVED is
|
||||
the audit-status tag.)*
|
||||
- **NORMAL-legacy** (post-WI-KK reclassification): Goals registered ONLY in
|
||||
`legacy/manager_server.py`; need `start_comfyui_legacy.sh` fixture. Current items:
|
||||
LGU2, LPP2 (2 Goals — fixture already exists from WI-FF; implementation pending
|
||||
a dedicated `test_e2e_secgate_legacy_default.py` module).
|
||||
- **T2-TASKLEVEL** (post-WI-KK reclassification): gate check lives in the worker /
|
||||
`get_risky_level` pipeline, not the HTTP handler. Requires queue-observation test
|
||||
pattern, not HTTP 403 check. Current items: IM4 (1 Goal — pattern TBD).
|
||||
- **T3 IRREDUCIBLE-NA**: no test-observable artifact exists. Current items: none.
|
||||
|
||||
Re-reading all items currently categorized "intentionally skipped (destructive)":
|
||||
**CV3, SR5, V4 are T1, not N/A**, and have been promoted to NORMAL coverage tasks in
|
||||
the Key-gaps bullets above.
|
||||
|
||||
### WI-KK SECGATE Harness Design (audit-embedded propagation plan)
|
||||
|
||||
WI-KK (#182) landed two artifacts that fundamentally reshape the T2 backlog:
|
||||
|
||||
1. **`tests/e2e/scripts/start_comfyui_strict.sh`** — a strict-mode ComfyUI launcher
|
||||
that patches `user/__manager/config.ini` to `security_level=strong`, leaves a
|
||||
`.before-strict` backup, and starts the server on the E2E port. Pair this with
|
||||
a module-scoped fixture that restores the backup in teardown (the model shown
|
||||
in `test_e2e_secgate_strict.py`) and any middle/middle+ gate becomes testable.
|
||||
2. **Research finding** (WI-KK #183): `security_utils.py` L14–40 returns
|
||||
`security_level in [WEAK, NORMAL_]` for the high+ check. Under `is_local_mode=True`
|
||||
(our 127.0.0.1 default), `security_level=normal` is NOT in that set, so high+
|
||||
operations return False → 403 **at the default config, no harness needed**.
|
||||
|
||||
Combining these two insights, the original "8 T2 SECGATE-PENDING Goals, harness
|
||||
should land as one cross-cutting item" collapses into a 4-sub-tier structure:
|
||||
|
||||
| WI-KK sub-tier | Goals | Test infrastructure | Status after WI-LL |
|
||||
|---|---|---|---|
|
||||
| T2-RESOLVED | SR4, CV4 | `test_e2e_secgate_strict.py` + `test_e2e_secgate_default.py` | PASS rows landed (§20, §21) |
|
||||
| T2-pending (harness-ready) | SR6, V5, UA2 | Same strict fixture as SR4 — mechanical addition | Deferred to a follow-up PR (low lift) |
|
||||
| NORMAL-legacy | LGU2, LPP2 | `start_comfyui_legacy.sh` (exists via WI-FF) | Deferred to `test_e2e_secgate_legacy_default.py` follow-up |
|
||||
| T2-TASKLEVEL | IM4 | Queue-observation pattern (not HTTP 403) — pattern TBD | Open design question; not a simple mechanical add |
|
||||
|
||||
Propagation plan (post-PR):
|
||||
1. Land SR6, V5, UA2 in `test_e2e_secgate_strict.py` using the SR4 template; adds 3 PASS rows, brings this audit to 136/151 TOTAL.
|
||||
2. Create `test_e2e_secgate_legacy_default.py` for LGU2 + LPP2; +2 PASS → 138/153.
|
||||
3. Design the IM4 queue-observation pattern (distinct from the HTTP 403 pattern used in §20/§21); +1 PASS → 139/154. This item may be reclassified to T3 IRREDUCIBLE-NA if the observable turns out to be log-only.
|
||||
|
||||
---
|
||||
|
||||
# Conclusion
|
||||
|
||||
**100% of tests have adequate verification** (excluding N/A; denominator = 116 after WI-MM + WI-NN bloat reductions — WI-MM net -9 PASS / -2 N/A / -1 section row; WI-NN parametrize-consolidation net -9 PASS / -4 N/A). The ⚠️ WEAK bucket is now empty. **Zero INADEQUATE tests remain** after Stage2 WI-D + WI-F resolution; Stage3 WI-G closed the config_api disk/restart gap (§4: 6 WEAK → PASS). The three reconciliation waves closed every remaining WEAK:
|
||||
|
||||
- **Wave1** (WI-L/M/N): 10 WEAK → PASS across 5 files (endpoint, git_clone, customnode_info, queue_lifecycle, version_mgmt) + snapshot_lifecycle 1 WEAK → PASS, with 1 dedup via WI-M.
|
||||
- **Wave2** (WI-P/Q): 7 WEAK → PASS (task_operations 6 for update/fix effect + params; snapshot_lifecycle 1 for save_snapshot disk + content verification).
|
||||
- **Wave3** (WI-T/U/W): 10 WEAK → PASS. WI-T Cluster C+G strengthened queue_lifecycle (3), customnode_info (1), and system_info (1) with field-level effect checks; WI-U Cluster H rewrote 3 legacy-ui Playwright specs (manager-menu 2, custom-nodes 1, model-manager 2) to verify UI state via dialog reopen / `<select>.value` assertions; WI-W fixed the TaskHistoryItem params-drop schema regression and re-enabled the skipped queue_lifecycle `history?id=...` test.
|
||||
|
||||
Cumulative Wave1+Wave2+Wave3 upgrade count: **27 WEAK → PASS** (10 + 7 + 10). Matrix delta across the three waves: PASS 54 → 94 (+40 including Stage2+Stage3 upstream), WEAK 36 → 0, INADEQUATE 5 → 0. WI-Z inventory reconciliation (Y1 + Y3) added 6 pre-existing PASS rows: PASS 94 → 100, total 109 → 115. WI-AA inventory reconciliation added 2 more pre-existing PASS rows (LB1/LB2): PASS 100 → 102, total 115 → 117. WI-GG added 26 per-invocation PASS rows for `test_e2e_csrf_legacy.py` (WI-FF deliverable): PASS 102 → 128, total 117 → 143. WI-JJ (FF-deferred items) added 3 legacy-side CSRF invocations for the 2 legacy install endpoints + `is_legacy_manager_ui` flag-value parity: PASS 128 → 131, total 143 → 146. WI-LL added 2 PASS rows for the WI-KK deliverables — SR4 via `test_e2e_secgate_strict.py` §20 and CV4 via `test_e2e_secgate_default.py` §21 — closing 2 of the 8 original T2 SECGATE-PENDING Goals and reclassifying the remaining 6 across 4 sub-tiers (see WI-KK SECGATE Harness Design block above): PASS 131 → 133, total 146 → 148.
|
||||
|
||||
- Check status code without verifying the actual effect (WEAK — 0%) ✅
|
||||
- Use direct API in UI tests (INADEQUATE — 0%) ✅
|
||||
- Are outside endpoint-effect scope (N/A — 15/148 ≈ 10.1% of total)
|
||||
|
||||
Remaining 🟢 gaps (not WEAK): ~~**SR3 snapshot-remove path-traversal**~~ (RESOLVED by WI-Z Y1 — `test_remove_path_traversal_rejected`) and **QH2 queue-history path-traversal** (destructive-safe security test, already present via `test_history_path_traversal_rejected` in queue_lifecycle § Key gaps). Design-Goal coverage gap (22/92 absent, 70/92 referenced) is tracked separately in § Design-Goal Coverage Gap and is not a test-quality issue.
|
||||
|
||||
Post-Wave3 + WI-Z + WI-AA + WI-GG + WI-JJ + WI-LL state: **100% adequate coverage achieved** (133/133 PASS, excluding 15 N/A). Audit is in a terminal state for the current 148 tests. Further coverage expansion (design-Goal additions, the 3 T2-pending harness-ready Goals, NORMAL-legacy Goals, T2-TASKLEVEL IM4) is new-work territory — propagation plan is documented in the WI-KK SECGATE Harness Design block above — not reconciliation.
|
||||
|
||||
> **WI-Z + WI-AA + WI-GG + WI-JJ + WI-LL note**: Total test count 109 → 115 (WI-Z Y1 +1 snapshot, Y3 +5 config_api) → 117 (WI-AA +2 LB1/LB2) → 143 (WI-GG +26 legacy CSRF per-invocation rows) → 146 (WI-JJ +3 legacy-side install/parity rows) → 148 (WI-LL +2 SECGATE PoC rows) reflects inventory reconciliation plus WI-KK's newly-landed secgate coverage. Cumulative **upgrade** count remains 27 (unchanged since Wave3). WI-LL is the first audit-reflect WI to also introduce a Classification-policy reshape (T2 SECGATE-PENDING 8 → 4 sub-tiers), not just row additions.
|
||||
|
||||
---
|
||||
*End of E2E Verification Audit*
|
||||
@@ -0,0 +1,427 @@
|
||||
# Report A — Endpoint Extraction + Scenario Mapping
|
||||
|
||||
**Generated**: 2026-04-18
|
||||
**Source files**:
|
||||
- `comfyui_manager/glob/manager_server.py` (glob v2 — primary/current API)
|
||||
- `comfyui_manager/legacy/manager_server.py` (legacy — `--enable-manager-legacy-ui`)
|
||||
|
||||
## Summary
|
||||
|
||||
| Category | Endpoints | Unique Scenarios |
|
||||
|---|---:|---:|
|
||||
| Glob v2 | 30 | 120 |
|
||||
| Legacy-only (not in glob) | 9 | 34 |
|
||||
| Legacy-shared (same path as glob) | 29 | — (see glob) |
|
||||
| **TOTAL unique HTTP handlers** | **39** | **154** |
|
||||
|
||||
Security-gated endpoints:
|
||||
- `middle`: reboot, snapshot/remove, _uninstall_custom_node, _update_custom_node
|
||||
- `middle+`: update_all, snapshot/restore, _install_custom_node, _install_model
|
||||
- `high+`: comfyui_switch_version, install/git_url, install/pip, non-safetensors model install, _fix_custom_node (raised from `high` to align the gate with the `SECURITY_MESSAGE_HIGH_P` log text — WI-#235; prior `middle` → `high` upgrade was commit `c8992e5d`)
|
||||
|
||||
---
|
||||
|
||||
# Section 1 — Glob v2 Endpoints (`comfyui_manager/glob/manager_server.py`)
|
||||
|
||||
## 1.1 Queue Management
|
||||
|
||||
### POST /v2/manager/queue/task
|
||||
- **Handler**: `queue_task` (L1218)
|
||||
- **Schema**: `QueueTaskItem` (Pydantic) — `{kind, ui_id, client_id, params}`
|
||||
- **Scenarios**:
|
||||
1. Success — valid task (kind=install/update/fix/disable/enable/uninstall/update_comfyui/install_model) → 200
|
||||
2. Validation error — malformed kind / missing ui_id / invalid params → 400 with ValidationError text
|
||||
3. Invalid JSON body → 500
|
||||
4. State: worker auto-starts when task added
|
||||
|
||||
### GET /v2/manager/queue/history_list
|
||||
- **Handler**: `get_history_list` (L1252)
|
||||
- **Scenarios**:
|
||||
1. Success — list of batch history file IDs (basename without .json) sorted by mtime desc → 200 `{ids: [...]}`
|
||||
2. Empty history directory → 200 `{ids: []}`
|
||||
3. History path inaccessible → 400
|
||||
|
||||
### GET /v2/manager/queue/history
|
||||
- **Handler**: `get_history` (L1281)
|
||||
- **Query**: `id` (batch file) | `client_id` | `ui_id` | `max_items` | `offset`
|
||||
- **Scenarios**:
|
||||
1. Success with `id=<batch_history_id>` → reads JSON file → 200
|
||||
2. Path-traversal attempt in `id` → 400 "Invalid history id"
|
||||
3. Filter by `ui_id` — returns single task history → 200 `{history: ...}`
|
||||
4. Filter by `client_id` — filters in-memory history dict → 200
|
||||
5. Pagination (`max_items`, `offset`) → 200 `{history: ...}`
|
||||
6. JSON serialization failure (in-memory TaskHistoryItem not serializable) → 400
|
||||
7. No query params — returns full current session history → 200
|
||||
|
||||
### POST /v2/manager/queue/reset
|
||||
- **Handler**: `reset_queue` (L1718)
|
||||
- **Scenarios**:
|
||||
1. Success — wipes pending + running + history → 200
|
||||
2. Idempotent — safe to call when queue already empty → 200
|
||||
|
||||
### GET /v2/manager/queue/status
|
||||
- **Handler**: `queue_count` (L1725)
|
||||
- **Query**: `client_id` (optional)
|
||||
- **Scenarios**:
|
||||
1. No filter — global counts (total/done/in_progress/pending, is_processing) → 200
|
||||
2. With `client_id` — response includes `client_id` echo + per-client counts → 200
|
||||
3. Unknown client_id — returns 0 counts with echo → 200
|
||||
|
||||
### POST /v2/manager/queue/start
|
||||
- **Handler**: `queue_start` (L1778)
|
||||
- **Scenarios**:
|
||||
1. Worker not running — starts worker → 200
|
||||
2. Worker already running → 201 (already in-progress)
|
||||
3. Empty queue — worker starts then idles → 200
|
||||
|
||||
### POST /v2/manager/queue/update_all
|
||||
- **Handler**: `update_all` (L1411)
|
||||
- **Security gate**: `middle+` → 403 otherwise
|
||||
- **Schema**: `UpdateAllQueryParams` — `{ui_id, client_id, mode?}`
|
||||
- **Scenarios**:
|
||||
1. Success — queues update tasks for all active nodes → 200 (synchronously; slow due to reload)
|
||||
2. Missing `ui_id`/`client_id` → 400 ValidationError
|
||||
3. Security blocked (level < middle+) → 403
|
||||
4. `mode=local` — uses local channel (no network)
|
||||
5. `mode=remote/cache` — fetches cached channel data
|
||||
6. Desktop version — skips comfyui-manager pack (reads `__COMFYUI_DESKTOP_VERSION__`)
|
||||
7. Empty node set — queues 0 tasks → 200
|
||||
|
||||
### POST /v2/manager/queue/update_comfyui
|
||||
- **Handler**: `update_comfyui` (L1791)
|
||||
- **Schema**: `UpdateComfyUIQueryParams` — `{client_id, ui_id, stable?}`
|
||||
- **Scenarios**:
|
||||
1. Success — queues `update-comfyui` task → 200
|
||||
2. Missing required params → 400 ValidationError
|
||||
3. `stable=true` — forces stable update regardless of config
|
||||
4. `stable=false` — forces nightly update
|
||||
5. No `stable` param — uses config policy (nightly-comfyui vs stable)
|
||||
|
||||
### POST /v2/manager/queue/install_model
|
||||
- **Handler**: `install_model` (L1875)
|
||||
- **Schema**: `ModelMetadata` (name, type, base, url, filename, save_path) + required `client_id` + `ui_id`
|
||||
- **Scenarios**:
|
||||
1. Success — valid request → queues `install-model` task → 200
|
||||
2. Missing `client_id` → 400 "Missing required field: client_id"
|
||||
3. Missing `ui_id` → 400 "Missing required field: ui_id"
|
||||
4. Invalid model metadata (missing name/url/filename) → 400 ValidationError
|
||||
5. Malformed JSON → 500
|
||||
|
||||
## 1.2 Custom Node Info
|
||||
|
||||
### GET /v2/customnode/getmappings
|
||||
- **Handler**: `fetch_customnode_mappings` (L1357)
|
||||
- **Query**: `mode` (required: local|cache|remote|nickname)
|
||||
- **Scenarios**:
|
||||
1. Success — returns `{pack_id: [node_list, metadata]}` dict → 200
|
||||
2. `mode=nickname` — applies nickname_filter → 200
|
||||
3. Missing `mode` → KeyError → 500
|
||||
4. Invalid `mode` value → may raise from get_data_by_mode → 400/500
|
||||
5. Missing nodes matched by `nodename_pattern` regex are appended
|
||||
|
||||
### GET /v2/customnode/fetch_updates
|
||||
- **Handler**: `fetch_updates` (L1393)
|
||||
- **Scenarios**:
|
||||
1. Always returns **410 Gone** with `{deprecated: true}` — deprecated endpoint
|
||||
2. Client should migrate to queue/update-based flow
|
||||
|
||||
### GET /v2/customnode/installed
|
||||
- **Handler**: `installed_list` (L1500)
|
||||
- **Query**: `mode` (default|imported)
|
||||
- **Scenarios**:
|
||||
1. Default mode — current installed packs snapshot → 200 dict
|
||||
2. `mode=imported` — startup-time installed packs (frozen snapshot) → 200
|
||||
3. Empty custom_nodes dir → 200 `{}`
|
||||
|
||||
### POST /v2/customnode/import_fail_info
|
||||
- **Handler**: `import_fail_info` (L1623)
|
||||
- **Body**: `{cnr_id?, url?}` — one required
|
||||
- **Scenarios**:
|
||||
1. Known failed pack via `cnr_id` — returns `{msg, traceback}` → 200
|
||||
2. Known failed pack via `url` — returns info → 200
|
||||
3. Unknown pack → 400 (no failure info available)
|
||||
4. Missing both cnr_id and url → 400 "Either 'cnr_id' or 'url' field is required"
|
||||
5. `cnr_id` not a string → 400 "'cnr_id' must be a string"
|
||||
6. Non-dict body → 400 "Request body must be a JSON object"
|
||||
|
||||
### POST /v2/customnode/import_fail_info_bulk
|
||||
- **Handler**: `import_fail_info_bulk` (L1657)
|
||||
- **Schema**: `ImportFailInfoBulkRequest` — `{cnr_ids: [], urls: []}`
|
||||
- **Scenarios**:
|
||||
1. Success with `cnr_ids` list — returns `{cnr_id: {error, traceback}|null}` → 200
|
||||
2. Success with `urls` list — returns `{url: {error, traceback}|null}` → 200
|
||||
3. Both lists empty → 400 "Either 'cnr_ids' or 'urls' field is required"
|
||||
4. Validation error (wrong types) → 400
|
||||
5. Each unknown pack → `null` in results dict (not an error)
|
||||
|
||||
## 1.3 Snapshots
|
||||
|
||||
### GET /v2/snapshot/getlist
|
||||
- **Handler**: `get_snapshot_list` (L1512)
|
||||
- **Scenarios**:
|
||||
1. Success — list of snapshot file stems (basename minus .json), sorted desc → 200 `{items: [...]}`
|
||||
2. Empty snapshot dir → 200 `{items: []}`
|
||||
|
||||
### GET /v2/snapshot/get_current
|
||||
- **Handler**: `get_current_snapshot_api` (L1575)
|
||||
- **Scenarios**:
|
||||
1. Success — returns current system state dict → 200
|
||||
2. Internal failure → 400
|
||||
|
||||
### POST /v2/snapshot/save
|
||||
- **Handler**: `save_snapshot` (L1585)
|
||||
- **Scenarios**:
|
||||
1. Success — creates timestamped snapshot file → 200
|
||||
2. Internal failure → 400
|
||||
3. Multiple rapid saves — each creates distinct timestamped file
|
||||
|
||||
### POST /v2/snapshot/remove
|
||||
- **Handler**: `remove_snapshot` (L1521)
|
||||
- **Security gate**: `middle` → 403
|
||||
- **Query**: `target` (snapshot file stem)
|
||||
- **Scenarios**:
|
||||
1. Success — removes existing file → 200
|
||||
2. Nonexistent target — 200 (no-op)
|
||||
3. Path traversal (`../x`) → 400 "Invalid target"
|
||||
4. Missing `target` query → exception → 400
|
||||
5. Security blocked (level < middle) → 403
|
||||
|
||||
### POST /v2/snapshot/restore
|
||||
- **Handler**: `restore_snapshot` (L1543)
|
||||
- **Security gate**: `middle+` → 403
|
||||
- **Query**: `target`
|
||||
- **Scenarios**:
|
||||
1. Success — copies snapshot to startup script path (applied on next reboot) → 200
|
||||
2. Nonexistent target → 400
|
||||
3. Path traversal → 400 "Invalid target"
|
||||
4. Security blocked → 403
|
||||
|
||||
## 1.4 Configuration
|
||||
|
||||
### GET /v2/manager/db_mode
|
||||
- **Handler**: `db_mode` (L1907)
|
||||
- **Scenarios**: Returns plain text current value ∈ {cache, channel, local, remote} → 200
|
||||
|
||||
### POST /v2/manager/db_mode
|
||||
- **Handler**: `set_db_mode_api` (L1912)
|
||||
- **Body**: `{value: <mode>}`
|
||||
- **Scenarios**:
|
||||
1. Valid value — persists to config.ini → 200
|
||||
2. Malformed JSON → 400 "Invalid request"
|
||||
3. Missing `value` key → 400 KeyError
|
||||
|
||||
### GET /v2/manager/policy/update
|
||||
- **Handler**: `update_policy` (L1923)
|
||||
- **Scenarios**: Returns plain text value ∈ {stable, stable-comfyui, nightly, nightly-comfyui} → 200
|
||||
|
||||
### POST /v2/manager/policy/update
|
||||
- **Handler**: `set_update_policy_api` (L1928)
|
||||
- **Body**: `{value: <policy>}`
|
||||
- **Scenarios**:
|
||||
1. Valid value → persists config → 200
|
||||
2. Malformed JSON / missing value → 400
|
||||
|
||||
### GET /v2/manager/channel_url_list
|
||||
- **Handler**: `channel_url_list` (L1939)
|
||||
- **Scenarios**:
|
||||
1. Success — `{selected: name, list: ["name::url", ...]}` → 200
|
||||
2. Selected URL doesn't match any channel → `selected="custom"`
|
||||
|
||||
### POST /v2/manager/channel_url_list
|
||||
- **Handler**: `set_channel_url` (L1954)
|
||||
- **Body**: `{value: <channel_name>}`
|
||||
- **Scenarios**:
|
||||
1. Known channel name → persists new URL → 200
|
||||
2. Unknown channel name → no-op → 200 (silent)
|
||||
3. Malformed JSON / missing value → 400
|
||||
|
||||
## 1.5 System
|
||||
|
||||
### GET /v2/manager/is_legacy_manager_ui
|
||||
- **Handler**: `is_legacy_manager_ui` (L1487)
|
||||
- **Scenarios**: Returns `{is_legacy_manager_ui: bool}` reflecting `--enable-manager-legacy-ui` flag → 200
|
||||
|
||||
### GET /v2/manager/version
|
||||
- **Handler**: `get_version` (L2009)
|
||||
- **Scenarios**: Returns plain text `core.version_str` → 200
|
||||
|
||||
### POST /v2/manager/reboot
|
||||
- **Handler**: `restart` (L1968)
|
||||
- **Security gate**: `middle` → 403
|
||||
- **Scenarios**:
|
||||
1. Success — triggers server process restart via execv → 200 (connection may drop)
|
||||
2. `__COMFY_CLI_SESSION__` env set — writes reboot marker + exit(0) instead of execv
|
||||
3. Desktop/Windows standalone variant — removes flag before restart
|
||||
4. Security blocked → 403
|
||||
|
||||
## 1.6 ComfyUI Version Management
|
||||
|
||||
### GET /v2/comfyui_manager/comfyui_versions
|
||||
- **Handler**: `comfyui_versions` (L1825)
|
||||
- **Scenarios**:
|
||||
1. Success — `{versions: [...], current: "<tag or hash>"}` → 200
|
||||
2. Git access failure → 400
|
||||
|
||||
### POST /v2/comfyui_manager/comfyui_switch_version
|
||||
- **Handler**: `comfyui_switch_version` (L1840)
|
||||
- **Security gate**: `high+` → 403
|
||||
- **Schema**: `ComfyUISwitchVersionParams` — `{ver, client_id, ui_id}` (JSON body; renamed in WI #261, migrated from query string in WI #258)
|
||||
- **Scenarios**:
|
||||
1. Success — queues update-comfyui task with target_version → 200
|
||||
2. Missing `ver`/`client_id`/`ui_id` → 400 ValidationError (JSON with `error` field)
|
||||
3. Security blocked → 403
|
||||
4. Internal exception → 400
|
||||
|
||||
---
|
||||
|
||||
# Section 2 — Legacy-Only Endpoints (`comfyui_manager/legacy/manager_server.py`)
|
||||
|
||||
Endpoints in this section exist **only in the legacy server** (not registered in glob). They are served when `--enable-manager-legacy-ui` is set.
|
||||
|
||||
### POST /v2/manager/queue/batch
|
||||
- **Handler**: `queue_batch` (L740)
|
||||
- **Body**: dict with keys ∈ {update_all, reinstall, install, uninstall, update, update_comfyui, disable, install_model, fix}, each with a list of per-pack payloads
|
||||
- **Scenarios**:
|
||||
1. Success with single kind (e.g. install) → appends to temp_queue_batch, finalizes, starts worker → 200 `{failed: []}`
|
||||
2. Mixed kinds in one batch — processes each sequentially
|
||||
3. Partial failure — some packs fail internally → 200 `{failed: [...ids]}`
|
||||
4. `update_all` with mode — runs legacy update_all flow inline
|
||||
5. `reinstall` — uninstall then install per pack; uninstall failure skips install
|
||||
6. `disable` — no security check inside `_disable_node`
|
||||
7. `install` with security gate fail → failed set entry
|
||||
8. Empty body `{}` → 200 `{failed: []}`
|
||||
|
||||
### GET /v2/customnode/getlist
|
||||
- **Handler**: `fetch_customnode_list` (L1018)
|
||||
- **Query**: `mode` (required), `skip_update?` (bool string)
|
||||
- **Scenarios**:
|
||||
1. Success — returns `{channel, node_packs}` with installed/update state populated → 200
|
||||
2. `skip_update=true` — skips git update check (faster)
|
||||
3. Removes comfyui-manager self-entry from results
|
||||
4. Channel lookup resolves to 'default'/'custom'/known name
|
||||
|
||||
### GET /customnode/alternatives
|
||||
- **Handler**: `fetch_customnode_alternatives` (L1072)
|
||||
- **Query**: `mode` (required)
|
||||
- **Scenarios**:
|
||||
1. Success — alter-list.json items keyed by id → 200
|
||||
|
||||
### GET /v2/externalmodel/getlist
|
||||
- **Handler**: `fetch_externalmodel_list` (L1143)
|
||||
- **Query**: `mode` (required)
|
||||
- **Scenarios**:
|
||||
1. Success — model-list.json with `installed` flag populated per file → 200
|
||||
2. HuggingFace sentinel filename — resolves from URL basename
|
||||
3. Custom save_path — checks under models/<save_path>
|
||||
|
||||
### GET /v2/customnode/versions/{node_name}
|
||||
- **Handler**: `get_cnr_versions` (L1262)
|
||||
- **Path param**: `node_name`
|
||||
- **Scenarios**:
|
||||
1. Known CNR pack — returns version list → 200
|
||||
2. Unknown pack — 400
|
||||
|
||||
### GET /v2/customnode/disabled_versions/{node_name}
|
||||
- **Handler**: `get_disabled_versions` (L1273)
|
||||
- **Scenarios**:
|
||||
1. Pack has nightly_inactive entry → version list includes "nightly"
|
||||
2. Pack has cnr_inactive entries → versions list
|
||||
3. No disabled versions → 400
|
||||
|
||||
### POST /v2/customnode/install/git_url
|
||||
- **Handler**: `install_custom_node_git_url` (L1502)
|
||||
- **Security gate**: `high+` → 403
|
||||
- **Body**: plain text URL
|
||||
- **Scenarios**:
|
||||
1. Success install → 200
|
||||
2. Already installed (skip action) → 200
|
||||
3. Clone failure → 400
|
||||
4. Security blocked → 403
|
||||
|
||||
### POST /v2/customnode/install/pip
|
||||
- **Handler**: `install_custom_node_pip` (L1522)
|
||||
- **Security gate**: `high+` → 403
|
||||
- **Body**: plain text space-separated packages
|
||||
- **Scenarios**:
|
||||
1. Success — pip install completes → 200
|
||||
2. Security blocked → 403
|
||||
|
||||
### GET /v2/manager/notice
|
||||
- **Handler**: `get_notice` (L1747)
|
||||
- **Scenarios**:
|
||||
1. Success — fetches GitHub wiki News, returns HTML → 200
|
||||
2. GitHub unreachable / non-200 → 200 "Unable to retrieve Notice" (plain text)
|
||||
3. No markdown-body div matched → 200 "Unable to retrieve Notice"
|
||||
4. Appends ComfyUI/Manager version footer
|
||||
5. Desktop variant — uses `__COMFYUI_DESKTOP_VERSION__`
|
||||
6. Non-git ComfyUI — prepends "Your ComfyUI isn't git repo" warning
|
||||
7. Outdated ComfyUI (required_commit_datetime > current) — prepends "too OUTDATED" warning
|
||||
|
||||
---
|
||||
|
||||
# Section 3 — Legacy-Shared Endpoints
|
||||
|
||||
These paths exist in BOTH glob and legacy files (29 endpoints). Semantics are typically equivalent but implementation may differ; see glob scenarios above. Notable differences:
|
||||
|
||||
- **queue/status** (L1379 legacy) — counts from `task_batch_queue[0]` (first batch only), not aggregated across batches like glob
|
||||
- **queue/start** (L1465 legacy) — calls `finalize_temp_queue_batch()` first, then starts worker thread
|
||||
- **update_all** (L904 legacy) — returns 401 if worker already running; auto-saves snapshot; uses temp_queue_batch instead of QueueTaskItem
|
||||
- **update_comfyui** (L1572 legacy) — no params; reads config.update_policy directly; always returns 200
|
||||
- **reboot** (L1796 legacy) — identical behavior to glob
|
||||
- **history** (L819 legacy) — only supports `id` query (no client_id/ui_id/pagination)
|
||||
- **import_fail_info** (L1289 legacy) — no basic dict validation; assumes cnr_id/url present (KeyError otherwise)
|
||||
|
||||
---
|
||||
|
||||
# Security Level Matrix
|
||||
|
||||
| Level | Glob endpoints | Legacy endpoints |
|
||||
|---|---|---|
|
||||
| **middle** | snapshot/remove, reboot | snapshot/remove, reboot, _uninstall, _update |
|
||||
| **middle+** | update_all, snapshot/restore, install_model | update_all, snapshot/restore, _install_custom_node, _install_model |
|
||||
| **high+** | comfyui_switch_version, _fix_custom_node | comfyui_switch_version, install/git_url, install/pip, non-safetensors model install, _fix |
|
||||
|
||||
> Note: `_fix` / `_fix_custom_node` security-level history: `middle` → `high` in commit `c8992e5d` (2026-04-04, which also added a previously-missing gate to the legacy handler); subsequent `high` → `high+` in WI-#235 to align the enforcement gate with the `SECURITY_MESSAGE_HIGH_P` log text (and tighten the gate for a state-mutating fix path). README 'Risky Level Table' has been updated in lockstep.
|
||||
|
||||
# Deprecated / Removed
|
||||
|
||||
- **GET /v2/customnode/fetch_updates** — glob returns 410; legacy still attempts fetch (may succeed but deprecated in concept)
|
||||
- **Individual queue/{install,uninstall,update,fix,disable,reinstall,abort_current}** — removed from legacy in recent work (replaced by queue/batch aggregator)
|
||||
- **GET /manager/notice** (v1, no /v2 prefix) — removed from legacy
|
||||
|
||||
---
|
||||
|
||||
# CSRF Method-Reject Contract Inventory
|
||||
|
||||
**Purpose**: Enumerate the 16 state-changing endpoints that must reject HTTP `GET` after commit `99caef55` (CSRF method-conversion mitigation; CVSS 8.1, reported by XlabAI / Tencent Xuanwu). This inventory supplements `verification_design.md` Section 10 (Goals CSRF-M1 / M2 / M3) and is the authoritative cross-reference for the contract enforced by `tests/e2e/test_e2e_csrf.py`.
|
||||
|
||||
**Data Source**: `tests/e2e/test_e2e_csrf.py::STATE_CHANGING_POST_ENDPOINTS` (L92-L109). Pre-99caef55 methods derived from `git log -S` history and the commit body of 99caef55. Security Level column cross-references the Security Level Matrix (§ above, L378-L382).
|
||||
|
||||
| # | Endpoint | Pre-99caef55 Method | Post-99caef55 Method | Security Level | Test Reference |
|
||||
|---|----------|---------------------|----------------------|----------------|----------------|
|
||||
| 1 | `/v2/manager/queue/start` | GET | POST | default | `TestStateChangingEndpointsRejectGet::test_get_is_rejected[/v2/manager/queue/start]` |
|
||||
| 2 | `/v2/manager/queue/reset` | GET | POST | default | `TestStateChangingEndpointsRejectGet::test_get_is_rejected[/v2/manager/queue/reset]` |
|
||||
| 3 | `/v2/manager/queue/update_all` | GET | POST | middle+ | `TestStateChangingEndpointsRejectGet::test_get_is_rejected[/v2/manager/queue/update_all]` |
|
||||
| 4 | `/v2/manager/queue/update_comfyui` | GET | POST | default | `TestStateChangingEndpointsRejectGet::test_get_is_rejected[/v2/manager/queue/update_comfyui]` |
|
||||
| 5 | `/v2/manager/queue/install_model` | POST | POST (pre-existing) | middle+ | `TestStateChangingEndpointsRejectGet::test_get_is_rejected[/v2/manager/queue/install_model]` |
|
||||
| 6 | `/v2/manager/queue/task` | POST | POST (pre-existing) | default | `TestStateChangingEndpointsRejectGet::test_get_is_rejected[/v2/manager/queue/task]` |
|
||||
| 7 | `/v2/snapshot/save` | GET | POST | default | `TestStateChangingEndpointsRejectGet::test_get_is_rejected[/v2/snapshot/save]` |
|
||||
| 8 | `/v2/snapshot/remove` | GET | POST | middle | `TestStateChangingEndpointsRejectGet::test_get_is_rejected[/v2/snapshot/remove]` |
|
||||
| 9 | `/v2/snapshot/restore` | GET | POST | middle+ | `TestStateChangingEndpointsRejectGet::test_get_is_rejected[/v2/snapshot/restore]` |
|
||||
| 10 | `/v2/manager/reboot` | GET | POST | middle | `TestStateChangingEndpointsRejectGet::test_get_is_rejected[/v2/manager/reboot]` |
|
||||
| 11 | `/v2/comfyui_manager/comfyui_switch_version` | GET | POST | high+ | `TestStateChangingEndpointsRejectGet::test_get_is_rejected[/v2/comfyui_manager/comfyui_switch_version]` |
|
||||
| 12 | `/v2/manager/db_mode` | GET (dual) | POST (write); GET preserved for read | default | `TestStateChangingEndpointsRejectGet::test_get_is_rejected[/v2/manager/db_mode]` |
|
||||
| 13 | `/v2/manager/policy/update` | GET (dual) | POST (write); GET preserved for read | default | `TestStateChangingEndpointsRejectGet::test_get_is_rejected[/v2/manager/policy/update]` |
|
||||
| 14 | `/v2/manager/channel_url_list` | GET (dual) | POST (write); GET preserved for read | default | `TestStateChangingEndpointsRejectGet::test_get_is_rejected[/v2/manager/channel_url_list]` |
|
||||
| 15 | `/v2/customnode/import_fail_info` | POST | POST (pre-existing) | default | `TestStateChangingEndpointsRejectGet::test_get_is_rejected[/v2/customnode/import_fail_info]` |
|
||||
| 16 | `/v2/customnode/import_fail_info_bulk` | POST | POST (pre-existing) | default | `TestStateChangingEndpointsRejectGet::test_get_is_rejected[/v2/customnode/import_fail_info_bulk]` |
|
||||
|
||||
**Conversion Breakdown** (reconciles commit 99caef55 body with 16-row fixture):
|
||||
- **Pure GET → POST conversions**: 9 endpoints (rows 1-4, 7-11) — confirmed write-only operations formerly exposed via GET.
|
||||
- **Dual-method endpoints** (GET + POST coexist after fix): 3 endpoints (rows 12-14) — the POST variant carries write semantics; GET preserved for read-only retrieval and is covered by Goal CSRF-M3.
|
||||
- **Pre-existing POST endpoints** (included in the fixture for contract completeness): 4 endpoints (rows 5-6, 15-16) — these were already POST before 99caef55 but remain part of the CSRF rejection contract so any future regression to GET is caught.
|
||||
|
||||
**Scope Note**: This inventory narrowly documents the method-restriction layer. Complementary CSRF defenses (Origin/Referer validation, same-site cookies, anti-CSRF tokens, cross-site form POST rejection) are out of scope for this contract and tracked in `verification_design.md` § 10.2.
|
||||
|
||||
---
|
||||
*End of Report A*
|
||||
@@ -0,0 +1,104 @@
|
||||
# Legacy UI — Channel Combo DOM Mapping Note
|
||||
|
||||
**Date**: 2026-04-20
|
||||
**Context**: WI-OO Item 3 follow-up — bloat sweep `dev:ci-022` B8 "Channel dropdown"
|
||||
test was title-mismatched (filter `/Cache|Local|Channel/` matched DB combo's
|
||||
option text, not the Channel combo itself). This record documents the correct
|
||||
DOM locators for any future reactivation.
|
||||
**Scope**: Record only. No test code added; no audit impact.
|
||||
|
||||
---
|
||||
|
||||
## 1. Source Locations
|
||||
|
||||
| Artifact | Path | Lines | Role |
|
||||
|----------|------|-------|------|
|
||||
| Channel combo creation + registration | `comfyui_manager/js/comfyui-manager.js` | 960-986 | Creates the `<select>`, installs title attr, fetches options from API, mounts into the settings panel via `createSettingsCombo` |
|
||||
| Settings row wrapper | `comfyui_manager/js/comfyui-gui-builder.js` | 15-27 | Exports `createSettingsCombo(label, content)` that wraps the combo in the label/input row |
|
||||
|
||||
## 2. DOM Structure
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Tag | `<select>` |
|
||||
| Classes | `cm-menu-combo p-select p-component p-inputwrapper p-inputwrapper-filled` |
|
||||
| `title` attribute | `"Configure the channel for retrieving data from the Custom Node list (including missing nodes) or the Model list."` (set at L961) |
|
||||
| Options source | `/v2/manager/channel_url_list` — populated asynchronously at L963-984 |
|
||||
| Option texts | Channel URL names (e.g. `default`, `recent`, custom URLs); NOT the word "Channel" |
|
||||
| Label wrapper | `div.setting-item > div.flex.flex-row.items-center.gap-2 > div.form-label.flex.grow.items-center > span.text-muted` with `textContent: "Channel"` (from `createSettingsCombo`) |
|
||||
| Render timing | Select element itself: **sync** at menu build time. Options: **async** after `channel_url_list` fetch resolves |
|
||||
|
||||
## 3. Why the Original `hasText: /Cache|Local|Channel/` Filter Failed
|
||||
|
||||
The removed test used `hasText` to find the Channel dropdown, but that matcher
|
||||
searches the element's rendered text (its `<option>` children's text in the case
|
||||
of a `<select>`). The Channel combo's options are channel URL names — they do
|
||||
not contain the words `Cache`, `Local`, or `Channel`.
|
||||
|
||||
In contrast, the DB (datasrc) combo located a few lines above
|
||||
(comfyui-manager.js:957, built from `this.datasrc_combo` which seeds options
|
||||
`Cache` / `Local` / `Remote`) did contain those literals, so the filter
|
||||
silently resolved to the wrong `<select>`. The test asserted visibility, which
|
||||
passed against the DB combo, masking the mismatch until WI-OO's audit exposed
|
||||
it as B8 title-mismatch bloat.
|
||||
|
||||
## 4. Stable Selector Candidates
|
||||
|
||||
Ordered by robustness (most stable first):
|
||||
|
||||
1. **Title attribute (recommended)** — unique per L961
|
||||
```ts
|
||||
select[title^="Configure the channel"]
|
||||
```
|
||||
The leading prefix `Configure the channel` appears nowhere else in the
|
||||
managed panel. Safe against minor title copy edits as long as the opening
|
||||
phrase is preserved.
|
||||
|
||||
2. **Label-based scope** — DOM-structure dependent
|
||||
```ts
|
||||
.setting-item:has(span.text-muted:text-is("Channel")) select
|
||||
```
|
||||
Works as long as `createSettingsCombo` keeps its current wrapper shape and
|
||||
the exact label text `"Channel"`.
|
||||
|
||||
3. **Class-only** — NOT unique
|
||||
Classes `cm-menu-combo p-select ...` are shared with the DB, Update-Policy,
|
||||
and Share combos. Using classes alone will match multiple elements and is
|
||||
brittle.
|
||||
|
||||
## 5. Async-Population Note
|
||||
|
||||
Options for the Channel combo are populated via an async `fetchApi` call to
|
||||
`/v2/manager/channel_url_list` at L963. Two testing consequences:
|
||||
|
||||
- A visibility assertion on the `<select>` resolves immediately — the element
|
||||
is appended synchronously at L960 and mounted at L986.
|
||||
- An assertion about option count or specific option values MUST wait for the
|
||||
fetch to resolve. Use `expect.poll` (or equivalent) with a reasonable
|
||||
timeout (≥5s) rather than an immediate `toHaveCount` check.
|
||||
|
||||
## 6. Proposed Test Skeleton (Reference Only)
|
||||
|
||||
Not added to any spec — kept here for future activation.
|
||||
|
||||
```ts
|
||||
test('shows Channel dropdown (async-populated)', async ({ page }) => {
|
||||
await openManagerMenu(page);
|
||||
const dialog = page.locator('#cm-manager-dialog').first();
|
||||
const channelCombo = dialog.locator('select[title^="Configure the channel"]');
|
||||
await expect(channelCombo).toBeVisible();
|
||||
await expect.poll(
|
||||
async () => await channelCombo.locator('option').count(),
|
||||
{ timeout: 5000 }
|
||||
).toBeGreaterThan(0);
|
||||
});
|
||||
```
|
||||
|
||||
## 7. Decision
|
||||
|
||||
Test **not** added. This aligns with the post-bloat-sweep net-removal
|
||||
direction established by WI-OO: re-introducing a Channel-dropdown visibility
|
||||
test would re-expand the surface the sweep explicitly trimmed. The record is
|
||||
preserved here so that, if future coverage expansion prioritizes the settings
|
||||
panel, reactivation needs only copy the skeleton above and choose selector
|
||||
option 1 from §4.
|
||||
@@ -0,0 +1,215 @@
|
||||
# Research: Cluster G Semantics — imported_mode + boolean CLI flag
|
||||
|
||||
**Scope**: Wave3 Cluster G pre-research for dev assertion design.
|
||||
**Researcher**: gteam-teng (Explore, read-only)
|
||||
**Date**: 2026-04-19
|
||||
**Targets**: 2 | **Status**: both resolved
|
||||
|
||||
---
|
||||
|
||||
## Target 1 — `/v2/customnode/installed?mode=imported` Semantics
|
||||
|
||||
### (i) Current source behavior — FROZEN AT STARTUP confirmed
|
||||
|
||||
**Source: `comfyui_manager/glob/manager_server.py`**
|
||||
|
||||
```python
|
||||
# L1510 — module-level evaluation at import time
|
||||
startup_time_installed_node_packs = core.get_installed_node_packs()
|
||||
|
||||
# L1513-1522
|
||||
@routes.get("/v2/customnode/installed")
|
||||
async def installed_list(request):
|
||||
mode = request.query.get("mode", "default")
|
||||
if mode == "imported":
|
||||
res = startup_time_installed_node_packs # frozen
|
||||
else:
|
||||
res = core.get_installed_node_packs() # live
|
||||
```
|
||||
|
||||
**Source: `comfyui_manager/glob/manager_core.py:1599-1632`** — `get_installed_node_packs()` scans filesystem via `os.listdir()` on every call (LIVE).
|
||||
|
||||
**Design intent**: "imported" mode returns the snapshot captured exactly once, at module import time (when `from .glob import manager_server` runs during ComfyUI startup). Default mode re-scans the filesystem. The divergence surfaces after a runtime install — default grows, imported does not. Used by `TaskQueue` (`manager_server.py:211`) to know what was loaded vs what is now on disk.
|
||||
|
||||
### (ii) Test-env expected value
|
||||
|
||||
At startup, before any install action, `imported == default` in content (same filesystem state, same scan logic). The seed pack `ComfyUI_SigmoidOffsetScheduler` MUST be present in both.
|
||||
|
||||
Schema per entry: `{cnr_id: str, ver: str, aux_id: Optional[str], enabled: bool}` — see `manager_core.py:1614` & `:1630`.
|
||||
|
||||
### (iii) Wave3 assertion code snippet (Cluster G)
|
||||
|
||||
**Strategy A — schema + seed check (cheap, deterministic, no install needed):**
|
||||
|
||||
```python
|
||||
def test_installed_imported_mode(self, comfyui):
|
||||
"""GET ?mode=imported returns startup snapshot with documented schema.
|
||||
|
||||
Frozen-at-startup invariant: at test time (no installs have occurred
|
||||
since server start), the imported snapshot must match the live listing
|
||||
in cardinality + key set, and each entry must carry the documented
|
||||
InstalledPack schema.
|
||||
"""
|
||||
# Frozen snapshot
|
||||
resp_imp = requests.get(
|
||||
f"{BASE_URL}/v2/customnode/installed",
|
||||
params={"mode": "imported"}, timeout=10,
|
||||
)
|
||||
assert resp_imp.status_code == 200
|
||||
imported = resp_imp.json()
|
||||
assert isinstance(imported, dict), f"expected dict, got {type(imported).__name__}"
|
||||
|
||||
# E2E seed pack must be in the startup snapshot
|
||||
seed = "ComfyUI_SigmoidOffsetScheduler"
|
||||
assert seed in imported, (
|
||||
f"seed pack {seed!r} missing from imported snapshot: keys={list(imported)}"
|
||||
)
|
||||
# Schema: same as default mode
|
||||
entry = imported[seed]
|
||||
for required in ("cnr_id", "ver", "enabled"):
|
||||
assert required in entry, f"{seed} missing {required!r}: {entry!r}"
|
||||
|
||||
# Frozen invariant (cheap form): imported at startup == default at startup
|
||||
# (no install has occurred, so they must agree on keys + core fields)
|
||||
resp_def = requests.get(f"{BASE_URL}/v2/customnode/installed", timeout=10)
|
||||
default = resp_def.json()
|
||||
assert set(imported.keys()) == set(default.keys()), (
|
||||
f"imported != default at startup: "
|
||||
f"only-imported={set(imported)-set(default)}, "
|
||||
f"only-default={set(default)-set(imported)}"
|
||||
)
|
||||
```
|
||||
|
||||
**Strategy B — true frozen invariant (expensive, OPTIONAL, skip by default):**
|
||||
|
||||
```python
|
||||
@pytest.mark.skip(reason=
|
||||
"Requires post-startup install; E2E runtime install is slow and gated by "
|
||||
"security_level. Enable via PYTEST_FULL_IMPORTED_MODE=1 for nightly runs.")
|
||||
def test_imported_mode_is_frozen_after_install(self, comfyui):
|
||||
"""After installing a new pack, imported mode MUST still match startup.
|
||||
|
||||
This is the true 'frozen' test — install a pack, then verify default mode
|
||||
sees it while imported mode does not (it was snapshotted before install).
|
||||
"""
|
||||
snap_before = requests.get(
|
||||
f"{BASE_URL}/v2/customnode/installed", params={"mode": "imported"}, timeout=10,
|
||||
).json()
|
||||
# ... trigger install of a fresh pack via /v2/customnode/install or FS manipulation ...
|
||||
snap_after = requests.get(
|
||||
f"{BASE_URL}/v2/customnode/installed", params={"mode": "imported"}, timeout=10,
|
||||
).json()
|
||||
assert snap_before == snap_after, "imported snapshot mutated — frozen invariant broken"
|
||||
live_after = requests.get(f"{BASE_URL}/v2/customnode/installed", timeout=10).json()
|
||||
assert set(live_after) - set(snap_after), "default mode did not reflect the new install"
|
||||
```
|
||||
|
||||
### (iv) Recommendation
|
||||
|
||||
- Adopt **Strategy A** as the WEAK-upgrade replacement — cheap, deterministic, ADEQUATE (positive path + field-level + cross-mode consistency).
|
||||
- Register **Strategy B** as `[E2E-DEBT]` in the scaffold; keep `@pytest.mark.skip` unless a nightly pipeline enables it.
|
||||
- Limitation to document: Strategy A cannot distinguish "frozen" from "live-and-coincidentally-equal" without a mid-session install — that's what Strategy B covers.
|
||||
|
||||
---
|
||||
|
||||
## Target 2 — `/v2/manager/is_legacy_manager_ui` boolean field (NOT /v2/manager/version)
|
||||
|
||||
**CORRECTION**: Dispatch text suggested `/v2/manager/version` as an example, but `test_returns_boolean_field` is defined inside `class TestIsLegacyManagerUI` (`tests/e2e/test_e2e_system_info.py:151-166`) and actually hits `/v2/manager/is_legacy_manager_ui`. `test_e2e_system_info.py::TestManagerVersion::test_version_returns_string` handles `/v2/manager/version` separately (returns `text/plain`, not JSON bool).
|
||||
|
||||
### (i) Current source behavior
|
||||
|
||||
**Source: `comfyui_manager/glob/manager_server.py:1500-1506`**
|
||||
|
||||
```python
|
||||
@routes.get("/v2/manager/is_legacy_manager_ui")
|
||||
async def is_legacy_manager_ui(request):
|
||||
return web.json_response(
|
||||
{"is_legacy_manager_ui": args.enable_manager_legacy_ui},
|
||||
content_type="application/json",
|
||||
status=200,
|
||||
)
|
||||
```
|
||||
|
||||
**`args`** is imported from `comfy.cli_args` (upstream ComfyUI argparse — `comfyui_manager/__init__.py:6`). The flag `--enable-manager-legacy-ui` is registered by ComfyUI's own cli_args module (not in this repo). `action='store_true'` means default is `False` (bool), not `None`.
|
||||
|
||||
**Same handler exists in legacy server** at `comfyui_manager/legacy/manager_server.py:995-1001` — identical body.
|
||||
|
||||
**Also read in glob at `__init__.py:19`** to gate `from .legacy import manager_server` import. This confirms the value is bool at module load time (used as an `if`).
|
||||
|
||||
### (ii) Test-env expected value — DETERMINISTIC
|
||||
|
||||
**Source: `tests/e2e/scripts/start_comfyui.sh:73-79`** (launch command):
|
||||
|
||||
```bash
|
||||
nohup "$PY" "$COMFY_DIR/main.py" \
|
||||
--cpu \
|
||||
--enable-manager \
|
||||
--port "$PORT" \
|
||||
> "$LOG_FILE" 2>&1 &
|
||||
```
|
||||
|
||||
The E2E launcher passes NO `--enable-manager-legacy-ui` flag. Therefore in every E2E run: `args.enable_manager_legacy_ui = False`.
|
||||
|
||||
No `tests/e2e/**` file references the flag (grep confirmed: 0 matches).
|
||||
|
||||
### (iii) Wave3 assertion code snippet
|
||||
|
||||
**Strengthen from `isinstance(bool)` → exact-value `is False`:**
|
||||
|
||||
```python
|
||||
def test_returns_boolean_field(self, comfyui):
|
||||
"""GET /v2/manager/is_legacy_manager_ui returns {is_legacy_manager_ui: False} in E2E.
|
||||
|
||||
E2E env deterministically omits --enable-manager-legacy-ui
|
||||
(start_comfyui.sh passes only --cpu --enable-manager --port),
|
||||
so args.enable_manager_legacy_ui defaults to False (store_true default).
|
||||
Strengthened from type-only check to exact-value check.
|
||||
"""
|
||||
resp = requests.get(
|
||||
f"{BASE_URL}/v2/manager/is_legacy_manager_ui", timeout=10,
|
||||
)
|
||||
assert resp.status_code == 200, f"Expected 200, got {resp.status_code}"
|
||||
data = resp.json()
|
||||
assert "is_legacy_manager_ui" in data, (
|
||||
f"Response missing 'is_legacy_manager_ui' field: {data}"
|
||||
)
|
||||
assert data["is_legacy_manager_ui"] is False, (
|
||||
f"E2E env omits --enable-manager-legacy-ui; expected False, "
|
||||
f"got {data['is_legacy_manager_ui']!r}. If E2E launcher changed, update assertion."
|
||||
)
|
||||
```
|
||||
|
||||
**Optional companion test (true-path coverage, currently out of scope):** A parametrized variant that restarts ComfyUI with `--enable-manager-legacy-ui` and asserts `is True`. Not recommended for Cluster G — server restart doubles suite runtime and the legacy path is already exercised by playwright `legacy-ui-*.spec.ts` tests.
|
||||
|
||||
### (iv) Recommendation
|
||||
|
||||
- Upgrade `isinstance(bool)` → `is False` as above. ADEQUATE (positive-path + field + exact value).
|
||||
- Document the launcher dependency in a comment (already in the snippet).
|
||||
- If the E2E launcher ever passes `--enable-manager-legacy-ui`, the assertion fails loudly with a clear message — correct behavior.
|
||||
|
||||
---
|
||||
|
||||
## Summary Table
|
||||
|
||||
| Target | Current test | Upgrade path | Complexity | E2E-debt? |
|
||||
|---|---|---|---|---|
|
||||
| T1 imported_mode (`test_installed_imported_mode`) | dict-type only (WEAK) | Schema + seed + cross-mode keyset equality (ADEQUATE) | LOW | Yes — frozen-after-install invariant skipped (Strategy B) |
|
||||
| T2 boolean flag (`test_returns_boolean_field`) | `isinstance(bool)` (WEAK) | `is False` with launcher-deterministic comment (ADEQUATE) | LOW | No |
|
||||
|
||||
## Constraints / Limitations
|
||||
|
||||
- Research performed as Explore agent (read-only). No tests executed, no code modified.
|
||||
- `comfy.cli_args` is upstream (ComfyUI), not in manager repo — flag default verified via usage pattern (store_true action) and the `if args.enable_manager_legacy_ui:` truthiness check at `__init__.py:19`, which would crash with `TypeError` on `None` truthiness on integer comparisons but works on falsy-default bool.
|
||||
- Target 2 CORRECTION: dispatch referenced `/v2/manager/version` but the target test actually hits `/v2/manager/is_legacy_manager_ui` — verified via source inspection of test class.
|
||||
|
||||
## Grep/Read evidence index
|
||||
|
||||
| # | Command | Finding |
|
||||
|---|---|---|
|
||||
| 1 | `Grep pattern=/customnode/installed path=glob/manager_server.py` | L1510 snapshot init, L1513-1520 handler |
|
||||
| 2 | `Read tests/e2e/test_e2e_customnode_info.py` | L224-237 current WEAK test |
|
||||
| 3 | `Grep pattern=is_legacy_manager_ui path=comfyui_manager` | L1500-1506 glob handler, L995-1001 legacy handler |
|
||||
| 4 | `Grep pattern=enable-manager-legacy-ui path=tests/e2e` | 0 matches — flag not passed in E2E |
|
||||
| 5 | `Read tests/e2e/scripts/start_comfyui.sh` | L73-79 launch command (no legacy flag) |
|
||||
| 6 | `Read comfyui_manager/__init__.py` | L19 uses flag as truthy gate |
|
||||
| 7 | `Read glob/manager_core.py:1599-1632` | `get_installed_node_packs()` live filesystem scan |
|
||||
@@ -0,0 +1,514 @@
|
||||
# Scenario × Functional Effect Mapping
|
||||
|
||||
**Generated**: 2026-04-18
|
||||
**Definition of "effect"**: The actual **functional purpose** of the feature — not just any side effect. A scenario is verified only when the intended outcome is observably achieved.
|
||||
|
||||
| Pattern | Effect definition |
|
||||
|---|---|
|
||||
| Success scenario | The feature's PURPOSE is fulfilled and observable |
|
||||
| Validation/security error | The purpose is NOT fulfilled + correct rejection signal |
|
||||
| State edge case | The purpose is correctly short-circuited or no-op |
|
||||
|
||||
Unless specified, status code alone is NOT sufficient evidence of effect.
|
||||
|
||||
---
|
||||
|
||||
# Section 1 — Glob v2 Endpoints
|
||||
|
||||
## 1.1 Queue Management (Install/Uninstall/Update/Fix/Disable/Enable/Model)
|
||||
|
||||
### POST /v2/manager/queue/task (kind=install)
|
||||
|
||||
Purpose: **install a custom node pack so it becomes loadable by ComfyUI**.
|
||||
|
||||
| Scenario | Functional effect to verify |
|
||||
|---|---|
|
||||
| Success (CNR pack) | (a) pack directory exists under `custom_nodes/`, (b) `.tracking` file present (CNR marker), (c) pack appears in GET `customnode/installed` with correct cnr_id + version, (d) worker `task_worker_lock` released after completion |
|
||||
| Success (nightly/URL) | (a) pack directory exists, (b) `.git` subdir present (git clone), (c) repo remote matches requested URL, (d) appears in installed list |
|
||||
| Success (skip_post_install + already disabled) | Pack moved from `.disabled/` back to active (enable shortcut), NOT a fresh install |
|
||||
| Validation error (bad `kind` value) | Task NOT queued (queue/status unchanged), queue/history does not contain this ui_id, pack NOT installed |
|
||||
| Validation error (missing ui_id/client_id) | Same: no queued task, no installation side-effect |
|
||||
| Worker auto-start | After task queued, `queue/status.is_processing=true` and eventually `done_count` increments |
|
||||
|
||||
### POST /v2/manager/queue/task (kind=uninstall)
|
||||
|
||||
Purpose: **remove an installed pack so it is no longer loaded**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | Pack directory no longer exists under `custom_nodes/`, pack absent from `customnode/installed`, no import error on next ComfyUI reload |
|
||||
| Target not installed | No-op or error — purpose already satisfied; no state change |
|
||||
| Unknown pack | No filesystem change |
|
||||
|
||||
### POST /v2/manager/queue/task (kind=update)
|
||||
|
||||
Purpose: **update an installed pack to a newer version**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | (a) pack directory still exists, (b) version actually changed (check `.tracking` content or pyproject version), (c) dependencies refreshed, (d) still loadable by ComfyUI |
|
||||
| Already up-to-date | No-op or confirmatory response; no downgrade |
|
||||
| Unknown pack / Update fails | No partial state (pack not removed nor corrupted) |
|
||||
|
||||
### POST /v2/manager/queue/task (kind=fix)
|
||||
|
||||
Purpose: **re-install dependencies of an existing pack without changing source**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | (a) pack directory unchanged (same HEAD/version), (b) dependencies present in venv after fix, (c) pack import succeeds on reload |
|
||||
| Missing dependencies pre-fix | After fix, imports succeed |
|
||||
|
||||
### POST /v2/manager/queue/task (kind=disable)
|
||||
|
||||
Purpose: **stop loading a pack without removing it, reversibly**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | (a) pack moved from `custom_nodes/<name>/` to `custom_nodes/.disabled/<name>/`, (b) on next ComfyUI reload, pack nodes NOT registered, (c) pack absent from `customnode/installed` (active) |
|
||||
| Already disabled | No-op; still in `.disabled/` |
|
||||
|
||||
### POST /v2/manager/queue/task (kind=enable)
|
||||
|
||||
Purpose: **restore a disabled pack to active, loadable state**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | (a) pack restored from `.disabled/` to active `custom_nodes/` (may be case-normalized CNR name), (b) on reload, nodes registered again, (c) appears in `customnode/installed` |
|
||||
| Not disabled (already active) | No-op; no regression |
|
||||
|
||||
### POST /v2/manager/queue/install_model
|
||||
|
||||
Purpose: **download a model file to the appropriate models/ subdirectory**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | (a) task queued (queue/status reflects), (b) eventually file downloaded to `models/<type>/<filename>`, (c) file size > 0, (d) visible via `externalmodel/getlist` with `installed=True` (legacy) |
|
||||
| Missing client_id/ui_id | Task NOT queued; no download attempted |
|
||||
| Invalid metadata | Task NOT queued |
|
||||
| Not in whitelist (legacy check) | Download rejected; no file written |
|
||||
| Non-safetensors + security<high+ | Rejected; no file written |
|
||||
|
||||
### POST /v2/manager/queue/update_all
|
||||
|
||||
Purpose: **queue update tasks for ALL currently active packs**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | queue/status.pending_count == N where N = (active_nodes + unknown_active_nodes - manager-skip). Each queued task has correct `kind=update` + correct `node_name` |
|
||||
| Security denied (<middle+) | 403; NO tasks queued; queue/status unchanged |
|
||||
| Missing params | 400; NO tasks queued |
|
||||
| mode=local | No remote fetch; uses local channel data |
|
||||
| Desktop build | `comfyui-manager` pack NOT in queued tasks |
|
||||
|
||||
### POST /v2/manager/queue/update_comfyui
|
||||
|
||||
Purpose: **queue a self-update task for ComfyUI core**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | (a) queue/status.total_count increased by 1, (b) the queued task has `kind=update-comfyui` with `params.is_stable` matching request/config |
|
||||
| Missing params | 400; no task queued |
|
||||
| stable=true overrides config | Task params.is_stable==True regardless of config policy |
|
||||
|
||||
### POST /v2/manager/queue/reset
|
||||
|
||||
Purpose: **clear all queued/running/history tasks**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | queue/status: total_count=0, done_count=0, pending_count=0, in_progress_count=0, is_processing=false |
|
||||
| Already empty | Same; idempotent |
|
||||
|
||||
### POST /v2/manager/queue/start
|
||||
|
||||
Purpose: **start the worker thread to process queued tasks**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Worker not running | queue/status.is_processing becomes true (may be momentary if queue empty); tasks transition pending → running → done |
|
||||
| Already running | 201; is_processing remains true; no duplicate worker spawned |
|
||||
| Empty queue | Worker starts and idles; no errors |
|
||||
|
||||
### GET /v2/manager/queue/status
|
||||
|
||||
Purpose: **accurately reflect the current queue state**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| No filter | Counts match actual internal queue state (cross-check via known queued tasks) |
|
||||
| With client_id filter | client_id echo + filtered counts correspond to only that client's tasks |
|
||||
| Fields shape | total/done/in_progress/pending/is_processing all present + correct types |
|
||||
|
||||
### GET /v2/manager/queue/history
|
||||
|
||||
Purpose: **retrieve completed task records for introspection**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| id=<batch_id> query | Returns JSON content of that batch file (not another's) |
|
||||
| Path traversal | file read DOES NOT occur; returns 400 |
|
||||
| ui_id filter | Returns the matching single task record |
|
||||
| client_id filter | Returns only that client's history |
|
||||
| Pagination | Result size ≤ max_items |
|
||||
| Serialization limitation | If 400 returned, server didn't crash; no corrupted state |
|
||||
|
||||
### GET /v2/manager/queue/history_list
|
||||
|
||||
Purpose: **list available batch history file IDs**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | Returned `ids` ⊆ files in `manager_batch_history_path` (mtime-desc sorted) |
|
||||
| Empty | ids=[] reflects empty dir |
|
||||
|
||||
## 1.2 Custom Node Info
|
||||
|
||||
### GET /v2/customnode/getmappings
|
||||
|
||||
Purpose: **provide node→pack mapping for the UI to resolve missing nodes**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success mode=local/cache/remote | Returned dict: values are `[node_list, metadata]`, all currently-loaded `NODE_CLASS_MAPPINGS` either present in a node_list OR matched by `nodename_pattern` regex |
|
||||
| mode=nickname | Nicknames filter applied (each entry has nickname field) |
|
||||
| Missing mode query | 500/KeyError; no partial data returned |
|
||||
|
||||
### GET /v2/customnode/fetch_updates (deprecated)
|
||||
|
||||
Purpose: **(deprecated) was previously used to fetch git updates**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Always | 410 + `{deprecated: true}`. No git fetch performed (no disk I/O on .git dirs) |
|
||||
|
||||
### GET /v2/customnode/installed
|
||||
|
||||
Purpose: **list currently-installed packs with metadata for the UI**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| mode=default | Dict reflects real filesystem scan of `custom_nodes/`: every dir with proper marker appears |
|
||||
| mode=imported | Returns snapshot frozen at startup (unchanged after runtime installs) — proves stability |
|
||||
| Newly installed pack | After install, default mode reflects it; imported mode does NOT |
|
||||
|
||||
### POST /v2/customnode/import_fail_info
|
||||
|
||||
Purpose: **return detailed traceback/message for a pack that failed to import at startup**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Known failed pack via cnr_id | 200 + body has `msg` + `traceback` matching `cm_global.error_dict[module]` |
|
||||
| Known failed via url | Same |
|
||||
| Unknown pack | 400 (no info); `error_dict` NOT mutated |
|
||||
| Missing fields / non-dict | 400 with appropriate text |
|
||||
|
||||
### POST /v2/customnode/import_fail_info_bulk
|
||||
|
||||
Purpose: **same as above but for multiple packs in one call**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| cnr_ids list | Each key maps to either {error, traceback} (if failed) or null (if no failure). Unknown cnr_ids → null |
|
||||
| urls list | Same semantics |
|
||||
| Empty lists | 400 |
|
||||
| Mixed types inside list | 400 or skip with per-item error |
|
||||
|
||||
## 1.3 Snapshots
|
||||
|
||||
### GET /v2/snapshot/get_current
|
||||
|
||||
Purpose: **capture and return the current system state (not persist it)**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | Returned dict contains `comfyui` (hash/tag), `git_custom_nodes` (list), `cnr_custom_nodes` (list), `pips`. Consistent with actual installed state |
|
||||
|
||||
### POST /v2/snapshot/save
|
||||
|
||||
Purpose: **persist current system state so it can be restored later**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | (a) new file created in `manager_snapshot_path` with timestamped name, (b) file content == get_current() at save time, (c) appears in `snapshot/getlist.items` |
|
||||
|
||||
### GET /v2/snapshot/getlist
|
||||
|
||||
Purpose: **list saved snapshots for UI selection**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | items list matches .json files in snapshot dir (without extension), sorted desc |
|
||||
| After save | New snapshot name appears at top |
|
||||
| After remove | Removed name absent |
|
||||
|
||||
### POST /v2/snapshot/remove
|
||||
|
||||
Purpose: **delete a saved snapshot permanently**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | File removed from disk; absent from getlist |
|
||||
| Nonexistent target | No change; 200 (no-op) |
|
||||
| Path traversal | File NOT removed; 400; any other files untouched |
|
||||
| Security denied | File NOT removed; 403 |
|
||||
|
||||
### POST /v2/snapshot/restore
|
||||
|
||||
Purpose: **schedule a snapshot to be applied on next server restart** (the actual restore happens at startup).
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | `restore-snapshot.json` copied to `manager_startup_script_path`. Next reboot → actual state reverts to snapshot (verifiable by reboot + get_current comparison) |
|
||||
| Nonexistent target | No marker file created; 400 |
|
||||
| Path traversal | No file operations; 400 |
|
||||
| Security denied | No marker file; 403 |
|
||||
|
||||
## 1.4 Configuration
|
||||
|
||||
### GET /v2/manager/db_mode
|
||||
|
||||
Purpose: **return current DB source mode config**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | Returned text == `core.get_config()["db_mode"]` value in `config.ini` |
|
||||
|
||||
### POST /v2/manager/db_mode
|
||||
|
||||
Purpose: **persist new DB mode to config.ini**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Valid value | (a) config.ini written to disk with new value, (b) GET returns new value, (c) survives process restart |
|
||||
| Malformed JSON / missing value | 400; config.ini UNCHANGED |
|
||||
|
||||
### GET/POST /v2/manager/policy/update
|
||||
|
||||
Purpose: **read/persist update policy (stable vs nightly)**.
|
||||
|
||||
Same verification pattern as db_mode but for `update_policy` key.
|
||||
|
||||
### GET /v2/manager/channel_url_list
|
||||
|
||||
Purpose: **return available channels + currently selected**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | `selected` matches channel whose URL == config.channel_url (else "custom"); `list` is all known channels as "name::url" |
|
||||
|
||||
### POST /v2/manager/channel_url_list
|
||||
|
||||
Purpose: **switch active channel by name**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Known name | config.channel_url written with new URL; GET.selected matches new name |
|
||||
| Unknown name | Silent no-op; 200; channel_url UNCHANGED (verify) |
|
||||
| Malformed | 400; channel_url UNCHANGED |
|
||||
|
||||
## 1.5 System
|
||||
|
||||
### GET /v2/manager/is_legacy_manager_ui
|
||||
|
||||
Purpose: **let UI know which Manager UI (legacy vs current) to load**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | `is_legacy_manager_ui` matches the CLI flag `--enable-manager-legacy-ui` that was passed |
|
||||
|
||||
### GET /v2/manager/version
|
||||
|
||||
Purpose: **report the Manager package version**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | Text == core.version_str (non-empty, semver-ish) |
|
||||
| Idempotent | Consecutive calls return identical value |
|
||||
|
||||
### POST /v2/manager/reboot
|
||||
|
||||
Purpose: **restart the server process**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | (a) server process actually exits, (b) new process binds same port, (c) new process serves requests, (d) pre-reboot state preserved (version, config) |
|
||||
| CLI session mode | `.reboot` marker file created before exit(0); process-manager restarts |
|
||||
| Security denied | 403; process continues (no restart) |
|
||||
|
||||
### GET /v2/comfyui_manager/comfyui_versions
|
||||
|
||||
Purpose: **enumerate available ComfyUI versions + current**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | `current` is a git tag/hash present in `.git` log; `versions` array non-empty; current ∈ versions |
|
||||
| Git failure | 400; no partial response |
|
||||
|
||||
### POST /v2/comfyui_manager/comfyui_switch_version
|
||||
|
||||
Purpose: **queue a task to switch ComfyUI to a target version (actual switch happens via worker)**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | (a) task queued with `params.target_version=<ver>`, (b) queue/status reflects, (c) eventually `.git` HEAD points at target commit/tag after worker runs |
|
||||
| Missing params | 400; no task queued |
|
||||
| Security denied (<high+) | 403; no task queued |
|
||||
|
||||
---
|
||||
|
||||
# Section 2 — Legacy-only Endpoints (UI → effect)
|
||||
|
||||
For these, the functional purpose is triggered by UI interaction. The effect MUST be observable both through the UI (state transitions, renders) AND/OR through the backend (filesystem, queue state).
|
||||
|
||||
### POST /v2/manager/queue/batch (legacy)
|
||||
|
||||
Purpose: **accept one aggregated request to enqueue multiple operations, then start the worker**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| install item(s) | Each pack installed (filesystem effect); `failed` list only contains actually-failed ids |
|
||||
| uninstall item(s) | Each pack removed |
|
||||
| update item(s) | Packs updated (version change verifiable) |
|
||||
| reinstall item(s) | Pack removed then re-installed (dir exists, .tracking present) |
|
||||
| disable | Pack in `.disabled/` |
|
||||
| install_model | Model file downloaded |
|
||||
| fix | Dependencies re-resolved |
|
||||
| update_comfyui | ComfyUI update task queued |
|
||||
| update_all | All active pack updates queued |
|
||||
| Mixed kinds | Each kind's effect achieved; `failed` contains only real failures |
|
||||
|
||||
### GET /v2/customnode/getlist (legacy)
|
||||
|
||||
Purpose: **feed the Custom Nodes Manager dialog with the list of available + installed packs**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | Response has `channel` + `node_packs`; each pack includes install state (installed/disabled), stars (github-stats), update availability (if skip_update=false) |
|
||||
| skip_update=true | No git fetch performed (check timing / no remote calls) |
|
||||
| Channel resolution | Maps URL back to name (default/custom/etc.) |
|
||||
|
||||
### GET /customnode/alternatives (legacy)
|
||||
|
||||
Purpose: **show alternative pack recommendations for a given pack**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | Response dict keyed by unified pack id; values from `alter-list.json` with markdown processed |
|
||||
|
||||
### GET /v2/externalmodel/getlist (legacy)
|
||||
|
||||
Purpose: **list available external models with install state for Model Manager dialog**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | Each model entry has `installed` ∈ {'True','False'}; True ⟺ file actually exists under appropriate models subdir |
|
||||
| HuggingFace sentinel | Filename resolved from URL basename; installed flag correct |
|
||||
| Custom save_path | Path resolved correctly |
|
||||
|
||||
### GET /v2/customnode/versions/{node_name} (legacy)
|
||||
|
||||
Purpose: **list all versions of a CNR pack for the user to pick**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Known CNR pack | Response array lists all available versions (latest first, typically) matches CNR registry |
|
||||
| Unknown pack | 400; no partial data |
|
||||
|
||||
### GET /v2/customnode/disabled_versions/{node_name} (legacy)
|
||||
|
||||
Purpose: **list versions of a pack currently in the disabled state for possible re-enable**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Has disabled versions | Response array matches actual `cnr_inactive_nodes[node]` keys + "nightly" if in `nightly_inactive_nodes` |
|
||||
| None disabled | 400 |
|
||||
|
||||
### POST /v2/customnode/install/git_url (legacy)
|
||||
|
||||
Purpose: **clone a pack from arbitrary git URL (dangerous; requires high+)**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | Repo cloned into `custom_nodes/`, `.git` dir present, repo remote matches URL |
|
||||
| Already installed | 200 skip; no duplicate; no overwrite |
|
||||
| Clone failure | 400; no partial dir left behind |
|
||||
| Security denied (<high+) | 403; no filesystem change |
|
||||
|
||||
### POST /v2/customnode/install/pip (legacy)
|
||||
|
||||
Purpose: **run `pip install <packages>` in the venv**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| Success | Packages are importable from the venv Python afterwards (or `pip list` shows them) |
|
||||
| Security denied (<high+) | 403; no pip invocation |
|
||||
|
||||
### GET /v2/manager/notice (legacy)
|
||||
|
||||
Purpose: **fetch the News wiki content and augment with version footer**.
|
||||
|
||||
| Scenario | Effect to verify |
|
||||
|---|---|
|
||||
| GitHub reachable | HTML returned; contains markdown-body content + ComfyUI/Manager version footer appended |
|
||||
| GitHub unreachable | "Unable to retrieve Notice"; no crash |
|
||||
| Non-git ComfyUI | Response starts with "Your ComfyUI isn't git repo" warning |
|
||||
| Outdated ComfyUI | Response starts with "too OUTDATED!!!" warning |
|
||||
| Desktop variant | Footer uses `__COMFYUI_DESKTOP_VERSION__` instead of commit hash |
|
||||
|
||||
---
|
||||
|
||||
# Section 3 — UI→effect Mapping (Legacy)
|
||||
|
||||
For Playwright tests, the "UI→effect" contract requires:
|
||||
|
||||
| UI action | Target endpoint | Effect to verify |
|
||||
|---|---|---|
|
||||
| Click Manager menu button | (none — UI only) | `#cm-manager-dialog` visible |
|
||||
| Click "Custom Nodes Manager" menu item | GET customnode/getlist + getmappings | `#cn-manager-dialog` + grid populated (rows > 0) |
|
||||
| Click "Model Manager" menu item | GET externalmodel/getlist | `#cmm-manager-dialog` + grid populated |
|
||||
| Click "Snapshot Manager" menu item | GET snapshot/getlist | `#snapshot-manager-dialog` + list populated |
|
||||
| Click "Install" on a pack row | GET customnode/versions/{id} → POST queue/batch (install) → WebSocket cm-queue-status | Pack dir exists on disk + row shows "Installed" state in UI + WebSocket `all-done` received |
|
||||
| Click "Uninstall" on installed row | POST queue/batch (uninstall) | Pack dir removed + row state updates to "Not Installed" |
|
||||
| Click "Disable" on row | POST queue/batch (disable) | Pack in `.disabled/` + row state "Disabled" |
|
||||
| Click "Update" on outdated row | POST queue/batch (update) | Pack version changes + row state update |
|
||||
| Click "Fix" on row | POST queue/batch (fix) | Dependencies restored |
|
||||
| Click "Try alternatives" | GET /customnode/alternatives | Alternatives list rendered |
|
||||
| Open "Versions" dropdown on row | GET customnode/versions/{id} | Version list rendered in UI |
|
||||
| Open "Disabled Versions" on row | GET customnode/disabled_versions/{id} | Disabled versions rendered |
|
||||
| Click "Install via Git URL" button + enter URL + confirm | POST customnode/install/git_url | Pack cloned; dir visible in UI |
|
||||
| Click "Install via pip" | POST customnode/install/pip | Package installed; no UI crash |
|
||||
| Click "Install" on Model Manager row | POST queue/install_model | Model file downloaded; row state "Installed" |
|
||||
| Change DB mode dropdown | POST db_mode | Config persisted; dropdown value persists after dialog reopen |
|
||||
| Change Update Policy dropdown | POST policy/update | Same |
|
||||
| Change Channel dropdown | POST channel_url_list | Same |
|
||||
| Click "Update All" button | POST queue/update_all | Multiple tasks queued; progress indicator shows count |
|
||||
| Click "Update ComfyUI" button | POST queue/update_comfyui | Task queued; status indicator |
|
||||
| Click "Save Snapshot" in Snapshot Manager | POST snapshot/save | New row in dialog list with timestamp |
|
||||
| Click "Remove" on snapshot row | POST snapshot/remove?target=X | Row disappears from list |
|
||||
| Click "Restore" on snapshot row | POST snapshot/restore?target=X | Marker file created; next reboot applies |
|
||||
| Click "Restart" button | POST manager/reboot | Server restarts; UI reconnects |
|
||||
| Open Manager menu with pending News | GET manager/notice | News panel visible with HTML content |
|
||||
| Filter/search in grid | (client-side) | Row count ≤ initial count |
|
||||
| Close dialog (X button / Esc) | (none) | Dialog hidden; no leaked DOM |
|
||||
|
||||
---
|
||||
|
||||
# Section 4 — Effects Not Easily Observable
|
||||
|
||||
Some purposes can only be proven via side-channel observation:
|
||||
|
||||
| Endpoint | Purpose | Why hard to verify |
|
||||
|---|---|---|
|
||||
| POST snapshot/restore | Apply snapshot at next reboot | Must actually reboot + compare post-state; destructive |
|
||||
| POST switch_version (positive) | Change ComfyUI version | Destructive; needs rollback |
|
||||
| POST manager/reboot | Restart process | Hard to assert "new process" vs "same process" cleanly; proxy: pid change or connection drop+rebind |
|
||||
| POST queue/start → worker runs | Tasks execute | Timing-dependent; must poll done_count |
|
||||
| GET manager/notice | Content from GitHub | External dependency; flaky |
|
||||
| POST install (network) | Actually installs | Depends on CNR/GitHub availability |
|
||||
| POST install_model (download) | File downloaded | Slow; large files; fake whitelist URL returns quick 404 |
|
||||
|
||||
For these, tests either (a) accept destructive as out-of-scope, (b) use timing/polling, or (c) mock at minimum granularity.
|
||||
|
||||
---
|
||||
*End of Scenario × Effect Mapping*
|
||||
@@ -0,0 +1,424 @@
|
||||
# Scenario Intent Mapping
|
||||
|
||||
**Generated**: 2026-04-18
|
||||
**Definition of "intent"**: For each scenario — **what real use case, user need, or protection concern does this scenario represent?** Answers "why does this scenario matter, what is it there to prove?"
|
||||
|
||||
Intent categories used:
|
||||
- **User capability** — the user wants to accomplish task X
|
||||
- **Data integrity** — the system must not corrupt state
|
||||
- **Security boundary** — privilege / access must be enforced
|
||||
- **Input resilience** — bad input must not crash or mis-operate
|
||||
- **Idempotency** — operation can be retried safely
|
||||
- **Observability** — the caller needs accurate state visibility
|
||||
- **Concurrency safety** — parallel calls don't interfere
|
||||
- **Recovery** — system can recover from failure / bad state
|
||||
|
||||
---
|
||||
|
||||
# Section 1 — Glob v2 Endpoints
|
||||
|
||||
## 1.1 Queue Management
|
||||
|
||||
### POST /v2/manager/queue/task (install)
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success (CNR) | User capability: install a registered pack at a specific version for reproducibility |
|
||||
| Success (nightly/URL) | User capability: install unreleased or private pack from arbitrary git URL |
|
||||
| Success (skip_post_install + already disabled) | Recovery: re-enable a previously disabled pack without full reinstall (optimization path) |
|
||||
| Validation error (bad kind) | Input resilience: prevent arbitrary op execution via malformed kind; ensure schema gate is the truth |
|
||||
| Validation error (missing ui_id/client_id) | Observability: every queued task must be traceable back to its originator |
|
||||
| Invalid JSON body | Input resilience: malformed bytes don't crash the server |
|
||||
| Worker auto-start | User capability: ease of use — installer doesn't need separate "start" call (legacy path does though) |
|
||||
|
||||
### POST /v2/manager/queue/task (uninstall)
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: remove a pack that's no longer needed or causing issues |
|
||||
| Target not installed | Idempotency: uninstall of non-present pack should not fail destructively |
|
||||
|
||||
### POST /v2/manager/queue/task (update)
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: upgrade to a newer release to get fixes/features |
|
||||
| Already up-to-date | Idempotency: safe to trigger update even when nothing new exists |
|
||||
| Update fails mid-way | Data integrity: don't leave pack in partially-updated state |
|
||||
|
||||
### POST /v2/manager/queue/task (fix)
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | Recovery: when dependencies drift or break, re-install them without re-cloning source |
|
||||
| Missing deps pre-fix | Recovery: fix should heal the environment |
|
||||
|
||||
### POST /v2/manager/queue/task (disable)
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: temporarily stop using a pack without losing it (reversible) |
|
||||
| Already disabled | Idempotency: re-disable is a no-op |
|
||||
|
||||
### POST /v2/manager/queue/task (enable)
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: restore a disabled pack to active use |
|
||||
| Not disabled | Idempotency: no-op when already active |
|
||||
|
||||
### POST /v2/manager/queue/install_model
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: download models from curated whitelist for model library |
|
||||
| Missing client_id/ui_id | Observability: every download is traceable |
|
||||
| Invalid metadata | Input resilience: malformed model requests rejected early |
|
||||
| Not in whitelist | Security boundary: prevent arbitrary URL downloads (supply-chain protection) |
|
||||
| Non-safetensors + lower security | Security boundary: block executable-format model files in lower-trust env |
|
||||
|
||||
### POST /v2/manager/queue/update_all
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: one-click update of all installed packs |
|
||||
| Security denied | Security boundary: bulk ops are more risky; require middle+ trust |
|
||||
| Missing params | Observability: must know who initiated bulk op |
|
||||
| mode=local | User capability: work offline using cached data |
|
||||
| Desktop build | Data integrity: don't self-update comfyui-manager in bundled builds |
|
||||
| Empty active set | Idempotency: safe to run on fresh install with nothing to update |
|
||||
|
||||
### POST /v2/manager/queue/update_comfyui
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: update ComfyUI core itself |
|
||||
| Missing params | Observability: traceability |
|
||||
| stable=true explicit | User capability: override policy for one-off stable update regardless of config |
|
||||
|
||||
### POST /v2/manager/queue/reset
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | Recovery: abort an in-progress batch; clear failed state |
|
||||
| Already empty | Idempotency: safe to call repeatedly as cleanup |
|
||||
|
||||
### POST /v2/manager/queue/start
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Worker not running | User capability: explicit trigger for the async worker |
|
||||
| Already running | Concurrency safety: don't spawn duplicate workers (data corruption risk) |
|
||||
| Empty queue | Idempotency: no error on empty queue |
|
||||
|
||||
### GET /v2/manager/queue/status
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| No filter | Observability: dashboard view of overall progress |
|
||||
| client_id filter | Observability: per-client progress for multi-user UI |
|
||||
| Unknown client_id | Input resilience: unknown id returns 0s, not error |
|
||||
|
||||
### GET /v2/manager/queue/history
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| id=<batch_id> | Observability: inspect an old batch for audit/debug |
|
||||
| Path traversal | Security boundary: prevent arbitrary file reads via history endpoint |
|
||||
| ui_id filter | Observability: detailed view for one task |
|
||||
| client_id filter | Observability: per-client history |
|
||||
| Pagination | Performance: avoid huge payload on long histories |
|
||||
| Serialization failure | Input resilience: fail cleanly (400) rather than crash |
|
||||
|
||||
### GET /v2/manager/queue/history_list
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | Observability: enumerate past batches |
|
||||
| Empty | Idempotency: no crash on empty history dir |
|
||||
| Path inaccessible | Input resilience: fail cleanly |
|
||||
|
||||
## 1.2 Custom Node Info
|
||||
|
||||
### GET /v2/customnode/getmappings
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success (mode=local/cache/remote) | User capability: UI resolves "missing nodes in workflow" to recommend packs |
|
||||
| mode=nickname | User capability: shorter display names for UI |
|
||||
| Missing mode | Input resilience: require explicit mode choice |
|
||||
|
||||
### GET /v2/customnode/fetch_updates (deprecated)
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Always 410 | API contract: signal clients to migrate to queue-based flow; don't silently break |
|
||||
|
||||
### GET /v2/customnode/installed
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| mode=default | Observability: current state for UI |
|
||||
| mode=imported | Observability: startup-time state for diff ("what changed since boot") |
|
||||
| Empty | Idempotency: no crash on empty install |
|
||||
|
||||
### POST /v2/customnode/import_fail_info
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Known failed pack | Recovery: show user exact traceback so they can decide fix vs report vs uninstall |
|
||||
| Unknown pack | Input resilience: 400 rather than empty success (distinguishable) |
|
||||
| Missing fields / non-dict | Input resilience: reject early |
|
||||
|
||||
### POST /v2/customnode/import_fail_info_bulk
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| cnr_ids list | Performance: batch lookup for dialog that shows multiple failed packs at once |
|
||||
| urls list | Same, for git-URL-installed packs |
|
||||
| Empty lists | Input resilience: require at least one query |
|
||||
| Null for unknown | Observability: distinguish "no failure info" from "lookup failed" |
|
||||
|
||||
## 1.3 Snapshots
|
||||
|
||||
### GET /v2/snapshot/get_current
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | Observability: inspect system state before taking a snapshot |
|
||||
| Failure | Input resilience: fail cleanly |
|
||||
|
||||
### POST /v2/snapshot/save
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: persist current state for later rollback |
|
||||
| Multiple saves | Observability: each save is independently retrievable |
|
||||
|
||||
### GET /v2/snapshot/getlist
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: choose which snapshot to restore/delete |
|
||||
| Empty | Idempotency: no crash on empty snapshot dir |
|
||||
|
||||
### POST /v2/snapshot/remove
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: housekeeping (remove old snapshots) |
|
||||
| Nonexistent target | Idempotency: re-delete should not error |
|
||||
| Path traversal | Security boundary: prevent deleting files outside snapshot dir |
|
||||
| Missing target | Input resilience |
|
||||
| Security denied | Security boundary: middle security required |
|
||||
|
||||
### POST /v2/snapshot/restore
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | Recovery: rollback to a known-good state after bad update |
|
||||
| Nonexistent | Input resilience |
|
||||
| Path traversal | Security boundary |
|
||||
| Security denied | Security boundary: middle+ required (restore is destructive) |
|
||||
|
||||
## 1.4 Configuration
|
||||
|
||||
### GET /v2/manager/db_mode
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | Observability: UI shows current mode setting |
|
||||
|
||||
### POST /v2/manager/db_mode
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Valid | User capability: switch between online/local DB for different network conditions |
|
||||
| Malformed | Input resilience |
|
||||
| Missing value | Input resilience: don't silently set unknown/empty |
|
||||
|
||||
### GET/POST /v2/manager/policy/update
|
||||
|
||||
Same as db_mode: observability of current policy + user choice to change update strategy (stable vs nightly) + input resilience.
|
||||
|
||||
### GET /v2/manager/channel_url_list
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | Observability: show available channels in UI dropdown |
|
||||
| "custom" selected | Input resilience: URL not in known list doesn't break display |
|
||||
|
||||
### POST /v2/manager/channel_url_list
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Known name | User capability: switch between upstream vs fork vs private channel |
|
||||
| Unknown name | Input resilience: silent no-op (don't crash on typo) |
|
||||
| Malformed | Input resilience |
|
||||
|
||||
## 1.5 System
|
||||
|
||||
### GET /v2/manager/is_legacy_manager_ui
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: frontend picks which UI variant to mount at page load |
|
||||
|
||||
### GET /v2/manager/version
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | Observability: display version in UI (troubleshooting / support) |
|
||||
| Idempotent | Data integrity: version doesn't change at runtime |
|
||||
|
||||
### POST /v2/manager/reboot
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: apply changes that require restart (snapshot restore, ComfyUI version switch) |
|
||||
| CLI session mode | Integration: cooperates with external process manager for clean restart |
|
||||
| Security denied | Security boundary: middle required (restart affects all users) |
|
||||
|
||||
### GET /v2/comfyui_manager/comfyui_versions
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: enumerate ComfyUI versions to pick one for rollback/upgrade |
|
||||
| Git failure | Input resilience: fail cleanly if ComfyUI isn't a git repo |
|
||||
|
||||
### POST /v2/comfyui_manager/comfyui_switch_version
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: switch ComfyUI to specific version (pin for reproducibility) |
|
||||
| Missing params | Observability |
|
||||
| Security denied | Security boundary: high+ required (massive blast radius — affects core behavior) |
|
||||
|
||||
---
|
||||
|
||||
# Section 2 — Legacy-only Endpoints
|
||||
|
||||
### POST /v2/manager/queue/batch
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Single-kind batch | User capability: execute multiple operations of same type in one round-trip |
|
||||
| Mixed-kind batch | User capability: apply a workflow (uninstall-then-install = reinstall) atomically |
|
||||
| Partial failure (`failed` list) | Observability: distinguish which packs in the batch failed from ones that succeeded |
|
||||
| Empty body | Idempotency: no-op if nothing to do |
|
||||
| update_all sub-key | User capability: trigger bulk update as part of batch |
|
||||
|
||||
### GET /v2/customnode/getlist
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: populate Custom Nodes Manager dialog with full available pack catalog |
|
||||
| skip_update=true | Performance: fast load when user doesn't need remote fetch |
|
||||
| Channel resolution | Observability: user sees which channel data came from |
|
||||
|
||||
### GET /customnode/alternatives
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: recommend alternative packs when one is discontinued/unavailable |
|
||||
|
||||
### GET /v2/externalmodel/getlist
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: browse curated model catalog |
|
||||
| `installed` flag per model | Observability: which models already present |
|
||||
| HuggingFace sentinel | User capability: HF-hosted models via standard URL |
|
||||
| Custom save_path | User capability: custom model placement |
|
||||
|
||||
### GET /v2/customnode/versions/{node_name}
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Known CNR | User capability: pick a specific version to install (stability over latest) |
|
||||
| Unknown pack | Input resilience |
|
||||
|
||||
### GET /v2/customnode/disabled_versions/{node_name}
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Has disabled | User capability: see what versions are available to re-enable without fresh install |
|
||||
| None | Input resilience |
|
||||
|
||||
### POST /v2/customnode/install/git_url
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: install arbitrary git pack (for advanced users / private packs) |
|
||||
| Already installed | Idempotency |
|
||||
| Clone failure | Input resilience: bad URL returns error; no corrupt state |
|
||||
| Security denied | Security boundary: high+ required (arbitrary code execution risk) |
|
||||
|
||||
### POST /v2/customnode/install/pip
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| Success | User capability: install pip packages needed by a pack |
|
||||
| Security denied | Security boundary: high+ required (arbitrary package execution risk) |
|
||||
|
||||
### GET /v2/manager/notice
|
||||
|
||||
| Scenario | Intent |
|
||||
|---|---|
|
||||
| GitHub reachable | User capability: see latest Manager news/changelog inline |
|
||||
| GitHub unreachable | Input resilience: don't block UI on external service failure |
|
||||
| Non-git ComfyUI | Observability: warn user that their install is non-standard |
|
||||
| Outdated ComfyUI | Observability: warn user they're too old to be safe |
|
||||
| Desktop variant | User capability: correct footer for desktop distribution |
|
||||
|
||||
---
|
||||
|
||||
# Section 3 — Cross-cutting Scenarios
|
||||
|
||||
Some scenarios recur across many endpoints with consistent intent:
|
||||
|
||||
| Scenario pattern | Applies to | Unified intent |
|
||||
|---|---|---|
|
||||
| Malformed JSON body | all POST endpoints accepting JSON | Input resilience — protect against corrupted bytes / wrong content-type |
|
||||
| Missing required field | all POST endpoints with schemas | Input resilience + Observability (traceability fields mandatory) |
|
||||
| Path traversal in target/id | snapshot/remove, snapshot/restore, queue/history | Security boundary — prevent arbitrary filesystem access |
|
||||
| Security level denial (middle/middle+/high+) | destructive endpoints | Security boundary — tier privileged ops per deployment risk profile |
|
||||
| Idempotent re-call on empty state | queue/reset, history_list, snapshot/getlist, installed | Idempotency — safe to poll or retry |
|
||||
| Repeated read returns same value | version, db_mode, policy/update | Data integrity — config/runtime state is stable |
|
||||
| Empty collection returned cleanly | history, getlist, installed, alternatives | Input resilience — empty is valid, not an error |
|
||||
|
||||
---
|
||||
|
||||
# Section 4 — Intent Coverage Summary
|
||||
|
||||
| Intent category | # scenarios | Notes |
|
||||
|---|---:|---|
|
||||
| User capability (positive user need) | 62 | The "happy paths" |
|
||||
| Input resilience | 32 | Mostly 400s for bad input |
|
||||
| Security boundary | 15 | Security levels + path traversal |
|
||||
| Idempotency | 14 | No-op / retry safety |
|
||||
| Observability | 16 | State visibility + traceability |
|
||||
| Data integrity | 8 | Config/state stability |
|
||||
| Recovery | 5 | Fix, restore, reset |
|
||||
| Concurrency safety | 2 | Worker dedup |
|
||||
|
||||
Total unique scenarios mapped: ~154 (matches Report A).
|
||||
|
||||
---
|
||||
|
||||
# Section 5 — Why This Mapping Matters
|
||||
|
||||
For each scenario, the **intent** drives the TEST design:
|
||||
- **User capability** scenarios need end-to-end effect verification (feature works as promised)
|
||||
- **Input resilience** scenarios need negative tests (bad inputs rejected cleanly)
|
||||
- **Security boundary** scenarios need permission gate tests (403 proven per security level)
|
||||
- **Idempotency** scenarios need repeat-call tests (no state drift)
|
||||
- **Observability** scenarios need response-correctness tests (UI can trust the data)
|
||||
- **Data integrity** scenarios need consistency tests (no runtime mutation of constants)
|
||||
- **Recovery** scenarios need fault-injection tests (broken state → fix heals it)
|
||||
- **Concurrency safety** scenarios need parallel-call tests (no duplicate workers/tasks)
|
||||
|
||||
Gaps in current E2E suite are best understood by intent: missing tests are typically for **security boundary** (403 gates), **input resilience edge cases** (path traversal, missing value keys), and **recovery** (fix/restore). These are the hardest to reach in simple E2E but matter most for production safety.
|
||||
|
||||
---
|
||||
*End of Scenario Intent Mapping*
|
||||
@@ -0,0 +1,177 @@
|
||||
# Test Bloat Inventory — Sweep Aggregate
|
||||
|
||||
**Generated**: 2026-04-20 (via `/pair-sweep test bloat identification`)
|
||||
**Scope**: 127 test functions across 21 files (14 pytest E2E + 7 Playwright specs)
|
||||
**Method**: Static analysis by 4-member team (4 chunks, 127 items, `cl-20260419-bloat-{teng,review,dev,dbg}`)
|
||||
**References**: `goal-report-bloat-sweep.md` (10 bloat code definitions B1-BA)
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
| Metric | Count | Rate |
|
||||
|---|---:|---:|
|
||||
| Total analyzed | **127** | 100% |
|
||||
| ✅ CLEAN | **94** | 74.0% |
|
||||
| ⚠️ BLOAT | **33** | 26.0% |
|
||||
| 🔴 Immediate remove/merge | **16** | 12.6% |
|
||||
| 🟡 Refactor / consolidate | **~10** | 7.9% |
|
||||
| 🟢 Borderline (retained with note) | **~7** | 5.5% |
|
||||
|
||||
**Post-action projection**: 127 → ~115 tests (−12 via remove/merge) with zero coverage loss. Bloat rate drops from 26% to ≤5%.
|
||||
|
||||
---
|
||||
|
||||
## Chunk Distribution
|
||||
|
||||
| Chunk | Member | Total | CLEAN | BLOAT | Top codes |
|
||||
|---|---|---:|---:|---:|---|
|
||||
| A (csrf/secgate/version) | dbg | 19 | 17 | 2 | B7, B9 |
|
||||
| B (endpoint/customnode/snapshot/git_clone) | reviewer | 29 | 23 | 6 | B1 (×4), B1/B5 (×1), B7 (×1) |
|
||||
| C (config_api/queue/task_ops/system) | teng | 45 | 30 | 15 | B1 (×5), B9 (×9), B8 (×1) |
|
||||
| D (uv_compile + Playwright) | dev | 34 | 24 | 10 | B9 (×5), B5 (×2), B1, B4+B5+BA, B8 |
|
||||
| **TOTAL** | — | **127** | **94** | **33** | B9 (primary) |
|
||||
|
||||
**Top bloat code**: **B9 Copy-paste** (14 occurrences across chunks) — largely from copy-paste test skeletons that can be parametrized.
|
||||
|
||||
---
|
||||
|
||||
## 🔴 Priority 1 — Immediate Remove (1 item)
|
||||
|
||||
| ID | File | Function | Reason | Verdict |
|
||||
|---|---|---|---|---|
|
||||
| dev:ci-013 | debug-install-flow.spec.ts | `capture install button API flow` | Zero `expect()` calls, only `console.log`. Diagnostic script committed as test — cannot fail. | B4+B5+BA |
|
||||
|
||||
**Action**: Delete the entire spec file OR move to `tools/` directory.
|
||||
|
||||
---
|
||||
|
||||
## 🔴 Priority 2 — Remove (subsumed by other tests) — 7 items
|
||||
|
||||
| ID | File | Function | Subsumed by | Code |
|
||||
|---|---|---|---|---|
|
||||
| reviewer:ci-004 | endpoint.py | (install_uninstall related)1 | ci-003 (WI-N strengthening adds API cross-check) | B1 |
|
||||
| reviewer:ci-005 | endpoint.py | install-uninstall-cycle | concat of ci-001+ci-002+ci-003 | B1 |
|
||||
| reviewer:ci-006 | endpoint.py | /system_stats smoke | fixture already polls until 200 | B5 |
|
||||
| reviewer:ci-009 | customnode_info.py | getmappings | subsumed by ci-008 first-5 schema (post-WI-M) | B1 |
|
||||
| teng:ci-005 | (config_api or queue) | strict subset of ci-002 disk check | ci-002 | B1 |
|
||||
| teng:ci-010 | subset of ci-007 + dup of ci-005 | ci-007 | B1 |
|
||||
| teng:ci-017 | weaker subset of ci-016 | ci-016 | B1 |
|
||||
| teng:ci-024 | subset of ci-016, misleading 'final' | ci-016 | B1, B8 |
|
||||
| teng:ci-028 | cycle covered by ci-026+ci-027 | ci-026+ci-027 | B1 |
|
||||
| dev:ci-010 | uv_compile.py | `test_uv_compile_conflict_attribution` | ci-012 (strict superset) | B1 |
|
||||
|
||||
**Action**: Delete these tests individually; confirm no unique assertion.
|
||||
|
||||
---
|
||||
|
||||
## 🟡 Priority 3 — Merge / Parametrize Clusters — 5 clusters (~12 tests → 4 tests)
|
||||
|
||||
### Cluster 1 — config_api roundtrip (3 → 1)
|
||||
`teng:ci-002, ci-007, ci-013` → `@pytest.mark.parametrize("endpoint,key,values", ...)`
|
||||
Estimated savings: 3 × ~60 lines → 1 parametrized test.
|
||||
|
||||
### Cluster 2 — config_api invalid-body (3 → 1)
|
||||
`teng:ci-003, ci-008, ci-015` → parametrize across `(endpoint, key)`.
|
||||
|
||||
### Cluster 3 — config_api junk-value (3 → 1)
|
||||
`teng:ci-004, ci-009, ci-014` → parametrize across `(endpoint, key, values)`.
|
||||
|
||||
### Cluster 4 — task_operations history (2 → 1)
|
||||
`teng:ci-030, ci-032` → parametrize across `(kind, ui_id)`.
|
||||
|
||||
### Cluster 5 — uv_compile verb (5 → 1)
|
||||
`dev:ci-004, ci-005, ci-006, ci-007, ci-011` → parametrize across verb (update/update_all/fix/fix_all/restore-dependencies).
|
||||
|
||||
### Cluster 6 — install_model missing-field (2 → 1)
|
||||
`teng:ci-034, ci-035` → parametrize across missing_field.
|
||||
|
||||
### Cluster 7 — version_mgmt response contract (4 → 1)
|
||||
`dbg:ci-013, ci-014, ci-015, ci-016` → merge into single `test_versions_response_contract`.
|
||||
|
||||
### Cluster 8 — snapshot: reviewer:ci-026 merge-with ci-022
|
||||
|
||||
**Total merge savings**: ~12 tests → 8 tests (−4 net).
|
||||
|
||||
---
|
||||
|
||||
## 🟡 Priority 4 — Refactor In-Place (3 items)
|
||||
|
||||
| ID | File | Function | Issue | Recommendation |
|
||||
|---|---|---|---|---|
|
||||
| dev:ci-003 | uv_compile.py | `test_reinstall_with_uv_compile` | OR-fallback masks "known issue — purge_node_state bug" | `@pytest.mark.xfail(reason=...)` OR split positive/already-exists |
|
||||
| dev:ci-008 | uv_compile.py | `test_uv_compile_no_packs` | `rc==0 OR "No custom node packs"` — OR-fallback | Split into 2 tests (empty tree rc==0 / non-empty rc==0 + substring) |
|
||||
| dev:ci-022 | manager-menu.spec.ts | `shows settings dropdowns (DB, Channel, Policy)` | Title promises 3 dropdowns, only asserts 2 | Add `channelCombo` assertion OR rename |
|
||||
| reviewer:ci-013 | customnode_info.py | (TODO stub) | L303 TODO makes test stub; skip mask hides incomplete impl | Resolve TODO or drop skip |
|
||||
| dbg:ci-012 | secgate_strict.py | `test_post_works_at_default_after_restore` | Entire body is `pytest.skip()` placeholder | **DELETE** function, preserve intent in module comment |
|
||||
| dbg:ci-018 | version_mgmt.py | `test_switch_version_missing_client_id` | Duplicates ci-017 (gate 403 before param validation) | Remove or parametrize with ci-017 |
|
||||
|
||||
---
|
||||
|
||||
## 🟢 Priority 5 — Borderline B9 Retained (intentional parallels) — 7 items
|
||||
|
||||
| ID | Reason for retention |
|
||||
|---|---|
|
||||
| dbg:ci-005-008 (csrf_legacy mirror csrf) | Different fixture → different SUT (legacy XOR glob mutex). Coverage necessary, not redundant. |
|
||||
| dev:ci-024 (Policy persist vs ci-023 DB) | Orthogonal target dropdowns; rollback paths differ. |
|
||||
| dev:ci-026 (model-manager open vs ci-014 custom-nodes open) | Different dialog id; structural-open verification per dialog is cheap. |
|
||||
| dev:ci-028 (model-manager search vs ci-017 custom-nodes search) | Different backend queries. |
|
||||
| dev:ci-032 (snapshot open vs ci-014/026) | Third dialog; each has distinct open-path pinning. |
|
||||
|
||||
---
|
||||
|
||||
## Key Findings / Patterns
|
||||
|
||||
1. **B9 Copy-paste dominates bloat** (~14 of 33 BLOAT items) — all concentrated in pytest uv_compile (5), config_api (9), version_mgmt (4). Parametrization fixes all.
|
||||
2. **Playwright >>> pytest for bloat rate**: Playwright 91% CLEAN vs pytest uv_compile 42% CLEAN. Playwright has `expect.poll` + `beforeEach` hoisting + state-based assertions. pytest uv_compile uses substring `in combined` as sole assertion in 5/12.
|
||||
3. **OR-fallback pattern** (2 tests: dev:ci-003, ci-008) masks which branch runs — AP-3-adjacent.
|
||||
4. **Intentional mutex parallels** (dbg chunk) kept as CLEAN — csrf.py + csrf_legacy.py test different SUT loaded via `__init__.py` mutex. Not redundant despite structural similarity.
|
||||
5. **`debug-install-flow.spec.ts`** is the single most egregious bloat — zero assertions, pure `console.log`. Not a test.
|
||||
|
||||
## Secondary Observations (non-bloat but flagged)
|
||||
|
||||
| Target | Observation | Scope |
|
||||
|---|---|---|
|
||||
| `test_e2e_uv_compile.py` | 12 CLI subprocess tests mis-filed under `tests/e2e/` (should be `tests/cli/`). Not B4 Dead — real functionality. | Relocation WI candidate |
|
||||
| `test_e2e_csrf.py` | Post-WI-HH correctly excludes 3 dual-purpose endpoints from STATE_CHANGING_POST_ENDPOINTS. | (already resolved) |
|
||||
| `test_e2e_secgate_strict.py::SR4 PoC` | Strongest negative-side check pattern (file-unchanged on disk). | Propagate pattern to SR6/V5/UA2 follow-ups |
|
||||
| `test_e2e_csrf_legacy.py` | 2 legacy-only endpoints (install/git_url, install/pip) per WI-JJ-B. | (already added) |
|
||||
|
||||
---
|
||||
|
||||
## Post-Action Projection
|
||||
|
||||
Applying 🔴 + 🟡:
|
||||
- 127 current tests
|
||||
- −1 remove (dev:ci-013 debug-install-flow)
|
||||
- −9 remove (subsumed tests, reviewer+teng+dev)
|
||||
- −4 merge/parametrize (7 clusters net savings: from 23 tests into 11 parametrized)
|
||||
|
||||
**Projected final count**: ~**113 tests** (−14, ~11% reduction) with zero coverage loss. Bloat rate target: ≤5%.
|
||||
|
||||
---
|
||||
|
||||
## Next Steps (follow-up WI candidates)
|
||||
|
||||
1. **WI-MM**: Apply 🔴 removals (1 remove + 9 subsumed + 1 delete PoC stub = 11 deletions) — low risk, high value
|
||||
2. **WI-NN**: Apply 🟡 parametrize clusters (5-7 clusters → significant line reduction)
|
||||
3. **WI-OO**: Apply 🟡 refactors (ci-003 xfail, ci-008 split, ci-022 rename/add, ci-013 TODO resolve, ci-018 merge/parametrize)
|
||||
4. **WI-PP (optional)**: Relocate `test_e2e_uv_compile.py` from `tests/e2e/` to `tests/cli/`
|
||||
|
||||
Each WI should update `reports/e2e_verification_audit.md` Summary Matrix + TOTAL (tests will decrease) and run `verify_audit_counts.py` PASS at completion.
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
- ✅ `cl-20260419-bloat-dbg`: 19/19 done
|
||||
- ✅ `cl-20260419-bloat-review`: 29/29 done
|
||||
- ✅ `cl-20260419-bloat-teng`: 45/45 done
|
||||
- ✅ `cl-20260419-bloat-dev`: 34/34 done
|
||||
- ✅ **Total**: 127/127 (100%)
|
||||
|
||||
Every item has verdict + evidence + recommendation in its respective checklist YAML.
|
||||
|
||||
---
|
||||
|
||||
*End of Test Bloat Inventory*
|
||||
@@ -0,0 +1,298 @@
|
||||
# Test Contract Audit
|
||||
|
||||
**Generated**: 2026-04-18
|
||||
**Contract**:
|
||||
- **Glob E2E** = endpoint call → effect verification (HTTP POST/GET + verify state change or response correctness)
|
||||
- **Legacy E2E** = UI interaction → effect verification (click/select/fill + verify state change)
|
||||
|
||||
Tests that call HTTP endpoints directly in the Playwright suite VIOLATE the legacy contract. Tests that check only status code without verifying effect VIOLATE the glob contract.
|
||||
|
||||
## Summary
|
||||
|
||||
| Contract violation | Count | Severity |
|
||||
|---|---:|---|
|
||||
| Playwright tests using direct API (bypass UI) | ~~9~~ → 5 | 🔴 contract breach (4 resolved in Stage2 WI-F; remaining 5 are in legacy-ui-snapshot helper functions + other files — see updated Section 1) |
|
||||
| Playwright tests partially UI-driven (mixed) | 2 | 🟡 weakened |
|
||||
| Glob tests missing effect verification | ~~1~~ → 0 | 🟡 status-only (resolved in Stage2 WI-D) |
|
||||
| Glob tests fully effect-verifying | 80 | ✓ compliant |
|
||||
| Security-contract tests (CSRF method-reject — 8 functions / 52 invocations — 26 glob + 26 legacy) | 8 | ✓ compliant (separate negative contract; glob + legacy server coverage) |
|
||||
|
||||
---
|
||||
|
||||
# Section 1 — Playwright Contract Audit (Legacy UI → Effect)
|
||||
|
||||
## ✅ VIOLATIONS — all 4 listed tests resolved in Stage2 WI-F
|
||||
|
||||
Historical record (2026-04-18, Stage2 WI-F). All 4 direct-API Playwright tests that previously violated the legacy contract have been removed from the suite or rewritten to click real UI elements:
|
||||
|
||||
| File | Test | Original violation | Resolution |
|
||||
|---|---|---|---|
|
||||
| legacy-ui-snapshot.spec.ts | `lists existing snapshots` | Direct `page.request.get('/v2/snapshot/getlist')` — no UI click | **DELETED**; backend `getlist` coverage owned by pytest `test_e2e_snapshot_lifecycle.py::test_getlist_after_save` (12/12 pytest regression PASS). |
|
||||
| legacy-ui-snapshot.spec.ts | `save snapshot via API and verify in list` | 100% `page.request.post/get` — zero UI interaction | **REWRITTEN** as `SS1 Save button creates a new snapshot row`: clicks dialog Save/Create button, polls `getlist` only as backend-effect confirmation helper (not as the test's primary action), cleans up via afterEach. Bonus: `UI Remove button deletes a snapshot row` added for row-delete UI coverage. |
|
||||
| legacy-ui-navigation.spec.ts | `API health check while dialogs are open` | `page.request.get('/v2/manager/version')` — direct API, not UI | **DELETED**; version coverage owned by `test_e2e_system_info.py::test_version_returns_string/test_version_is_stable`. |
|
||||
| legacy-ui-navigation.spec.ts | `system endpoints accessible from browser context` | 2× `page.request.get` — direct API | **DELETED**; fully redundant with `test_e2e_system_info.py` suite. |
|
||||
|
||||
**Verification**: `npx playwright test --list --grep '<any of the 4 titles>'` → `Total: 0 tests in 0 files`. Current spec listing: 5 tests in 2 files, all UI-driven.
|
||||
|
||||
**Residual note**: The snapshot spec still uses `page.request.get('/v2/snapshot/getlist')` inside the `getSnapshotNames` helper and in the `beforeEach/afterEach` for deterministic seeding/cleanup. This is acceptable because (a) the TEST ACTION is a UI button click, and (b) the API use is confined to backend-effect observation, matching the hybrid pattern also used in legacy-ui-manager-menu's dropdown tests (the mixed-pattern row below, which remains WEAKENED but is a known follow-up).
|
||||
|
||||
## 🟡 WEAKENED — tests that mix UI + direct API
|
||||
|
||||
These tests DO perform UI interaction (e.g., `selectOption`) but use direct API for verification/cleanup. The UI→effect part is present but the effect is validated via API, not via UI rendering.
|
||||
|
||||
| File | Test | Mixed pattern | Recommended |
|
||||
|---|---|---|---|
|
||||
| legacy-ui-manager-menu.spec.ts | `DB mode dropdown round-trips via API` | `selectOption(newValue)` (UI ✓) → `page.request.get` (verify ✗) | Verify via UI: re-open dialog, read `.value` from `<select>` element. Optional: also check page reload reflects persisted value. |
|
||||
| legacy-ui-manager-menu.spec.ts | `Update Policy dropdown round-trips via API` | Same pattern | Same |
|
||||
|
||||
## ✓ CORRECT — pure UI→effect tests
|
||||
|
||||
| File | Test | UI action | Effect verified |
|
||||
|---|---|---|---|
|
||||
| legacy-ui-manager-menu.spec.ts | `opens via Manager button and shows 3-column layout` | Click Manager button | Dialog `#cm-manager-dialog` visible + expected buttons |
|
||||
| legacy-ui-manager-menu.spec.ts | `shows settings dropdowns (DB, Channel, Policy)` | Open Manager menu | 3 `<select>` elements visible |
|
||||
| legacy-ui-manager-menu.spec.ts | `closes and reopens without duplicating` | Close + reopen dialog | ≤2 dialog instances in DOM |
|
||||
| legacy-ui-custom-nodes.spec.ts | `opens from Manager menu and renders grid` | Click "Custom Nodes Manager" | `#cn-manager-dialog` + grid visible |
|
||||
| legacy-ui-custom-nodes.spec.ts | `loads custom node list (non-empty)` | Open dialog, wait | `.tg-row` count > 0 |
|
||||
| legacy-ui-custom-nodes.spec.ts | `filter dropdown changes displayed nodes` | `selectOption('Installed')` | Filtered count ≤ initial |
|
||||
| legacy-ui-custom-nodes.spec.ts | `search input filters the grid` | `fill('ComfyUI-Manager')` | Filtered count ≤ initial |
|
||||
| legacy-ui-custom-nodes.spec.ts | `footer buttons are present` | Open dialog | Install via Git URL / Restart button visible |
|
||||
| legacy-ui-model-manager.spec.ts | `opens from Manager menu and renders grid` | Click "Model Manager" | `#cmm-manager-dialog` + grid |
|
||||
| legacy-ui-model-manager.spec.ts | `loads model list (non-empty)` | Open dialog | Rows > 0 |
|
||||
| legacy-ui-model-manager.spec.ts | `search input filters the model grid` | `fill('stable diffusion')` | Filtered ≤ initial |
|
||||
| legacy-ui-model-manager.spec.ts | `filter dropdown is present with expected options` | Open dialog | Options length > 0 |
|
||||
| legacy-ui-snapshot.spec.ts | `opens snapshot manager from Manager menu` | Click "Snapshot Manager" | `#snapshot-manager-dialog` visible |
|
||||
| legacy-ui-snapshot.spec.ts | `SS1 Save button creates a new snapshot row` | Click dialog Save/Create button | New snapshot appears in UI row + backend list (hybrid UI-action + backend-effect confirm) |
|
||||
| legacy-ui-snapshot.spec.ts | `UI Remove button deletes a snapshot row` | Click in-row Remove/Delete button (dialog confirm accepted) | Snapshot absent from UI row AND backend list |
|
||||
| legacy-ui-navigation.spec.ts | `Manager menu → Custom Nodes → close → Manager still visible` | Nested dialog nav | Manager reopenable |
|
||||
| legacy-ui-navigation.spec.ts | `Manager menu → Model Manager → close → reopen` | Close + reopen | Model Manager reappears |
|
||||
| debug-install-flow.spec.ts | `capture install button API flow` | Click Install → select version → Select | Captures full API sequence (debug) |
|
||||
|
||||
## Playwright Contract Summary (post Stage2 WI-F)
|
||||
|
||||
- **Compliant** (UI→effect): 17 / 20 tests (85%)
|
||||
- **Mixed** (UI + direct API): 2 / 20 tests (10%)
|
||||
- **Violating** (direct API only): 0 / 20 tests (0%) ✅ — WI-F resolved all 4
|
||||
- **Debug/instrumentation** (acceptable exception): 1 / 20 tests (5%)
|
||||
|
||||
Net change from previous audit (22 tests → 20 tests): `legacy-ui-navigation` lost 2 deleted INADEQUATE tests; `legacy-ui-snapshot` kept 3 total (1 existing PASS + 2 new UI-driven PASS that replaced the 2 original INADEQUATE). The "Mixed" WEAKENED rows (2 manager-menu dropdown tests) remain and should be addressed in a follow-up WI.
|
||||
|
||||
---
|
||||
|
||||
# Section 1.5 — Security-Contract Tests (CSRF Method-Reject)
|
||||
|
||||
`tests/e2e/test_e2e_csrf.py` follows a NEGATIVE-assertion contract:
|
||||
state-changing POST endpoints MUST reject HTTP GET. Unlike the glob
|
||||
endpoint→effect contract (positive response + state change), the CSRF
|
||||
contract verifies ABSENCE of acceptance.
|
||||
|
||||
**Contract**:
|
||||
- GET on state-changing POST endpoint → status_code ∈ (400,403,404,405)
|
||||
- POST counterpart → status_code == 200 (sanity)
|
||||
- GET on read-only endpoint → status_code == 200 (negative control)
|
||||
|
||||
**Reference**: commit 99caef55 — "mitigate CSRF on state-changing
|
||||
endpoints + version SSOT" (CVSS 8.1, reported by XlabAI-Tencent-Xuanwu).
|
||||
Commit applied the GET→POST conversion to BOTH `glob/manager_server.py`
|
||||
(~91 lines) and `legacy/manager_server.py` (~92 lines); the legacy-side
|
||||
regression guard is exercised by `test_e2e_csrf_legacy.py` (added in WI-FF,
|
||||
integrated into this audit in WI-GG).
|
||||
|
||||
**Scope clarification per file docstrings**:
|
||||
ONLY the GET-rejection layer. NOT covered: Origin/Referer validation,
|
||||
same-site cookies, anti-CSRF tokens, cross-site form POST. Do NOT
|
||||
read a PASS here as "CSRF fully solved". Both glob and legacy suites
|
||||
share the same scope — the split exists solely because `comfyui_manager/__init__.py`
|
||||
loads `glob.manager_server` XOR `legacy.manager_server` (mutex via
|
||||
`--enable-manager-legacy-ui`), so each route table requires its own server
|
||||
lifecycle to exercise.
|
||||
|
||||
| File | Tests | Contract verdict |
|
||||
|---|---:|---|
|
||||
| test_e2e_csrf.py::TestStateChangingEndpointsRejectGet | 13 (parametrized; 3 dual-purpose endpoints removed in WI-HH — legitimately covered only in the allow-GET class) | ✓ compliant — negative-path security contract (glob) |
|
||||
| test_e2e_csrf.py::TestCsrfPostWorks | 2 | ✓ compliant — positive sanity (glob) |
|
||||
| test_e2e_csrf.py::TestCsrfReadEndpointsStillAllowGet | 11 (parametrized) | ✓ compliant — negative control for over-correction (glob) |
|
||||
| test_e2e_csrf_legacy.py::TestLegacyStateChangingEndpointsRejectGet | 13 (parametrized — queue/task→queue/batch; 3 dual-purpose excluded) | ✓ compliant — negative-path security contract (legacy) |
|
||||
| test_e2e_csrf_legacy.py::TestLegacyCsrfPostWorks | 2 | ✓ compliant — positive sanity (legacy) |
|
||||
| test_e2e_csrf_legacy.py::TestLegacyCsrfReadEndpointsStillAllowGet | 11 (parametrized) | ✓ compliant — negative control (legacy) |
|
||||
|
||||
**Endpoint-list differences** (legacy vs glob, per `test_e2e_csrf_legacy.py` docstring L23–36):
|
||||
- `/v2/manager/queue/task` → dropped (glob-only; legacy uses `queue/batch`)
|
||||
- `/v2/manager/queue/batch` → added (legacy task-enqueue)
|
||||
- `/v2/manager/db_mode`, `/v2/manager/policy/update`, `/v2/manager/channel_url_list` → dropped from reject-GET (CSRF contract applies only to POST write-path; same GET-read + POST-write split as glob, so these 3 legitimately appear in the allow-GET class only). `test_e2e_csrf.py` currently lists them in BOTH classes; WI-HH tracks the glob-side correction.
|
||||
|
||||
---
|
||||
|
||||
# Section 2 — Glob pytest Contract Audit (Endpoint → Effect)
|
||||
|
||||
## 🟡 Missing effect verification
|
||||
|
||||
| File | Test | Missing effect |
|
||||
|---|---|---|
|
||||
| test_e2e_task_operations.py | `test_install_model_accepts_valid_request` | Only checks 200 status. Does NOT verify task was actually queued (could be via GET queue/status total_count≥1). |
|
||||
|
||||
Adding one line (`status check`) would fix this.
|
||||
|
||||
## ✓ Effect-verifying tests (80 of 81)
|
||||
|
||||
### Install/Uninstall (pack-level effects)
|
||||
- test_e2e_endpoint.test_install_via_endpoint → `_pack_exists` + `_has_tracking` ✓
|
||||
- test_e2e_endpoint.test_uninstall_via_endpoint → `_pack_exists == False` ✓
|
||||
- test_e2e_endpoint.test_install_uninstall_cycle → both ✓
|
||||
- test_e2e_git_clone.test_01_nightly_install → `_pack_exists` + `.git` dir ✓
|
||||
- test_e2e_git_clone.test_03_nightly_uninstall → `_pack_exists == False` ✓
|
||||
|
||||
### Disable/Enable (state-level effects)
|
||||
- test_e2e_task_operations.test_disable_pack → `_pack_disabled` + `_pack_exists == False` ✓
|
||||
- test_e2e_task_operations.test_enable_pack → `_pack_exists` + `!_pack_disabled` ✓
|
||||
- test_e2e_task_operations.test_disable_enable_cycle → both transitions ✓
|
||||
|
||||
### Update/Fix (post-state verification)
|
||||
- test_e2e_task_operations.test_update_installed_pack → `_pack_exists` after ✓
|
||||
- test_e2e_task_operations.test_fix_installed_pack → `_pack_exists` after ✓
|
||||
|
||||
### Queue state
|
||||
- test_e2e_queue_lifecycle.test_reset_queue → status.pending_count == 0 ✓
|
||||
- test_e2e_queue_lifecycle.test_queue_task_and_history → done_count > 0 ✓
|
||||
- test_e2e_queue_lifecycle.test_start_queue_already_idle → status code ✓ (idempotent effect)
|
||||
- test_e2e_task_operations.test_update_comfyui_queues_task → total_count ≥ 1 ✓
|
||||
|
||||
### Config round-trips (persistence effect)
|
||||
- test_e2e_config_api.test_set_and_restore_db_mode → GET reflects POST ✓
|
||||
- test_e2e_config_api.test_set_and_restore_update_policy → same ✓
|
||||
- test_e2e_config_api.test_set_and_restore_channel → same ✓
|
||||
|
||||
### Snapshot state
|
||||
- test_e2e_snapshot_lifecycle.test_save_snapshot + test_getlist_after_save → save creates, getlist reflects ✓
|
||||
- test_e2e_snapshot_lifecycle.test_remove_snapshot → removed item absent from list ✓
|
||||
|
||||
### System state
|
||||
- test_e2e_system_info.test_reboot_and_recovery → health check recovers ✓
|
||||
- test_e2e_system_info.test_version_is_stable → consecutive calls idempotent ✓
|
||||
|
||||
### Response-correctness (read endpoints)
|
||||
All GET endpoint tests verify response schema and content. Examples:
|
||||
- getmappings, installed, queue/status, queue/history, snapshot/get_current, etc. → response shape + field presence asserted ✓
|
||||
|
||||
### Validation/Negative (error-path effects)
|
||||
- All `test_*_invalid_body`, `test_*_missing_params`, `test_*_returns_400`, `test_fetch_updates_returns_deprecated` verify the error RESPONSE effect (status code + optional body fields) ✓
|
||||
|
||||
## Glob pytest Contract Summary
|
||||
|
||||
- **Compliant** (endpoint→effect, positive contract): 81 / 81 tests (100%) — Stage2 WI-D upgraded `test_install_model_accepts_valid_request` to effect-verifying.
|
||||
- **Weak** (status-only, no effect): ~~1~~ → 0 / 81 tests (resolved)
|
||||
- **Security-contract** (CSRF method-reject, separate negative contract): 8 / 8 test functions (52 / 52 parametrized invocations — 26 glob + 26 legacy) — all compliant. References: `tests/e2e/test_e2e_csrf.py` (glob, 99caef55 ~91-line diff; 3 dual-purpose endpoints removed from reject-GET fixture in WI-HH to match the GET-read + POST-write split, so glob count dropped from 29 → 26) + `tests/e2e/test_e2e_csrf_legacy.py` (legacy, 99caef55 ~92-line diff — added in WI-FF, audited in WI-GG). See Section 1.5 above.
|
||||
|
||||
---
|
||||
|
||||
# Section 3 — Reclassification Plan
|
||||
|
||||
## Tests to move out of Playwright suite (STATUS: ALL 4 RESOLVED — Stage2 WI-F)
|
||||
|
||||
1. ~~`legacy-ui-snapshot.spec.ts::lists existing snapshots`~~ → **DELETED**; backend `getlist` coverage owned by `test_e2e_snapshot_lifecycle.py::test_getlist_after_save`.
|
||||
2. ~~`legacy-ui-snapshot.spec.ts::save snapshot via API and verify in list`~~ → **REWRITTEN** as UI-driven `SS1 Save button creates a new snapshot row`; also `UI Remove button deletes a snapshot row` added.
|
||||
3. ~~`legacy-ui-navigation.spec.ts::API health check while dialogs are open`~~ → **DELETED**; version coverage in `test_e2e_system_info.py::test_version_returns_string`.
|
||||
4. ~~`legacy-ui-navigation.spec.ts::system endpoints accessible from browser context`~~ → **DELETED**; redundant with pytest system_info suite.
|
||||
|
||||
Verification: `npx playwright test --list --grep '<any of the 4 titles>'` → 0 tests. pytest counterparts regression: 12/12 PASS (`test_e2e_snapshot_lifecycle.py` + `test_e2e_system_info.py`).
|
||||
|
||||
## Tests to rewrite for UI-only verification
|
||||
|
||||
2 mixed tests in `legacy-ui-manager-menu.spec.ts` should verify via UI instead of API:
|
||||
|
||||
1. `DB mode dropdown round-trips via API` → after selectOption, re-open dialog and check `<select>.value` matches
|
||||
2. `Update Policy dropdown round-trips via API` → same pattern
|
||||
|
||||
Keep the restore step (via API) for cleanup — that is acceptable as teardown.
|
||||
|
||||
## New UI→effect tests needed (currently missing)
|
||||
|
||||
Based on Report A legacy endpoints and legacy UI flows, these UI→effect tests are missing:
|
||||
|
||||
| Legacy UI flow | Endpoint triggered | Effect to verify |
|
||||
|---|---|---|
|
||||
| Click "Install" in Custom Nodes Manager row | POST queue/batch | Pack appears in filesystem (via test hooks) + "Installed" badge in UI |
|
||||
| Click "Uninstall" button | POST queue/batch | Pack removed + row shows "Not Installed" |
|
||||
| Click "Update All" in Manager menu | POST queue/update_all | "Updating" indicator appears + queue progress WebSocket |
|
||||
| Click "Install via Git URL" button + enter URL | POST customnode/install/git_url | Pack cloned (if endpoint still exists) |
|
||||
| Click "Restart" in Manager menu | POST manager/reboot | Server restart + UI reconnect |
|
||||
| Click "Save" in Snapshot Manager | POST snapshot/save | Snapshot row appears in UI list |
|
||||
| Click "Delete" row action in Snapshot Manager | POST snapshot/remove?target=X | Row disappears from UI list |
|
||||
|
||||
→ The existing `debug-install-flow.spec.ts` provides the instrumentation template. These can be built from it with assertions added.
|
||||
|
||||
---
|
||||
|
||||
# Section 4 — Revised Coverage Verdict
|
||||
|
||||
Applying the strict contract:
|
||||
|
||||
## Glob v2 coverage (endpoint → effect)
|
||||
|
||||
| Status | Count |
|
||||
|---|---:|
|
||||
| Effect-verified | 27/30 |
|
||||
| Status-only (weakened) | 1/30 (install_model) |
|
||||
| Intentionally skipped destructive | 2/30 (snapshot/restore, switch_version positive) |
|
||||
|
||||
## Legacy coverage (UI → effect)
|
||||
|
||||
Strict UI→effect tests for legacy endpoints:
|
||||
|
||||
| Endpoint | UI→effect test exists? |
|
||||
|---|---|
|
||||
| POST queue/batch | ⚠️ debug only (no assertion); NO production test |
|
||||
| GET customnode/getlist | ✓ via `loads custom node list (non-empty)` |
|
||||
| GET /customnode/alternatives | ✗ |
|
||||
| GET externalmodel/getlist | ✓ via `loads model list (non-empty)` |
|
||||
| GET customnode/versions/{node_name} | ⚠️ debug only |
|
||||
| GET customnode/disabled_versions/{node_name} | ✗ |
|
||||
| POST customnode/install/git_url | ✗ (no "Install via Git URL" test) |
|
||||
| POST customnode/install/pip | ✗ |
|
||||
| GET manager/notice | ✗ |
|
||||
| GET db_mode / POST db_mode (via UI) | 🟡 mixed (UI selectOption + API verify) |
|
||||
| GET policy/update / POST policy/update (via UI) | 🟡 mixed |
|
||||
| GET snapshot/getlist (via dialog) | ✓ (opens dialog) |
|
||||
| POST snapshot/save (via UI button) | ✗ (only API-driven test exists) |
|
||||
| POST snapshot/remove (via UI) | ✗ (only API-driven cleanup) |
|
||||
| POST manager/reboot (via UI "Restart" button) | ✗ |
|
||||
|
||||
**Strict legacy coverage**: 3/15 endpoints fully UI→effect verified.
|
||||
|
||||
---
|
||||
|
||||
# Section 5 — Action Items (Prioritized)
|
||||
|
||||
## 🔴 Contract violations (fix or remove)
|
||||
|
||||
1. DELETE 2 Playwright tests in `legacy-ui-snapshot.spec.ts` (API-only — redundant with pytest E2E)
|
||||
2. DELETE 2 Playwright tests in `legacy-ui-navigation.spec.ts` (API-only health checks)
|
||||
3. FIX install_model status-only test in `test_e2e_task_operations.py`
|
||||
|
||||
## 🟡 Weaken→strengthen
|
||||
|
||||
4. Rewrite 2 mixed `legacy-ui-manager-menu.spec.ts` dropdown tests to verify UI state (not API round-trip)
|
||||
|
||||
## 🟢 New UI→effect tests (recommended)
|
||||
|
||||
5. Add UI-driven install/uninstall test (click button → verify pack effect + UI state)
|
||||
6. Add UI-driven snapshot save/remove test (via Snapshot Manager dialog buttons)
|
||||
7. Add UI-driven "Restart" button test (verify server restart)
|
||||
8. Add UI-driven "Update All" flow test
|
||||
|
||||
## ✓ JS call-site verification (2026-04-18 re-audit)
|
||||
|
||||
All 5 endpoints initially flagged as "dead code" are CONFIRMED ACTIVE in legacy UI JS:
|
||||
|
||||
| Endpoint | JS file:line |
|
||||
|---|---|
|
||||
| /customnode/alternatives | custom-nodes-manager.js:1885 |
|
||||
| /v2/customnode/disabled_versions/{name} | custom-nodes-manager.js:1401 |
|
||||
| /v2/customnode/install/git_url | common.js:248 |
|
||||
| /v2/customnode/install/pip | common.js:213 |
|
||||
| /v2/manager/notice | comfyui-manager.js:418 |
|
||||
|
||||
→ Action: **add UI→effect tests** (not remove). Previous "dead code candidate" recommendation retracted.
|
||||
|
||||
---
|
||||
*End of Test Contract Audit*
|
||||
@@ -0,0 +1,834 @@
|
||||
# Verification Design — Per-Goal Verification Items
|
||||
|
||||
**Generated**: 2026-04-18
|
||||
**Source**: Derived from `endpoint_scenarios.md`, `scenario_intents.md`, `scenario_effects.md`
|
||||
**Purpose**: For each scenario's achievement goal, design the concrete verification items (assertions + observables + tools) required to prove the goal is met.
|
||||
|
||||
**Verification item format** (used throughout):
|
||||
```
|
||||
Goal: <what scenario intends to achieve>
|
||||
Precondition: <state required before the action>
|
||||
Action: <call / UI interaction>
|
||||
Observable: <what can be inspected>
|
||||
Assertion: <pass/fail criterion>
|
||||
Negative check: <what must NOT happen>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Section 1 — Queue Management Verification Design
|
||||
|
||||
## 1.1 POST /v2/manager/queue/task (kind=install)
|
||||
|
||||
### Goal A1 — Install a CNR pack to loadable state
|
||||
- **Precondition**: pack dir absent; `.tracking` absent
|
||||
- **Action**: POST queue/task kind=install params={id, version, selected_version, mode, channel} → POST queue/start
|
||||
- **Observable**: filesystem `custom_nodes/<pack>/` + `.tracking` file content; `customnode/installed` response; WebSocket `cm-queue-status all-done`
|
||||
- **Assertion**: pack dir exists AND `.tracking` present AND `installed[pack_id].cnr_id == pack_id` AND version matches
|
||||
- **Negative check**: no leftover `.trash_*` dirs; no ModuleNotFoundError in server log
|
||||
|
||||
### Goal A2 — Install a nightly (URL) pack via git clone
|
||||
- **Precondition**: pack dir absent
|
||||
- **Action**: POST queue/task kind=install selected_version=nightly with `repository` URL
|
||||
- **Observable**: pack dir + `.git/` subdir; `.git/config` remote URL
|
||||
- **Assertion**: `.git` exists AND remote URL matches request
|
||||
- **Negative check**: no ModuleNotFoundError (git_helper.py subprocess ran cleanly)
|
||||
|
||||
### Goal A3 — Skip install when already disabled (re-enable shortcut)
|
||||
- **Precondition**: pack in `.disabled/`
|
||||
- **Action**: POST queue/task kind=install params.skip_post_install=true for known cnr_id
|
||||
- **Observable**: pack moved back to active custom_nodes/; no fresh clone
|
||||
- **Assertion**: pack active + no new download; existing `.tracking` preserved
|
||||
- **Negative check**: no re-download (verify via timing or lack of network log)
|
||||
|
||||
### Goal A4 — Reject bad kind (validation error)
|
||||
- **Precondition**: queue empty or known state
|
||||
- **Action**: POST queue/task with kind="garbage"
|
||||
- **Observable**: response status; `queue/status.total_count` before/after
|
||||
- **Assertion**: 400 + ValidationError text; total_count unchanged
|
||||
- **Negative check**: no pack changes; no task in history
|
||||
|
||||
### Goal A5 — Reject missing ui_id/client_id (traceability gate)
|
||||
- **Action**: POST queue/task without ui_id OR without client_id
|
||||
- **Assertion**: 400; no task queued (verify via status counts)
|
||||
|
||||
### Goal A6 — Worker auto-starts on task queue
|
||||
- **Precondition**: worker idle
|
||||
- **Action**: POST queue/task (any valid)
|
||||
- **Observable**: `queue/status.is_processing` polling
|
||||
- **Assertion**: within N seconds is_processing becomes true; eventually done_count increments
|
||||
|
||||
## 1.2 POST /v2/manager/queue/task (kind=uninstall)
|
||||
|
||||
### Goal U1 — Remove installed pack
|
||||
- **Precondition**: pack present; in `installed` list
|
||||
- **Action**: POST queue/task kind=uninstall params.node_name=<cnr_id>
|
||||
- **Observable**: filesystem pack dir; `customnode/installed`
|
||||
- **Assertion**: pack dir absent AND cnr_id absent from installed
|
||||
- **Negative check**: other packs untouched; no orphan files in .disabled/
|
||||
|
||||
### Goal U2 — Idempotent uninstall of missing pack
|
||||
- **Precondition**: pack absent
|
||||
- **Action**: POST queue/task kind=uninstall for non-existent pack
|
||||
- **Assertion**: task completes without raising fatal error; state unchanged
|
||||
|
||||
## 1.3 POST /v2/manager/queue/task (kind=update)
|
||||
|
||||
### Goal UP1 — Upgrade pack to newer version
|
||||
- **Precondition**: pack installed at version X; version Y > X available
|
||||
- **Action**: POST queue/task kind=update params.node_name, node_ver
|
||||
- **Observable**: `.tracking` file content (version field); CNR API version
|
||||
- **Assertion**: pack still present; new version recorded; dependencies updated
|
||||
- **Negative check**: no partial state (no half-cloned dir)
|
||||
|
||||
### Goal UP2 — Idempotent when already up-to-date
|
||||
- **Precondition**: pack at latest version
|
||||
- **Action**: POST queue/task kind=update
|
||||
- **Assertion**: no version downgrade; no pack removal; task completes
|
||||
|
||||
## 1.4 POST /v2/manager/queue/task (kind=fix)
|
||||
|
||||
### Goal F1 — Reinstall dependencies of existing pack
|
||||
- **Precondition**: pack present; dependencies broken (simulate by removing from venv)
|
||||
- **Action**: POST queue/task kind=fix params.node_name, node_ver
|
||||
- **Observable**: venv site-packages for pack's requirements; import succeeds
|
||||
- **Assertion**: after fix, all declared requirements importable; pack dir unchanged (same HEAD)
|
||||
|
||||
## 1.5 POST /v2/manager/queue/task (kind=disable)
|
||||
|
||||
### Goal D1 — Move pack to disabled state reversibly
|
||||
- **Precondition**: pack active
|
||||
- **Action**: POST queue/task kind=disable params.node_name, is_unknown
|
||||
- **Observable**: `custom_nodes/.disabled/<pack>/` exists; `custom_nodes/<pack>/` absent
|
||||
- **Assertion**: pack relocated to .disabled/; not in active installed list; on reload NODE_CLASS_MAPPINGS lacks its nodes
|
||||
- **Negative check**: pack files preserved (not deleted)
|
||||
|
||||
### Goal D2 — Idempotent disable of already-disabled pack
|
||||
- **Precondition**: pack in .disabled/
|
||||
- **Action**: POST queue/task kind=disable
|
||||
- **Assertion**: pack still in .disabled/; no duplicate; no error
|
||||
|
||||
## 1.6 POST /v2/manager/queue/task (kind=enable)
|
||||
|
||||
### Goal E1 — Restore pack from .disabled/ to active
|
||||
- **Precondition**: pack in .disabled/
|
||||
- **Action**: POST queue/task kind=enable params.cnr_id
|
||||
- **Observable**: `custom_nodes/<pack>/` exists (may be lowercase variant); .disabled/ entry gone
|
||||
- **Assertion**: pack active; appears in installed; on next reload nodes load
|
||||
- **Negative check**: no duplicate entries
|
||||
|
||||
## 1.7 POST /v2/manager/queue/install_model
|
||||
|
||||
### Goal IM1 — Download model to correct subdirectory
|
||||
- **Precondition**: model file absent; URL in whitelist
|
||||
- **Action**: POST queue/install_model with valid ModelMetadata + client_id + ui_id
|
||||
- **Observable**: `models/<type>/<filename>` file; file size > 0
|
||||
- **Assertion**: file exists + non-zero size; externalmodel/getlist shows installed=True
|
||||
- **Negative check**: no orphan partial downloads
|
||||
|
||||
### Goal IM2 — Reject missing client_id / ui_id
|
||||
- **Action**: POST install_model without one of them
|
||||
- **Assertion**: 400; no task queued; no download attempted
|
||||
|
||||
### Goal IM3 — Reject non-whitelist URL (legacy)
|
||||
- **Action**: POST install_model with URL NOT in whitelist
|
||||
- **Assertion**: 400; no file written
|
||||
|
||||
### Goal IM4 — Block non-safetensors below high+ security
|
||||
- **Precondition**: security_level < high+
|
||||
- **Action**: POST install_model with filename="*.ckpt"
|
||||
- **Assertion**: 403; no file written
|
||||
|
||||
## 1.8 POST /v2/manager/queue/update_all
|
||||
|
||||
### Goal UA1 — Queue update tasks for all active packs
|
||||
- **Precondition**: N active packs installed; queue empty
|
||||
- **Action**: POST queue/update_all with ui_id, client_id, mode
|
||||
- **Observable**: `queue/status.pending_count` delta
|
||||
- **Assertion**: pending_count increased by N (minus manager-skip if desktop); each queued task has kind=update + correct node_name
|
||||
- **Negative check**: comfyui-manager NOT in queue if __COMFYUI_DESKTOP_VERSION__ set
|
||||
|
||||
### Goal UA2 — Security gate (<middle+)
|
||||
- **Precondition**: security_level < middle+
|
||||
- **Action**: POST queue/update_all
|
||||
- **Assertion**: 403; no tasks queued
|
||||
|
||||
### Goal UA3 — Require ui_id + client_id
|
||||
- **Action**: POST queue/update_all without required params
|
||||
- **Assertion**: 400 ValidationError; no queue change
|
||||
|
||||
## 1.9 POST /v2/manager/queue/update_comfyui
|
||||
|
||||
### Goal UC1 — Queue ComfyUI self-update task
|
||||
- **Action**: POST queue/update_comfyui with client_id, ui_id
|
||||
- **Observable**: queue/status.total_count; queued task's params
|
||||
- **Assertion**: total_count += 1; queued task has kind=update-comfyui with is_stable matching config or override
|
||||
- **Negative check**: actual git operation doesn't start (task queued only)
|
||||
|
||||
### Goal UC2 — Explicit stable flag override
|
||||
- **Action**: POST queue/update_comfyui?stable=true (config says nightly)
|
||||
- **Assertion**: queued task.params.is_stable == true
|
||||
|
||||
## 1.10 POST /v2/manager/queue/reset
|
||||
|
||||
### Goal R1 — Clear all queued/running/history tasks
|
||||
- **Precondition**: queue has N tasks
|
||||
- **Action**: POST queue/reset
|
||||
- **Observable**: queue/status all counts
|
||||
- **Assertion**: total_count=done_count=in_progress_count=pending_count=0; is_processing=false
|
||||
- **Negative check**: history tasks also cleared
|
||||
|
||||
### Goal R2 — Idempotent on empty queue
|
||||
- **Precondition**: queue empty
|
||||
- **Action**: POST queue/reset (2 times)
|
||||
- **Assertion**: 200 both times; state still empty
|
||||
|
||||
## 1.11 POST /v2/manager/queue/start
|
||||
|
||||
### Goal S1 — Start worker when idle
|
||||
- **Precondition**: is_processing=false; queue has tasks
|
||||
- **Action**: POST queue/start
|
||||
- **Assertion**: 200; is_processing=true; tasks begin processing (done_count eventually increments)
|
||||
|
||||
### Goal S2 — Don't spawn duplicate worker
|
||||
- **Precondition**: is_processing=true
|
||||
- **Action**: POST queue/start
|
||||
- **Assertion**: 201; still is_processing=true; same worker pid (observable via logs)
|
||||
- **Negative check**: no duplicate task processing (each task runs once)
|
||||
|
||||
## 1.12 GET /v2/manager/queue/status
|
||||
|
||||
### Goal QS1 — Accurate overall counts
|
||||
- **Precondition**: known queue state (e.g., 3 pending, 1 running, 2 done)
|
||||
- **Action**: GET queue/status
|
||||
- **Assertion**: counts match actual internal queue state
|
||||
|
||||
### Goal QS2 — Client-filtered counts
|
||||
- **Precondition**: tasks from multiple client_ids
|
||||
- **Action**: GET queue/status?client_id=X
|
||||
- **Assertion**: response.client_id == X; counts reflect only X's tasks
|
||||
|
||||
## 1.13 GET /v2/manager/queue/history
|
||||
|
||||
### Goal QH1 — Retrieve batch history by id
|
||||
- **Precondition**: batch file exists at manager_batch_history_path
|
||||
- **Action**: GET queue/history?id=<batch_id>
|
||||
- **Assertion**: response JSON matches file contents
|
||||
|
||||
### Goal QH2 — Reject path traversal
|
||||
- **Action**: GET queue/history?id=../../../etc/passwd
|
||||
- **Assertion**: 400 "Invalid history id"; no file read attempt
|
||||
|
||||
### Goal QH3 — Filter by ui_id / client_id / pagination
|
||||
- **Action**: GET queue/history with respective query params
|
||||
- **Assertion**: results properly filtered/paginated
|
||||
|
||||
## 1.14 GET /v2/manager/queue/history_list
|
||||
|
||||
### Goal QHL1 — List batch IDs sorted by mtime
|
||||
- **Precondition**: files exist in history dir
|
||||
- **Action**: GET queue/history_list
|
||||
- **Assertion**: ids list == filenames (stem) sorted by mtime desc
|
||||
|
||||
### Goal QHL2 — Empty list when no history
|
||||
- **Precondition**: empty dir
|
||||
- **Assertion**: `{ids: []}`; 200
|
||||
|
||||
---
|
||||
|
||||
# Section 2 — CustomNode Info Verification Design
|
||||
|
||||
## 2.1 GET /v2/customnode/getmappings
|
||||
|
||||
### Goal CM1 — Return comprehensive node→pack mapping
|
||||
- **Action**: GET getmappings?mode=local (or cache/remote)
|
||||
- **Observable**: response dict; current NODE_CLASS_MAPPINGS
|
||||
- **Assertion**: every currently-loaded node appears in some entry's node_list OR matches a nodename_pattern regex
|
||||
- **Negative check**: no stale entries for uninstalled packs
|
||||
|
||||
### Goal CM2 — Nickname mode applies filter
|
||||
- **Action**: GET getmappings?mode=nickname
|
||||
- **Assertion**: entries include nickname field filtered by nickname_filter rules
|
||||
|
||||
### Goal CM3 — Require explicit mode
|
||||
- **Action**: GET getmappings (no mode param)
|
||||
- **Assertion**: server error (500/KeyError); no partial data
|
||||
|
||||
## 2.2 GET /v2/customnode/fetch_updates (deprecated)
|
||||
|
||||
### Goal FU1 — Signal deprecation to clients
|
||||
- **Action**: GET fetch_updates?mode=local
|
||||
- **Assertion**: 410 status; body `{deprecated: true, error: ..., message: ...}`
|
||||
- **Negative check**: no git fetch executed (no mtime changes on .git/)
|
||||
|
||||
## 2.3 GET /v2/customnode/installed
|
||||
|
||||
### Goal IL1 — Current installed packs
|
||||
- **Precondition**: known packs installed
|
||||
- **Action**: GET installed
|
||||
- **Assertion**: response dict keys include all installed pack identifiers; each entry has cnr_id + version + enabled
|
||||
|
||||
### Goal IL2 — imported mode is startup-frozen
|
||||
- **Precondition**: install a new pack post-startup
|
||||
- **Action**: GET installed?mode=imported
|
||||
- **Assertion**: new pack absent from response (startup snapshot unchanged)
|
||||
- **Negative check**: default mode DOES show new pack (proves they differ)
|
||||
|
||||
## 2.4 POST /v2/customnode/import_fail_info
|
||||
|
||||
### Goal IF1 — Return failure info for failed pack
|
||||
- **Precondition**: pack exists in `cm_global.error_dict`
|
||||
- **Action**: POST import_fail_info {cnr_id}
|
||||
- **Assertion**: 200; response has `msg` + `traceback`; content matches error_dict entry
|
||||
|
||||
### Goal IF2 — Reject unknown pack
|
||||
- **Action**: POST import_fail_info {cnr_id: "unknown-12345"}
|
||||
- **Assertion**: 400; no response body with info
|
||||
|
||||
### Goal IF3 — Validate request body shape
|
||||
- **Action**: POST import_fail_info with non-dict OR missing both fields OR wrong type
|
||||
- **Assertion**: 400 with specific error text
|
||||
|
||||
## 2.5 POST /v2/customnode/import_fail_info_bulk
|
||||
|
||||
### Goal IFB1 — Return per-pack lookup results
|
||||
- **Action**: POST import_fail_info_bulk {cnr_ids: [known, unknown]}
|
||||
- **Assertion**: 200; response maps known→info dict, unknown→null
|
||||
|
||||
### Goal IFB2 — Reject empty lists
|
||||
- **Action**: POST with {cnr_ids:[], urls:[]}
|
||||
- **Assertion**: 400 "Either 'cnr_ids' or 'urls' field is required"
|
||||
|
||||
---
|
||||
|
||||
# Section 3 — Snapshot Verification Design
|
||||
|
||||
## 3.1 GET /v2/snapshot/get_current
|
||||
|
||||
### Goal SG1 — Capture current state accurately
|
||||
- **Precondition**: known set of packs installed
|
||||
- **Action**: GET snapshot/get_current
|
||||
- **Assertion**: response has comfyui (hash/tag), git_custom_nodes (list), cnr_custom_nodes (list), pips; entries match actual installed state
|
||||
|
||||
## 3.2 POST /v2/snapshot/save
|
||||
|
||||
### Goal SS1 — Persist current state to retrievable file
|
||||
- **Precondition**: snapshot dir exists
|
||||
- **Action**: POST snapshot/save
|
||||
- **Observable**: new file in manager_snapshot_path; snapshot/getlist response
|
||||
- **Assertion**: new timestamped file created; content matches get_current() at save time; appears in getlist.items
|
||||
- **Negative check**: other snapshots untouched
|
||||
|
||||
### Goal SS2 — Multiple saves create distinct files
|
||||
- **Action**: POST save; wait 1s; POST save
|
||||
- **Assertion**: 2 distinct new entries in getlist
|
||||
|
||||
## 3.3 GET /v2/snapshot/getlist
|
||||
|
||||
### Goal SL1 — List matches filesystem
|
||||
- **Action**: GET snapshot/getlist
|
||||
- **Assertion**: items == .json files in snapshot dir (stems), desc sorted
|
||||
|
||||
## 3.4 POST /v2/snapshot/remove
|
||||
|
||||
### Goal SR1 — Delete snapshot permanently
|
||||
- **Precondition**: target snapshot exists
|
||||
- **Action**: POST snapshot/remove?target=<id>
|
||||
- **Assertion**: file removed from disk; absent from getlist
|
||||
- **Negative check**: other snapshots untouched
|
||||
|
||||
### Goal SR2 — Idempotent on missing target
|
||||
- **Action**: POST remove?target=nonexistent
|
||||
- **Assertion**: 200 no-op
|
||||
|
||||
### Goal SR3 — Reject path traversal
|
||||
- **Action**: POST remove?target=../../etc/passwd
|
||||
- **Assertion**: 400 "Invalid target"; no file ops outside snapshot dir
|
||||
|
||||
### Goal SR4 — Security gate (<middle)
|
||||
- **Precondition**: security_level < middle
|
||||
- **Action**: POST remove
|
||||
- **Assertion**: 403; target file untouched
|
||||
- **Test reference**: `tests/e2e/test_e2e_secgate_strict.py::TestSecurityGate403_SR4::test_remove_returns_403` (WI-KK PoC, audit-integrated by WI-LL — uses `start_comfyui_strict.sh` strict-mode fixture)
|
||||
|
||||
## 3.5 POST /v2/snapshot/restore
|
||||
|
||||
### Goal SR5 — Schedule snapshot restore for next reboot
|
||||
- **Precondition**: target snapshot exists
|
||||
- **Action**: POST restore?target=<id>
|
||||
- **Observable**: manager_startup_script_path for `restore-snapshot.json`
|
||||
- **Assertion**: restore-snapshot.json exists; content == target snapshot; (downstream: reboot → state reverts)
|
||||
|
||||
### Goal SR6 — Security gate (<middle+)
|
||||
- **Assertion**: 403; no marker file
|
||||
|
||||
---
|
||||
|
||||
# Section 4 — Config Verification Design
|
||||
|
||||
## 4.1 GET /v2/manager/db_mode
|
||||
|
||||
### Goal C1 — Return current config value
|
||||
- **Action**: GET db_mode
|
||||
- **Assertion**: text response ∈ {cache, channel, local, remote}; matches config.ini[db_mode]
|
||||
|
||||
## 4.2 POST /v2/manager/db_mode
|
||||
|
||||
### Goal C2 — Persist new value to config.ini
|
||||
- **Precondition**: original value X
|
||||
- **Action**: POST db_mode {value: Y} where Y ≠ X
|
||||
- **Observable**: GET db_mode after POST; config.ini file content; GET after process restart
|
||||
- **Assertion**: GET returns Y; config.ini updated; survives restart (restart + re-GET returns Y)
|
||||
- **Cleanup**: restore X
|
||||
|
||||
### Goal C3 — Reject malformed JSON / missing value
|
||||
- **Action**: POST with non-JSON body OR {foo: bar}
|
||||
- **Assertion**: 400; config.ini unchanged
|
||||
|
||||
## 4.3-4.4 GET/POST /v2/manager/policy/update
|
||||
|
||||
Same verification pattern as db_mode, with policy values ∈ {stable, stable-comfyui, nightly, nightly-comfyui}.
|
||||
|
||||
## 4.5 GET /v2/manager/channel_url_list
|
||||
|
||||
### Goal C4 — Return selected + available channels
|
||||
- **Action**: GET channel_url_list
|
||||
- **Assertion**: response has selected (str) + list (array of "name::url" strings); selected == name whose URL matches config.channel_url else "custom"
|
||||
|
||||
## 4.6 POST /v2/manager/channel_url_list
|
||||
|
||||
### Goal C5 — Switch channel by name
|
||||
- **Precondition**: original channel X
|
||||
- **Action**: POST {value: Y} where Y is a known channel name
|
||||
- **Observable**: GET after POST
|
||||
- **Assertion**: selected == Y; config.channel_url == channels[Y]
|
||||
- **Cleanup**: restore X
|
||||
|
||||
### Goal C6 — Silent no-op on unknown name
|
||||
- **Action**: POST {value: "nonexistent-channel-xyz"}
|
||||
- **Assertion**: 200; selected UNCHANGED (verify via GET)
|
||||
|
||||
---
|
||||
|
||||
# Section 5 — System + ComfyUI Version Verification Design
|
||||
|
||||
## 5.1 GET /v2/manager/version
|
||||
|
||||
### Goal V1 — Consistent version string
|
||||
- **Action**: GET version × 2 consecutive calls
|
||||
- **Assertion**: both return same non-empty string == core.version_str
|
||||
|
||||
## 5.2 GET /v2/manager/is_legacy_manager_ui
|
||||
|
||||
### Goal V2 — Reflect CLI flag
|
||||
- **Precondition**: server started with/without --enable-manager-legacy-ui
|
||||
- **Action**: GET is_legacy_manager_ui
|
||||
- **Assertion**: response.is_legacy_manager_ui matches the actual CLI flag
|
||||
|
||||
## 5.3 POST /v2/manager/reboot
|
||||
|
||||
### Goal V3 — Process restart + recovery
|
||||
- **Precondition**: server running; pre_version captured
|
||||
- **Action**: POST reboot
|
||||
- **Observable**: connection behavior; health endpoint polling; post-reboot version
|
||||
- **Assertion**: connection drops or 200; within N seconds server responds again; post_version == pre_version (verifies core state preserved)
|
||||
- **Negative check**: config unchanged across reboot
|
||||
|
||||
### Goal V4 — COMFY_CLI_SESSION mode
|
||||
- **Precondition**: __COMFY_CLI_SESSION__ env set
|
||||
- **Action**: POST reboot
|
||||
- **Assertion**: `.reboot` marker file created; process exits with code 0 (external manager restarts)
|
||||
|
||||
### Goal V5 — Security gate (<middle)
|
||||
- **Action**: POST reboot at low security
|
||||
- **Assertion**: 403; server continues (no restart occurs)
|
||||
|
||||
## 5.4 GET /v2/comfyui_manager/comfyui_versions
|
||||
|
||||
### Goal CV1 — Enumerate versions + current
|
||||
- **Action**: GET comfyui_versions
|
||||
- **Assertion**: response `{versions, current}`; versions list non-empty; each item is string; current ∈ versions; matches actual `.git` tags/commits
|
||||
|
||||
### Goal CV2 — Fail cleanly on non-git
|
||||
- **Precondition**: ComfyUI dir not a git repo (simulate)
|
||||
- **Action**: GET
|
||||
- **Assertion**: 400; no partial response
|
||||
|
||||
## 5.5 POST /v2/comfyui_manager/comfyui_switch_version
|
||||
|
||||
### Goal CV3 — Queue update-comfyui task with target_version
|
||||
- **Precondition**: security_level ≥ high+
|
||||
- **Action**: POST switch_version?ver=X&client_id=Y&ui_id=Z
|
||||
- **Observable**: queue/status; queued task params
|
||||
- **Assertion**: task queued with params.target_version=X
|
||||
- **Note**: actual switch (destructive) NOT verified in E2E
|
||||
|
||||
### Goal CV4 — Security gate (<high+)
|
||||
- **Action**: POST switch_version at lower security
|
||||
- **Assertion**: 403; no task queued
|
||||
- **Test reference**: `tests/e2e/test_e2e_secgate_default.py::TestSecurityGate403_CV4::test_switch_version_returns_403_at_default` (WI-KK demo, audit-integrated by WI-LL — runs at default `security_level=normal`; WI-KK research showed `is_local_mode=True + normal` is already outside the `[WEAK, NORMAL_]` allowed set for high+, so no harness is needed)
|
||||
|
||||
### Goal CV5 — Validate params
|
||||
- **Action**: POST without ver OR without client_id/ui_id
|
||||
- **Assertion**: 400 with error details; no task queued
|
||||
|
||||
---
|
||||
|
||||
# Section 6 — Legacy-only Verification Design (UI → effect)
|
||||
|
||||
All verification here assumes Playwright UI interaction as the Action. Effect is observed through both UI state and backend filesystem/state.
|
||||
|
||||
## 6.1 POST /v2/manager/queue/batch
|
||||
|
||||
### Goal LB1 — Install via UI row "Install" button
|
||||
- **Precondition**: pack not installed; Custom Nodes Manager dialog open
|
||||
- **Action (UI)**: filter to Not Installed → click Install on target row → click "Select" on version dialog
|
||||
- **Observable**: backend: custom_nodes/<pack>/ + `.tracking`; UI: row badge changes to "Installed"; WebSocket: cm-queue-status all-done
|
||||
- **Assertion**: backend install effect + UI state updates
|
||||
|
||||
### Goal LB2 — Uninstall via UI
|
||||
- **Precondition**: pack installed
|
||||
- **Action (UI)**: click "Uninstall" button on row
|
||||
- **Observable**: backend: pack dir removed; UI: row shows "Not Installed"
|
||||
- **Assertion**: both states consistent
|
||||
|
||||
### Goal LB3 — Disable via UI
|
||||
- **Action (UI)**: click "Disable" button on row
|
||||
- **Assertion**: pack in .disabled/; UI row shows "Disabled"
|
||||
|
||||
### Goal LB4 — Update all via menu
|
||||
- **Action (UI)**: Manager menu → "Update All"
|
||||
- **Observable**: queue/status; progress indicator in UI
|
||||
- **Assertion**: N tasks queued; UI progress reflects N; eventual completion
|
||||
|
||||
### Goal LB5 — Batch partial failure reporting
|
||||
- **Action (UI)**: batch with one guaranteed-fail pack
|
||||
- **Observable**: response.failed array; UI notification
|
||||
- **Assertion**: failed list contains only failed pack id; others succeeded
|
||||
|
||||
## 6.2 GET /v2/customnode/getlist
|
||||
|
||||
### Goal LG1 — Populate Custom Nodes Manager dialog
|
||||
- **Action (UI)**: click "Custom Nodes Manager" from Manager menu
|
||||
- **Observable**: UI grid rows
|
||||
- **Assertion**: rows > 0; each row has Title + Installed/Not-Installed state + Install/Uninstall button
|
||||
|
||||
### Goal LG2 — skip_update=true optimizes loading
|
||||
- **Action (UI)**: open dialog with flag set (URL param or setting)
|
||||
- **Observable**: loading time; network log
|
||||
- **Assertion**: no git fetch calls; load time < N seconds
|
||||
|
||||
## 6.3 GET /customnode/alternatives
|
||||
|
||||
### Goal LA1 — Show alternatives on dedicated UI flow
|
||||
- **Action (UI)**: trigger alternatives display (specific button or context)
|
||||
- **Observable**: alternatives panel/dialog
|
||||
- **Assertion**: populated with alter-list.json entries (at least one if data exists)
|
||||
|
||||
## 6.4 GET /v2/externalmodel/getlist
|
||||
|
||||
### Goal LM1 — Populate Model Manager dialog
|
||||
- **Action (UI)**: open Model Manager
|
||||
- **Observable**: UI grid
|
||||
- **Assertion**: rows > 0; each row has `installed` flag accurately reflecting filesystem
|
||||
|
||||
### Goal LM2 — Install flag correctness
|
||||
- **Precondition**: pre-existing model file at known path
|
||||
- **Action (UI)**: open Model Manager
|
||||
- **Assertion**: that model row shows "Installed"
|
||||
|
||||
## 6.5 GET /v2/customnode/versions/{node_name}
|
||||
|
||||
### Goal LV1 — Version dropdown on Install click
|
||||
- **Action (UI)**: click Install on a CNR pack row
|
||||
- **Observable**: version selector (select multiple) in modal
|
||||
- **Assertion**: dropdown options match CNR registry versions for that pack; "latest" option present
|
||||
|
||||
### Goal LV2 — Unknown pack → 400 (UI should handle)
|
||||
- **Action**: direct API call with bogus node_name
|
||||
- **Assertion**: 400; UI shows error or skips
|
||||
|
||||
## 6.6 GET /v2/customnode/disabled_versions/{node_name}
|
||||
|
||||
### Goal LDV1 — Show disabled versions for re-enable
|
||||
- **Precondition**: pack has disabled versions
|
||||
- **Action (UI)**: open pack's "Disabled Versions" dropdown
|
||||
- **Observable**: dropdown options
|
||||
- **Assertion**: options match backend disabled_versions response
|
||||
|
||||
## 6.7 POST /v2/customnode/install/git_url
|
||||
|
||||
### Goal LGU1 — Install via "Install via Git URL" button
|
||||
- **Precondition**: security_level ≥ high+; pack absent
|
||||
- **Action (UI)**: click button → enter URL → confirm
|
||||
- **Observable**: custom_nodes/<pack>/ + .git/
|
||||
- **Assertion**: pack cloned; UI reflects
|
||||
|
||||
### Goal LGU2 — Security gate
|
||||
- **Precondition**: security_level < high+
|
||||
- **Assertion**: 403 from API; UI shows error; no clone occurs
|
||||
|
||||
## 6.8 POST /v2/customnode/install/pip
|
||||
|
||||
### Goal LPP1 — Install pip packages via UI
|
||||
- **Action (UI)**: trigger pip install flow with known package
|
||||
- **Observable**: venv site-packages
|
||||
- **Assertion**: package importable after install
|
||||
|
||||
### Goal LPP2 — Security gate
|
||||
- **Assertion**: 403 at lower security; no pip invocation
|
||||
|
||||
## 6.9 GET /v2/manager/notice
|
||||
|
||||
### Goal LN1 — Fetch and display News
|
||||
- **Precondition**: GitHub reachable
|
||||
- **Action**: GET manager/notice (triggered on Manager menu open)
|
||||
- **Assertion**: HTML response; contains expected markdown-body content; footer has ComfyUI + Manager version
|
||||
|
||||
### Goal LN2 — Graceful on GitHub unreachable
|
||||
- **Precondition**: network blocked
|
||||
- **Assertion**: "Unable to retrieve Notice" returned; no server crash; UI shows message
|
||||
|
||||
### Goal LN3 — Non-git ComfyUI warning
|
||||
- **Precondition**: ComfyUI dir not a git repo
|
||||
- **Assertion**: response starts with "isn't git repo" warning paragraph
|
||||
|
||||
### Goal LN4 — Outdated ComfyUI warning
|
||||
- **Precondition**: comfy_ui_commit_datetime.date() < required_commit_datetime.date()
|
||||
- **Assertion**: response starts with "too OUTDATED!!!" paragraph
|
||||
|
||||
---
|
||||
|
||||
# Section 7 — Cross-cutting Intent Pattern Templates
|
||||
|
||||
Reusable verification templates organized by intent category. Apply the template to any scenario of that type.
|
||||
|
||||
## 7.1 User Capability Template
|
||||
|
||||
**Goal**: User accomplishes task X.
|
||||
|
||||
**Verification structure**:
|
||||
1. **Precondition check**: state is F0 (function not yet performed)
|
||||
2. **Invoke action**: endpoint call or UI interaction
|
||||
3. **Wait for completion**: poll status endpoint OR WebSocket event OR filesystem observer
|
||||
4. **Observable assertions** (in order):
|
||||
- Response status 2xx
|
||||
- Response body schema correct
|
||||
- Side-effect state is F1 (function performed)
|
||||
- Secondary state changes (counters, history, lists) reflect the change
|
||||
5. **Negative checks**:
|
||||
- No orphan state (partial writes, stale locks)
|
||||
- Other unrelated state unchanged (blast radius contained)
|
||||
6. **Cleanup**: restore F0 where possible (for idempotent tests)
|
||||
|
||||
**Examples in this report**: A1, A2, U1, UP1, D1, E1, IM1, UA1, R1, S1, SS1, SR1, C2, C5, V3, CV3, LB1-LB5, LGU1, LPP1
|
||||
|
||||
## 7.2 Input Resilience Template
|
||||
|
||||
**Goal**: Bad input must be rejected without side effects.
|
||||
|
||||
**Verification structure**:
|
||||
1. **Precondition**: system in known-good state S0
|
||||
2. **Action**: send malformed/invalid input (craft variants per endpoint):
|
||||
- Malformed JSON (raw text, wrong Content-Type)
|
||||
- Missing required field
|
||||
- Wrong type for field
|
||||
- Out-of-range value
|
||||
- Extraneous unexpected field (should be ignored)
|
||||
3. **Observable assertions**:
|
||||
- Response 4xx (400 typical; 500 only if server constraint like KeyError)
|
||||
- Error message identifies the problem (if surfaced)
|
||||
4. **Negative checks** (the critical part):
|
||||
- State S1 == S0 (no mutation)
|
||||
- Queue/history not populated
|
||||
- Filesystem unchanged
|
||||
- No downstream side effects (no email, no download, no process start)
|
||||
|
||||
**Examples**: A4, A5, U1 (missing target), IM2, UA3, UC2 edges, QH2, IF2, IF3, IFB2, C3, LV2
|
||||
|
||||
## 7.3 Security Boundary Template
|
||||
|
||||
**Goal**: Operations requiring privilege X must fail at lower privilege.
|
||||
|
||||
**Verification structure**:
|
||||
1. **Precondition**: server running at specific `security_level` below the gate
|
||||
2. **Action**: invoke the privileged endpoint
|
||||
3. **Observable assertions**:
|
||||
- Response 403
|
||||
- Server log contains security-denied message
|
||||
4. **Critical negative checks**:
|
||||
- NO task queued
|
||||
- NO filesystem change
|
||||
- NO network operation initiated
|
||||
- NO config change
|
||||
- NO process change (no restart, no exit)
|
||||
5. **Positive counterpart**: at or above required level, operation succeeds
|
||||
|
||||
**Security tiers to cover**:
|
||||
- `middle` — reboot, snapshot/remove, _uninstall, _update
|
||||
- `middle+` — update_all, snapshot/restore, _install_custom_node, _install_model
|
||||
- `high+` — comfyui_switch_version, install/git_url, install/pip, non-safetensors model, _fix (`middle` → `high` in commit `c8992e5d`; subsequent `high` → `high+` in WI-#235 to align the gate with the `SECURITY_MESSAGE_HIGH_P` log text)
|
||||
|
||||
**Path-traversal sub-template** (within security boundary):
|
||||
1. Action: request with `../`, absolute path, encoded traversal
|
||||
2. Assertion: 400 "Invalid target" or similar
|
||||
3. Negative check: target file unchanged; no files outside the endpoint's scope accessed
|
||||
|
||||
**Examples**: IM4, UA2, SR3, SR4, SR6, V5, CV4, LGU2, LPP2, QH2
|
||||
|
||||
## 7.4 Observability Template
|
||||
|
||||
**Goal**: Caller can trust the response to reflect actual system state.
|
||||
|
||||
**Verification structure**:
|
||||
1. **Establish known state**: inject a known delta (queue a task, install a pack, save a snapshot)
|
||||
2. **Read endpoint**: GET the observability endpoint
|
||||
3. **Assertion**: response reflects the known delta
|
||||
4. **Consistency check**: read again immediately; result identical (no race/jitter)
|
||||
5. **Schema check**: all expected fields present with correct types
|
||||
|
||||
**Required-identity fields** (traceability sub-pattern):
|
||||
- client_id + ui_id must be present in inputs for all state-mutating endpoints
|
||||
- Verify that history/status queries can locate tasks by these IDs
|
||||
|
||||
**Examples**: QS1, QS2, QH1, QH3, IL1, IL2, SL1, C1, V1, V2, CV1, LG1, LM1, LM2, LV1, LDV1
|
||||
|
||||
## 7.5 Idempotency Template
|
||||
|
||||
**Goal**: Re-invoking an operation on an already-satisfied state is safe.
|
||||
|
||||
**Verification structure**:
|
||||
1. **Invoke operation** to reach state F1
|
||||
2. **Invoke same operation again** (no intermediate state change)
|
||||
3. **Assertions**:
|
||||
- Response 2xx (same or semantically equivalent status, e.g., 200/201)
|
||||
- State remains F1 (no regression, no duplicate)
|
||||
- No error raised
|
||||
4. **Stress variant**: invoke N times in rapid succession; ensure no race condition
|
||||
|
||||
**Examples**: U2, UP2, D2, E2, R2, S2, SS2 (distinct but both succeed), SR2, C6
|
||||
|
||||
## 7.6 Data Integrity Template
|
||||
|
||||
**Goal**: Config / runtime constants remain stable and consistent.
|
||||
|
||||
**Verification structure**:
|
||||
1. **Read constant twice** (consecutive or across operations)
|
||||
2. **Assertion**: identical values
|
||||
3. **Persistence across restart**: set value → reboot → re-read
|
||||
4. **Assertion**: value persisted
|
||||
|
||||
**Examples**: V1 (version idempotent), C2 (db_mode survives restart), CV1 (versions list stable)
|
||||
|
||||
## 7.7 Recovery Template
|
||||
|
||||
**Goal**: System can recover from failure or drifted state.
|
||||
|
||||
**Verification structure**:
|
||||
1. **Induce failure state**: delete dependency, corrupt a file, disable a pack
|
||||
2. **Invoke recovery operation**: fix, restore, re-enable
|
||||
3. **Assertion**: state is healthy again; pack/module loads
|
||||
4. **Negative check**: no data loss; original files preserved where applicable
|
||||
|
||||
**Examples**: F1 (fix dependency), E1 (enable disabled), SR5 (restore snapshot)
|
||||
|
||||
## 7.8 Concurrency Safety Template
|
||||
|
||||
**Goal**: Parallel or duplicated operations don't corrupt state.
|
||||
|
||||
**Verification structure**:
|
||||
1. **Setup parallel trigger**: fire multiple start/reset/task commands in rapid succession
|
||||
2. **Assertions**:
|
||||
- No duplicate worker thread (single pid in logs)
|
||||
- No task processed twice (idempotent task-id check)
|
||||
- No race-condition data corruption (e.g., half-written config)
|
||||
3. **Stress test variant**: N parallel requests, assert linearized outcome
|
||||
|
||||
**Examples**: S2 (duplicate start), R1 (reset during processing)
|
||||
|
||||
---
|
||||
|
||||
# Section 8 — Verification Matrix Summary
|
||||
|
||||
| Intent category | Template | Scenarios using | Test type needed |
|
||||
|---|---|---:|---|
|
||||
| User capability | 7.1 | 62 | E2E effect verification (real install/remove/save etc.) |
|
||||
| Input resilience | 7.2 | 32 | Negative tests with negative-check assertions |
|
||||
| Security boundary | 7.3 | 15 | Permission tests at each security level |
|
||||
| Observability | 7.4 | 16 | GET correctness + traceability tests |
|
||||
| Idempotency | 7.5 | 14 | Repeat-call tests |
|
||||
| Data integrity | 7.6 | 8 | Cross-restart persistence tests |
|
||||
| Recovery | 7.7 | 5 | Fault-injection tests |
|
||||
| Concurrency safety | 7.8 | 2 | Parallel-call stress tests |
|
||||
|
||||
---
|
||||
|
||||
# Section 9 — Practical Implementation Priority
|
||||
|
||||
Based on security impact + existing coverage gaps:
|
||||
|
||||
## 🔴 Must-add (security + integrity)
|
||||
1. **Path traversal tests** for snapshot/remove, snapshot/restore, queue/history (Section 7.3 path sub-template)
|
||||
2. **Security gate 403 tests** for each tier — requires running with restricted security_level in separate test run
|
||||
3. **Config persistence across restart** — set db_mode → reboot → verify (Section 7.6)
|
||||
|
||||
## 🟡 Should-add (coverage quality)
|
||||
4. **UI-driven install/uninstall flow** (LB1-LB3) — convert debug-install-flow.spec.ts to assertion test
|
||||
5. **install_model effect** — current test only checks 200; add queue/status verification (Goal IM1)
|
||||
6. **Fix recovery test** — induce broken dependency, verify fix heals (Goal F1)
|
||||
|
||||
## 🟢 Nice-to-have
|
||||
7. Concurrency tests (duplicate queue/start; parallel task queueing)
|
||||
8. get_current snapshot content fidelity — compare to actual installed state
|
||||
9. Update version bump verification — test install v1.0.0 → update → expect v1.0.1 marker
|
||||
|
||||
---
|
||||
|
||||
# Section 10 — Security Mitigation Contracts
|
||||
|
||||
Layer: CSRF method-rejection (GET→POST conversion, commit `99caef55`). This section formalizes the contract exercised by TWO test files — one per server variant (mutex loading via `--enable-manager-legacy-ui`):
|
||||
|
||||
- `tests/e2e/test_e2e_csrf.py` — glob server (4 functions / 26 parametrized invocations — post-WI-HH; was 29 before the 3 dual-purpose endpoints were removed from the reject-GET fixture)
|
||||
- `tests/e2e/test_e2e_csrf_legacy.py` — legacy server (4 functions / 26 parametrized invocations; WI-FF added, WI-GG audit-integrated)
|
||||
|
||||
Scope is deliberately narrow — see each test file's SCOPE docstring: only the method-reject layer, not Origin/Referer/same-site/anti-token defenses (those are handled by `origin_only_middleware` at the aiohttp layer and are out of scope here). The full 16-endpoint inventory is recorded in `reports/endpoint_scenarios.md` under "CSRF Method-Reject Contract Inventory"; the legacy file substitutes `/v2/manager/queue/batch` for `/v2/manager/queue/task` (glob-only) and scopes the 3 dual-purpose endpoints (`db_mode`, `policy/update`, `channel_url_list`) to the ALLOW-GET class only.
|
||||
|
||||
## 10.1 CSRF Method-Reject Contract
|
||||
|
||||
### Goal CSRF-M1 — State-changing endpoints reject HTTP GET (13 endpoints, post-WI-HH)
|
||||
- **Precondition**: ComfyUI running; the 13 state-changing endpoints from `STATE_CHANGING_POST_ENDPOINTS` declared as `@routes.post(...)` in `comfyui_manager/glob/manager_server.py` (mirror in `comfyui_manager/legacy/manager_server.py`) — post-WI-HH count; excludes the 3 dual-purpose endpoints (`db_mode`, `policy/update`, `channel_url_list`) whose GET path legitimately reads the current value
|
||||
- **Action**: HTTP GET to each endpoint path (no body, `allow_redirects=False`, module-scoped ComfyUI fixture)
|
||||
- **Observable**: HTTP status code; response body (first 200 chars on failure)
|
||||
- **Assertion**: `status_code not in range(200, 400)` AND `status_code in (400, 403, 404, 405)` for every path in the 13-endpoint fixture (post-WI-HH; the legacy counterpart also iterates 13 paths after substituting `queue/batch` for `queue/task`)
|
||||
- **Negative check**: no 2xx success and no 3xx redirect on any state-changing path (blocks `<img src=...>`, link-click, and redirect-based cross-origin triggers — CVSS 8.1 vector from XlabAI/Tencent Xuanwu report)
|
||||
- **Test reference** (glob): `tests/e2e/test_e2e_csrf.py::TestStateChangingEndpointsRejectGet::test_get_is_rejected` (1 function × 13 parametrized invocations — post-WI-HH)
|
||||
- **Test reference** (legacy): `tests/e2e/test_e2e_csrf_legacy.py::TestLegacyStateChangingEndpointsRejectGet::test_get_is_rejected` (1 function × 13 parametrized invocations — `queue/batch` replaces `queue/task`; the 3 dual-purpose endpoints are covered via the read-path under CSRF-M3)
|
||||
|
||||
### Goal CSRF-M2 — POST counterparts still succeed (positive control)
|
||||
- **Precondition**: ComfyUI running; clean snapshot state
|
||||
- **Action**: POST `/v2/manager/queue/reset` (no-op reset), then POST `/v2/snapshot/save` (creates a snapshot, cleaned up via `/v2/snapshot/remove`)
|
||||
- **Observable**: HTTP 200 response on both POSTs; on save, the new entry is observable via `/v2/snapshot/getlist`
|
||||
- **Assertion**: `status_code == 200` for both POSTs; cleanup removes the just-created snapshot
|
||||
- **Negative check**: the CSRF fix must NOT break the legitimate POST path (regression guard — functional equivalence of the converted endpoints must hold)
|
||||
- **Test reference** (glob): `tests/e2e/test_e2e_csrf.py::TestCsrfPostWorks` (2 functions: `test_queue_reset_post_works`, `test_snapshot_save_post_works`)
|
||||
- **Test reference** (legacy): `tests/e2e/test_e2e_csrf_legacy.py::TestLegacyCsrfPostWorks` (2 functions — same endpoints, legacy server fixture)
|
||||
|
||||
### Goal CSRF-M3 — Read-only endpoints still allow GET (negative control, 11 endpoints)
|
||||
- **Precondition**: ComfyUI running
|
||||
- **Action**: HTTP GET to 11 read-only endpoints: `/v2/manager/version`, `/v2/manager/db_mode`, `/v2/manager/policy/update`, `/v2/manager/channel_url_list`, `/v2/manager/queue/status`, `/v2/manager/queue/history_list`, `/v2/manager/is_legacy_manager_ui`, `/v2/customnode/installed`, `/v2/snapshot/getlist`, `/v2/snapshot/get_current`, `/v2/comfyui_manager/comfyui_versions`
|
||||
- **Observable**: HTTP status code
|
||||
- **Assertion**: `status_code == 200` for every read-only path
|
||||
- **Negative check**: the CSRF fix must NOT over-correct by making pure-read endpoints POST-only (would break UI flows that `<img>`-safe-read via GET). Note: the 3 dual-purpose endpoints (`db_mode`, `policy/update`, `channel_url_list`) appear in BOTH CSRF-M1 (write path requires POST; plain GET is rejected) AND CSRF-M3 (read path still answers GET 200); this dual appearance is intentional — they were split into GET (read) + POST (write) by 99caef55.
|
||||
- **Test reference** (glob): `tests/e2e/test_e2e_csrf.py::TestCsrfReadEndpointsStillAllowGet::test_get_read_endpoint_succeeds` (1 function × 11 parametrized invocations)
|
||||
- **Test reference** (legacy): `tests/e2e/test_e2e_csrf_legacy.py::TestLegacyCsrfReadEndpointsStillAllowGet::test_get_read_endpoint_succeeds` (1 function × 11 parametrized invocations — same endpoint list)
|
||||
|
||||
## 10.2 Out-of-Scope CSRF Layers (tracked for future verification)
|
||||
- **Origin/Referer validation** — `origin_only_middleware` (aiohttp layer); not exercised by `test_e2e_csrf.py`
|
||||
- **Same-site cookie enforcement** — browser-layer concern; not server-testable in isolation
|
||||
- **Anti-CSRF token verification** — not implemented in current codebase
|
||||
- **Cross-site form POST defense** — subsumed by Origin validation above
|
||||
|
||||
These remain Goals for future work; do not infer coverage from Goals CSRF-M1/M2/M3.
|
||||
|
||||
---
|
||||
*End of Verification Design Report*
|
||||
Reference in New Issue
Block a user