wangbo/ComfyUI-Manager
1
0
Fork 0
mirror of https://github.com/Comfy-Org/ComfyUI-Manager.git synced 2026-05-09 16:42:32 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
01799f8cac
ComfyUI-Manager/reports/coverage_gaps.md
Dr.Lt.Data 4410ebc6a6
Some checks are pending
Publish to PyPI / build-and-publish (push) Waiting to run
Details
Python Linting / Run Ruff (push) Waiting to run
Details
fix(security): harden CSRF with Content-Type gate and expand E2E coverage (#2818) 
Defense-in-depth over GET→POST alone: reject the three CORS-safelisted
simple-form Content-Types (x-www-form-urlencoded, multipart/form-data,
text/plain) on 16 no-body POST handlers (glob + legacy) to block
<form method=POST> CSRF that bypasses method-only gating. Move
comfyui_switch_version to a JSON body so the preflight requirement applies.
Split db_mode/policy/update/channel_url_list into GET(read) + POST(write).
Tighten do_fix (high → high+) and gate three previously-ungated config
setters at middle. Resynchronize openapi.yaml (27 paths, 30 operations,
ComfyUISwitchVersionParams as a shared $ref component). Add E2E harness
variants, Playwright config, CSRF/secgate suites, 39-endpoint coverage,
and a CHANGELOG.

Breaking: legacy per-op POST routes (install/uninstall/fix/disable/update/
reinstall/abort_current) are removed; callers already use queue/batch.
Legacy /manager/notice (v1) is removed; /v2/manager/notice is retained.

Reported-by: XlabAI Team of Tencent Xuanwu Lab
CVSS: 8.1 (AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:H/A:H)
2026-04-22 05:04:30 +09:00

204 lines
16 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Coverage Gap Analysis — Report A × Report B
**Generated**: 2026-04-18 (WI-AA inventory update 2026-04-19: +2 LB1/LB2 tests → 119 total; WI-GG update 2026-04-20: +26 legacy CSRF parametrized rows → 145 total; WI-JJ update 2026-04-20: +3 legacy CSRF parity/install rows; WI-LL update 2026-04-20: +2 SECGATE rows → 148 audit rows / 126 test functions)
**Inputs**: `reports/endpoint_scenarios.md` (39 handlers, 154 scenarios) + `reports/e2e_test_coverage.md` (126 test functions / 148 audit rows — the row-count delta reflects audit-side per-invocation rendering of legacy CSRF plus the 2 new SECGATE PoC rows; function-level progression is recorded in `e2e_test_coverage.md`)
> **WI-AA (2026-04-19)**: `tests/playwright/legacy-ui-install.spec.ts` (2 tests: LB1 Install button triggers install effect, LB2 Uninstall button triggers uninstall effect) has been **integrated into the audit** (see `e2e_verification_audit.md` §16). These tests drive the install/uninstall action via the Custom Nodes Manager dialog UI and verify the resulting backend state via `/v2/customnode/installed`. They close the long-standing gap noted in this document's Section 4 for UI-driven install/uninstall effect coverage on the legacy UI. Prior coverage-gap mentions of "missing UI→effect for install/uninstall buttons" are now RESOLVED.
>
> **WI-GG (2026-04-20)**: `tests/e2e/test_e2e_csrf_legacy.py` (4 test functions / 26 parametrized invocations: 13 reject-GET + 2 POST-works + 11 allow-GET) — from WI-FF — has been **integrated into the audit** (see `e2e_verification_audit.md` §19). This extends the CSRF-Mitigation Layer Coverage block below from glob-only to glob + legacy, closing the regression-guard gap that a legacy-side `@routes.post` revert would have slipped past CI. LB1/LB2 classification as RESOLVED is unchanged.
>
> **WI-LL (2026-04-20)**: `tests/e2e/test_e2e_secgate_strict.py` (SR4 PoC — strict-mode fixture) + `tests/e2e/test_e2e_secgate_default.py` (CV4 demo — no harness needed) — from WI-KK — have been **integrated into the audit** (see `e2e_verification_audit.md` §20, §21). Two of the original 8 T2 SECGATE-PENDING Goals are now RESOLVED (SR4, CV4); the remaining 6 are reclassified into 4 sub-tiers (T2-pending-harness-ready: SR6/V5/UA2; NORMAL-legacy: LGU2/LPP2; T2-TASKLEVEL: IM4). Section 4 🟢 Low Priority "Security level 403 gates ... impractical in standard E2E env" is now PARTIAL — see the classification policy + propagation plan in `e2e_verification_audit.md`.
## Summary
| Metric | Count |
|---|---:|
| Glob v2 endpoints fully covered (row has no ✗ in Missing scenarios) | 15/30 |
| Glob v2 endpoints partially covered (some ✓ + some ✗) | 14/30 |
| Glob v2 endpoints NOT covered (positive) | 1/30 |
| Legacy-only endpoints fully covered | 0/9 |
| Legacy-only endpoints partially covered (indirect only) | 4/9 |
| Legacy-only endpoints NOT covered | 5/9 |
| Orphan tests (non-endpoint-direct; pytest + Playwright UI-only) | 29 |
---
# Section 1 — Endpoint × Test Coverage Matrix (Glob v2)
Legend: **✓** direct assertion test, **~** indirect / partial, **✗** not tested
| # | Endpoint | Direct test | Missing scenarios |
|---|---|---|---|
| 1 | POST queue/task (install) | ✓ test_e2e_endpoint, test_e2e_git_clone, test_e2e_task_operations | ✗ 400 ValidationError (bad kind/schema), ✗ 500 on malformed JSON |
| 2 | POST queue/task (uninstall) | ✓ test_e2e_endpoint, test_e2e_task_operations | (shared with #1) |
| 3 | POST queue/task (update/fix/disable/enable) | ✓ test_e2e_task_operations | (shared) |
| 4 | GET queue/history_list | ✓ test_e2e_queue_lifecycle | ✗ 400 on inaccessible history path |
| 5 | GET queue/history | ✓ test_e2e_queue_lifecycle, test_e2e_task_operations | ✗ `id=<batch_id>` file-based query, ✗ path traversal rejection |
| 6 | GET customnode/getmappings | ✓ test_e2e_customnode_info | ✗ `mode=nickname`, ✗ missing `mode` KeyError→500 |
| 7 | GET customnode/fetch_updates | ✓ test_e2e_customnode_info (410) | (fully covered — deprecated endpoint) |
| 8 | POST queue/update_all | ✓ test_e2e_task_operations | ✗ 403 security gate, ✗ `mode=local` vs remote distinction |
| 9 | GET is_legacy_manager_ui | ✓ test_e2e_system_info, playwright navigation | (fully covered) |
| 10 | GET customnode/installed | ✓ test_e2e_endpoint, test_e2e_customnode_info (both modes) | (fully covered) |
| 11 | GET snapshot/getlist | ✓ test_e2e_snapshot_lifecycle, playwright snapshot | (fully covered) |
| 12 | POST snapshot/remove | ✓ test_e2e_snapshot_lifecycle | ✗ path traversal "Invalid target" 400, ✗ 403 security gate, ✗ missing `target` query |
| 13 | POST snapshot/restore | ✗ **intentionally skipped (destructive)** | ALL scenarios (positive, path traversal, 403) |
| 14 | GET snapshot/get_current | ✓ test_e2e_snapshot_lifecycle | (fully covered) |
| 15 | POST snapshot/save | ✓ test_e2e_snapshot_lifecycle, playwright snapshot | (fully covered) |
| 16 | POST customnode/import_fail_info | ✓ test_e2e_customnode_info (negative only) | ✗ positive path (returning actual failure info) — requires seed failed import |
| 17 | POST customnode/import_fail_info_bulk | ✓ test_e2e_customnode_info | ✗ positive path with real failure info |
| 18 | POST queue/reset | ✓ test_e2e_queue_lifecycle | (fully covered) |
| 19 | GET queue/status | ✓ test_e2e_queue_lifecycle | (fully covered) |
| 20 | POST queue/start | ✓ test_e2e_queue_lifecycle (idle + lifecycle) | (fully covered) |
| 21 | POST queue/update_comfyui | ✓ test_e2e_task_operations | (fully covered) |
| 22 | GET comfyui_versions | ✓ test_e2e_version_mgmt (4 tests) | ✗ 400 on git-access failure |
| 23 | POST comfyui_switch_version | ✓ test_e2e_version_mgmt (negative only) | ✗ **positive path (intentionally skipped)**, ✗ 403 security gate |
| 24 | POST queue/install_model | ✓ test_e2e_task_operations | (fully covered) |
| 25 | GET db_mode | ✓ test_e2e_config_api, playwright manager-menu | (fully covered) |
| 26 | POST db_mode | ✓ test_e2e_config_api, playwright manager-menu | ✗ missing `value` KeyError→400 (only malformed JSON tested) |
| 27 | GET policy/update | ✓ test_e2e_config_api, playwright | (fully covered) |
| 28 | POST policy/update | ✓ test_e2e_config_api, playwright | ✗ missing `value` key |
| 29 | GET channel_url_list | ✓ test_e2e_config_api | (fully covered) |
| 30 | POST channel_url_list | ✓ test_e2e_config_api | ✗ unknown channel name (silent no-op) |
| 31 | POST manager/reboot | ✓ test_e2e_system_info | ✗ __COMFY_CLI_SESSION__ env branch |
| 32 | GET manager/version | ✓ test_e2e_system_info, playwright navigation | (fully covered) |
Note: queue/task has 3 rows above per kind; 30 glob endpoints = 32 row entries (queue/task counted per-kind).
## Glob v2 Summary
- **Fully covered** (row has no ✗ in Missing scenarios column): 15 endpoints
(is_legacy_manager_ui, customnode/installed, snapshot/getlist, snapshot/get_current, snapshot/save, queue/reset, queue/status, queue/start, queue/update_comfyui, queue/install_model, fetch_updates, get db_mode, get policy/update, get channel_url_list, get manager/version)
- **Partially covered** (some ✓ + some ✗ in Missing scenarios): 14 endpoints (row-level collapse of per-kind queue/task into 1 endpoint)
- **Intentionally skipped** (destructive, counted under NOT covered): 1 (snapshot/restore); switch_version has ✓ negative + skipped-positive so it falls under partial, not skipped
- Sum: 15 + 14 + 1 = 30 ✓
### CSRF-Mitigation Layer Coverage (separate from positive-path coverage above)
The 16 state-changing POST endpoints — commit 99caef55 converted 12+ of
these from GET→POST (the remainder such as queue/task, import_fail_info,
and import_fail_info_bulk were already POST but are included for contract
completeness) — are independently covered. Commit 99caef55 applied the
conversion to BOTH `comfyui_manager/glob/manager_server.py` (~91 lines)
and `comfyui_manager/legacy/manager_server.py` (~92 lines), so two test
files are required (the server-loading is mutex on `--enable-manager-legacy-ui`):
**Glob server**: `tests/e2e/test_e2e_csrf.py` (4 functions / 26 parametrized invocations — post-WI-HH; was 29 before the 3 dual-purpose endpoints were scoped out of the reject-GET fixture)
| Contract | Test | Coverage |
|---|---|---|
| Reject GET on 13 state-changing POST endpoints (glob; post-WI-HH) | TestStateChangingEndpointsRejectGet (parametrized ×13) | ✓ full |
| POST counterpart sanity (glob) | TestCsrfPostWorks (2 tests) | ~ spot-check (queue/reset + snapshot/save only) |
| Read-only GET still allowed — negative control (glob) | TestCsrfReadEndpointsStillAllowGet (parametrized ×11) | ✓ full |
**Legacy server** (WI-FF, audit-integrated in WI-GG): `tests/e2e/test_e2e_csrf_legacy.py` (4 functions / 26 parametrized invocations)
| Contract | Test | Coverage |
|---|---|---|
| Reject GET on 13 state-changing POST endpoints (legacy; queue/task→queue/batch, dual-purpose endpoints scoped to ALLOW-GET only) | TestLegacyStateChangingEndpointsRejectGet (parametrized ×13) | ✓ full |
| POST counterpart sanity (legacy) | TestLegacyCsrfPostWorks (2 tests — queue/reset + snapshot/save) | ~ spot-check |
| Read-only GET still allowed — negative control (legacy) | TestLegacyCsrfReadEndpointsStillAllowGet (parametrized ×11) | ✓ full |
**Important**: POST `snapshot/restore` and POST `comfyui_switch_version` are
listed as "intentionally skipped (destructive)" for POSITIVE-path coverage,
but their CSRF reject-GET contract IS covered by BOTH test files —
the destructive-skip only applies to the success-path assertion, not to
the security layer. The legacy-side coverage closes the regression-guard
gap that a revert of any legacy `@routes.post` back to `@routes.get` would
otherwise have slipped past CI.
---
# Section 2 — Endpoint × Test Coverage Matrix (Legacy-only)
| # | Endpoint | Test coverage | Gap |
|---|---|---|---|
| 1 | POST queue/batch | ~ debug-install-flow captures API sequence indirectly | ✗ No dedicated assertion test for batch semantics |
| 2 | GET customnode/getlist | ~ playwright custom-nodes triggers it via UI | ✗ No direct assertion on response shape, `skip_update` param, channel resolution |
| 3 | GET /customnode/alternatives | ✗ NOT COVERED | ALL scenarios |
| 4 | GET externalmodel/getlist | ~ playwright model-manager triggers via UI | ✗ No direct assertion on `installed` flag population, save_path resolution |
| 5 | GET customnode/versions/{node_name} | ~ debug-install-flow captures it | ✗ 400 on unknown pack, no direct test |
| 6 | GET customnode/disabled_versions/{node_name} | ✗ NOT COVERED | ALL scenarios |
| 7 | POST customnode/install/git_url | ✗ NOT COVERED | ALL (high+ security, may be intentional) |
| 8 | POST customnode/install/pip | ✗ NOT COVERED | ALL (high+ security, may be intentional) |
| 9 | GET manager/notice | ✗ NOT COVERED | ALL scenarios |
## Legacy-only Summary
- **Fully covered**: 0/9
- **Indirect-only** (triggered via UI flow but no direct assertion): 4/9
- **Not covered**: 5/9
---
# Section 3 — Orphan Tests (not mapped to HTTP endpoints)
Tests that do not directly assert HTTP endpoint behavior:
| Test file | Tests | Purpose |
|---|---:|---|
| `tests/cli/test_uv_compile.py` | 8 | `cm-cli --uv-compile` CLI entrypoint (not HTTP). Relocated from `tests/e2e/` in WI-PP (see Recommendations §4 — now ACTIONED). |
| `test_e2e_endpoint.py::test_startup_resolver_ran` | 1 | Log file assertion (server log contains UnifiedDepResolver) |
| `test_e2e_endpoint.py::test_comfyui_started` | 1 | `/system_stats` (ComfyUI core endpoint, not Manager) |
| `test_e2e_git_clone.py::test_02_no_module_error` | 1 | Log file regression check |
| Playwright UI-only tests (no API assertion) | ~14 of 22 | UI rendering, dialog lifecycle, filter/search — no HTTP assertion |
Total orphan tests (non-endpoint-direct): ~29 / 115 (25%)
---
# Section 4 — Critical Gaps (Prioritized)
## 🔴 High Priority — Legacy Endpoints with ZERO UI→effect Test Coverage
These endpoints ARE actively called by legacy UI JavaScript but have no Playwright test exercising the UI flow:
| Endpoint | JS call site | Missing UI→effect test |
|---|---|---|
| POST /v2/customnode/install/git_url | `common.js:248` | "Install via Git URL" button flow |
| POST /v2/customnode/install/pip | `common.js:213` | pip install UI flow |
| GET /v2/customnode/disabled_versions/{node_name} | `custom-nodes-manager.js:1401` | Node row "Disabled Versions" dropdown |
| GET /customnode/alternatives | `custom-nodes-manager.js:1885` | Alternatives display in custom nodes dialog |
| GET /v2/manager/notice | `comfyui-manager.js:418` | Notice display on Manager menu open |
→ These are **NOT dead code** — they require **UI→effect tests added**, not removal.
## 🟡 Medium Priority — Missing Scenarios (Covered Endpoints)
1. **queue/task 400 ValidationError** — no test verifies schema rejection. Adding `test_queue_task_invalid_body` with malformed `kind` value would be trivial.
2. **queue/history with `id=<batch_id>` + path traversal** — file-based history path not exercised.
3. **snapshot/remove path traversal** — security-critical but not asserted.
4. **comfyui_versions 400 on git failure** — would need to simulate git unavailable.
5. **POST db_mode/policy/update missing `value` key** — currently only tests malformed JSON; KeyError path untested.
## 🟢 Low Priority — Acceptable Gaps
1. **snapshot/restore + switch_version positive** — intentionally skipped (destructive). Acceptable.
2. ~~**Security level 403 gates** — require running with locked-down security_level; impractical in standard E2E env.~~ **PARTIAL (WI-LL via WI-KK, 2026-04-20)**: SR4 (snapshot/remove middle) + CV4 (comfyui_switch_version high+) are now covered via `test_e2e_secgate_strict.py` and `test_e2e_secgate_default.py` respectively. Remaining SECGATE Goals are reclassified (`e2e_verification_audit.md` classification-policy block): SR6/V5/UA2 are T2-pending-harness-ready (mechanical additions to strict.py); LGU2/LPP2 are NORMAL-legacy (needs `start_comfyui_legacy.sh`); IM4 is T2-TASKLEVEL (queue-observation pattern, not HTTP 403).
3. **import_fail_info positive path** — would require seeding a failed module import; complex setup.
---
# Section 5 — Recommendations
1. **Add UI→effect Playwright tests** for the 5 JS-called legacy endpoints (install/git_url, install/pip, disabled_versions, alternatives, manager/notice). All are live in legacy UI flows.
2. **Add minimal gap tests** for queue/task ValidationError + snapshot/remove path traversal — both are security-relevant.
3. **Convert debug-install-flow.spec.ts** from logging-only into an assertion test (verify queue/batch payload structure + cm-queue-status WebSocket events).
4. ~~**Consider moving uv_compile tests** to a separate CLI test directory (tests/cli/) — they are not E2E HTTP tests.~~ **ACTIONED (WI-PP)**: file now lives at `tests/cli/test_uv_compile.py`. CI workflow `.github/workflows/e2e.yml` updated to the new path. Placement hygiene restored.
---
# Section 6 — Cross-Report Consistency Check
| Report A claim | Report B claim | Status |
|---|---|---|
| 30 glob v2 endpoints | Row-level: 15 fully + 14 partial + 1 NOT covered = 30 (matches Summary L10-12) | ✓ consistent under row-no-✗ definition |
| 9 legacy-only endpoints | "4 covered, 5 not covered" | ✓ consistent |
| 154 scenarios total | (per-test scenario breakdown) | ⚠️ scenario-level count not aggregated in Report B; recommend adding |
| Security gates (middle/middle+/high+) | Security 403 paths not tested | ⚠️ gap confirmed |
| Deprecated endpoints flagged | fetch_updates 410 tested | ✓ consistent |
Internal accounting reconciled under the single "row has no ✗ in Missing scenarios column" definition for "fully covered". Earlier drafts of this report mixed three counting schemes (Summary 24/5/1, body-list 11/15/2, Section 6 27/30) that did not agree; this revision uses the row-level count uniformly (15/14/1 = 30 glob v2 endpoints) and aligns Summary L10-12, body L63-66, and Section 6 L172. Report A (endpoint_scenarios.md) and Report B (e2e_test_coverage.md) align on endpoint counts (30 glob v2 + 9 legacy = 39) and coverage categories under this definition.
---
*End of Coverage Gap Analysis*
Reference in New Issue View Git Blame Copy Permalink