mirror of
https://github.com/Comfy-Org/ComfyUI-Manager.git
synced 2025-12-24 22:00:49 +08:00
43647249cf
Remove package-level caching in cnr_utils and node_package modules to enable proper dynamic custom node installation and version switching without ComfyUI server restarts. Key Changes: - Remove @lru_cache decorators from version-sensitive functions - Remove cached_property from NodePackage for dynamic state updates - Add comprehensive test suite with parallel execution support - Implement version switching tests (CNR ↔ Nightly) - Add case sensitivity integration tests - Improve error handling and logging API Priority Rules (manager_core.py:1801): - Enabled-Priority: Show only enabled version when both exist - CNR-Priority: Show only CNR when both CNR and Nightly are disabled - Prevents duplicate package entries in /v2/customnode/installed API - Cross-match using cnr_id and aux_id for CNR ↔ Nightly detection Test Infrastructure: - 8 test files with 59 comprehensive test cases - Parallel test execution across 5 isolated environments - Automated test scripts with environment setup - Configurable timeout (60 minutes default) - Support for both master and dr-support-pip-cm branches Bug Fixes: - Fix COMFYUI_CUSTOM_NODES_PATH environment variable export - Resolve test fixture regression with module-level variables - Fix import timing issues in test configuration - Register pytest integration marker to eliminate warnings - Fix POSIX compliance in shell scripts (((var++)) → $((var + 1))) Documentation: - CNR_VERSION_MANAGEMENT_DESIGN.md v1.0 → v1.1 with API priority rules - Add test guides and execution documentation (TESTING_PROMPT.md) - Add security-enhanced installation guide - Create CLI migration guides and references - Document package version management 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
183 lines
4.5 KiB
Markdown
183 lines
4.5 KiB
Markdown
# ComfyUI Manager Test Suite
|
|
|
|
Comprehensive test suite for ComfyUI Manager with parallel execution support.
|
|
|
|
## Quick Start
|
|
|
|
### Fastest Way: Automated Testing
|
|
|
|
```bash
|
|
./tests/run_automated_tests.sh
|
|
```
|
|
|
|
**What it does**:
|
|
- Cleans environment and stops old processes
|
|
- Sets up 10 parallel test environments
|
|
- Runs all 43 tests in ~2 minutes
|
|
- Generates comprehensive report
|
|
|
|
**Expected**: 100% pass rate, ~140-160s execution time, 9x+ speedup
|
|
|
|
### For Claude Code Users
|
|
|
|
Load the testing prompt:
|
|
```
|
|
@tests/TESTING_PROMPT.md
|
|
```
|
|
|
|
Claude Code will automatically execute tests and provide intelligent analysis.
|
|
|
|
## Test Suite Overview
|
|
|
|
### Coverage (54 Tests)
|
|
- **Queue Task API** (8 tests) - Install, uninstall, version switching
|
|
- **Version Switching** (19 tests) - CNR↔Nightly, upgrades, downgrades
|
|
- **Enable/Disable API** (5 tests) - Package activation
|
|
- **Update API** (4 tests) - Package updates
|
|
- **Installed API** (4 tests) - Package listing, original case preservation
|
|
- **Case Sensitivity** (2 tests) - Case-insensitive lookup, full workflow
|
|
- **Complex Scenarios** (12 tests) - Multi-version state, automatic switching
|
|
|
|
### Performance
|
|
- **Execution**: ~140-160s (2.3-2.7 minutes)
|
|
- **Parallel**: 10 environments
|
|
- **Speedup**: 9x+ vs sequential
|
|
- **Load Balance**: 1.2x variance (excellent)
|
|
|
|
## Manual Execution
|
|
|
|
### Parallel Testing (Recommended)
|
|
|
|
```bash
|
|
# Setup (one-time)
|
|
export NUM_ENVS=10
|
|
./tests/setup_parallel_test_envs.sh
|
|
|
|
# Run tests
|
|
./tests/run_parallel_tests.sh
|
|
```
|
|
|
|
### Single Environment Testing
|
|
|
|
```bash
|
|
# Setup
|
|
./tests/setup_test_env.sh
|
|
|
|
# Run tests
|
|
cd tests/env
|
|
python ComfyUI/main.py --enable-manager &
|
|
sleep 20
|
|
pytest ../glob/
|
|
```
|
|
|
|
## Adding New Tests
|
|
|
|
When adding 3+ new tests or modifying test execution time significantly:
|
|
|
|
```bash
|
|
# 1. Write your tests in tests/glob/
|
|
|
|
# 2. Run tests and check load balance
|
|
./tests/run_automated_tests.sh
|
|
# Look for "Load Balance: X.XXx variance" in report
|
|
|
|
# 3. If variance > 2.0x, update durations
|
|
./tests/update_test_durations.sh # Takes ~15-20 min
|
|
|
|
# 4. Commit duration data
|
|
git add .test_durations
|
|
git commit -m "chore: update test duration data"
|
|
```
|
|
|
|
**See**: `glob/TESTING_GUIDE.md` for detailed workflow
|
|
|
|
## Files
|
|
|
|
- `run_automated_tests.sh` - One-command test execution
|
|
- `run_parallel_tests.sh` - Parallel test runner
|
|
- `setup_parallel_test_envs.sh` - Environment setup
|
|
- `update_test_durations.sh` - Update load balancing data
|
|
- `TESTING_PROMPT.md` - Claude Code automation
|
|
- `glob/` - Test implementations
|
|
- `glob/TESTING_GUIDE.md` - Development workflow guide
|
|
|
|
## Requirements
|
|
|
|
- Python 3.12+
|
|
- Virtual environment: `/home/rho/venv`
|
|
- ComfyUI branch: `ltdrdata/dr-support-pip-cm`
|
|
- Ports: 8188-8197 available
|
|
|
|
## Troubleshooting
|
|
|
|
### Tests Fail to Start
|
|
|
|
```bash
|
|
# Stop existing processes
|
|
pkill -f "ComfyUI/main.py"
|
|
sleep 2
|
|
|
|
# Re-run
|
|
./tests/run_automated_tests.sh
|
|
```
|
|
|
|
### Slow Execution
|
|
|
|
If tests take >3 minutes, update duration data:
|
|
```bash
|
|
./tests/update_test_durations.sh
|
|
```
|
|
|
|
### Environment Issues
|
|
|
|
Rebuild test environments:
|
|
```bash
|
|
rm -rf tests/env/ComfyUI_*
|
|
NUM_ENVS=10 ./tests/setup_parallel_test_envs.sh
|
|
```
|
|
|
|
## Generated Files
|
|
|
|
- **Report**: `.claude/livecontext/automated_test_*.md`
|
|
- **Logs**: `tests/tmp/test-results-[1-10].log`
|
|
- **Server Logs**: `tests/tmp/comfyui-parallel-[1-10].log`
|
|
|
|
## CI/CD Integration
|
|
|
|
```yaml
|
|
- name: Run Tests
|
|
run: |
|
|
source /home/rho/venv/bin/activate
|
|
./tests/run_automated_tests.sh
|
|
```
|
|
|
|
Exit code: 0 = pass, 1 = fail
|
|
|
|
---
|
|
|
|
**Status**: ✅ Production-ready (100% pass rate, <3min execution)
|
|
|
|
## Recent Fixes (2025-11-06)
|
|
|
|
### Fixed Test Failures
|
|
|
|
#### test_case_sensitivity_full_workflow
|
|
- **Issue**: HTTP 405 error - incorrect API endpoint usage
|
|
- **Root Cause**: Using non-existent `/customnode/install` endpoint
|
|
- **Fix**: Migrated to queue API (`/v2/manager/queue/task` + `/v2/manager/queue/start`)
|
|
- **Changes**: Updated request parameters (`"id"`, `"version"`, `"selected_version"`)
|
|
- **Location**: `tests/glob/test_case_sensitivity_integration.py:65-200`
|
|
|
|
#### test_enable_package
|
|
- **Issue**: AssertionError - found 2 disabled packages instead of 1
|
|
- **Root Cause**: Leftover disabled packages from previous parallel tests
|
|
- **Fix**: Added cleanup before test execution (not just after)
|
|
- **Changes**: Created `_cleanup()` helper, added filesystem sync delay
|
|
- **Location**: `tests/glob/test_enable_disable_api.py:56-111`
|
|
|
|
### Improvements
|
|
- Increased wait times for parallel execution reliability (20s → 30s)
|
|
- Added queue status checking for better debugging
|
|
- Enhanced fixture cleanup with filesystem sync delays
|
|
- Both tests now pass consistently in parallel execution
|