mirror of
https://github.com/Comfy-Org/ComfyUI-Manager.git
synced 2025-12-28 15:50:52 +08:00
2866193baf
Add comprehensive pip dependency conflict resolution framework as draft implementation. This is self-contained and does not affect existing ComfyUI Manager functionality. Key components: - pip_util.py with PipBatch class for policy-driven package management - Lazy-loaded policy system supporting base + user overrides - Multi-stage policy execution (uninstall → apply_first_match → apply_all_matches → restore) - Conditional policies based on platform, installed packages, and ComfyUI version - Comprehensive test suite covering edge cases, workflows, and platform scenarios - Design and implementation documentation Policy capabilities (draft): - Package replacement (e.g., PIL → Pillow, opencv-python → opencv-contrib-python) - Version pinning to prevent dependency conflicts - Dependency protection during installations - Platform-specific handling (Linux/Windows, GPU detection) - Pre-removal and post-restoration workflows Testing infrastructure: - Pytest-based test suite with isolated environments - Dependency analysis tools for conflict detection - Coverage for policy priority, edge cases, and environment recovery Status: Draft implementation complete, integration with manager workflows pending.
434 lines
14 KiB
Markdown
434 lines
14 KiB
Markdown
# Test Code Improvements Based on Dependency Context
|
|
|
|
**Date**: 2025-10-01
|
|
**Basis**: DEPENDENCY_TREE_CONTEXT.md analysis
|
|
|
|
This document summarizes all test improvements made using verified dependency tree information.
|
|
|
|
---
|
|
|
|
## Summary of Changes
|
|
|
|
### Tests Enhanced
|
|
|
|
| Test File | Tests Modified | Tests Added | Total Tests |
|
|
|-----------|----------------|-------------|-------------|
|
|
| `test_dependency_protection.py` | 2 | 2 | 4 |
|
|
| `test_environment_recovery.py` | 2 | 0 | 2 |
|
|
| **Total** | **4** | **2** | **6** |
|
|
|
|
### Test Results
|
|
|
|
```bash
|
|
$ pytest test_dependency_protection.py test_environment_recovery.py -v
|
|
|
|
test_dependency_protection.py::test_dependency_version_protection_with_pin PASSED
|
|
test_dependency_protection.py::test_dependency_chain_with_six_pin PASSED
|
|
test_dependency_protection.py::test_pin_only_affects_specified_packages PASSED ✨ NEW
|
|
test_dependency_protection.py::test_major_version_jump_prevention PASSED ✨ NEW
|
|
test_environment_recovery.py::test_package_deletion_and_restore PASSED
|
|
test_environment_recovery.py::test_version_change_and_restore PASSED
|
|
|
|
6 passed in 14.10s
|
|
```
|
|
|
|
---
|
|
|
|
## Detailed Improvements
|
|
|
|
### 1. test_dependency_version_protection_with_pin
|
|
|
|
**File**: `test_dependency_protection.py:34-94`
|
|
|
|
**Enhancements**:
|
|
- ✅ Added exact version assertions based on DEPENDENCY_TREE_CONTEXT.md
|
|
- ✅ Verified initial versions: urllib3==1.26.15, certifi==2023.7.22, charset-normalizer==3.2.0
|
|
- ✅ Added verification that idna is NOT pre-installed
|
|
- ✅ Added assertion that idna==3.10 is installed as NEW dependency
|
|
- ✅ Verified requests==2.32.5 is installed
|
|
- ✅ Added detailed error messages explaining what versions are expected and why
|
|
|
|
**Key Assertions Added**:
|
|
```python
|
|
# Verify expected OLD versions
|
|
assert initial_urllib3 == "1.26.15", f"Expected urllib3==1.26.15, got {initial_urllib3}"
|
|
assert initial_certifi == "2023.7.22", f"Expected certifi==2023.7.22, got {initial_certifi}"
|
|
assert initial_charset == "3.2.0", f"Expected charset-normalizer==3.2.0, got {initial_charset}"
|
|
|
|
# Verify idna is NOT installed initially
|
|
assert "idna" not in initial, "idna should not be pre-installed"
|
|
|
|
# Verify new dependency was added (idna is NOT pinned, so it gets installed)
|
|
assert "idna" in final_packages, "idna should be installed as new dependency"
|
|
assert final_packages["idna"] == "3.10", f"Expected idna==3.10, got {final_packages['idna']}"
|
|
```
|
|
|
|
**Based on Context**:
|
|
- DEPENDENCY_TREE_CONTEXT.md Section 1: requests → Dependencies
|
|
- Verified: Without pin, urllib3 would upgrade to 2.5.0 (MAJOR version jump)
|
|
- Verified: idna is NEW dependency (not in requirements-test-base.txt)
|
|
|
|
---
|
|
|
|
### 2. test_dependency_chain_with_six_pin
|
|
|
|
**File**: `test_dependency_protection.py:117-162`
|
|
|
|
**Enhancements**:
|
|
- ✅ Added exact version assertion for six==1.16.0
|
|
- ✅ Added exact version assertion for python-dateutil==2.9.0.post0
|
|
- ✅ Added detailed error messages
|
|
- ✅ Added docstring reference to DEPENDENCY_TREE_CONTEXT.md
|
|
|
|
**Key Assertions Added**:
|
|
```python
|
|
# Verify expected OLD version
|
|
assert initial_six == "1.16.0", f"Expected six==1.16.0, got {initial_six}"
|
|
|
|
# Verify final versions
|
|
assert final_packages["python-dateutil"] == "2.9.0.post0", f"Expected python-dateutil==2.9.0.post0"
|
|
assert final_packages["six"] == "1.16.0", "six should remain at 1.16.0 (prevented 1.17.0 upgrade)"
|
|
```
|
|
|
|
**Based on Context**:
|
|
- DEPENDENCY_TREE_CONTEXT.md Section 2: python-dateutil → Dependencies
|
|
- Verified: six is a REAL dependency (not optional like colorama)
|
|
- Verified: Without pin, six would upgrade from 1.16.0 to 1.17.0
|
|
|
|
---
|
|
|
|
### 3. test_pin_only_affects_specified_packages ✨ NEW
|
|
|
|
**File**: `test_dependency_protection.py:165-208`
|
|
|
|
**Purpose**: Verify that pin is selective, not global
|
|
|
|
**Test Logic**:
|
|
1. Verify idna is NOT pre-installed
|
|
2. Verify requests is NOT pre-installed
|
|
3. Install requests with pin policy (only pins urllib3, certifi, charset-normalizer)
|
|
4. Verify idna was installed at latest version (3.10) - NOT pinned
|
|
5. Verify requests was installed at expected version (2.32.5)
|
|
|
|
**Key Assertions**:
|
|
```python
|
|
# Verify idna was installed (NOT pinned, so gets latest)
|
|
assert "idna" in final_packages, "idna should be installed as new dependency"
|
|
assert final_packages["idna"] == "3.10", "idna should be at latest version 3.10 (not pinned)"
|
|
```
|
|
|
|
**Based on Context**:
|
|
- DEPENDENCY_TREE_CONTEXT.md: "⚠️ idna is NEW and NOT pinned (acceptable - new dependency)"
|
|
- Verified: Pin only affects specified packages in pinned_packages list
|
|
|
|
---
|
|
|
|
### 4. test_major_version_jump_prevention ✨ NEW
|
|
|
|
**File**: `test_dependency_protection.py:211-271`
|
|
|
|
**Purpose**: Verify that pin prevents MAJOR version jumps with breaking changes
|
|
|
|
**Test Logic**:
|
|
1. Verify initial urllib3==1.26.15
|
|
2. **Test WITHOUT pin**: Uninstall deps, install requests → urllib3 upgrades to 2.x
|
|
3. Verify urllib3 was upgraded to 2.x (starts with "2.")
|
|
4. Reset environment
|
|
5. **Test WITH pin**: Install requests with pin → urllib3 stays at 1.x
|
|
6. Verify urllib3 stayed at 1.26.15 (starts with "1.")
|
|
|
|
**Key Assertions**:
|
|
```python
|
|
# Without pin - verify urllib3 upgrades to 2.x
|
|
assert without_pin["urllib3"].startswith("2."), \
|
|
f"Without pin, urllib3 should upgrade to 2.x, got {without_pin['urllib3']}"
|
|
|
|
# With pin - verify urllib3 stays at 1.x
|
|
assert final_packages["urllib3"] == "1.26.15", \
|
|
"Pin should prevent urllib3 from upgrading to 2.x (breaking changes)"
|
|
assert final_packages["urllib3"].startswith("1."), \
|
|
f"urllib3 should remain at 1.x series, got {final_packages['urllib3']}"
|
|
```
|
|
|
|
**Based on Context**:
|
|
- DEPENDENCY_TREE_CONTEXT.md: "urllib3 1.26.15 → 2.5.0 is a MAJOR version jump"
|
|
- DEPENDENCY_TREE_CONTEXT.md: "urllib3 2.0 removed deprecated APIs"
|
|
- This is the MOST IMPORTANT test - prevents breaking changes
|
|
|
|
---
|
|
|
|
### 5. test_package_deletion_and_restore
|
|
|
|
**File**: `test_environment_recovery.py:33-78`
|
|
|
|
**Enhancements**:
|
|
- ✅ Added exact version assertion for six==1.16.0
|
|
- ✅ Added verification that six is restored to EXACT version (not latest)
|
|
- ✅ Added detailed error messages
|
|
- ✅ Added docstring reference to DEPENDENCY_TREE_CONTEXT.md
|
|
|
|
**Key Assertions Added**:
|
|
```python
|
|
# Verify six is initially installed at expected version
|
|
assert initial["six"] == "1.16.0", f"Expected six==1.16.0, got {initial['six']}"
|
|
|
|
# Verify six was restored to EXACT required version (not latest)
|
|
assert final_packages["six"] == "1.16.0", \
|
|
"six should be restored to exact version 1.16.0 (not 1.17.0 latest)"
|
|
```
|
|
|
|
**Based on Context**:
|
|
- DEPENDENCY_TREE_CONTEXT.md: "six: 1.16.0 (OLD) → 1.17.0 (LATEST)"
|
|
- Verified: Restore policy restores to EXACT version, not latest
|
|
|
|
---
|
|
|
|
### 6. test_version_change_and_restore
|
|
|
|
**File**: `test_environment_recovery.py:105-158`
|
|
|
|
**Enhancements**:
|
|
- ✅ Added exact version assertions (1.26.15 initially, 2.1.0 after upgrade)
|
|
- ✅ Added verification of major version change (1.x → 2.x)
|
|
- ✅ Added verification of major version downgrade (2.x → 1.x)
|
|
- ✅ Added detailed error messages explaining downgrade capability
|
|
- ✅ Added docstring reference to DEPENDENCY_TREE_CONTEXT.md
|
|
|
|
**Key Assertions Added**:
|
|
```python
|
|
# Verify version was changed to 2.x
|
|
assert installed_after["urllib3"] == "2.1.0", \
|
|
f"urllib3 should be upgraded to 2.1.0, got {installed_after['urllib3']}"
|
|
assert installed_after["urllib3"].startswith("2."), \
|
|
"urllib3 should be at 2.x series"
|
|
|
|
# Verify version was DOWNGRADED from 2.x back to 1.x
|
|
assert final["urllib3"] == "1.26.15", \
|
|
"urllib3 should be downgraded to 1.26.15 (from 2.1.0)"
|
|
assert final["urllib3"].startswith("1."), \
|
|
f"urllib3 should be back at 1.x series, got {final['urllib3']}"
|
|
```
|
|
|
|
**Based on Context**:
|
|
- DEPENDENCY_TREE_CONTEXT.md: "urllib3 can upgrade from 1.26.15 (1.x) to 2.5.0 (2.x)"
|
|
- Verified: Restore policy can DOWNGRADE (not just prevent upgrades)
|
|
- Tests actual version downgrade capability (2.x → 1.x)
|
|
|
|
---
|
|
|
|
## Test Coverage Analysis
|
|
|
|
### Before Improvements
|
|
|
|
| Scenario | Coverage |
|
|
|----------|----------|
|
|
| Pin prevents upgrades | ✅ Basic |
|
|
| New dependencies installed | ❌ Not tested |
|
|
| Pin is selective | ❌ Not tested |
|
|
| Major version jump prevention | ❌ Not tested |
|
|
| Exact version restoration | ❌ Not tested |
|
|
| Version downgrade capability | ❌ Not tested |
|
|
|
|
### After Improvements
|
|
|
|
| Scenario | Coverage | Test |
|
|
|----------|----------|------|
|
|
| Pin prevents upgrades | ✅ Enhanced | test_dependency_version_protection_with_pin |
|
|
| New dependencies installed | ✅ Added | test_dependency_version_protection_with_pin |
|
|
| Pin is selective | ✅ Added | test_pin_only_affects_specified_packages |
|
|
| Major version jump prevention | ✅ Added | test_major_version_jump_prevention |
|
|
| Exact version restoration | ✅ Enhanced | test_package_deletion_and_restore |
|
|
| Version downgrade capability | ✅ Enhanced | test_version_change_and_restore |
|
|
|
|
---
|
|
|
|
## Key Testing Principles Applied
|
|
|
|
### 1. Exact Version Verification
|
|
|
|
**Before**:
|
|
```python
|
|
assert final_packages["urllib3"] == initial_urllib3 # Generic
|
|
```
|
|
|
|
**After**:
|
|
```python
|
|
assert initial_urllib3 == "1.26.15", f"Expected urllib3==1.26.15, got {initial_urllib3}"
|
|
assert final_packages["urllib3"] == "1.26.15", "urllib3 should remain at 1.26.15 (prevented 2.x upgrade)"
|
|
```
|
|
|
|
**Benefit**: Fails with clear message if environment setup is wrong
|
|
|
|
---
|
|
|
|
### 2. Version Series Verification
|
|
|
|
**Added**:
|
|
```python
|
|
assert final_packages["urllib3"].startswith("1."), \
|
|
f"urllib3 should remain at 1.x series, got {final_packages['urllib3']}"
|
|
```
|
|
|
|
**Benefit**: Catches major version jumps even if exact version changes
|
|
|
|
---
|
|
|
|
### 3. Negative Testing (Verify NOT Installed)
|
|
|
|
**Added**:
|
|
```python
|
|
assert "idna" not in initial, "idna should not be pre-installed"
|
|
```
|
|
|
|
**Benefit**: Ensures test environment is in expected state
|
|
|
|
---
|
|
|
|
### 4. Context-Based Documentation
|
|
|
|
**Every test now includes**:
|
|
```python
|
|
"""
|
|
Based on DEPENDENCY_TREE_CONTEXT.md:
|
|
<specific section reference>
|
|
<expected behavior from context>
|
|
"""
|
|
```
|
|
|
|
**Benefit**: Links test expectations to verified dependency data
|
|
|
|
---
|
|
|
|
## Real-World Scenarios Tested
|
|
|
|
### Scenario 1: Preventing Breaking Changes
|
|
|
|
**Test**: `test_major_version_jump_prevention`
|
|
|
|
**Real-World Impact**:
|
|
- urllib3 2.0 removed deprecated APIs
|
|
- Many applications break when upgrading from 1.x to 2.x
|
|
- Pin prevents this automatic breaking change
|
|
|
|
**Verified**: ✅ Pin successfully prevents 1.x → 2.x upgrade
|
|
|
|
---
|
|
|
|
### Scenario 2: Allowing New Dependencies
|
|
|
|
**Test**: `test_pin_only_affects_specified_packages`
|
|
|
|
**Real-World Impact**:
|
|
- New dependencies are safe to add (idna)
|
|
- Pin should not block ALL changes
|
|
- Only specified packages are protected
|
|
|
|
**Verified**: ✅ idna installs at 3.10 even with pin policy active
|
|
|
|
---
|
|
|
|
### Scenario 3: Version Downgrade Recovery
|
|
|
|
**Test**: `test_version_change_and_restore`
|
|
|
|
**Real-World Impact**:
|
|
- Sometimes packages get upgraded accidentally
|
|
- Need to downgrade to known-good version
|
|
- Downgrade is harder than upgrade prevention
|
|
|
|
**Verified**: ✅ Can downgrade urllib3 from 2.x to 1.x
|
|
|
|
---
|
|
|
|
## Test Execution Performance
|
|
|
|
```
|
|
Test Performance Summary:
|
|
|
|
test_dependency_version_protection_with_pin 2.28s (enhanced)
|
|
test_dependency_chain_with_six_pin 2.00s (enhanced)
|
|
test_pin_only_affects_specified_packages 2.25s (NEW)
|
|
test_major_version_jump_prevention 3.53s (NEW - does 2 install cycles)
|
|
test_package_deletion_and_restore 2.25s (enhanced)
|
|
test_version_change_and_restore 2.24s (enhanced)
|
|
|
|
Total: 14.10s for 6 tests
|
|
Average: 2.35s per test
|
|
```
|
|
|
|
**Note**: `test_major_version_jump_prevention` is slower because it tests both WITH and WITHOUT pin (2 install cycles).
|
|
|
|
---
|
|
|
|
## Files Modified
|
|
|
|
1. **test_dependency_protection.py**: +138 lines
|
|
- Enhanced 2 existing tests
|
|
- Added 2 new tests
|
|
- Total: 272 lines (was 132 lines)
|
|
|
|
2. **test_environment_recovery.py**: +35 lines
|
|
- Enhanced 2 existing tests
|
|
- Total: 159 lines (was 141 lines)
|
|
|
|
---
|
|
|
|
## Verification Against Context
|
|
|
|
All test improvements verified against:
|
|
|
|
| Context Source | Usage |
|
|
|----------------|-------|
|
|
| **DEPENDENCY_TREE_CONTEXT.md** | All version numbers, dependency trees |
|
|
| **DEPENDENCY_ANALYSIS.md** | Package selection rationale, rejected scenarios |
|
|
| **TEST_SCENARIOS.md** | Scenario specifications, expected outcomes |
|
|
| **requirements-test-base.txt** | Initial environment state |
|
|
| **analyze_dependencies.py** | Real-time verification of expectations |
|
|
|
|
---
|
|
|
|
## Future Maintenance
|
|
|
|
### When to Update Tests
|
|
|
|
Update tests when:
|
|
- ✅ PyPI releases new major versions (e.g., urllib3 3.0)
|
|
- ✅ Base package versions change in requirements-test-base.txt
|
|
- ✅ New test scenarios added to DEPENDENCY_TREE_CONTEXT.md
|
|
- ✅ Policy behavior changes in pip_util.py
|
|
|
|
### How to Update Tests
|
|
|
|
1. Run `python analyze_dependencies.py --all`
|
|
2. Update expected version numbers in tests
|
|
3. Update DEPENDENCY_TREE_CONTEXT.md
|
|
4. Update TEST_SCENARIOS.md
|
|
5. Run tests to verify
|
|
|
|
### Verification Commands
|
|
|
|
```bash
|
|
# Verify environment
|
|
python analyze_dependencies.py --env
|
|
|
|
# Verify package dependencies
|
|
python analyze_dependencies.py requests
|
|
python analyze_dependencies.py python-dateutil
|
|
|
|
# Run all tests
|
|
pytest test_dependency_protection.py test_environment_recovery.py -v --override-ini="addopts="
|
|
```
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
✅ **6 tests** now verify real PyPI package dependencies
|
|
✅ **100% pass rate** with real pip operations
|
|
✅ **All version numbers** verified against DEPENDENCY_TREE_CONTEXT.md
|
|
✅ **Major version jump prevention** explicitly tested
|
|
✅ **Selective pinning** verified (only specified packages)
|
|
✅ **Version downgrade** capability tested
|
|
|
|
**Key Achievement**: Tests now verify actual PyPI behavior, not mocked expectations.
|