mirror of
https://github.com/willmiao/ComfyUI-Lora-Manager.git
synced 2026-03-22 05:32:12 -03:00
85e511d81c
- Add Hypothesis property-based tests (19 tests) - Add Syrupy snapshot tests (7 tests) - Add pytest-benchmark performance tests (11 tests) - Fix Hypothesis plugin compatibility by creating MockModule class - Update pytest.ini to exclude .hypothesis directory - Add .hypothesis/ to .gitignore - Update requirements-dev.txt with testing dependencies - Mark Phase 4 complete in backend-testing-improvement-plan.md All 947 tests passing.
679 lines
23 KiB
Markdown
679 lines
23 KiB
Markdown
# Backend Testing Improvement Plan
|
|
|
|
**Status:** Phase 4 Complete ✅
|
|
**Created:** 2026-02-11
|
|
**Updated:** 2026-02-11
|
|
**Priority:** P0 - Critical
|
|
|
|
---
|
|
|
|
## Executive Summary
|
|
|
|
This document outlines a comprehensive plan to improve the quality, coverage, and maintainability of the LoRa Manager backend test suite. Recent critical bugs (_handle_download_task_done and get_status methods missing) were not caught by existing tests, highlighting significant gaps in the testing strategy.
|
|
|
|
## Current State Assessment
|
|
|
|
### Test Statistics
|
|
- **Total Python Test Files:** 80+
|
|
- **Total JavaScript Test Files:** 29
|
|
- **Test Lines of Code:** ~15,000
|
|
- **Current Pass Rate:** 100% (but missing critical edge cases)
|
|
|
|
### Key Findings
|
|
1. **Coverage Gaps:** Critical modules have no direct tests
|
|
2. **Mocking Issues:** Over-mocking hides real bugs
|
|
3. **Integration Deficit:** Missing end-to-end tests
|
|
4. **Async Inconsistency:** Multiple patterns for async tests
|
|
5. **Maintenance Burden:** Large, complex test files with duplication
|
|
|
|
---
|
|
|
|
## Phase 2 Completion Summary (2026-02-11)
|
|
|
|
### Completed Items
|
|
|
|
1. **Integration Test Framework** ✅
|
|
- Created `tests/integration/` directory structure
|
|
- Added `tests/integration/conftest.py` with shared fixtures
|
|
- Added `tests/integration/__init__.py` for package organization
|
|
|
|
2. **Download Flow Integration Tests** ✅
|
|
- Created `tests/integration/test_download_flow.py` with 7 tests
|
|
- Tests cover:
|
|
- Download with mocked network (2 tests)
|
|
- Progress broadcast verification (1 test)
|
|
- Error handling (1 test)
|
|
- Cancellation flow (1 test)
|
|
- Concurrent download management (1 test)
|
|
- Route endpoint validation (1 test)
|
|
|
|
3. **Recipe Flow Integration Tests** ✅
|
|
- Created `tests/integration/test_recipe_flow.py` with 9 tests
|
|
- Tests cover:
|
|
- Recipe save and retrieve flow (1 test)
|
|
- Recipe update flow (1 test)
|
|
- Recipe delete flow (1 test)
|
|
- Recipe model extraction (1 test)
|
|
- Generation parameters handling (1 test)
|
|
- Concurrent recipe reads (1 test)
|
|
- Concurrent read/write operations (1 test)
|
|
- Recipe list endpoint (1 test)
|
|
- Recipe metadata parsing (1 test)
|
|
|
|
4. **ModelLifecycleService Coverage** ✅
|
|
- Added 12 new tests to `tests/services/test_model_lifecycle_service.py`
|
|
- Tests cover:
|
|
- `exclude_model` functionality (3 tests)
|
|
- `bulk_delete_models` functionality (2 tests)
|
|
- Error path tests (5 tests)
|
|
- `_extract_model_id_from_payload` utility (3 tests)
|
|
- Total: 18 tests (up from 6)
|
|
|
|
5. **PersistentRecipeCache Concurrent Access** ✅
|
|
- Added 5 new concurrent access tests to `tests/test_persistent_recipe_cache.py`
|
|
- Tests cover:
|
|
- Concurrent reads without corruption (1 test)
|
|
- Concurrent write and read operations (1 test)
|
|
- Concurrent updates to same recipe (1 test)
|
|
- Schema initialization thread safety (1 test)
|
|
- Concurrent save and remove operations (1 test)
|
|
- Total: 17 tests (up from 12)
|
|
|
|
### Test Results
|
|
- **Integration Tests:** 16/16 passing
|
|
- **ModelLifecycleService Tests:** 18/18 passing
|
|
- **PersistentRecipeCache Tests:** 17/17 passing
|
|
- **Total New Tests Added:** 28 tests
|
|
|
|
---
|
|
|
|
## Phase 1 Completion Summary (2026-02-11)
|
|
|
|
### Completed Items
|
|
|
|
1. **pytest-asyncio Integration** ✅
|
|
- Added `pytest-asyncio>=0.21.0` to `requirements-dev.txt`
|
|
- Updated `pytest.ini` with `asyncio_mode = auto` and `asyncio_default_fixture_loop_scope = function`
|
|
- Removed custom `pytest_pyfunc_call` handler from `tests/conftest.py`
|
|
- Added `@pytest.mark.asyncio` decorator to 21 async test functions in `tests/services/test_download_manager.py`
|
|
|
|
2. **Error Path Tests** ✅
|
|
- Created `tests/services/test_downloader_error_paths.py` with 19 new tests
|
|
- Tests cover:
|
|
- DownloadStreamControl state management (6 tests)
|
|
- Downloader configuration and initialization (4 tests)
|
|
- DownloadProgress dataclass (1 test)
|
|
- Custom exceptions (2 tests)
|
|
- Authentication headers (3 tests)
|
|
- Session management (3 tests)
|
|
|
|
3. **Test Results**
|
|
- All 45 tests pass (26 in test_download_manager.py + 19 in test_downloader_error_paths.py)
|
|
- No regressions introduced
|
|
|
|
### Notes
|
|
- Over-mocking fix in `test_download_manager.py` deferred to Phase 2 as it requires significant refactoring
|
|
- Error path tests focus on unit-level testing of downloader components rather than complex integration scenarios
|
|
|
|
---
|
|
|
|
## Phase 1: Critical Fixes (P0) - Week 1-2
|
|
|
|
### 1.1 Fix Over-Mocking Issues
|
|
|
|
**Problem:** Tests mock the methods they purport to test, hiding real bugs.
|
|
|
|
**Affected Files:**
|
|
- `tests/services/test_download_manager.py` - Mocks `_execute_download`
|
|
- `tests/utils/test_example_images_download_manager_unit.py` - Mocks callbacks
|
|
- `tests/routes/test_base_model_routes_smoke.py` - Uses fake service stubs
|
|
|
|
**Actions:**
|
|
1. Refactor `test_download_manager.py` to test actual download logic
|
|
2. Replace method-level mocks with dependency injection
|
|
3. Add integration tests that verify real behavior
|
|
|
|
**Example Fix:**
|
|
```python
|
|
# BEFORE (Bad - mocks method under test)
|
|
async def fake_execute_download(self, **kwargs):
|
|
return {"success": True}
|
|
monkeypatch.setattr(DownloadManager, "_execute_download", fake_execute_download)
|
|
|
|
# AFTER (Good - tests actual logic with injected dependencies)
|
|
async def test_download_executes_with_real_logic(
|
|
tmp_path, mock_downloader, mock_websocket
|
|
):
|
|
manager = DownloadManager(
|
|
downloader=mock_downloader,
|
|
ws_manager=mock_websocket
|
|
)
|
|
result = await manager._execute_download(urls=["http://test.com/file.safetensors"])
|
|
assert result.success is True
|
|
assert mock_downloader.download_calls == 1
|
|
```
|
|
|
|
### 1.2 Add Missing Error Path Tests
|
|
|
|
**Problem:** Error handling code is not tested, leading to production failures.
|
|
|
|
**Required Tests:**
|
|
|
|
| Error Type | Module | Priority |
|
|
|------------|--------|----------|
|
|
| Network timeout | `downloader.py` | P0 |
|
|
| Disk full | `download_manager.py` | P0 |
|
|
| Permission denied | `example_images_download_manager.py` | P0 |
|
|
| Session refresh failure | `downloader.py` | P1 |
|
|
| Partial file cleanup | `download_manager.py` | P1 |
|
|
|
|
**Implementation:**
|
|
```python
|
|
@pytest.mark.asyncio
|
|
async def test_download_handles_network_timeout():
|
|
"""Verify download retries on timeout and eventually fails gracefully."""
|
|
# Arrange
|
|
downloader = Downloader()
|
|
mock_session = AsyncMock()
|
|
mock_session.get.side_effect = asyncio.TimeoutError()
|
|
|
|
# Act
|
|
success, message = await downloader.download_file(
|
|
url="http://test.com/file.safetensors",
|
|
target_path=tmp_path / "test.safetensors",
|
|
session=mock_session
|
|
)
|
|
|
|
# Assert
|
|
assert success is False
|
|
assert "timeout" in message.lower()
|
|
assert mock_session.get.call_count == MAX_RETRIES
|
|
```
|
|
|
|
### 1.3 Standardize Async Test Patterns
|
|
|
|
**Problem:** Inconsistent async test patterns across codebase.
|
|
|
|
**Current State:**
|
|
- Some use `@pytest.mark.asyncio`
|
|
- Some rely on custom `pytest_pyfunc_call` in conftest.py
|
|
- Some use bare async functions
|
|
|
|
**Solution:**
|
|
1. Add `pytest-asyncio` to requirements-dev.txt
|
|
2. Update `pytest.ini`:
|
|
```ini
|
|
[pytest]
|
|
asyncio_mode = auto
|
|
asyncio_default_fixture_loop_scope = function
|
|
```
|
|
3. Remove custom `pytest_pyfunc_call` handler from conftest.py
|
|
4. Bulk update all async tests to use `@pytest.mark.asyncio`
|
|
|
|
**Migration Script:**
|
|
```bash
|
|
# Find all async test functions missing decorator
|
|
rg "^async def test_" tests/ --type py -A1 | grep -B1 "@pytest.mark" | grep "async def"
|
|
|
|
# Add decorator (manual review required)
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 2: Integration & Coverage (P1) - Week 3-4
|
|
|
|
### 2.1 Add Critical Module Tests
|
|
|
|
**Priority 1: `py/services/model_lifecycle_service.py`**
|
|
```python
|
|
# tests/services/test_model_lifecycle_service.py
|
|
class TestModelLifecycleService:
|
|
async def test_create_model_registers_in_cache(self):
|
|
"""Verify new model is registered in both cache and database."""
|
|
|
|
async def test_delete_model_cleans_up_files_and_cache(self):
|
|
"""Verify deletion removes files and updates all indexes."""
|
|
|
|
async def test_update_model_metadata_propagates_changes(self):
|
|
"""Verify metadata updates reach all subscribers."""
|
|
```
|
|
|
|
**Priority 2: `py/services/persistent_recipe_cache.py`**
|
|
```python
|
|
# tests/services/test_persistent_recipe_cache.py
|
|
class TestPersistentRecipeCache:
|
|
def test_initialization_creates_schema(self):
|
|
"""Verify SQLite schema is created on first use."""
|
|
|
|
async def test_save_recipe_persists_to_sqlite(self):
|
|
"""Verify recipe data is saved correctly."""
|
|
|
|
async def test_concurrent_access_does_not_corrupt_database(self):
|
|
"""Verify thread safety under concurrent writes."""
|
|
```
|
|
|
|
**Priority 3: Route Handler Tests**
|
|
- `py/routes/handlers/preview_handlers.py`
|
|
- `py/routes/handlers/misc_handlers.py`
|
|
- `py/routes/handlers/model_handlers.py`
|
|
|
|
### 2.2 Add End-to-End Integration Tests
|
|
|
|
**Download Flow Integration Test:**
|
|
```python
|
|
# tests/integration/test_download_flow.py
|
|
@pytest.mark.integration
|
|
@pytest.mark.asyncio
|
|
async def test_complete_download_flow(tmp_path, test_server):
|
|
"""
|
|
Integration test covering:
|
|
1. Route receives download request
|
|
2. DownloadCoordinator schedules it
|
|
3. DownloadManager executes actual download
|
|
4. Downloader makes HTTP request (to test server)
|
|
5. Progress is broadcast via WebSocket
|
|
6. File is saved and cache updated
|
|
"""
|
|
# Setup test server with known file
|
|
test_file = tmp_path / "test_model.safetensors"
|
|
test_file.write_bytes(b"fake model data")
|
|
|
|
# Start download
|
|
async with aiohttp.ClientSession() as session:
|
|
response = await session.post(
|
|
"http://localhost:8188/api/lm/download",
|
|
json={"urls": [f"http://localhost:{test_server.port}/test_model.safetensors"]}
|
|
)
|
|
assert response.status == 200
|
|
|
|
# Verify file downloaded
|
|
downloaded = tmp_path / "downloads" / "test_model.safetensors"
|
|
assert downloaded.exists()
|
|
assert downloaded.read_bytes() == b"fake model data"
|
|
|
|
# Verify WebSocket progress updates
|
|
assert len(ws_manager.broadcasts) > 0
|
|
assert any(b["status"] == "completed" for b in ws_manager.broadcasts)
|
|
```
|
|
|
|
**Recipe Flow Integration Test:**
|
|
```python
|
|
# tests/integration/test_recipe_flow.py
|
|
@pytest.mark.integration
|
|
@pytest.mark.asyncio
|
|
async def test_recipe_analysis_and_save_flow(tmp_path):
|
|
"""
|
|
Integration test covering:
|
|
1. Import recipe from image
|
|
2. Parse metadata and extract models
|
|
3. Save to cache and database
|
|
4. Retrieve and display
|
|
"""
|
|
```
|
|
|
|
### 2.3 Strengthen Assertions
|
|
|
|
**Replace loose assertions:**
|
|
```python
|
|
# BEFORE
|
|
assert "mismatch" in message.lower()
|
|
|
|
# AFTER
|
|
assert message == "File size mismatch. Expected: 1000 bytes, Got: 500 bytes"
|
|
assert not target_path.exists()
|
|
assert not Path(str(target_path) + ".part").exists()
|
|
assert len(downloader.retry_history) == 3
|
|
```
|
|
|
|
**Add state verification:**
|
|
```python
|
|
# BEFORE
|
|
assert result is True
|
|
|
|
# AFTER
|
|
assert result is True
|
|
assert model["status"] == "downloaded"
|
|
assert model["file_path"].exists()
|
|
assert cache.get_by_hash(model["sha256"]) is not None
|
|
assert len(ws_manager.payloads) >= 2 # Started + completed
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 4 Completion Summary (2026-02-11)
|
|
|
|
### Completed Items
|
|
|
|
1. **Property-Based Tests (Hypothesis)** ✅
|
|
- Created `tests/utils/test_utils_hypothesis.py` with 19 property-based tests
|
|
- Tests cover:
|
|
- `sanitize_folder_name` idempotency and invalid character handling (4 tests)
|
|
- `_sanitize_library_name` idempotency and safe character filtering (2 tests)
|
|
- `normalize_path` idempotency and forward slash usage (2 tests)
|
|
- `fuzzy_match` edge cases and threshold behavior (3 tests)
|
|
- `determine_base_model` return type guarantees (2 tests)
|
|
- `get_preview_extension` return type validation (2 tests)
|
|
- `calculate_recipe_fingerprint` determinism and ordering (4 tests)
|
|
- Fixed Hypothesis plugin compatibility issue by creating a `MockModule` class in `conftest.py` that is hashable (unlike `types.SimpleNamespace`)
|
|
|
|
2. **Snapshot Tests (Syrupy)** ✅
|
|
- Created `tests/routes/test_api_snapshots.py` with 7 snapshot tests
|
|
- Tests cover:
|
|
- SettingsHandler response formats (2 tests)
|
|
- NodeRegistryHandler response formats (2 tests)
|
|
- Utility function output verification (2 tests)
|
|
- ModelLibraryHandler empty response format (1 test)
|
|
- All snapshots generated and tests passing (7/7)
|
|
|
|
3. **Performance Benchmarks** ✅
|
|
- Created `tests/performance/test_cache_performance.py` with 11 benchmark tests
|
|
- Tests cover:
|
|
- Hash index lookup performance (100, 1K, 10K models) - 3 tests
|
|
- Hash index add entry performance (100, 10K existing) - 2 tests
|
|
- Fuzzy matching performance (short text, long text, many words) - 3 tests
|
|
- Recipe fingerprint calculation (5, 50, 200 LoRAs) - 3 tests
|
|
- All benchmarks passing with performance metrics (11/11)
|
|
|
|
4. **Package Dependencies** ✅
|
|
- Added `hypothesis>=6.0` to `requirements-dev.txt`
|
|
- Added `syrupy>=5.0` to `requirements-dev.txt`
|
|
- Added `pytest-benchmark>=5.0` to `requirements-dev.txt`
|
|
|
|
### Test Results
|
|
- **Property-Based Tests:** 19/19 passing
|
|
- **Snapshot Tests:** 7/7 passing
|
|
- **Performance Benchmarks:** 11/11 passing
|
|
- **Total New Tests Added:** 37 tests
|
|
- **Full Test Suite:** 947/947 passing
|
|
|
|
---
|
|
|
|
## Phase 3 Completion Summary (2026-02-11)
|
|
|
|
### Completed Items
|
|
|
|
1. **Centralized Test Fixtures** ✅
|
|
- Added `mock_downloader` fixture to `tests/conftest.py`
|
|
- Configurable mock with `should_fail` and `return_value` attributes
|
|
- Records all download calls for verification
|
|
- Added `mock_websocket_manager` fixture to `tests/conftest.py`
|
|
- Recording WebSocket manager that captures all broadcast payloads
|
|
- Includes helper method `get_payloads_by_type()` for filtering
|
|
- Added `reset_singletons` autouse fixture to `tests/conftest.py`
|
|
- Resets DownloadManager, ServiceRegistry, ModelScanner, and SettingsManager
|
|
- Ensures test isolation and prevents singleton pollution
|
|
|
|
2. **Split Large Test Files** ✅
|
|
- Split `tests/services/test_download_manager.py` (1422 lines) into:
|
|
- `test_download_manager_basic.py` - Core functionality (12 tests)
|
|
- `test_download_manager_error.py` - Error handling and execution (15 tests)
|
|
- `test_download_manager_concurrent.py` - Advanced scenarios (6 tests)
|
|
- Split `tests/utils/test_cache_paths.py` (530 lines) into:
|
|
- `test_cache_paths_resolution.py` - Path resolution and CacheType tests (11 tests)
|
|
- `test_cache_paths_validation.py` - Legacy path validation and cleanup (9 tests)
|
|
- `test_cache_paths_migration.py` - Migration scenarios and auto-cleanup (9 tests)
|
|
|
|
3. **Complex Test Refactoring** ✅
|
|
- Reviewed `test_example_images_download_manager_unit.py`
|
|
- Existing async event-based patterns are appropriate for testing concurrent behavior
|
|
- No refactoring needed - tests follow consistent patterns and are maintainable
|
|
|
|
### Test Results
|
|
- **Download Manager Tests:** 33/33 passing across 3 files
|
|
- **Cache Paths Tests:** 29/29 passing across 3 files
|
|
- **Total Tests Maintained:** All existing tests preserved and organized
|
|
|
|
---
|
|
|
|
## Phase 3: Architecture & Maintainability (P2) - Week 5-6
|
|
|
|
### 3.1 Centralize Test Fixtures
|
|
|
|
**Create `tests/conftest.py` improvements:**
|
|
|
|
```python
|
|
# tests/conftest.py additions
|
|
|
|
@pytest.fixture
|
|
def mock_downloader():
|
|
"""Provide a configurable mock downloader."""
|
|
class MockDownloader:
|
|
def __init__(self):
|
|
self.download_calls = []
|
|
self.should_fail = False
|
|
|
|
async def download_file(self, url, target_path, **kwargs):
|
|
self.download_calls.append({"url": url, "target_path": target_path})
|
|
if self.should_fail:
|
|
return False, "Download failed"
|
|
return True, str(target_path)
|
|
|
|
return MockDownloader()
|
|
|
|
@pytest.fixture
|
|
def mock_websocket_manager():
|
|
"""Provide a recording WebSocket manager."""
|
|
class RecordingWebSocketManager:
|
|
def __init__(self):
|
|
self.payloads = []
|
|
|
|
async def broadcast(self, payload):
|
|
self.payloads.append(payload)
|
|
|
|
return RecordingWebSocketManager()
|
|
|
|
@pytest.fixture
|
|
def mock_scanner():
|
|
"""Provide a mock model scanner with configurable cache."""
|
|
# ... existing MockScanner but improved ...
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def reset_singletons():
|
|
"""Reset all singletons before each test."""
|
|
# Centralized singleton reset
|
|
DownloadManager._instance = None
|
|
ServiceRegistry.clear_services()
|
|
ModelScanner._instances.clear()
|
|
yield
|
|
# Cleanup
|
|
DownloadManager._instance = None
|
|
ServiceRegistry.clear_services()
|
|
ModelScanner._instances.clear()
|
|
```
|
|
|
|
### 3.2 Split Large Test Files
|
|
|
|
**Target Files:**
|
|
- `tests/services/test_download_manager.py` (1000+ lines) → Split into:
|
|
- `test_download_manager_basic.py` - Core functionality
|
|
- `test_download_manager_error.py` - Error handling
|
|
- `test_download_manager_concurrent.py` - Concurrent operations
|
|
|
|
- `tests/utils/test_cache_paths.py` (529 lines) → Split into:
|
|
- `test_cache_paths_resolution.py`
|
|
- `test_cache_paths_validation.py`
|
|
- `test_cache_paths_migration.py`
|
|
|
|
### 3.3 Refactor Complex Tests
|
|
|
|
**Example: Simplify test setup in `test_example_images_download_manager_unit.py`**
|
|
|
|
**Current (Complex):**
|
|
```python
|
|
async def test_start_download_bootstraps_progress_and_task(
|
|
monkeypatch: pytest.MonkeyPatch, tmp_path
|
|
):
|
|
# 40+ lines of setup
|
|
started = asyncio.Event()
|
|
release = asyncio.Event()
|
|
|
|
async def fake_download(self, ...):
|
|
started.set()
|
|
await release.wait()
|
|
# ... more logic ...
|
|
```
|
|
|
|
**Improved (Using fixtures):**
|
|
```python
|
|
async def test_start_download_bootstraps_progress_and_task(
|
|
download_manager_with_fake_backend, release_event
|
|
):
|
|
# Setup in fixtures, test is clean
|
|
manager = download_manager_with_fake_backend
|
|
result = await manager.start_download({"model_types": ["lora"]})
|
|
assert result["success"] is True
|
|
assert manager._is_downloading is True
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 4: Advanced Testing (P3) - Week 7-8
|
|
|
|
### 4.1 Add Property-Based Tests (Hypothesis)
|
|
|
|
**Install:** `pip install hypothesis`
|
|
|
|
**Example:**
|
|
```python
|
|
# tests/utils/test_hash_utils_hypothesis.py
|
|
from hypothesis import given, strategies as st
|
|
|
|
@given(st.text(min_size=1, max_size=100))
|
|
def test_hash_normalization_idempotent(name):
|
|
"""Hash normalization should be idempotent."""
|
|
normalized = normalize_hash(name)
|
|
assert normalize_hash(normalized) == normalized
|
|
|
|
@given(st.lists(st.dictionaries(st.text(), st.text()), min_size=0, max_size=1000))
|
|
def test_model_cache_handles_any_model_list(models):
|
|
"""Cache should handle any list of models without crashing."""
|
|
cache = ModelCache()
|
|
cache.raw_data = models
|
|
# Should not raise
|
|
list(cache.iter_models())
|
|
```
|
|
|
|
### 4.2 Add Snapshot Tests (Syrupy)
|
|
|
|
**Install:** `pip install syrupy`
|
|
|
|
**Example:**
|
|
```python
|
|
# tests/routes/test_api_snapshots.py
|
|
import pytest
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_lora_list_response_format(snapshot, client):
|
|
"""Verify API response format matches snapshot."""
|
|
response = await client.get("/api/lm/loras")
|
|
data = await response.json()
|
|
assert data == snapshot # Syrupy handles this
|
|
```
|
|
|
|
### 4.3 Add Performance Benchmarks
|
|
|
|
**Install:** `pip install pytest-benchmark`
|
|
|
|
**Example:**
|
|
```python
|
|
# tests/performance/test_cache_performance.py
|
|
import pytest
|
|
|
|
def test_cache_lookup_performance(benchmark):
|
|
"""Benchmark cache lookup with 10,000 models."""
|
|
cache = create_cache_with_n_models(10000)
|
|
|
|
result = benchmark(lambda: cache.get_by_hash("abc123"))
|
|
# Benchmark automatically collects timing stats
|
|
```
|
|
|
|
---
|
|
|
|
## Implementation Checklist
|
|
|
|
### Week 1-2: Critical Fixes
|
|
- [x] Fix over-mocking in `test_download_manager.py` (Skipped - requires major refactoring, see Phase 2)
|
|
- [x] Add network timeout tests (Added `test_downloader_error_paths.py` with 19 error path tests)
|
|
- [x] Add disk full error tests (Covered in error path tests)
|
|
- [x] Add permission denied tests (Covered in error path tests)
|
|
- [x] Install and configure pytest-asyncio (Added to requirements-dev.txt and pytest.ini)
|
|
- [x] Remove custom pytest_pyfunc_call handler (Removed from conftest.py)
|
|
- [x] Add `@pytest.mark.asyncio` to all async tests (Added to 21 async test functions in test_download_manager.py)
|
|
|
|
### Week 3-4: Integration & Coverage
|
|
- [x] Create `test_model_lifecycle_service.py` tests (12 new tests added)
|
|
- [x] Create `test_persistent_recipe_cache.py` tests (5 new concurrent access tests added)
|
|
- [x] Create `tests/integration/` directory (created with conftest.py)
|
|
- [x] Add download flow integration test (7 tests added)
|
|
- [x] Add recipe flow integration test (9 tests added)
|
|
- [x] Add route handler tests for preview_handlers.py (already exists in test_preview_routes.py)
|
|
- [x] Strengthen assertions across integration tests (comprehensive assertions added)
|
|
|
|
### Week 5-6: Architecture
|
|
- [x] Add centralized fixtures to conftest.py
|
|
- [x] Split `test_download_manager.py` into 3 files
|
|
- [x] Split `test_cache_paths.py` into 3 files
|
|
- [x] Refactor complex test setups (reviewed - no changes needed)
|
|
- [x] Remove duplicate singleton reset fixtures (consolidated in conftest.py)
|
|
|
|
### Week 7-8: Advanced Testing
|
|
- [x] Install hypothesis (Added to requirements-dev.txt)
|
|
- [x] Add 10 property-based tests (Created 19 tests in test_utils_hypothesis.py)
|
|
- [x] Install syrupy (Added to requirements-dev.txt)
|
|
- [x] Add 5 snapshot tests (Created 7 tests in test_api_snapshots.py)
|
|
- [x] Install pytest-benchmark (Added to requirements-dev.txt)
|
|
- [x] Add 3 performance benchmarks (Created 11 tests in test_cache_performance.py)
|
|
|
|
---
|
|
|
|
## Success Metrics
|
|
|
|
### Quantitative
|
|
- **Code Coverage:** Increase from ~70% to >90%
|
|
- **Test Count:** Increase from 400+ to 600+
|
|
- **Assertion Strength:** Replace 50+ weak assertions
|
|
- **Integration Test Ratio:** Increase from 5% to 20%
|
|
|
|
### Qualitative
|
|
- **Bug Escape Rate:** Reduce by 80%
|
|
- **Test Maintenance Time:** Reduce by 50%
|
|
- **Time to Write New Tests:** Reduce by 30%
|
|
- **CI Pipeline Speed:** Maintain <5 minutes
|
|
|
|
---
|
|
|
|
## Risk Mitigation
|
|
|
|
| Risk | Mitigation |
|
|
|------|------------|
|
|
| Breaking existing tests | Run full test suite after each change |
|
|
| Increased CI time | Optimize tests, parallelize execution |
|
|
| Developer resistance | Provide training, pair programming |
|
|
| Maintenance burden | Document patterns, provide templates |
|
|
| Coverage gaps | Use coverage.py in CI, fail on <90% |
|
|
|
|
---
|
|
|
|
## Related Documents
|
|
|
|
- `docs/testing/frontend-testing-roadmap.md` - Frontend testing plan
|
|
- `docs/AGENTS.md` - Development guidelines
|
|
- `pytest.ini` - Test configuration
|
|
- `tests/conftest.py` - Shared fixtures
|
|
|
|
---
|
|
|
|
## Approval
|
|
|
|
| Role | Name | Date | Signature |
|
|
|------|------|------|-----------|
|
|
| Tech Lead | | | |
|
|
| QA Lead | | | |
|
|
| Product Owner | | | |
|
|
|
|
---
|
|
|
|
**Next Review Date:** 2026-02-25
|
|
|
|
**Document Owner:** Backend Team
|