alphard/ComfyUI-Lora-Manager
1
0
Fork 0
mirror of https://github.com/willmiao/ComfyUI-Lora-Manager.git synced 2026-09-21 11:11:26 -03:00
Code Issues Packages Projects Releases Wiki Activity

chore(skill): remove lora-manager-e2e skill, keep sandbox helpers in scripts/e2e/

Browse Source
This commit is contained in:
Will Miao
2026-09-05 07:10:43 +08:00
parent 121d8d5cea
commit 139231e225
8 changed files with 20 additions and 1073 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/lora-manager-e2e/SKILL.md
-146
View File
@@ -1,146 +0,0 @@
---
name: lora-manager-e2e
description: "End-to-end testing and validation for LoRa Manager features. Use ONLY for sandboxed E2E validation of LoRa Manager standalone mode: start the standalone server on a free port with --settings-path, drive the web UI (http://127.0.0.1:{PORT}/loras) via Chrome DevTools MCP, and verify frontend-to-backend integration. NOT for UI behavior checks that unit tests (Vitest/jsdom) can cover. Trigger keywords: E2E, standalone, Chrome DevTools MCP, lora-manager-e2e, sandbox."
---
# LoRa Manager E2E Testing
End-to-end testing of LoRa Manager standalone mode using Chrome DevTools MCP.
## When to Use — and When NOT To
E2E runs are slow and token-heavy. Reach for them only when the question genuinely
spans server + browser (routing, scan persistence, websocket updates, EXIF writes).
- **Default to unit/component tests first**: `npm run test:js` (Vitest/jsdom) covers
DOM rendering, modal behavior, event handling and API-client calls deterministically
in seconds. Backend logic goes through `pytest`. A UI-behavior question answered by
jsdom MUST NOT be escalated to E2E.
- **Use E2E only when** the behavior cannot be observed without a live server and a
real browser, e.g. template rendering through the aiohttp server, scanner → SQLite
persistence → API → DOM round-trips, or real EXIF/image writes.
- If you start an E2E and realize a unit test would answer the question, stop and
switch.
**Browser driver is fixed: Chrome DevTools MCP.** Do not substitute kimi-webbridge —
it operates on the user's real browser (real tabs, real sessions, synthetic
`isTrusted=false` events), which breaks the isolation this skill requires and lacks
the console/network inspection E2E debugging relies on. kimi-webbridge is for
interactive browsing with the user's real login sessions, not for sandboxed E2E.
## Conventions
- **`{PORT}`**: default candidate `8188`, but it is **commonly occupied by a live
ComfyUI** — always check first (`ss -tlnp | grep ':{PORT}'`) and use a free port
(e.g. `8199`). Substitute the chosen port everywhere below. Never kill a process
you did not start for this E2E.
- **`<repo-root>`**: the repository/worktree root; run all commands from there.
- **`<sandbox>`**: a throwaway dir, e.g. `/tmp/opencode/<plan>-e2e`.
## SANDBOX (MANDATORY)
> Every E2E run MUST target a throwaway sandbox, never real user data.
1. **Explicit settings directory**: always launch with `--settings-path <sandbox>/settings`.
This pins ALL runtime data (`settings.json`, `cache/`, `backups/`, `logs/`, `stats/`,
`wildcards/`) under the sandbox. **Never** create `<repo-root>/settings.json` — the repo
folder is usually the real ComfyUI plugin folder and a portable settings file there is
read by the real instance.
2. **Sandboxed library paths**: point `folder_paths` / `recipes_path` /
`example_images_path` at disposable dirs under `<sandbox>` — never the real library,
real recipe dir, or real settings:
```json
{
"folder_paths": {
"loras": ["<sandbox>/models/loras"],
"checkpoints": ["<sandbox>/models/checkpoints"],
"unet": ["<sandbox>/models/checkpoints"],
"diffusers": []
},
"recipes_path": "<sandbox>/recipes",
"example_images_path": "<sandbox>/example_images"
}
```
3. **Real-data protection proof**: before starting and after finishing, snapshot the real
config and recipe library and confirm they are byte-identical; also confirm
`<repo-root>` gained no `settings.json` or `cache/`:
```bash
sha256sum ~/.config/ComfyUI-LoRA-Manager/settings.json > <sandbox>/settings.before.sha256
ls ~/models/recipes/*.recipe.json 2>/dev/null | wc -l > <sandbox>/recipes-count.before.txt
# AFTER the run: record again and diff. Any change = the run leaked into real data.
```
## Quick Start
```bash
cd <repo-root>
# 1. Sandbox
mkdir -p <sandbox>/settings <sandbox>/models/{loras,checkpoints} <sandbox>/{recipes,example_images}
# write <sandbox>/settings/settings.json per the SANDBOX example
# 2. Port
ss -tlnp | grep ':{PORT}' || echo "port {PORT} is free"
# 3. Server — MUST be fully detached (a plain background & dies with the shell);
# the helper enforces this and manages its own pidfile
python .agents/skills/lora-manager-e2e/scripts/start_server.py \
--port {PORT} --settings-path <sandbox>/settings --wait --timeout 30 --detach
ss -tlnp | grep ':{PORT}' # verify listening BEFORE proceeding
# 4. Chrome with remote debugging, then connect Chrome DevTools MCP (verify via list_pages)
google-chrome --remote-debugging-port=9222 --user-data-dir=/tmp/chrome-lora-manager http://127.0.0.1:{PORT}/loras
```
Then drive the UI with the MCP tools (`take_snapshot`, `click`, `fill`, `fill_form`,
`evaluate_script`, `wait_for`, `list_network_requests`, `list_console_messages`) —
see [references/mcp-cheatsheet.md](references/mcp-cheatsheet.md) for patterns.
Server restart after config/fixture changes:
```bash
python .agents/skills/lora-manager-e2e/scripts/start_server.py \
--port {PORT} --settings-path <sandbox>/settings --restart --wait --detach
# then reload the browser page (ignoreCache=True)
```
`--restart` only kills the E2E server the script itself started (via its pidfile) and
aborts instead of killing unrelated processes on the port.
## Abort Rule
A sandboxed E2E should finish in well under 30 minutes. If any phase exceeds ~2x its
expected duration (server readiness > 60 s, MCP connect > 2 min, a single scenario >
10 min), or any single tool call fails 3+ times in a row, **STOP** — do not retry
blindly. Report `BLOCKED` with the phase, last observed state (server PID,
`ss -tlnp` output, page snapshot, last API response) and suspected cause. A clean
BLOCKED report beats an hour of retries.
## Troubleshooting
- **"browser is already running" / `list_pages` fails**: a stale Chrome holds the
profile dir. Find it (`ps -ef | grep -i '[c]hrome.*user-data-dir'`), confirm it is a
leftover QA Chrome (not the live ComfyUI, not your current MCP browser), kill only
that PID, then retry `list_pages`.
- **MCP refuses to write screenshots into the worktree**: save to `/tmp` via
`take_screenshot(filePath="/tmp/...")` and copy into the evidence dir from the shell.
## Cleanup
1. Stop the standalone server: `kill <recorded-pid>` (only the PID you started), then
confirm `ss -tlnp | grep ':{PORT}'` is empty.
2. Close browser pages (keep at least one open).
3. `rm -rf <sandbox>`; verify `<repo-root>` gained no `settings.json` or `cache/`.
4. Re-run the real-data protection check from the SANDBOX section and record the result.
## References & Scripts
- [references/mcp-cheatsheet.md](references/mcp-cheatsheet.md) — Chrome DevTools MCP
command patterns (navigation, waiting, snapshots, forms, network, console, performance).
- [references/test-scenarios.md](references/test-scenarios.md) — detailed test scenarios
(list display, metadata editing, recipes, settings, import/export).
- [references/recipe-rematch-fixtures.md](references/recipe-rematch-fixtures.md) —
fixture format, fresh-state reset and known gaps for recipe rematch/repair E2E runs.
- `scripts/start_server.py` — start/restart the standalone server
(`--port --settings-path --restart --wait --timeout --detach`); refuses to touch
unrelated processes on the port.
- `scripts/wait_for_server.py` — poll readiness (`--port --timeout`).
.agents/skills/lora-manager-e2e/references/mcp-cheatsheet.md
-360
View File
@@ -1,360 +0,0 @@
# Chrome DevTools MCP Cheatsheet for LoRa Manager
Quick reference for common MCP commands used in LoRa Manager E2E testing.
> **Port convention**: `{PORT}` is the port chosen for the E2E run (default candidate `8188`, but only if actually free — see the SKILL.md Port Selection section; use e.g. `8199` when `8188` is occupied by a live ComfyUI). Always run against the **sandboxed** standalone server, never a live instance.
## Navigation
```python
# Navigate to LoRA list page
navigate_page(type="url", url="http://127.0.0.1:{PORT}/loras")
# Reload page with cache clear
navigate_page(type="reload", ignoreCache=True)
# Go back/forward
navigate_page(type="back")
navigate_page(type="forward")
```
## Waiting
```python
# Wait for text to appear
wait_for(text="LoRAs", timeout=10000)
# Wait for specific element (via evaluate_script)
evaluate_script(function="""
() => {
return new Promise((resolve) => {
const check = () => {
if (document.querySelector('.lora-card')) {
resolve(true);
} else {
setTimeout(check, 100);
}
};
check();
});
}
""")
```
## Taking Snapshots
```python
# Full page snapshot
snapshot = take_snapshot()
# Verbose snapshot (more details)
snapshot = take_snapshot(verbose=True)
# Save to file
take_snapshot(filePath="test-snapshots/page-load.json")
```
## Element Interaction
```python
# Click element
click(uid="element-uid-from-snapshot")
# Double click
click(uid="element-uid", dblClick=True)
# Fill input
fill(uid="search-input", value="test query")
# Fill multiple inputs
fill_form(elements=[
{"uid": "input-1", "value": "value 1"},
{"uid": "input-2", "value": "value 2"},
])
# Hover
hover(uid="lora-card-1")
# Upload file
upload_file(uid="file-input", filePath="/path/to/file.safetensors")
```
## Keyboard Input
```python
# Press key
press_key(key="Enter")
press_key(key="Escape")
press_key(key="Tab")
# Keyboard shortcuts
press_key(key="Control+A") # Select all
press_key(key="Control+F") # Find
```
## JavaScript Evaluation
```python
# Simple evaluation
result = evaluate_script(function="() => document.title")
# Async evaluation
result = evaluate_script(function="""
async () => {
const response = await fetch('/loras/api/list');
return await response.json();
}
""")
# Check element existence
exists = evaluate_script(function="""
() => document.querySelector('.lora-card') !== null
""")
# Get element count
count = evaluate_script(function="""
() => document.querySelectorAll('.lora-card').length
""")
```
## Network Monitoring
```python
# List all network requests
requests = list_network_requests()
# Filter by resource type
xhr_requests = list_network_requests(resourceTypes=["xhr", "fetch"])
# Get specific request details
details = get_network_request(reqid=123)
# Include preserved requests from previous navigations
all_requests = list_network_requests(includePreservedRequests=True)
```
## Console Monitoring
```python
# List all console messages
messages = list_console_messages()
# Filter by type
errors = list_console_messages(types=["error", "warn"])
# Include preserved messages
all_messages = list_console_messages(includePreservedMessages=True)
# Get specific message
details = get_console_message(msgid=1)
```
## Performance Testing
```python
# Start trace with page reload
performance_start_trace(reload=True, autoStop=False)
# Start trace without reload
performance_start_trace(reload=False, autoStop=True, filePath="trace.json.gz")
# Stop trace
results = performance_stop_trace()
# Stop and save
performance_stop_trace(filePath="trace-results.json.gz")
# Analyze specific insight
insight = performance_analyze_insight(
insightSetId="results.insightSets[0].id",
insightName="LCPBreakdown"
)
```
## Page Management
```python
# List open pages
pages = list_pages()
# Select a page
select_page(pageId=0, bringToFront=True)
# Create new page
new_page(url="http://127.0.0.1:{PORT}/loras")
# Close page (keep at least one open!)
close_page(pageId=1)
# Resize page
resize_page(width=1920, height=1080)
```
## Screenshots
```python
# Full page screenshot
take_screenshot(fullPage=True)
# Viewport screenshot
take_screenshot()
# Element screenshot
take_screenshot(uid="lora-card-1")
# Save to file
take_screenshot(filePath="screenshots/page.png", format="png")
# JPEG with quality
take_screenshot(filePath="screenshots/page.jpg", format="jpeg", quality=90)
```
## Dialog Handling
```python
# Accept dialog
handle_dialog(action="accept")
# Accept with text input
handle_dialog(action="accept", promptText="user input")
# Dismiss dialog
handle_dialog(action="dismiss")
```
## Device Emulation
```python
# Mobile viewport
emulate(viewport={"width": 375, "height": 667, "isMobile": True, "hasTouch": True})
# Tablet viewport
emulate(viewport={"width": 768, "height": 1024, "isMobile": True, "hasTouch": True})
# Desktop viewport
emulate(viewport={"width": 1920, "height": 1080})
# Network throttling
emulate(networkConditions="Slow 3G")
emulate(networkConditions="Fast 4G")
# CPU throttling
emulate(cpuThrottlingRate=4) # 4x slowdown
# Geolocation
emulate(geolocation={"latitude": 37.7749, "longitude": -122.4194})
# User agent
emulate(userAgent="Mozilla/5.0 (Custom)")
# Reset emulation
emulate(viewport=None, networkConditions="No emulation", userAgent=None)
```
## Drag and Drop
```python
# Drag element to another
drag(from_uid="draggable-item", to_uid="drop-zone")
```
## Common LoRa Manager Test Patterns
### Verify LoRA Cards Loaded
```python
navigate_page(type="url", url="http://127.0.0.1:{PORT}/loras")
wait_for(text="LoRAs", timeout=10000)
# Check if cards loaded
result = evaluate_script(function="""
() => {
const cards = document.querySelectorAll('.lora-card');
return {
count: cards.length,
hasData: cards.length > 0
};
}
""")
```
### Search and Verify Results
```python
fill(uid="search-input", value="character")
press_key(key="Enter")
wait_for(timeout=2000) # Wait for debounce
# Check results
result = evaluate_script(function="""
() => {
const cards = document.querySelectorAll('.lora-card');
const names = Array.from(cards).map(c => c.dataset.name || c.textContent);
return { count: cards.length, names };
}
""")
```
### Check API Response
```python
# Trigger API call
evaluate_script(function="""
() => window.loraApiCallPromise = fetch('/loras/api/list').then(r => r.json())
""")
# Wait and get result
import time
time.sleep(1)
result = evaluate_script(function="""
async () => await window.loraApiCallPromise
""")
```
### Monitor Console for Errors
```python
# Before test: clear console (navigate reloads)
navigate_page(type="reload")
# ... perform actions ...
# Check for errors
errors = list_console_messages(types=["error"])
assert len(errors) == 0, f"Console errors: {errors}"
```
## Troubleshooting
### Stale profile lock ("browser is already running" / `list_pages` fails)
A Chrome profile held by a stale Chrome from a prior MCP session makes `list_pages`
fail with "browser is already running". Fix:
1. Find the stale Chrome that owns the profile dir (e.g. `~/.config/chrome-dev-profile`):
```bash
ps -ef | grep -i '[c]hrome.*user-data-dir'
```
2. Confirm it is a QA Chrome from a completed task (NOT the live ComfyUI server, NOT
your current MCP instance).
3. Kill ONLY that stale Chrome (`kill <stale-pid>`), then retry `list_pages`.
### Screenshot-write restrictions
The MCP may refuse to write into paths outside its configured workspace roots
(e.g. `.omo/evidence/screenshots/` under a worktree that canonicalizes to an unmapped
path). Save the screenshot to `/tmp` via the MCP, then copy it into the evidence dir:
```bash
# MCP: take_screenshot(filePath="/tmp/<plan>-e2e/recipe-b-after.png", format="png")
# Shell:
mkdir -p <repo-root>/.omo/evidence/screenshots
cp /tmp/<plan>-e2e/recipe-b-after.png <repo-root>/.omo/evidence/screenshots/
```
### Time budgets & abort rule
See SKILL.md "Time Budgets & Abort Guidance": if a phase exceeds ~2x its budget or a
tool call retries 3+ times in a row, STOP and report BLOCKED with the last observed
state (server PID + `ss -tlnp`, page snapshot, last API response). Do not loop.
.agents/skills/lora-manager-e2e/references/recipe-rematch-fixtures.md
-72
View File
@@ -1,72 +0,0 @@
# Recipe Rematch/Repair E2E — Fixtures, Fresh State, Known Gaps
Specialized guidance for recipe rematch/repair E2E runs, extracted from the SKILL.md
main flow. Read the SKILL.md SANDBOX section first — everything here assumes a
sandboxed run.
## Fixture Rules (validated by the task-8 E2E)
Seed the **sandboxed** `recipes_path` with hand-written fixture recipes:
1. **Filename constraint**: each file MUST be named `f"{id}.recipe.json"` **and** the
in-JSON `id` field MUST equal the filename. Discovery accepts any `*.recipe.json`,
but persistence resolves the path via `get_recipe_json_path` and
`_save_recipe_persistently` returns `False` on a mismatch → the fixture would be
counted as an error.
- `recipe-a.recipe.json` → in-JSON `"id": "recipe-a"`
2. **File format**: mirror an existing recipe JSON — top-level `id`, `file_path`,
`title`, `loras`, `fingerprint`, `gen_params`; lora entries per the persistence
conventions (`hash`, `file_name`, `modelVersionId`, `isDeleted`, ...).
3. **Companion image**: each recipe needs an image (e.g. a `.webp` generated with PIL)
referenced by `file_path`, used for EXIF verification
(`ExifUtils.append_recipe_metadata` writes a `"Recipe metadata: ..."` marker; a
freshly generated `.webp` with no marker is the clean "untouched" control).
4. **autov3 three-state contract**: for L3 (autov3-only, renamed-file) fixtures the
local model's `.metadata.json` sidecar MUST have the `autov3` key **ABSENT** (the
"unchecked" state), NOT `""` — `""` is the TERMINAL "checked but unavailable" state
that L3 deliberately skips. The scanner computes + persists `autov3` from the file
header during the normal library scan (`model_scanner.py` `_process_model_file`), so
the live L3 match resolves through the local autov3/hash cache; the
computed-autov3 branch for unchecked items is covered by the unit suite.
5. **Fixture design for a rematch run** (mirrors the task-8 E2E):
- `recipe-a`: lora entry `isDeleted=True`, `hash` = 12-char autov3 computed from the
local model (`calculate_autov3`, `py/utils/file_utils.py`), whose local model file
was RENAMED after the recipe was written so `file_name` differs (proves L3 match
without filename).
- `recipe-b`: parser-convention checkpoint entry (uses `id`, no `modelVersionId`)
matching a local checkpoint via L2 — the local checkpoint's `.metadata.json` MUST
carry civitai version data with that `id` so `version_index` contains it (L2
cannot match otherwise).
- `recipe-c`: healthy recipe (no deleted entries) → must remain untouched.
The scanner computes and persists model hashes during the library scan, so the sandbox
model dirs just need the model files + `.metadata.json` sidecars. With
`--settings-path`, all derived data lands under the sandbox settings dir (`cache/`,
`backups/`, `logs/`, `stats/`, `wildcards/`), and NO `cache/` appears in the repo root.
## Fresh State Between Entry-Point Runs
Each entry point (global / per-recipe / selection-bulk) must start from the same
deleted state. Between runs (keep a pristine copy in `<sandbox>/recipes-before/`):
```bash
# 1. Reset fixtures to the before-state snapshot
cp <sandbox>/recipes-before/*.recipe.json <sandbox>/recipes/
# 2. Clear the recipe/FTS caches (with --settings-path these live under the sandbox
# settings dir, NOT <repo-root>/cache)
rm -f <sandbox>/settings/cache/recipe/*.sqlite
rm -rf <sandbox>/settings/cache/fts/*
# 3. Restart the server (fresh process, fresh scan)
python .agents/skills/lora-manager-e2e/scripts/start_server.py \
--port {PORT} --settings-path <sandbox>/settings --restart --wait --timeout 30 --detach
# 4. Re-verify the server is listening + reload the browser page
```
## Cancellation Testing (KNOWN GAP)
Testing the rematch-cancel path E2E requires a run long enough to cancel mid-flight. A
tiny 3-recipe fixture set completes in **seconds** — too fast to reliably cancel. The
cancel path is currently **unit-covered only** (`rematch_all_recipes` cancellation
tests); do not block an E2E run on cancel-path verification. If you must attempt it,
you would need an artificially large/deferred fixture set to create a cancellable
window — treat this as a research task, not part of the standard E2E.
.agents/skills/lora-manager-e2e/references/test-scenarios.md
-280
View File
@@ -1,280 +0,0 @@
# LoRa Manager E2E Test Scenarios
This document provides detailed test scenarios for end-to-end validation of LoRa Manager features.
> **Run preconditions (from SKILL.md)**: every run uses the **sandboxed** standalone
> server on a free port `{PORT}` (default candidate `8188`, only if actually free — pick
> e.g. `8199` when `8188` is occupied by a live ComfyUI). Fixtures live in the sandboxed
> `recipes_path` as `f"{id}.recipe.json"` files with matching in-JSON `id`; the real user
> config and real library are never touched (record protection proof before/after).
> Abort if a phase exceeds ~2x its budget or a tool call retries 3+ times (SKILL.md
> "Time Budgets & Abort Guidance").
## Table of Contents
1. [LoRA List Page](#lora-list-page)
2. [Model Details](#model-details)
3. [Recipes](#recipes)
4. [Settings](#settings)
5. [Import/Export](#importexport)
---
## LoRA List Page
### Scenario: Page Load and Display
**Objective**: Verify the LoRA list page loads correctly and displays models.
**Steps**:
1. Navigate to `http://127.0.0.1:{PORT}/loras`
2. Wait for page title "LoRAs" to appear
3. Take snapshot to verify:
- Header with "LoRAs" title is visible
- Search/filter controls are present
- Grid/list view toggle exists
- LoRA cards are displayed (if models exist)
- Pagination controls (if applicable)
**Expected Result**: Page loads without errors, UI elements are present.
### Scenario: Search Functionality
**Objective**: Verify search filters LoRA models correctly.
**Steps**:
1. Ensure at least one LoRA exists with known name (e.g., "test-character")
2. Navigate to LoRA list page
3. Enter search term in search box: "test"
4. Press Enter or click search button
5. Wait for results to update
**Expected Result**: Only LoRAs matching search term are displayed.
**Verification Script**:
```python
# After search, verify filtered results
evaluate_script(function="""
() => {
const cards = document.querySelectorAll('.lora-card');
const names = Array.from(cards).map(c => c.dataset.name);
return { count: cards.length, names };
}
""")
```
### Scenario: Filter by Tags
**Objective**: Verify tag filtering works correctly.
**Steps**:
1. Navigate to LoRA list page
2. Click on a tag (e.g., "character", "style")
3. Wait for filtered results
**Expected Result**: Only LoRAs with selected tag are displayed.
### Scenario: View Mode Toggle
**Objective**: Verify grid/list view toggle works.
**Steps**:
1. Navigate to LoRA list page
2. Click list view button
3. Verify list layout
4. Click grid view button
5. Verify grid layout
**Expected Result**: View mode changes correctly, layout updates.
---
## Model Details
### Scenario: Open Model Details
**Objective**: Verify clicking a LoRA opens its details.
**Steps**:
1. Navigate to LoRA list page
2. Click on a LoRA card
3. Wait for details panel/modal to open
**Expected Result**: Details panel shows:
- Model name
- Preview image
- Metadata (trigger words, tags, etc.)
- Action buttons (edit, delete, etc.)
### Scenario: Edit Model Metadata
**Objective**: Verify metadata editing works end-to-end.
**Steps**:
1. Open a LoRA's details
2. Click "Edit" button
3. Modify trigger words field
4. Add/remove tags
5. Save changes
6. Refresh page
7. Reopen the same LoRA
**Expected Result**: Changes persist after refresh.
### Scenario: Delete Model
**Objective**: Verify model deletion works.
**Steps**:
1. Open a LoRA's details
2. Click "Delete" button
3. Confirm deletion in dialog
4. Wait for removal
**Expected Result**: Model removed from list, success message shown.
---
## Recipes
### Scenario: Recipe List Display
**Objective**: Verify recipes page loads and displays recipes.
**Steps**:
1. Navigate to `http://127.0.0.1:{PORT}/recipes`
2. Wait for "Recipes" title
3. Take snapshot
**Expected Result**: Recipe list displayed with cards/items.
### Scenario: Create New Recipe
**Objective**: Verify recipe creation workflow.
**Steps**:
1. Navigate to recipes page
2. Click "New Recipe" button
3. Fill recipe form:
- Name: "Test Recipe"
- Description: "E2E test recipe"
- Add LoRA models
4. Save recipe
5. Verify recipe appears in list
**Expected Result**: New recipe created and displayed.
### Scenario: Apply Recipe
**Objective**: Verify applying a recipe to ComfyUI.
**Steps**:
1. Open a recipe
2. Click "Apply" or "Load in ComfyUI"
3. Verify action completes
**Expected Result**: Recipe applied successfully.
---
## Settings
### Scenario: Settings Page Load
**Objective**: Verify settings page displays correctly.
**Steps**:
1. Navigate to `http://127.0.0.1:{PORT}/settings`
2. Wait for "Settings" title
3. Take snapshot
**Expected Result**: Settings form with various options displayed.
### Scenario: Change Setting and Restart
**Objective**: Verify settings persist after restart.
**Steps**:
1. Navigate to settings page
2. Change a setting (e.g., default view mode)
3. Save settings
4. Restart server: `python scripts/start_server.py --port {PORT} --restart --wait --timeout 30 --detach`
5. Refresh browser page
6. Navigate to settings
**Expected Result**: Changed setting value persists.
---
## Import/Export
### Scenario: Export Models List
**Objective**: Verify export functionality.
**Steps**:
1. Navigate to LoRA list
2. Click "Export" button
3. Select format (JSON/CSV)
4. Download file
**Expected Result**: File downloaded with correct data.
### Scenario: Import Models
**Objective**: Verify import functionality.
**Steps**:
1. Prepare import file
2. Navigate to import page
3. Upload file
4. Verify import results
**Expected Result**: Models imported successfully, confirmation shown.
---
## API Integration Tests
### Scenario: Verify API Endpoints
**Objective**: Verify backend API responds correctly.
**Test via browser console**:
```javascript
// List LoRAs
fetch('/loras/api/list').then(r => r.json()).then(console.log)
// Get LoRA details
fetch('/loras/api/detail/<id>').then(r => r.json()).then(console.log)
// Search LoRAs
fetch('/loras/api/search?q=test').then(r => r.json()).then(console.log)
```
**Expected Result**: APIs return valid JSON with expected structure.
---
## Console Error Monitoring
During all tests, monitor browser console for errors:
```python
# Check for JavaScript errors
messages = list_console_messages(types=["error"])
assert len(messages) == 0, f"Console errors found: {messages}"
```
## Network Request Verification
Verify key API calls are made:
```python
# List XHR requests
requests = list_network_requests(resourceTypes=["xhr", "fetch"])
# Look for specific endpoints
lora_list_requests = [r for r in requests if "/api/list" in r.get("url", "")]
assert len(lora_list_requests) > 0, "LoRA list API not called"
```
.agents/skills/lora-manager-e2e/scripts/example_e2e_test.py
-215
View File
@@ -1,215 +0,0 @@
#!/usr/bin/env python3
"""
Example E2E test demonstrating LoRa Manager testing workflow.
This script shows how to:
1. Start the standalone server
2. Use Chrome DevTools MCP to interact with the UI
3. Verify functionality end-to-end
Note: This is a template. Actual execution requires Chrome DevTools MCP.
Port: pick a FREE port for the run — 8188 is commonly occupied by a live
ComfyUI (see the skill's Port Selection section). Set PORT below to e.g. 8199
when 8188 is taken. Always run against a SANDBOXED standalone server.
"""
import subprocess
import sys
# Choose the E2E port. 8188 is only the default candidate; use 8199 (or any
# free port checked with `ss -tlnp`) when 8188 is occupied by a live ComfyUI.
PORT = "8188"
def run_test():
"""Run example E2E test flow."""
print("=" * 60)
print("LoRa Manager E2E Test Example")
print("=" * 60)
# Step 1: Start server (detached so it survives the shell)
print("\n[1/5] Starting LoRa Manager standalone server...")
result = subprocess.run(
[sys.executable, "start_server.py", "--port", PORT, "--wait", "--timeout", "30", "--detach"],
capture_output=True,
text=True,
)
if result.returncode != 0:
print(f"Failed to start server: {result.stderr}")
return 1
print("Server ready!")
# Step 2: Open Chrome (manual step - show command)
print("\n[2/5] Open Chrome with debug mode:")
print(
f"google-chrome --remote-debugging-port=9222 "
f"--user-data-dir=/tmp/chrome-lora-manager http://127.0.0.1:{PORT}/loras"
)
print("(In actual test, this would be automated via MCP)")
# Step 3: Navigate and verify page load
print("\n[3/5] Page Load Verification:")
print(
f"""
MCP Commands to execute:
1. navigate_page(type="url", url="http://127.0.0.1:{PORT}/loras")
2. wait_for(text="LoRAs", timeout=10000)
3. snapshot = take_snapshot()
"""
)
# Step 4: Test search functionality
print("\n[4/5] Search Functionality Test:")
print(
"""
MCP Commands to execute:
1. fill(uid="search-input", value="test")
2. press_key(key="Enter")
3. wait_for(text="Results", timeout=5000)
4. result = evaluate_script(function=`
() => {
const cards = document.querySelectorAll('.lora-card');
return { count: cards.length };
}
`)
"""
)
# Step 5: Verify API
print("\n[5/5] API Verification:")
print(
"""
MCP Commands to execute:
1. api_result = evaluate_script(function=`
async () => {
const response = await fetch('/loras/api/list');
const data = await response.json();
return { count: data.length, status: response.status };
}
`)
2. Verify api_result['status'] == 200
"""
)
print("\n" + "=" * 60)
print("Test flow completed!")
print("=" * 60)
return 0
def example_restart_flow():
"""Example: Testing configuration change that requires restart."""
print("\n" + "=" * 60)
print("Example: Server Restart Flow")
print("=" * 60)
print(
f"""
Scenario: Change setting and verify after restart
Steps:
1. Navigate to settings page
- navigate_page(type="url", url="http://127.0.0.1:{PORT}/settings")
2. Change a setting (e.g., theme)
- fill(uid="theme-select", value="dark")
- click(uid="save-settings-button")
3. Restart server
- subprocess.run([python, "start_server.py", "--port", "{PORT}", "--restart", "--wait", "--detach"])
4. Refresh browser
- navigate_page(type="reload", ignoreCache=True)
- wait_for(text="LoRAs", timeout=15000)
5. Verify setting persisted
- navigate_page(type="url", url="http://127.0.0.1:{PORT}/settings")
- theme = evaluate_script(function="() => document.querySelector('#theme-select').value")
- assert theme == "dark"
"""
)
def example_modal_interaction():
"""Example: Testing modal dialog interaction."""
print("\n" + "=" * 60)
print("Example: Modal Dialog Interaction")
print("=" * 60)
print(
"""
Scenario: Add new LoRA via modal
Steps:
1. Open modal
- click(uid="add-lora-button")
- wait_for(text="Add LoRA", timeout=3000)
2. Fill form
- fill_form(elements=[
{"uid": "lora-name", "value": "Test Character"},
{"uid": "lora-path", "value": "/models/test.safetensors"},
])
3. Submit
- click(uid="modal-submit-button")
4. Verify success
- wait_for(text="Successfully added", timeout=5000)
- snapshot = take_snapshot()
"""
)
def example_network_monitoring():
"""Example: Network request monitoring."""
print("\n" + "=" * 60)
print("Example: Network Request Monitoring")
print("=" * 60)
print(
f"""
Scenario: Verify API calls during user interaction
Steps:
1. Clear network log (implicit on navigation)
- navigate_page(type="url", url="http://127.0.0.1:{PORT}/loras")
2. Perform action that triggers API call
- fill(uid="search-input", value="character")
- press_key(key="Enter")
3. List network requests
- requests = list_network_requests(resourceTypes=["xhr", "fetch"])
4. Find search API call
- search_requests = [r for r in requests if "/api/search" in r.get("url", "")]
- assert len(search_requests) > 0, "Search API was not called"
5. Get request details
- if search_requests:
details = get_network_request(reqid=search_requests[0]["reqid"])
- Verify request method, response status, etc.
"""
)
if __name__ == "__main__":
print("LoRa Manager E2E Test Examples\n")
print("This script demonstrates E2E testing patterns.\n")
print("Note: Actual execution requires Chrome DevTools MCP connection.\n")
run_test()
example_restart_flow()
example_modal_interaction()
example_network_monitoring()
print("\n" + "=" * 60)
print("All examples shown!")
print("=" * 60)
.agents/skills/lora-manager-e2e/scripts/start_server.py
-357
View File
@@ -1,357 +0,0 @@
#!/usr/bin/env python3
"""
Start or restart LoRa Manager standalone server for E2E testing.
Backward-compatible CLI: --port, --restart, --wait, --timeout all work as before.
New options: --detach (setsid-style fully detached launch, survives shell death).
Safety rules implemented here:
- Never kill processes the script did not start. The script tracks the PIDs it
manages in a pidfile (/tmp/lora-manager-e2e-server-{PORT}.pid).
- If the port is held by an unrelated process (e.g. a live ComfyUI) the script
reports the conflict and exits early instead of killing it.
- --restart only kills managed PIDs; if unrelated processes still hold the port
afterwards, the script reports them and aborts.
"""
from __future__ import annotations
import argparse
import os
import signal
import socket
import subprocess
import sys
import time
PIDFILE_PREFIX = "/tmp/lora-manager-e2e-server"
def pidfile_path(port: int) -> str:
"""Path of the pidfile that records PIDs this script started for a port."""
return f"{PIDFILE_PREFIX}-{port}.pid"
def read_managed_pids(port: int) -> list[int]:
"""Read PIDs this script previously managed for the port (may be stale)."""
path = pidfile_path(port)
if not os.path.exists(path):
return []
try:
with open(path, "r", encoding="utf-8") as fh:
return [int(line.strip()) for line in fh if line.strip().isdigit()]
except (OSError, ValueError):
return []
def write_managed_pids(port: int, pids: list[int]) -> None:
"""Record PIDs this script manages for the port."""
try:
with open(pidfile_path(port), "w", encoding="utf-8") as fh:
for pid in pids:
fh.write(f"{pid}\n")
except OSError as exc:
print(f"Warning: could not write pidfile for port {port}: {exc}")
def clear_managed_pids(port: int) -> None:
"""Remove the pidfile for the port (no longer managed)."""
path = pidfile_path(port)
try:
if os.path.exists(path):
os.remove(path)
except OSError as exc:
print(f"Warning: could not remove pidfile {path}: {exc}")
def process_alive(pid: int) -> bool:
"""Return True if a process with the given pid exists."""
try:
os.kill(pid, 0)
return True
except ProcessLookupError:
return False
except PermissionError:
return True # exists but owned by someone else
def find_server_process(port: int) -> list[int]:
"""Find PIDs of processes listening on the given port."""
try:
result = subprocess.run(
["lsof", "-ti", f":{port}"],
capture_output=True,
text=True,
check=False,
)
if result.returncode == 0 and result.stdout.strip():
return [int(pid) for pid in result.stdout.strip().split("\n") if pid]
except FileNotFoundError:
# lsof not available, try netstat
try:
result = subprocess.run(
["netstat", "-tlnp"],
capture_output=True,
text=True,
check=False,
)
pids = []
for line in result.stdout.split("\n"):
if f":{port}" in line:
parts = line.split()
for part in parts:
if "/" in part:
try:
pid = int(part.split("/")[0])
pids.append(pid)
except ValueError:
pass
return pids
except FileNotFoundError:
pass
return []
def describe_processes(pids: list[int]) -> str:
"""Human-readable description of a pid list (pid + command line)."""
descriptions = []
for pid in pids:
cmdline = ""
try:
with open(f"/proc/{pid}/cmdline", "rb") as fh:
raw = fh.read().replace(b"\x00", b" ").decode("utf-8", "replace")
cmdline = raw.strip()
except OSError:
pass
descriptions.append(f"pid {pid}{' (' + cmdline + ')' if cmdline else ''}")
return ", ".join(descriptions) if descriptions else "none"
def kill_pids(pids: list[int], what: str) -> None:
"""Send SIGTERM (then SIGKILL) to the given PIDs, only after reporting."""
for pid in pids:
print(f"Sent SIGTERM to {what} pid {pid}")
try:
os.kill(pid, signal.SIGTERM)
except ProcessLookupError:
pass
# Wait for processes to terminate
deadline = time.time() + 5
while time.time() < deadline:
if not any(process_alive(pid) for pid in pids):
break
time.sleep(0.2)
# Force kill if still running
for pid in pids:
if process_alive(pid):
try:
os.kill(pid, signal.SIGKILL)
print(f"Sent SIGKILL to {what} pid {pid}")
except ProcessLookupError:
pass
def is_server_ready(port: int, timeout: float = 2.0) -> bool:
"""Check if server is accepting connections."""
try:
with socket.create_connection(("127.0.0.1", port), timeout=timeout):
return True
except (socket.timeout, ConnectionRefusedError, OSError):
return False
def wait_for_server(port: int, timeout: int = 30) -> bool:
"""Wait for server to become ready."""
start = time.time()
last_report = 0.0
while time.time() - start < timeout:
if is_server_ready(port):
return True
# Report progress every ~5s so a slow boot is visible, not silent.
elapsed = time.time() - start
if elapsed - last_report >= 5:
print(f" ...still waiting ({int(elapsed)}s/{timeout}s)")
last_report = elapsed
time.sleep(0.5)
return False
def main() -> int:
parser = argparse.ArgumentParser(
description="Start LoRa Manager standalone server for E2E testing"
)
parser.add_argument(
"--port",
type=int,
default=8188,
help="Server port (default: 8188)",
)
parser.add_argument(
"--restart",
action="store_true",
help="Kill the E2E server previously managed by this script for the port "
"(tracked via pidfile) before starting; refuse to kill unrelated processes",
)
parser.add_argument(
"--wait",
action="store_true",
help="Wait for server to be ready before exiting",
)
parser.add_argument(
"--timeout",
type=int,
default=30,
help="Timeout for waiting (default: 30)",
)
parser.add_argument(
"--detach",
action="store_true",
help="Launch the server fully detached (setsid-style) so it survives shell "
"death. REQUIRED for E2E: a plain background process dies with the shell",
)
parser.add_argument(
"--settings-path",
type=str,
default=None,
metavar="DIR",
help="Explicit settings directory passed to standalone.py (--settings-path, "
"equivalent to LORA_MANAGER_SETTINGS_DIR). settings.json, cache/, "
"wildcards/, backups/, logs/, stats/ all live under this directory instead "
"of the project root or the user config dir. Recommended for sandboxed E2E "
"so the real instance and the repo stay untouched",
)
args = parser.parse_args()
# Get project root (parent of .agents directory)
script_dir = os.path.dirname(os.path.abspath(__file__))
skill_dir = os.path.dirname(script_dir)
project_root = os.path.dirname(os.path.dirname(os.path.dirname(skill_dir)))
managed_pids = read_managed_pids(args.port)
# Restart if requested: kill ONLY managed PIDs.
if args.restart:
alive_managed = [pid for pid in managed_pids if process_alive(pid)]
if alive_managed:
print(
f"Killing E2E server previously started by this script on port "
f"{args.port} ({describe_processes(alive_managed)})..."
)
kill_pids(alive_managed, "managed E2E server")
else:
print(
f"No live managed E2E server for port {args.port} "
f"(pidfile: {pidfile_path(args.port)})"
)
time.sleep(1)
# Refuse to kill anything the script did not manage.
remaining = find_server_process(args.port)
if remaining:
print(
f"ERROR: port {args.port} is still held by process(es) this script "
f"did not start: {describe_processes(remaining)}."
)
print(
"These may be unrelated (e.g. a live ComfyUI). The script will NOT "
"kill them. Pick a different --port, or stop them manually if you "
"are certain they are stale E2E servers."
)
return 2
clear_managed_pids(args.port)
# Port conflict check before starting: never blind-kill.
port_pids = find_server_process(args.port)
if port_pids:
alive_managed = [pid for pid in port_pids if pid in managed_pids]
unmanaged = [pid for pid in port_pids if pid not in managed_pids]
if alive_managed and not unmanaged:
print(
f"Server already running on port {args.port} "
f"({describe_processes(alive_managed)}, started by this script). "
f"Use --restart to recycle it."
)
return 0
print(
f"ERROR: port {args.port} is already in use by process(es): "
f"{describe_processes(port_pids)}."
)
print(
"This is likely an unrelated process (e.g. a live ComfyUI holding 8188). "
"The script will NOT kill it. Pick a free port with --port, e.g. 8199."
)
return 2
# Start server
print(f"Starting LoRa Manager standalone server on port {args.port}...")
cmd = [
sys.executable,
"standalone.py",
"--host",
"127.0.0.1",
"--port",
str(args.port),
]
if args.settings_path:
settings_dir = os.path.abspath(os.path.expanduser(args.settings_path))
if os.path.exists(settings_dir) and not os.path.isdir(settings_dir):
print(
f"ERROR: --settings-path '{settings_dir}' exists but is not a directory."
)
return 2
os.makedirs(settings_dir, exist_ok=True)
cmd.extend(["--settings-path", settings_dir])
print(f"Settings directory: {settings_dir}")
if args.detach:
# Fully detached launch: new session (setsid), no controlling terminal,
# stdin from /dev/null, stdout/stderr to a log file. Survives the shell.
log_dir = os.path.join(script_dir, "logs")
os.makedirs(log_dir, exist_ok=True)
log_path = os.path.join(log_dir, f"server-{args.port}.log")
with open(log_path, "ab") as log_fh:
process = subprocess.Popen(
cmd,
cwd=project_root,
stdin=subprocess.DEVNULL,
stdout=log_fh,
stderr=subprocess.STDOUT,
start_new_session=True,
close_fds=True,
)
print(f"Detached server process started with PID {process.pid} (setsid)")
print(f"Log: {log_path}")
else:
# Plain background process (legacy behavior): dies with the shell.
process = subprocess.Popen(
cmd,
cwd=project_root,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
start_new_session=True,
)
print(f"Server process started with PID {process.pid}")
print(
"NOTE: not detached — this process dies when the launching shell exits. "
"For E2E use --detach."
)
write_managed_pids(args.port, [process.pid])
# Wait for ready if requested
if args.wait:
print(f"Waiting for server to be ready (timeout: {args.timeout}s)...")
if wait_for_server(args.port, args.timeout):
print(f"Server ready at http://127.0.0.1:{args.port}/loras")
return 0
print(f"Timeout waiting for server on port {args.port}")
return 1
print(f"Server starting at http://127.0.0.1:{args.port}/loras")
return 0
if __name__ == "__main__":
sys.exit(main())
.agents/skills/lora-manager-e2e/scripts/wait_for_server.py
-71
View File
@@ -1,71 +0,0 @@
#!/usr/bin/env python3
"""
Wait for LoRa Manager server to become ready.
Timeout is configurable via --timeout (default 30s); the script polls the port
until the server accepts connections or the timeout expires.
"""
from __future__ import annotations
import argparse
import socket
import sys
import time
def is_server_ready(port: int, timeout: float = 2.0) -> bool:
"""Check if server is accepting connections."""
try:
with socket.create_connection(("127.0.0.1", port), timeout=timeout):
return True
except (socket.timeout, ConnectionRefusedError, OSError):
return False
def wait_for_server(port: int, timeout: int = 30) -> bool:
"""Wait for server to become ready."""
start = time.time()
last_report = 0.0
while time.time() - start < timeout:
if is_server_ready(port):
return True
# Report progress every ~5s so a slow boot is visible, not silent.
elapsed = time.time() - start
if elapsed - last_report >= 5:
print(f" ...still waiting ({int(elapsed)}s/{timeout}s)")
last_report = elapsed
time.sleep(0.5)
return False
def main() -> int:
parser = argparse.ArgumentParser(
description="Wait for LoRa Manager server to become ready"
)
parser.add_argument(
"--port",
type=int,
default=8188,
help="Server port (default: 8188)",
)
parser.add_argument(
"--timeout",
type=int,
default=30,
help="Timeout in seconds (default: 30)",
)
args = parser.parse_args()
print(f"Waiting for server on port {args.port} (timeout: {args.timeout}s)...")
if wait_for_server(args.port, args.timeout):
print(f"Server ready at http://127.0.0.1:{args.port}/loras")
return 0
print(f"Timeout: Server not ready after {args.timeout}s")
return 1
if __name__ == "__main__":
sys.exit(main())