ComfyUI-Lora-Manager/.agents/skills/lora-manager-e2e/SKILL.md

name: lora-manager-e2e description: End-to-end testing and validation for LoRa Manager features. Use when performing automated E2E validation of LoRa Manager standalone mode in a SANDBOXED, disposable configuration: check the port, start/restart the standalone server on a free port, use Chrome DevTools MCP to interact with the web UI (http://127.0.0.1:{PORT}/loras), and verify frontend-to-backend functionality. Covers workflow validation, UI interaction testing, and integration testing between the standalone Python backend and the browser frontend. Trigger keywords: E2E, standalone, Chrome DevTools MCP, lora-manager-e2e, sandbox.

LoRa Manager E2E Testing

This skill provides workflows and utilities for end-to-end testing of LoRa Manager using Chrome DevTools MCP.

Conventions Used in This Document

• {PORT}: The server port. The default candidate is 8188, but 8188 is commonly occupied by a live ComfyUI process and MUST NOT be assumed to be free. Always check availability first (see Port Selection) and use a free port (e.g. 8199) for the E2E run. Substitute the actual port for every {PORT} in the commands below.
• <repo-root>: The repository/worktree root. Always run commands from the repo or worktree root; never assume a specific absolute path (paths such as /home/<user>/... differ per machine). The E2E scripts resolve the project root themselves, but fixture/settings paths are relative to <repo-root>.

SANDBOX (MANDATORY)

Read this section before running anything. Every E2E run MUST target a throwaway sandbox, never the real user data. A fresh subagent that skips this section WILL permanently mutate real user recipes.

1. Portable settings: create <repo-root>/settings.json (gitignored) with "use_portable_settings": true plus sandboxed folder_paths (lora/checkpoint roots) and recipes_path. This keeps the configuration inside the repo instead of the real user config dir (~/.config/ComfyUI-LoRA-Manager/settings.json).
2. Sandboxed paths: point folder_paths / recipes_path / example_images_path at disposable dirs — e.g. under /tmp/opencode/<plan-name>-e2e/ (or worktree-local dirs). NEVER point the E2E at the real library (~/models/...), real recipe dir, or real settings.
3. Never touch the real config: the real user config at ~/.config/ComfyUI-LoRA-Manager/settings.json and the real recipe dir must remain byte-identical before and after the run.
4. Record real-data protection proof before starting and after finishing:

   # BEFORE: snapshot real config + recipe library state
   sha256sum ~/.config/ComfyUI-LoRA-Manager/settings.json > /tmp/opencode/<plan>-e2e/settings.before.sha256
   ls ~/models/recipes/*.recipe.json 2>/dev/null | wc -l > /tmp/opencode/<plan>-e2e/recipes-count.before.txt
   find ~/models/recipes -name '*.recipe.json' -newermt "$(date -Iseconds)" | head   # expect empty after run
   # AFTER: record again, then diff the two snapshots. Any change = the run leaked into real data.
   

   Also confirm <repo-root>/git status stays clean for settings.json/cache/ (both are gitignored).

Portable Settings Example

{
  "use_portable_settings": true,
  "folder_paths": {
    "loras": ["/tmp/opencode/<plan>-e2e/models/loras"],
    "checkpoints": ["/tmp/opencode/<plan>-e2e/models/checkpoints"],
    "unet": ["/tmp/opencode/<plan>-e2e/models/checkpoints"],
    "diffusers": []
  },
  "recipes_path": "/tmp/opencode/<plan>-e2e/recipes",
  "example_images_path": "/tmp/opencode/<plan>-e2e/example_images"
}

The scanner computes and persists model hashes during the library scan, so the sandbox model dirs just need the model files + .metadata.json sidecars (see Fixture + Fresh-State Guidance).

Time Budgets & Abort Guidance

A fresh subagent should complete a sandboxed standalone E2E in well under 30 minutes. Budget each phase:

Phase	Expected duration	Abort if
Port check + sandbox setup	< 2 min	—
Server start (detached) + readiness	< 30 s	> 60 s (2x) → stop
Chrome DevTools MCP connect	< 1 min	> 2 min → stop
Per entry-point run (after fixtures ready)	< 5 min	> 10 min (2x) → stop
Fixture reset + cache clear between runs	< 1 min	> 2 min → stop

Abort rule: if a phase exceeds ~2x its budget, OR any single tool call fails/retries 3+ times in a row, STOP. Do not loop or retry blindly. Report BLOCKED with: the phase, the last observed state (server PID + ss -tlnp output, page snapshot, last API response), and the suspected cause. Record the partial state as evidence; a clean BLOCKED report is more valuable than an hour of retries.

Prerequisites

• LoRa Manager project cloned and dependencies installed (pip install -r requirements.txt) — run everything from <repo-root>
• Chrome browser available for debugging
• Chrome DevTools MCP connected
• ss (or lsof/netstat) available for port checks: ss -tlnp

Port Selection

8188 is only the default candidate. Verify it is actually free before every run:

# Is anything listening on 8188?
ss -tlnp | grep ':8188' || echo "8188 is free"

• If a process holds 8188 (e.g. a live ComfyUI — pid 6575 on this machine), pick a different free port, e.g. 8199:

  ss -tlnp | grep ':8199' || echo "8199 is free"
  

• Never kill a process you did not start for this E2E. The live ComfyUI is off-limits. Pick a free port instead.
• Use your chosen port for all subsequent commands (server, Chrome launch, browser URLs).

Quick Start Workflow (sandboxed)

1. Prepare the sandbox

cd <repo-root>                       # ALWAYS run from the repo/worktree root
mkdir -p /tmp/opencode/<plan>-e2e/models/{loras,checkpoints}
mkdir -p /tmp/opencode/<plan>-e2e/{recipes,example_images,recipes-before}
# write <repo-root>/settings.json per the portable-settings example above
# record real-data protection proof (see SANDBOX section)

2. Check port availability

ss -tlnp | grep ':{PORT}' || echo "port {PORT} is free"

If {PORT} is occupied by an unrelated process, pick a free one and use it everywhere below. When in doubt use 8199.

3. Start LoRa Manager Standalone (detached)

The standalone server dies with the shell unless launched fully detached — a plain background & from the bash tool is killed when the tool call returns. Launch via the helper script:

python .agents/skills/lora-manager-e2e/scripts/start_server.py --port {PORT} --wait --timeout 30 --detach

Or manually (equivalent detached form):

setsid nohup python standalone.py --port {PORT} --host 127.0.0.1 < /dev/null \
  >> /tmp/opencode/<plan>-e2e/server.log 2>&1 &
echo "started"   # record the printed/pidfile PID for cleanup

Verify it is listening before proceeding (readiness poll is not a substitute for this):

ss -tlnp | grep ':{PORT}'

Record the server PID for cleanup: the helper script writes it to /tmp/lora-manager-e2e-server-{PORT}.pid; a manual setsid launch has no pidfile, so capture it explicitly (e.g. from ss -tlnp).

4. Open Chrome Debug Mode

# Chrome with remote debugging on port 9222 (note the {PORT} URL)
google-chrome --remote-debugging-port=9222 --user-data-dir=/tmp/chrome-lora-manager http://127.0.0.1:{PORT}/loras

5. Connect Chrome DevTools MCP

Ensure the MCP server is connected to Chrome at http://localhost:9222. Verify with list_pages — if it fails with "browser is already running", see Chrome DevTools MCP Troubleshooting.

6. Navigate and Interact

Use Chrome DevTools MCP tools to:

• Take snapshots: take_snapshot
• Click elements: click
• Fill forms: fill or fill_form
• Evaluate scripts: evaluate_script
• Wait for elements: wait_for

Common E2E Test Patterns

Pattern: Full Page Load Verification

# Navigate to LoRA list page
navigate_page(type="url", url="http://127.0.0.1:{PORT}/loras")

# Wait for page to load
wait_for(text="LoRAs", timeout=10000)

# Take snapshot to verify UI state
snapshot = take_snapshot()

Pattern: Restart Server for Configuration Changes

# Stop current server (if running), start with new configuration.
# --restart only kills the E2E server this script started before (via its pidfile);
# it refuses to blindly kill unrelated processes on the port.
python .agents/skills/lora-manager-e2e/scripts/start_server.py --port {PORT} --restart --wait --detach

# Wait and refresh browser
navigate_page(type="reload", ignoreCache=True)
wait_for(text="LoRAs", timeout=15000)

Pattern: Verify Backend API via Frontend

# Execute script in browser to call backend API
result = evaluate_script(function="""
async () => {
  const response = await fetch('/loras/api/list');
  const data = await response.json();
  return { count: data.length, firstItem: data[0]?.name };
}
""")

Pattern: Form Submission Flow

# Fill a form (e.g., search or filter)
fill_form(elements=[
    {"uid": "search-input", "value": "character"},
])

# Click submit button
click(uid="search-button")

# Wait for results
wait_for(text="Results", timeout=5000)

# Verify results via snapshot
snapshot = take_snapshot()

Pattern: Modal Dialog Interaction

# Open modal (e.g., add LoRA)
click(uid="add-lora-button")

# Wait for modal to appear
wait_for(text="Add LoRA", timeout=3000)

# Fill modal form
fill_form(elements=[
    {"uid": "lora-name", "value": "Test LoRA"},
    {"uid": "lora-path", "value": "/path/to/lora.safetensors"},
])

# Submit
click(uid="modal-submit-button")

# Wait for success message or close
wait_for(text="Success", timeout=5000)

Fixture + Fresh-State Guidance

For rematch/repair E2E runs, seed the sandboxed recipes_path with hand-written fixture recipes. Rules (validated by the task-8 E2E):

1. Filename constraint: each file MUST be named f"{id}.recipe.json" and the in-JSON id field MUST equal the filename. Discovery accepts any *.recipe.json, but persistence resolves the path via get_recipe_json_path and _save_recipe_persistently returns False on a mismatch → the fixture would be counted as an error.
   • recipe-a.recipe.json → in-JSON "id": "recipe-a"
2. File format: mirror an existing recipe JSON — top-level id, file_path, title, loras, fingerprint, gen_params; lora entries per the persistence conventions (hash, file_name, modelVersionId, isDeleted, ...).
3. Companion image: each recipe needs an image (e.g. a .webp generated with PIL) referenced by file_path, used for EXIF verification (ExifUtils.append_recipe_metadata writes a "Recipe metadata: ..." marker; a freshly generated .webp with no marker is the clean "untouched" control).
4. autov3 three-state contract: for L3 (autov3-only, renamed-file) fixtures the local model's .metadata.json sidecar MUST have the autov3 key ABSENT (the "unchecked" state), NOT "" — "" is the TERMINAL "checked but unavailable" state that L3 deliberately skips. The scanner computes + persists autov3 from the file header during the normal library scan (model_scanner.py _process_model_file), so the live L3 match resolves through the local autov3/hash cache; the computed-autov3 branch for unchecked items is covered by the unit suite.
5. Fixture design for a rematch run (mirrors the task-8 E2E):
   • recipe-a: lora entry isDeleted=True, hash = 12-char autov3 computed from the local model (calculate_autov3, py/utils/file_utils.py), whose local model file was RENAMED after the recipe was written so file_name differs (proves L3 match without filename).
   • recipe-b: parser-convention checkpoint entry (uses id, no modelVersionId) matching a local checkpoint via L2 — the local checkpoint's .metadata.json MUST carry civitai version data with that id so version_index contains it (L2 cannot match otherwise).
   • recipe-c: healthy recipe (no deleted entries) → must remain untouched.

Fresh state between entry-point runs

Each entry point (global / per-recipe / selection-bulk) must start from the same deleted state. Between runs:

# 1. Reset fixtures to the before-state snapshot (copy back from recipes-before/)
cp /tmp/opencode/<plan>-e2e/recipes-before/*.recipe.json /tmp/opencode/<plan>-e2e/recipes/
# 2. Clear the recipe/FTS caches so the stale in-memory/library state is gone
rm -f <repo-root>/cache/recipe/*.sqlite
rm -rf <repo-root>/cache/fts/*
# 3. Restart the server (fresh process, fresh scan)
python .agents/skills/lora-manager-e2e/scripts/start_server.py --port {PORT} --restart --wait --timeout 30 --detach
# 4. Re-verify server listening + reload the browser page

Server Lifecycle

• Detached launch is mandatory: the standalone server dies with the shell unless launched via setsid (or the helper script's --detach). Use setsid nohup python standalone.py --port {PORT} --host 127.0.0.1 ... < /dev/null &.
• Verify with ss -tlnp after every (re)start; do not proceed on a blind "server starting" message.
• Never kill pre-existing processes — only kill the E2E server PID you started (start_server.py --restart kills only PIDs it manages via its pidfile). The live ComfyUI or a stale QA Chrome must never be killed as part of cleanup unless explicitly identified as such (see Chrome troubleshooting).
• Record your PID for cleanup: note the PID printed/pidfile, and stop exactly that PID at the end (kill <PID>, then confirm with ss -tlnp that {PORT} is released).

Chrome DevTools MCP Troubleshooting

Stale profile lock ("browser is already running" / list_pages fails)

A Chrome profile can be held by a stale Chrome from a prior MCP session, which makes list_pages fail with "browser is already running":

1. Identify the stale Chrome — it owns the profile dir in --user-data-dir (e.g. ~/.config/chrome-dev-profile). Find its process:

   ps -ef | grep -i '[c]hrome.*user-data-dir'
   

2. Confirm it is a QA Chrome from a completed task (its parent is an old MCP/browser process, it is NOT the live ComfyUI server, and it is NOT your current MCP instance).
3. Kill ONLY that stale Chrome:

   kill <stale-chrome-pid>
   

   Never kill the live server or unrelated processes.
4. Retry list_pages. The current MCP will spawn a fresh browser.

Screenshot-write restrictions

The chrome-devtools MCP may refuse to write into paths outside its configured workspace roots (e.g. the worktree .omo/evidence/... canonicalizing to an unmapped path). Workaround:

# 1. Save the screenshot to /tmp via the MCP
#    take_screenshot(filePath="/tmp/<plan>-e2e/recipe-b-after.png", format="png")
# 2. Copy it into the evidence dir from the shell
mkdir -p <repo-root>/.omo/evidence/screenshots
cp /tmp/<plan>-e2e/recipe-b-after.png <repo-root>/.omo/evidence/screenshots/

Cancellation Testing (KNOWN GAP)

Testing the rematch-cancel path E2E requires a run long enough to cancel mid-flight. A tiny 3-recipe fixture set completes in seconds — too fast to reliably cancel. The cancel path is currently unit-covered only (rematch_all_recipes cancellation tests); do not block an E2E run on cancel-path verification. If you must attempt it, you would need an artificially large/deferred fixture set to create a cancellable window — treat this as a research task, not part of the standard E2E.

Available Scripts

scripts/start_server.py

Starts or restarts the LoRa Manager standalone server for E2E testing.

python scripts/start_server.py [--port PORT] [--restart] [--wait] [--timeout SECONDS] [--detach]

Options:

• --port: Server port (default: 8188). The script exits early with a clear message if the port is already in use by an unrelated process.
• --restart: Kill the E2E server this script previously managed (tracked via /tmp/lora-manager-e2e-server-{PORT}.pid) before starting. If unrelated processes still hold the port after that, the script reports them and aborts instead of killing them.
• --wait: Wait for the server to be ready before exiting.
• --timeout: Readiness wait timeout in seconds (default: 30).
• --detach: Launch the server fully detached (setsid-style, survives shell death — REQUIRED for E2E). Default off: a normal background process that dies with the shell.

scripts/wait_for_server.py

Polls the server until ready or timeout.

python scripts/wait_for_server.py [--port PORT] [--timeout SECONDS]

Test Scenarios Reference

See references/test-scenarios.md for detailed test scenarios including:

• LoRA list display and filtering
• Model metadata editing
• Recipe creation and management
• Settings configuration
• Import/export functionality

Network Request Verification

Use list_network_requests and get_network_request to verify API calls:

# List recent XHR/fetch requests
requests = list_network_requests(resourceTypes=["xhr", "fetch"])

# Get details of specific request
details = get_network_request(reqid=123)

Console Message Monitoring

# Check for errors or warnings
messages = list_console_messages(types=["error", "warn"])

Performance Testing

# Start performance trace
performance_start_trace(reload=True, autoStop=False)

# Perform actions...

# Stop and analyze
results = performance_stop_trace()

Cleanup

Always ensure proper cleanup after tests:

1. Stop the standalone server: kill <recorded-pid> (only the PID you started), then confirm ss -tlnp | grep ':{PORT}' is empty.
2. Close browser pages (keep at least one open).
3. Remove the sandbox: rm -rf /tmp/opencode/<plan>-e2e and <repo-root>/settings.json + <repo-root>/cache (both gitignored).
4. Re-run the real-data protection check from the SANDBOX section and record the result in your evidence.