mirror of
https://github.com/willmiao/ComfyUI-Lora-Manager.git
synced 2026-09-21 11:11:26 -03:00
Compare commits
8 Commits
574dfbbe55
...
e914a0e19d
| Author | SHA1 | Date | |
|---|---|---|---|
 Will Miao
|
e914a0e19d | ||
|
Will Miao
|
2ba04bb1bd | ||
|
Will Miao
|
1b7314591a | ||
|
Will Miao
|
2bfb987312 | ||
|
Will Miao
|
df34efafbc | ||
|
Will Miao
|
c2a2048c8b | ||
|
Will Miao
|
1e1921cabb | ||
|
Will Miao
|
ee233548e5 |
@@ -1,385 +1,146 @@
|
||||
---
|
||||
name: lora-manager-e2e
|
||||
description: "End-to-end testing and validation for LoRa Manager features. Use when performing automated E2E validation of LoRa Manager standalone mode in a SANDBOXED, disposable configuration: check the port, start/restart the standalone server on a free port, use Chrome DevTools MCP to interact with the web UI (http://127.0.0.1:{PORT}/loras), and verify frontend-to-backend functionality. Covers workflow validation, UI interaction testing, and integration testing between the standalone Python backend and the browser frontend. Trigger keywords: E2E, standalone, Chrome DevTools MCP, lora-manager-e2e, sandbox."
|
||||
description: "End-to-end testing and validation for LoRa Manager features. Use ONLY for sandboxed E2E validation of LoRa Manager standalone mode: start the standalone server on a free port with --settings-path, drive the web UI (http://127.0.0.1:{PORT}/loras) via Chrome DevTools MCP, and verify frontend-to-backend integration. NOT for UI behavior checks that unit tests (Vitest/jsdom) can cover. Trigger keywords: E2E, standalone, Chrome DevTools MCP, lora-manager-e2e, sandbox."
|
||||
---
|
||||
|
||||
# LoRa Manager E2E Testing
|
||||
|
||||
This skill provides workflows and utilities for end-to-end testing of LoRa Manager using Chrome DevTools MCP.
|
||||
End-to-end testing of LoRa Manager standalone mode using Chrome DevTools MCP.
|
||||
|
||||
## Conventions Used in This Document
|
||||
## When to Use — and When NOT To
|
||||
|
||||
- **`{PORT}`**: The server port. The default candidate is `8188`, but **`8188` is commonly occupied by a live ComfyUI process** and MUST NOT be assumed to be free. Always check availability first (see [Port Selection](#port-selection)) and use a free port (e.g. `8199`) for the E2E run. Substitute the actual port for every `{PORT}` in the commands below.
|
||||
- **`<repo-root>`**: The repository/worktree root. Always run commands from the repo or worktree root; never assume a specific absolute path (paths such as `/home/<user>/...` differ per machine). The E2E scripts resolve the project root themselves, but fixture/settings paths are relative to `<repo-root>`.
|
||||
- **`<settings-dir>`**: The sandboxed explicit settings directory passed via `--settings-path` (see [SANDBOX](#sandbox-mandatory)); substitute the actual path (e.g. `/tmp/opencode/<plan>-e2e/settings`) for every `{PATH}` in commands below that target the sandbox config.
|
||||
E2E runs are slow and token-heavy. Reach for them only when the question genuinely
|
||||
spans server + browser (routing, scan persistence, websocket updates, EXIF writes).
|
||||
|
||||
- **Default to unit/component tests first**: `npm run test:js` (Vitest/jsdom) covers
|
||||
DOM rendering, modal behavior, event handling and API-client calls deterministically
|
||||
in seconds. Backend logic goes through `pytest`. A UI-behavior question answered by
|
||||
jsdom MUST NOT be escalated to E2E.
|
||||
- **Use E2E only when** the behavior cannot be observed without a live server and a
|
||||
real browser, e.g. template rendering through the aiohttp server, scanner → SQLite
|
||||
persistence → API → DOM round-trips, or real EXIF/image writes.
|
||||
- If you start an E2E and realize a unit test would answer the question, stop and
|
||||
switch.
|
||||
|
||||
**Browser driver is fixed: Chrome DevTools MCP.** Do not substitute kimi-webbridge —
|
||||
it operates on the user's real browser (real tabs, real sessions, synthetic
|
||||
`isTrusted=false` events), which breaks the isolation this skill requires and lacks
|
||||
the console/network inspection E2E debugging relies on. kimi-webbridge is for
|
||||
interactive browsing with the user's real login sessions, not for sandboxed E2E.
|
||||
|
||||
## Conventions
|
||||
|
||||
- **`{PORT}`**: default candidate `8188`, but it is **commonly occupied by a live
|
||||
ComfyUI** — always check first (`ss -tlnp | grep ':{PORT}'`) and use a free port
|
||||
(e.g. `8199`). Substitute the chosen port everywhere below. Never kill a process
|
||||
you did not start for this E2E.
|
||||
- **`<repo-root>`**: the repository/worktree root; run all commands from there.
|
||||
- **`<sandbox>`**: a throwaway dir, e.g. `/tmp/opencode/<plan>-e2e`.
|
||||
|
||||
## SANDBOX (MANDATORY)
|
||||
|
||||
> **Read this section before running anything.** Every E2E run MUST target a throwaway sandbox, never the real user data. A fresh subagent that skips this section WILL permanently mutate real user recipes.
|
||||
> Every E2E run MUST target a throwaway sandbox, never real user data.
|
||||
|
||||
1. **Explicit settings directory (preferred)**: launch the standalone server with `--settings-path <sandbox>/settings` (or set `LORA_MANAGER_SETTINGS_DIR`). This pins ALL runtime data — `settings.json`, `cache/`, `wildcards/`, `backups/`, `logs/`, `stats/` — under that directory, independent of portable mode and of the real user config dir. **Do NOT** write `<repo-root>/settings.json` for sandboxing: the repo folder is usually the real ComfyUI plugin folder, and a portable `settings.json` there is read by the real instance — exactly the conflict this E2E must avoid.
|
||||
2. **Sandboxed paths**: point `folder_paths` / `recipes_path` / `example_images_path` at disposable dirs under the sandbox — e.g. `<sandbox>/models/loras`, `<sandbox>/recipes`. NEVER point the E2E at the real library (`~/models/...`), real recipe dir, or real settings.
|
||||
3. **Never touch the real config**: the real user config at `~/.config/ComfyUI-LoRA-Manager/settings.json` and the real recipe dir must remain byte-identical before and after the run.
|
||||
4. **Record real-data protection proof** before starting and after finishing:
|
||||
```bash
|
||||
# BEFORE: snapshot real config + recipe library state
|
||||
sha256sum ~/.config/ComfyUI-LoRA-Manager/settings.json > /tmp/opencode/<plan>-e2e/settings.before.sha256
|
||||
ls ~/models/recipes/*.recipe.json 2>/dev/null | wc -l > /tmp/opencode/<plan>-e2e/recipes-count.before.txt
|
||||
find ~/models/recipes -name '*.recipe.json' -newermt "$(date -Iseconds)" | head # expect empty after run
|
||||
# AFTER: record again, then diff the two snapshots. Any change = the run leaked into real data.
|
||||
1. **Explicit settings directory**: always launch with `--settings-path <sandbox>/settings`.
|
||||
This pins ALL runtime data (`settings.json`, `cache/`, `backups/`, `logs/`, `stats/`,
|
||||
`wildcards/`) under the sandbox. **Never** create `<repo-root>/settings.json` — the repo
|
||||
folder is usually the real ComfyUI plugin folder and a portable settings file there is
|
||||
read by the real instance.
|
||||
2. **Sandboxed library paths**: point `folder_paths` / `recipes_path` /
|
||||
`example_images_path` at disposable dirs under `<sandbox>` — never the real library,
|
||||
real recipe dir, or real settings:
|
||||
|
||||
```json
|
||||
{
|
||||
"folder_paths": {
|
||||
"loras": ["<sandbox>/models/loras"],
|
||||
"checkpoints": ["<sandbox>/models/checkpoints"],
|
||||
"unet": ["<sandbox>/models/checkpoints"],
|
||||
"diffusers": []
|
||||
},
|
||||
"recipes_path": "<sandbox>/recipes",
|
||||
"example_images_path": "<sandbox>/example_images"
|
||||
}
|
||||
```
|
||||
Also confirm `<repo-root>/git status` stays clean (`settings.json`/`cache/` are gitignored and must not be created by the run).
|
||||
|
||||
### Sandbox Settings (via `--settings-path`)
|
||||
3. **Real-data protection proof**: before starting and after finishing, snapshot the real
|
||||
config and recipe library and confirm they are byte-identical; also confirm
|
||||
`<repo-root>` gained no `settings.json` or `cache/`:
|
||||
|
||||
Write this file as `<sandbox>/settings/settings.json` — `<settings-dir>` in the commands below:
|
||||
```bash
|
||||
sha256sum ~/.config/ComfyUI-LoRA-Manager/settings.json > <sandbox>/settings.before.sha256
|
||||
ls ~/models/recipes/*.recipe.json 2>/dev/null | wc -l > <sandbox>/recipes-count.before.txt
|
||||
# AFTER the run: record again and diff. Any change = the run leaked into real data.
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"folder_paths": {
|
||||
"loras": ["/tmp/opencode/<plan>-e2e/models/loras"],
|
||||
"checkpoints": ["/tmp/opencode/<plan>-e2e/models/checkpoints"],
|
||||
"unet": ["/tmp/opencode/<plan>-e2e/models/checkpoints"],
|
||||
"diffusers": []
|
||||
},
|
||||
"recipes_path": "/tmp/opencode/<plan>-e2e/recipes",
|
||||
"example_images_path": "/tmp/opencode/<plan>-e2e/example_images"
|
||||
}
|
||||
```
|
||||
|
||||
The scanner computes and persists model hashes during the library scan, so the sandbox model dirs just need the model files + `.metadata.json` sidecars (see [Fixture + Fresh-State Guidance](#fixture--fresh-state-guidance)). With `--settings-path`, all derived data lands under `<settings-dir>` (`cache/`, `backups/`, `logs/`, `stats/`, `wildcards/`), and NO `cache/` appears in `<repo-root>`.
|
||||
|
||||
## Time Budgets & Abort Guidance
|
||||
|
||||
A fresh subagent should complete a sandboxed standalone E2E **in well under 30 minutes**. Budget each phase:
|
||||
|
||||
| Phase | Expected duration | Abort if |
|
||||
| --- | --- | --- |
|
||||
| Port check + sandbox setup | < 2 min | — |
|
||||
| Server start (detached) + readiness | < 30 s | > 60 s (2x) → stop |
|
||||
| Chrome DevTools MCP connect | < 1 min | > 2 min → stop |
|
||||
| Per entry-point run (after fixtures ready) | < 5 min | > 10 min (2x) → stop |
|
||||
| Fixture reset + cache clear between runs | < 1 min | > 2 min → stop |
|
||||
|
||||
**Abort rule**: if a phase exceeds ~2x its budget, OR any single tool call fails/retries 3+ times in a row, **STOP**. Do not loop or retry blindly. Report `BLOCKED` with: the phase, the last observed state (server PID + `ss -tlnp` output, page snapshot, last API response), and the suspected cause. Record the partial state as evidence; a clean BLOCKED report is more valuable than an hour of retries.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- LoRa Manager project cloned and dependencies installed (`pip install -r requirements.txt`) — run everything from `<repo-root>`
|
||||
- Chrome browser available for debugging
|
||||
- Chrome DevTools MCP connected
|
||||
- `ss` (or `lsof`/`netstat`) available for port checks: `ss -tlnp`
|
||||
|
||||
## Port Selection
|
||||
|
||||
`8188` is only the *default candidate*. Verify it is actually free before every run:
|
||||
|
||||
```bash
|
||||
# Is anything listening on 8188?
|
||||
ss -tlnp | grep ':8188' || echo "8188 is free"
|
||||
```
|
||||
|
||||
- If a process holds `8188` (e.g. a live ComfyUI — pid 6575 on this machine), pick a different free port, e.g. `8199`:
|
||||
```bash
|
||||
ss -tlnp | grep ':8199' || echo "8199 is free"
|
||||
```
|
||||
- **Never** kill a process you did not start for this E2E. The live ComfyUI is off-limits. Pick a free port instead.
|
||||
- Use your chosen port for **all** subsequent commands (server, Chrome launch, browser URLs).
|
||||
|
||||
## Quick Start Workflow (sandboxed)
|
||||
|
||||
### 1. Prepare the sandbox
|
||||
|
||||
```bash
|
||||
cd <repo-root> # ALWAYS run from the repo/worktree root
|
||||
mkdir -p /tmp/opencode/<plan>-e2e/settings
|
||||
mkdir -p /tmp/opencode/<plan>-e2e/models/{loras,checkpoints}
|
||||
mkdir -p /tmp/opencode/<plan>-e2e/{recipes,example_images,recipes-before}
|
||||
# write <sandbox>/settings/settings.json per the sandbox-settings example above
|
||||
# record real-data protection proof (see SANDBOX section)
|
||||
```
|
||||
|
||||
### 2. Check port availability
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
cd <repo-root>
|
||||
# 1. Sandbox
|
||||
mkdir -p <sandbox>/settings <sandbox>/models/{loras,checkpoints} <sandbox>/{recipes,example_images}
|
||||
# write <sandbox>/settings/settings.json per the SANDBOX example
|
||||
# 2. Port
|
||||
ss -tlnp | grep ':{PORT}' || echo "port {PORT} is free"
|
||||
```
|
||||
|
||||
If `{PORT}` is occupied by an unrelated process, pick a free one and use it everywhere below. When in doubt use `8199`.
|
||||
|
||||
### 3. Start LoRa Manager Standalone (detached)
|
||||
|
||||
The standalone server **dies with the shell unless launched fully detached** — a plain background `&` from the bash tool is killed when the tool call returns. Launch via the helper script (note `--settings-path`):
|
||||
|
||||
```bash
|
||||
# 3. Server — MUST be fully detached (a plain background & dies with the shell);
|
||||
# the helper enforces this and manages its own pidfile
|
||||
python .agents/skills/lora-manager-e2e/scripts/start_server.py \
|
||||
--port {PORT} --settings-path /tmp/opencode/<plan>-e2e/settings \
|
||||
--wait --timeout 30 --detach
|
||||
```
|
||||
|
||||
Or manually (equivalent detached form):
|
||||
|
||||
```bash
|
||||
setsid nohup python standalone.py --port {PORT} --settings-path /tmp/opencode/<plan>-e2e/settings --host 127.0.0.1 < /dev/null \
|
||||
>> /tmp/opencode/<plan>-e2e/server.log 2>&1 &
|
||||
echo "started" # record the printed/pidfile PID for cleanup
|
||||
```
|
||||
|
||||
Verify it is listening **before** proceeding (readiness poll is not a substitute for this):
|
||||
|
||||
```bash
|
||||
ss -tlnp | grep ':{PORT}'
|
||||
```
|
||||
|
||||
Record the server PID for cleanup: the helper script writes it to `/tmp/lora-manager-e2e-server-{PORT}.pid`; a manual `setsid` launch has no pidfile, so capture it explicitly (e.g. from `ss -tlnp`).
|
||||
|
||||
### 4. Open Chrome Debug Mode
|
||||
|
||||
```bash
|
||||
# Chrome with remote debugging on port 9222 (note the {PORT} URL)
|
||||
--port {PORT} --settings-path <sandbox>/settings --wait --timeout 30 --detach
|
||||
ss -tlnp | grep ':{PORT}' # verify listening BEFORE proceeding
|
||||
# 4. Chrome with remote debugging, then connect Chrome DevTools MCP (verify via list_pages)
|
||||
google-chrome --remote-debugging-port=9222 --user-data-dir=/tmp/chrome-lora-manager http://127.0.0.1:{PORT}/loras
|
||||
```
|
||||
|
||||
### 5. Connect Chrome DevTools MCP
|
||||
Then drive the UI with the MCP tools (`take_snapshot`, `click`, `fill`, `fill_form`,
|
||||
`evaluate_script`, `wait_for`, `list_network_requests`, `list_console_messages`) —
|
||||
see [references/mcp-cheatsheet.md](references/mcp-cheatsheet.md) for patterns.
|
||||
|
||||
Ensure the MCP server is connected to Chrome at `http://localhost:9222`. Verify with `list_pages` — if it fails with "browser is already running", see [Chrome DevTools MCP Troubleshooting](#chrome-devtools-mcp-troubleshooting).
|
||||
Server restart after config/fixture changes:
|
||||
|
||||
### 6. Navigate and Interact
|
||||
|
||||
Use Chrome DevTools MCP tools to:
|
||||
- Take snapshots: `take_snapshot`
|
||||
- Click elements: `click`
|
||||
- Fill forms: `fill` or `fill_form`
|
||||
- Evaluate scripts: `evaluate_script`
|
||||
- Wait for elements: `wait_for`
|
||||
|
||||
## Common E2E Test Patterns
|
||||
|
||||
### Pattern: Full Page Load Verification
|
||||
|
||||
```python
|
||||
# Navigate to LoRA list page
|
||||
navigate_page(type="url", url="http://127.0.0.1:{PORT}/loras")
|
||||
|
||||
# Wait for page to load
|
||||
wait_for(text="LoRAs", timeout=10000)
|
||||
|
||||
# Take snapshot to verify UI state
|
||||
snapshot = take_snapshot()
|
||||
```
|
||||
|
||||
### Pattern: Restart Server for Configuration Changes
|
||||
|
||||
```python
|
||||
# Stop current server (if running), start with new configuration.
|
||||
# --restart only kills the E2E server this script started before (via its pidfile);
|
||||
# it refuses to blindly kill unrelated processes on the port.
|
||||
```bash
|
||||
python .agents/skills/lora-manager-e2e/scripts/start_server.py \
|
||||
--port {PORT} --settings-path /tmp/opencode/<plan>-e2e/settings \
|
||||
--restart --wait --detach
|
||||
|
||||
# Wait and refresh browser
|
||||
navigate_page(type="reload", ignoreCache=True)
|
||||
wait_for(text="LoRAs", timeout=15000)
|
||||
--port {PORT} --settings-path <sandbox>/settings --restart --wait --detach
|
||||
# then reload the browser page (ignoreCache=True)
|
||||
```
|
||||
|
||||
### Pattern: Verify Backend API via Frontend
|
||||
`--restart` only kills the E2E server the script itself started (via its pidfile) and
|
||||
aborts instead of killing unrelated processes on the port.
|
||||
|
||||
```python
|
||||
# Execute script in browser to call backend API
|
||||
result = evaluate_script(function="""
|
||||
async () => {
|
||||
const response = await fetch('/loras/api/list');
|
||||
const data = await response.json();
|
||||
return { count: data.length, firstItem: data[0]?.name };
|
||||
}
|
||||
""")
|
||||
```
|
||||
## Abort Rule
|
||||
|
||||
### Pattern: Form Submission Flow
|
||||
A sandboxed E2E should finish in well under 30 minutes. If any phase exceeds ~2x its
|
||||
expected duration (server readiness > 60 s, MCP connect > 2 min, a single scenario >
|
||||
10 min), or any single tool call fails 3+ times in a row, **STOP** — do not retry
|
||||
blindly. Report `BLOCKED` with the phase, last observed state (server PID,
|
||||
`ss -tlnp` output, page snapshot, last API response) and suspected cause. A clean
|
||||
BLOCKED report beats an hour of retries.
|
||||
|
||||
```python
|
||||
# Fill a form (e.g., search or filter)
|
||||
fill_form(elements=[
|
||||
{"uid": "search-input", "value": "character"},
|
||||
])
|
||||
## Troubleshooting
|
||||
|
||||
# Click submit button
|
||||
click(uid="search-button")
|
||||
|
||||
# Wait for results
|
||||
wait_for(text="Results", timeout=5000)
|
||||
|
||||
# Verify results via snapshot
|
||||
snapshot = take_snapshot()
|
||||
```
|
||||
|
||||
### Pattern: Modal Dialog Interaction
|
||||
|
||||
```python
|
||||
# Open modal (e.g., add LoRA)
|
||||
click(uid="add-lora-button")
|
||||
|
||||
# Wait for modal to appear
|
||||
wait_for(text="Add LoRA", timeout=3000)
|
||||
|
||||
# Fill modal form
|
||||
fill_form(elements=[
|
||||
{"uid": "lora-name", "value": "Test LoRA"},
|
||||
{"uid": "lora-path", "value": "/path/to/lora.safetensors"},
|
||||
])
|
||||
|
||||
# Submit
|
||||
click(uid="modal-submit-button")
|
||||
|
||||
# Wait for success message or close
|
||||
wait_for(text="Success", timeout=5000)
|
||||
```
|
||||
|
||||
## Fixture + Fresh-State Guidance
|
||||
|
||||
For rematch/repair E2E runs, seed the **sandboxed** `recipes_path` with hand-written fixture recipes. Rules (validated by the task-8 E2E):
|
||||
|
||||
1. **Filename constraint**: each file MUST be named `f"{id}.recipe.json"` **and** the in-JSON `id` field MUST equal the filename. Discovery accepts any `*.recipe.json`, but persistence resolves the path via `get_recipe_json_path` and `_save_recipe_persistently` returns `False` on a mismatch → the fixture would be counted as an error.
|
||||
- `recipe-a.recipe.json` → in-JSON `"id": "recipe-a"`
|
||||
2. **File format**: mirror an existing recipe JSON — top-level `id`, `file_path`, `title`, `loras`, `fingerprint`, `gen_params`; lora entries per the persistence conventions (`hash`, `file_name`, `modelVersionId`, `isDeleted`, ...).
|
||||
3. **Companion image**: each recipe needs an image (e.g. a `.webp` generated with PIL) referenced by `file_path`, used for EXIF verification (`ExifUtils.append_recipe_metadata` writes a `"Recipe metadata: ..."` marker; a freshly generated `.webp` with no marker is the clean "untouched" control).
|
||||
4. **autov3 three-state contract**: for L3 (autov3-only, renamed-file) fixtures the local model's `.metadata.json` sidecar MUST have the `autov3` key **ABSENT** (the "unchecked" state), NOT `""` — `""` is the TERMINAL "checked but unavailable" state that L3 deliberately skips. The scanner computes + persists `autov3` from the file header during the normal library scan (`model_scanner.py` `_process_model_file`), so the live L3 match resolves through the local autov3/hash cache; the computed-autov3 branch for unchecked items is covered by the unit suite.
|
||||
5. **Fixture design for a rematch run** (mirrors the task-8 E2E):
|
||||
- `recipe-a`: lora entry `isDeleted=True`, `hash` = 12-char autov3 computed from the local model (`calculate_autov3`, `py/utils/file_utils.py`), whose local model file was RENAMED after the recipe was written so `file_name` differs (proves L3 match without filename).
|
||||
- `recipe-b`: parser-convention checkpoint entry (uses `id`, no `modelVersionId`) matching a local checkpoint via L2 — the local checkpoint's `.metadata.json` MUST carry civitai version data with that `id` so `version_index` contains it (L2 cannot match otherwise).
|
||||
- `recipe-c`: healthy recipe (no deleted entries) → must remain untouched.
|
||||
|
||||
### Fresh state between entry-point runs
|
||||
|
||||
Each entry point (global / per-recipe / selection-bulk) must start from the same deleted state. Between runs:
|
||||
|
||||
```bash
|
||||
# 1. Reset fixtures to the before-state snapshot (copy back from recipes-before/)
|
||||
cp /tmp/opencode/<plan>-e2e/recipes-before/*.recipe.json /tmp/opencode/<plan>-e2e/recipes/
|
||||
# 2. Clear the recipe/FTS caches so the stale in-memory/library state is gone.
|
||||
# With --settings-path these live under the sandbox settings dir, NOT <repo-root>/cache.
|
||||
rm -f /tmp/opencode/<plan>-e2e/settings/cache/recipe/*.sqlite
|
||||
rm -rf /tmp/opencode/<plan>-e2e/settings/cache/fts/*
|
||||
# 3. Restart the server (fresh process, fresh scan)
|
||||
python .agents/skills/lora-manager-e2e/scripts/start_server.py \
|
||||
--port {PORT} --settings-path /tmp/opencode/<plan>-e2e/settings \
|
||||
--restart --wait --timeout 30 --detach
|
||||
# 4. Re-verify server listening + reload the browser page
|
||||
```
|
||||
|
||||
## Server Lifecycle
|
||||
|
||||
- **Detached launch is mandatory**: the standalone server dies with the shell unless launched via `setsid` (or the helper script's `--detach`). Use `setsid nohup python standalone.py --port {PORT} --settings-path <sandbox>/settings --host 127.0.0.1 ... < /dev/null &`.
|
||||
- **Always pass `--settings-path`** pointing at the sandbox settings dir — this is what keeps the run fully sandboxed (see [SANDBOX](#sandbox-mandatory)).
|
||||
- **Verify with `ss -tlnp`** after every (re)start; do not proceed on a blind "server starting" message.
|
||||
- **Never kill pre-existing processes** — only kill the E2E server PID you started (`start_server.py --restart` kills only PIDs it manages via its pidfile). The live ComfyUI or a stale QA Chrome must never be killed as part of cleanup unless explicitly identified as such (see Chrome troubleshooting).
|
||||
- **Record your PID for cleanup**: note the PID printed/pidfile, and stop exactly that PID at the end (`kill <PID>`, then confirm with `ss -tlnp` that `{PORT}` is released).
|
||||
|
||||
## Chrome DevTools MCP Troubleshooting
|
||||
|
||||
### Stale profile lock ("browser is already running" / `list_pages` fails)
|
||||
|
||||
A Chrome profile can be held by a stale Chrome from a prior MCP session, which makes `list_pages` fail with "browser is already running":
|
||||
|
||||
1. Identify the stale Chrome — it owns the profile dir in `--user-data-dir` (e.g. `~/.config/chrome-dev-profile`). Find its process:
|
||||
```bash
|
||||
ps -ef | grep -i '[c]hrome.*user-data-dir'
|
||||
```
|
||||
2. Confirm it is a QA Chrome from a completed task (its parent is an old MCP/browser process, it is NOT the live ComfyUI server, and it is NOT your current MCP instance).
|
||||
3. Kill ONLY that stale Chrome:
|
||||
```bash
|
||||
kill <stale-chrome-pid>
|
||||
```
|
||||
Never kill the live server or unrelated processes.
|
||||
4. Retry `list_pages`. The current MCP will spawn a fresh browser.
|
||||
|
||||
### Screenshot-write restrictions
|
||||
|
||||
The chrome-devtools MCP may refuse to write into paths outside its configured workspace roots (e.g. the worktree `.omo/evidence/...` canonicalizing to an unmapped path). Workaround:
|
||||
|
||||
```bash
|
||||
# 1. Save the screenshot to /tmp via the MCP
|
||||
# take_screenshot(filePath="/tmp/<plan>-e2e/recipe-b-after.png", format="png")
|
||||
# 2. Copy it into the evidence dir from the shell
|
||||
mkdir -p <repo-root>/.omo/evidence/screenshots
|
||||
cp /tmp/<plan>-e2e/recipe-b-after.png <repo-root>/.omo/evidence/screenshots/
|
||||
```
|
||||
|
||||
## Cancellation Testing (KNOWN GAP)
|
||||
|
||||
Testing the rematch-cancel path E2E requires a run long enough to cancel mid-flight. A tiny 3-recipe fixture set completes in **seconds** — too fast to reliably cancel. The cancel path is currently **unit-covered only** (`rematch_all_recipes` cancellation tests); do not block an E2E run on cancel-path verification. If you must attempt it, you would need an artificially large/deferred fixture set to create a cancellable window — treat this as a research task, not part of the standard E2E.
|
||||
|
||||
## Available Scripts
|
||||
|
||||
### scripts/start_server.py
|
||||
|
||||
Starts or restarts the LoRa Manager standalone server for E2E testing.
|
||||
|
||||
```bash
|
||||
python scripts/start_server.py [--port PORT] [--settings-path DIR] [--restart] [--wait] [--timeout SECONDS] [--detach]
|
||||
```
|
||||
|
||||
Options:
|
||||
- `--port`: Server port (default: 8188). The script exits early with a clear message if the port is already in use by an unrelated process.
|
||||
- `--settings-path`: Explicit sandbox settings directory passed through to `standalone.py` (equivalent to `LORA_MANAGER_SETTINGS_DIR`). Creates the directory if needed and refuses to start if the path exists as a file. **Use this for every sandboxed E2E run.**
|
||||
- `--restart`: Kill the E2E server this script previously managed (tracked via `/tmp/lora-manager-e2e-server-{PORT}.pid`) before starting. If unrelated processes still hold the port after that, the script reports them and aborts instead of killing them.
|
||||
- `--wait`: Wait for the server to be ready before exiting.
|
||||
- `--timeout`: Readiness wait timeout in seconds (default: 30).
|
||||
- `--detach`: Launch the server fully detached (`setsid`-style, survives shell death — REQUIRED for E2E). Default off: a normal background process that dies with the shell.
|
||||
|
||||
### scripts/wait_for_server.py
|
||||
|
||||
Polls the server until ready or timeout.
|
||||
|
||||
```bash
|
||||
python scripts/wait_for_server.py [--port PORT] [--timeout SECONDS]
|
||||
```
|
||||
|
||||
## Test Scenarios Reference
|
||||
|
||||
See [references/test-scenarios.md](references/test-scenarios.md) for detailed test scenarios including:
|
||||
- LoRA list display and filtering
|
||||
- Model metadata editing
|
||||
- Recipe creation and management
|
||||
- Settings configuration
|
||||
- Import/export functionality
|
||||
|
||||
## Network Request Verification
|
||||
|
||||
Use `list_network_requests` and `get_network_request` to verify API calls:
|
||||
|
||||
```python
|
||||
# List recent XHR/fetch requests
|
||||
requests = list_network_requests(resourceTypes=["xhr", "fetch"])
|
||||
|
||||
# Get details of specific request
|
||||
details = get_network_request(reqid=123)
|
||||
```
|
||||
|
||||
## Console Message Monitoring
|
||||
|
||||
```python
|
||||
# Check for errors or warnings
|
||||
messages = list_console_messages(types=["error", "warn"])
|
||||
```
|
||||
|
||||
## Performance Testing
|
||||
|
||||
```python
|
||||
# Start performance trace
|
||||
performance_start_trace(reload=True, autoStop=False)
|
||||
|
||||
# Perform actions...
|
||||
|
||||
# Stop and analyze
|
||||
results = performance_stop_trace()
|
||||
```
|
||||
- **"browser is already running" / `list_pages` fails**: a stale Chrome holds the
|
||||
profile dir. Find it (`ps -ef | grep -i '[c]hrome.*user-data-dir'`), confirm it is a
|
||||
leftover QA Chrome (not the live ComfyUI, not your current MCP browser), kill only
|
||||
that PID, then retry `list_pages`.
|
||||
- **MCP refuses to write screenshots into the worktree**: save to `/tmp` via
|
||||
`take_screenshot(filePath="/tmp/...")` and copy into the evidence dir from the shell.
|
||||
|
||||
## Cleanup
|
||||
|
||||
Always ensure proper cleanup after tests:
|
||||
1. Stop the standalone server: `kill <recorded-pid>` (only the PID you started), then confirm `ss -tlnp | grep ':{PORT}'` is empty.
|
||||
1. Stop the standalone server: `kill <recorded-pid>` (only the PID you started), then
|
||||
confirm `ss -tlnp | grep ':{PORT}'` is empty.
|
||||
2. Close browser pages (keep at least one open).
|
||||
3. Remove the sandbox: `rm -rf /tmp/opencode/<plan>-e2e`. Verify `<repo-root>` has NOT gained a `settings.json` or `cache/` (with `--settings-path` they never appear there).
|
||||
4. Re-run the real-data protection check from the SANDBOX section and record the result in your evidence.
|
||||
3. `rm -rf <sandbox>`; verify `<repo-root>` gained no `settings.json` or `cache/`.
|
||||
4. Re-run the real-data protection check from the SANDBOX section and record the result.
|
||||
|
||||
## References & Scripts
|
||||
|
||||
- [references/mcp-cheatsheet.md](references/mcp-cheatsheet.md) — Chrome DevTools MCP
|
||||
command patterns (navigation, waiting, snapshots, forms, network, console, performance).
|
||||
- [references/test-scenarios.md](references/test-scenarios.md) — detailed test scenarios
|
||||
(list display, metadata editing, recipes, settings, import/export).
|
||||
- [references/recipe-rematch-fixtures.md](references/recipe-rematch-fixtures.md) —
|
||||
fixture format, fresh-state reset and known gaps for recipe rematch/repair E2E runs.
|
||||
- `scripts/start_server.py` — start/restart the standalone server
|
||||
(`--port --settings-path --restart --wait --timeout --detach`); refuses to touch
|
||||
unrelated processes on the port.
|
||||
- `scripts/wait_for_server.py` — poll readiness (`--port --timeout`).
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
# Recipe Rematch/Repair E2E — Fixtures, Fresh State, Known Gaps
|
||||
|
||||
Specialized guidance for recipe rematch/repair E2E runs, extracted from the SKILL.md
|
||||
main flow. Read the SKILL.md SANDBOX section first — everything here assumes a
|
||||
sandboxed run.
|
||||
|
||||
## Fixture Rules (validated by the task-8 E2E)
|
||||
|
||||
Seed the **sandboxed** `recipes_path` with hand-written fixture recipes:
|
||||
|
||||
1. **Filename constraint**: each file MUST be named `f"{id}.recipe.json"` **and** the
|
||||
in-JSON `id` field MUST equal the filename. Discovery accepts any `*.recipe.json`,
|
||||
but persistence resolves the path via `get_recipe_json_path` and
|
||||
`_save_recipe_persistently` returns `False` on a mismatch → the fixture would be
|
||||
counted as an error.
|
||||
- `recipe-a.recipe.json` → in-JSON `"id": "recipe-a"`
|
||||
2. **File format**: mirror an existing recipe JSON — top-level `id`, `file_path`,
|
||||
`title`, `loras`, `fingerprint`, `gen_params`; lora entries per the persistence
|
||||
conventions (`hash`, `file_name`, `modelVersionId`, `isDeleted`, ...).
|
||||
3. **Companion image**: each recipe needs an image (e.g. a `.webp` generated with PIL)
|
||||
referenced by `file_path`, used for EXIF verification
|
||||
(`ExifUtils.append_recipe_metadata` writes a `"Recipe metadata: ..."` marker; a
|
||||
freshly generated `.webp` with no marker is the clean "untouched" control).
|
||||
4. **autov3 three-state contract**: for L3 (autov3-only, renamed-file) fixtures the
|
||||
local model's `.metadata.json` sidecar MUST have the `autov3` key **ABSENT** (the
|
||||
"unchecked" state), NOT `""` — `""` is the TERMINAL "checked but unavailable" state
|
||||
that L3 deliberately skips. The scanner computes + persists `autov3` from the file
|
||||
header during the normal library scan (`model_scanner.py` `_process_model_file`), so
|
||||
the live L3 match resolves through the local autov3/hash cache; the
|
||||
computed-autov3 branch for unchecked items is covered by the unit suite.
|
||||
5. **Fixture design for a rematch run** (mirrors the task-8 E2E):
|
||||
- `recipe-a`: lora entry `isDeleted=True`, `hash` = 12-char autov3 computed from the
|
||||
local model (`calculate_autov3`, `py/utils/file_utils.py`), whose local model file
|
||||
was RENAMED after the recipe was written so `file_name` differs (proves L3 match
|
||||
without filename).
|
||||
- `recipe-b`: parser-convention checkpoint entry (uses `id`, no `modelVersionId`)
|
||||
matching a local checkpoint via L2 — the local checkpoint's `.metadata.json` MUST
|
||||
carry civitai version data with that `id` so `version_index` contains it (L2
|
||||
cannot match otherwise).
|
||||
- `recipe-c`: healthy recipe (no deleted entries) → must remain untouched.
|
||||
|
||||
The scanner computes and persists model hashes during the library scan, so the sandbox
|
||||
model dirs just need the model files + `.metadata.json` sidecars. With
|
||||
`--settings-path`, all derived data lands under the sandbox settings dir (`cache/`,
|
||||
`backups/`, `logs/`, `stats/`, `wildcards/`), and NO `cache/` appears in the repo root.
|
||||
|
||||
## Fresh State Between Entry-Point Runs
|
||||
|
||||
Each entry point (global / per-recipe / selection-bulk) must start from the same
|
||||
deleted state. Between runs (keep a pristine copy in `<sandbox>/recipes-before/`):
|
||||
|
||||
```bash
|
||||
# 1. Reset fixtures to the before-state snapshot
|
||||
cp <sandbox>/recipes-before/*.recipe.json <sandbox>/recipes/
|
||||
# 2. Clear the recipe/FTS caches (with --settings-path these live under the sandbox
|
||||
# settings dir, NOT <repo-root>/cache)
|
||||
rm -f <sandbox>/settings/cache/recipe/*.sqlite
|
||||
rm -rf <sandbox>/settings/cache/fts/*
|
||||
# 3. Restart the server (fresh process, fresh scan)
|
||||
python .agents/skills/lora-manager-e2e/scripts/start_server.py \
|
||||
--port {PORT} --settings-path <sandbox>/settings --restart --wait --timeout 30 --detach
|
||||
# 4. Re-verify the server is listening + reload the browser page
|
||||
```
|
||||
|
||||
## Cancellation Testing (KNOWN GAP)
|
||||
|
||||
Testing the rematch-cancel path E2E requires a run long enough to cancel mid-flight. A
|
||||
tiny 3-recipe fixture set completes in **seconds** — too fast to reliably cancel. The
|
||||
cancel path is currently **unit-covered only** (`rematch_all_recipes` cancellation
|
||||
tests); do not block an E2E run on cancel-path verification. If you must attempt it,
|
||||
you would need an artificially large/deferred fixture set to create a cancellable
|
||||
window — treat this as a research task, not part of the standard E2E.
|
||||
@@ -2,6 +2,10 @@
|
||||
|
||||
This file provides guidance for agentic coding assistants working in this repository.
|
||||
|
||||
## Overview
|
||||
|
||||
ComfyUI LoRA Manager is a comprehensive LoRA management system for ComfyUI that combines a Python backend with browser-based widgets. It provides model organization, downloading from CivitAI/CivArchive, recipe management, and one-click workflow integration.
|
||||
|
||||
## Development Commands
|
||||
|
||||
### Backend Development
|
||||
@@ -28,16 +32,21 @@ COVERAGE_FILE=coverage/backend/.coverage pytest \
|
||||
--cov=py --cov=standalone \
|
||||
--cov-report=term-missing \
|
||||
--cov-report=html:coverage/backend/html \
|
||||
--cov-report=xml:coverage/backend/coverage.xml
|
||||
--cov-report=xml:coverage/backend/coverage.xml \
|
||||
--cov-report=json:coverage/backend/coverage.json
|
||||
```
|
||||
|
||||
### Frontend Development (LoRA Manager Web UI)
|
||||
|
||||
```bash
|
||||
# Install dependencies (root and Vue widgets)
|
||||
npm install
|
||||
cd vue-widgets && npm install && cd ..
|
||||
|
||||
npm test # Run all tests (JS + Vue)
|
||||
npm run test:js # Run JS tests only
|
||||
npm run test:watch # Watch mode
|
||||
npm run test:vue # Run Vue widget tests only
|
||||
npm run test:watch # Watch mode (JS tests only)
|
||||
npm run test:coverage # Generate coverage report
|
||||
```
|
||||
|
||||
@@ -54,88 +63,159 @@ npm run test:watch # Watch mode
|
||||
npm run test:coverage # Generate coverage report
|
||||
```
|
||||
|
||||
## Python Code Style
|
||||
### Localization
|
||||
|
||||
### Imports & Formatting
|
||||
```bash
|
||||
# Sync translation keys after UI string updates
|
||||
python scripts/sync_translation_keys.py
|
||||
```
|
||||
|
||||
Locale files are in `locales/` (en, zh-CN, zh-TW, ja, ko, fr, de, es, ru, he).
|
||||
|
||||
## Code Style
|
||||
|
||||
### Python
|
||||
|
||||
#### Imports & Formatting
|
||||
|
||||
- Use `from __future__ import annotations` for forward references
|
||||
- Group imports: standard library, third-party, local (blank line separated)
|
||||
- Use `TYPE_CHECKING` guard for type-checking-only imports
|
||||
- Absolute imports within `py/`: `from ..services import X`
|
||||
- PEP 8 with 4-space indentation, type hints required
|
||||
|
||||
### Naming Conventions
|
||||
#### Naming Conventions
|
||||
|
||||
- Files: `snake_case.py`, Classes: `PascalCase`, Functions/vars: `snake_case`
|
||||
- Constants: `UPPER_SNAKE_CASE`, Private: `_protected`, `__mangled`
|
||||
|
||||
### Error Handling & Async
|
||||
#### Error Handling & Async
|
||||
|
||||
- Use `logging.getLogger(__name__)`, define custom exceptions in `py/services/errors.py`
|
||||
- `async def` for I/O, `@pytest.mark.asyncio` for async tests
|
||||
- Singleton with `asyncio.Lock`: see `ModelScanner.get_instance()`
|
||||
- Return `aiohttp.web.json_response` or `web.Response`
|
||||
|
||||
### Testing
|
||||
### JavaScript/TypeScript
|
||||
|
||||
- `pytest` with `--import-mode=importlib`
|
||||
- Fixtures in `tests/conftest.py`, use `tmp_path_factory` for isolation
|
||||
- Mark tests needing real paths: `@pytest.mark.no_settings_dir_isolation`
|
||||
- Mock ComfyUI dependencies via conftest patterns
|
||||
|
||||
## JavaScript/TypeScript Code Style
|
||||
|
||||
### Imports & Modules
|
||||
#### Imports & Modules
|
||||
|
||||
- ES modules: `import { app } from "../../scripts/app.js"` for ComfyUI
|
||||
- Vue: `import { ref, computed } from 'vue'`, type imports: `import type { Foo }`
|
||||
- Export named functions: `export function foo() {}`
|
||||
|
||||
### Naming & Formatting
|
||||
#### Naming & Formatting
|
||||
|
||||
- camelCase for functions/vars/props, PascalCase for classes
|
||||
- Constants: `UPPER_SNAKE_CASE`, Files: `snake_case.js` or `kebab-case.js`
|
||||
- 2-space indentation preferred (follow existing file conventions)
|
||||
- Vue Single File Components: `<script setup lang="ts">` preferred
|
||||
|
||||
### Widget Development
|
||||
#### Widget Development
|
||||
|
||||
- Prefer vanilla JS for `web/comfyui/` widgets; avoid framework dependencies (except the Vue widgets in `vue-widgets/`)
|
||||
- ComfyUI: `app.registerExtension()`, `node.addDOMWidget(name, type, element, options)`
|
||||
- Event handlers via `addEventListener` or widget callbacks
|
||||
- Shared utilities: `web/comfyui/utils.js`
|
||||
- Dual-mode rendering patterns (canvas vs Vue): see `docs/comfyui-dual-mode-widgets.md`
|
||||
|
||||
### Vue Composables Pattern
|
||||
#### Vue Composables Pattern
|
||||
|
||||
- Use composition API: `useXxxState(widget)`, return reactive refs and methods
|
||||
- Guard restoration loops with flag: `let isRestoring = false`
|
||||
- Build config from state: `const buildConfig = (): Config => { ... }`
|
||||
|
||||
## Architecture Patterns
|
||||
## Architecture
|
||||
|
||||
### Dual Mode Operation
|
||||
|
||||
The system runs in two modes:
|
||||
- **ComfyUI plugin mode**: Integrates with ComfyUI's PromptServer, uses `folder_paths` for model discovery
|
||||
- **Standalone mode**: `standalone.py` mocks ComfyUI dependencies, reads paths from `settings.json`
|
||||
- Detection: `os.environ.get("LORA_MANAGER_STANDALONE", "0") == "1"`
|
||||
|
||||
### Backend Entry Points
|
||||
|
||||
- `__init__.py` — ComfyUI plugin entry: registers nodes via `NODE_CLASS_MAPPINGS`, sets `WEB_DIRECTORY`, calls `LoraManager.add_routes()`
|
||||
- `standalone.py` — Standalone server: mocks `folder_paths` and node modules, starts aiohttp server
|
||||
- `py/lora_manager.py` — Main `LoraManager` class that registers all HTTP routes
|
||||
|
||||
### Service Layer
|
||||
|
||||
- `ServiceRegistry` singleton for DI, services use `get_instance()` classmethod
|
||||
- `BaseModelService` abstract base → `LoraService`, `CheckpointService`, `EmbeddingService`
|
||||
- `ModelScanner` base → `LoraScanner`, `CheckpointScanner`, `EmbeddingScanner` for file discovery with hash-based deduplication
|
||||
- `PersistentModelCache` (SQLite) for metadata persistence
|
||||
- `MetadataSyncService` — background sync from CivitAI/CivArchive APIs
|
||||
- `SettingsManager` — settings with schema migration support
|
||||
- `WebSocketManager` — real-time progress broadcasting
|
||||
- `ModelServiceFactory` — creates the right service for each model type
|
||||
- Use cases in `py/services/use_cases/` orchestrate complex business logic (auto-organize, bulk refresh, downloads)
|
||||
- Separate scanners (discovery) from services (business logic)
|
||||
- Handlers in `py/routes/handlers/` are pure functions with deps as params
|
||||
|
||||
### Model Types & Routes
|
||||
|
||||
- `BaseModelService` base for LoRA, Checkpoint, Embedding
|
||||
- `ModelScanner` for file discovery, hash deduplication
|
||||
- `PersistentModelCache` (SQLite) for persistence
|
||||
- Route registrars: `ModelRouteRegistrar`, endpoints: `/loras/*`, `/checkpoints/*`, `/embeddings/*`
|
||||
- WebSocket via `WebSocketManager` for real-time updates
|
||||
- API endpoints follow `/loras/*`, `/checkpoints/*`, `/embeddings/*` patterns
|
||||
- Route registrars organize endpoints by domain: `ModelRouteRegistrar`, `RecipeRouteRegistrar`, etc.
|
||||
- Request handlers in `py/routes/handlers/` implement route logic
|
||||
- All routes use aiohttp, return `web.json_response` or `web.Response`
|
||||
|
||||
### Recipe System
|
||||
|
||||
- Base: `py/recipes/base.py`, Enrichment: `RecipeEnrichmentService`
|
||||
- Parsers: `py/recipes/parsers/`
|
||||
- Base: `py/recipes/base.py`, Enrichment: `RecipeEnrichmentService` in `py/recipes/enrichment.py`
|
||||
- Parsers: `py/recipes/parsers/` for PNG metadata, JSON, and workflow formats
|
||||
|
||||
### Custom Nodes
|
||||
|
||||
- Location: `py/nodes/`, all nodes registered in `__init__.py`
|
||||
- Each node class has a `NAME` class attribute used as key in `NODE_CLASS_MAPPINGS`
|
||||
- Standard ComfyUI node pattern: `INPUT_TYPES()` classmethod, `RETURN_TYPES`, `FUNCTION`
|
||||
|
||||
### Configuration
|
||||
|
||||
- `py/config.py` manages folder paths for models and handles symlink mappings
|
||||
- Auto-saves paths to `settings.json` in ComfyUI mode
|
||||
|
||||
### Frontend UI Architecture
|
||||
|
||||
#### 1. LoRA Manager Web UI
|
||||
- Location: `./static/` (JS/CSS) and `./templates/` (HTML)
|
||||
- Tech: Vanilla JS + CSS, served by the hosting server (ComfyUI app in plugin mode, `standalone.py` in standalone mode)
|
||||
- Tests: `tests/frontend/**/*.test.js` (vitest + jsdom)
|
||||
|
||||
#### 2. ComfyUI Custom Node Widgets
|
||||
- Location: `./web/comfyui/` (Vanilla JS) + `./vue-widgets/` (Vue)
|
||||
- Primary styles: `./web/comfyui/lm_styles.css` (NOT `./static/css/`)
|
||||
- Vue widgets: Vue 3 + TypeScript + PrimeVue + vue-i18n, e.g. `LoraPoolWidget`, `LoraRandomizerWidget`, `LoraCyclerWidget`, `AutocompleteTextWidget`
|
||||
- Vue builds to `./web/comfyui/vue-widgets/`; auto-built on ComfyUI startup via `py/vue_widget_builder.py`, typecheck via `vue-tsc`
|
||||
- Widget registration: `app.registerExtension()` and `getCustomWidgets` hooks; `node.addDOMWidget(...)` embeds HTML in LiteGraph nodes
|
||||
- See `docs/dom_widget_dev_guide.md` for the DOMWidget development guide
|
||||
|
||||
## Testing
|
||||
|
||||
### Backend (pytest)
|
||||
|
||||
- Config in `pytest.ini`: `--import-mode=importlib`, testpaths=`tests`
|
||||
- Fixtures in `tests/conftest.py` mock ComfyUI dependencies; use `tmp_path_factory` for isolation
|
||||
- Markers: `@pytest.mark.asyncio`, `@pytest.mark.no_settings_dir_isolation` (tests needing real settings paths)
|
||||
|
||||
### Frontend (vitest)
|
||||
|
||||
- Vanilla JS tests: `tests/frontend/**/*.test.js` with jsdom; setup in `tests/frontend/setup.js`
|
||||
- Vue widget tests: `vue-widgets/tests/**/*.test.ts` with jsdom + `@vue/test-utils`
|
||||
|
||||
## Key Integration Points
|
||||
|
||||
- **Settings:** Stored in the user config directory (via `platformdirs`) or portable mode (`"use_portable_settings": true`)
|
||||
- **CivitAI/CivArchive:** API clients for metadata sync and model downloads; CivitAI API key stored in settings
|
||||
- **Symlinks:** Config scans symlinks to map virtual→physical paths; fingerprinting prevents redundant rescans
|
||||
- **WebSocket:** Broadcasts real-time progress for downloads, scans, and metadata sync
|
||||
- **Model scanning flow:** Walk folders → compute hashes → deduplicate → extract safetensors metadata → cache in SQLite → background CivitAI sync → WebSocket broadcast
|
||||
|
||||
## Important Notes
|
||||
|
||||
- ALWAYS use English for comments (per copilot-instructions.md)
|
||||
- Dual mode: ComfyUI plugin (folder_paths) vs standalone (settings.json)
|
||||
- Detection: `os.environ.get("LORA_MANAGER_STANDALONE", "0") == "1"`
|
||||
- Run `python scripts/sync_translation_keys.py` after adding UI strings to `locales/en.json`
|
||||
- Symlinks require normalized paths.
|
||||
**Business paths vs real paths**: All stored paths and operation routing use the
|
||||
@@ -150,16 +230,4 @@ npm run test:coverage # Generate coverage report
|
||||
- Follow the style of recent repository commits when writing commit messages
|
||||
- Prefer the repo's existing `feat(...)`, `fix(...)`, `chore:` style where applicable
|
||||
- If the user has provided a GitHub issue link or issue ID for the task, mention that issue in the commit message, for example `(#871)`
|
||||
- When unrelated local changes exist, stage and commit only the files relevant to the requested task
|
||||
|
||||
## Frontend UI Architecture
|
||||
|
||||
### 1. LoRA Manager Web UI
|
||||
- Location: `./static/` and `./templates/`
|
||||
- Tech: Vanilla JS + CSS, served by the hosting server (ComfyUI app in plugin mode, `standalone.py` in standalone mode)
|
||||
- Tests via npm in root directory
|
||||
|
||||
### 2. ComfyUI Custom Node Widgets
|
||||
- Location: `./web/comfyui/` (Vanilla JS) + `./vue-widgets/` (Vue)
|
||||
- Primary styles: `./web/comfyui/lm_styles.css` (NOT `./static/css/`)
|
||||
- Vue builds to `./web/comfyui/vue-widgets/`, typecheck via `vue-tsc`
|
||||
- When unrelated local changes exist, stage and commit only the files relevant to the requested task
|
||||
@@ -1,189 +0,0 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Overview
|
||||
|
||||
ComfyUI LoRA Manager is a comprehensive LoRA management system for ComfyUI that combines a Python backend with browser-based widgets. It provides model organization, downloading from CivitAI/CivArchive, recipe management, and one-click workflow integration.
|
||||
|
||||
## Development Commands
|
||||
|
||||
### Backend
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
pip install -r requirements-dev.txt
|
||||
|
||||
# Run standalone server (port 8188 by default)
|
||||
python standalone.py --port 8188
|
||||
|
||||
# Run all backend tests
|
||||
pytest
|
||||
|
||||
# Run specific test file or function
|
||||
pytest tests/test_recipes.py
|
||||
pytest tests/test_recipes.py::test_function_name
|
||||
|
||||
# Run backend tests with coverage
|
||||
COVERAGE_FILE=coverage/backend/.coverage pytest \
|
||||
--cov=py \
|
||||
--cov=standalone \
|
||||
--cov-report=term-missing \
|
||||
--cov-report=html:coverage/backend/html \
|
||||
--cov-report=xml:coverage/backend/coverage.xml \
|
||||
--cov-report=json:coverage/backend/coverage.json
|
||||
```
|
||||
|
||||
### Frontend
|
||||
|
||||
There are three test suites run by `npm test`: vanilla JS tests (vitest at root) and Vue widget tests (`vue-widgets/` vitest).
|
||||
|
||||
```bash
|
||||
npm install
|
||||
cd vue-widgets && npm install && cd ..
|
||||
|
||||
# Run all frontend tests (JS + Vue)
|
||||
npm test
|
||||
|
||||
# Run only vanilla JS tests
|
||||
npm run test:js
|
||||
|
||||
# Run only Vue widget tests
|
||||
npm run test:vue
|
||||
|
||||
# Watch mode (JS tests only)
|
||||
npm run test:watch
|
||||
|
||||
# Frontend coverage
|
||||
npm run test:coverage
|
||||
|
||||
# Build Vue widgets (output to web/comfyui/vue-widgets/)
|
||||
cd vue-widgets && npm run build
|
||||
|
||||
# Vue widget dev mode (watch + rebuild)
|
||||
cd vue-widgets && npm run dev
|
||||
|
||||
# Typecheck Vue widgets
|
||||
cd vue-widgets && npm run typecheck
|
||||
```
|
||||
|
||||
### Localization
|
||||
|
||||
```bash
|
||||
# Sync translation keys after UI string updates
|
||||
python scripts/sync_translation_keys.py
|
||||
```
|
||||
|
||||
Locale files are in `locales/` (en, zh-CN, zh-TW, ja, ko, fr, de, es, ru, he).
|
||||
|
||||
## Architecture
|
||||
|
||||
### Dual Mode Operation
|
||||
|
||||
The system runs in two modes:
|
||||
- **ComfyUI plugin mode**: Integrates with ComfyUI's PromptServer, uses `folder_paths` for model discovery
|
||||
- **Standalone mode**: `standalone.py` mocks ComfyUI dependencies, reads paths from `settings.json`
|
||||
- Detection: `os.environ.get("LORA_MANAGER_STANDALONE", "0") == "1"`
|
||||
|
||||
### Backend (Python)
|
||||
|
||||
**Entry points:**
|
||||
- `__init__.py` — ComfyUI plugin entry: registers nodes via `NODE_CLASS_MAPPINGS`, sets `WEB_DIRECTORY`, calls `LoraManager.add_routes()`
|
||||
- `standalone.py` — Standalone server: mocks `folder_paths` and node modules, starts aiohttp server
|
||||
- `py/lora_manager.py` — Main `LoraManager` class that registers all HTTP routes
|
||||
|
||||
**Service layer** (`py/services/`):
|
||||
- `ServiceRegistry` singleton for dependency injection; services follow `get_instance()` singleton pattern
|
||||
- `BaseModelService` abstract base → `LoraService`, `CheckpointService`, `EmbeddingService`
|
||||
- `ModelScanner` base → `LoraScanner`, `CheckpointScanner`, `EmbeddingScanner` for file discovery with hash-based deduplication
|
||||
- `PersistentModelCache` — SQLite-based metadata cache
|
||||
- `MetadataSyncService` — Background sync from CivitAI/CivArchive APIs
|
||||
- `SettingsManager` — Settings with schema migration support
|
||||
- `WebSocketManager` — Real-time progress broadcasting
|
||||
- `ModelServiceFactory` — Creates the right service for each model type
|
||||
- Use cases in `py/services/use_cases/` orchestrate complex business logic (auto-organize, bulk refresh, downloads)
|
||||
|
||||
**Routes** (`py/routes/`):
|
||||
- Route registrars organize endpoints by domain: `ModelRouteRegistrar`, `RecipeRouteRegistrar`, etc.
|
||||
- Request handlers in `py/routes/handlers/` implement route logic
|
||||
- API endpoints follow `/loras/*`, `/checkpoints/*`, `/embeddings/*` patterns
|
||||
- All routes use aiohttp, return `web.json_response` or `web.Response`
|
||||
|
||||
**Recipe system** (`py/recipes/`):
|
||||
- `base.py` — Recipe metadata structure
|
||||
- `enrichment.py` — Enriches recipes with model metadata
|
||||
- `parsers/` — Parsers for PNG metadata, JSON, and workflow formats
|
||||
|
||||
**Custom nodes** (`py/nodes/`):
|
||||
- Each node class has a `NAME` class attribute used as key in `NODE_CLASS_MAPPINGS`
|
||||
- Standard ComfyUI node pattern: `INPUT_TYPES()` classmethod, `RETURN_TYPES`, `FUNCTION`
|
||||
- All nodes registered in `__init__.py`
|
||||
|
||||
**Configuration** (`py/config.py`):
|
||||
- Manages folder paths for models, handles symlink mappings
|
||||
- Auto-saves paths to settings.json in ComfyUI mode
|
||||
|
||||
### Frontend — Two Distinct UI Systems
|
||||
|
||||
#### 1. Standalone Manager Web UI
|
||||
- **Location:** `static/` (JS/CSS) and `templates/` (HTML)
|
||||
- **Tech:** Vanilla JS + CSS, served by standalone server
|
||||
- **Structure:** `static/js/core.js` (shared), `loras.js`, `checkpoints.js`, `embeddings.js`, `recipes.js`, `statistics.js`
|
||||
- **Tests:** `tests/frontend/**/*.test.js` (vitest + jsdom)
|
||||
|
||||
#### 2. ComfyUI Custom Node Widgets
|
||||
- **Vanilla JS widgets:** `web/comfyui/*.js` — ES modules extending ComfyUI's LiteGraph UI
|
||||
- `loras_widget.js` / `loras_widget_events.js` — Main LoRA selection widget
|
||||
- `autocomplete.js` — Trigger word and embedding autocomplete
|
||||
- `preview_tooltip.js` — Model card preview tooltips
|
||||
- `top_menu_extension.js` — "Launch LoRA Manager" menu item
|
||||
- `utils.js` — Shared utilities and API helpers
|
||||
- Widget styling in `web/comfyui/lm_styles.css` (NOT `static/css/`)
|
||||
- **Vue widgets:** `vue-widgets/src/` → built to `web/comfyui/vue-widgets/`
|
||||
- Vue 3 + TypeScript + PrimeVue + vue-i18n
|
||||
- Vite build with CSS-injected-by-JS plugin
|
||||
- Components: `LoraPoolWidget`, `LoraRandomizerWidget`, `LoraCyclerWidget`, `AutocompleteTextWidget`
|
||||
- Auto-built on ComfyUI startup via `py/vue_widget_builder.py`
|
||||
- Tests: `vue-widgets/tests/**/*.test.ts` (vitest)
|
||||
|
||||
**Widget registration pattern:**
|
||||
- Widgets use `app.registerExtension()` and `getCustomWidgets` hooks
|
||||
- `node.addDOMWidget(name, type, element, options)` embeds HTML in LiteGraph nodes
|
||||
- See `docs/dom_widget_dev_guide.md` for DOMWidget development guide
|
||||
|
||||
## Code Style
|
||||
|
||||
**Python:**
|
||||
- PEP 8, 4-space indentation, English comments only
|
||||
- Use `from __future__ import annotations` for forward references
|
||||
- Use `TYPE_CHECKING` guard for type-checking-only imports
|
||||
- Loggers via `logging.getLogger(__name__)`
|
||||
- Custom exceptions in `py/services/errors.py`
|
||||
- Async patterns: `async def` for I/O, `@pytest.mark.asyncio` for async tests
|
||||
- Singleton pattern with class-level `asyncio.Lock` (see `ModelScanner.get_instance()`)
|
||||
|
||||
**JavaScript:**
|
||||
- ES modules, camelCase functions/variables, PascalCase classes
|
||||
- Widget files use `*_widget.js` suffix
|
||||
- Prefer vanilla JS for `web/comfyui/` widgets, avoid framework dependencies (except Vue widgets)
|
||||
|
||||
## Testing
|
||||
|
||||
**Backend (pytest):**
|
||||
- Config in `pytest.ini`: `--import-mode=importlib`, testpaths=`tests`
|
||||
- Fixtures in `tests/conftest.py` handle ComfyUI dependency mocking
|
||||
- Markers: `@pytest.mark.asyncio`, `@pytest.mark.no_settings_dir_isolation`
|
||||
- Uses `tmp_path_factory` for directory isolation
|
||||
|
||||
**Frontend (vitest):**
|
||||
- Vanilla JS tests: `tests/frontend/**/*.test.js` with jsdom
|
||||
- Vue widget tests: `vue-widgets/tests/**/*.test.ts` with jsdom + @vue/test-utils
|
||||
- Setup in `tests/frontend/setup.js`
|
||||
|
||||
## Key Integration Points
|
||||
|
||||
- **Settings:** Stored in user directory (via `platformdirs`) or portable mode (`"use_portable_settings": true`)
|
||||
- **CivitAI/CivArchive:** API clients for metadata sync and model downloads; CivitAI API key in settings
|
||||
- **Symlink handling:** Config scans symlinks to map virtual→physical paths; fingerprinting prevents redundant rescans
|
||||
- **WebSocket:** Broadcasts real-time progress for downloads, scans, and metadata sync
|
||||
- **Model scanning flow:** Walk folders → compute hashes → deduplicate → extract safetensors metadata → cache in SQLite → background CivitAI sync → WebSocket broadcast
|
||||
@@ -0,0 +1,337 @@
|
||||
# Plan: Global Rate-Limit Abidance for Recipe Ingest & Metadata Fetching
|
||||
|
||||
**Issue:** [#1085 — Large Recipe Ingest Appears to not abide by vendor rate limits, possibly a few other errors?](https://github.com/willmiao/ComfyUI-Lora-Manager/issues/1085)
|
||||
**Status:** v2 — reviewed; decisions recorded in §10. **Phase 1 implemented**
|
||||
(2026-08-27, commit `c2a2048c`): coordinator + downloader gate + Fix C
|
||||
failover semantics + helper double-wait fix + settings. **Phase 2
|
||||
implemented** (2026-08-27): batch-import rate-limit failures map to
|
||||
`SKIPPED` + `rate_limited` WebSocket flag + UI slowdown hint (toast + status
|
||||
text, i18n keys synced); `download_to_memory` / `get_response_headers` /
|
||||
`download_file` register 429 cooldowns. Changes vs v1: Fix C moved to
|
||||
Phase 1, helper double-wait resolved in Phase 1, gate/guard ordering
|
||||
specified.
|
||||
**Scope:** HTTP API traffic to CivitAI (`civitai.red`) and CivArchive (`civarchive.com`) from metadata fetching (bulk refresh, metadata sync, recipe analysis/enrichment, usage-control lookups). Large binary downloads (model files / preview images via `download_file`) are out of scope for *pacing* (they are already single-connection transfers) but their 429 responses should still be *registered*.
|
||||
|
||||
> Context: a first batch of fixes for this issue was already committed as
|
||||
> `ee233548` ("fix(recipes): enforce batch-import concurrency bound and harden
|
||||
> ingest errors (#1085)"): the batch-import concurrency controller now shares a
|
||||
> real semaphore (bounds 1–5 actually apply), the Comfy parser tolerates
|
||||
> list/`None` `ckpt_name`, CivArchive treats empty error payloads as failures,
|
||||
> and offline-cooldown short-circuits log at DEBUG. This plan covers the two
|
||||
> remaining orchestration-level fixes:
|
||||
> **Fix 2** — slow down globally when a vendor rate limit is hit (respect
|
||||
> `Retry-After`, queue instead of hammering); **Fix 3** — stop immediately
|
||||
> failing over to CivArchive when CivitAI is rate-limited.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem Statement
|
||||
|
||||
During a large recipe ingest (e.g. importing the example-images directory,
|
||||
which can be thousands of images), the manager fires one metadata request per
|
||||
checkpoint + per LoRA per image through the fallback provider chain
|
||||
(`civitai_api → civarchive_api → sqlite`). Consequences observed in #1085:
|
||||
|
||||
1. **CivitAI gets hammered** → 429s. The consumer then *immediately* tries
|
||||
CivArchive for the same lookup, so **CivArchive gets hammered too** before
|
||||
it was ever naturally needed (its only real job is recovering metadata for
|
||||
models deleted from CivitAI).
|
||||
2. Requests are retried per-call after `Retry-After`, but **each concurrent
|
||||
call sleeps independently** → thundering herd: thousands of coroutines wake
|
||||
at the same moment and re-flood the vendor.
|
||||
3. While CivArchive is in the `ConnectivityGuard` cooldown, every batch item
|
||||
short-circuits and is marked `FAILED` — the batch import's success/failure
|
||||
accounting is polluted by a transient vendor state (log spam was fixed in
|
||||
`ee233548`; the item-failure accounting is not).
|
||||
4. `ConnectivityGuard` (`py/services/connectivity_guard.py`) only treats
|
||||
transport-level unreachability as offline; **HTTP 429 is invisible to it**,
|
||||
so nothing ever intentionally paces request rate.
|
||||
|
||||
User expectation from the issue: *"once a vendor rate limit time out is hit,
|
||||
you should trigger a slow down with intentional reduction in request rate"*.
|
||||
|
||||
## 2. Current State (verified against code)
|
||||
|
||||
### 2.1 Where 429s are surfaced
|
||||
|
||||
- `Downloader.make_request` (`py/services/downloader.py:1120-1132`): HTTP 429 →
|
||||
returns `RateLimitError(message, retry_after=…)` parsed from `Retry-After`
|
||||
(missing header defaults to `None`).
|
||||
- `CivitaiClient._make_request` (`py/services/civitai_client.py:97-100`):
|
||||
converts `RateLimitError` to a raise immediately; no waiting. Transient
|
||||
5xx/connection errors are retried 3× with 1s/2s/4s backoff.
|
||||
- `CivArchiveClient._make_request` (`py/services/civarchive_client.py`):
|
||||
raises `RateLimitError` with `provider="civarchive_api"` when not set.
|
||||
- `_RateLimitRetryHelper` (`py/services/model_metadata_provider.py:45-102`):
|
||||
per-call retry loop — sleeps `retry_after` (capped at 1800 s; `≥120 s` ⇒ no
|
||||
retry), then re-raises. Because every concurrent call runs its own helper,
|
||||
they sleep in parallel and re-fire in parallel.
|
||||
- `FallbackMetadataProvider` (`py/services/model_metadata_provider.py:488-508,
|
||||
564-584` etc.): on a final `RateLimitError` from one provider it logs
|
||||
"skipping to next provider" and **continues to the next network provider** —
|
||||
this is the direct cause of the CivArchive flood.
|
||||
- `MetadataSyncService.fetch_and_update_model`
|
||||
(`py/services/metadata_sync_service.py:248-333`): manually iterates
|
||||
`provider_attempts`; on `RateLimitError` it `continue`s to the next provider
|
||||
(same failover problem), then reports `"Rate limited"` when nothing
|
||||
succeeded.
|
||||
- `Downloader.make_request` has a per-destination scope already available:
|
||||
`_guard_destination(url)` returns the hostname (`downloader.py:1194-1199`),
|
||||
used by `ConnectivityGuard`.
|
||||
|
||||
### 2.2 What pacing exists today
|
||||
|
||||
- `ConnectivityGuard`: per-destination cooldown (30 s base, ×2 per extra
|
||||
failure batch, 300 s cap) triggered only by transport errors
|
||||
(`connectivity_guard.py:168-197`).
|
||||
- `AdaptiveConcurrencyController` (batch import, fixed in `ee233548`): shared
|
||||
semaphore enforces 1–5 concurrent items; *duration*-based adjustment only —
|
||||
it never sees HTTP statuses, so it cannot distinguish "slow because rate
|
||||
limited" from "slow because big image".
|
||||
- No token bucket, no minimum inter-request interval, no shared
|
||||
`Retry-After` gate anywhere (`grep` for throttle/token-bucket/rate-limiter:
|
||||
0 hits).
|
||||
|
||||
## 3. Requirements & Constraints
|
||||
|
||||
R1. **Respect `Retry-After`.** After a 429, no further request to that
|
||||
destination may be sent before the vendor's retry window elapses.
|
||||
R2. **No thundering herd.** Concurrent waiters must share one wake-up (gate),
|
||||
not sleep independently.
|
||||
R3. **No double load.** A CivitAI 429 must not trigger a CivArchive request
|
||||
for the same lookup. CivArchive should only be consulted when CivitAI
|
||||
legitimately has no answer (404 / "not found"), or when CivitAI is
|
||||
unreachable long-term.
|
||||
R4. **No spurious item failures.** A rate-limited request must not turn a
|
||||
batch-import item into `FAILED`; it should wait (bounded) and retry, or at
|
||||
worst be `SKIPPED` with a clear "rate limited" reason (re-runnable import).
|
||||
R5. **Never hang forever.** All waiting is bounded by a configurable cap; on
|
||||
expiry the caller receives the `RateLimitError` and can decide.
|
||||
R6. **Keep legitimate failover.** Deleted-model recovery via CivArchive/sqlite
|
||||
must keep working (404 paths unchanged).
|
||||
R7. **Single choke point.** The pacing gate should live where every API call
|
||||
passes (the `Downloader`), so bulk refresh, metadata sync, recipe
|
||||
analysis, and usage-control lookups all benefit without per-feature work.
|
||||
|
||||
## 4. Approach Comparison
|
||||
|
||||
### A. Reactive gate — shared `Retry-After` deadman clock (recommended core)
|
||||
|
||||
A process-wide, per-destination coordinator records the *next-allowed-send*
|
||||
timestamp from each 429 (`now + max(retry_after, backoff)`). Every request
|
||||
through `Downloader.make_request` consults the gate *before sending* and *when
|
||||
a 429 arrives*; waiters block on a shared `asyncio.Event` that fires when the
|
||||
cooldown expires.
|
||||
|
||||
- Pros: single choke point (R7); herd-free (R2); honors server guidance (R1);
|
||||
no guessing at vendor limits; covers all providers automatically; reuses
|
||||
existing per-destination scoping.
|
||||
- Cons: still experiences 429s before slowing down (reactive); long
|
||||
`Retry-After` windows (CivArchive has been observed at ~1500 s) need a sane
|
||||
wait cap + skip/retry UX.
|
||||
|
||||
### B. Preemptive pacing — minimum inter-request interval (recommended companion)
|
||||
|
||||
Per-destination token bucket (simplest form: capacity 1 — at least `N` seconds
|
||||
between consecutive API requests; `N` configurable, default ~0.75 s ≈ 80
|
||||
r/min ceiling).
|
||||
|
||||
- Pros: prevents most 429s before they happen — exactly the "intentional
|
||||
reduction in request rate" the issue asks for; trivial to implement on top
|
||||
of A's coordinator.
|
||||
- Cons: adds latency to bulk operations (thousands of models × `N`); the *exact*
|
||||
vendor limits are unknown (CivitAI anonymous vs keyed vs `civitai.red`
|
||||
mirror differ), so the default must be conservative-but-not-crippling and
|
||||
settings-tunable.
|
||||
|
||||
### C. Fallback semantics change — stop network→network failover on 429 (must-do, low risk)
|
||||
|
||||
`FallbackMetadataProvider` (and `MetadataSyncService.fetch_and_update_model`'s
|
||||
manual loop) must treat a final `RateLimitError` as a **terminal, non-failover
|
||||
result** for network providers. Local-only providers (sqlite archive DB) may
|
||||
stay as a last resort (no vendor cost).
|
||||
|
||||
- Pros: directly removes the CivArchive flood; small, surgical change.
|
||||
- Cons: none significant; requires care to keep 404-failover intact (R6).
|
||||
|
||||
### Rejected / deferred
|
||||
|
||||
- **Per-feature retry queues** (batch import pauses & resumes whole batches):
|
||||
richer UX but much larger change (batch state machine, WebSocket states);
|
||||
unnecessary once A+B make requests wait at the choke point. Defer unless
|
||||
review finds the bounded-wait UX insufficient.
|
||||
- **Full token bucket with burst credit**: overkill; capacity-1 interval is
|
||||
enough given the shared semaphore already caps concurrency at 5.
|
||||
- **Retrying in `connectivity_guard`**: wrong layer — the guard is about
|
||||
transport reachability, not vendor quota.
|
||||
|
||||
## 5. Recommended Architecture
|
||||
|
||||
New singleton **`RateLimitCoordinator`** (`py/services/rate_limit_coordinator.py`,
|
||||
mirroring `ConnectivityGuard`'s singleton + per-destination patterns):
|
||||
|
||||
```
|
||||
state per destination (hostname):
|
||||
next_allowed_send: float (monotonic) # from 429 Retry-After + backoff
|
||||
consecutive_429: int # for backoff growth
|
||||
last_send_at: float # for min-interval pacing
|
||||
waiters: list[Future] | asyncio.Event # shared wake-up per cooldown cycle
|
||||
```
|
||||
|
||||
API:
|
||||
|
||||
- `async wait_for_slot(destination, request_started_within_window: bool)`
|
||||
— called by `Downloader.make_request` *before* sending (blocks until
|
||||
`min(now >= next_allowed_send)` and inter-request interval elapses) and
|
||||
re-armable after a 429.
|
||||
- `register_rate_limit(destination, retry_after: float | None)`
|
||||
— called on 429: `next_allowed_send = max(now + retry_after_or_backoff, current)`;
|
||||
`consecutive_429 += 1`; backoff = `retry_after` honored, else exponential
|
||||
`30 · 2^(n-1)` capped at 1800 s; creates/re-arms the shared wake-up event.
|
||||
- `register_success(destination)` — resets `consecutive_429` (called from the
|
||||
existing 200 path in `make_request`).
|
||||
- `remaining_seconds(destination)`, `in_cooldown(destination)` — for tests and
|
||||
diagnostics.
|
||||
|
||||
Enforcement points:
|
||||
|
||||
1. **`Downloader.make_request`** (`downloader.py:1102-1132`): ordering inside
|
||||
the method is **connectivity-guard fail-fast first** (offline short-circuit
|
||||
costs nothing to check), **then** `await coordinator.wait_for_slot(destination)`
|
||||
before `session.request`. On 429: `coordinator.register_rate_limit(...)`,
|
||||
then *wait for the gate and re-send* (loop, bounded by
|
||||
`rate_limit_max_wait_seconds`, default 300; `retry_after ≥ cap` ⇒ fail
|
||||
immediately). After the loop, return the `RateLimitError` to the caller
|
||||
(unchanged contract) **with `exc.gate_handled = True` set** so downstream
|
||||
retry helpers know the wait already happened. 200 path calls
|
||||
`register_success`.
|
||||
2. **`Downloader.download_to_memory` / `get_response_headers`** (phase 2):
|
||||
register 429s (so API calls queue); waiting only in `make_request`
|
||||
initially.
|
||||
3. **`FallbackMetadataProvider`** (`model_metadata_provider.py`): remove
|
||||
network→network failover on `RateLimitError` — re-raise; only sqlite stays
|
||||
as a local last resort (implementation: per-method `except RateLimitError`
|
||||
handler that marks the chain rate-limited and stops iterating).
|
||||
4. **`MetadataSyncService.fetch_and_update_model`**
|
||||
(`metadata_sync_service.py:248-333`): on `RateLimitError` from the default
|
||||
provider, stop appending further network providers (sqlite may remain);
|
||||
the existing `any_rate_limited` merge already produces `"Rate limited"`.
|
||||
5. **Batch import** (`batch_import_service.py`): no structural change needed —
|
||||
items now wait inside `make_request`; optionally (phase 2) map residual
|
||||
rate-limit failures (after the wait cap) to `SKIPPED` with
|
||||
`"rate limited (retry_after=…s); re-run the import later"` instead of
|
||||
`FAILED`, and surface a `rate_limited` flag in the WebSocket progress
|
||||
broadcast.
|
||||
6. **`_RateLimitRetryHelper` retries** (`model_metadata_provider.py`):
|
||||
**Phase 1** — when the raised `RateLimitError` carries `gate_handled = True`
|
||||
(set by the downloader after honoring the gate), the helper skips its own
|
||||
`retry_after` sleep and re-raises immediately, eliminating the double wait.
|
||||
The wiring stays so a `RateLimitError` still propagates cleanly; full
|
||||
demotion/removal can follow once the gate proves out.
|
||||
|
||||
Settings (`settings.json`, schema extension in `SettingsManager`):
|
||||
|
||||
| key | default | meaning |
|
||||
|---|---|---|
|
||||
| `rate_limit_gate_enabled` | `true` | master switch for the coordinator |
|
||||
| `rate_limit_max_wait_seconds` | `300` | how long `make_request` waits on a 429 gate before returning the error |
|
||||
| `rate_limit_min_interval_seconds` | `0.75` | minimum seconds between API requests per destination (pacing, R6-friendly conservative default) |
|
||||
|
||||
## 6. Changes by File
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `py/services/rate_limit_coordinator.py` (new) | coordinator singleton + per-destination state + tests seam |
|
||||
| `py/services/downloader.py` | gate pre-check + 429 register/wait/retry loop + `register_success`; log the 429 notice at INFO once per cooldown, then DEBUG |
|
||||
| `py/services/model_metadata_provider.py` | `FallbackMetadataProvider`: stop network failover on `RateLimitError`; helper skips its sleep when the error is marked `gate_handled` |
|
||||
| `py/services/metadata_sync_service.py` | `fetch_and_update_model`: same failover semantics; keep sqlite last resort |
|
||||
| `py/services/batch_import_service.py` | (phase 2) rate-limit failures → `SKIPPED` + `rate_limited` progress flag |
|
||||
| `py/services/settings_manager.py` | new settings keys + defaults |
|
||||
| `tests/services/test_rate_limit_coordinator.py` (new) | gate unit tests |
|
||||
| `tests/services/test_civitai_client.py` / `test_civarchive_client.py` | provider-level 429 behavior |
|
||||
| `tests/services/test_metadata_service.py` | failover-chain tests |
|
||||
| `tests/services/test_batch_import_service.py` | SKIPPED-on-rate-limit |
|
||||
|
||||
## 7. Impact, Risks, Open Questions
|
||||
|
||||
- **Behavior change**: with the gate in `make_request`, any request can block
|
||||
up to the wait cap — UI actions that call the API (e.g. a model-details
|
||||
fetch) may take longer during cooldowns. Mitigation: bounded cap + INFO log
|
||||
+ the existing async request handling already tolerates slow responses.
|
||||
**Decided (§10): interactive requests take the same bounded wait** — one
|
||||
behavior, no call-source plumbing; cooldowns are usually short.
|
||||
- **Gate waits occupy batch slots**: with the 1–5 batch semaphore, all slots
|
||||
can park on a gate simultaneously, freezing visible progress for up to one
|
||||
wait cap per wave. Bounded and acceptable; the phase-2 `SKIPPED` mapping +
|
||||
WebSocket `rate_limited` flag (both confirmed in scope, §10) make the stall
|
||||
visible and recoverable.
|
||||
- **Rate limit reality check**: CivitAI anonymous vs keyed limits, and whether
|
||||
`civitai.red` differs, is unverified. Default pacing `0.75 s/req` is a
|
||||
conservative guess (R6). Open question for maintainer: preferred default
|
||||
and whether an API-keyed ceiling should be higher.
|
||||
- **Long CivArchive windows**: `Retry-After ~1500 s` observed in code
|
||||
comments. **Decided (§10): keep the 300 s default cap** — such lookups
|
||||
fail/skip rather than park a request path for 25 minutes; batch import maps
|
||||
them to `SKIPPED` (phase 2) so the user can re-run later.
|
||||
- **Double waiting**: `_RateLimitRetryHelper` + gate could stack waits.
|
||||
**Resolved in Phase 1**: the downloader marks gate-honored errors with
|
||||
`gate_handled = True` and the helper skips its own sleep for those.
|
||||
- **Downloads**: `download_file` 429s return an error to download managers
|
||||
unchanged (already handled); only *registration* is proposed, so future
|
||||
API calls queue behind a large `Retry-After` from a download burst.
|
||||
|
||||
## 8. Test Plan
|
||||
|
||||
1. **Coordinator unit tests** (new file):
|
||||
- 429 with `retry_after` → `wait_for_slot` blocks ~that long, then passes.
|
||||
- N concurrent waiters all wake together (herd test, wall-clock ≈ one
|
||||
window, not N windows).
|
||||
- Consecutive 429s grow backoff; `register_success` resets.
|
||||
- Missing `Retry-After` → default backoff path.
|
||||
- Wait cap: request fails after `rate_limit_max_wait_seconds` with
|
||||
`RateLimitError`.
|
||||
2. **Downloader tests** (mock aiohttp session): 429 then 200 → `make_request`
|
||||
returns success after gate delay; two back-to-back calls to the same
|
||||
destination are spaced ≥ `min_interval`; different destinations are not
|
||||
spaced.
|
||||
3. **Provider tests**: `FallbackMetadataProvider.get_model_version_info` —
|
||||
Civitai raises `RateLimitError` → CivArchive mock **not called**; 404 still
|
||||
falls through to CivArchive; sqlite still tried after network 429.
|
||||
4. **Sync-service test**: `fetch_and_update_model` with a rate-limited default
|
||||
provider → result error contains `"Rate limited"` and sqlite attempt state
|
||||
unchanged.
|
||||
5. **Batch-import test**: analysis provider 429s first, then succeeds →
|
||||
item ends `SUCCESS` (wait path), and post-cap 429 → `SKIPPED` with
|
||||
rate-limit reason (phase 2).
|
||||
6. Full regression: `pytest tests/services tests/routes tests/standalone`
|
||||
(currently 1582 passing).
|
||||
|
||||
## 9. Implementation Phases
|
||||
|
||||
- **Phase 1 (this plan, reviewed):** `RateLimitCoordinator` +
|
||||
`Downloader.make_request` integration (guard fail-fast → gate pre-check
|
||||
pacing → 429 register/wait/retry loop with cap → `gate_handled` marking) +
|
||||
settings + **Fix C failover semantics** (`FallbackMetadataProvider`,
|
||||
`fetch_and_update_model` — moved up from phase 2: smallest diff, kills the
|
||||
CivArchive flood immediately, independent of coordinator correctness) +
|
||||
`_RateLimitRetryHelper` double-wait fix + coordinator/downloader/provider/
|
||||
sync tests.
|
||||
- **Phase 2:** batch-import `SKIPPED`-on-rate-limit + `rate_limited` WebSocket
|
||||
progress flag + slowdown hint (confirmed, §10),
|
||||
`download_to_memory`/HEAD 429 registration, batch tests.
|
||||
- **Phase 3:** full regression + docs + commit referencing `(#1085)`.
|
||||
|
||||
## 10. Review Checklist — Decisions (2026-08-27)
|
||||
|
||||
- [x] Default pacing interval `0.75 s` — **accepted** as conservative default;
|
||||
tunable via `rate_limit_min_interval_seconds`. Revisit if CivitAI
|
||||
publishes keyed/anonymous ceilings.
|
||||
- [x] Wait cap `300 s` — **accepted**; long-window CivArchive lookups fail →
|
||||
batch import marks them `SKIPPED` with a rate-limit reason (phase 2).
|
||||
- [x] Interactive API calls also wait (bounded) — **yes**, same behavior for
|
||||
all callers.
|
||||
- [x] Keep sqlite as last resort behind a network rate limit — **yes**
|
||||
(local-only, no vendor cost).
|
||||
- [x] UI hint — **yes**: WebSocket `rate_limited` flag + "rate limited —
|
||||
slowing down" hint in batch-import progress (phase 2); INFO logging
|
||||
regardless.
|
||||
@@ -1033,6 +1033,8 @@
|
||||
"start": "Start Import",
|
||||
"startImport": "Start Import",
|
||||
"importing": "Importing...",
|
||||
"rateLimitedSlowdown": "[TODO: Translate] Rate limited — slowing down...",
|
||||
"rateLimitedHint": "[TODO: Translate] Some items were skipped due to metadata provider rate limits. Re-run the import later to retry them.",
|
||||
"progress": "Progress",
|
||||
"total": "Total",
|
||||
"success": "Success",
|
||||
@@ -2038,6 +2040,7 @@
|
||||
"batchImportCancelFailed": "Failed to cancel batch import: {message}",
|
||||
"batchImportNoUrls": "Please enter at least one URL or file path",
|
||||
"batchImportNoDirectory": "Please enter a directory path",
|
||||
"batchImportRateLimited": "[TODO: Translate] Metadata provider rate limit reached — requests are being slowed and some items may be skipped. You can re-run the import later.",
|
||||
"batchImportBrowseFailed": "Failed to browse directory: {message}",
|
||||
"batchImportDirectorySelected": "Directory selected: {path}",
|
||||
"noRecipesSelected": "Keine Rezepte ausgewählt",
|
||||
|
||||
@@ -1033,6 +1033,8 @@
|
||||
"start": "Start Import",
|
||||
"startImport": "Start Import",
|
||||
"importing": "Importing...",
|
||||
"rateLimitedSlowdown": "Rate limited — slowing down...",
|
||||
"rateLimitedHint": "Some items were skipped due to metadata provider rate limits. Re-run the import later to retry them.",
|
||||
"progress": "Progress",
|
||||
"total": "Total",
|
||||
"success": "Success",
|
||||
@@ -2038,6 +2040,7 @@
|
||||
"batchImportCancelFailed": "Failed to cancel batch import: {message}",
|
||||
"batchImportNoUrls": "Please enter at least one URL or file path",
|
||||
"batchImportNoDirectory": "Please enter a directory path",
|
||||
"batchImportRateLimited": "Metadata provider rate limit reached — requests are being slowed and some items may be skipped. You can re-run the import later.",
|
||||
"batchImportBrowseFailed": "Failed to browse directory: {message}",
|
||||
"batchImportDirectorySelected": "Directory selected: {path}",
|
||||
"noRecipesSelected": "No recipes selected",
|
||||
|
||||
@@ -1033,6 +1033,8 @@
|
||||
"start": "Start Import",
|
||||
"startImport": "Start Import",
|
||||
"importing": "Importing...",
|
||||
"rateLimitedSlowdown": "[TODO: Translate] Rate limited — slowing down...",
|
||||
"rateLimitedHint": "[TODO: Translate] Some items were skipped due to metadata provider rate limits. Re-run the import later to retry them.",
|
||||
"progress": "Progress",
|
||||
"total": "Total",
|
||||
"success": "Success",
|
||||
@@ -2038,6 +2040,7 @@
|
||||
"batchImportCancelFailed": "Failed to cancel batch import: {message}",
|
||||
"batchImportNoUrls": "Please enter at least one URL or file path",
|
||||
"batchImportNoDirectory": "Please enter a directory path",
|
||||
"batchImportRateLimited": "[TODO: Translate] Metadata provider rate limit reached — requests are being slowed and some items may be skipped. You can re-run the import later.",
|
||||
"batchImportBrowseFailed": "Failed to browse directory: {message}",
|
||||
"batchImportDirectorySelected": "Directory selected: {path}",
|
||||
"noRecipesSelected": "No se han seleccionado recetas",
|
||||
|
||||
@@ -1033,6 +1033,8 @@
|
||||
"start": "Start Import",
|
||||
"startImport": "Start Import",
|
||||
"importing": "Importing...",
|
||||
"rateLimitedSlowdown": "[TODO: Translate] Rate limited — slowing down...",
|
||||
"rateLimitedHint": "[TODO: Translate] Some items were skipped due to metadata provider rate limits. Re-run the import later to retry them.",
|
||||
"progress": "Progress",
|
||||
"total": "Total",
|
||||
"success": "Success",
|
||||
@@ -2038,6 +2040,7 @@
|
||||
"batchImportCancelFailed": "Failed to cancel batch import: {message}",
|
||||
"batchImportNoUrls": "Please enter at least one URL or file path",
|
||||
"batchImportNoDirectory": "Please enter a directory path",
|
||||
"batchImportRateLimited": "[TODO: Translate] Metadata provider rate limit reached — requests are being slowed and some items may be skipped. You can re-run the import later.",
|
||||
"batchImportBrowseFailed": "Failed to browse directory: {message}",
|
||||
"batchImportDirectorySelected": "Directory selected: {path}",
|
||||
"noRecipesSelected": "Aucune recette sélectionnée",
|
||||
|
||||
@@ -1033,6 +1033,8 @@
|
||||
"start": "Start Import",
|
||||
"startImport": "Start Import",
|
||||
"importing": "Importing...",
|
||||
"rateLimitedSlowdown": "[TODO: Translate] Rate limited — slowing down...",
|
||||
"rateLimitedHint": "[TODO: Translate] Some items were skipped due to metadata provider rate limits. Re-run the import later to retry them.",
|
||||
"progress": "Progress",
|
||||
"total": "Total",
|
||||
"success": "Success",
|
||||
@@ -2038,6 +2040,7 @@
|
||||
"batchImportCancelFailed": "Failed to cancel batch import: {message}",
|
||||
"batchImportNoUrls": "Please enter at least one URL or file path",
|
||||
"batchImportNoDirectory": "Please enter a directory path",
|
||||
"batchImportRateLimited": "[TODO: Translate] Metadata provider rate limit reached — requests are being slowed and some items may be skipped. You can re-run the import later.",
|
||||
"batchImportBrowseFailed": "Failed to browse directory: {message}",
|
||||
"batchImportDirectorySelected": "Directory selected: {path}",
|
||||
"noRecipesSelected": "לא נבחרו מתכונים",
|
||||
|
||||
@@ -1033,6 +1033,8 @@
|
||||
"start": "Start Import",
|
||||
"startImport": "Start Import",
|
||||
"importing": "Importing...",
|
||||
"rateLimitedSlowdown": "[TODO: Translate] Rate limited — slowing down...",
|
||||
"rateLimitedHint": "[TODO: Translate] Some items were skipped due to metadata provider rate limits. Re-run the import later to retry them.",
|
||||
"progress": "Progress",
|
||||
"total": "Total",
|
||||
"success": "Success",
|
||||
@@ -2038,6 +2040,7 @@
|
||||
"batchImportCancelFailed": "Failed to cancel batch import: {message}",
|
||||
"batchImportNoUrls": "Please enter at least one URL or file path",
|
||||
"batchImportNoDirectory": "Please enter a directory path",
|
||||
"batchImportRateLimited": "[TODO: Translate] Metadata provider rate limit reached — requests are being slowed and some items may be skipped. You can re-run the import later.",
|
||||
"batchImportBrowseFailed": "Failed to browse directory: {message}",
|
||||
"batchImportDirectorySelected": "Directory selected: {path}",
|
||||
"noRecipesSelected": "レシピが選択されていません",
|
||||
|
||||
@@ -1033,6 +1033,8 @@
|
||||
"start": "Start Import",
|
||||
"startImport": "Start Import",
|
||||
"importing": "Importing...",
|
||||
"rateLimitedSlowdown": "[TODO: Translate] Rate limited — slowing down...",
|
||||
"rateLimitedHint": "[TODO: Translate] Some items were skipped due to metadata provider rate limits. Re-run the import later to retry them.",
|
||||
"progress": "Progress",
|
||||
"total": "Total",
|
||||
"success": "Success",
|
||||
@@ -2038,6 +2040,7 @@
|
||||
"batchImportCancelFailed": "Failed to cancel batch import: {message}",
|
||||
"batchImportNoUrls": "Please enter at least one URL or file path",
|
||||
"batchImportNoDirectory": "Please enter a directory path",
|
||||
"batchImportRateLimited": "[TODO: Translate] Metadata provider rate limit reached — requests are being slowed and some items may be skipped. You can re-run the import later.",
|
||||
"batchImportBrowseFailed": "Failed to browse directory: {message}",
|
||||
"batchImportDirectorySelected": "Directory selected: {path}",
|
||||
"noRecipesSelected": "선택한 레시피가 없습니다",
|
||||
|
||||
@@ -1033,6 +1033,8 @@
|
||||
"start": "Start Import",
|
||||
"startImport": "Start Import",
|
||||
"importing": "Importing...",
|
||||
"rateLimitedSlowdown": "[TODO: Translate] Rate limited — slowing down...",
|
||||
"rateLimitedHint": "[TODO: Translate] Some items were skipped due to metadata provider rate limits. Re-run the import later to retry them.",
|
||||
"progress": "Progress",
|
||||
"total": "Total",
|
||||
"success": "Success",
|
||||
@@ -2038,6 +2040,7 @@
|
||||
"batchImportCancelFailed": "Failed to cancel batch import: {message}",
|
||||
"batchImportNoUrls": "Please enter at least one URL or file path",
|
||||
"batchImportNoDirectory": "Please enter a directory path",
|
||||
"batchImportRateLimited": "[TODO: Translate] Metadata provider rate limit reached — requests are being slowed and some items may be skipped. You can re-run the import later.",
|
||||
"batchImportBrowseFailed": "Failed to browse directory: {message}",
|
||||
"batchImportDirectorySelected": "Directory selected: {path}",
|
||||
"noRecipesSelected": "Рецепты не выбраны",
|
||||
|
||||
@@ -1033,6 +1033,8 @@
|
||||
"start": "开始导入",
|
||||
"startImport": "开始导入",
|
||||
"importing": "正在导入配方...",
|
||||
"rateLimitedSlowdown": "[TODO: Translate] Rate limited — slowing down...",
|
||||
"rateLimitedHint": "[TODO: Translate] Some items were skipped due to metadata provider rate limits. Re-run the import later to retry them.",
|
||||
"progress": "进度",
|
||||
"total": "总计",
|
||||
"success": "成功",
|
||||
@@ -2038,6 +2040,7 @@
|
||||
"batchImportCancelFailed": "取消批量导入失败：{message}",
|
||||
"batchImportNoUrls": "请输入至少一个 URL 或文件路径",
|
||||
"batchImportNoDirectory": "请输入目录路径",
|
||||
"batchImportRateLimited": "[TODO: Translate] Metadata provider rate limit reached — requests are being slowed and some items may be skipped. You can re-run the import later.",
|
||||
"batchImportBrowseFailed": "浏览目录失败：{message}",
|
||||
"batchImportDirectorySelected": "已选择目录：{path}",
|
||||
"noRecipesSelected": "未选择任何配方",
|
||||
|
||||
@@ -1033,6 +1033,8 @@
|
||||
"start": "開始匯入",
|
||||
"startImport": "開始匯入",
|
||||
"importing": "匯入中...",
|
||||
"rateLimitedSlowdown": "[TODO: Translate] Rate limited — slowing down...",
|
||||
"rateLimitedHint": "[TODO: Translate] Some items were skipped due to metadata provider rate limits. Re-run the import later to retry them.",
|
||||
"progress": "進度",
|
||||
"total": "總計",
|
||||
"success": "成功",
|
||||
@@ -2038,6 +2040,7 @@
|
||||
"batchImportCancelFailed": "取消批量匯入失敗：{message}",
|
||||
"batchImportNoUrls": "請輸入至少一個 URL 或檔案路徑",
|
||||
"batchImportNoDirectory": "請輸入目錄路徑",
|
||||
"batchImportRateLimited": "[TODO: Translate] Metadata provider rate limit reached — requests are being slowed and some items may be skipped. You can re-run the import later.",
|
||||
"batchImportBrowseFailed": "瀏覽目錄失敗：{message}",
|
||||
"batchImportDirectorySelected": "已選擇目錄：{path}",
|
||||
"noRecipesSelected": "未選取任何食譜",
|
||||
|
||||
+28
-18
@@ -40,24 +40,34 @@ class ComfyMetadataParser(RecipeMetadataParser):
|
||||
checkpoint_node = next(iter(checkpoint_nodes.values()))
|
||||
if 'inputs' in checkpoint_node and 'ckpt_name' in checkpoint_node['inputs']:
|
||||
checkpoint_name = checkpoint_node['inputs']['ckpt_name']
|
||||
checkpoint_match = re.search(r'civitai:(\d+)@(\d+)', checkpoint_name)
|
||||
if checkpoint_match:
|
||||
checkpoint_id = checkpoint_match.group(1)
|
||||
checkpoint_version_id = checkpoint_match.group(2)
|
||||
checkpoint = {
|
||||
'id': checkpoint_version_id,
|
||||
'modelId': checkpoint_id,
|
||||
'name': f"Checkpoint {checkpoint_id}",
|
||||
'version': '',
|
||||
'type': 'checkpoint'
|
||||
}
|
||||
if metadata_provider:
|
||||
try:
|
||||
civitai_info_tuple = await metadata_provider.get_model_version_info(checkpoint_version_id)
|
||||
civitai_info, _ = civitai_info_tuple if isinstance(civitai_info_tuple, tuple) else (civitai_info_tuple, None)
|
||||
checkpoint = await self.populate_checkpoint_from_civitai(checkpoint, civitai_info)
|
||||
except Exception as e:
|
||||
logger.error(f"Error fetching Civitai info for checkpoint: {e}")
|
||||
# Some ComfyUI workflows serialize ckpt_name as a
|
||||
# single-element list (e.g. ["model.safetensors"]) or leave
|
||||
# the value unset (None). Neither is a string, so skip the
|
||||
# CivitAI-URN lookup instead of crashing re.search with a
|
||||
# TypeError that fails the whole image import.
|
||||
if isinstance(checkpoint_name, list):
|
||||
checkpoint_name = (
|
||||
checkpoint_name[0] if checkpoint_name else None
|
||||
)
|
||||
if isinstance(checkpoint_name, str):
|
||||
checkpoint_match = re.search(r'civitai:(\d+)@(\d+)', checkpoint_name)
|
||||
if checkpoint_match:
|
||||
checkpoint_id = checkpoint_match.group(1)
|
||||
checkpoint_version_id = checkpoint_match.group(2)
|
||||
checkpoint = {
|
||||
'id': checkpoint_version_id,
|
||||
'modelId': checkpoint_id,
|
||||
'name': f"Checkpoint {checkpoint_id}",
|
||||
'version': '',
|
||||
'type': 'checkpoint'
|
||||
}
|
||||
if metadata_provider:
|
||||
try:
|
||||
civitai_info_tuple = await metadata_provider.get_model_version_info(checkpoint_version_id)
|
||||
civitai_info, _ = civitai_info_tuple if isinstance(civitai_info_tuple, tuple) else (civitai_info_tuple, None)
|
||||
checkpoint = await self.populate_checkpoint_from_civitai(checkpoint, civitai_info)
|
||||
except Exception as e:
|
||||
logger.error(f"Error fetching Civitai info for checkpoint: {e}")
|
||||
|
||||
recipe_base_model = checkpoint.get('baseModel') if checkpoint else None
|
||||
loras = []
|
||||
|
||||
@@ -71,6 +71,9 @@ class BatchImportProgress:
|
||||
tags: List[str] = field(default_factory=list)
|
||||
skip_no_metadata: bool = False
|
||||
skip_duplicates: bool = False
|
||||
# Set once any item is skipped due to vendor rate limiting (#1085); lets
|
||||
# the UI surface a "slowing down / try again later" hint.
|
||||
rate_limited: bool = False
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
@@ -82,6 +85,7 @@ class BatchImportProgress:
|
||||
"skipped": self.skipped,
|
||||
"current_item": self.current_item,
|
||||
"status": self.status,
|
||||
"rate_limited": self.rate_limited,
|
||||
"started_at": self.started_at,
|
||||
"finished_at": self.finished_at,
|
||||
"progress_percent": round((self.completed / self.total) * 100, 1)
|
||||
@@ -118,6 +122,10 @@ class AdaptiveConcurrencyController:
|
||||
self._task_durations: List[float] = []
|
||||
self._recent_errors = 0
|
||||
self._recent_successes = 0
|
||||
# Batch-wide shared semaphore; created lazily on first use so the
|
||||
# controller can also be constructed outside a running event loop.
|
||||
self._semaphore: Optional[asyncio.Semaphore] = None
|
||||
self._semaphore_capacity = initial_concurrency
|
||||
|
||||
def record_result(self, duration: float, success: bool) -> None:
|
||||
self._task_durations.append(duration)
|
||||
@@ -146,7 +154,37 @@ class AdaptiveConcurrencyController:
|
||||
self._recent_successes = 0
|
||||
|
||||
def get_semaphore(self) -> asyncio.Semaphore:
|
||||
return asyncio.Semaphore(self.current_concurrency)
|
||||
"""Return the batch-wide shared semaphore.
|
||||
|
||||
The same semaphore instance is returned for every item of a batch so
|
||||
the configured concurrency bounds are actually enforced. Previously a
|
||||
fresh semaphore was created per call, letting every item run
|
||||
concurrently and hammering remote metadata providers without any
|
||||
limit.
|
||||
"""
|
||||
if self._semaphore is None:
|
||||
self._semaphore = asyncio.Semaphore(self.current_concurrency)
|
||||
self._semaphore_capacity = self.current_concurrency
|
||||
return self._semaphore
|
||||
|
||||
async def apply_concurrency(self) -> None:
|
||||
"""Synchronize the shared semaphore capacity with ``current_concurrency``.
|
||||
|
||||
Call after ``record_result`` (once per completed item). Growing the
|
||||
capacity is immediate (release). Shrinking requires acquiring a permit
|
||||
and holding it, which is best-effort while other tasks are still
|
||||
running — the capacity converges on subsequent calls.
|
||||
"""
|
||||
semaphore = self.get_semaphore()
|
||||
while self._semaphore_capacity < self.current_concurrency:
|
||||
semaphore.release()
|
||||
self._semaphore_capacity += 1
|
||||
while self._semaphore_capacity > self.current_concurrency:
|
||||
try:
|
||||
await asyncio.wait_for(semaphore.acquire(), timeout=0.01)
|
||||
except (asyncio.TimeoutError, asyncio.CancelledError):
|
||||
break
|
||||
self._semaphore_capacity -= 1
|
||||
|
||||
|
||||
class BatchImportService:
|
||||
@@ -349,6 +387,13 @@ class BatchImportService:
|
||||
ext = os.path.splitext(filename)[1].lower()
|
||||
return ext in self.SUPPORTED_EXTENSIONS
|
||||
|
||||
@staticmethod
|
||||
def _is_rate_limit_error(error: Optional[str]) -> bool:
|
||||
"""Return True when an error payload represents vendor rate limiting."""
|
||||
if not error:
|
||||
return False
|
||||
return "rate limit" in error.lower()
|
||||
|
||||
async def _run_batch_import(
|
||||
self,
|
||||
*,
|
||||
@@ -394,6 +439,9 @@ class BatchImportService:
|
||||
self._concurrency_controller.record_result(
|
||||
duration, result.get("success", False)
|
||||
)
|
||||
# Keep the shared batch semaphore in sync with the adaptively
|
||||
# adjusted concurrency so the bounds actually take effect.
|
||||
await self._concurrency_controller.apply_concurrency()
|
||||
|
||||
if result.get("success"):
|
||||
item.status = ImportStatus.SUCCESS
|
||||
@@ -404,6 +452,17 @@ class BatchImportService:
|
||||
item.status = ImportStatus.SKIPPED
|
||||
item.error_message = result.get("error")
|
||||
progress.skipped += 1
|
||||
elif self._is_rate_limit_error(result.get("error")):
|
||||
# Vendor rate limit is a transient, external condition —
|
||||
# do not pollute the failure count with it (#1085). The
|
||||
# import can simply be re-run later.
|
||||
item.status = ImportStatus.SKIPPED
|
||||
item.error_message = (
|
||||
f"Rate limited by metadata provider; "
|
||||
f"re-run the import later ({result.get('error')})"
|
||||
)
|
||||
progress.skipped += 1
|
||||
progress.rate_limited = True
|
||||
else:
|
||||
item.status = ImportStatus.FAILED
|
||||
item.error_message = result.get("error")
|
||||
@@ -411,11 +470,21 @@ class BatchImportService:
|
||||
|
||||
except Exception as e:
|
||||
self._logger.error(f"Error importing {item.source}: {e}")
|
||||
item.status = ImportStatus.FAILED
|
||||
item.error_message = str(e)
|
||||
item.duration = time.time() - start_time
|
||||
progress.failed += 1
|
||||
if self._is_rate_limit_error(str(e)):
|
||||
item.status = ImportStatus.SKIPPED
|
||||
item.error_message = (
|
||||
f"Rate limited by metadata provider; "
|
||||
f"re-run the import later ({e})"
|
||||
)
|
||||
progress.skipped += 1
|
||||
progress.rate_limited = True
|
||||
else:
|
||||
item.status = ImportStatus.FAILED
|
||||
item.error_message = str(e)
|
||||
progress.failed += 1
|
||||
self._concurrency_controller.record_result(item.duration, False)
|
||||
await self._concurrency_controller.apply_concurrency()
|
||||
|
||||
progress.completed += 1
|
||||
self._logger.info(
|
||||
|
||||
@@ -7,6 +7,7 @@ import logging
|
||||
import asyncio
|
||||
from copy import deepcopy
|
||||
from typing import Any, Optional, Dict, Tuple, List, cast
|
||||
from .connectivity_guard import is_expected_offline_error
|
||||
from .model_metadata_provider import CivArchiveModelMetadataProvider, ModelMetadataProviderManager
|
||||
from .downloader import get_downloader
|
||||
from .errors import RateLimitError
|
||||
@@ -46,7 +47,11 @@ class CivArchiveClient:
|
||||
"""Call CivArchive API and return JSON payload"""
|
||||
success, payload = await self._make_request(path, params=params)
|
||||
if not success:
|
||||
error = payload if isinstance(payload, str) else "Request failed"
|
||||
# Normalize empty-string failure payloads (e.g. a throttled
|
||||
# connection dropped without a message) so callers never see a
|
||||
# falsy error alongside a None payload — that combination used to
|
||||
# crash downstream None.get() calls.
|
||||
error = payload if isinstance(payload, str) and payload else "Request failed"
|
||||
return None, error
|
||||
if not isinstance(payload, dict):
|
||||
return None, "Invalid response structure"
|
||||
@@ -298,6 +303,8 @@ class CivArchiveClient:
|
||||
|
||||
async def _resolve_version_from_files(self, payload: Dict[str, Any]) -> Optional[Dict[str, Any]]:
|
||||
"""Fallback to fetch version data when only file metadata is available"""
|
||||
if not isinstance(payload, dict):
|
||||
return None
|
||||
data = self._normalize_payload(payload)
|
||||
files = data.get("files") or payload.get("files") or []
|
||||
if not isinstance(files, list):
|
||||
@@ -332,10 +339,13 @@ class CivArchiveClient:
|
||||
"""Find model by SHA256 hash value using CivArchive API"""
|
||||
try:
|
||||
payload, error = await self._request_json(f"/sha256/{model_hash.lower()}")
|
||||
if error:
|
||||
if "not found" in error.lower():
|
||||
# Treat a missing payload as an error even when the error string is
|
||||
# falsy; passing None into the split/transform helpers below used to
|
||||
# crash with "'NoneType' object has no attribute 'get'".
|
||||
if error is not None or payload is None:
|
||||
if error and "not found" in error.lower():
|
||||
return None, "Model not found"
|
||||
return None, error
|
||||
return None, error or "Request failed"
|
||||
|
||||
context, version_data, fallback_files = self._split_context(cast(Dict[str, Any], payload))
|
||||
transformed = self._transform_version(context, version_data, fallback_files)
|
||||
@@ -352,7 +362,14 @@ class CivArchiveClient:
|
||||
except RateLimitError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error fetching CivArchive model by hash {model_hash[:10]}: {e}")
|
||||
if is_expected_offline_error(str(e)):
|
||||
logger.debug(
|
||||
"Skipping CivArchive model by hash %s while offline: %s",
|
||||
model_hash[:10],
|
||||
e,
|
||||
)
|
||||
else:
|
||||
logger.error(f"Error fetching CivArchive model by hash {model_hash[:10]}: {e}")
|
||||
return None, str(e)
|
||||
|
||||
async def get_model_versions(self, model_id: str) -> Optional[Dict[str, Any]]:
|
||||
@@ -362,7 +379,14 @@ class CivArchiveClient:
|
||||
if error or payload is None:
|
||||
if error and "not found" in error.lower():
|
||||
return None
|
||||
logger.error(f"Error fetching CivArchive model versions for {model_id}: {error}")
|
||||
if is_expected_offline_error(error):
|
||||
logger.debug(
|
||||
"Skipping CivArchive model versions fetch for %s while offline: %s",
|
||||
model_id,
|
||||
error,
|
||||
)
|
||||
else:
|
||||
logger.error(f"Error fetching CivArchive model versions for {model_id}: {error}")
|
||||
return None
|
||||
|
||||
data = self._normalize_payload(payload)
|
||||
@@ -426,7 +450,19 @@ class CivArchiveClient:
|
||||
if error or payload is None:
|
||||
if error and "not found" in error.lower():
|
||||
return None
|
||||
logger.error(f"Error fetching CivArchive model version via API {model_id}/{version_id}: {error}")
|
||||
# The connectivity guard short-circuits requests during its
|
||||
# offline cooldown; that is an expected, transient state, so
|
||||
# log it as DEBUG instead of spamming one ERROR per request
|
||||
# (batch imports can hit this thousands of times).
|
||||
if is_expected_offline_error(error):
|
||||
logger.debug(
|
||||
"Skipping CivArchive model version fetch %s/%s while offline: %s",
|
||||
model_id,
|
||||
version_id,
|
||||
error,
|
||||
)
|
||||
else:
|
||||
logger.error(f"Error fetching CivArchive model version via API {model_id}/{version_id}: {error}")
|
||||
return None
|
||||
|
||||
context, version_data, fallback_files = self._split_context(payload)
|
||||
|
||||
+125
-57
@@ -32,6 +32,7 @@ from .connectivity_guard import (
|
||||
ConnectivityGuard,
|
||||
)
|
||||
from .errors import RateLimitError
|
||||
from .rate_limit_coordinator import RateLimitCoordinator
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -595,6 +596,21 @@ class Downloader:
|
||||
False,
|
||||
"File not found - the download link may be invalid or expired.",
|
||||
)
|
||||
elif response.status == 429:
|
||||
# Register the vendor's cooldown so API calls through
|
||||
# make_request queue behind it (#1085). The download
|
||||
# itself fails as before; retry policy stays with the
|
||||
# caller (download manager).
|
||||
retry_after = self._extract_retry_after(response.headers)
|
||||
coordinator = await RateLimitCoordinator.get_instance()
|
||||
if coordinator.enabled:
|
||||
coordinator.register_rate_limit(
|
||||
self._guard_destination(url), retry_after
|
||||
)
|
||||
logger.warning(
|
||||
f"Rate limited (429) for {url}, retry_after={retry_after}"
|
||||
)
|
||||
return False, f"Download rate limited (429), retry after {retry_after}s"
|
||||
else:
|
||||
logger.error(
|
||||
f"Download failed for {url} with status {response.status}"
|
||||
@@ -972,6 +988,11 @@ class Downloader:
|
||||
elif response.status == 429:
|
||||
raw_retry_after = response.headers.get("Retry-After")
|
||||
retry_after = _parse_retry_after(raw_retry_after or "")
|
||||
# Register the vendor's cooldown so API calls through
|
||||
# make_request queue behind it (#1085).
|
||||
coordinator = await RateLimitCoordinator.get_instance()
|
||||
if coordinator.enabled:
|
||||
coordinator.register_rate_limit(destination, retry_after)
|
||||
if raw_retry_after:
|
||||
logger.warning(
|
||||
"Rate limited (429) for %s, Retry-After: %ss", url, retry_after
|
||||
@@ -1041,6 +1062,14 @@ class Downloader:
|
||||
if response.status == 200:
|
||||
guard.register_success(destination)
|
||||
return True, dict(response.headers)
|
||||
elif response.status == 429:
|
||||
# Register the vendor's cooldown so API calls through
|
||||
# make_request queue behind it (#1085).
|
||||
retry_after = self._extract_retry_after(response.headers)
|
||||
coordinator = await RateLimitCoordinator.get_instance()
|
||||
if coordinator.enabled:
|
||||
coordinator.register_rate_limit(destination, retry_after)
|
||||
return False, f"Head request rate limited (429), retry after {retry_after}s"
|
||||
else:
|
||||
return False, f"Head request failed with status {response.status}"
|
||||
|
||||
@@ -1074,74 +1103,113 @@ class Downloader:
|
||||
|
||||
Returns:
|
||||
Tuple[bool, Union[Dict, str]]: (success, response data or error message)
|
||||
|
||||
When the rate-limit gate is enabled (``rate_limit_gate_enabled``),
|
||||
requests are paced per destination and 429 responses are honored by
|
||||
waiting out the ``Retry-After`` window (bounded by
|
||||
``rate_limit_max_wait_seconds``) before re-sending. A ``RateLimitError``
|
||||
returned after gate involvement is marked with ``gate_handled = True``
|
||||
so downstream retry helpers do not wait a second time.
|
||||
"""
|
||||
guard = await ConnectivityGuard.get_instance()
|
||||
destination = self._guard_destination(url)
|
||||
# Fail fast on transport-level outages before pacing: there is no
|
||||
# point waiting out a vendor cooldown while the network is down.
|
||||
if guard.should_block_request(destination):
|
||||
return False, OFFLINE_COOLDOWN_ERROR
|
||||
|
||||
try:
|
||||
session = await self.session
|
||||
# Debug log for proxy mode at request time
|
||||
if self.proxy_url:
|
||||
logger.debug(f"[make_request] Using app-level proxy: {self.proxy_url}")
|
||||
else:
|
||||
logger.debug(
|
||||
"[make_request] Using system-level proxy (trust_env) if configured."
|
||||
)
|
||||
coordinator = await RateLimitCoordinator.get_instance()
|
||||
gate_enabled = coordinator.enabled
|
||||
# Safety bound on the wait-and-resend loop; each 429 normally exits
|
||||
# via the wait cap in wait_for_slot, this covers pathological 429s
|
||||
# with tiny Retry-After values.
|
||||
max_resend_attempts = 5
|
||||
attempt = 0
|
||||
|
||||
# Prepare headers
|
||||
headers = self._get_auth_headers(use_auth)
|
||||
if custom_headers:
|
||||
headers.update(custom_headers)
|
||||
while True:
|
||||
if gate_enabled:
|
||||
try:
|
||||
await coordinator.wait_for_slot(destination)
|
||||
except RateLimitError as exc:
|
||||
exc.gate_handled = True
|
||||
return False, exc
|
||||
|
||||
# Add proxy to kwargs if not already present
|
||||
if "proxy" not in kwargs:
|
||||
kwargs["proxy"] = self.proxy_url
|
||||
|
||||
async with session.request(
|
||||
method, url, headers=headers, **kwargs
|
||||
) as response:
|
||||
if response.status == 200:
|
||||
guard.register_success(destination)
|
||||
# Try to parse as JSON, fall back to text
|
||||
try:
|
||||
data = await response.json()
|
||||
return True, data
|
||||
except:
|
||||
text = await response.text()
|
||||
return True, text
|
||||
elif response.status == 401:
|
||||
return False, "Unauthorized access - invalid or missing API key"
|
||||
elif response.status == 403:
|
||||
return False, "Access forbidden"
|
||||
elif response.status == 404:
|
||||
return False, "Resource not found"
|
||||
elif response.status == 429:
|
||||
retry_after = self._extract_retry_after(response.headers)
|
||||
error_msg = "Request rate limited"
|
||||
logger.warning(
|
||||
"Rate limit encountered for %s %s; retry_after=%s",
|
||||
method,
|
||||
url,
|
||||
retry_after,
|
||||
)
|
||||
return False, RateLimitError(
|
||||
error_msg,
|
||||
retry_after=retry_after,
|
||||
)
|
||||
try:
|
||||
session = await self.session
|
||||
# Debug log for proxy mode at request time
|
||||
if self.proxy_url:
|
||||
logger.debug(f"[make_request] Using app-level proxy: {self.proxy_url}")
|
||||
else:
|
||||
return False, f"Request failed with status {response.status}"
|
||||
logger.debug(
|
||||
"[make_request] Using system-level proxy (trust_env) if configured."
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
if guard.is_network_unreachable_error(e):
|
||||
guard.register_network_failure(e, destination)
|
||||
if guard.should_block_request(destination):
|
||||
return False, OFFLINE_COOLDOWN_ERROR
|
||||
logger.debug("Network unavailable for %s %s: %s", method, url, e)
|
||||
# Prepare headers
|
||||
headers = self._get_auth_headers(use_auth)
|
||||
if custom_headers:
|
||||
headers.update(custom_headers)
|
||||
|
||||
# Add proxy to kwargs if not already present
|
||||
if "proxy" not in kwargs:
|
||||
kwargs["proxy"] = self.proxy_url
|
||||
|
||||
async with session.request(
|
||||
method, url, headers=headers, **kwargs
|
||||
) as response:
|
||||
if response.status == 200:
|
||||
guard.register_success(destination)
|
||||
if gate_enabled:
|
||||
coordinator.register_success(destination)
|
||||
# Try to parse as JSON, fall back to text
|
||||
try:
|
||||
data = await response.json()
|
||||
return True, data
|
||||
except:
|
||||
text = await response.text()
|
||||
return True, text
|
||||
elif response.status == 401:
|
||||
return False, "Unauthorized access - invalid or missing API key"
|
||||
elif response.status == 403:
|
||||
return False, "Access forbidden"
|
||||
elif response.status == 404:
|
||||
return False, "Resource not found"
|
||||
elif response.status == 429:
|
||||
retry_after = self._extract_retry_after(response.headers)
|
||||
error_msg = "Request rate limited"
|
||||
if not gate_enabled:
|
||||
logger.warning(
|
||||
"Rate limit encountered for %s %s; retry_after=%s",
|
||||
method,
|
||||
url,
|
||||
retry_after,
|
||||
)
|
||||
return False, RateLimitError(
|
||||
error_msg,
|
||||
retry_after=retry_after,
|
||||
)
|
||||
# The coordinator logs the cooldown notice (INFO once
|
||||
# per window, DEBUG on extension).
|
||||
coordinator.register_rate_limit(destination, retry_after)
|
||||
attempt += 1
|
||||
if attempt >= max_resend_attempts:
|
||||
error = RateLimitError(error_msg, retry_after=retry_after)
|
||||
error.gate_handled = True
|
||||
return False, error
|
||||
# Loop back: wait_for_slot blocks until the cooldown
|
||||
# elapses (or raises once the wait exceeds the cap).
|
||||
continue
|
||||
else:
|
||||
return False, f"Request failed with status {response.status}"
|
||||
|
||||
except Exception as e:
|
||||
if guard.is_network_unreachable_error(e):
|
||||
guard.register_network_failure(e, destination)
|
||||
if guard.should_block_request(destination):
|
||||
return False, OFFLINE_COOLDOWN_ERROR
|
||||
logger.debug("Network unavailable for %s %s: %s", method, url, e)
|
||||
return False, str(e)
|
||||
logger.error(f"Error making {method} request to {url}: {e}")
|
||||
return False, str(e)
|
||||
logger.error(f"Error making {method} request to {url}: {e}")
|
||||
return False, str(e)
|
||||
|
||||
async def close(self):
|
||||
"""Close the HTTP session"""
|
||||
|
||||
@@ -245,16 +245,23 @@ class MetadataSyncService:
|
||||
civitai_api_not_found = False
|
||||
any_rate_limited = False
|
||||
|
||||
skip_network_providers = False
|
||||
for provider_name, provider in provider_attempts:
|
||||
if skip_network_providers and provider_name != "sqlite":
|
||||
# A network provider was already rate-limited; failing
|
||||
# over to another network provider just spreads the flood
|
||||
# (#1085). The local sqlite archive stays as last resort.
|
||||
continue
|
||||
try:
|
||||
civitai_metadata_candidate, error = await provider.get_model_by_hash(sha256)
|
||||
except RateLimitError as exc:
|
||||
logger.warning(
|
||||
"Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
|
||||
"Provider %s is rate-limited (retry_after=%.0fs); not failing over to other network providers",
|
||||
provider_name or provider.__class__.__name__,
|
||||
exc.retry_after or 0,
|
||||
)
|
||||
any_rate_limited = True
|
||||
skip_network_providers = True
|
||||
continue
|
||||
except Exception as exc: # pragma: no cover - defensive logging
|
||||
logger.error("Provider %s failed for hash %s: %s", provider_name, sha256, exc)
|
||||
|
||||
@@ -66,6 +66,14 @@ class _RateLimitRetryHelper:
|
||||
except RateLimitError as exc:
|
||||
attempt += 1
|
||||
|
||||
# The downloader's rate-limit gate already applied the wait
|
||||
# policy for this request (waited out the vendor window or
|
||||
# deliberately refused because it exceeds the cap). Sleeping
|
||||
# again here would double the wait — just propagate.
|
||||
if getattr(exc, "gate_handled", False):
|
||||
exc.provider = exc.provider or label
|
||||
raise
|
||||
|
||||
# Determine effective retry limit based on rate-limit magnitude
|
||||
effective_retry_limit = self._retry_limit # default: 3
|
||||
if exc.retry_after is not None and exc.retry_after >= 120.0:
|
||||
@@ -101,6 +109,12 @@ class _RateLimitRetryHelper:
|
||||
|
||||
return min(self._max_delay, max(0.0, base_delay))
|
||||
|
||||
|
||||
# Labels of providers that are free to consult even while a network provider
|
||||
# is rate-limited (local lookups, no vendor cost).
|
||||
_LOCAL_PROVIDER_LABELS = frozenset({"sqlite"})
|
||||
|
||||
|
||||
class ModelMetadataProvider(ABC):
|
||||
"""Base abstract class for all model metadata providers"""
|
||||
|
||||
@@ -451,7 +465,14 @@ class SQLiteModelMetadataProvider(ModelMetadataProvider):
|
||||
return None
|
||||
|
||||
class FallbackMetadataProvider(ModelMetadataProvider):
|
||||
"""Try providers in order, return first successful result."""
|
||||
"""Try providers in order, return first successful result.
|
||||
|
||||
Rate-limit policy (#1085): once a *network* provider raises
|
||||
``RateLimitError``, the chain stops consulting further network providers —
|
||||
failing over would just spread the flood to the next vendor. Local-only
|
||||
providers (see ``_LOCAL_PROVIDER_LABELS``) are still allowed as a last
|
||||
resort because they cost the vendor nothing.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -486,7 +507,10 @@ class FallbackMetadataProvider(ModelMetadataProvider):
|
||||
)
|
||||
|
||||
async def get_model_by_hash(self, model_hash: str) -> Tuple[Optional[Dict[str, Any]], Optional[str]]:
|
||||
rate_limited = False
|
||||
for provider, label in self._iter_providers():
|
||||
if rate_limited and label not in _LOCAL_PROVIDER_LABELS:
|
||||
continue
|
||||
try:
|
||||
result, error = await self._call_with_rate_limit(
|
||||
label,
|
||||
@@ -496,8 +520,9 @@ class FallbackMetadataProvider(ModelMetadataProvider):
|
||||
if result:
|
||||
return result, error
|
||||
except RateLimitError as exc:
|
||||
rate_limited = True
|
||||
logger.warning(
|
||||
"Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
|
||||
"Provider %s is rate-limited (retry_after=%.0fs); not failing over to other network providers",
|
||||
label,
|
||||
exc.retry_after or 0,
|
||||
)
|
||||
@@ -505,11 +530,18 @@ class FallbackMetadataProvider(ModelMetadataProvider):
|
||||
except Exception as e:
|
||||
logger.debug("Provider %s failed for get_model_by_hash: %s", label, e)
|
||||
continue
|
||||
if rate_limited:
|
||||
# Distinct from "Model not found": callers must not mistake a
|
||||
# rate-limited lookup for a confirmed deletion.
|
||||
return None, "Rate limited"
|
||||
return None, "Model not found"
|
||||
|
||||
async def get_model_versions(self, model_id: str) -> Optional[Dict[str, Any]]:
|
||||
not_found_confirmed = False
|
||||
rate_limited = False
|
||||
for provider, label in self._iter_providers():
|
||||
if rate_limited and label not in _LOCAL_PROVIDER_LABELS:
|
||||
continue
|
||||
try:
|
||||
result = await self._call_with_rate_limit(
|
||||
label,
|
||||
@@ -519,8 +551,9 @@ class FallbackMetadataProvider(ModelMetadataProvider):
|
||||
if result:
|
||||
return result
|
||||
except RateLimitError as exc:
|
||||
rate_limited = True
|
||||
logger.warning(
|
||||
"Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
|
||||
"Provider %s is rate-limited (retry_after=%.0fs); not failing over to other network providers",
|
||||
label,
|
||||
exc.retry_after or 0,
|
||||
)
|
||||
@@ -539,7 +572,10 @@ class FallbackMetadataProvider(ModelMetadataProvider):
|
||||
return None
|
||||
|
||||
async def get_model_version(self, model_id: Optional[int] = None, version_id: Optional[int] = None) -> Optional[Dict[str, Any]]:
|
||||
rate_limited = False
|
||||
for provider, label in self._iter_providers():
|
||||
if rate_limited and label not in _LOCAL_PROVIDER_LABELS:
|
||||
continue
|
||||
try:
|
||||
result = await self._call_with_rate_limit(
|
||||
label,
|
||||
@@ -550,8 +586,9 @@ class FallbackMetadataProvider(ModelMetadataProvider):
|
||||
if result:
|
||||
return result
|
||||
except RateLimitError as exc:
|
||||
rate_limited = True
|
||||
logger.warning(
|
||||
"Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
|
||||
"Provider %s is rate-limited (retry_after=%.0fs); not failing over to other network providers",
|
||||
label,
|
||||
exc.retry_after or 0,
|
||||
)
|
||||
@@ -562,7 +599,10 @@ class FallbackMetadataProvider(ModelMetadataProvider):
|
||||
return None
|
||||
|
||||
async def get_model_version_info(self, version_id: str) -> Tuple[Optional[Dict[str, Any]], Optional[str]]:
|
||||
rate_limited = False
|
||||
for provider, label in self._iter_providers():
|
||||
if rate_limited and label not in _LOCAL_PROVIDER_LABELS:
|
||||
continue
|
||||
try:
|
||||
result, error = await self._call_with_rate_limit(
|
||||
label,
|
||||
@@ -572,8 +612,9 @@ class FallbackMetadataProvider(ModelMetadataProvider):
|
||||
if result:
|
||||
return result, error
|
||||
except RateLimitError as exc:
|
||||
rate_limited = True
|
||||
logger.warning(
|
||||
"Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
|
||||
"Provider %s is rate-limited (retry_after=%.0fs); not failing over to other network providers",
|
||||
label,
|
||||
exc.retry_after or 0,
|
||||
)
|
||||
@@ -581,12 +622,17 @@ class FallbackMetadataProvider(ModelMetadataProvider):
|
||||
except Exception as e:
|
||||
logger.debug("Provider %s failed for get_model_version_info: %s", label, e)
|
||||
continue
|
||||
if rate_limited:
|
||||
return None, "Rate limited"
|
||||
return None, "No provider could retrieve the data"
|
||||
|
||||
async def get_model_versions_by_hashes(
|
||||
self, hashes: List[str]
|
||||
) -> Optional[List[Dict[str, Any]]]:
|
||||
rate_limited = False
|
||||
for provider, label in self._iter_providers():
|
||||
if rate_limited and label not in _LOCAL_PROVIDER_LABELS:
|
||||
continue
|
||||
try:
|
||||
result = await self._call_with_rate_limit(
|
||||
label,
|
||||
@@ -598,8 +644,9 @@ class FallbackMetadataProvider(ModelMetadataProvider):
|
||||
except NotImplementedError:
|
||||
continue
|
||||
except RateLimitError as exc:
|
||||
rate_limited = True
|
||||
logger.warning(
|
||||
"Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
|
||||
"Provider %s is rate-limited (retry_after=%.0fs); not failing over to other network providers",
|
||||
label,
|
||||
exc.retry_after or 0,
|
||||
)
|
||||
@@ -614,7 +661,10 @@ class FallbackMetadataProvider(ModelMetadataProvider):
|
||||
return None
|
||||
|
||||
async def get_user_models(self, username: str, cursor: Optional[str] = None) -> Optional[Dict[str, Any]]:
|
||||
rate_limited = False
|
||||
for provider, label in self._iter_providers():
|
||||
if rate_limited and label not in _LOCAL_PROVIDER_LABELS:
|
||||
continue
|
||||
try:
|
||||
result = await self._call_with_rate_limit(
|
||||
label,
|
||||
@@ -625,8 +675,9 @@ class FallbackMetadataProvider(ModelMetadataProvider):
|
||||
if result is not None:
|
||||
return result
|
||||
except RateLimitError as exc:
|
||||
rate_limited = True
|
||||
logger.warning(
|
||||
"Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
|
||||
"Provider %s is rate-limited (retry_after=%.0fs); not failing over to other network providers",
|
||||
label,
|
||||
exc.retry_after or 0,
|
||||
)
|
||||
|
||||
@@ -0,0 +1,213 @@
|
||||
"""Process-wide, per-destination rate-limit gate for outbound API traffic.
|
||||
|
||||
Implements the pacing/gating layer designed in
|
||||
``docs/plans/issue-1085-rate-limit-design.md``:
|
||||
|
||||
- **Reactive gate**: a 429 response arms ``next_allowed_send`` from the
|
||||
vendor's ``Retry-After`` (or exponential backoff when the header is
|
||||
missing); subsequent requests to the same destination wait out the window.
|
||||
- **Preemptive pacing**: a minimum inter-request interval per destination
|
||||
spaces consecutive sends so bursts never form in the first place.
|
||||
- **Herd-free**: waiters are serialized through a per-destination lock, so
|
||||
each one claims a distinct send slot instead of thousands of coroutines
|
||||
waking up together.
|
||||
- **Bounded**: waits longer than ``rate_limit_max_wait_seconds`` are refused
|
||||
by raising :class:`RateLimitError`, leaving the final decision to callers.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Dict, Optional
|
||||
|
||||
from .errors import RateLimitError
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_MIN_INTERVAL_SECONDS = 0.75
|
||||
DEFAULT_MAX_WAIT_SECONDS = 300.0
|
||||
BASE_BACKOFF_SECONDS = 30.0
|
||||
MAX_BACKOFF_SECONDS = 1800.0
|
||||
|
||||
|
||||
@dataclass
|
||||
class _DestinationState:
|
||||
"""Rate-limit bookkeeping for one destination (hostname)."""
|
||||
|
||||
next_allowed_send: float = 0.0 # time.monotonic() timestamp
|
||||
consecutive_429: int = 0
|
||||
last_send_at: float = 0.0 # time.monotonic() timestamp
|
||||
lock: asyncio.Lock = field(default_factory=asyncio.Lock)
|
||||
|
||||
|
||||
class RateLimitCoordinator:
|
||||
"""Coordinates outbound request pacing per destination.
|
||||
|
||||
Singleton mirroring :class:`ConnectivityGuard`'s pattern. All waits are
|
||||
bounded by the ``rate_limit_max_wait_seconds`` setting; when the required
|
||||
wait exceeds the cap, :meth:`wait_for_slot` raises :class:`RateLimitError`
|
||||
instead of parking the caller.
|
||||
"""
|
||||
|
||||
_instance: "RateLimitCoordinator | None" = None
|
||||
_instance_lock = asyncio.Lock()
|
||||
|
||||
@classmethod
|
||||
async def get_instance(cls) -> "RateLimitCoordinator":
|
||||
async with cls._instance_lock:
|
||||
if cls._instance is None:
|
||||
cls._instance = cls()
|
||||
return cls._instance
|
||||
|
||||
def __init__(self) -> None:
|
||||
if hasattr(self, "_initialized"):
|
||||
return
|
||||
self._initialized = True
|
||||
self._states: Dict[str, _DestinationState] = {}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Settings (read live so settings edits apply without a restart)
|
||||
|
||||
@staticmethod
|
||||
def _setting(key: str, default):
|
||||
try:
|
||||
from .settings_manager import get_settings_manager
|
||||
|
||||
return get_settings_manager().get(key, default)
|
||||
except Exception: # pragma: no cover - defensive: settings unavailable
|
||||
return default
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
return bool(self._setting("rate_limit_gate_enabled", True))
|
||||
|
||||
@property
|
||||
def min_interval_seconds(self) -> float:
|
||||
try:
|
||||
return max(0.0, float(self._setting("rate_limit_min_interval_seconds", DEFAULT_MIN_INTERVAL_SECONDS)))
|
||||
except (TypeError, ValueError):
|
||||
return DEFAULT_MIN_INTERVAL_SECONDS
|
||||
|
||||
@property
|
||||
def max_wait_seconds(self) -> float:
|
||||
try:
|
||||
return max(0.0, float(self._setting("rate_limit_max_wait_seconds", DEFAULT_MAX_WAIT_SECONDS)))
|
||||
except (TypeError, ValueError):
|
||||
return DEFAULT_MAX_WAIT_SECONDS
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# State helpers
|
||||
|
||||
@staticmethod
|
||||
def _normalize(destination: Optional[str]) -> str:
|
||||
if destination is None or not destination.strip():
|
||||
return "__global__"
|
||||
return destination.lower().strip()
|
||||
|
||||
def _state_for(self, destination: Optional[str]) -> _DestinationState:
|
||||
key = self._normalize(destination)
|
||||
if key not in self._states:
|
||||
self._states[key] = _DestinationState()
|
||||
return self._states[key]
|
||||
|
||||
def reset(self) -> None:
|
||||
"""Drop all per-destination state. Test seam."""
|
||||
self._states.clear()
|
||||
|
||||
def in_cooldown(self, destination: Optional[str] = None) -> bool:
|
||||
return self.remaining_seconds(destination) > 0
|
||||
|
||||
def remaining_seconds(self, destination: Optional[str] = None) -> float:
|
||||
state = self._state_for(destination)
|
||||
return max(0.0, state.next_allowed_send - time.monotonic())
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Gate operations
|
||||
|
||||
async def wait_for_slot(self, destination: Optional[str] = None) -> None:
|
||||
"""Block until this caller may send the next request to *destination*.
|
||||
|
||||
Waits for both the rate-limit cooldown (``next_allowed_send``) and the
|
||||
minimum inter-request interval (``last_send_at + min_interval``).
|
||||
Waiters queue on the per-destination lock, so concurrent callers are
|
||||
spaced out instead of stampeding when a cooldown expires.
|
||||
|
||||
Raises:
|
||||
RateLimitError: when the required wait exceeds
|
||||
``rate_limit_max_wait_seconds``.
|
||||
"""
|
||||
state = self._state_for(destination)
|
||||
deadline = time.monotonic() + self.max_wait_seconds
|
||||
async with state.lock:
|
||||
now = time.monotonic()
|
||||
wake_at = max(
|
||||
state.next_allowed_send,
|
||||
state.last_send_at + self.min_interval_seconds,
|
||||
)
|
||||
if wake_at > deadline:
|
||||
raise RateLimitError(
|
||||
f"Rate limit wait for '{self._normalize(destination)}' "
|
||||
f"exceeds the {self.max_wait_seconds:.0f}s cap",
|
||||
retry_after=wake_at - now,
|
||||
)
|
||||
delay = wake_at - now
|
||||
if delay > 0:
|
||||
logger.debug(
|
||||
"Rate-limit gate: pacing request to '%s' by %.2fs",
|
||||
self._normalize(destination),
|
||||
delay,
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
state.last_send_at = time.monotonic()
|
||||
|
||||
def register_rate_limit(
|
||||
self,
|
||||
destination: Optional[str],
|
||||
retry_after: Optional[float] = None,
|
||||
) -> float:
|
||||
"""Record a 429 for *destination* and arm the cooldown window.
|
||||
|
||||
Honors the vendor's ``Retry-After`` when present; otherwise grows an
|
||||
exponential backoff (30s base, doubling per consecutive 429, capped at
|
||||
1800s). Returns the cooldown duration in seconds.
|
||||
"""
|
||||
state = self._state_for(destination)
|
||||
state.consecutive_429 += 1
|
||||
if retry_after is not None and retry_after > 0:
|
||||
backoff = min(MAX_BACKOFF_SECONDS, float(retry_after))
|
||||
else:
|
||||
backoff = min(
|
||||
MAX_BACKOFF_SECONDS,
|
||||
BASE_BACKOFF_SECONDS * (2 ** (state.consecutive_429 - 1)),
|
||||
)
|
||||
now = time.monotonic()
|
||||
already_cooling = state.next_allowed_send > now
|
||||
state.next_allowed_send = max(state.next_allowed_send, now + backoff)
|
||||
if already_cooling:
|
||||
logger.debug(
|
||||
"Rate-limit cooldown for '%s' extended by %.0fs (consecutive_429=%d)",
|
||||
self._normalize(destination),
|
||||
backoff,
|
||||
state.consecutive_429,
|
||||
)
|
||||
else:
|
||||
logger.info(
|
||||
"Rate limited by '%s'; pausing requests for %.0fs",
|
||||
self._normalize(destination),
|
||||
backoff,
|
||||
)
|
||||
return backoff
|
||||
|
||||
def register_success(self, destination: Optional[str]) -> None:
|
||||
"""Reset rate-limit state after a successful request.
|
||||
|
||||
A 200 proves the vendor is accepting traffic again, so any armed
|
||||
cooldown window is cleared alongside the backoff counter (mirrors
|
||||
``ConnectivityGuard.register_success`` semantics).
|
||||
"""
|
||||
state = self._state_for(destination)
|
||||
state.consecutive_429 = 0
|
||||
state.next_allowed_send = 0.0
|
||||
@@ -70,6 +70,9 @@ DEFAULT_SETTINGS: Dict[str, Any] = {
|
||||
"enable_metadata_archive_db": False,
|
||||
"enable_civarchive_api": True,
|
||||
"metadata_provider_order": "civitai_archive_sqlite",
|
||||
"rate_limit_gate_enabled": True,
|
||||
"rate_limit_max_wait_seconds": 300,
|
||||
"rate_limit_min_interval_seconds": 0.75,
|
||||
"proxy_enabled": False,
|
||||
"proxy_host": "",
|
||||
"proxy_port": "",
|
||||
|
||||
+100
-21
@@ -49,34 +49,113 @@
|
||||
-ms-user-select: none;
|
||||
}
|
||||
|
||||
/* Remove bulk base model modal specific styles - now using shared components */
|
||||
/* Use shared metadata editing styles instead */
|
||||
/* ── Bulk base model modal — dedicated inline-list layout ───────────────
|
||||
Unlike the single-model modal (overlay dropdown), the bulk modal renders
|
||||
the option list inline so it never covers the footer buttons and only the
|
||||
list itself scrolls. Dropdown internals reuse the shared .base-model-*
|
||||
styles from lora-modal.css. */
|
||||
|
||||
/* Override for bulk base model select to ensure proper width */
|
||||
.bulk-base-model-select {
|
||||
#bulkBaseModelModal .modal-content {
|
||||
width: min(720px, calc(100vw - 2rem));
|
||||
height: min(640px, calc(100vh - var(--header-height, 48px) - 5.5rem));
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
#bulkBaseModelModal .modal-header {
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
#bulkBaseModelModal .modal-body {
|
||||
flex: 1;
|
||||
min-height: 0;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
#bulkBaseModelModal .bulk-add-tags-info {
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
#bulkBaseModelModal .bulk-base-model-label {
|
||||
flex-shrink: 0;
|
||||
display: block;
|
||||
font-weight: 500;
|
||||
margin-bottom: var(--space-1);
|
||||
color: var(--text-color);
|
||||
}
|
||||
|
||||
.bulk-base-model-picker {
|
||||
width: 100%;
|
||||
max-width: 100%;
|
||||
padding: 6px 10px;
|
||||
}
|
||||
|
||||
#bulkBaseModelModal .bulk-base-model-picker {
|
||||
flex: 1;
|
||||
min-height: 0;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
#bulkBaseModelModal .bulk-base-model-picker .base-model-search-wrapper {
|
||||
flex: 1;
|
||||
min-height: 0;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
width: 100%;
|
||||
z-index: auto;
|
||||
}
|
||||
|
||||
#bulkBaseModelModal .base-model-search-input-wrapper {
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
/* Inline list instead of overlay dropdown */
|
||||
#bulkBaseModelModal .base-model-dropdown {
|
||||
position: relative;
|
||||
top: auto;
|
||||
left: auto;
|
||||
right: auto;
|
||||
flex: 1;
|
||||
min-height: 0;
|
||||
max-height: none;
|
||||
overflow-y: auto;
|
||||
margin-top: var(--space-1);
|
||||
border: 1px solid var(--lora-border);
|
||||
border-radius: var(--border-radius-xs);
|
||||
border: 1px solid var(--border-color);
|
||||
background-color: var(--lora-surface);
|
||||
color: var(--text-color);
|
||||
font-size: 0.95em;
|
||||
height: 32px;
|
||||
box-shadow: none;
|
||||
z-index: auto;
|
||||
overscroll-behavior: contain;
|
||||
scrollbar-gutter: stable;
|
||||
scrollbar-width: thin;
|
||||
scrollbar-color: var(--lora-border) transparent;
|
||||
}
|
||||
|
||||
.bulk-base-model-select:focus {
|
||||
border-color: var(--lora-accent);
|
||||
outline: none;
|
||||
#bulkBaseModelModal .base-model-dropdown::-webkit-scrollbar {
|
||||
width: 8px;
|
||||
}
|
||||
|
||||
/* Dark theme support for bulk base model select */
|
||||
[data-theme="dark"] .bulk-base-model-select {
|
||||
background-color: rgba(30, 30, 30, 0.9);
|
||||
color: var(--text-color);
|
||||
#bulkBaseModelModal .base-model-dropdown::-webkit-scrollbar-thumb {
|
||||
background: var(--lora-border);
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
[data-theme="dark"] .bulk-base-model-select option {
|
||||
background-color: #2d2d2d;
|
||||
color: var(--text-color);
|
||||
#bulkBaseModelModal .base-model-dropdown::-webkit-scrollbar-track {
|
||||
background: transparent;
|
||||
}
|
||||
|
||||
/* The shared dropdown header uses opacity for its muted look, which makes the
|
||||
sticky background translucent — scrolled items bleed through. Keep the
|
||||
muted text color but restore a fully opaque background (bulk scope only). */
|
||||
#bulkBaseModelModal .base-model-dropdown-header {
|
||||
opacity: 1;
|
||||
color: var(--text-muted, var(--text-color));
|
||||
}
|
||||
|
||||
#bulkBaseModelModal .bulk-base-model-footer {
|
||||
flex-shrink: 0;
|
||||
padding-top: var(--space-2);
|
||||
margin-top: var(--space-2);
|
||||
border-top: 1px solid var(--lora-border);
|
||||
}
|
||||
@@ -0,0 +1,404 @@
|
||||
/**
|
||||
* BaseModelPicker.js
|
||||
* Shared searchable base model picker used by the single-model metadata modal
|
||||
* (commit mode) and the bulk base model modal (change mode).
|
||||
*/
|
||||
|
||||
import { BASE_MODEL_CATEGORIES, getMergedBaseModels, BASE_MODELS_UPDATED_EVENT } from '../../utils/constants.js';
|
||||
import { translate } from '../../utils/i18nHelpers.js';
|
||||
|
||||
// ── Filename-based base model inference ──────────────────────────────────────
|
||||
// Rules are ordered by specificity — first match wins for dedup.
|
||||
// Each rule checks the filename (lowercased) for a regex pattern and suggests
|
||||
// the associated base model values.
|
||||
|
||||
export const BASE_MODEL_FILENAME_RULES = [
|
||||
{ pattern: /flux\.?\s*2\s*klein/i, models: ['Flux.2 Klein 9B', 'Flux.2 Klein 9B-base', 'Flux.2 Klein 4B', 'Flux.2 Klein 4B-base'] },
|
||||
{ pattern: /flux\.?\s*2/i, models: ['Flux.2 D', 'Flux.2 Klein 9B', 'Flux.2 Klein 4B'] },
|
||||
{ pattern: /flux\.?\s*1\s*(dev|d)\b/i, models: ['Flux.1 D'] },
|
||||
{ pattern: /flux\.?\s*1\s*(schnell|s)\b/i, models: ['Flux.1 S'] },
|
||||
{ pattern: /flux/i, models: ['Flux.1 D', 'Flux.1 S', 'Flux.2 D'] },
|
||||
{ pattern: /sdxl/i, models: ['SDXL 1.0', 'SDXL Lightning', 'SDXL Hyper'] },
|
||||
{ pattern: /sd\s*1[._-\s]?5/i, models: ['SD 1.5'] },
|
||||
{ pattern: /sd\s*1[._-\s]?4/i, models: ['SD 1.4'] },
|
||||
{ pattern: /sd\s*1/i, models: ['SD 1.5', 'SD 1.4', 'SD 1.5 LCM', 'SD 1.5 Hyper'] },
|
||||
{ pattern: /sd\s*3[._-\s]?5/i, models: ['SD 3.5', 'SD 3.5 Medium', 'SD 3.5 Large', 'SD 3.5 Large Turbo'] },
|
||||
{ pattern: /sd\s*3/i, models: ['SD 3', 'SD 3.5'] },
|
||||
{ pattern: /wan\s*\.?\s*video/i, models: ['Wan Video', 'Wan Video 1.3B t2v', 'Wan Video 14B t2v', 'Wan Video 14B i2v 480p', 'Wan Video 14B i2v 720p'] },
|
||||
{ pattern: /hunyuan\s*\.?\s*video/i, models: ['Hunyuan Video'] },
|
||||
{ pattern: /ltxv/i, models: ['LTXV', 'LTXV2', 'LTXV 2.3'] },
|
||||
{ pattern: /cogvideo/i, models: ['CogVideoX'] },
|
||||
{ pattern: /pony/i, models: ['Pony', 'Pony V7'] },
|
||||
{ pattern: /illustrious/i, models: ['Illustrious'] },
|
||||
{ pattern: /noobai/i, models: ['NoobAI'] },
|
||||
{ pattern: /pixart/i, models: ['PixArt a', 'PixArt E'] },
|
||||
{ pattern: /aura\s*\.?\s*flow/i, models: ['AuraFlow'] },
|
||||
{ pattern: /kolors/i, models: ['Kolors'] },
|
||||
{ pattern: /hunyuan\s*1/i, models: ['Hunyuan 1'] },
|
||||
{ pattern: /lumina/i, models: ['Lumina'] },
|
||||
{ pattern: /hidream/i, models: ['HiDream'] },
|
||||
{ pattern: /qwen/i, models: ['Qwen'] },
|
||||
{ pattern: /chroma/i, models: ['Chroma'] },
|
||||
{ pattern: /anima/i, models: ['Anima'] },
|
||||
{ pattern: /sd\s*2[._-\s]?[01]/i, models: ['SD 2.0', 'SD 2.1'] },
|
||||
{ pattern: /mochi/i, models: ['Mochi'] },
|
||||
{ pattern: /svd/i, models: ['SVD'] },
|
||||
{ pattern: /zimage/i, models: ['ZImageTurbo', 'ZImageBase'] },
|
||||
{ pattern: /nucleus/i, models: ['Nucleus'] },
|
||||
{ pattern: /krea/i, models: ['Flux.1 Krea', 'Krea 2'] },
|
||||
{ pattern: /ernie/i, models: ['Ernie', 'Ernie Turbo'] },
|
||||
];
|
||||
|
||||
/**
|
||||
* Infer likely base model(s) from a filename + model name string.
|
||||
* Returns a deduplicated array in match-priority order.
|
||||
* @param {string} filename
|
||||
* @returns {string[]}
|
||||
*/
|
||||
export function inferBaseModelsFromFilename(filename) {
|
||||
if (!filename || typeof filename !== 'string') return [];
|
||||
const seen = new Set();
|
||||
const results = [];
|
||||
for (const rule of BASE_MODEL_FILENAME_RULES) {
|
||||
if (rule.pattern.test(filename)) {
|
||||
for (const model of rule.models) {
|
||||
if (!seen.has(model)) {
|
||||
seen.add(model);
|
||||
results.push(model);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Infer likely base model(s) from a set of file paths (bulk selection).
|
||||
* Each path contributes its basename to the inference; models are deduplicated
|
||||
* and sorted by how many selected paths matched them (most matches first).
|
||||
* Returns an empty array when nothing matches.
|
||||
* @param {string[]} filePaths
|
||||
* @returns {string[]}
|
||||
*/
|
||||
export function inferBaseModelsFromFilepaths(filePaths) {
|
||||
if (!Array.isArray(filePaths) || filePaths.length === 0) return [];
|
||||
const hitCounts = new Map(); // model -> number of paths that matched it
|
||||
for (const filePath of filePaths) {
|
||||
if (!filePath || typeof filePath !== 'string') continue;
|
||||
const basename = filePath.split(/[\\/]/).pop();
|
||||
for (const model of inferBaseModelsFromFilename(basename)) {
|
||||
hitCounts.set(model, (hitCounts.get(model) || 0) + 1);
|
||||
}
|
||||
}
|
||||
return Array.from(hitCounts.keys())
|
||||
.sort((a, b) => hitCounts.get(b) - hitCounts.get(a));
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the full categorized option list. Reads BASE_MODEL_CATEGORIES and
|
||||
* getMergedBaseModels() fresh on every call so late-arriving dynamic models
|
||||
* are picked up; uncategorized dynamic entries land in "Other (API)".
|
||||
* @returns {Array<{value: string, label: string, category: string}>}
|
||||
*/
|
||||
function buildCategorizedOptions() {
|
||||
const allModels = [];
|
||||
const categorizedModels = new Set();
|
||||
|
||||
Object.entries(BASE_MODEL_CATEGORIES).forEach(([category, models]) => {
|
||||
models.forEach(model => {
|
||||
allModels.push({ value: model, label: model, category });
|
||||
categorizedModels.add(model);
|
||||
});
|
||||
});
|
||||
|
||||
const uncategorizedModels = getMergedBaseModels().filter(model => !categorizedModels.has(model));
|
||||
uncategorizedModels.forEach(model => {
|
||||
allModels.push({ value: model, label: model, category: 'Other (API)' });
|
||||
});
|
||||
|
||||
return allModels;
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a searchable base model picker.
|
||||
*
|
||||
* Two commit semantics are supported:
|
||||
* - 'commit' (default): selecting an item immediately calls onCommit(value).
|
||||
* Escape or an outside click calls onDismiss().
|
||||
* - 'change': selecting an item updates the internal value and calls
|
||||
* onChange(value); the caller owns when the selected value is persisted.
|
||||
* Typed text doubles as a custom value unless allowCustomValue is false,
|
||||
* in which case it is search-only.
|
||||
*
|
||||
* @param {Object} options
|
||||
* @param {string[]} [options.suggestions] - Models shown in the Suggested section
|
||||
* @param {string} [options.initialValue] - Initially selected value
|
||||
* @param {'commit'|'change'} [options.mode] - Commit semantics
|
||||
* @param {boolean} [options.allowCustomValue=true] - Accept typed text as a custom value
|
||||
* @param {(value: string) => void} [options.onCommit] - Commit-mode commit callback
|
||||
* @param {(value: string) => void} [options.onChange] - Called whenever the value changes
|
||||
* @param {() => void} [options.onDismiss] - Commit-mode dismiss callback (Escape/outside click)
|
||||
* @returns {{ element: HTMLElement, getValue: Function, setValue: Function, refreshOptions: Function, destroy: Function }}
|
||||
*/
|
||||
export function createBaseModelPicker(options = {}) {
|
||||
const {
|
||||
suggestions = [],
|
||||
initialValue = '',
|
||||
mode = 'commit',
|
||||
allowCustomValue = true,
|
||||
onCommit = null,
|
||||
onChange = null,
|
||||
onDismiss = null,
|
||||
} = options;
|
||||
|
||||
const isCommitMode = mode !== 'change';
|
||||
let currentValue = initialValue || '';
|
||||
let currentFilter = '';
|
||||
let destroyed = false;
|
||||
|
||||
// ── Build widget DOM ────────────────────────────────────────────────────
|
||||
const wrapper = document.createElement('div');
|
||||
wrapper.className = 'base-model-search-wrapper';
|
||||
|
||||
const inputWrapper = document.createElement('div');
|
||||
inputWrapper.className = 'base-model-search-input-wrapper';
|
||||
const searchIcon = document.createElement('i');
|
||||
searchIcon.className = 'fas fa-search search-icon';
|
||||
searchIcon.setAttribute('aria-hidden', 'true');
|
||||
inputWrapper.appendChild(searchIcon);
|
||||
const searchInput = document.createElement('input');
|
||||
searchInput.type = 'text';
|
||||
searchInput.className = 'base-model-search-input';
|
||||
searchInput.placeholder = translate('modals.model.metadata.baseModelSearchPlaceholder', {}, 'Search base model…');
|
||||
searchInput.autocomplete = 'off';
|
||||
searchInput.spellcheck = false;
|
||||
inputWrapper.appendChild(searchInput);
|
||||
wrapper.appendChild(inputWrapper);
|
||||
|
||||
const dropdown = document.createElement('div');
|
||||
dropdown.className = 'base-model-dropdown';
|
||||
wrapper.appendChild(dropdown);
|
||||
|
||||
// ── Render ──────────────────────────────────────────────────────────────
|
||||
function renderDropdown(filterText) {
|
||||
currentFilter = filterText || '';
|
||||
const lowerFilter = currentFilter.toLowerCase().trim();
|
||||
const allModels = buildCategorizedOptions();
|
||||
const suggestedSet = new Set(suggestions);
|
||||
dropdown.innerHTML = '';
|
||||
let hasVisibleItems = false;
|
||||
const fragment = document.createDocumentFragment();
|
||||
|
||||
// 1. Suggested section (filtered by search)
|
||||
const suggestedToShow = lowerFilter
|
||||
? suggestions.filter(m => m.toLowerCase().includes(lowerFilter))
|
||||
: suggestions;
|
||||
|
||||
if (suggestedToShow.length > 0) {
|
||||
const section = document.createElement('div');
|
||||
section.className = 'base-model-dropdown-section';
|
||||
|
||||
const header = document.createElement('div');
|
||||
header.className = 'base-model-dropdown-header suggested-header';
|
||||
header.innerHTML = '<i class="fas fa-star" aria-hidden="true"></i> ' +
|
||||
translate('modals.model.metadata.baseModelSuggested', {}, 'Suggested');
|
||||
section.appendChild(header);
|
||||
|
||||
suggestedToShow.forEach(model => {
|
||||
const item = document.createElement('div');
|
||||
item.className = 'base-model-dropdown-item';
|
||||
if (model === currentValue) item.classList.add('selected');
|
||||
item.dataset.value = model;
|
||||
item.textContent = model;
|
||||
section.appendChild(item);
|
||||
hasVisibleItems = true;
|
||||
});
|
||||
|
||||
fragment.appendChild(section);
|
||||
}
|
||||
|
||||
// 2. Categorized options (deduplicated against suggestions)
|
||||
const categoryMap = {};
|
||||
allModels.forEach(m => {
|
||||
if (suggestedSet.has(m.value)) return; // already shown in Suggested
|
||||
if (lowerFilter && !m.label.toLowerCase().includes(lowerFilter)) return;
|
||||
if (!categoryMap[m.category]) categoryMap[m.category] = [];
|
||||
categoryMap[m.category].push(m);
|
||||
});
|
||||
|
||||
Object.entries(categoryMap).forEach(([category, items]) => {
|
||||
if (items.length === 0) return;
|
||||
const section = document.createElement('div');
|
||||
section.className = 'base-model-dropdown-section';
|
||||
|
||||
const header = document.createElement('div');
|
||||
header.className = 'base-model-dropdown-header';
|
||||
header.textContent = category;
|
||||
section.appendChild(header);
|
||||
|
||||
items.forEach(m => {
|
||||
const item = document.createElement('div');
|
||||
item.className = 'base-model-dropdown-item';
|
||||
if (m.value === currentValue) item.classList.add('selected');
|
||||
item.dataset.value = m.value;
|
||||
item.textContent = m.label;
|
||||
section.appendChild(item);
|
||||
hasVisibleItems = true;
|
||||
});
|
||||
|
||||
fragment.appendChild(section);
|
||||
});
|
||||
|
||||
// 3. Empty state
|
||||
if (!hasVisibleItems) {
|
||||
const empty = document.createElement('div');
|
||||
empty.className = 'base-model-dropdown-empty';
|
||||
empty.textContent = translate('modals.model.metadata.baseModelNoMatch', {}, 'No matching base models');
|
||||
fragment.appendChild(empty);
|
||||
}
|
||||
|
||||
dropdown.appendChild(fragment);
|
||||
|
||||
// Scroll the selected item into view
|
||||
const selected = dropdown.querySelector('.base-model-dropdown-item.selected');
|
||||
if (selected) {
|
||||
selected.scrollIntoView({ block: 'nearest' });
|
||||
}
|
||||
}
|
||||
|
||||
// Initial render — show everything
|
||||
renderDropdown('');
|
||||
|
||||
// ── Value handling ──────────────────────────────────────────────────────
|
||||
function applySelection(value) {
|
||||
currentValue = value;
|
||||
if (isCommitMode) {
|
||||
if (typeof onCommit === 'function') onCommit(value);
|
||||
return;
|
||||
}
|
||||
// Change mode: mirror the selection into the input and notify only.
|
||||
searchInput.value = value;
|
||||
// Filter the list down to the selected item instead of resetting to
|
||||
// the full list (which scroll-jumps to the selection). Custom values
|
||||
// that are not in the option list keep the full list visible.
|
||||
const isKnownValue = suggestions.includes(value) ||
|
||||
buildCategorizedOptions().some(m => m.value === value);
|
||||
renderDropdown(isKnownValue ? value : '');
|
||||
if (typeof onChange === 'function') onChange(value);
|
||||
}
|
||||
|
||||
// ── Events ──────────────────────────────────────────────────────────────
|
||||
let filterTimeout;
|
||||
searchInput.addEventListener('input', () => {
|
||||
clearTimeout(filterTimeout);
|
||||
filterTimeout = setTimeout(() => {
|
||||
renderDropdown(searchInput.value);
|
||||
// Change mode with custom values: typed text is the live value.
|
||||
if (!isCommitMode && allowCustomValue) {
|
||||
currentValue = searchInput.value;
|
||||
if (typeof onChange === 'function') onChange(currentValue);
|
||||
}
|
||||
}, 50);
|
||||
});
|
||||
|
||||
// Click to select
|
||||
dropdown.addEventListener('click', (e) => {
|
||||
const item = e.target.closest('.base-model-dropdown-item');
|
||||
if (!item) return;
|
||||
applySelection(item.dataset.value);
|
||||
});
|
||||
|
||||
// Keyboard navigation
|
||||
searchInput.addEventListener('keydown', (e) => {
|
||||
const items = Array.from(dropdown.querySelectorAll('.base-model-dropdown-item'));
|
||||
const activeIdx = items.findIndex(el => el.classList.contains('active'));
|
||||
|
||||
if (e.key === 'ArrowDown') {
|
||||
e.preventDefault();
|
||||
items.forEach(el => el.classList.remove('active'));
|
||||
const next = Math.min(activeIdx + 1, items.length - 1);
|
||||
if (items[next]) {
|
||||
items[next].classList.add('active');
|
||||
items[next].scrollIntoView({ block: 'nearest' });
|
||||
}
|
||||
} else if (e.key === 'ArrowUp') {
|
||||
e.preventDefault();
|
||||
items.forEach(el => el.classList.remove('active'));
|
||||
const prev = Math.max(activeIdx - 1, 0);
|
||||
if (items[prev]) {
|
||||
items[prev].classList.add('active');
|
||||
items[prev].scrollIntoView({ block: 'nearest' });
|
||||
}
|
||||
} else if (e.key === 'Enter') {
|
||||
e.preventDefault();
|
||||
const activeItem = items.find(el => el.classList.contains('active'));
|
||||
if (activeItem) {
|
||||
applySelection(activeItem.dataset.value);
|
||||
} else if (allowCustomValue && searchInput.value.trim()) {
|
||||
applySelection(searchInput.value.trim());
|
||||
}
|
||||
} else if (e.key === 'Escape') {
|
||||
e.preventDefault();
|
||||
if (isCommitMode && typeof onDismiss === 'function') {
|
||||
onDismiss();
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
// Commit mode: outside click commits typed text (when custom values are
|
||||
// allowed) or dismisses. Deferred to avoid the opening click itself.
|
||||
const outsideClickHandler = (e) => {
|
||||
if (wrapper.contains(e.target)) return;
|
||||
const typedValue = searchInput.value.trim();
|
||||
if (allowCustomValue && typedValue) {
|
||||
applySelection(typedValue);
|
||||
} else if (typeof onDismiss === 'function') {
|
||||
onDismiss();
|
||||
}
|
||||
};
|
||||
let outsideClickTimer = null;
|
||||
if (isCommitMode) {
|
||||
outsideClickTimer = setTimeout(() => {
|
||||
outsideClickTimer = null;
|
||||
if (!destroyed) {
|
||||
document.addEventListener('click', outsideClickHandler);
|
||||
}
|
||||
}, 0);
|
||||
}
|
||||
|
||||
// Refresh when dynamic base models arrive late; keeps the current search text.
|
||||
const handleBaseModelsUpdated = () => {
|
||||
if (destroyed) return;
|
||||
refreshOptions();
|
||||
};
|
||||
window.addEventListener(BASE_MODELS_UPDATED_EVENT, handleBaseModelsUpdated);
|
||||
|
||||
// ── Public API ──────────────────────────────────────────────────────────
|
||||
function getValue() {
|
||||
return currentValue;
|
||||
}
|
||||
|
||||
function setValue(value) {
|
||||
currentValue = value || '';
|
||||
searchInput.value = currentValue;
|
||||
renderDropdown('');
|
||||
}
|
||||
|
||||
function refreshOptions() {
|
||||
renderDropdown(currentFilter);
|
||||
}
|
||||
|
||||
function destroy() {
|
||||
if (destroyed) return;
|
||||
destroyed = true;
|
||||
clearTimeout(filterTimeout);
|
||||
if (outsideClickTimer) {
|
||||
clearTimeout(outsideClickTimer);
|
||||
outsideClickTimer = null;
|
||||
}
|
||||
document.removeEventListener('click', outsideClickHandler);
|
||||
window.removeEventListener(BASE_MODELS_UPDATED_EVENT, handleBaseModelsUpdated);
|
||||
}
|
||||
|
||||
return { element: wrapper, getValue, setValue, refreshOptions, destroy };
|
||||
}
|
||||
@@ -3,75 +3,9 @@
|
||||
* Handles model metadata editing functionality - General version
|
||||
*/
|
||||
|
||||
import { BASE_MODEL_CATEGORIES, getMergedBaseModels } from '../../utils/constants.js';
|
||||
import { showToast } from '../../utils/uiHelpers.js';
|
||||
import { getModelApiClient } from '../../api/modelApiFactory.js';
|
||||
import { translate } from '../../utils/i18nHelpers.js';
|
||||
|
||||
// ── Filename-based base model inference ──────────────────────────────────────
|
||||
// Rules are ordered by specificity — first match wins for dedup.
|
||||
// Each rule checks the filename (lowercased) for a regex pattern and suggests
|
||||
// the associated base model values.
|
||||
|
||||
const BASE_MODEL_FILENAME_RULES = [
|
||||
{ pattern: /flux\.?\s*2\s*klein/i, models: ['Flux.2 Klein 9B', 'Flux.2 Klein 9B-base', 'Flux.2 Klein 4B', 'Flux.2 Klein 4B-base'] },
|
||||
{ pattern: /flux\.?\s*2/i, models: ['Flux.2 D', 'Flux.2 Klein 9B', 'Flux.2 Klein 4B'] },
|
||||
{ pattern: /flux\.?\s*1\s*(dev|d)\b/i, models: ['Flux.1 D'] },
|
||||
{ pattern: /flux\.?\s*1\s*(schnell|s)\b/i, models: ['Flux.1 S'] },
|
||||
{ pattern: /flux/i, models: ['Flux.1 D', 'Flux.1 S', 'Flux.2 D'] },
|
||||
{ pattern: /sdxl/i, models: ['SDXL 1.0', 'SDXL Lightning', 'SDXL Hyper'] },
|
||||
{ pattern: /sd\s*1[._-\s]?5/i, models: ['SD 1.5'] },
|
||||
{ pattern: /sd\s*1[._-\s]?4/i, models: ['SD 1.4'] },
|
||||
{ pattern: /sd\s*1/i, models: ['SD 1.5', 'SD 1.4', 'SD 1.5 LCM', 'SD 1.5 Hyper'] },
|
||||
{ pattern: /sd\s*3[._-\s]?5/i, models: ['SD 3.5', 'SD 3.5 Medium', 'SD 3.5 Large', 'SD 3.5 Large Turbo'] },
|
||||
{ pattern: /sd\s*3/i, models: ['SD 3', 'SD 3.5'] },
|
||||
{ pattern: /wan\s*\.?\s*video/i, models: ['Wan Video', 'Wan Video 1.3B t2v', 'Wan Video 14B t2v', 'Wan Video 14B i2v 480p', 'Wan Video 14B i2v 720p'] },
|
||||
{ pattern: /hunyuan\s*\.?\s*video/i, models: ['Hunyuan Video'] },
|
||||
{ pattern: /ltxv/i, models: ['LTXV', 'LTXV2', 'LTXV 2.3'] },
|
||||
{ pattern: /cogvideo/i, models: ['CogVideoX'] },
|
||||
{ pattern: /pony/i, models: ['Pony', 'Pony V7'] },
|
||||
{ pattern: /illustrious/i, models: ['Illustrious'] },
|
||||
{ pattern: /noobai/i, models: ['NoobAI'] },
|
||||
{ pattern: /pixart/i, models: ['PixArt a', 'PixArt E'] },
|
||||
{ pattern: /aura\s*\.?\s*flow/i, models: ['AuraFlow'] },
|
||||
{ pattern: /kolors/i, models: ['Kolors'] },
|
||||
{ pattern: /hunyuan\s*1/i, models: ['Hunyuan 1'] },
|
||||
{ pattern: /lumina/i, models: ['Lumina'] },
|
||||
{ pattern: /hidream/i, models: ['HiDream'] },
|
||||
{ pattern: /qwen/i, models: ['Qwen'] },
|
||||
{ pattern: /chroma/i, models: ['Chroma'] },
|
||||
{ pattern: /anima/i, models: ['Anima'] },
|
||||
{ pattern: /sd\s*2[._-\s]?[01]/i, models: ['SD 2.0', 'SD 2.1'] },
|
||||
{ pattern: /mochi/i, models: ['Mochi'] },
|
||||
{ pattern: /svd/i, models: ['SVD'] },
|
||||
{ pattern: /zimage/i, models: ['ZImageTurbo', 'ZImageBase'] },
|
||||
{ pattern: /nucleus/i, models: ['Nucleus'] },
|
||||
{ pattern: /krea/i, models: ['Flux.1 Krea', 'Krea 2'] },
|
||||
{ pattern: /ernie/i, models: ['Ernie', 'Ernie Turbo'] },
|
||||
];
|
||||
|
||||
/**
|
||||
* Infer likely base model(s) from a filename + model name string.
|
||||
* Returns a deduplicated array in match-priority order.
|
||||
* @param {string} filename
|
||||
* @returns {string[]}
|
||||
*/
|
||||
function inferBaseModelsFromFilename(filename) {
|
||||
if (!filename || typeof filename !== 'string') return [];
|
||||
const seen = new Set();
|
||||
const results = [];
|
||||
for (const rule of BASE_MODEL_FILENAME_RULES) {
|
||||
if (rule.pattern.test(filename)) {
|
||||
for (const model of rule.models) {
|
||||
if (!seen.has(model)) {
|
||||
seen.add(model);
|
||||
results.push(model);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return results;
|
||||
}
|
||||
import { inferBaseModelsFromFilename, createBaseModelPicker } from './BaseModelPicker.js';
|
||||
|
||||
/**
|
||||
* Resolve the active file path for the currently open model modal.
|
||||
@@ -321,199 +255,16 @@ export function setupBaseModelEditing(filePath) {
|
||||
// Handle edit button click
|
||||
editBtn.addEventListener('click', () => {
|
||||
baseModelDisplay.classList.add('editing');
|
||||
|
||||
|
||||
// Store the original value to check for changes later
|
||||
const originalValue = baseModelContent.textContent.trim();
|
||||
|
||||
// ── Build the full option list ────────────────────────────────────────
|
||||
const allModels = []; // { value, label, category }
|
||||
const categorizedModels = new Set();
|
||||
|
||||
Object.entries(BASE_MODEL_CATEGORIES).forEach(([category, models]) => {
|
||||
models.forEach(model => {
|
||||
allModels.push({ value: model, label: model, category });
|
||||
categorizedModels.add(model);
|
||||
});
|
||||
});
|
||||
|
||||
const mergedModels = getMergedBaseModels();
|
||||
const uncategorizedModels = mergedModels.filter(model => !categorizedModels.has(model));
|
||||
if (uncategorizedModels.length > 0) {
|
||||
uncategorizedModels.forEach(model => {
|
||||
allModels.push({ value: model, label: model, category: 'Other (API)' });
|
||||
});
|
||||
}
|
||||
|
||||
// ── Filename-based inference ──────────────────────────────────────────
|
||||
|
||||
// Filename-based inference for the Suggested section
|
||||
const fileName = (document.querySelector('.file-name-content')?.textContent || '') + ' ' +
|
||||
(document.querySelector('.model-name-content')?.textContent || '');
|
||||
const inferredModels = inferBaseModelsFromFilename(fileName);
|
||||
const inferredSet = new Set(inferredModels);
|
||||
|
||||
// ── Build search widget DOM ───────────────────────────────────────────
|
||||
const wrapper = document.createElement('div');
|
||||
wrapper.className = 'base-model-search-wrapper';
|
||||
|
||||
// Search input row
|
||||
const inputWrapper = document.createElement('div');
|
||||
inputWrapper.className = 'base-model-search-input-wrapper';
|
||||
const searchIcon = document.createElement('i');
|
||||
searchIcon.className = 'fas fa-search search-icon';
|
||||
searchIcon.setAttribute('aria-hidden', 'true');
|
||||
inputWrapper.appendChild(searchIcon);
|
||||
const searchInput = document.createElement('input');
|
||||
searchInput.type = 'text';
|
||||
searchInput.className = 'base-model-search-input';
|
||||
searchInput.placeholder = translate('modals.model.metadata.baseModelSearchPlaceholder', {}, 'Search base model…');
|
||||
searchInput.autocomplete = 'off';
|
||||
searchInput.spellcheck = false;
|
||||
inputWrapper.appendChild(searchInput);
|
||||
wrapper.appendChild(inputWrapper);
|
||||
|
||||
// Dropdown list
|
||||
const dropdown = document.createElement('div');
|
||||
dropdown.className = 'base-model-dropdown';
|
||||
wrapper.appendChild(dropdown);
|
||||
|
||||
// ── Render ────────────────────────────────────────────────────────────
|
||||
function renderDropdown(filterText) {
|
||||
const lowerFilter = (filterText || '').toLowerCase().trim();
|
||||
dropdown.innerHTML = '';
|
||||
let hasVisibleItems = false;
|
||||
const fragment = document.createDocumentFragment();
|
||||
|
||||
// 1. Suggested section (filename-inferred, filtered by search)
|
||||
let suggestedToShow = inferredModels;
|
||||
if (lowerFilter) {
|
||||
suggestedToShow = inferredModels.filter(m =>
|
||||
m.toLowerCase().includes(lowerFilter)
|
||||
);
|
||||
}
|
||||
|
||||
if (suggestedToShow.length > 0) {
|
||||
const section = document.createElement('div');
|
||||
section.className = 'base-model-dropdown-section';
|
||||
|
||||
const header = document.createElement('div');
|
||||
header.className = 'base-model-dropdown-header suggested-header';
|
||||
header.innerHTML = '<i class="fas fa-star" aria-hidden="true"></i> ' +
|
||||
translate('modals.model.metadata.baseModelSuggested', {}, 'Suggested');
|
||||
section.appendChild(header);
|
||||
|
||||
suggestedToShow.forEach(model => {
|
||||
const item = document.createElement('div');
|
||||
item.className = 'base-model-dropdown-item';
|
||||
if (model === originalValue) item.classList.add('selected');
|
||||
item.dataset.value = model;
|
||||
item.textContent = model;
|
||||
section.appendChild(item);
|
||||
hasVisibleItems = true;
|
||||
});
|
||||
|
||||
fragment.appendChild(section);
|
||||
}
|
||||
|
||||
// 2. Categorized options (deduplicated against suggestions)
|
||||
const categoryMap = {};
|
||||
allModels.forEach(m => {
|
||||
if (inferredSet.has(m.value)) return; // already shown in Suggested
|
||||
if (lowerFilter && !m.label.toLowerCase().includes(lowerFilter)) return;
|
||||
if (!categoryMap[m.category]) categoryMap[m.category] = [];
|
||||
categoryMap[m.category].push(m);
|
||||
});
|
||||
|
||||
Object.entries(categoryMap).forEach(([category, items]) => {
|
||||
if (items.length === 0) return;
|
||||
const section = document.createElement('div');
|
||||
section.className = 'base-model-dropdown-section';
|
||||
|
||||
const header = document.createElement('div');
|
||||
header.className = 'base-model-dropdown-header';
|
||||
header.textContent = category;
|
||||
section.appendChild(header);
|
||||
|
||||
items.forEach(m => {
|
||||
const item = document.createElement('div');
|
||||
item.className = 'base-model-dropdown-item';
|
||||
if (m.value === originalValue) item.classList.add('selected');
|
||||
item.dataset.value = m.value;
|
||||
item.textContent = m.label;
|
||||
section.appendChild(item);
|
||||
hasVisibleItems = true;
|
||||
});
|
||||
|
||||
fragment.appendChild(section);
|
||||
});
|
||||
|
||||
// 3. Empty state
|
||||
if (!hasVisibleItems) {
|
||||
const empty = document.createElement('div');
|
||||
empty.className = 'base-model-dropdown-empty';
|
||||
empty.textContent = translate('modals.model.metadata.baseModelNoMatch', {}, 'No matching base models');
|
||||
fragment.appendChild(empty);
|
||||
}
|
||||
|
||||
dropdown.appendChild(fragment);
|
||||
|
||||
// Scroll the selected item into view
|
||||
const selected = dropdown.querySelector('.base-model-dropdown-item.selected');
|
||||
if (selected) {
|
||||
selected.scrollIntoView({ block: 'nearest' });
|
||||
}
|
||||
}
|
||||
|
||||
// Initial render — show everything
|
||||
renderDropdown('');
|
||||
|
||||
// ── Events ────────────────────────────────────────────────────────────
|
||||
let filterTimeout;
|
||||
searchInput.addEventListener('input', () => {
|
||||
clearTimeout(filterTimeout);
|
||||
filterTimeout = setTimeout(() => renderDropdown(searchInput.value), 50);
|
||||
});
|
||||
|
||||
// Click to select
|
||||
dropdown.addEventListener('click', (e) => {
|
||||
const item = e.target.closest('.base-model-dropdown-item');
|
||||
if (!item) return;
|
||||
baseModelContent.textContent = item.dataset.value;
|
||||
cleanup();
|
||||
const finalValue = baseModelContent.textContent.trim();
|
||||
if (finalValue !== originalValue) {
|
||||
saveBaseModel(
|
||||
getActiveModalFilePath(baseModelContent.dataset.filePath),
|
||||
originalValue
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
// Replace content with search widget
|
||||
baseModelContent.style.display = 'none';
|
||||
editBtn.style.display = 'none';
|
||||
baseModelDisplay.insertBefore(wrapper, editBtn);
|
||||
searchInput.focus();
|
||||
|
||||
// ── Cleanup ───────────────────────────────────────────────────────────
|
||||
function cleanup() {
|
||||
if (wrapper.parentNode === baseModelDisplay) {
|
||||
baseModelDisplay.removeChild(wrapper);
|
||||
}
|
||||
baseModelContent.style.display = '';
|
||||
editBtn.style.display = '';
|
||||
baseModelDisplay.classList.remove('editing');
|
||||
document.removeEventListener('click', outsideClickHandler);
|
||||
}
|
||||
|
||||
// Outside click → save typed/custom value if any
|
||||
const outsideClickHandler = function(e) {
|
||||
if (wrapper.contains(e.target)) return;
|
||||
|
||||
// If user typed a custom value (not just empty), apply it
|
||||
const typedValue = searchInput.value.trim();
|
||||
if (typedValue) {
|
||||
baseModelContent.textContent = typedValue;
|
||||
}
|
||||
cleanup();
|
||||
|
||||
const saveIfChanged = () => {
|
||||
const finalValue = baseModelContent.textContent.trim();
|
||||
if (finalValue !== originalValue) {
|
||||
saveBaseModel(
|
||||
@@ -522,56 +273,39 @@ export function setupBaseModelEditing(filePath) {
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
// Defer listener to avoid the opening click itself
|
||||
setTimeout(() => {
|
||||
document.addEventListener('click', outsideClickHandler);
|
||||
}, 0);
|
||||
|
||||
// Keyboard navigation
|
||||
searchInput.addEventListener('keydown', function onKeydown(e) {
|
||||
const items = Array.from(dropdown.querySelectorAll('.base-model-dropdown-item'));
|
||||
const activeIdx = items.findIndex(el => el.classList.contains('active'));
|
||||
|
||||
if (e.key === 'ArrowDown') {
|
||||
e.preventDefault();
|
||||
items.forEach(el => el.classList.remove('active'));
|
||||
const next = Math.min(activeIdx + 1, items.length - 1);
|
||||
if (items[next]) {
|
||||
items[next].classList.add('active');
|
||||
items[next].scrollIntoView({ block: 'nearest' });
|
||||
}
|
||||
} else if (e.key === 'ArrowUp') {
|
||||
e.preventDefault();
|
||||
items.forEach(el => el.classList.remove('active'));
|
||||
const prev = Math.max(activeIdx - 1, 0);
|
||||
if (items[prev]) {
|
||||
items[prev].classList.add('active');
|
||||
items[prev].scrollIntoView({ block: 'nearest' });
|
||||
}
|
||||
} else if (e.key === 'Enter') {
|
||||
e.preventDefault();
|
||||
const activeItem = items.find(el => el.classList.contains('active'));
|
||||
if (activeItem) {
|
||||
activeItem.click();
|
||||
} else if (searchInput.value.trim()) {
|
||||
// Custom value typed
|
||||
baseModelContent.textContent = searchInput.value.trim();
|
||||
cleanup();
|
||||
const finalValue = baseModelContent.textContent.trim();
|
||||
if (finalValue !== originalValue) {
|
||||
saveBaseModel(
|
||||
getActiveModalFilePath(baseModelContent.dataset.filePath),
|
||||
originalValue
|
||||
);
|
||||
}
|
||||
}
|
||||
} else if (e.key === 'Escape') {
|
||||
e.preventDefault();
|
||||
|
||||
const picker = createBaseModelPicker({
|
||||
suggestions: inferredModels,
|
||||
initialValue: originalValue,
|
||||
mode: 'commit',
|
||||
onCommit: (value) => {
|
||||
baseModelContent.textContent = value;
|
||||
cleanup();
|
||||
saveIfChanged();
|
||||
},
|
||||
onDismiss: () => {
|
||||
// Escape or empty outside click: restore the original value
|
||||
baseModelContent.textContent = originalValue;
|
||||
cleanup();
|
||||
}
|
||||
},
|
||||
});
|
||||
|
||||
function cleanup() {
|
||||
picker.destroy();
|
||||
if (picker.element.parentNode === baseModelDisplay) {
|
||||
baseModelDisplay.removeChild(picker.element);
|
||||
}
|
||||
baseModelContent.style.display = '';
|
||||
editBtn.style.display = '';
|
||||
baseModelDisplay.classList.remove('editing');
|
||||
}
|
||||
|
||||
// Replace content with search widget
|
||||
baseModelContent.style.display = 'none';
|
||||
editBtn.style.display = 'none';
|
||||
baseModelDisplay.insertBefore(picker.element, editBtn);
|
||||
const searchInput = picker.element.querySelector('.base-model-search-input');
|
||||
if (searchInput) searchInput.focus();
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -1426,6 +1426,32 @@ export function initVersionsTab({
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* True when the downloaded version is the newest version in the model's
|
||||
* remote version set, i.e. the one whose install flips the backend
|
||||
* update-available flag off. Unknown version sets fall back to "latest"
|
||||
* so the post-download in-place reconciliation still runs by default.
|
||||
* (#1078)
|
||||
*/
|
||||
function versionIsLatestAvailable(version) {
|
||||
if (!controller.record || !Array.isArray(controller.record.versions)) {
|
||||
return true;
|
||||
}
|
||||
const versions = controller.record.versions;
|
||||
if (versions.length === 0) {
|
||||
return true;
|
||||
}
|
||||
const target = Number(version?.versionId);
|
||||
if (!Number.isFinite(target)) {
|
||||
return true;
|
||||
}
|
||||
const maxId = versions.reduce(
|
||||
(max, v) => Math.max(max, Number(v?.versionId) || 0),
|
||||
0
|
||||
);
|
||||
return target >= maxId;
|
||||
}
|
||||
|
||||
async function handleDownloadVersion(button, versionId) {
|
||||
if (!controller.record) {
|
||||
return;
|
||||
@@ -1451,6 +1477,7 @@ export function initVersionsTab({
|
||||
targetFolder: resolveTemplatePath ? '' : (pathInfo?.targetFolder || ''),
|
||||
useDefaultPaths: resolveTemplatePath ? true : null,
|
||||
useSaveDirAsRoot: resolveTemplatePath,
|
||||
isLatestVersion: versionIsLatestAvailable(version),
|
||||
});
|
||||
|
||||
if (success) {
|
||||
|
||||
+2
-1
@@ -20,7 +20,7 @@ import { BulkContextMenu } from './components/ContextMenu/BulkContextMenu.js';
|
||||
import { createPageContextMenu, createGlobalContextMenu } from './components/ContextMenu/index.js';
|
||||
import { initializeEventManagement } from './utils/eventManagementInit.js';
|
||||
import { civitaiBaseModelApi } from './api/civitaiBaseModelApi.js';
|
||||
import { setDynamicBaseModels } from './utils/constants.js';
|
||||
import { setDynamicBaseModels, BASE_MODELS_UPDATED_EVENT } from './utils/constants.js';
|
||||
|
||||
// Core application class
|
||||
export class AppCore {
|
||||
@@ -134,6 +134,7 @@ export class AppCore {
|
||||
const result = await civitaiBaseModelApi.getBaseModels();
|
||||
if (result && result.models) {
|
||||
setDynamicBaseModels(result.models, result.last_updated);
|
||||
window.dispatchEvent(new CustomEvent(BASE_MODELS_UPDATED_EVENT));
|
||||
console.log(`AppCore: Loaded ${result.merged_count} base models (${result.hardcoded_count} hardcoded + ${result.remote_count} remote)`);
|
||||
}
|
||||
} catch (error) {
|
||||
|
||||
@@ -427,6 +427,12 @@ export class BatchImportManager {
|
||||
this.progress = progress;
|
||||
this.updateProgressUI(progress);
|
||||
|
||||
// Surface vendor rate limiting once per import (#1085): requests are
|
||||
// being paced and some items may be skipped rather than failed.
|
||||
if (progress.rate_limited && !(prev && prev.rate_limited)) {
|
||||
showToast('toast.recipes.batchImportRateLimited', {}, 'warning');
|
||||
}
|
||||
|
||||
// Only log when something actually changed (and on the first update),
|
||||
// so per-second polling does not spam the console with identical lines.
|
||||
const changed =
|
||||
@@ -495,7 +501,9 @@ export class BatchImportManager {
|
||||
const statusText = document.getElementById('batchStatusText');
|
||||
if (statusText) {
|
||||
if (progress.status === 'running') {
|
||||
statusText.textContent = translate('recipes.batchImport.importing', {}, 'Importing...');
|
||||
statusText.textContent = progress.rate_limited
|
||||
? translate('recipes.batchImport.rateLimitedSlowdown', {}, 'Rate limited — slowing down...')
|
||||
: translate('recipes.batchImport.importing', {}, 'Importing...');
|
||||
} else if (progress.status === 'completed') {
|
||||
statusText.textContent = translate('recipes.batchImport.completed', {}, 'Import completed');
|
||||
} else if (progress.status === 'cancelled') {
|
||||
|
||||
@@ -6,7 +6,7 @@ import { modalManager } from './ModalManager.js';
|
||||
import { getModelApiClient, resetAndReload } from '../api/modelApiFactory.js';
|
||||
import { RecipeSidebarApiClient, updateRecipeMetadata, extractRecipeId } from '../api/recipeApi.js';
|
||||
import { MODEL_TYPES, MODEL_CONFIG } from '../api/apiConfig.js';
|
||||
import { BASE_MODEL_CATEGORIES } from '../utils/constants.js';
|
||||
import { createBaseModelPicker, inferBaseModelsFromFilepaths } from '../components/shared/BaseModelPicker.js';
|
||||
import { getPriorityTagSuggestions } from '../utils/priorityTagHelpers.js';
|
||||
import { eventManager } from '../utils/EventManager.js';
|
||||
import { translate } from '../utils/i18nHelpers.js';
|
||||
@@ -30,6 +30,10 @@ export class BulkManager {
|
||||
// toggleCardSelection, cleared in clearSelection.
|
||||
this.bulkAnchorFilepath = null;
|
||||
|
||||
// Bulk base model picker state
|
||||
this.bulkBaseModelPicker = null;
|
||||
this.bulkBaseModelValue = '';
|
||||
|
||||
// Drag detection properties
|
||||
this.dragThreshold = 5; // Pixels to move before considering it a drag
|
||||
this.dragDelayMs = 100; // Minimum hold time before a drag is treated as a marquee
|
||||
@@ -1830,47 +1834,35 @@ export class BulkManager {
|
||||
* Initialize bulk base model interface
|
||||
*/
|
||||
initializeBulkBaseModelInterface() {
|
||||
const select = document.getElementById('bulkBaseModelSelect');
|
||||
if (!select) return;
|
||||
const container = document.getElementById('bulkBaseModelPicker');
|
||||
if (!container) return;
|
||||
|
||||
// Clear existing options
|
||||
select.innerHTML = '';
|
||||
// Reset any previous picker instance
|
||||
this.cleanupBulkBaseModelModal();
|
||||
container.innerHTML = '';
|
||||
|
||||
// Add placeholder option
|
||||
const placeholderOption = document.createElement('option');
|
||||
placeholderOption.value = '';
|
||||
placeholderOption.textContent = 'Select a base model...';
|
||||
placeholderOption.disabled = true;
|
||||
placeholderOption.selected = true;
|
||||
select.appendChild(placeholderOption);
|
||||
|
||||
// Create option groups for better organization
|
||||
Object.entries(BASE_MODEL_CATEGORIES).forEach(([category, models]) => {
|
||||
const optgroup = document.createElement('optgroup');
|
||||
optgroup.label = category;
|
||||
|
||||
models.forEach(model => {
|
||||
const option = document.createElement('option');
|
||||
option.value = model;
|
||||
option.textContent = model;
|
||||
optgroup.appendChild(option);
|
||||
});
|
||||
|
||||
select.appendChild(optgroup);
|
||||
const suggestions = inferBaseModelsFromFilepaths(Array.from(state.selectedModels));
|
||||
this.bulkBaseModelValue = '';
|
||||
this.bulkBaseModelPicker = createBaseModelPicker({
|
||||
suggestions,
|
||||
mode: 'change',
|
||||
onChange: (value) => {
|
||||
this.bulkBaseModelValue = value;
|
||||
},
|
||||
});
|
||||
container.appendChild(this.bulkBaseModelPicker.element);
|
||||
this.bulkBaseModelPicker.element.querySelector('.base-model-search-input')?.focus();
|
||||
}
|
||||
|
||||
/**
|
||||
* Save bulk base model changes
|
||||
*/
|
||||
async saveBulkBaseModel() {
|
||||
const select = document.getElementById('bulkBaseModelSelect');
|
||||
if (!select || !select.value) {
|
||||
const newBaseModel = (this.bulkBaseModelValue || this.bulkBaseModelPicker?.getValue() || '').trim();
|
||||
if (!newBaseModel) {
|
||||
showToast('toast.models.baseModelNotSelected', {}, 'warning');
|
||||
return;
|
||||
}
|
||||
|
||||
const newBaseModel = select.value;
|
||||
const selectedCount = state.selectedModels.size;
|
||||
|
||||
if (selectedCount === 0) {
|
||||
@@ -1938,9 +1930,14 @@ export class BulkManager {
|
||||
* Cleanup bulk base model modal
|
||||
*/
|
||||
cleanupBulkBaseModelModal() {
|
||||
const select = document.getElementById('bulkBaseModelSelect');
|
||||
if (select) {
|
||||
select.innerHTML = '';
|
||||
if (this.bulkBaseModelPicker) {
|
||||
this.bulkBaseModelPicker.destroy();
|
||||
this.bulkBaseModelPicker = null;
|
||||
}
|
||||
this.bulkBaseModelValue = '';
|
||||
const container = document.getElementById('bulkBaseModelPicker');
|
||||
if (container) {
|
||||
container.innerHTML = '';
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1077,6 +1077,7 @@ export class DownloadManager {
|
||||
deferReload = false,
|
||||
suppressSuccessToast = false,
|
||||
suppressFailureSummary = false,
|
||||
isLatestVersion = null,
|
||||
}) {
|
||||
const config = this.apiClient?.apiConfig?.config;
|
||||
|
||||
@@ -1085,7 +1086,7 @@ export class DownloadManager {
|
||||
}
|
||||
|
||||
const displayName = versionName || `#${versionId}`;
|
||||
const retryParams = { modelId, versionId, versionName, modelRoot, targetFolder, useDefaultPaths, useSaveDirAsRoot, source, fileParams, closeModal: false, deferReload, suppressSuccessToast, suppressFailureSummary };
|
||||
const retryParams = { modelId, versionId, versionName, modelRoot, targetFolder, useDefaultPaths, useSaveDirAsRoot, source, fileParams, closeModal: false, deferReload, suppressSuccessToast, suppressFailureSummary, isLatestVersion };
|
||||
this._lastDownloadError = null;
|
||||
let ws = null;
|
||||
let updateProgress = () => { };
|
||||
@@ -1228,22 +1229,16 @@ export class DownloadManager {
|
||||
}
|
||||
|
||||
if (!deferReload) {
|
||||
const pageState = this.apiClient.getPageState();
|
||||
|
||||
if (!useDefaultPaths && targetFolder) {
|
||||
pageState.activeFolder = targetFolder;
|
||||
setStorageItem(`${this.apiClient.modelType}_activeFolder`, targetFolder);
|
||||
|
||||
document.querySelectorAll('.folder-tags .tag').forEach(tag => {
|
||||
const isActive = tag.dataset.folder === targetFolder;
|
||||
tag.classList.toggle('active', isActive);
|
||||
if (isActive && !tag.parentNode.classList.contains('collapsed')) {
|
||||
tag.scrollIntoView({ behavior: 'smooth', block: 'nearest' });
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
await resetAndReload(true);
|
||||
// In-place view update instead of a full page reload: the
|
||||
// download only flips the update flag for one model, so we
|
||||
// reconcile its cards without resetting the listing, the
|
||||
// scroll position or the sidebar's active folder (#1078).
|
||||
// The legacy code hijacked `pageState.activeFolder` here
|
||||
// whenever a custom target folder was used.
|
||||
await this._reconcileViewAfterDownload({
|
||||
modelId,
|
||||
isLatestVersion: isLatestVersion ?? this._isDownloadingLatestVersion(versionId),
|
||||
});
|
||||
}
|
||||
|
||||
return true;
|
||||
@@ -1285,6 +1280,175 @@ export class DownloadManager {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reconcile the current model listing after a successful download,
|
||||
* without resetting the whole page (#1078).
|
||||
*
|
||||
* The legacy behaviour re-loaded page 1 and scrolled to the top after
|
||||
* every download, and hijacked the sidebar's active folder whenever a
|
||||
* custom target folder was used. In-place reconciliation only touches
|
||||
* the cards that can change as a result of the download:
|
||||
*
|
||||
* - Updates view: once the newest eligible version is installed the
|
||||
* model no longer qualifies, so its cards are removed from the list
|
||||
* (the update flag is model-level, so every visible card of the
|
||||
* model disappears at once).
|
||||
* - Normal listing: the card stays; only the update flag is cleared.
|
||||
* - The model is not in the current view (different folder / filter /
|
||||
* window): nothing changes, which also covers brand-new models whose
|
||||
* card did not exist before.
|
||||
*
|
||||
* The sidebar folder tree is refreshed separately so folder counts
|
||||
* stay accurate without touching the model listing or scroll position.
|
||||
*
|
||||
* @param {object} opts
|
||||
* @param {string|number} opts.modelId CivitAI model id of the downloaded model.
|
||||
* @param {boolean} [opts.isLatestVersion=true] True when the downloaded
|
||||
* version is the newest known remote version, so the update flag can
|
||||
* be cleared. When false (user deliberately picked an older version)
|
||||
* the list is left untouched.
|
||||
* @param {boolean} [opts.refreshSidebar=true] Whether to refresh the
|
||||
* sidebar folder tree afterwards (batch callers batch this into a
|
||||
* single refresh).
|
||||
* @returns {Promise<boolean>} True when an in-place update was applied.
|
||||
*/
|
||||
async _reconcileViewAfterDownload({ modelId, isLatestVersion = true, refreshSidebar = true } = {}) {
|
||||
const scroller = state?.virtualScroller;
|
||||
const items = Array.isArray(scroller?.items) ? scroller.items : [];
|
||||
|
||||
// No virtual scroller (page without one, not on a listing page,
|
||||
// recipes duplicates mode, ...) — fall back to the legacy reload.
|
||||
if (!scroller || items.length === 0 || typeof scroller.removeMultipleItemsByFilePath !== 'function') {
|
||||
await resetAndReload(true);
|
||||
return false;
|
||||
}
|
||||
|
||||
if (modelId == null) {
|
||||
// No CivitAI identity (e.g. HF downloads) — nothing to reconcile.
|
||||
await this._refreshSidebarAfterReconcile(refreshSidebar);
|
||||
return false;
|
||||
}
|
||||
|
||||
const key = String(modelId);
|
||||
const matches = items.filter(item => {
|
||||
const civitai = item?.civitai;
|
||||
return civitai != null && String(civitai.modelId) === key;
|
||||
});
|
||||
|
||||
if (matches.length === 0) {
|
||||
// Downloaded model is not visible in the current view — keep the
|
||||
// listing untouched, only refresh folder counts.
|
||||
await this._refreshSidebarAfterReconcile(refreshSidebar);
|
||||
return false;
|
||||
}
|
||||
|
||||
const pageState = this.apiClient?.getPageState ? this.apiClient.getPageState() : null;
|
||||
const updatesView = pageState?.showUpdateAvailableOnly === true;
|
||||
|
||||
if (updatesView && isLatestVersion) {
|
||||
const paths = matches.map(match => match.file_path).filter(Boolean);
|
||||
if (paths.length > 0) {
|
||||
scroller.removeMultipleItemsByFilePath(paths);
|
||||
}
|
||||
} else if (!updatesView && isLatestVersion) {
|
||||
for (const match of matches) {
|
||||
if (match.file_path) {
|
||||
scroller.updateSingleItem(match.file_path, { update_available: false });
|
||||
}
|
||||
}
|
||||
}
|
||||
// isLatestVersion === false: deliberately downloading an older
|
||||
// version keeps the update flag — nothing changes in the list.
|
||||
|
||||
await this._refreshSidebarAfterReconcile(refreshSidebar);
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reconcile the listing after a batch download. CivitAI models are
|
||||
* matched card-by-card via `_reconcileViewAfterDownload`; HF
|
||||
* downloads (no CivitAI identity to match) keep the legacy reload.
|
||||
*/
|
||||
async _reconcileBatchViewAfterDownload(completedCivitaiItems = [], hfCompletedCount = 0) {
|
||||
if (hfCompletedCount > 0) {
|
||||
await resetAndReload(true);
|
||||
return;
|
||||
}
|
||||
const scroller = state?.virtualScroller;
|
||||
if (!scroller || !Array.isArray(scroller.items)) {
|
||||
await resetAndReload(true);
|
||||
return;
|
||||
}
|
||||
const seen = new Set();
|
||||
for (const item of completedCivitaiItems) {
|
||||
const modelId = item?.modelId;
|
||||
if (modelId == null || seen.has(String(modelId))) {
|
||||
continue;
|
||||
}
|
||||
seen.add(String(modelId));
|
||||
await this._reconcileViewAfterDownload({
|
||||
modelId,
|
||||
isLatestVersion: this._isVersionLatest(item.selectedVersion?.id, item.versions),
|
||||
refreshSidebar: false,
|
||||
});
|
||||
}
|
||||
await this._refreshSidebarAfterReconcile(true);
|
||||
}
|
||||
|
||||
/**
|
||||
* Refresh the sidebar folder tree (counts only — never the model
|
||||
* listing). Lazy import keeps SidebarManager out of DownloadManager's
|
||||
* load graph (it transitively imports BulkManager and friends).
|
||||
*/
|
||||
async _refreshSidebarAfterReconcile(shouldRefresh) {
|
||||
if (shouldRefresh === false) {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const { sidebarManager } = await import('../components/SidebarManager.js');
|
||||
if (sidebarManager && typeof sidebarManager.refresh === 'function') {
|
||||
await sidebarManager.refresh();
|
||||
}
|
||||
} catch (error) {
|
||||
console.debug('Failed to refresh sidebar after download:', error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* True when `versionId` is the newest known remote version of the
|
||||
* versions list. Unknown/missing lists are treated as "latest" so the
|
||||
* common download-the-update flow reconciles by default; callers that
|
||||
* know the remote version set pass an explicit flag instead.
|
||||
*/
|
||||
_isVersionLatest(versionId, versions) {
|
||||
if (!Array.isArray(versions) || versions.length === 0) {
|
||||
return true;
|
||||
}
|
||||
let maxId = null;
|
||||
for (const version of versions) {
|
||||
const id = Number(version?.id ?? version?.versionId);
|
||||
if (!Number.isFinite(id)) {
|
||||
continue;
|
||||
}
|
||||
if (maxId === null || id > maxId) {
|
||||
maxId = id;
|
||||
}
|
||||
}
|
||||
if (maxId === null) {
|
||||
return true;
|
||||
}
|
||||
const target = Number(versionId);
|
||||
if (!Number.isFinite(target)) {
|
||||
return true;
|
||||
}
|
||||
return target >= maxId;
|
||||
}
|
||||
|
||||
/** True when the currently selected version is the newest remote one. */
|
||||
_isDownloadingLatestVersion(versionId) {
|
||||
return this._isVersionLatest(versionId, this.versions);
|
||||
}
|
||||
|
||||
/**
|
||||
* Download multiple selected files of the same version sequentially,
|
||||
* reusing the location-step choices for every file. Per-file toasts,
|
||||
@@ -1364,7 +1528,15 @@ export class DownloadManager {
|
||||
});
|
||||
}
|
||||
|
||||
await resetAndReload(true);
|
||||
// Full success: reconcile the model's cards in place. On partial
|
||||
// failure keep the listing untouched so the still-outdated version
|
||||
// flags survive until the user retries the remaining files.
|
||||
if (failedItems.length === 0) {
|
||||
await this._reconcileViewAfterDownload({
|
||||
modelId: this.modelId,
|
||||
isLatestVersion: this._isDownloadingLatestVersion(this.currentVersion?.id),
|
||||
});
|
||||
}
|
||||
return failedItems.length === 0;
|
||||
}
|
||||
|
||||
@@ -1961,6 +2133,11 @@ export class DownloadManager {
|
||||
let failedDownloads = 0;
|
||||
let cancelled = false;
|
||||
const failedItems = [];
|
||||
// Successful CivitAI items are reconciled in place afterwards
|
||||
// (their cards can be matched by model id); HF items keep the
|
||||
// legacy full reload because they have no CivitAI identity (#1078).
|
||||
const completedCivitaiItems = [];
|
||||
let hfCompletedCount = 0;
|
||||
|
||||
loadingManager.showCancelButton(async () => {
|
||||
if (cancelled) return;
|
||||
@@ -2065,6 +2242,11 @@ export class DownloadManager {
|
||||
} else {
|
||||
completedDownloads++;
|
||||
updateProgress(100, completedDownloads, '');
|
||||
if (isHf) {
|
||||
hfCompletedCount++;
|
||||
} else {
|
||||
completedCivitaiItems.push(item);
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
if (!cancelled) {
|
||||
@@ -2095,7 +2277,7 @@ export class DownloadManager {
|
||||
});
|
||||
}
|
||||
|
||||
await resetAndReload(true);
|
||||
await this._reconcileBatchViewAfterDownload(completedCivitaiItems, hfCompletedCount);
|
||||
}
|
||||
|
||||
async downloadVersionWithDefaults(modelType, modelId, versionId, {
|
||||
@@ -2104,7 +2286,8 @@ export class DownloadManager {
|
||||
modelRoot = '',
|
||||
targetFolder = '',
|
||||
useDefaultPaths = null,
|
||||
useSaveDirAsRoot = false
|
||||
useSaveDirAsRoot = false,
|
||||
isLatestVersion = null,
|
||||
} = {}) {
|
||||
console.warn('[download] downloadVersionWithDefaults: NO fileParams will be sent — backend will always use primary file. '
|
||||
+ 'modelType=%s, modelId=%s, versionId=%s, versionName="%s"',
|
||||
@@ -2129,6 +2312,7 @@ export class DownloadManager {
|
||||
useSaveDirAsRoot,
|
||||
source,
|
||||
closeModal: false,
|
||||
isLatestVersion,
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -87,6 +87,10 @@ export const BASE_MODELS = {
|
||||
UNKNOWN: "Other"
|
||||
};
|
||||
|
||||
// Window event dispatched after dynamic base models are (re)loaded from the API.
|
||||
// Pickers listen for it to refresh their option lists when data arrives late.
|
||||
export const BASE_MODELS_UPDATED_EVENT = 'lora-manager:base-models-updated';
|
||||
|
||||
// Custom dataTransfer MIME type tagging internal model-card drags (move-to-folder).
|
||||
// Preview-drop handlers use it to ignore drags that did not come from the OS file system.
|
||||
export const MODEL_CARD_DRAG_MIME_TYPE = 'application/x-lora-manager-model-card';
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
<div id="bulkBaseModelModal" class="modal" style="display: none;">
|
||||
<div class="modal-content modal-content">
|
||||
<div class="modal-content">
|
||||
<div class="modal-header">
|
||||
<h2>{{ t('modals.bulkBaseModel.title') }}</h2>
|
||||
<span class="close" onclick="modalManager.closeModal('bulkBaseModelModal')">×</span>
|
||||
@@ -8,31 +8,18 @@
|
||||
<div class="bulk-add-tags-info">
|
||||
<p>{{ t('modals.bulkBaseModel.description') }} <span id="bulkBaseModelCount">0</span> {{ t('modals.bulkBaseModel.models') }}</p>
|
||||
</div>
|
||||
|
||||
<div class="model-tags-container bulk-tags-container edit-mode">
|
||||
<div class="metadata-edit-container" style="display: block;">
|
||||
<div class="metadata-edit-content">
|
||||
<div class="metadata-edit-header">
|
||||
<label>{{ t('modals.bulkBaseModel.selectBaseModel') }}</label>
|
||||
</div>
|
||||
<div class="setting-control">
|
||||
<div class="select-control">
|
||||
<select id="bulkBaseModelSelect" class="bulk-base-model-select">
|
||||
<!-- Options will be populated dynamically -->
|
||||
</select>
|
||||
</div>
|
||||
</div>
|
||||
<div class="metadata-edit-controls">
|
||||
<button class="metadata-save-btn bulk-save-base-model-btn" onclick="bulkManager.saveBulkBaseModel()">
|
||||
<i class="fas fa-save"></i> {{ t('modals.bulkBaseModel.save') }}
|
||||
</button>
|
||||
<button class="btn btn-secondary" onclick="modalManager.closeModal('bulkBaseModelModal')">
|
||||
{{ t('modals.bulkBaseModel.cancel') }}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<label class="bulk-base-model-label">{{ t('modals.bulkBaseModel.selectBaseModel') }}</label>
|
||||
<div id="bulkBaseModelPicker" class="bulk-base-model-picker">
|
||||
<!-- Picker will be initialized dynamically -->
|
||||
</div>
|
||||
</div>
|
||||
<div class="modal-actions bulk-base-model-footer">
|
||||
<button class="cancel-btn" onclick="modalManager.closeModal('bulkBaseModelModal')">
|
||||
{{ t('modals.bulkBaseModel.cancel') }}
|
||||
</button>
|
||||
<button class="primary-btn bulk-save-base-model-btn" onclick="bulkManager.saveBulkBaseModel()">
|
||||
<i class="fas fa-save"></i> {{ t('modals.bulkBaseModel.save') }}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -0,0 +1,343 @@
|
||||
import { describe, it, beforeEach, afterEach, expect, vi } from 'vitest';
|
||||
|
||||
vi.mock('../../../static/js/utils/i18nHelpers.js', () => ({
|
||||
translate: vi.fn((key, params, fallback) => (typeof fallback === 'string' ? fallback : key)),
|
||||
}));
|
||||
|
||||
import {
|
||||
createBaseModelPicker,
|
||||
inferBaseModelsFromFilename,
|
||||
inferBaseModelsFromFilepaths,
|
||||
} from '../../../static/js/components/shared/BaseModelPicker.js';
|
||||
import {
|
||||
setDynamicBaseModels,
|
||||
clearDynamicBaseModels,
|
||||
BASE_MODELS_UPDATED_EVENT,
|
||||
} from '../../../static/js/utils/constants.js';
|
||||
|
||||
// jsdom does not implement scrollIntoView
|
||||
Element.prototype.scrollIntoView = Element.prototype.scrollIntoView || vi.fn();
|
||||
|
||||
const flushDebounce = () => new Promise((resolve) => setTimeout(resolve, 70));
|
||||
|
||||
function mountPicker(options = {}) {
|
||||
const picker = createBaseModelPicker(options);
|
||||
document.body.appendChild(picker.element);
|
||||
return picker;
|
||||
}
|
||||
|
||||
function getInput(picker) {
|
||||
return picker.element.querySelector('.base-model-search-input');
|
||||
}
|
||||
|
||||
function getDropdown(picker) {
|
||||
return picker.element.querySelector('.base-model-dropdown');
|
||||
}
|
||||
|
||||
function getItemValues(picker) {
|
||||
return Array.from(picker.element.querySelectorAll('.base-model-dropdown-item'))
|
||||
.map((el) => el.dataset.value);
|
||||
}
|
||||
|
||||
describe('inferBaseModelsFromFilepaths', () => {
|
||||
it('returns an empty array for empty or invalid input', () => {
|
||||
expect(inferBaseModelsFromFilepaths([])).toEqual([]);
|
||||
expect(inferBaseModelsFromFilepaths(null)).toEqual([]);
|
||||
expect(inferBaseModelsFromFilepaths(['/models/zzz_unknown.safetensors'])).toEqual([]);
|
||||
});
|
||||
|
||||
it('deduplicates and sorts by hit count across selected paths', () => {
|
||||
const result = inferBaseModelsFromFilepaths([
|
||||
'/loras/flux1_dev_alpha.safetensors',
|
||||
'/loras/another_flux_model.safetensors',
|
||||
'C:\\models\\sdxl_style.safetensors',
|
||||
]);
|
||||
|
||||
// Flux.1 D matched two paths, so it ranks first; entries are deduplicated
|
||||
expect(result[0]).toBe('Flux.1 D');
|
||||
expect(new Set(result).size).toBe(result.length);
|
||||
expect(result).toContain('SDXL 1.0');
|
||||
});
|
||||
|
||||
it('infers base models from a single filename', () => {
|
||||
expect(inferBaseModelsFromFilename('my_pony_lora.safetensors')).toContain('Pony');
|
||||
expect(inferBaseModelsFromFilename('')).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('createBaseModelPicker', () => {
|
||||
beforeEach(() => {
|
||||
clearDynamicBaseModels();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
clearDynamicBaseModels();
|
||||
});
|
||||
|
||||
it('groups uncategorized dynamic models under "Other (API)"', () => {
|
||||
setDynamicBaseModels(['MiniMax H3'], new Date().toISOString());
|
||||
const picker = mountPicker();
|
||||
|
||||
const headers = Array.from(picker.element.querySelectorAll('.base-model-dropdown-header'));
|
||||
const otherHeader = headers.find((el) => el.textContent === 'Other (API)');
|
||||
expect(otherHeader).toBeTruthy();
|
||||
|
||||
const section = otherHeader.closest('.base-model-dropdown-section');
|
||||
const values = Array.from(section.querySelectorAll('.base-model-dropdown-item'))
|
||||
.map((el) => el.dataset.value);
|
||||
expect(values).toContain('MiniMax H3');
|
||||
|
||||
picker.destroy();
|
||||
});
|
||||
|
||||
it('filters options case-insensitively after the debounce', async () => {
|
||||
setDynamicBaseModels(['MiniMax H3'], new Date().toISOString());
|
||||
const picker = mountPicker();
|
||||
|
||||
const input = getInput(picker);
|
||||
input.value = 'MINIMAX';
|
||||
input.dispatchEvent(new Event('input', { bubbles: true }));
|
||||
await flushDebounce();
|
||||
|
||||
expect(getItemValues(picker)).toEqual(['MiniMax H3']);
|
||||
|
||||
picker.destroy();
|
||||
});
|
||||
|
||||
it('shows the empty state when nothing matches', async () => {
|
||||
const picker = mountPicker();
|
||||
|
||||
const input = getInput(picker);
|
||||
input.value = 'no-such-model-xyz';
|
||||
input.dispatchEvent(new Event('input', { bubbles: true }));
|
||||
await flushDebounce();
|
||||
|
||||
expect(getItemValues(picker)).toEqual([]);
|
||||
expect(getDropdown(picker).querySelector('.base-model-dropdown-empty')).toBeTruthy();
|
||||
|
||||
picker.destroy();
|
||||
});
|
||||
|
||||
it('commits immediately on item click in commit mode', () => {
|
||||
const onCommit = vi.fn();
|
||||
const picker = mountPicker({ onCommit });
|
||||
|
||||
const item = Array.from(picker.element.querySelectorAll('.base-model-dropdown-item'))
|
||||
.find((el) => el.dataset.value === 'SDXL 1.0');
|
||||
item.click();
|
||||
|
||||
expect(onCommit).toHaveBeenCalledWith('SDXL 1.0');
|
||||
|
||||
picker.destroy();
|
||||
});
|
||||
|
||||
it('supports keyboard navigation and Enter to commit the active item', () => {
|
||||
const onCommit = vi.fn();
|
||||
const picker = mountPicker({ onCommit });
|
||||
const input = getInput(picker);
|
||||
|
||||
input.dispatchEvent(new KeyboardEvent('keydown', { key: 'ArrowDown', bubbles: true }));
|
||||
input.dispatchEvent(new KeyboardEvent('keydown', { key: 'ArrowDown', bubbles: true }));
|
||||
input.dispatchEvent(new KeyboardEvent('keydown', { key: 'ArrowUp', bubbles: true }));
|
||||
|
||||
const active = picker.element.querySelector('.base-model-dropdown-item.active');
|
||||
expect(active).toBeTruthy();
|
||||
|
||||
input.dispatchEvent(new KeyboardEvent('keydown', { key: 'Enter', bubbles: true }));
|
||||
expect(onCommit).toHaveBeenCalledWith(active.dataset.value);
|
||||
|
||||
picker.destroy();
|
||||
});
|
||||
|
||||
it('commits a custom typed value on Enter', () => {
|
||||
const onCommit = vi.fn();
|
||||
const picker = mountPicker({ onCommit });
|
||||
const input = getInput(picker);
|
||||
|
||||
input.value = 'My Custom Model';
|
||||
input.dispatchEvent(new KeyboardEvent('keydown', { key: 'Enter', bubbles: true }));
|
||||
|
||||
expect(onCommit).toHaveBeenCalledWith('My Custom Model');
|
||||
expect(picker.getValue()).toBe('My Custom Model');
|
||||
|
||||
picker.destroy();
|
||||
});
|
||||
|
||||
it('keeps typed text search-only on Enter when allowCustomValue is false', () => {
|
||||
const onCommit = vi.fn();
|
||||
const picker = mountPicker({ onCommit, allowCustomValue: false });
|
||||
const input = getInput(picker);
|
||||
|
||||
input.value = 'My Custom Model';
|
||||
input.dispatchEvent(new KeyboardEvent('keydown', { key: 'Enter', bubbles: true }));
|
||||
|
||||
expect(onCommit).not.toHaveBeenCalled();
|
||||
expect(picker.getValue()).toBe('');
|
||||
|
||||
picker.destroy();
|
||||
});
|
||||
|
||||
it('commits the typed custom value on outside click', async () => {
|
||||
const onCommit = vi.fn();
|
||||
const onDismiss = vi.fn();
|
||||
const picker = mountPicker({ onCommit, onDismiss, initialValue: 'SD 1.5' });
|
||||
const input = getInput(picker);
|
||||
|
||||
input.value = 'My Custom Model';
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
document.body.click();
|
||||
|
||||
expect(onCommit).toHaveBeenCalledWith('My Custom Model');
|
||||
expect(onDismiss).not.toHaveBeenCalled();
|
||||
expect(picker.getValue()).toBe('My Custom Model');
|
||||
|
||||
picker.destroy();
|
||||
});
|
||||
|
||||
it('dismisses without committing typed search text on outside click when allowCustomValue is false', async () => {
|
||||
const onCommit = vi.fn();
|
||||
const onDismiss = vi.fn();
|
||||
const picker = mountPicker({ onCommit, onDismiss, initialValue: 'SD 1.5', allowCustomValue: false });
|
||||
const input = getInput(picker);
|
||||
|
||||
input.value = 'My Custom Model';
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
document.body.click();
|
||||
|
||||
expect(onDismiss).toHaveBeenCalledTimes(1);
|
||||
expect(onCommit).not.toHaveBeenCalled();
|
||||
expect(picker.getValue()).toBe('SD 1.5');
|
||||
|
||||
picker.destroy();
|
||||
});
|
||||
|
||||
it('dismisses without committing on Escape', () => {
|
||||
const onCommit = vi.fn();
|
||||
const onDismiss = vi.fn();
|
||||
const picker = mountPicker({ onCommit, onDismiss, initialValue: 'SD 1.5' });
|
||||
const input = getInput(picker);
|
||||
|
||||
input.dispatchEvent(new KeyboardEvent('keydown', { key: 'Escape', bubbles: true }));
|
||||
|
||||
expect(onDismiss).toHaveBeenCalledTimes(1);
|
||||
expect(onCommit).not.toHaveBeenCalled();
|
||||
|
||||
picker.destroy();
|
||||
});
|
||||
|
||||
it('refreshes options when dynamic models arrive late and keeps the search text', async () => {
|
||||
const picker = mountPicker();
|
||||
const input = getInput(picker);
|
||||
|
||||
input.value = 'minimax';
|
||||
input.dispatchEvent(new Event('input', { bubbles: true }));
|
||||
await flushDebounce();
|
||||
expect(getItemValues(picker)).toEqual([]);
|
||||
|
||||
// Dynamic models arrive after the picker is already open
|
||||
setDynamicBaseModels(['MiniMax H3'], new Date().toISOString());
|
||||
window.dispatchEvent(new CustomEvent(BASE_MODELS_UPDATED_EVENT));
|
||||
|
||||
expect(input.value).toBe('minimax');
|
||||
expect(getItemValues(picker)).toEqual(['MiniMax H3']);
|
||||
|
||||
picker.destroy();
|
||||
});
|
||||
|
||||
it('stops reacting to updates after destroy', () => {
|
||||
const picker = mountPicker();
|
||||
picker.destroy();
|
||||
|
||||
setDynamicBaseModels(['MiniMax H3'], new Date().toISOString());
|
||||
window.dispatchEvent(new CustomEvent(BASE_MODELS_UPDATED_EVENT));
|
||||
|
||||
expect(getItemValues(picker)).not.toContain('MiniMax H3');
|
||||
});
|
||||
|
||||
it('renders filename-based suggestions in a Suggested section', () => {
|
||||
const suggestions = inferBaseModelsFromFilename('flux1_dev_model.safetensors');
|
||||
const picker = mountPicker({ suggestions });
|
||||
|
||||
const suggestedHeader = picker.element.querySelector('.base-model-dropdown-header.suggested-header');
|
||||
expect(suggestedHeader).toBeTruthy();
|
||||
|
||||
const section = suggestedHeader.closest('.base-model-dropdown-section');
|
||||
const values = Array.from(section.querySelectorAll('.base-model-dropdown-item'))
|
||||
.map((el) => el.dataset.value);
|
||||
expect(values).toContain('Flux.1 D');
|
||||
|
||||
// Suggested entries are deduplicated out of the categorized sections
|
||||
expect(getItemValues(picker).filter((v) => v === 'Flux.1 D')).toHaveLength(1);
|
||||
|
||||
picker.destroy();
|
||||
});
|
||||
|
||||
it('change mode only notifies via onChange and tracks the value', () => {
|
||||
const onCommit = vi.fn();
|
||||
const onChange = vi.fn();
|
||||
const picker = mountPicker({ mode: 'change', onCommit, onChange });
|
||||
|
||||
const item = Array.from(picker.element.querySelectorAll('.base-model-dropdown-item'))
|
||||
.find((el) => el.dataset.value === 'SDXL 1.0');
|
||||
item.click();
|
||||
|
||||
expect(onCommit).not.toHaveBeenCalled();
|
||||
expect(onChange).toHaveBeenCalledWith('SDXL 1.0');
|
||||
expect(picker.getValue()).toBe('SDXL 1.0');
|
||||
|
||||
// The list collapses to the selected item instead of resetting to the
|
||||
// full list (avoids a scroll jump in the bulk modal's inline list)
|
||||
expect(getItemValues(picker)).toEqual(['SDXL 1.0']);
|
||||
|
||||
picker.destroy();
|
||||
});
|
||||
|
||||
it('treats typed text as the live value in change mode', async () => {
|
||||
const onChange = vi.fn();
|
||||
const picker = mountPicker({ mode: 'change', onChange });
|
||||
const input = getInput(picker);
|
||||
|
||||
input.value = 'Typed Custom';
|
||||
input.dispatchEvent(new Event('input', { bubbles: true }));
|
||||
await flushDebounce();
|
||||
|
||||
expect(onChange).toHaveBeenCalledWith('Typed Custom');
|
||||
expect(picker.getValue()).toBe('Typed Custom');
|
||||
|
||||
input.dispatchEvent(new KeyboardEvent('keydown', { key: 'Enter', bubbles: true }));
|
||||
expect(picker.getValue()).toBe('Typed Custom');
|
||||
// Custom values are not in the option list — the full list stays visible
|
||||
expect(getItemValues(picker).length).toBeGreaterThan(1);
|
||||
|
||||
picker.destroy();
|
||||
});
|
||||
|
||||
it('keeps typed text as search-only in change mode when allowCustomValue is false', async () => {
|
||||
const onChange = vi.fn();
|
||||
const picker = mountPicker({ mode: 'change', onChange, allowCustomValue: false });
|
||||
const input = getInput(picker);
|
||||
|
||||
input.value = 'Typed Custom';
|
||||
input.dispatchEvent(new Event('input', { bubbles: true }));
|
||||
await flushDebounce();
|
||||
input.dispatchEvent(new KeyboardEvent('keydown', { key: 'Enter', bubbles: true }));
|
||||
|
||||
expect(onChange).not.toHaveBeenCalled();
|
||||
expect(picker.getValue()).toBe('');
|
||||
expect(getDropdown(picker).querySelector('.base-model-dropdown-empty')).toBeTruthy();
|
||||
|
||||
picker.destroy();
|
||||
});
|
||||
|
||||
it('setValue updates the input and selected marker', () => {
|
||||
const picker = mountPicker({ mode: 'change' });
|
||||
|
||||
picker.setValue('SD 3.5');
|
||||
|
||||
expect(picker.getValue()).toBe('SD 3.5');
|
||||
const selected = picker.element.querySelector('.base-model-dropdown-item.selected');
|
||||
expect(selected?.dataset.value).toBe('SD 3.5');
|
||||
|
||||
picker.destroy();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,297 @@
|
||||
import { describe, it, beforeEach, afterEach, expect, vi } from 'vitest';
|
||||
import { renderTemplate } from '../utils/domFixtures.js';
|
||||
|
||||
const showToastMock = vi.fn();
|
||||
const translateMock = vi.fn((key, params, fallback) => (typeof fallback === 'string' ? fallback : key));
|
||||
|
||||
const loadingManagerStub = {
|
||||
showSimpleLoading: vi.fn(),
|
||||
showCancelButton: vi.fn(),
|
||||
hide: vi.fn(),
|
||||
};
|
||||
|
||||
const stateStub = {
|
||||
currentPageType: 'loras',
|
||||
bulkMode: false,
|
||||
selectedModels: new Set(),
|
||||
loadingManager: loadingManagerStub,
|
||||
virtualScroller: { updateSingleItem: vi.fn() },
|
||||
global: { settings: {} },
|
||||
};
|
||||
|
||||
const saveModelMetadataMock = vi.fn();
|
||||
const getModelApiClientMock = vi.fn(() => ({ saveModelMetadata: saveModelMetadataMock }));
|
||||
const updateRecipeMetadataMock = vi.fn(() => Promise.resolve({ success: true }));
|
||||
const showModalMock = vi.fn();
|
||||
const closeModalMock = vi.fn();
|
||||
|
||||
vi.mock('../../../static/js/state/index.js', () => ({
|
||||
state: stateStub,
|
||||
getCurrentPageState: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('../../../static/js/utils/uiHelpers.js', () => ({
|
||||
showToast: showToastMock,
|
||||
copyToClipboard: vi.fn(),
|
||||
sendLoraToWorkflow: vi.fn(),
|
||||
sendEmbeddingToWorkflow: vi.fn(),
|
||||
buildLoraSyntax: vi.fn(),
|
||||
getNSFWLevelName: vi.fn(() => 'Unknown'),
|
||||
}));
|
||||
|
||||
vi.mock('../../../static/js/api/modelApiFactory.js', () => ({
|
||||
getModelApiClient: getModelApiClientMock,
|
||||
resetAndReload: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('../../../static/js/api/recipeApi.js', () => ({
|
||||
RecipeSidebarApiClient: class {},
|
||||
updateRecipeMetadata: updateRecipeMetadataMock,
|
||||
extractRecipeId: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('../../../static/js/api/apiConfig.js', () => ({
|
||||
MODEL_TYPES: { LORA: 'loras', CHECKPOINT: 'checkpoints', EMBEDDING: 'embeddings' },
|
||||
MODEL_CONFIG: {},
|
||||
}));
|
||||
|
||||
vi.mock('../../../static/js/managers/ModalManager.js', () => ({
|
||||
modalManager: { showModal: showModalMock, closeModal: closeModalMock },
|
||||
}));
|
||||
|
||||
vi.mock('../../../static/js/components/shared/ModelCard.js', () => ({
|
||||
updateCardsForBulkMode: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('../../../static/js/utils/i18nHelpers.js', () => ({
|
||||
translate: translateMock,
|
||||
}));
|
||||
|
||||
vi.mock('../../../static/js/utils/priorityTagHelpers.js', () => ({
|
||||
getPriorityTagSuggestions: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('../../../static/js/components/shared/NsfwLevelSelector.js', () => ({
|
||||
getNsfwLevelSelector: vi.fn(),
|
||||
}));
|
||||
|
||||
import {
|
||||
setDynamicBaseModels,
|
||||
clearDynamicBaseModels,
|
||||
} from '../../../static/js/utils/constants.js';
|
||||
|
||||
// jsdom does not implement scrollIntoView
|
||||
Element.prototype.scrollIntoView = Element.prototype.scrollIntoView || vi.fn();
|
||||
|
||||
function getPickerContainer() {
|
||||
return document.getElementById('bulkBaseModelPicker');
|
||||
}
|
||||
|
||||
function clickDropdownItem(value) {
|
||||
const item = Array.from(document.querySelectorAll('.base-model-dropdown-item'))
|
||||
.find((el) => el.dataset.value === value);
|
||||
expect(item, `dropdown item for "${value}"`).toBeTruthy();
|
||||
item.click();
|
||||
}
|
||||
|
||||
describe('BulkManager bulk base model', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
clearDynamicBaseModels();
|
||||
stateStub.currentPageType = 'loras';
|
||||
stateStub.bulkMode = false;
|
||||
stateStub.selectedModels.clear();
|
||||
saveModelMetadataMock.mockResolvedValue(undefined);
|
||||
updateRecipeMetadataMock.mockResolvedValue({ success: true });
|
||||
renderTemplate('components/modals/bulk_base_model_modal.html');
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
clearDynamicBaseModels();
|
||||
});
|
||||
|
||||
async function createBulkManager() {
|
||||
const { BulkManager } = await import('../../../static/js/managers/BulkManager.js');
|
||||
return new BulkManager();
|
||||
}
|
||||
|
||||
it('warns when opening the modal without a selection', async () => {
|
||||
const bulk = await createBulkManager();
|
||||
bulk.showBulkBaseModelModal();
|
||||
|
||||
expect(showToastMock).toHaveBeenCalledWith('toast.models.noModelsSelected', {}, 'warning');
|
||||
expect(showModalMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('initializes the picker inside the modal container', async () => {
|
||||
const bulk = await createBulkManager();
|
||||
stateStub.selectedModels.add('/models/sdxl_a.safetensors');
|
||||
stateStub.selectedModels.add('/models/sdxl_b.safetensors');
|
||||
|
||||
bulk.showBulkBaseModelModal();
|
||||
|
||||
expect(document.getElementById('bulkBaseModelCount').textContent).toBe('2');
|
||||
const container = getPickerContainer();
|
||||
expect(container.querySelector('.base-model-search-wrapper')).toBeTruthy();
|
||||
expect(bulk.bulkBaseModelPicker).toBeTruthy();
|
||||
|
||||
// Filename-based suggestions from the selected paths
|
||||
const suggestedHeader = container.querySelector('.base-model-dropdown-header.suggested-header');
|
||||
expect(suggestedHeader).toBeTruthy();
|
||||
const suggestedSection = suggestedHeader.closest('.base-model-dropdown-section');
|
||||
expect(suggestedSection.querySelector('.base-model-dropdown-item')?.dataset.value).toBe('SDXL 1.0');
|
||||
|
||||
bulk.cleanupBulkBaseModelModal();
|
||||
});
|
||||
|
||||
it('offers dynamic models such as MiniMax H3 under "Other (API)"', async () => {
|
||||
setDynamicBaseModels(['MiniMax H3'], new Date().toISOString());
|
||||
const bulk = await createBulkManager();
|
||||
stateStub.selectedModels.add('/models/test.safetensors');
|
||||
|
||||
bulk.showBulkBaseModelModal();
|
||||
|
||||
const headers = Array.from(document.querySelectorAll('.base-model-dropdown-header'));
|
||||
const otherHeader = headers.find((el) => el.textContent === 'Other (API)');
|
||||
expect(otherHeader).toBeTruthy();
|
||||
const section = otherHeader.closest('.base-model-dropdown-section');
|
||||
const values = Array.from(section.querySelectorAll('.base-model-dropdown-item'))
|
||||
.map((el) => el.dataset.value);
|
||||
expect(values).toContain('MiniMax H3');
|
||||
|
||||
bulk.cleanupBulkBaseModelModal();
|
||||
});
|
||||
|
||||
it('saves a dynamic base model through the model API on model pages', async () => {
|
||||
setDynamicBaseModels(['MiniMax H3'], new Date().toISOString());
|
||||
const bulk = await createBulkManager();
|
||||
stateStub.currentPageType = 'loras';
|
||||
stateStub.selectedModels.add('/models/a.safetensors');
|
||||
stateStub.selectedModels.add('/models/b.safetensors');
|
||||
|
||||
bulk.showBulkBaseModelModal();
|
||||
clickDropdownItem('MiniMax H3');
|
||||
expect(bulk.bulkBaseModelValue).toBe('MiniMax H3');
|
||||
|
||||
await bulk.saveBulkBaseModel();
|
||||
|
||||
expect(closeModalMock).toHaveBeenCalledWith('bulkBaseModelModal');
|
||||
expect(saveModelMetadataMock).toHaveBeenCalledTimes(2);
|
||||
expect(saveModelMetadataMock).toHaveBeenCalledWith('/models/a.safetensors', { base_model: 'MiniMax H3' });
|
||||
expect(saveModelMetadataMock).toHaveBeenCalledWith('/models/b.safetensors', { base_model: 'MiniMax H3' });
|
||||
expect(updateRecipeMetadataMock).not.toHaveBeenCalled();
|
||||
expect(showToastMock).toHaveBeenCalledWith(
|
||||
'toast.models.bulkBaseModelUpdateSuccess',
|
||||
{ count: 2 },
|
||||
'success'
|
||||
);
|
||||
});
|
||||
|
||||
it('saves through the recipe API when on the recipes page', async () => {
|
||||
const bulk = await createBulkManager();
|
||||
stateStub.currentPageType = 'recipes';
|
||||
stateStub.selectedModels.add('/recipes/test.webp');
|
||||
|
||||
bulk.showBulkBaseModelModal();
|
||||
clickDropdownItem('SD 1.5');
|
||||
|
||||
await bulk.saveBulkBaseModel();
|
||||
|
||||
expect(updateRecipeMetadataMock).toHaveBeenCalledWith('/recipes/test.webp', { base_model: 'SD 1.5' });
|
||||
expect(updateRecipeMetadataMock).toHaveBeenCalledTimes(1);
|
||||
expect(saveModelMetadataMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('warns and skips saving when no base model is selected', async () => {
|
||||
const bulk = await createBulkManager();
|
||||
stateStub.selectedModels.add('/models/a.safetensors');
|
||||
|
||||
bulk.showBulkBaseModelModal();
|
||||
await bulk.saveBulkBaseModel();
|
||||
|
||||
expect(showToastMock).toHaveBeenCalledWith('toast.models.baseModelNotSelected', {}, 'warning');
|
||||
expect(saveModelMetadataMock).not.toHaveBeenCalled();
|
||||
expect(closeModalMock).not.toHaveBeenCalledWith('bulkBaseModelModal');
|
||||
|
||||
bulk.cleanupBulkBaseModelModal();
|
||||
});
|
||||
|
||||
it('accepts arbitrary typed values in bulk mode, matching the single-model modal', async () => {
|
||||
const bulk = await createBulkManager();
|
||||
stateStub.selectedModels.add('/models/a.safetensors');
|
||||
|
||||
bulk.showBulkBaseModelModal();
|
||||
const input = document.querySelector('#bulkBaseModelPicker .base-model-search-input');
|
||||
input.value = 'Not A Listed Model';
|
||||
input.dispatchEvent(new Event('input', { bubbles: true }));
|
||||
await new Promise((resolve) => setTimeout(resolve, 70));
|
||||
|
||||
expect(bulk.bulkBaseModelValue).toBe('Not A Listed Model');
|
||||
|
||||
await bulk.saveBulkBaseModel();
|
||||
|
||||
expect(saveModelMetadataMock).toHaveBeenCalledWith('/models/a.safetensors', { base_model: 'Not A Listed Model' });
|
||||
expect(closeModalMock).toHaveBeenCalledWith('bulkBaseModelModal');
|
||||
|
||||
bulk.cleanupBulkBaseModelModal();
|
||||
});
|
||||
|
||||
it('uses a dedicated layout without the settings-page control wrapper', () => {
|
||||
const modal = document.getElementById('bulkBaseModelModal');
|
||||
expect(modal).toBeTruthy();
|
||||
expect(modal.querySelector('.setting-control')).toBeNull();
|
||||
expect(modal.querySelector('.metadata-edit-container')).toBeNull();
|
||||
const footer = modal.querySelector('.bulk-base-model-footer');
|
||||
expect(footer).toBeTruthy();
|
||||
// Buttons follow the app-wide modal-actions convention
|
||||
expect(footer.classList.contains('modal-actions')).toBe(true);
|
||||
expect(footer.querySelector('.primary-btn.bulk-save-base-model-btn')).toBeTruthy();
|
||||
expect(footer.querySelector('.cancel-btn')).toBeTruthy();
|
||||
});
|
||||
|
||||
it('focuses the search input when the modal opens', async () => {
|
||||
const bulk = await createBulkManager();
|
||||
stateStub.selectedModels.add('/models/a.safetensors');
|
||||
|
||||
bulk.showBulkBaseModelModal();
|
||||
|
||||
const input = document.querySelector('#bulkBaseModelPicker .base-model-search-input');
|
||||
expect(document.activeElement).toBe(input);
|
||||
|
||||
bulk.cleanupBulkBaseModelModal();
|
||||
});
|
||||
|
||||
it('reports partial failures', async () => {
|
||||
const bulk = await createBulkManager();
|
||||
stateStub.selectedModels.add('/models/ok.safetensors');
|
||||
stateStub.selectedModels.add('/models/fail.safetensors');
|
||||
saveModelMetadataMock
|
||||
.mockResolvedValueOnce(undefined)
|
||||
.mockRejectedValueOnce(new Error('boom'));
|
||||
|
||||
bulk.showBulkBaseModelModal();
|
||||
clickDropdownItem('SDXL 1.0');
|
||||
await bulk.saveBulkBaseModel();
|
||||
|
||||
expect(showToastMock).toHaveBeenCalledWith(
|
||||
'toast.models.bulkBaseModelUpdatePartial',
|
||||
{ success: 1, failed: 1 },
|
||||
'warning'
|
||||
);
|
||||
});
|
||||
|
||||
it('destroys the picker and clears the staged value on cleanup', async () => {
|
||||
const bulk = await createBulkManager();
|
||||
stateStub.selectedModels.add('/models/a.safetensors');
|
||||
|
||||
bulk.showBulkBaseModelModal();
|
||||
clickDropdownItem('SDXL 1.0');
|
||||
expect(bulk.bulkBaseModelValue).toBe('SDXL 1.0');
|
||||
|
||||
bulk.cleanupBulkBaseModelModal();
|
||||
|
||||
expect(bulk.bulkBaseModelPicker).toBeNull();
|
||||
expect(bulk.bulkBaseModelValue).toBe('');
|
||||
expect(getPickerContainer().innerHTML).toBe('');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,466 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
const {
|
||||
DOWNLOAD_MANAGER_MODULE,
|
||||
MODAL_MANAGER_MODULE,
|
||||
UI_HELPERS_MODULE,
|
||||
STATE_MODULE,
|
||||
LOADING_MANAGER_MODULE,
|
||||
API_FACTORY_MODULE,
|
||||
STORAGE_HELPERS_MODULE,
|
||||
FOLDER_TREE_MANAGER_MODULE,
|
||||
I18N_HELPERS_MODULE,
|
||||
SUMMARY_MODULE,
|
||||
SIDEBAR_MANAGER_MODULE,
|
||||
mockApiClient,
|
||||
mockScroller,
|
||||
stateMock,
|
||||
mockLoadingManager,
|
||||
showToastMock,
|
||||
resetAndReloadMock,
|
||||
setStorageItemMock,
|
||||
sidebarRefreshMock,
|
||||
} = vi.hoisted(() => {
|
||||
const mockScroller = {
|
||||
items: [],
|
||||
removeItemByFilePath: vi.fn(),
|
||||
removeMultipleItemsByFilePath: vi.fn(),
|
||||
updateSingleItem: vi.fn(),
|
||||
};
|
||||
|
||||
const stateMock = {
|
||||
currentPageType: 'loras',
|
||||
global: { settings: {} },
|
||||
loadingManager: null,
|
||||
virtualScroller: mockScroller,
|
||||
};
|
||||
|
||||
const mockApiClient = {
|
||||
apiConfig: {
|
||||
config: {
|
||||
displayName: 'LoRA',
|
||||
singularName: 'lora',
|
||||
},
|
||||
},
|
||||
modelType: 'loras',
|
||||
getPageState: vi.fn(() => ({})),
|
||||
downloadModel: vi.fn(),
|
||||
downloadHfModel: vi.fn(),
|
||||
cancelDownload: vi.fn(),
|
||||
};
|
||||
|
||||
const mockLoadingManager = {
|
||||
showSimpleLoading: vi.fn(),
|
||||
setStatus: vi.fn(),
|
||||
hide: vi.fn(),
|
||||
restoreProgressBar: vi.fn(),
|
||||
showDownloadProgress: vi.fn(() => vi.fn()),
|
||||
showCancelButton: vi.fn(),
|
||||
};
|
||||
|
||||
return {
|
||||
DOWNLOAD_MANAGER_MODULE: new URL('../../../static/js/managers/DownloadManager.js', import.meta.url).pathname,
|
||||
MODAL_MANAGER_MODULE: new URL('../../../static/js/managers/ModalManager.js', import.meta.url).pathname,
|
||||
UI_HELPERS_MODULE: new URL('../../../static/js/utils/uiHelpers.js', import.meta.url).pathname,
|
||||
STATE_MODULE: new URL('../../../static/js/state/index.js', import.meta.url).pathname,
|
||||
LOADING_MANAGER_MODULE: new URL('../../../static/js/managers/LoadingManager.js', import.meta.url).pathname,
|
||||
API_FACTORY_MODULE: new URL('../../../static/js/api/modelApiFactory.js', import.meta.url).pathname,
|
||||
STORAGE_HELPERS_MODULE: new URL('../../../static/js/utils/storageHelpers.js', import.meta.url).pathname,
|
||||
FOLDER_TREE_MANAGER_MODULE: new URL('../../../static/js/components/FolderTreeManager.js', import.meta.url).pathname,
|
||||
I18N_HELPERS_MODULE: new URL('../../../static/js/utils/i18nHelpers.js', import.meta.url).pathname,
|
||||
SUMMARY_MODULE: new URL('../../../static/js/components/DownloadBatchSummaryModal.js', import.meta.url).pathname,
|
||||
SIDEBAR_MANAGER_MODULE: new URL('../../../static/js/components/SidebarManager.js', import.meta.url).pathname,
|
||||
mockApiClient,
|
||||
mockScroller,
|
||||
stateMock,
|
||||
mockLoadingManager,
|
||||
showToastMock: vi.fn(),
|
||||
resetAndReloadMock: vi.fn(),
|
||||
setStorageItemMock: vi.fn(),
|
||||
sidebarRefreshMock: vi.fn(),
|
||||
};
|
||||
});
|
||||
|
||||
vi.mock(MODAL_MANAGER_MODULE, () => ({
|
||||
modalManager: {
|
||||
showModal: vi.fn(),
|
||||
closeModal: vi.fn(),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock(UI_HELPERS_MODULE, () => ({
|
||||
showToast: showToastMock,
|
||||
}));
|
||||
|
||||
vi.mock(STATE_MODULE, () => ({
|
||||
state: stateMock,
|
||||
}));
|
||||
|
||||
vi.mock(LOADING_MANAGER_MODULE, () => ({
|
||||
LoadingManager: vi.fn(() => mockLoadingManager),
|
||||
}));
|
||||
|
||||
vi.mock(API_FACTORY_MODULE, () => ({
|
||||
getModelApiClient: vi.fn(() => mockApiClient),
|
||||
resetAndReload: resetAndReloadMock,
|
||||
}));
|
||||
|
||||
vi.mock(STORAGE_HELPERS_MODULE, () => ({
|
||||
getStorageItem: vi.fn((_key, defaultValue) => defaultValue),
|
||||
setStorageItem: setStorageItemMock,
|
||||
}));
|
||||
|
||||
vi.mock(FOLDER_TREE_MANAGER_MODULE, () => ({
|
||||
FolderTreeManager: vi.fn(() => ({
|
||||
clearSelection: vi.fn(),
|
||||
init: vi.fn(),
|
||||
})),
|
||||
}));
|
||||
|
||||
vi.mock(I18N_HELPERS_MODULE, () => ({
|
||||
translate: vi.fn((_, __, fallback) => fallback ?? ''),
|
||||
}));
|
||||
|
||||
vi.mock(SUMMARY_MODULE, () => ({
|
||||
showDownloadBatchSummary: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock(SIDEBAR_MANAGER_MODULE, () => ({
|
||||
sidebarManager: { refresh: sidebarRefreshMock },
|
||||
}));
|
||||
|
||||
/** Minimal WebSocket stub: executeDownloadWithProgress never awaits open. */
|
||||
class FakeWebSocket {
|
||||
constructor(url) {
|
||||
this.url = url;
|
||||
this.readyState = 0; // CONNECTING
|
||||
}
|
||||
close() {}
|
||||
}
|
||||
|
||||
/** Build a card item as returned by the backend listing endpoint. */
|
||||
function makeItem(filePath, modelId, base) {
|
||||
return {
|
||||
file_path: filePath,
|
||||
civitai: { modelId, ...(base ? { baseModel: base } : {}) },
|
||||
update_available: true,
|
||||
};
|
||||
}
|
||||
|
||||
describe('DownloadManager post-download in-place reconciliation (#1078)', () => {
|
||||
let DownloadManager;
|
||||
let manager;
|
||||
|
||||
beforeEach(async () => {
|
||||
document.body.innerHTML = '';
|
||||
stateMock.virtualScroller = mockScroller;
|
||||
mockScroller.items = [];
|
||||
mockScroller.removeItemByFilePath.mockReset();
|
||||
mockScroller.removeMultipleItemsByFilePath.mockReset();
|
||||
mockScroller.updateSingleItem.mockReset();
|
||||
mockApiClient.getPageState.mockReset();
|
||||
mockApiClient.getPageState.mockReturnValue({});
|
||||
mockApiClient.downloadModel.mockReset();
|
||||
resetAndReloadMock.mockReset();
|
||||
resetAndReloadMock.mockResolvedValue(undefined);
|
||||
sidebarRefreshMock.mockReset();
|
||||
sidebarRefreshMock.mockResolvedValue(undefined);
|
||||
setStorageItemMock.mockReset();
|
||||
showToastMock.mockClear();
|
||||
vi.stubGlobal('WebSocket', FakeWebSocket);
|
||||
|
||||
vi.resetModules();
|
||||
({ DownloadManager } = await import(DOWNLOAD_MANAGER_MODULE));
|
||||
manager = new DownloadManager();
|
||||
manager.apiClient = mockApiClient;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
document.body.innerHTML = '';
|
||||
vi.unstubAllGlobals();
|
||||
stateMock.virtualScroller = mockScroller;
|
||||
});
|
||||
|
||||
describe('_isVersionLatest', () => {
|
||||
it('treats unknown/empty version lists as latest', () => {
|
||||
expect(manager._isVersionLatest('250', [])).toBe(true);
|
||||
expect(manager._isVersionLatest('250', null)).toBe(true);
|
||||
expect(manager._isVersionLatest(undefined, undefined)).toBe(true);
|
||||
});
|
||||
|
||||
it('returns true only for the newest known remote version', () => {
|
||||
const versions = [{ id: 100 }, { id: 250 }, { id: 30 }];
|
||||
expect(manager._isVersionLatest(250, versions)).toBe(true);
|
||||
expect(manager._isVersionLatest('250', versions)).toBe(true);
|
||||
expect(manager._isVersionLatest(100, versions)).toBe(false);
|
||||
expect(manager._isVersionLatest(30, versions)).toBe(false);
|
||||
});
|
||||
|
||||
it('supports record-style version objects (versionId field)', () => {
|
||||
const versions = [{ versionId: 10 }, { versionId: 20 }];
|
||||
expect(manager._isVersionLatest(20, versions)).toBe(true);
|
||||
expect(manager._isVersionLatest(10, versions)).toBe(false);
|
||||
});
|
||||
|
||||
it('returns true when the target id is unknown', () => {
|
||||
const versions = [{ id: 100 }];
|
||||
expect(manager._isVersionLatest('not-a-number', versions)).toBe(true);
|
||||
expect(manager._isVersionLatest(undefined, versions)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('_reconcileViewAfterDownload', () => {
|
||||
it('removes the model cards in the Updates view when the latest version was installed', async () => {
|
||||
stateMock.virtualScroller.items = [
|
||||
makeItem('/models/loras/old.safetensors', 837884),
|
||||
makeItem('/models/loras/unrelated.safetensors', 999999),
|
||||
];
|
||||
mockApiClient.getPageState.mockReturnValue({ showUpdateAvailableOnly: true });
|
||||
|
||||
const result = await manager._reconcileViewAfterDownload({ modelId: 837884, isLatestVersion: true });
|
||||
|
||||
expect(result).toBe(true);
|
||||
expect(mockScroller.removeMultipleItemsByFilePath).toHaveBeenCalledWith([
|
||||
'/models/loras/old.safetensors',
|
||||
]);
|
||||
expect(mockScroller.updateSingleItem).not.toHaveBeenCalled();
|
||||
expect(resetAndReloadMock).not.toHaveBeenCalled();
|
||||
// Sidebar (folder counts) is refreshed, but never the model listing.
|
||||
expect(sidebarRefreshMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('patches the update flag instead of removing cards in a normal listing', async () => {
|
||||
stateMock.virtualScroller.items = [
|
||||
makeItem('/models/loras/old.safetensors', 837884),
|
||||
];
|
||||
mockApiClient.getPageState.mockReturnValue({ showUpdateAvailableOnly: false });
|
||||
|
||||
const result = await manager._reconcileViewAfterDownload({ modelId: 837884, isLatestVersion: true });
|
||||
|
||||
expect(result).toBe(true);
|
||||
expect(mockScroller.updateSingleItem).toHaveBeenCalledWith(
|
||||
'/models/loras/old.safetensors',
|
||||
{ update_available: false }
|
||||
);
|
||||
expect(mockScroller.removeMultipleItemsByFilePath).not.toHaveBeenCalled();
|
||||
expect(resetAndReloadMock).not.toHaveBeenCalled();
|
||||
expect(sidebarRefreshMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('leaves the list untouched when an older version was deliberately downloaded', async () => {
|
||||
stateMock.virtualScroller.items = [makeItem('/models/loras/old.safetensors', 837884)];
|
||||
mockApiClient.getPageState.mockReturnValue({ showUpdateAvailableOnly: true });
|
||||
|
||||
const result = await manager._reconcileViewAfterDownload({ modelId: 837884, isLatestVersion: false });
|
||||
|
||||
expect(result).toBe(true);
|
||||
expect(mockScroller.removeMultipleItemsByFilePath).not.toHaveBeenCalled();
|
||||
expect(mockScroller.updateSingleItem).not.toHaveBeenCalled();
|
||||
expect(resetAndReloadMock).not.toHaveBeenCalled();
|
||||
expect(sidebarRefreshMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('keeps the listing untouched when the downloaded model is not in the current view', async () => {
|
||||
stateMock.virtualScroller.items = [makeItem('/models/loras/other.safetensors', 999999)];
|
||||
mockApiClient.getPageState.mockReturnValue({ showUpdateAvailableOnly: true });
|
||||
|
||||
const result = await manager._reconcileViewAfterDownload({ modelId: 837884, isLatestVersion: true });
|
||||
|
||||
expect(result).toBe(false);
|
||||
expect(mockScroller.removeMultipleItemsByFilePath).not.toHaveBeenCalled();
|
||||
expect(mockScroller.updateSingleItem).not.toHaveBeenCalled();
|
||||
expect(resetAndReloadMock).not.toHaveBeenCalled();
|
||||
expect(sidebarRefreshMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('no-ops for models without a CivitAI identity (HF downloads)', async () => {
|
||||
stateMock.virtualScroller.items = [makeItem('/models/loras/a.safetensors', 123)];
|
||||
mockApiClient.getPageState.mockReturnValue({ showUpdateAvailableOnly: true });
|
||||
|
||||
const result = await manager._reconcileViewAfterDownload({ modelId: null, isLatestVersion: true });
|
||||
|
||||
expect(result).toBe(false);
|
||||
expect(mockScroller.removeMultipleItemsByFilePath).not.toHaveBeenCalled();
|
||||
expect(resetAndReloadMock).not.toHaveBeenCalled();
|
||||
expect(sidebarRefreshMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('falls back to a full reload when no virtual scroller is available', async () => {
|
||||
stateMock.virtualScroller = undefined;
|
||||
|
||||
const result = await manager._reconcileViewAfterDownload({ modelId: 837884, isLatestVersion: true });
|
||||
|
||||
expect(result).toBe(false);
|
||||
expect(resetAndReloadMock).toHaveBeenCalledWith(true);
|
||||
expect(mockScroller.removeMultipleItemsByFilePath).not.toHaveBeenCalled();
|
||||
expect(sidebarRefreshMock).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('_reconcileBatchViewAfterDownload', () => {
|
||||
it('reconciles each distinct successful CivitAI model and refreshes the sidebar once', async () => {
|
||||
stateMock.virtualScroller.items = [
|
||||
makeItem('/models/loras/a.safetensors', 111),
|
||||
makeItem('/models/loras/b.safetensors', 222),
|
||||
];
|
||||
mockApiClient.getPageState.mockReturnValue({ showUpdateAvailableOnly: true });
|
||||
|
||||
await manager._reconcileBatchViewAfterDownload([
|
||||
{ modelId: '111', selectedVersion: { id: 40 }, versions: [{ id: 40 }, { id: 10 }] },
|
||||
{ modelId: '111', selectedVersion: { id: 40 }, versions: [{ id: 40 }, { id: 10 }] },
|
||||
{ modelId: '222', selectedVersion: { id: 7 }, versions: [{ id: 7 }] },
|
||||
], 0);
|
||||
|
||||
expect(mockScroller.removeMultipleItemsByFilePath).toHaveBeenCalledTimes(2);
|
||||
expect(mockScroller.removeMultipleItemsByFilePath).toHaveBeenCalledWith(['/models/loras/a.safetensors']);
|
||||
expect(mockScroller.removeMultipleItemsByFilePath).toHaveBeenCalledWith(['/models/loras/b.safetensors']);
|
||||
// One sidebar refresh for the whole batch, not one per model.
|
||||
expect(sidebarRefreshMock).toHaveBeenCalledTimes(1);
|
||||
expect(resetAndReloadMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('falls back to a full reload when any HF download completed', async () => {
|
||||
stateMock.virtualScroller.items = [makeItem('/models/loras/a.safetensors', 111)];
|
||||
mockApiClient.getPageState.mockReturnValue({ showUpdateAvailableOnly: true });
|
||||
|
||||
await manager._reconcileBatchViewAfterDownload([
|
||||
{ modelId: '111', selectedVersion: { id: 40 }, versions: [{ id: 40 }, { id: 10 }] },
|
||||
], 1);
|
||||
|
||||
expect(resetAndReloadMock).toHaveBeenCalledWith(true);
|
||||
expect(mockScroller.removeMultipleItemsByFilePath).not.toHaveBeenCalled();
|
||||
expect(sidebarRefreshMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('falls back to a full reload when no virtual scroller exists', async () => {
|
||||
stateMock.virtualScroller = undefined;
|
||||
|
||||
await manager._reconcileBatchViewAfterDownload([
|
||||
{ modelId: '111', selectedVersion: { id: 40 }, versions: [{ id: 40 }, { id: 10 }] },
|
||||
], 0);
|
||||
|
||||
expect(resetAndReloadMock).toHaveBeenCalledWith(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('executeDownloadWithProgress success path', () => {
|
||||
it('reconciles in place instead of hijacking the active folder or reloading', async () => {
|
||||
stateMock.virtualScroller.items = [makeItem('/models/loras/old.safetensors', 837884)];
|
||||
const pageState = { showUpdateAvailableOnly: true };
|
||||
mockApiClient.getPageState.mockReturnValue(pageState);
|
||||
mockApiClient.downloadModel.mockResolvedValue({ success: true });
|
||||
manager.versions = [{ id: 250 }, { id: 100 }];
|
||||
|
||||
const result = await manager.executeDownloadWithProgress({
|
||||
modelId: 837884,
|
||||
versionId: 250,
|
||||
versionName: 'v2',
|
||||
targetFolder: 'Some/SubFolder',
|
||||
useDefaultPaths: false,
|
||||
source: 'civitai',
|
||||
});
|
||||
|
||||
expect(result).toBe(true);
|
||||
// The download destination folder must never become the active folder.
|
||||
expect(pageState).toEqual({ showUpdateAvailableOnly: true });
|
||||
expect(setStorageItemMock).not.toHaveBeenCalledWith(
|
||||
expect.stringContaining('_activeFolder'),
|
||||
expect.anything()
|
||||
);
|
||||
// Card reconciled in place; no full page reload, no scroll reset.
|
||||
expect(mockScroller.removeMultipleItemsByFilePath).toHaveBeenCalledWith([
|
||||
'/models/loras/old.safetensors',
|
||||
]);
|
||||
expect(resetAndReloadMock).not.toHaveBeenCalled();
|
||||
expect(sidebarRefreshMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('falls back to a full reload when no virtual scroller is available', async () => {
|
||||
stateMock.virtualScroller = undefined;
|
||||
mockApiClient.downloadModel.mockResolvedValue({ success: true });
|
||||
|
||||
const result = await manager.executeDownloadWithProgress({
|
||||
modelId: 837884,
|
||||
versionId: 250,
|
||||
source: 'civitai',
|
||||
});
|
||||
|
||||
expect(result).toBe(true);
|
||||
expect(resetAndReloadMock).toHaveBeenCalledWith(true);
|
||||
});
|
||||
|
||||
it('passes through an explicit isLatestVersion flag', async () => {
|
||||
stateMock.virtualScroller.items = [makeItem('/models/loras/old.safetensors', 837884)];
|
||||
mockApiClient.getPageState.mockReturnValue({ showUpdateAvailableOnly: true });
|
||||
mockApiClient.downloadModel.mockResolvedValue({ success: true });
|
||||
manager.versions = [{ id: 250 }];
|
||||
|
||||
await manager.executeDownloadWithProgress({
|
||||
modelId: 837884,
|
||||
versionId: 100,
|
||||
source: 'civitai',
|
||||
isLatestVersion: false,
|
||||
});
|
||||
|
||||
// Deliberately downloading an older version keeps the card.
|
||||
expect(mockScroller.removeMultipleItemsByFilePath).not.toHaveBeenCalled();
|
||||
expect(mockScroller.updateSingleItem).not.toHaveBeenCalled();
|
||||
expect(resetAndReloadMock).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('_downloadSelectedFilesSequentially success path', () => {
|
||||
it('reconciles in place once all files of the latest version are downloaded', async () => {
|
||||
stateMock.virtualScroller.items = [makeItem('/models/loras/old.safetensors', 837884)];
|
||||
mockApiClient.getPageState.mockReturnValue({ showUpdateAvailableOnly: true });
|
||||
mockApiClient.downloadModel.mockResolvedValue({ success: true });
|
||||
manager.modelId = '837884';
|
||||
manager.currentVersion = { id: 201 };
|
||||
manager.versions = [{ id: 201 }, { id: 100 }];
|
||||
manager.source = 'civitai';
|
||||
manager.selectedFiles = [
|
||||
{ id: 1, name: 'a.safetensors', type: 'Model', sizeKB: 10 },
|
||||
{ id: 2, name: 'b.safetensors', type: 'Model', sizeKB: 10 },
|
||||
];
|
||||
|
||||
const result = await manager._downloadSelectedFilesSequentially({
|
||||
modelRoot: '/models/loras',
|
||||
targetFolder: '',
|
||||
useDefaultPaths: true,
|
||||
});
|
||||
|
||||
expect(result).toBe(true);
|
||||
expect(resetAndReloadMock).not.toHaveBeenCalled();
|
||||
expect(mockScroller.removeMultipleItemsByFilePath).toHaveBeenCalledWith([
|
||||
'/models/loras/old.safetensors',
|
||||
]);
|
||||
expect(sidebarRefreshMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('keeps the listing untouched on partial multi-file failure', async () => {
|
||||
stateMock.virtualScroller.items = [makeItem('/models/loras/old.safetensors', 837884)];
|
||||
mockApiClient.getPageState.mockReturnValue({ showUpdateAvailableOnly: true });
|
||||
mockApiClient.downloadModel
|
||||
.mockResolvedValueOnce({ success: true })
|
||||
.mockResolvedValueOnce({ success: false, error: 'rate limited' });
|
||||
manager.modelId = '837884';
|
||||
manager.currentVersion = { id: 201 };
|
||||
manager.versions = [{ id: 201 }, { id: 100 }];
|
||||
manager.source = 'civitai';
|
||||
manager.selectedFiles = [
|
||||
{ id: 1, name: 'a.safetensors', type: 'Model', sizeKB: 10 },
|
||||
{ id: 2, name: 'b.safetensors', type: 'Model', sizeKB: 10 },
|
||||
];
|
||||
|
||||
const result = await manager._downloadSelectedFilesSequentially({
|
||||
modelRoot: '/models/loras',
|
||||
targetFolder: '',
|
||||
useDefaultPaths: true,
|
||||
});
|
||||
|
||||
expect(result).toBe(false);
|
||||
expect(mockScroller.removeMultipleItemsByFilePath).not.toHaveBeenCalled();
|
||||
expect(resetAndReloadMock).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -154,6 +154,65 @@ class TestAdaptiveConcurrencyController:
|
||||
controller.record_result(duration=5.0, success=True)
|
||||
assert controller.current_concurrency == 3
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_semaphore_returns_shared_instance(self):
|
||||
controller = AdaptiveConcurrencyController(initial_concurrency=3)
|
||||
# Every item of a batch must receive the same semaphore so the
|
||||
# concurrency bound is actually enforced batch-wide.
|
||||
first = controller.get_semaphore()
|
||||
second = controller.get_semaphore()
|
||||
assert first is second
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_shared_semaphore_limits_concurrent_tasks(self):
|
||||
controller = AdaptiveConcurrencyController(initial_concurrency=3)
|
||||
semaphore = controller.get_semaphore()
|
||||
active = 0
|
||||
peak = 0
|
||||
|
||||
async def worker():
|
||||
nonlocal active, peak
|
||||
async with semaphore:
|
||||
active += 1
|
||||
peak = max(peak, active)
|
||||
await asyncio.sleep(0.05)
|
||||
active -= 1
|
||||
|
||||
await asyncio.gather(*[worker() for _ in range(10)])
|
||||
assert peak == 3
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_apply_concurrency_increases_capacity(self):
|
||||
controller = AdaptiveConcurrencyController(initial_concurrency=3)
|
||||
semaphore = controller.get_semaphore()
|
||||
controller.record_result(duration=0.5, success=True) # 3 -> 4
|
||||
await controller.apply_concurrency()
|
||||
|
||||
acquired = await asyncio.gather(
|
||||
*[asyncio.wait_for(semaphore.acquire(), timeout=0.2) for _ in range(4)]
|
||||
)
|
||||
assert all(acquired)
|
||||
with pytest.raises(asyncio.TimeoutError):
|
||||
await asyncio.wait_for(semaphore.acquire(), timeout=0.05)
|
||||
for _ in range(4):
|
||||
semaphore.release()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_apply_concurrency_decreases_capacity(self):
|
||||
controller = AdaptiveConcurrencyController(initial_concurrency=3)
|
||||
semaphore = controller.get_semaphore()
|
||||
controller.record_result(duration=1.0, success=False) # 3 -> 2
|
||||
await controller.apply_concurrency()
|
||||
|
||||
acquired = await asyncio.gather(
|
||||
*[asyncio.wait_for(semaphore.acquire(), timeout=0.2) for _ in range(2)]
|
||||
)
|
||||
assert all(acquired)
|
||||
with pytest.raises(asyncio.TimeoutError):
|
||||
await asyncio.wait_for(semaphore.acquire(), timeout=0.05)
|
||||
for _ in range(2):
|
||||
semaphore.release()
|
||||
|
||||
|
||||
class TestBatchImportProgress:
|
||||
def test_to_dict(self):
|
||||
@@ -596,3 +655,103 @@ class TestInputValidation:
|
||||
assert service._validate_local_path("../etc/passwd") is False
|
||||
assert service._validate_local_path("relative/path.png") is False
|
||||
assert service._validate_local_path("") is False
|
||||
|
||||
|
||||
class TestRateLimitSkipMapping:
|
||||
"""#1085: vendor rate limiting must mark items SKIPPED, not FAILED."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_services(self):
|
||||
ws_manager = MockWebSocketManager()
|
||||
persistence_service = MockPersistenceService()
|
||||
logger = logging.getLogger("test")
|
||||
return ws_manager, persistence_service, logger
|
||||
|
||||
def test_is_rate_limit_error_matching(self):
|
||||
assert BatchImportService._is_rate_limit_error("Rate limited") is True
|
||||
assert BatchImportService._is_rate_limit_error(
|
||||
"Rate limit wait for 'civarchive.com' exceeds the 300s cap"
|
||||
) is True
|
||||
assert BatchImportService._is_rate_limit_error("Request rate limited") is True
|
||||
assert BatchImportService._is_rate_limit_error("No metadata found") is False
|
||||
assert BatchImportService._is_rate_limit_error(None) is False
|
||||
assert BatchImportService._is_rate_limit_error("") is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rate_limited_item_becomes_skipped_and_sets_flag(self, mock_services):
|
||||
ws_manager, persistence_service, logger = mock_services
|
||||
analysis_service = MockAnalysisService(
|
||||
{
|
||||
"https://example.com/limited.png": MockAnalysisResult(
|
||||
{"error": "Rate limited"}
|
||||
),
|
||||
}
|
||||
)
|
||||
service = BatchImportService(
|
||||
analysis_service=analysis_service, # pyright: ignore[reportArgumentType]
|
||||
persistence_service=persistence_service,
|
||||
ws_manager=ws_manager,
|
||||
logger=logger,
|
||||
)
|
||||
|
||||
operation_id = await service.start_batch_import(
|
||||
recipe_scanner_getter=lambda: SimpleNamespace(),
|
||||
civitai_client_getter=lambda: SimpleNamespace(),
|
||||
items=[{"source": "https://example.com/limited.png"}],
|
||||
)
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
# The operation may already be cleaned up; inspect the broadcasts.
|
||||
final = next(
|
||||
(
|
||||
b
|
||||
for b in reversed(ws_manager.broadcasts)
|
||||
if b.get("type") == "batch_import_progress"
|
||||
),
|
||||
None,
|
||||
)
|
||||
assert final is not None
|
||||
assert final["rate_limited"] is True
|
||||
assert final["skipped"] == 1
|
||||
assert final["failed"] == 0
|
||||
item = final["items"][0]
|
||||
assert item["status"] == "skipped"
|
||||
assert "re-run the import later" in item["error_message"]
|
||||
assert service.get_progress(operation_id) is None or True
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_non_rate_limit_error_stays_failed(self, mock_services):
|
||||
ws_manager, persistence_service, logger = mock_services
|
||||
analysis_service = MockAnalysisService(
|
||||
{
|
||||
"https://example.com/broken.png": MockAnalysisResult(
|
||||
{"error": "No metadata found"}
|
||||
),
|
||||
}
|
||||
)
|
||||
service = BatchImportService(
|
||||
analysis_service=analysis_service, # pyright: ignore[reportArgumentType]
|
||||
persistence_service=persistence_service,
|
||||
ws_manager=ws_manager,
|
||||
logger=logger,
|
||||
)
|
||||
|
||||
await service.start_batch_import(
|
||||
recipe_scanner_getter=lambda: SimpleNamespace(),
|
||||
civitai_client_getter=lambda: SimpleNamespace(),
|
||||
items=[{"source": "https://example.com/broken.png"}],
|
||||
)
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
final = next(
|
||||
(
|
||||
b
|
||||
for b in reversed(ws_manager.broadcasts)
|
||||
if b.get("type") == "batch_import_progress"
|
||||
),
|
||||
None,
|
||||
)
|
||||
assert final is not None
|
||||
assert final["rate_limited"] is False
|
||||
assert final["failed"] == 1
|
||||
assert final["skipped"] == 0
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import copy
|
||||
import logging
|
||||
from typing import Any, Dict
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
@@ -257,3 +258,49 @@ async def test_get_model_by_hash_propagates_rate_limit(downloader):
|
||||
|
||||
assert exc_info.value.retry_after == 5
|
||||
assert exc_info.value.provider == "civarchive_api"
|
||||
|
||||
|
||||
async def test_get_model_by_hash_empty_error_payload(downloader):
|
||||
"""An empty-string failure payload must surface as a proper error.
|
||||
|
||||
Regression test: (None, "") used to fall through the falsy-error check and
|
||||
crash in _resolve_version_from_files with "'NoneType' object has no
|
||||
attribute 'get'".
|
||||
"""
|
||||
|
||||
async def fake_make_request(method, url, use_auth=False, **kwargs):
|
||||
return False, ""
|
||||
|
||||
downloader.make_request = fake_make_request
|
||||
|
||||
client = await CivArchiveClient.get_instance()
|
||||
|
||||
result, error = await client.get_model_by_hash("empty-error")
|
||||
|
||||
assert result is None
|
||||
assert error == "Request failed"
|
||||
|
||||
|
||||
async def test_get_model_version_offline_cooldown_logged_as_debug(downloader, caplog):
|
||||
"""Cooldown short-circuits must not spam an ERROR per request."""
|
||||
|
||||
async def fake_make_request(method, url, use_auth=False, **kwargs):
|
||||
return False, "offline_cooldown"
|
||||
|
||||
downloader.make_request = fake_make_request
|
||||
|
||||
client = await CivArchiveClient.get_instance()
|
||||
|
||||
with caplog.at_level(logging.DEBUG, logger="py.services.civarchive_client"):
|
||||
result = await client.get_model_version(model_id=1, version_id=123)
|
||||
|
||||
assert result is None
|
||||
module_records = [r for r in caplog.records if r.name == "py.services.civarchive_client"]
|
||||
assert not any(
|
||||
r.levelno >= logging.ERROR and "Error fetching CivArchive model version" in r.getMessage()
|
||||
for r in module_records
|
||||
)
|
||||
assert any(
|
||||
r.levelno == logging.DEBUG and "while offline" in r.getMessage()
|
||||
for r in module_records
|
||||
)
|
||||
|
||||
@@ -277,3 +277,72 @@ async def test_parse_metadata_without_extra_metadata(monkeypatch):
|
||||
assert "error" not in result
|
||||
assert result["loras"] == []
|
||||
assert result["checkpoint"]["id"] == "456"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_parse_metadata_with_list_ckpt_name(monkeypatch):
|
||||
"""ckpt_name serialized as a single-element list must not crash re.search."""
|
||||
checkpoint_info = {
|
||||
"id": 456,
|
||||
"modelId": 123,
|
||||
"model": {"name": "Checkpoint", "type": "checkpoint"},
|
||||
"name": "v1",
|
||||
"baseModel": "SDXL 1.0",
|
||||
}
|
||||
|
||||
async def fake_metadata_provider():
|
||||
class Provider:
|
||||
async def get_model_version_info(self, version_id):
|
||||
assert version_id == "456"
|
||||
return checkpoint_info, None
|
||||
|
||||
return Provider()
|
||||
|
||||
monkeypatch.setattr(
|
||||
"py.recipes.parsers.comfy.get_default_metadata_provider",
|
||||
fake_metadata_provider,
|
||||
)
|
||||
|
||||
metadata_json = {
|
||||
"1": {
|
||||
"class_type": "CheckpointLoaderSimple",
|
||||
"inputs": {"ckpt_name": ["urn:air:sdxl:checkpoint:civitai:123@456"]},
|
||||
}
|
||||
}
|
||||
|
||||
result = await ComfyMetadataParser().parse_metadata(json.dumps(metadata_json))
|
||||
|
||||
assert "error" not in result
|
||||
assert result["checkpoint"] is not None
|
||||
assert int(result["checkpoint"]["id"]) == 456
|
||||
assert int(result["checkpoint"]["modelId"]) == 123
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_parse_metadata_with_none_ckpt_name(monkeypatch):
|
||||
"""Missing (None) ckpt_name must not crash re.search with a TypeError."""
|
||||
|
||||
async def fake_metadata_provider():
|
||||
class Provider:
|
||||
async def get_model_version_info(self, version_id):
|
||||
raise AssertionError("Checkpoint lookup must be skipped")
|
||||
|
||||
return Provider()
|
||||
|
||||
monkeypatch.setattr(
|
||||
"py.recipes.parsers.comfy.get_default_metadata_provider",
|
||||
fake_metadata_provider,
|
||||
)
|
||||
|
||||
metadata_json = {
|
||||
"1": {
|
||||
"class_type": "CheckpointLoaderSimple",
|
||||
"inputs": {"ckpt_name": None},
|
||||
}
|
||||
}
|
||||
|
||||
result = await ComfyMetadataParser().parse_metadata(json.dumps(metadata_json))
|
||||
|
||||
assert "error" not in result
|
||||
assert result["checkpoint"] is None
|
||||
assert result["loras"] == []
|
||||
|
||||
@@ -812,3 +812,56 @@ async def test_fetch_and_update_model_does_not_overwrite_api_metadata_with_archi
|
||||
|
||||
helpers.metadata_manager.save_metadata.assert_awaited()
|
||||
update_cache.assert_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_and_update_model_keeps_sqlite_last_resort_after_civarchive_rate_limit(tmp_path):
|
||||
"""A CivArchive 429 must not block the local sqlite last resort (#1085)."""
|
||||
civarchive_provider = SimpleNamespace(
|
||||
get_model_by_hash=AsyncMock(
|
||||
side_effect=RateLimitError("limited", retry_after=30)
|
||||
),
|
||||
get_model_version=AsyncMock(),
|
||||
)
|
||||
sqlite_payload = {
|
||||
"source": "archive_db",
|
||||
"model": {"name": "Recovered", "description": "", "tags": []},
|
||||
"images": [],
|
||||
"baseModel": "sdxl",
|
||||
}
|
||||
sqlite_provider = SimpleNamespace(
|
||||
get_model_by_hash=AsyncMock(return_value=(sqlite_payload, None)),
|
||||
get_model_version=AsyncMock(),
|
||||
)
|
||||
|
||||
async def select_provider(name: str):
|
||||
if name == "civarchive_api":
|
||||
return civarchive_provider
|
||||
if name == "sqlite":
|
||||
return sqlite_provider
|
||||
raise AssertionError(f"unexpected provider request: {name}")
|
||||
|
||||
helpers = build_service(
|
||||
settings_values={"enable_metadata_archive_db": True},
|
||||
provider_selector=AsyncMock(side_effect=select_provider),
|
||||
)
|
||||
|
||||
model_path = tmp_path / "model.safetensors"
|
||||
model_data = {
|
||||
"civitai_deleted": True,
|
||||
"db_checked": False,
|
||||
"file_path": str(model_path),
|
||||
}
|
||||
update_cache = AsyncMock()
|
||||
|
||||
ok, error = await helpers.service.fetch_and_update_model(
|
||||
sha256="cafe",
|
||||
file_path=str(model_path),
|
||||
model_data=model_data,
|
||||
update_cache_func=update_cache,
|
||||
)
|
||||
|
||||
assert ok and error is None
|
||||
civarchive_provider.get_model_by_hash.assert_awaited_once()
|
||||
sqlite_provider.get_model_by_hash.assert_awaited_once()
|
||||
assert model_data["metadata_source"] == "archive_db"
|
||||
|
||||
@@ -101,7 +101,9 @@ async def test_fallback_retries_same_provider_on_rate_limit(monkeypatch):
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fallback_continues_to_next_provider_on_rate_limit(monkeypatch):
|
||||
"""After exhausting retries on primary, fallback should continue to secondary."""
|
||||
"""#1085: a rate-limited network provider no longer fails over to another
|
||||
network provider (that just spreads the flood); local providers such as
|
||||
sqlite remain as a last resort."""
|
||||
sleep_mock = AsyncMock()
|
||||
monkeypatch.setattr(provider_module.asyncio, "sleep", sleep_mock)
|
||||
monkeypatch.setattr(provider_module.random, "uniform", lambda *_: 0.0)
|
||||
@@ -114,13 +116,26 @@ async def test_fallback_continues_to_next_provider_on_rate_limit(monkeypatch):
|
||||
rate_limit_retry_limit=2,
|
||||
)
|
||||
|
||||
# After Change A: no longer raises; falls through to secondary
|
||||
result, error = await fallback.get_model_by_hash("abc")
|
||||
|
||||
# Secondary is a network provider: it must NOT be consulted after the 429.
|
||||
assert result is None
|
||||
assert error == "Rate limited"
|
||||
assert primary.calls == 2 # retry_limit exhausted on primary
|
||||
assert secondary.calls == 0 # no network failover
|
||||
|
||||
# A local sqlite provider behind the rate-limited one is still allowed.
|
||||
sqlite = TrackingProvider()
|
||||
fallback = FallbackMetadataProvider(
|
||||
[("primary", AlwaysRateLimitedProvider()), ("sqlite", sqlite)],
|
||||
rate_limit_retry_limit=2,
|
||||
)
|
||||
|
||||
result, error = await fallback.get_model_by_hash("abc")
|
||||
|
||||
assert error is None
|
||||
assert result == {"id": "secondary"}
|
||||
assert primary.calls == 2 # retry_limit exhausted on primary
|
||||
assert secondary.calls == 1 # secondary IS called now
|
||||
assert sqlite.calls == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -0,0 +1,463 @@
|
||||
"""Tests for the per-destination rate-limit gate (#1085).
|
||||
|
||||
Covers the RateLimitCoordinator itself, its integration into
|
||||
``Downloader.make_request``, the failover semantics change in
|
||||
``FallbackMetadataProvider``, and the ``_RateLimitRetryHelper`` double-wait
|
||||
fix.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from datetime import datetime
|
||||
from types import SimpleNamespace
|
||||
from typing import Any, Dict, Optional
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
|
||||
from py.services.connectivity_guard import ConnectivityGuard
|
||||
from py.services.downloader import Downloader
|
||||
from py.services.errors import RateLimitError
|
||||
from py.services.model_metadata_provider import (
|
||||
FallbackMetadataProvider,
|
||||
_RateLimitRetryHelper,
|
||||
)
|
||||
from py.services.rate_limit_coordinator import RateLimitCoordinator
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_singletons():
|
||||
RateLimitCoordinator._instance = None
|
||||
ConnectivityGuard._instance = None
|
||||
yield
|
||||
RateLimitCoordinator._instance = None
|
||||
ConnectivityGuard._instance = None
|
||||
|
||||
|
||||
def _patch_gate_settings(monkeypatch, **overrides):
|
||||
"""Override the coordinator's settings reads for the test."""
|
||||
monkeypatch.setattr(
|
||||
RateLimitCoordinator,
|
||||
"_setting",
|
||||
staticmethod(lambda key, default: overrides.get(key, default)),
|
||||
)
|
||||
|
||||
|
||||
async def _make_coordinator(monkeypatch, **overrides) -> RateLimitCoordinator:
|
||||
_patch_gate_settings(monkeypatch, **overrides)
|
||||
return await RateLimitCoordinator.get_instance()
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# Coordinator unit tests
|
||||
|
||||
|
||||
async def test_pacing_enforces_min_interval(monkeypatch):
|
||||
coordinator = await _make_coordinator(
|
||||
monkeypatch, rate_limit_min_interval_seconds=0.1
|
||||
)
|
||||
start = time.monotonic()
|
||||
await coordinator.wait_for_slot("example.com")
|
||||
await coordinator.wait_for_slot("example.com")
|
||||
elapsed = time.monotonic() - start
|
||||
assert elapsed >= 0.1
|
||||
|
||||
|
||||
async def test_pacing_is_per_destination(monkeypatch):
|
||||
coordinator = await _make_coordinator(
|
||||
monkeypatch, rate_limit_min_interval_seconds=0.2
|
||||
)
|
||||
await coordinator.wait_for_slot("a.example.com")
|
||||
start = time.monotonic()
|
||||
await coordinator.wait_for_slot("b.example.com")
|
||||
elapsed = time.monotonic() - start
|
||||
assert elapsed < 0.1
|
||||
|
||||
|
||||
async def test_register_rate_limit_arms_cooldown_and_waits(monkeypatch):
|
||||
coordinator = await _make_coordinator(
|
||||
monkeypatch,
|
||||
rate_limit_min_interval_seconds=0.0,
|
||||
rate_limit_max_wait_seconds=5.0,
|
||||
)
|
||||
coordinator.register_rate_limit("example.com", retry_after=0.15)
|
||||
assert coordinator.in_cooldown("example.com")
|
||||
assert 0.1 < coordinator.remaining_seconds("example.com") <= 0.15
|
||||
|
||||
start = time.monotonic()
|
||||
await coordinator.wait_for_slot("example.com")
|
||||
elapsed = time.monotonic() - start
|
||||
assert elapsed >= 0.14
|
||||
assert not coordinator.in_cooldown("example.com")
|
||||
|
||||
|
||||
async def test_concurrent_waiters_share_one_cooldown_window(monkeypatch):
|
||||
"""Herd test: N waiters wake after ~one window, not N windows."""
|
||||
coordinator = await _make_coordinator(
|
||||
monkeypatch,
|
||||
rate_limit_min_interval_seconds=0.0,
|
||||
rate_limit_max_wait_seconds=5.0,
|
||||
)
|
||||
coordinator.register_rate_limit("example.com", retry_after=0.2)
|
||||
|
||||
start = time.monotonic()
|
||||
await asyncio.gather(
|
||||
*(coordinator.wait_for_slot("example.com") for _ in range(4))
|
||||
)
|
||||
elapsed = time.monotonic() - start
|
||||
# 4 independent windows would take ~0.8s; a shared window is ~0.2s.
|
||||
assert 0.19 <= elapsed < 0.5
|
||||
|
||||
|
||||
async def test_backoff_grows_on_consecutive_429_and_resets_on_success(
|
||||
monkeypatch,
|
||||
):
|
||||
coordinator = await _make_coordinator(
|
||||
monkeypatch, rate_limit_min_interval_seconds=0.0
|
||||
)
|
||||
coordinator.register_rate_limit("example.com", retry_after=None)
|
||||
first = coordinator.remaining_seconds("example.com")
|
||||
assert 29.0 < first <= 30.0
|
||||
|
||||
coordinator.register_rate_limit("example.com", retry_after=None)
|
||||
second = coordinator.remaining_seconds("example.com")
|
||||
assert 59.0 < second <= 60.0
|
||||
|
||||
coordinator.register_success("example.com")
|
||||
coordinator.register_rate_limit("example.com", retry_after=None)
|
||||
third = coordinator.remaining_seconds("example.com")
|
||||
assert 29.0 < third <= 30.0
|
||||
|
||||
|
||||
async def test_wait_beyond_cap_raises_rate_limit_error(monkeypatch):
|
||||
coordinator = await _make_coordinator(
|
||||
monkeypatch,
|
||||
rate_limit_min_interval_seconds=0.0,
|
||||
rate_limit_max_wait_seconds=0.05,
|
||||
)
|
||||
coordinator.register_rate_limit("example.com", retry_after=30.0)
|
||||
|
||||
start = time.monotonic()
|
||||
with pytest.raises(RateLimitError) as excinfo:
|
||||
await coordinator.wait_for_slot("example.com")
|
||||
elapsed = time.monotonic() - start
|
||||
assert elapsed < 1.0 # refused immediately instead of parking
|
||||
assert excinfo.value.retry_after is not None
|
||||
assert excinfo.value.retry_after > 1.0
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# Downloader integration tests
|
||||
|
||||
|
||||
class _FakeResponse:
|
||||
def __init__(
|
||||
self,
|
||||
status: int,
|
||||
payload: Optional[Dict[str, Any]] = None,
|
||||
headers: Optional[Dict[str, str]] = None,
|
||||
):
|
||||
self.status = status
|
||||
self._payload = payload
|
||||
self.headers = headers or {}
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, exc_type, exc, tb):
|
||||
return False
|
||||
|
||||
async def json(self):
|
||||
if self._payload is None:
|
||||
raise ValueError("no json payload")
|
||||
return self._payload
|
||||
|
||||
async def text(self):
|
||||
return ""
|
||||
|
||||
|
||||
class _FakeSession:
|
||||
def __init__(self, responses):
|
||||
self._responses = list(responses)
|
||||
self.requests = []
|
||||
|
||||
def request(self, method, url, headers=None, **kwargs):
|
||||
self.requests.append({"method": method, "url": url})
|
||||
assert self._responses, "unexpected extra request"
|
||||
return self._responses.pop(0)
|
||||
|
||||
def get(self, url, headers=None, **kwargs):
|
||||
return self.request("GET", url, headers=headers, **kwargs)
|
||||
|
||||
def head(self, url, headers=None, **kwargs):
|
||||
return self.request("HEAD", url, headers=headers, **kwargs)
|
||||
|
||||
async def close(self):
|
||||
return None
|
||||
|
||||
|
||||
def _build_downloader(responses) -> Downloader:
|
||||
downloader = Downloader()
|
||||
fake_session = _FakeSession(responses)
|
||||
downloader._session = fake_session # pyright: ignore[reportAttributeAccessIssue]
|
||||
downloader._session_created_at = datetime.now()
|
||||
downloader._proxy_url = None
|
||||
|
||||
async def _noop_create_session():
|
||||
downloader._session = fake_session # pyright: ignore[reportAttributeAccessIssue]
|
||||
downloader._session_created_at = datetime.now()
|
||||
downloader._proxy_url = None
|
||||
|
||||
downloader._create_session = _noop_create_session # type: ignore[assignment]
|
||||
return downloader
|
||||
|
||||
|
||||
async def test_make_request_waits_out_429_then_resends(monkeypatch):
|
||||
_patch_gate_settings(
|
||||
monkeypatch,
|
||||
rate_limit_gate_enabled=True,
|
||||
rate_limit_min_interval_seconds=0.0,
|
||||
rate_limit_max_wait_seconds=5.0,
|
||||
)
|
||||
downloader = _build_downloader(
|
||||
[
|
||||
_FakeResponse(429, headers={"Retry-After": "1"}),
|
||||
_FakeResponse(200, payload={"ok": True}),
|
||||
]
|
||||
)
|
||||
|
||||
start = time.monotonic()
|
||||
success, payload = await downloader.make_request(
|
||||
"GET", "https://api.example.com/models/1"
|
||||
)
|
||||
elapsed = time.monotonic() - start
|
||||
|
||||
assert success is True
|
||||
assert payload == {"ok": True}
|
||||
assert len(downloader._session.requests) == 2
|
||||
assert elapsed >= 0.9
|
||||
|
||||
|
||||
async def test_make_request_paces_consecutive_calls(monkeypatch):
|
||||
_patch_gate_settings(
|
||||
monkeypatch,
|
||||
rate_limit_gate_enabled=True,
|
||||
rate_limit_min_interval_seconds=0.15,
|
||||
rate_limit_max_wait_seconds=5.0,
|
||||
)
|
||||
downloader = _build_downloader(
|
||||
[_FakeResponse(200, payload={}), _FakeResponse(200, payload={})]
|
||||
)
|
||||
|
||||
start = time.monotonic()
|
||||
await downloader.make_request("GET", "https://api.example.com/a")
|
||||
await downloader.make_request("GET", "https://api.example.com/b")
|
||||
elapsed = time.monotonic() - start
|
||||
assert elapsed >= 0.14
|
||||
|
||||
|
||||
async def test_make_request_gate_disabled_returns_429_immediately(monkeypatch):
|
||||
_patch_gate_settings(monkeypatch, rate_limit_gate_enabled=False)
|
||||
downloader = _build_downloader(
|
||||
[_FakeResponse(429, headers={"Retry-After": "30"})]
|
||||
)
|
||||
|
||||
start = time.monotonic()
|
||||
success, payload = await downloader.make_request(
|
||||
"GET", "https://api.example.com/models/1"
|
||||
)
|
||||
elapsed = time.monotonic() - start
|
||||
|
||||
assert success is False
|
||||
assert isinstance(payload, RateLimitError)
|
||||
assert payload.retry_after == 30.0
|
||||
# Gate was off: the error is NOT marked, so retry helpers keep their
|
||||
# legacy behavior.
|
||||
assert getattr(payload, "gate_handled", False) is False
|
||||
assert len(downloader._session.requests) == 1
|
||||
assert elapsed < 1.0
|
||||
|
||||
|
||||
async def test_make_request_refuses_wait_beyond_cap(monkeypatch):
|
||||
_patch_gate_settings(
|
||||
monkeypatch,
|
||||
rate_limit_gate_enabled=True,
|
||||
rate_limit_min_interval_seconds=0.0,
|
||||
rate_limit_max_wait_seconds=0.2,
|
||||
)
|
||||
downloader = _build_downloader(
|
||||
[_FakeResponse(429, headers={"Retry-After": "3600"})]
|
||||
)
|
||||
|
||||
start = time.monotonic()
|
||||
success, payload = await downloader.make_request(
|
||||
"GET", "https://api.example.com/models/1"
|
||||
)
|
||||
elapsed = time.monotonic() - start
|
||||
|
||||
assert success is False
|
||||
assert isinstance(payload, RateLimitError)
|
||||
assert payload.gate_handled is True
|
||||
assert len(downloader._session.requests) == 1
|
||||
assert elapsed < 1.0
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# FallbackMetadataProvider failover semantics (Fix C)
|
||||
|
||||
|
||||
def _stub_provider(*, result=None, error=None, exc: Exception | None = None):
|
||||
if exc is not None:
|
||||
call = AsyncMock(side_effect=exc)
|
||||
else:
|
||||
call = AsyncMock(return_value=(result, error))
|
||||
return SimpleNamespace(get_model_by_hash=call)
|
||||
|
||||
|
||||
async def test_fallback_does_not_fail_over_to_network_provider_on_429(monkeypatch):
|
||||
# The stub error is not gate_handled, so the retry helper would sleep
|
||||
# retry_after between attempts; patch it out (the helper's own behavior
|
||||
# is covered by the double-wait tests below).
|
||||
monkeypatch.setattr(
|
||||
"py.services.model_metadata_provider.asyncio.sleep", AsyncMock()
|
||||
)
|
||||
civitai = _stub_provider(exc=RateLimitError("limited", retry_after=30))
|
||||
civarchive = _stub_provider(result={"id": 1}, error=None)
|
||||
sqlite = _stub_provider(result=None, error="not in archive")
|
||||
|
||||
fallback = FallbackMetadataProvider(
|
||||
[
|
||||
("civitai_api", civitai),
|
||||
("civarchive_api", civarchive),
|
||||
("sqlite", sqlite),
|
||||
]
|
||||
)
|
||||
|
||||
result, error = await fallback.get_model_by_hash("deadbeef")
|
||||
|
||||
assert result is None
|
||||
assert error == "Rate limited"
|
||||
civarchive.get_model_by_hash.assert_not_called() # no network failover
|
||||
sqlite.get_model_by_hash.assert_called_once() # local last resort kept
|
||||
|
||||
|
||||
async def test_fallback_still_fails_over_on_not_found():
|
||||
civitai = _stub_provider(result=None, error="Model not found")
|
||||
civarchive = _stub_provider(result={"id": 1}, error=None)
|
||||
|
||||
fallback = FallbackMetadataProvider(
|
||||
[("civitai_api", civitai), ("civarchive_api", civarchive)]
|
||||
)
|
||||
|
||||
result, _ = await fallback.get_model_by_hash("deadbeef")
|
||||
|
||||
assert result == {"id": 1}
|
||||
civarchive.get_model_by_hash.assert_called_once()
|
||||
|
||||
|
||||
async def test_fallback_404_failover_still_works_after_rate_limit_change():
|
||||
"""A 404 from the first network provider still reaches the second."""
|
||||
civitai = _stub_provider(result=None, error="Resource not found")
|
||||
civarchive = _stub_provider(result={"id": 2}, error=None)
|
||||
|
||||
fallback = FallbackMetadataProvider(
|
||||
[("civitai_api", civitai), ("civarchive_api", civarchive)]
|
||||
)
|
||||
|
||||
result, _ = await fallback.get_model_by_hash("deadbeef")
|
||||
assert result == {"id": 2}
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# _RateLimitRetryHelper double-wait fix
|
||||
|
||||
|
||||
async def test_retry_helper_does_not_sleep_for_gate_handled_errors():
|
||||
calls = 0
|
||||
|
||||
async def failing():
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
error = RateLimitError("limited", retry_after=30)
|
||||
error.gate_handled = True
|
||||
raise error
|
||||
|
||||
helper = _RateLimitRetryHelper()
|
||||
start = time.monotonic()
|
||||
with pytest.raises(RateLimitError) as excinfo:
|
||||
await helper.run("civitai_api", failing)
|
||||
elapsed = time.monotonic() - start
|
||||
|
||||
assert calls == 1 # propagated immediately, no retry loop
|
||||
assert elapsed < 1.0
|
||||
assert excinfo.value.provider == "civitai_api"
|
||||
|
||||
|
||||
async def test_retry_helper_keeps_legacy_retry_for_ungated_errors():
|
||||
calls = 0
|
||||
|
||||
async def failing():
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
raise RateLimitError("limited", retry_after=None)
|
||||
|
||||
helper = _RateLimitRetryHelper(
|
||||
retry_limit=2, base_delay=0.01, max_delay=0.05, jitter_ratio=0.0
|
||||
)
|
||||
with pytest.raises(RateLimitError):
|
||||
await helper.run("civitai_api", failing)
|
||||
|
||||
assert calls == 2 # legacy retry behavior unchanged
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# Download-path 429 registration (Phase 2)
|
||||
|
||||
|
||||
class _FakeDownloadResponse(_FakeResponse):
|
||||
async def read(self):
|
||||
return b"data"
|
||||
|
||||
|
||||
async def test_download_to_memory_429_registers_cooldown(monkeypatch):
|
||||
_patch_gate_settings(
|
||||
monkeypatch,
|
||||
rate_limit_gate_enabled=True,
|
||||
rate_limit_min_interval_seconds=0.0,
|
||||
)
|
||||
downloader = _build_downloader(
|
||||
[_FakeDownloadResponse(429, headers={"Retry-After": "120"})]
|
||||
)
|
||||
|
||||
success, error, _ = await downloader.download_to_memory(
|
||||
"https://api.example.com/preview.png"
|
||||
)
|
||||
|
||||
assert success is False
|
||||
assert "Rate limited" in error
|
||||
coordinator = await RateLimitCoordinator.get_instance()
|
||||
remaining = coordinator.remaining_seconds("api.example.com")
|
||||
assert 110.0 < remaining <= 120.0
|
||||
|
||||
|
||||
async def test_get_response_headers_429_registers_cooldown(monkeypatch):
|
||||
_patch_gate_settings(
|
||||
monkeypatch,
|
||||
rate_limit_gate_enabled=True,
|
||||
rate_limit_min_interval_seconds=0.0,
|
||||
)
|
||||
downloader = _build_downloader(
|
||||
[_FakeResponse(429, headers={"Retry-After": "60"})]
|
||||
)
|
||||
|
||||
success, error = await downloader.get_response_headers(
|
||||
"https://api.example.com/model/file.safetensors"
|
||||
)
|
||||
|
||||
assert success is False
|
||||
assert "rate limited" in error.lower()
|
||||
coordinator = await RateLimitCoordinator.get_instance()
|
||||
remaining = coordinator.remaining_seconds("api.example.com")
|
||||
assert 50.0 < remaining <= 60.0
|
||||
Reference in New Issue
Block a user