diff --git a/docs/i18n-translation-guidelines.md b/docs/i18n-translation-guidelines.md index 568465ca..843a0a7a 100644 --- a/docs/i18n-translation-guidelines.md +++ b/docs/i18n-translation-guidelines.md @@ -151,6 +151,12 @@ Locales: `en`, `zh-CN`, `zh-TW`, `ja`, `ko`, `fr`, `de`, `es`, `ru`, `he` (RTL). > nothing is tracked for alerts). Register follows each file's existing norm: 你 (zh-CN), > 您 (zh-TW), Sie (de), tú (es), вы (ru). No remaining `[TODO: Translate]` placeholders. +> **Status (2026-10, scoped scan):** the Refresh dropdown gained a per-root scope section +> (`loras.controls.refresh.scopeSection` / `.rootOffline` / `.rootModels`) and the scan result +> toasts (`toast.api.refreshCompleteScoped`, `.refreshKeptUnreachable`, `.scanRootUnreachable`), +> 6 keys in total. They are `[TODO: Translate]` placeholders in the other 9 locales for now — +> the feature owner has not asked for this pass yet. + > **Status (2026-10, reconcile walk progress):** a regular Refresh now reports the reconcile > walk per model root (which roots are being checked, how many model files have been seen, and > an ETA) and splits the progress bar into walk (0-50 %) and new-file (50-99 %) phases, which diff --git a/docs/plans/issue-1108-scoped-scan.md b/docs/plans/issue-1108-scoped-scan.md new file mode 100644 index 00000000..c121045f --- /dev/null +++ b/docs/plans/issue-1108-scoped-scan.md @@ -0,0 +1,283 @@ +# Plan: Scoped Scan (one root / one folder at a time) + +**Issue:** [#1108](https://github.com/willmiao/ComfyUI-Lora-Manager/issues/1108) — scan a single +folder/root instead of the whole library. +**Status:** **P1 implemented** (2026-10-07, uncommitted working tree). P2 (folder scope, Wave 5) +is not implemented. Supersedes the earlier draft's per-root *status panel* and persisted +*unreachable-subtree* state (both dropped, see "Must NOT have"). + +Implementation notes / deviations from the draft below: + +* The scope dataclass is public (`ReconcileScope`) because it crosses the scanner → service → + route boundary; the draft called it `_ReconcileScope`. +* `_reconcile_cache()` now returns a summary dict, exposed as + `ModelScanner.last_reconcile_summary` and returned by `BaseModelService.scan_models()`, so the + scan HTTP response carries the counts (the WS `completed` payload carries the same fields). +* Root labels are computed over the **whole configured set** and passed into the walk tracker, so + the progress line and the menu never disagree. +* Discovery: `Config._dedupe_existing_paths()` drops roots that do not exist **at config load**, so + a drive that is already off when LM starts is not in `get_model_roots()` at all. Two consequences: + it cannot appear in the refresh menu (nothing to select), and its cached entries are kept because + they are outside every configured root prefix. The `root_unreachable` path therefore covers the + *mid-session* case (drive switched off while LM runs), which is the workflow that motivated the + issue; both paths keep the entries, which is what matters. +* Known limitation to keep in mind: after a scoped scan (or one with unreadable paths) the recorded + empty-folder list is unioned instead of replaced, so an empty folder deleted on disk can linger in + the sidebar until the next full refresh. Models are unaffected. + +## TL;DR (For humans) + +**What you'll get:** the Refresh ▾ menu can scan **one model root** (one drive) and leave every +other root untouched. Refreshing while a drive is switched off no longer deletes that drive's +models from the cache/DB — they are kept and reported. Browsing the grid while a drive is off no +longer silently strips `preview_url` from that drive's models. + +**Why this approach:** a scan scope is a **path scope**, not a device scope, so it also works for +layouts where one root contains symlinks to other drives. The safety net collapses into one rule — +*anything this walk could not read is left untouched* — and that rule is nearly free: the root +existence check already exists, `os.walk(onerror=...)` only fires on failure, and the cached-entry +prefix attribution is already computed for the walk-progress weights. + +**What it will NOT do:** no active probing of nested symlinks; no persisted "unreachable subtree" +table; no per-card offline badges (separate change); no root enable/disable switch; the main +Refresh button keeps meaning "scan everything"; no configuration-convention/documentation push to +re-layout libraries that use cross-drive symlinks (dropped by request). + +**Effort:** Medium (~1.5–2 days for P1 including tests) +**Risk:** Medium-low — the risk sits in the folder-tree (`all_folders`) merge under a scoped scan +and in the exact pruning predicate; both get explicit tests. + +**Decisions to sanity-check:** +1. A full Refresh now treats a *configured but unreachable* root as "keep + report" instead of + "everything under it was deleted". This is a behaviour change (release-note it). +2. Keep the near-zero-cost **first-level symlink** reachability check (reuses + `config._path_mappings`); nested symlinks stay uncovered. +3. Root labels become set-aware (`G: loras` / `usb/loras`, auto-deduped by parent segments). + This also changes the walk-progress line. + +## Scope + +### Must have (P1) + +- **Scope-aware reconcile**: `_reconcile_cache(scope=...)` where scope is + `{roots: [path] | None, folder: rel | None}`. Files inside the scope reconcile normally + (add / repair / remove); everything outside is neither re-read nor removed. +- **Path-level pruning guard**: cached entries under a path this walk could not read are excluded + from `missing` and reported. Three sources: + - configured root that fails the reachability check (already filtered at + `model_scanner.py:1468`), + - directories `os.walk` failed to enter (`onerror` collector; covers Windows junctions to an + offline drive, permission errors, I/O errors), + - first-level symlink mappings whose target is not a directory (`config._path_mappings`). +- **Folder tree correctness under scope**: `all_folders` becomes + `(old outside scope) ∪ (old under unreachable paths) ∪ (discovered)`, so a scoped scan cannot + collapse the sidebar tree and an offline root keeps its folders. +- **Result payload** for the toast: `added`, `removed`, `repaired`, `scanned_roots` (labels), + `skipped_roots` (`{path, label, kept}`), `unavailable_paths` (`{path, kept}`, capped), + `kept_unreachable`. +- **API**: `GET /api/lm/{prefix}/scan?full_rebuild=false&roots=` (repeatable); + `GET /api/lm/{prefix}/roots` gains `root_details: [{path, label, reachable, models}]` while + keeping `roots: [str]` unchanged (other callers depend on it). +- **Frontend**: Refresh ▾ menu lists the current page's roots (label + model count, offline rows + greyed out and non-clickable), main button unchanged, root items disabled while a scan runs; + scoped completion toast reports counts and any kept/skipped paths. +- **Preview fix**: a preview 404 must not clear `preview_url` when the file's parent directory is + itself unreachable (`preview_handlers.py:57` + `_cleanup_stale_preview_url`). +- **Set-aware root labels** (`_root_display_labels(roots)`), shared by the progress broadcasts and + `/roots`: last path segment, Windows drive prefix (`G: loras`), deduped by prepending real parent + segments (`usb/loras`, `a/models/loras`), `(2)` fallback by sorted path, 40-char cap, full path in + the tooltip. +- i18n keys in `locales/en.json` + `python scripts/sync_translation_keys.py`. +- pytest + vitest coverage for every item above. + +### Must have (P2, optional follow-up) + +- `folder=` parameter: walk `/` for every reachable root that contains it (the + sidebar's unified tree has no root identity, so "this folder" means "this relative path in all + roots"), prune only inside that prefix. +- Sidebar folder context menu entry "Scan this folder" (`templates/components/context_menu.html` + `#sidebarFolderContextMenu`, above `check-folder-updates`; update + `tests/frontend/regression/sidebarFolderContextMenu.test.js` expectations). + +### Must NOT have (guardrails) + +- NO per-entry symlink probing during the walk — `DirEntry.is_symlink()` costs a syscall per entry + on Windows, the exact cost the reconcile optimisation removed. Nested symlinks stay uncovered. +- NO persisted `unavailable_paths` table, no DB schema change, no new cache state. +- NO configuration-convention guidance for cross-drive symlinks (dropped by request). +- NO per-card/modal offline badges in this plan (separate change; needs the L1 prefix list which + P1 produces, so it can build on this later). +- NO root enable/disable setting, NO status panel in Settings. +- NO change to `full_rebuild=true` (it still walks everything and replaces the cache); + `roots` + `full_rebuild=true` is rejected with 400. +- NO new dependency, NO change to the extension-facing endpoints. +- NO `os.path.realpath` for scope/prune routing — business paths only (AGENTS.md rule). + +## Todos + +### Wave 1 — backend semantics and safety net + +1. **Scope plumbing in the scanner.** + What to do: add `_ReconcileScope` (dataclass: `roots: Optional[List[str]]`, `folder: Optional[str]`) + and `_reconcile_cache(scope=None)`. Resolve the effective root list once: configured roots ∩ + scope.roots, filtered by reachability; missing ones become `skipped_roots` (never pruned). + `_walk_roots_for_reconcile` / `_walk_root_group_sync` / `_walk_root_for_reconcile` take the scope + so the walk can start at `/` while still computing `folder`/`file_path` relative to + the **root** (`_process_model_file(path, root_path)` must keep receiving the root). + References: `py/services/model_scanner.py:1453` (`_reconcile_cache`), `:1468` (root filter), + `:1832` (`_walk_root_group_sync`), `:365` (`_walk_root_for_reconcile`), `:134` + (`_count_cached_entries_per_root`), `:126` (`_normalized_root_prefix`). + Done when: a scoped reconcile over 2 roots leaves the other root's `raw_data`, hash index and + folder list byte-identical. + +2. **Pruning predicate + unreachable collection.** + What to do: `missing = {p for p in cached_paths - found_paths if in_scope(p) and not + under_unreachable(p)}`; collect unreachable prefixes from (a) skipped roots, (b) an `onerror` + callback on `os.walk` (record `oserror.filename` as a business path), (c) `config._path_mappings` + entries whose target fails `os.path.isdir` (mapping is keyed target→link; protect the **link** + prefix). Count the kept entries per prefix for reporting. + References: `py/services/model_scanner.py:1679` (`missing_files = cached_paths - found_paths`), + `:398` (`os.walk(..., followlinks=True)`), `py/config.py:167` (`_path_mappings`), + `:801` (`add_path_mapping`, target→link), `:750-775` (first-level-only symlink scan). + Done when: with a root removed from disk (or a directory replaced by an unreadable junction), + `raw_data` keeps those entries, `removed` is 0, and the counts land in the result payload. + +3. **Folder-tree merge under scope.** + What to do: replace the unconditional `sorted_discovered` assignment with the union rule from + "Must have"; keep the `folders_changed` comparison and the persist path unchanged. Unscoped + scans with nothing unreachable must produce exactly today's list. + References: `py/services/model_scanner.py` (walk loop records `discovered_folders`; `model_scanner.py:1738` + (`sorted_discovered`) and `:1739` (`folders_changed`) after the dedup pass). + Done when: a scoped scan keeps folders outside the scope, an offline root keeps its subtree, and + a full scan still drops folders deleted on disk. + +4. **Result payload + progress/cancel/complete messages.** + What to do: extend the `completed` broadcast with `scanned_roots`, `skipped_roots`, + `unavailable_paths`, `kept_unreachable`, `repaired`; the walk tracker already receives only the + scoped roots, so the progress line shows that root's label with `roots_total=1`. + References: `py/services/model_scanner.py` `_broadcast_scan_progress` (`:576`), + `_ReconcileWalkTracker` (`:266`), the completed broadcast at the end of `_reconcile_cache`. + Done when: a scoped scan emits `stage=reconcile_scan` with the single scoped label and a + `completed` payload carrying the new fields. + +5. **API surface.** + What to do: `scan_models` accepts repeated `roots` query params (400 on unknown root, 400 when + combined with `full_rebuild=true`); `get_model_roots` adds `root_details` (label, reachable, + model count via the shared prefix attribution) without touching `roots`. + References: `py/routes/handlers/model_handlers.py:1139` (`scan_models`), `:1165` + (`get_model_roots`), `py/routes/model_route_registrar.py:61-62`, `static/js/api/apiConfig.js:114`. + Done when: `tests/routes/test_lora_routes.py`-style coverage passes for the new params and the + backward-compatible `/roots` payload. + +6. **Preview 404 must not prune when the parent is unreachable.** + What to do: in `serve_preview`, before `_cleanup_stale_preview_url`, check + `os.path.isdir(os.path.dirname(resolved))`; if the directory itself is missing/unreachable, + return 404 **without** clearing caches. (Centralized sidecar mode puts the preview in the mirror + tree, so the check must use the preview file's own parent.) + References: `py/routes/handlers/preview_handlers.py:57` (404 branch), `:74-110` + (`_cleanup_stale_preview_url` → `clear_preview_by_path` + `_persist_current_cache`), + `tests/routes/test_preview_routes.py`. + Done when: a test asserts the cache keeps `preview_url` when the parent dir is absent and still + clears it when the file was really deleted from a reachable directory. + +### Wave 2 — labels + +7. **`_root_display_labels(roots)` (set-aware, order-independent).** + What to do: replace the single-root `_root_display_label`; compute over the sorted root list; + prepend real parent segments until unique; `(2)`/`(3)` fallback; 40-char middle-ellipsis; return + `{path: label}`. Use it in `_ReconcileWalkTracker.__init__` (progress) and in the `/roots` + handler so both always agree. + References: `py/services/model_scanner.py:117` (`_root_display_label`, current single-root + version), `:266` (`_ReconcileWalkTracker`). + Done when: unit tests cover `usb/loras` + `ssd/loras`, `a/models/loras` + `b/models/loras`, + Windows `G:\x\loras` + `H:\y\loras` → `G: loras` / `H: loras`, and the identical-path fallback. + +### Wave 3 — frontend + +8. **Refresh ▾ scope section.** + What to do: add an empty container to `templates/components/controls.html` inside + `.dropdown-menu` (`#refreshScopeMenu`) + a section title; on dropdown open, fetch + `endpoints.roots`, render one row per root (`data-action="scan-root"`, `data-root=""`, + label + model count, `disabled` + "Offline" for unreachable), hide the section on the recipes + page (recipes have no roots); wire clicks by **event delegation** on the menu (the existing + `full-rebuild` item is bound with a direct `querySelector`, dynamic rows cannot be); + invalidate the cached list after a scan finishes; add CSS for the section + `max-height`/ + `overflow-y` so 4+ roots stay usable. + References: `templates/components/controls.html:67-76`, `static/js/components/controls/PageControls.js:131` + (`data-action="refresh"`), `:220-260` (`initDropdowns` + `full-rebuild` wiring), + `static/js/api/apiConfig.js:118` (`roots` endpoint), `static/js/api/baseModelApi.js:1318` + (`fetchModelRoots`). + Done when: the menu lists roots with counts, an offline root is visibly disabled, and clicking a + row starts a scoped refresh. + +9. **Scoped request + toast.** + What to do: `refreshModels(fullRebuild, {roots})` appends repeated `roots` params; the + `completed` payload drives the toast (`added`/`removed`/`kept_unreachable`/`skipped_roots`); + offline rows toast `… is not reachable right now. Nothing was changed.` without a request. + References: `static/js/api/baseModelApi.js:514-602` (refreshModels, URL build at `:601`, toast + at `:618`), `static/js/components/controls/PageControls.js:485`. + Done when: the frontend test asserts the scoped URL and both toast variants. + +10. **i18n.** + What to do: add `loras.controls.refresh.scopeSection`, `.rootOffline`, `.rootModels`, + `toast.api.refreshCompleteScoped`, `toast.api.refreshKeptUnreachable`, + `toast.api.scanRootUnreachable` to `locales/en.json`, run + `python scripts/sync_translation_keys.py`, then stop (placeholders are the expected state until + translations are requested). + Done when: `pytest tests/i18n` passes and every locale has the keys. + +### Wave 4 — tests and verification + +11. **Backend tests** (`tests/services/test_model_scanner.py`, `tests/routes/test_lora_routes.py`, + `tests/routes/test_preview_routes.py`): + `test_reconcile_scoped_scan_leaves_other_roots_untouched`, + `test_reconcile_scoped_scan_removes_deleted_files_in_scope`, + `test_reconcile_scoped_scan_preserves_folder_tree_outside_scope`, + `test_reconcile_keeps_entries_of_unreachable_root`, + `test_reconcile_keeps_entries_under_unreadable_dir`, + `test_reconcile_keeps_entries_under_offline_first_level_symlink`, + `test_root_display_labels_dedupe_by_parent_segments`, + `test_scan_models_accepts_roots_param`, `test_scan_rejects_unknown_root`, + `test_scan_rejects_roots_with_full_rebuild`, `test_roots_endpoint_reports_details`, + `test_preview_404_keeps_cache_when_parent_dir_missing`. + +12. **Frontend tests**: extend `tests/frontend/api/baseModelApi.refresh.test.js` (scoped URL, both + toasts) and add `tests/frontend/components/controls/pageControls.scanRoots.test.js` + (rows rendered from `/roots`, offline row disabled, delegation wiring). + +13. **Full suites + sandbox eyeball**: `pytest -q`, `npx vitest run`, `npm run test:vue`; then a + sandboxed standalone instance with two roots and a slowed walk + (`LM_WALK_DELAY_S`-style `sitecustomize` hook) to eyeball the scoped progress line, the offline + row and the toast — the user verifies by eye. + +### Wave 5 — P2 (folder scope), only if approved + +14. `folder=` in the scan endpoint + `_ReconcileScope.folder`, the sidebar menu entry, the + context-menu regression test update, and the "folder in several roots" semantics (scan every + reachable root that has the relative path; report per root). + +## Known limitations (accepted) + +- **Nested symlinks** (a symlink *below* a root pointing at another drive) are only covered when + `os.walk` fails to enter the target (Windows junctions, permission errors). A broken symlink that + `os.walk` classifies as a non-directory (typical on POSIX) is not detected: its entries are pruned + as today and come back on the next scan once the target is reachable (sidecars carry the sha256, + so no re-hash). Verify the Windows junction behaviour before writing release notes. +- Only **first-level** symlinks are known to `config` (by design, `config.py:750-775`), so the + symlink reachability check inherits that limit. +- A scoped scan of a folder does not re-read metadata for unchanged files (unchanged behaviour). +- Empty folders removed on disk can linger in the sidebar after a scoped scan (the folder list is + unioned rather than replaced whenever the scan did not verify every root); the next full refresh + drops them. +- `root_details[].reachable` is a live `os.path.exists()` per root, so a root that disappears + mid-session shows as offline; a root that was already gone at startup is filtered out by + `Config` and therefore absent from the list. + +## Verification checklist + +- [ ] `pytest -q` green, `npx vitest run` green, `npm run test:vue` green. +- [ ] `python scripts/sync_translation_keys.py --dry-run` reports no pending changes. +- [ ] Sandbox: 2 roots, one offline → scoped scan of the online root reports `added`, keeps the + offline root's models, and the grid/sidebar still show them. +- [ ] Sandbox: preview of an offline root's model returns 404 and the DB keeps `preview_url`. +- [ ] Release note wording agreed for the behaviour change (decision 1). diff --git a/locales/de.json b/locales/de.json index 9912af2e..ae043631 100644 --- a/locales/de.json +++ b/locales/de.json @@ -918,7 +918,10 @@ "refresh": { "title": "Modelliste aktualisieren", "full": "Cache neu aufbauen", - "fullTooltip": "Alle Modelldetails aus Metadatendateien neu laden – nutzen, wenn die Bibliothek veraltet wirkt oder nach manuellen Änderungen." + "fullTooltip": "Alle Modelldetails aus Metadatendateien neu laden – nutzen, wenn die Bibliothek veraltet wirkt oder nach manuellen Änderungen.", + "scopeSection": "[TODO: Translate] Scan one folder", + "rootOffline": "[TODO: Translate] Offline", + "rootModels": "[TODO: Translate] {count} models" }, "fetch": { "title": "Metadaten von CivitAI abrufen", @@ -2804,7 +2807,10 @@ "exampleImagesDownloadFailed": "Fehler beim Herunterladen der Beispielbilder: {message}", "moveFailed": "Element konnte nicht verschoben werden: {message}", "copiedToClipboard": "In die Zwischenablage kopiert", - "downloadStarted": "Download gestartet" + "downloadStarted": "Download gestartet", + "refreshCompleteScoped": "[TODO: Translate] Scanned {scope}: {added} new, {removed} removed", + "refreshKeptUnreachable": "[TODO: Translate] {count} models kept: {paths} not reachable", + "scanRootUnreachable": "[TODO: Translate] {scope} is not reachable right now. Nothing was changed." }, "agent": { "llmNotConfigured": "KI-Anbieter nicht konfiguriert. Aktivieren Sie ihn unter Einstellungen → KI-Anbieter.", diff --git a/locales/en.json b/locales/en.json index 14caaeda..ff08bf1e 100644 --- a/locales/en.json +++ b/locales/en.json @@ -918,7 +918,10 @@ "refresh": { "title": "Refresh model list", "full": "Rebuild Cache", - "fullTooltip": "Reload all model details from metadata files—use if the library looks out of date or after manual edits." + "fullTooltip": "Reload all model details from metadata files—use if the library looks out of date or after manual edits.", + "scopeSection": "Scan one folder", + "rootOffline": "Offline", + "rootModels": "{count} models" }, "fetch": { "title": "Fetch metadata from CivitAI", @@ -2804,7 +2807,10 @@ "exampleImagesDownloadFailed": "Failed to download example images: {message}", "moveFailed": "Failed to move item: {message}", "copiedToClipboard": "Copied to clipboard", - "downloadStarted": "Download started" + "downloadStarted": "Download started", + "refreshCompleteScoped": "Scanned {scope}: {added} new, {removed} removed", + "refreshKeptUnreachable": "{count} models kept: {paths} not reachable", + "scanRootUnreachable": "{scope} is not reachable right now. Nothing was changed." }, "agent": { "llmNotConfigured": "AI provider not configured. Enable it in Settings → AI Provider.", diff --git a/locales/es.json b/locales/es.json index 1248cadb..42229a4b 100644 --- a/locales/es.json +++ b/locales/es.json @@ -918,7 +918,10 @@ "refresh": { "title": "Actualizar lista de modelos", "full": "Reconstruir caché", - "fullTooltip": "Vuelve a cargar todos los detalles desde los archivos de metadatos; úsalo si la biblioteca parece desactualizada o tras ediciones manuales." + "fullTooltip": "Vuelve a cargar todos los detalles desde los archivos de metadatos; úsalo si la biblioteca parece desactualizada o tras ediciones manuales.", + "scopeSection": "[TODO: Translate] Scan one folder", + "rootOffline": "[TODO: Translate] Offline", + "rootModels": "[TODO: Translate] {count} models" }, "fetch": { "title": "Obtener metadatos de CivitAI", @@ -2804,7 +2807,10 @@ "exampleImagesDownloadFailed": "Error al descargar imágenes de ejemplo: {message}", "moveFailed": "No se pudo mover el elemento: {message}", "copiedToClipboard": "Copiado al portapapeles", - "downloadStarted": "Descarga iniciada" + "downloadStarted": "Descarga iniciada", + "refreshCompleteScoped": "[TODO: Translate] Scanned {scope}: {added} new, {removed} removed", + "refreshKeptUnreachable": "[TODO: Translate] {count} models kept: {paths} not reachable", + "scanRootUnreachable": "[TODO: Translate] {scope} is not reachable right now. Nothing was changed." }, "agent": { "llmNotConfigured": "Proveedor de IA no configurado. Actívelo en Configuración → Proveedor de IA.", diff --git a/locales/fr.json b/locales/fr.json index 4ff0e3fa..5aff7118 100644 --- a/locales/fr.json +++ b/locales/fr.json @@ -918,7 +918,10 @@ "refresh": { "title": "Actualiser la liste des modèles", "full": "Reconstruire le cache", - "fullTooltip": "Recharge tous les détails des modèles depuis les fichiers de métadonnées — à utiliser si la bibliothèque paraît obsolète ou après des modifications manuelles." + "fullTooltip": "Recharge tous les détails des modèles depuis les fichiers de métadonnées — à utiliser si la bibliothèque paraît obsolète ou après des modifications manuelles.", + "scopeSection": "[TODO: Translate] Scan one folder", + "rootOffline": "[TODO: Translate] Offline", + "rootModels": "[TODO: Translate] {count} models" }, "fetch": { "title": "Récupérer les métadonnées depuis CivitAI", @@ -2804,7 +2807,10 @@ "exampleImagesDownloadFailed": "Échec du téléchargement des images d'exemple : {message}", "moveFailed": "Échec du déplacement de l'élément : {message}", "copiedToClipboard": "Copié dans le presse-papiers", - "downloadStarted": "Téléchargement démarré" + "downloadStarted": "Téléchargement démarré", + "refreshCompleteScoped": "[TODO: Translate] Scanned {scope}: {added} new, {removed} removed", + "refreshKeptUnreachable": "[TODO: Translate] {count} models kept: {paths} not reachable", + "scanRootUnreachable": "[TODO: Translate] {scope} is not reachable right now. Nothing was changed." }, "agent": { "llmNotConfigured": "Fournisseur d'IA non configuré. Activez-le dans Paramètres → Fournisseur d'IA.", diff --git a/locales/he.json b/locales/he.json index ed15bde8..acd6452d 100644 --- a/locales/he.json +++ b/locales/he.json @@ -918,7 +918,10 @@ "refresh": { "title": "רענן רשימת מודלים", "full": "בניית מטמון מחדש", - "fullTooltip": "טוען מחדש את כל פרטי המודלים מקבצי המטא-נתונים – לשימוש אם הספרייה נראית לא מעודכנת או לאחר עריכות ידניות." + "fullTooltip": "טוען מחדש את כל פרטי המודלים מקבצי המטא-נתונים – לשימוש אם הספרייה נראית לא מעודכנת או לאחר עריכות ידניות.", + "scopeSection": "[TODO: Translate] Scan one folder", + "rootOffline": "[TODO: Translate] Offline", + "rootModels": "[TODO: Translate] {count} models" }, "fetch": { "title": "אחזר מטא-נתונים מ-CivitAI", @@ -2804,7 +2807,10 @@ "exampleImagesDownloadFailed": "הורדת תמונות הדוגמה נכשלה: {message}", "moveFailed": "לא ניתן היה להעביר את הפריט: {message}", "copiedToClipboard": "הועתק ללוח", - "downloadStarted": "ההורדה החלה" + "downloadStarted": "ההורדה החלה", + "refreshCompleteScoped": "[TODO: Translate] Scanned {scope}: {added} new, {removed} removed", + "refreshKeptUnreachable": "[TODO: Translate] {count} models kept: {paths} not reachable", + "scanRootUnreachable": "[TODO: Translate] {scope} is not reachable right now. Nothing was changed." }, "agent": { "llmNotConfigured": "ספק AI לא הוגדר. הפעל אותו בהגדרות → ספק AI.", diff --git a/locales/ja.json b/locales/ja.json index d0553547..f26122d0 100644 --- a/locales/ja.json +++ b/locales/ja.json @@ -918,7 +918,10 @@ "refresh": { "title": "モデルリストを更新", "full": "キャッシュを再構築", - "fullTooltip": "メタデータファイルから全モデル情報を再読み込みします。リストが古いと感じるときや手動編集後に使用してください。" + "fullTooltip": "メタデータファイルから全モデル情報を再読み込みします。リストが古いと感じるときや手動編集後に使用してください。", + "scopeSection": "[TODO: Translate] Scan one folder", + "rootOffline": "[TODO: Translate] Offline", + "rootModels": "[TODO: Translate] {count} models" }, "fetch": { "title": "CivitAIからメタデータを取得", @@ -2804,7 +2807,10 @@ "exampleImagesDownloadFailed": "例画像のダウンロードに失敗しました:{message}", "moveFailed": "アイテムを移動できませんでした: {message}", "copiedToClipboard": "クリップボードにコピーしました", - "downloadStarted": "ダウンロードを開始しました" + "downloadStarted": "ダウンロードを開始しました", + "refreshCompleteScoped": "[TODO: Translate] Scanned {scope}: {added} new, {removed} removed", + "refreshKeptUnreachable": "[TODO: Translate] {count} models kept: {paths} not reachable", + "scanRootUnreachable": "[TODO: Translate] {scope} is not reachable right now. Nothing was changed." }, "agent": { "llmNotConfigured": "AIプロバイダーが設定されていません。設定 → AIプロバイダーで有効にしてください。", diff --git a/locales/ko.json b/locales/ko.json index 89c30126..5d325b20 100644 --- a/locales/ko.json +++ b/locales/ko.json @@ -918,7 +918,10 @@ "refresh": { "title": "모델 목록 새로고침", "full": "캐시 재구성", - "fullTooltip": "메타데이터 파일에서 모든 모델 정보를 다시 불러옵니다. 라이브러리가 오래되어 보이거나 수동 수정 후에 사용하세요." + "fullTooltip": "메타데이터 파일에서 모든 모델 정보를 다시 불러옵니다. 라이브러리가 오래되어 보이거나 수동 수정 후에 사용하세요.", + "scopeSection": "[TODO: Translate] Scan one folder", + "rootOffline": "[TODO: Translate] Offline", + "rootModels": "[TODO: Translate] {count} models" }, "fetch": { "title": "CivitAI에서 메타데이터 가져오기", @@ -2804,7 +2807,10 @@ "exampleImagesDownloadFailed": "예시 이미지 다운로드 실패: {message}", "moveFailed": "항목을 이동하지 못했습니다: {message}", "copiedToClipboard": "클립보드에 복사됨", - "downloadStarted": "다운로드 시작됨" + "downloadStarted": "다운로드 시작됨", + "refreshCompleteScoped": "[TODO: Translate] Scanned {scope}: {added} new, {removed} removed", + "refreshKeptUnreachable": "[TODO: Translate] {count} models kept: {paths} not reachable", + "scanRootUnreachable": "[TODO: Translate] {scope} is not reachable right now. Nothing was changed." }, "agent": { "llmNotConfigured": "AI 제공자가 설정되지 않았습니다. 설정 → AI 제공자에서 활성화하세요.", diff --git a/locales/ru.json b/locales/ru.json index 2c86f25e..eb14ce66 100644 --- a/locales/ru.json +++ b/locales/ru.json @@ -918,7 +918,10 @@ "refresh": { "title": "Обновить список моделей", "full": "Перестроить кэш", - "fullTooltip": "Перечитывает все данные моделей из файлов метаданных — используйте, если библиотека выглядит устаревшей или после ручных правок." + "fullTooltip": "Перечитывает все данные моделей из файлов метаданных — используйте, если библиотека выглядит устаревшей или после ручных правок.", + "scopeSection": "[TODO: Translate] Scan one folder", + "rootOffline": "[TODO: Translate] Offline", + "rootModels": "[TODO: Translate] {count} models" }, "fetch": { "title": "Получить метаданные с CivitAI", @@ -2804,7 +2807,10 @@ "exampleImagesDownloadFailed": "Не удалось загрузить примеры изображений: {message}", "moveFailed": "Не удалось переместить элемент: {message}", "copiedToClipboard": "Скопировано в буфер обмена", - "downloadStarted": "Загрузка начата" + "downloadStarted": "Загрузка начата", + "refreshCompleteScoped": "[TODO: Translate] Scanned {scope}: {added} new, {removed} removed", + "refreshKeptUnreachable": "[TODO: Translate] {count} models kept: {paths} not reachable", + "scanRootUnreachable": "[TODO: Translate] {scope} is not reachable right now. Nothing was changed." }, "agent": { "llmNotConfigured": "Поставщик ИИ не настроен. Включите его в Настройки → Поставщик ИИ.", diff --git a/locales/zh-CN.json b/locales/zh-CN.json index 8018e0dc..7fedcd22 100644 --- a/locales/zh-CN.json +++ b/locales/zh-CN.json @@ -918,7 +918,10 @@ "refresh": { "title": "刷新模型列表", "full": "重建缓存", - "fullTooltip": "从元数据文件重新加载所有模型信息;用于列表过时或手动编辑后。" + "fullTooltip": "从元数据文件重新加载所有模型信息;用于列表过时或手动编辑后。", + "scopeSection": "[TODO: Translate] Scan one folder", + "rootOffline": "[TODO: Translate] Offline", + "rootModels": "[TODO: Translate] {count} models" }, "fetch": { "title": "从 CivitAI 获取元数据", @@ -2804,7 +2807,10 @@ "exampleImagesDownloadFailed": "示例图片下载失败:{message}", "moveFailed": "移动条目失败:{message}", "copiedToClipboard": "已复制到剪贴板", - "downloadStarted": "下载已开始" + "downloadStarted": "下载已开始", + "refreshCompleteScoped": "[TODO: Translate] Scanned {scope}: {added} new, {removed} removed", + "refreshKeptUnreachable": "[TODO: Translate] {count} models kept: {paths} not reachable", + "scanRootUnreachable": "[TODO: Translate] {scope} is not reachable right now. Nothing was changed." }, "agent": { "llmNotConfigured": "AI 提供商未配置。请在 设置 → AI 提供商 中进行配置。", diff --git a/locales/zh-TW.json b/locales/zh-TW.json index 0ca955b0..1f92cd9e 100644 --- a/locales/zh-TW.json +++ b/locales/zh-TW.json @@ -918,7 +918,10 @@ "refresh": { "title": "重新整理模型列表", "full": "重建快取", - "fullTooltip": "從中繼資料檔重新載入所有模型資訊;適用於清單過時或手動編輯後。" + "fullTooltip": "從中繼資料檔重新載入所有模型資訊;適用於清單過時或手動編輯後。", + "scopeSection": "[TODO: Translate] Scan one folder", + "rootOffline": "[TODO: Translate] Offline", + "rootModels": "[TODO: Translate] {count} models" }, "fetch": { "title": "從 CivitAI 取得 metadata", @@ -2804,7 +2807,10 @@ "exampleImagesDownloadFailed": "下載範例圖片失敗:{message}", "moveFailed": "移動項目失敗:{message}", "copiedToClipboard": "已複製到剪貼簿", - "downloadStarted": "下載已開始" + "downloadStarted": "下載已開始", + "refreshCompleteScoped": "[TODO: Translate] Scanned {scope}: {added} new, {removed} removed", + "refreshKeptUnreachable": "[TODO: Translate] {count} models kept: {paths} not reachable", + "scanRootUnreachable": "[TODO: Translate] {scope} is not reachable right now. Nothing was changed." }, "agent": { "llmNotConfigured": "AI 提供者尚未設定。請在 設定 → AI 提供者 中進行設定。", diff --git a/py/config.py b/py/config.py index 8ec4b9af..20f9e1a2 100644 --- a/py/config.py +++ b/py/config.py @@ -798,6 +798,16 @@ class Config: except Exception as e: logger.error(f"Error scanning links in {root}: {e}") + def iter_path_mappings(self) -> List[Tuple[str, str]]: + """Return the known ``(physical target, virtual link)`` symlink pairs. + + Only symlinks directly under a model root are tracked (see + :meth:`_scan_symbolic_links`), so callers must treat this as a partial + view of the on-disk link layout — enough to notice that a linked drive + went away, not enough to resolve nested links. + """ + return list(self._path_mappings.items()) + def add_path_mapping(self, link_path: str, target_path: str): """Add a symbolic link path mapping target_path: actual target path diff --git a/py/routes/handlers/model_handlers.py b/py/routes/handlers/model_handlers.py index 2072fb2c..1adc9d39 100644 --- a/py/routes/handlers/model_handlers.py +++ b/py/routes/handlers/model_handlers.py @@ -26,6 +26,7 @@ from ...services.connectivity_guard import ( ) from ...services.metadata_sync_service import MetadataSyncService from ...services.model_file_service import ModelMoveService +from ...services.model_scanner import ReconcileScope from ...services.preview_asset_service import PreviewAssetService from ...services.service_registry import ServiceRegistry from ...services.settings_manager import SettingsManager, get_settings_manager @@ -1139,8 +1140,30 @@ class ModelQueryHandler: async def scan_models(self, request: web.Request) -> web.Response: try: full_rebuild = request.query.get("full_rebuild", "false").lower() == "true" - await self._service.scan_models( - force_refresh=True, rebuild_cache=full_rebuild + requested_roots = [ + value for value in request.query.getall("roots", []) if value + ] + if requested_roots and full_rebuild: + return web.json_response( + { + "error": "Scoped scans are not supported with " + "full_rebuild=true; a full rebuild always walks every root" + }, + status=400, + ) + + scope = None + if requested_roots: + configured = self._service.get_model_roots() + unknown = [root for root in requested_roots if root not in configured] + if unknown: + return web.json_response( + {"error": "Unknown model root(s)", "roots": unknown}, status=400 + ) + scope = ReconcileScope(roots=tuple(requested_roots)) + + summary = await self._service.scan_models( + force_refresh=True, rebuild_cache=full_rebuild, scope=scope ) _broadcast_models_changed() if self._service.scanner.is_cancelled(): @@ -1150,12 +1173,13 @@ class ModelQueryHandler: "message": f"{self._service.model_type.capitalize()} scan cancelled", } ) - return web.json_response( - { - "status": "success", - "message": f"{self._service.model_type.capitalize()} scan completed", - } - ) + payload: Dict[str, Any] = { + "status": "success", + "message": f"{self._service.model_type.capitalize()} scan completed", + } + if summary: + payload.update(summary) + return web.json_response(payload) except Exception as exc: self._logger.error( "Error scanning %ss: %s", self._service.model_type, exc, exc_info=True @@ -1165,7 +1189,14 @@ class ModelQueryHandler: async def get_model_roots(self, request: web.Request) -> web.Response: try: roots = self._service.get_model_roots() - return web.json_response({"success": True, "roots": roots}) + try: + root_details = self._service.describe_model_roots() + except Exception as exc: # pragma: no cover - defensive + self._logger.debug("Root details unavailable: %s", exc) + root_details = [] + return web.json_response( + {"success": True, "roots": roots, "root_details": root_details} + ) except Exception as exc: self._logger.error( "Error getting %s roots: %s", diff --git a/py/routes/handlers/preview_handlers.py b/py/routes/handlers/preview_handlers.py index c4a9edf4..be28c4f8 100644 --- a/py/routes/handlers/preview_handlers.py +++ b/py/routes/handlers/preview_handlers.py @@ -54,7 +54,18 @@ class PreviewHandler: if not resolved.is_file(): logger.debug("Preview file not found at %s", str(resolved)) - asyncio.create_task(self._cleanup_stale_preview_url(normalized)) + if resolved.parent.is_dir(): + # The file is really gone from a reachable directory, so the + # cached preview_url is stale and can be cleared. + asyncio.create_task(self._cleanup_stale_preview_url(normalized)) + else: + # The directory itself is unreachable (drive switched off, + # unmounted share). Nothing was deleted: keep the cached + # preview_url so the card recovers when the drive is back. + logger.debug( + "Preview directory unreachable, keeping cached preview_url: %s", + str(resolved.parent), + ) raise web.HTTPNotFound(text="Preview file not found") # aiohttp's FileResponse handles range requests, content headers, and diff --git a/py/services/base_model_service.py b/py/services/base_model_service.py index 228192aa..b91645d5 100644 --- a/py/services/base_model_service.py +++ b/py/services/base_model_service.py @@ -27,6 +27,7 @@ from ..utils.civitai_utils import build_civitai_model_page_url logger = logging.getLogger(__name__) if TYPE_CHECKING: + from .model_scanner import ReconcileScope from .model_update_service import ModelUpdateService @@ -936,12 +937,25 @@ class BaseModelService(ABC): return self.scanner.get_hash_by_path(file_path) async def scan_models( - self, force_refresh: bool = False, rebuild_cache: bool = False - ): - """Trigger model scanning""" - return await self.scanner.get_cached_data( - force_refresh=force_refresh, rebuild_cache=rebuild_cache + self, + force_refresh: bool = False, + rebuild_cache: bool = False, + scope: Optional["ReconcileScope"] = None, + ) -> Optional[Dict[str, Any]]: + """Trigger model scanning, optionally restricted to a scope. + + Returns the reconcile summary (added / removed / repaired / skipped + roots / kept entries) for incremental scans, ``None`` for a full + rebuild. + """ + await self.scanner.get_cached_data( + force_refresh=force_refresh, rebuild_cache=rebuild_cache, scope=scope ) + return self.scanner.last_reconcile_summary + + def describe_model_roots(self) -> List[Dict[str, Any]]: + """Describe configured roots (label / reachability / cached count).""" + return self.scanner.describe_model_roots() async def get_model_info_by_name(self, name: str): """Get model information by name""" diff --git a/py/services/model_scanner.py b/py/services/model_scanner.py index 477873b6..b982e015 100644 --- a/py/services/model_scanner.py +++ b/py/services/model_scanner.py @@ -114,13 +114,84 @@ def _new_file_pass_progress(processed: int, total: int) -> int: return _WALK_PROGRESS_SHARE + min(49, int(ratio * 49)) -def _root_display_label(root: str) -> str: - """Short label identifying a model root in progress messages.""" - drive, _tail = os.path.splitdrive(root) - if drive: - return drive - normalized = root.rstrip("/\\") - return os.path.basename(normalized) or root +# Maximum length of a root label in progress messages and menus; the full path +# always stays available in the tooltip. +_ROOT_LABEL_MAX_LENGTH = 40 + + +def _truncate_label(label: str, limit: int = _ROOT_LABEL_MAX_LENGTH) -> str: + """Middle-truncate an over-long label.""" + if len(label) <= limit: + return label + keep = max(limit - 3, 1) + head = (keep + 1) // 2 + tail = keep - head + if not tail: + return f"{label[:head]}..." + return f"{label[:head]}...{label[-tail:]}" + + +def _root_path_parts(root: str) -> Tuple[str, List[str]]: + """Split a root into its drive prefix (Windows) and its path segments.""" + drive, tail = os.path.splitdrive(root) + normalized = tail.replace("\\", "/").strip("/") + return drive, [part for part in normalized.split("/") if part] + + +def _root_display_labels(roots: Sequence[str]) -> Dict[str, str]: + """Return a unique, human-friendly label for every root in the set. + + The label starts as the last path segment (prefixed with the drive letter on + Windows, e.g. ``G: loras``) and grows leftwards with **real parent path + segments** until it is unique inside the set (``usb/loras`` vs + ``ssd/loras``). Roots that still render identically (Windows case variants, + a duplicated config entry) get a deterministic numeric suffix. + + Labels are a property of the whole set, so they are computed from the sorted + root list: the same configuration always yields the same labels, whatever + order the roots are walked in. + """ + if not roots: + return {} + + parts_by_root = {root: _root_path_parts(root) for root in roots} + depth = {root: (1 if parts_by_root[root][1] else 0) for root in roots} + + def _render(root: str) -> str: + drive, parts = parts_by_root[root] + if not parts: + return drive or root + take = max(min(depth[root], len(parts)), 1) + tail = "/".join(parts[len(parts) - take:]) + return f"{drive} {tail}" if drive else tail + + # Grow the depth of colliding labels until they can be told apart. + while True: + groups: Dict[str, List[str]] = {} + for root in roots: + groups.setdefault(_render(root), []).append(root) + colliding = [group for group in groups.values() if len(group) > 1] + if not colliding: + break + grew = False + for group in colliding: + for root in group: + if depth[root] < len(parts_by_root[root][1]): + depth[root] += 1 + grew = True + if not grew: + break + + labels = {root: _render(root) for root in roots} + fallback: Dict[str, List[str]] = {} + for root in sorted(roots): + fallback.setdefault(labels[root], []).append(root) + for label, group in fallback.items(): + if len(group) > 1: + for index, root in enumerate(group, start=1): + labels[root] = f"{label} ({index})" + + return {root: _truncate_label(label) for root, label in labels.items()} def _normalized_root_prefix(root: str) -> str: @@ -131,34 +202,136 @@ def _normalized_root_prefix(root: str) -> str: return prefix.lower() if _CASE_INSENSITIVE_PATHS else prefix -def _count_cached_entries_per_root( - cached_paths: Set[str], roots: Sequence[str] -) -> Dict[str, int]: - """Attribute cached entries to model roots (longest prefix wins). +def _path_matches_prefix(path: str, prefix: str) -> bool: + """Match a cached business path against a normalized scope prefix.""" + candidate = path.lower() if _CASE_INSENSITIVE_PATHS else path + return candidate.startswith(prefix) + +def _scope_prefix(root: str, folder: Optional[str] = None) -> str: + """Normalized prefix covering a root, or a folder inside that root.""" + prefix = _normalized_root_prefix(root) + if folder: + relative = folder.replace("\\", "/").strip("/") + if relative: + if _CASE_INSENSITIVE_PATHS: + relative = relative.lower() + prefix = f"{prefix}{relative}/" + return prefix + + +def _scope_walk_path(root: str, folder: Optional[str] = None) -> str: + """Filesystem path a scoped walk starts at (the root itself when unscoped).""" + if not folder: + return root + relative = folder.replace("\\", "/").strip("/") + if not relative: + return root + return os.path.join(root, *relative.split("/")) + + +@dataclass(frozen=True) +class ReconcileScope: + """Restrict a reconcile to some of the configured roots and/or a folder. + + ``roots=None`` means "every configured root". ``folder`` is a root-relative + folder in forward-slash form and applies to each root in the scope. + """ + + roots: Optional[Tuple[str, ...]] = None + folder: Optional[str] = None + + +class _UnreachablePaths: + """Prefixes this reconcile could not read; their entries are never pruned. + + Populated from three sources: a configured root that is not reachable, a + directory ``os.walk`` failed to enter (permissions, I/O error, a Windows + junction to an offline drive), and a known first-level symlink whose target + is not a directory. Cached entries under these prefixes are reported instead + of removed, so switching a drive off can no longer wipe its models. + """ + + def __init__(self) -> None: + self._lock = threading.Lock() + self._prefixes: List[str] = [] + self._details: Dict[str, Dict[str, Any]] = {} + + def add(self, path: str, reason: str) -> None: + prefix = _normalized_root_prefix(path) + with self._lock: + if prefix in self._details: + return + self._details[prefix] = { + 'path': path.replace(os.sep, '/'), + 'reason': reason, + 'kept': 0, + } + self._prefixes.append(prefix) + + def match(self, cached_path: str) -> Optional[str]: + """Return the prefix covering ``cached_path``, if any.""" + for prefix in self._prefixes: + if _path_matches_prefix(cached_path, prefix): + return prefix + return None + + def record_kept(self, prefix: str, amount: int = 1) -> None: + detail = self._details.get(prefix) + if detail is not None: + detail['kept'] += amount + + def total_kept(self) -> int: + return sum(detail['kept'] for detail in self._details.values()) + + def count(self) -> int: + return len(self._details) + + def has_entries(self) -> bool: + return bool(self._details) + + def payload(self, limit: int = 5) -> List[Dict[str, Any]]: + """Most affected prefixes first, capped for the progress payload.""" + entries = sorted( + self._details.values(), key=lambda detail: detail['kept'], reverse=True + ) + return [dict(entry) for entry in entries[:limit]] + + +def _count_cached_entries_for_prefixes( + cached_paths: Set[str], prefixes: Sequence[Tuple[str, str]] +) -> Dict[str, int]: + """Count cached entries per ``(key, normalized prefix)`` pair. + + Longest prefix wins, so a folder scope counts only the entries inside it. Used as the walk-workload weight for progress reporting: the walk itself is what discovers the real file count, so the cached entry count is the only estimate available up front. """ - counts: Dict[str, int] = {root: 0 for root in roots} - if not roots: + counts: Dict[str, int] = {key: 0 for key, _prefix in prefixes} + if not prefixes: return counts - prefixes = sorted( - ((_normalized_root_prefix(root), root) for root in roots), - key=lambda item: len(item[0]), - reverse=True, - ) + ordered = sorted(prefixes, key=lambda item: len(item[1]), reverse=True) case_insensitive = _CASE_INSENSITIVE_PATHS for path in cached_paths: candidate = path.lower() if case_insensitive else path - for prefix, root in prefixes: + for key, prefix in ordered: if candidate.startswith(prefix): - counts[root] += 1 + counts[key] += 1 break return counts +def _count_cached_entries_per_root( + cached_paths: Set[str], roots: Sequence[str] +) -> Dict[str, int]: + """Attribute cached entries to model roots (longest prefix wins).""" + return _count_cached_entries_for_prefixes( + cached_paths, [(root, _normalized_root_prefix(root)) for root in roots] + ) + + def _root_device_key(root: str) -> str: """Group key identifying the storage device that holds a root. @@ -273,10 +446,19 @@ class _ReconcileWalkTracker: they finish. """ - def __init__(self, roots: Sequence[str], expected: Mapping[str, int]) -> None: + def __init__( + self, + roots: Sequence[str], + expected: Mapping[str, int], + labels: Optional[Mapping[str, str]] = None, + ) -> None: self._lock = threading.Lock() self._roots: List[str] = list(roots) - self._labels = {root: _root_display_label(root) for root in self._roots} + # Callers that already labelled the full configured root set pass those + # labels in, so the progress line and the refresh menu never disagree on + # how a root is named. + resolved = dict(labels) if labels else _root_display_labels(self._roots) + self._labels = {root: resolved.get(root) or root for root in self._roots} self._expected = { root: max(int(expected.get(root, 0) or 0), 0) for root in self._roots } @@ -373,15 +555,28 @@ def _walk_root_for_reconcile( excluded_models: Set[str], is_cancelled: Callable[[], bool], report_progress: Callable[[int], None], + walk_path: Optional[str] = None, + unreachable: Optional[_UnreachablePaths] = None, ) -> _RootWalkResult: """Walk one model root and classify every model file found. + ``walk_path`` defaults to the root itself; a folder scope starts the walk + deeper but still resolves ``folder``/``file_path`` against ``root_path`` so + cache entries keep their library-relative shape. + Runs synchronously (worker thread) and never mutates scanner state: all cache updates happen on the event loop once every root has been walked. """ result = _RootWalkResult(root_path=root_path) stale_seen: Set[str] = set() files_since_report = 0 + walk_start = walk_path or root_path + + def _on_walk_error(error: OSError) -> None: + """Record a directory the walk could not enter (offline/denied).""" + filename = getattr(error, 'filename', None) + if filename and unreachable is not None: + unreachable.add(str(filename), 'unreadable_dir') def mark_stale_if_needed(cached_path: str) -> None: """Queue a cached path for file_name repair when it drifted.""" @@ -395,7 +590,9 @@ def _walk_root_for_reconcile( stale_seen.add(cached_path) result.stale_paths.append(cached_path) - for root, dirnames, files in os.walk(root_path, followlinks=True): + for root, dirnames, files in os.walk( + walk_start, followlinks=True, onerror=_on_walk_error + ): dirnames[:] = [d for d in dirnames if not _is_excluded_dir(d)] real_root = os.path.realpath(root) @@ -554,6 +751,8 @@ class ModelScanner: self._defer_persist_depth = 0 self._deferred_persist_pending = False self._autov3_backfill_scheduled = False # One-time AutoV3 backfill trigger per process + # Summary of the last incremental reconcile (see last_reconcile_summary) + self._last_reconcile_summary: Optional[Dict[str, Any]] = None # Guard against concurrent all-folders backfill walks (cold fallback # for persisted snapshots that predate folder recording). self._all_folders_backfill_running = False @@ -1334,12 +1533,19 @@ class ModelScanner: asyncio.set_event_loop(None) loop.close() - async def get_cached_data(self, force_refresh: bool = False, rebuild_cache: bool = False) -> ModelCache: + async def get_cached_data( + self, + force_refresh: bool = False, + rebuild_cache: bool = False, + scope: Optional[ReconcileScope] = None, + ) -> ModelCache: """Get cached model data, refresh if needed Args: force_refresh: Whether to refresh the cache rebuild_cache: Whether to completely rebuild the cache + scope: Restrict an incremental reconcile to some roots / a folder + (ignored by a full rebuild, which always walks everything) """ # If cache is not initialized, return an empty cache # Actual initialization should be done via initialize_in_background @@ -1353,12 +1559,55 @@ class ModelScanner: # If force refresh is requested, initialize the cache directly if force_refresh: if rebuild_cache: + self._last_reconcile_summary = None await self._initialize_cache() else: - await self._reconcile_cache() + self._last_reconcile_summary = await self._reconcile_cache(scope=scope) return cast(ModelCache, self._cache) + @property + def last_reconcile_summary(self) -> Optional[Dict[str, Any]]: + """Summary of the most recent incremental reconcile (None otherwise). + + Carries the counts the refresh toast reports (added / removed / + repaired) plus the scan scope and the paths that were left untouched + because they were unreachable. + """ + return self._last_reconcile_summary + + def describe_model_roots(self) -> List[Dict[str, Any]]: + """Describe the configured roots for the refresh scope menu. + + ``models`` is the cached entry count per root (the same attribution the + walk-progress weights use) and ``reachable`` is a live filesystem check, + so a switched-off drive shows as offline without touching the cache. + """ + roots: List[str] = [] + seen: Set[str] = set() + for root in self.get_model_roots(): + if not root or root in seen: + continue + seen.add(root) + roots.append(root) + + labels = _root_display_labels(roots) + cached_paths = ( + {item.get('file_path', '') for item in self._cache.raw_data} + if self._cache is not None + else set() + ) + counts = _count_cached_entries_per_root(cached_paths, roots) + return [ + { + 'path': root.replace(os.sep, '/'), + 'label': labels[root], + 'reachable': os.path.exists(root), + 'models': counts.get(root, 0), + } + for root in roots + ] + async def _initialize_cache(self) -> None: """Initialize or refresh the cache""" self._is_initializing = True # Set flag @@ -1450,8 +1699,15 @@ class ModelScanner: finally: self._is_initializing = False # Unset flag - async def _reconcile_cache(self) -> None: - """Fast cache reconciliation - only process differences between cache and filesystem""" + async def _reconcile_cache(self, scope: Optional[ReconcileScope] = None) -> Optional[Dict[str, Any]]: + """Fast cache reconciliation - only process differences between cache and filesystem. + + ``scope`` restricts the reconcile to some of the configured roots and/or + a folder inside them: entries outside the scope are neither re-read nor + removed, so scanning one drive can never touch another. Paths the walk + could not read are kept and reported instead of being treated as + deleted. Returns a summary of the run (also broadcast to the UI). + """ self.reset_cancellation() self._is_initializing = True # Set flag for reconciliation duration try: @@ -1465,30 +1721,69 @@ class ModelScanner: cached_paths = {item['file_path'] for item in self._cache.raw_data} path_to_item = {item['file_path']: item for item in self._cache.raw_data} - # Every configured root that is currently reachable. A root that is - # missing (drive switched off, unmounted share) is skipped, so its - # cached entries are reported as missing below. - roots: List[str] = [] + # Snapshot the configured roots once for this phase: scope + # resolution, reachability, symlink relevance and the new-file pass + # all read from this list. + configured_roots: List[str] = [] seen_roots: Set[str] = set() for root_path in self.get_model_roots(): if not root_path or root_path in seen_roots: continue - if not os.path.exists(root_path): - continue seen_roots.add(root_path) + configured_roots.append(root_path) + + all_labels = _root_display_labels(configured_roots) + scope_roots = [ + root for root in configured_roots + if scope is None or scope.roots is None or root in scope.roots + ] + scope_folder = scope.folder if scope is not None else None + + # A configured root that is not reachable (drive switched off, + # unmounted share) is skipped instead of being treated as deleted. + unreachable = _UnreachablePaths() + roots: List[str] = [] + skipped_roots: List[Dict[str, Any]] = [] + for root_path in scope_roots: + if not os.path.exists(root_path): + unreachable.add(root_path, 'root_unreachable') + skipped_roots.append({ + 'path': root_path.replace(os.sep, '/'), + 'label': all_labels.get(root_path, root_path), + 'reason': 'root_unreachable', + }) + continue roots.append(root_path) + self._collect_offline_symlink_prefixes(roots, unreachable) + + # Scope prefixes come from every root the caller asked for (even the + # unreachable ones) so their entries are recognised as "in scope but + # unreadable" and reported instead of silently disappearing. + scope_prefixes = [ + (root, _scope_prefix(root, scope_folder)) for root in scope_roots + ] + walk_prefixes = [ + (root, _scope_prefix(root, scope_folder)) for root in roots + ] + # Roots on different devices are walked by parallel workers (a cold # or slow drive then no longer serializes the others); roots sharing # a device stay sequential so directory claims remain deterministic. tracker = _ReconcileWalkTracker( - roots, _count_cached_entries_per_root(cached_paths, roots) + roots, + _count_cached_entries_for_prefixes(cached_paths, walk_prefixes), + labels={root: all_labels.get(root, root) for root in roots}, ) walk_results = await self._walk_roots_for_reconcile( roots=roots, + walk_targets={ + root: _scope_walk_path(root, scope_folder) for root in roots + }, tracker=tracker, cached_paths=cached_paths, path_to_item=path_to_item, + unreachable=unreachable, ) # Final walk snapshot: the bar reaches the walk share, then the @@ -1675,8 +1970,21 @@ class ModelScanner: exc, ) - # Find missing files (in cache but not in filesystem) - missing_files = cached_paths - found_paths + # Find missing files (in cache but not in filesystem). Only paths + # inside this scan's scope are candidates, and paths the walk could + # not read are kept and reported instead of being removed. + missing_files: Set[str] = set() + for path in cached_paths - found_paths: + if not any( + _path_matches_prefix(path, prefix) + for _root, prefix in scope_prefixes + ): + continue + unreachable_prefix = unreachable.match(path) + if unreachable_prefix is not None: + unreachable.record_kept(unreachable_prefix) + continue + missing_files.add(path) total_removed = 0 if missing_files: @@ -1732,13 +2040,28 @@ class ModelScanner: self._cache.raw_data = list(reversed(deduped)) total_removed += dedup_removed - # The walk above visited every directory, so refresh the recorded - # folder list (including empty folders) even when no model files - # changed — e.g. an empty folder was created or removed externally. + # The walk above visited every directory *in scope*, so refresh the + # recorded folder list (including empty folders) when nothing was + # left unverified — e.g. an empty folder was created or removed + # externally. A scoped scan, or one with unreadable paths, can only + # add to the list: dropping folders this walk never looked at would + # empty the sidebar for every drive that was not scanned. sorted_discovered = sorted(discovered_folders, key=lambda x: x.lower()) - folders_changed = self._cache.all_folders != sorted_discovered + fully_verified = ( + len(roots) == len(configured_roots) + and scope_folder is None + and not unreachable.has_entries() + ) + if fully_verified: + merged_folders = sorted_discovered + else: + merged_folders = sorted( + set(self._cache.all_folders or []) | set(sorted_discovered), + key=lambda x: x.lower(), + ) + folders_changed = self._cache.all_folders != merged_folders if folders_changed: - self._cache.all_folders = sorted_discovered + self._cache.all_folders = merged_folders # Resort cache if changes were made if total_added > 0 or total_removed > 0: @@ -1755,16 +2078,33 @@ class ModelScanner: elif folders_changed: await self._persist_current_cache() + summary: Dict[str, Any] = { + 'added': total_added, + 'removed': total_removed, + 'repaired': total_repaired, + 'scanned_roots': [all_labels.get(root, root) for root in roots], + 'skipped_roots': skipped_roots, + 'unavailable_paths': unreachable.payload(), + 'unavailable_paths_total': unreachable.count(), + 'kept_unreachable': unreachable.total_kept(), + } logger.info( f"{self.model_type.capitalize()} Scanner: Cache reconciliation completed in " f"{time.time() - start_time:.2f} seconds. Added {total_added}, " f"removed {total_removed}, repaired {total_repaired} models." + + ( + f" Skipped {len(skipped_roots)} unreachable root(s), kept " + f"{summary['kept_unreachable']} model(s) under unreadable paths." + if skipped_roots or unreachable.has_entries() + else "" + ) ) await self._broadcast_scan_progress( 'completed', 'process_new', 100, False, - added=total_added, removed=total_removed, elapsed_seconds=time.time() - start_time, + **summary, ) + return summary except Exception as e: logger.error(f"{self.model_type.capitalize()} Scanner: Error reconciling cache: {e}", exc_info=True) await self._broadcast_scan_progress( @@ -1779,9 +2119,11 @@ class ModelScanner: self, *, roots: List[str], + walk_targets: Mapping[str, str], tracker: _ReconcileWalkTracker, cached_paths: Set[str], path_to_item: Mapping[str, Dict[str, Any]], + unreachable: _UnreachablePaths, ) -> List[_RootWalkResult]: """Walk every root off the event loop and return results in root order. @@ -1810,6 +2152,8 @@ class ModelScanner: dir_claims, excluded_models, path_to_item, + walk_targets, + unreachable, ) for group in _group_roots_by_device(roots) ] @@ -1837,6 +2181,8 @@ class ModelScanner: dir_claims: _RealDirClaims, excluded_models: Set[str], path_to_item: Mapping[str, Dict[str, Any]], + walk_targets: Mapping[str, str], + unreachable: _UnreachablePaths, ) -> List[_RootWalkResult]: """Walk the roots of one device sequentially (worker-thread entry point).""" results: List[_RootWalkResult] = [] @@ -1852,6 +2198,7 @@ class ModelScanner: try: result = _walk_root_for_reconcile( root_path=root_path, + walk_path=walk_targets.get(root_path, root_path), file_extensions=self.file_extensions, cached_paths=lookups.cached_paths, path_to_item=path_to_item, @@ -1860,6 +2207,7 @@ class ModelScanner: excluded_models=excluded_models, is_cancelled=self.is_cancelled, report_progress=report_progress, + unreachable=unreachable, ) finally: tracker.finish_root(root_path) @@ -1869,6 +2217,41 @@ class ModelScanner: break return results + def _collect_offline_symlink_prefixes( + self, roots: Sequence[str], unreachable: _UnreachablePaths + ) -> None: + """Flag known first-level symlinks whose target is not a directory. + + ``config`` deliberately tracks only symlinks directly under a root (see + ``Config._scan_symbolic_links``), so this costs one ``isdir`` per known + mapping and covers "the linked drive is switched off" for the layouts the + scanner already knows about. Nested symlinks stay out of scope. + """ + try: + mappings = config.iter_path_mappings() + except Exception as exc: # pragma: no cover - defensive logging + logger.debug( + "%s Scanner: symlink map unavailable: %s", + self.model_type.capitalize(), + exc, + ) + return + if not mappings or not roots: + return + + root_prefixes = [_normalized_root_prefix(root) for root in roots] + for target, link in mappings: + link_prefix = _normalized_root_prefix(link) + if not any( + link_prefix == root_prefix + or _path_matches_prefix(link_prefix, root_prefix) + for root_prefix in root_prefixes + ): + continue + if os.path.isdir(target): + continue + unreachable.add(link, 'symlink_target_offline') + async def _monitor_walk_progress(self, tracker: _ReconcileWalkTracker) -> None: """Broadcast walk progress while the root workers are running.""" while True: diff --git a/static/css/layout.css b/static/css/layout.css index 5e0ed103..c28d191b 100644 --- a/static/css/layout.css +++ b/static/css/layout.css @@ -700,3 +700,48 @@ body.sticky-controls .sticky-topbar { padding: 10px 0; } } + +/* Refresh dropdown: per-root scan scope (issue #1108). The list can hold one + entry per configured model root, so it scrolls instead of growing past the + viewport. Offline roots stay clickable (clicking explains the state) and are + only visually de-emphasised — unlike .dropdown-item.disabled, which swallows + the click via pointer-events: none. */ +.dropdown-separator { + height: 1px; + margin: 5px 0; + background-color: var(--border-color); +} + +.dropdown-section-title { + padding: 4px 15px 2px; + font-size: 0.85em; + font-weight: 600; + color: var(--text-color); + opacity: 0.6; +} + +.dropdown-scope-list { + max-height: 45vh; + overflow-y: auto; +} + +.scan-root-item.is-offline { + cursor: default; + opacity: 0.6; +} + +.scan-root-label { + margin-right: 12px; +} + +.scan-root-count, +.scan-root-offline { + float: right; + margin-left: 16px; + font-size: 0.9em; + opacity: 0.7; +} + +.scan-root-item.is-offline .scan-root-offline { + opacity: 0.9; +} diff --git a/static/js/api/baseModelApi.js b/static/js/api/baseModelApi.js index cce6ffa3..e00f8494 100644 --- a/static/js/api/baseModelApi.js +++ b/static/js/api/baseModelApi.js @@ -511,8 +511,9 @@ export class BaseModelApiClient { } } - async refreshModels(fullRebuild = false) { + async refreshModels(fullRebuild = false, { roots = null } = {}) { const abortController = new AbortController(); + const scopeRoots = Array.isArray(roots) ? roots.filter(Boolean) : []; const displayName = this.apiConfig.config.displayName; const singularName = this.apiConfig.config.singularName; const actionText = translate( @@ -600,6 +601,9 @@ export class BaseModelApiClient { const url = new URL(this.apiConfig.endpoints.scan, window.location.origin); url.searchParams.append('full_rebuild', fullRebuild); + for (const root of scopeRoots) { + url.searchParams.append('roots', root); + } const response = await fetch(url, { signal: abortController.signal }); @@ -615,7 +619,7 @@ export class BaseModelApiClient { resetAndReload(true); - showToast('toast.api.refreshComplete', { action: actionText }, 'success'); + this._showRefreshSummary(data, actionText, scopeRoots); } catch (error) { if (error.name === 'AbortError') { showToast('toast.api.operationCancelled', {}, 'info'); @@ -632,6 +636,59 @@ export class BaseModelApiClient { } } + /** + * Report what a finished scan did. + * + * A scoped scan names the folder it scanned and how many models changed; a + * scan that could not read part of its scope says so instead of silently + * looking like "nothing found" (the entries are kept, not deleted). + * @param {Object} summary - Scan response payload + * @param {string} actionText - Localized "Refresh" / "Full rebuild" + * @param {Array} scopeRoots - Roots the scan was restricted to + */ + _showRefreshSummary(summary, actionText, scopeRoots = []) { + const payload = summary || {}; + const scannedRoots = Array.isArray(payload.scanned_roots) ? payload.scanned_roots : []; + const scopeLabel = scannedRoots.length ? scannedRoots.join(', ') : ''; + + if (scopeRoots.length && scopeLabel) { + showToast( + 'toast.api.refreshCompleteScoped', + { + scope: scopeLabel, + added: Number(payload.added || 0), + removed: Number(payload.removed || 0), + }, + 'success' + ); + } else { + showToast('toast.api.refreshComplete', { action: actionText }, 'success'); + } + + const keptCount = Number(payload.kept_unreachable || 0); + if (keptCount > 0) { + const unavailable = Array.isArray(payload.unavailable_paths) + ? payload.unavailable_paths + : []; + const skipped = Array.isArray(payload.skipped_roots) ? payload.skipped_roots : []; + // Skipped roots already carry a short label; the remaining + // unreadable folders only have a path. + const names = ( + skipped.length + ? skipped.map(entry => entry?.label || entry?.path) + : unavailable.map(entry => entry?.path) + ).filter(Boolean); + showToast( + 'toast.api.refreshKeptUnreachable', + { + count: keptCount.toLocaleString(), + paths: names.slice(0, 3).join(', ') || '—', + }, + 'info' + ); + } + } + /** * Connect to the shared fetch-progress WebSocket for scan progress updates. * Returns null when the connection cannot be established (silent fallback). diff --git a/static/js/components/controls/CheckpointsControls.js b/static/js/components/controls/CheckpointsControls.js index 51473c31..4e65b57a 100644 --- a/static/js/components/controls/CheckpointsControls.js +++ b/static/js/components/controls/CheckpointsControls.js @@ -33,9 +33,6 @@ export class CheckpointsControls extends PageControls { return await resetAndReload(updateFolders); }, - refreshModels: async (fullRebuild = false) => { - return await getModelApiClient().refreshModels(fullRebuild); - }, // Add fetch from Civitai functionality for checkpoints fetchFromCivitai: async () => { diff --git a/static/js/components/controls/EmbeddingsControls.js b/static/js/components/controls/EmbeddingsControls.js index 6b8042df..4c200cba 100644 --- a/static/js/components/controls/EmbeddingsControls.js +++ b/static/js/components/controls/EmbeddingsControls.js @@ -30,9 +30,6 @@ export class EmbeddingsControls extends PageControls { return await resetAndReload(updateFolders); }, - refreshModels: async (fullRebuild = false) => { - return await getModelApiClient().refreshModels(fullRebuild); - }, // Add fetch from Civitai functionality for embeddings fetchFromCivitai: async () => { diff --git a/static/js/components/controls/LorasControls.js b/static/js/components/controls/LorasControls.js index b79139ac..e54a11f8 100644 --- a/static/js/components/controls/LorasControls.js +++ b/static/js/components/controls/LorasControls.js @@ -37,9 +37,6 @@ export class LorasControls extends PageControls { return await resetAndReload(updateFolders); }, - refreshModels: async (fullRebuild = false) => { - return await getModelApiClient().refreshModels(fullRebuild); - }, // LoRA-specific API functions fetchFromCivitai: async () => { diff --git a/static/js/components/controls/OtherControls.js b/static/js/components/controls/OtherControls.js index c9bb6d60..dae878ee 100644 --- a/static/js/components/controls/OtherControls.js +++ b/static/js/components/controls/OtherControls.js @@ -31,9 +31,6 @@ export class OtherControls extends PageControls { return await resetAndReload(updateFolders); }, - refreshModels: async (fullRebuild = false) => { - return await getModelApiClient().refreshModels(fullRebuild); - }, // Add fetch from Civitai functionality for other models fetchFromCivitai: async () => { diff --git a/static/js/components/controls/PageControls.js b/static/js/components/controls/PageControls.js index 294651dd..089ad9c6 100644 --- a/static/js/components/controls/PageControls.js +++ b/static/js/components/controls/PageControls.js @@ -5,7 +5,9 @@ import { showToast, openCivitaiByMetadata, isTypingContext } from '../../utils/u import { eventManager } from '../../utils/EventManager.js'; import { performModelUpdateCheck } from '../../utils/updateCheckHelpers.js'; import { sidebarManager } from '../SidebarManager.js'; +import { getModelApiClient } from '../../api/modelApiFactory.js'; import { initSortDropdown, applySortToSelect, randomizeSortValue } from './SortDropdown.js'; +import { renderScanScopeMenu, resolveScanScopeTarget } from './ScanScopeMenu.js'; /** * PageControls class - Unified control management for model pages @@ -82,7 +84,18 @@ export class PageControls { * @param {Object} api - API methods for the page */ registerAPI(api) { - this.api = api; + this.api = { + ...api, + // These two are identical for every page and must stay in sync with + // the scan API: the refresh scope menu reads the root list, and a + // scoped refresh has to pass `roots` through. They are defined here + // rather than in each page facade so a page cannot silently drop + // them (which is exactly how the scope menu ended up empty). + fetchModelRoots: async () => getModelApiClient().fetchModelRoots(), + // Forward the exact arguments: an unscoped refresh must keep calling + // the client with a single argument. + refreshModels: async (...args) => getModelApiClient().refreshModels(...args), + }; console.log(`API methods registered for ${this.pageType} page`); // Initialize sidebar manager after API is registered @@ -233,7 +246,10 @@ export class PageControls { }); // Toggle current dropdown - dropdownGroup.classList.toggle('active'); + const isOpen = dropdownGroup.classList.toggle('active'); + if (isOpen) { + this.loadScanScopeMenu(); + } }); }); @@ -248,6 +264,21 @@ export class PageControls { }); } + // Per-root scan rows are rendered dynamically, so they are wired by + // delegation instead of a direct querySelector per item. + const scopeMenu = document.getElementById('refreshScopeMenu'); + if (scopeMenu) { + scopeMenu.addEventListener('click', (e) => { + const item = e.target.closest('[data-action="scan-root"]'); + if (!item) { + return; + } + e.stopPropagation(); + this.handleScanRootSelection(item); + document.querySelector('.dropdown-group.active')?.classList.remove('active'); + }); + } + const checkUpdatesOption = document.getElementById('checkUpdatesMenuItem'); if (checkUpdatesOption) { checkUpdatesOption.addEventListener('click', async (e) => { @@ -266,6 +297,55 @@ export class PageControls { }); } + /** + * Render the per-root scan entries of the refresh dropdown. + * + * Roots come from the backend (label, model count, reachability) so the menu + * always names a root exactly like the scan progress line does. The list is + * cached until the next scan finishes. + */ + async loadScanScopeMenu() { + const menu = document.getElementById('refreshScopeMenu'); + if (!menu || typeof this.api?.fetchModelRoots !== 'function') { + return; + } + + if (this._scanScopeDetails) { + this.renderScanScopeMenu(this._scanScopeDetails); + return; + } + if (this._scanScopeLoading) { + return; + } + + this._scanScopeLoading = true; + try { + const payload = await this.api.fetchModelRoots(); + const details = Array.isArray(payload?.root_details) ? payload.root_details : []; + this._scanScopeDetails = details; + renderScanScopeMenu(menu, details); + } catch (error) { + console.error('Error loading model roots:', error); + menu.innerHTML = ''; + } finally { + this._scanScopeLoading = false; + } + } + + handleScanRootSelection(item) { + const { rootPath, label, offline } = resolveScanScopeTarget(item); + if (!rootPath) { + return; + } + if (offline) { + // showToast(key, params, type): the sentence must NOT be passed here, + // it used to land in the `type` slot and produced an unstyled toast. + showToast('toast.api.scanRootUnreachable', { scope: label }, 'info'); + return; + } + this.refreshModels(false, { roots: [rootPath] }); + } + async handleCheckModelUpdates(menuItem) { if (this._updateCheckInProgress) { return; @@ -482,14 +562,23 @@ export class PageControls { * Refresh models list * @param {boolean} fullRebuild - Whether to perform a full rebuild */ - async refreshModels(fullRebuild = false) { + async refreshModels(fullRebuild = false, { roots = null } = {}) { if (!this.api) { console.error('API methods not registered'); return; } + // Root labels / reachability may have changed with the scan, and the + // model counts definitely did. + this._scanScopeDetails = null; + const scopedRoots = Array.isArray(roots) ? roots.filter(Boolean) : []; + try { - await this.api.refreshModels(fullRebuild); + if (scopedRoots.length) { + await this.api.refreshModels(fullRebuild, { roots: scopedRoots }); + } else { + await this.api.refreshModels(fullRebuild); + } // Refresh sidebar after rebuild if (this.sidebarManager) { diff --git a/static/js/components/controls/ScanScopeMenu.js b/static/js/components/controls/ScanScopeMenu.js new file mode 100644 index 00000000..8b03cf94 --- /dev/null +++ b/static/js/components/controls/ScanScopeMenu.js @@ -0,0 +1,72 @@ +// ScanScopeMenu.js - per-root entries of the Refresh dropdown (issue #1108) +import { translate } from '../../utils/i18nHelpers.js'; + +/** + * Render one row per configured model root. + * + * Rows carry the root path in `data-root` and are wired by delegation, because + * the list is rebuilt every time the root set changes. An offline root stays + * clickable (the click explains why nothing can be scanned) and is only + * de-emphasised visually. + * + * @param {HTMLElement|null} menu - Container element (#refreshScopeMenu) + * @param {Array} details - `root_details` from GET /api/lm/{type}/roots + */ +export function renderScanScopeMenu(menu, details = []) { + if (!menu) { + return; + } + menu.innerHTML = ''; + + details.forEach(detail => { + if (!detail || !detail.path) { + return; + } + const item = document.createElement('div'); + item.className = `dropdown-item scan-root-item${detail.reachable ? '' : ' is-offline'}`; + item.dataset.action = 'scan-root'; + item.dataset.root = detail.path; + item.title = detail.path; + + const icon = document.createElement('i'); + icon.className = `fas ${detail.reachable ? 'fa-folder-open' : 'fa-plug-circle-xmark'}`; + item.appendChild(icon); + + const label = document.createElement('span'); + label.className = 'scan-root-label'; + label.textContent = detail.label || detail.path; + item.appendChild(label); + + const meta = document.createElement('span'); + if (detail.reachable) { + const count = Number(detail.models || 0).toLocaleString(); + meta.className = 'scan-root-count'; + meta.textContent = translate( + 'loras.controls.refresh.rootModels', + { count }, + `${count} models` + ); + } else { + meta.className = 'scan-root-offline'; + meta.textContent = translate('loras.controls.refresh.rootOffline', {}, 'Offline'); + } + item.appendChild(meta); + + menu.appendChild(item); + }); +} + +/** + * Read what a clicked row refers to. + * @param {HTMLElement} item - Row rendered by renderScanScopeMenu + * @returns {{rootPath: string, label: string, offline: boolean}} + */ +export function resolveScanScopeTarget(item) { + const rootPath = item?.dataset?.root || ''; + const label = item?.querySelector?.('.scan-root-label')?.textContent || rootPath; + return { + rootPath, + label, + offline: Boolean(item?.classList?.contains('is-offline')), + }; +} diff --git a/static/js/utils/uiHelpers.js b/static/js/utils/uiHelpers.js index 1f932b8c..278f70e6 100644 --- a/static/js/utils/uiHelpers.js +++ b/static/js/utils/uiHelpers.js @@ -139,9 +139,14 @@ export async function copyToClipboard(text, successMessage = null) { * @param {string} type - Toast type (info/success/warning/error) * @returns {HTMLElement} The toast element (not yet attached to the DOM) */ +// Only these carry styling (icon + accent border). An unknown type is coerced +// to `info` instead of ending up as a bare `.toast` box — a wrong argument +// order at a call site used to leak the message itself into the class list. +const TOAST_TYPES = new Set(['info', 'success', 'warning', 'error', 'copy']); + function createToastElement(message, type) { const toast = document.createElement('div'); - toast.className = `toast toast-${type}`; + toast.className = `toast toast-${TOAST_TYPES.has(type) ? type : 'info'}`; toast.textContent = message; return toast; } diff --git a/templates/components/controls.html b/templates/components/controls.html index b542371d..997d6b95 100644 --- a/templates/components/controls.html +++ b/templates/components/controls.html @@ -73,6 +73,12 @@ + {% if page_id != 'recipes' %} + + + + + {% endif %} diff --git a/tests/frontend/api/baseModelApi.refresh.test.js b/tests/frontend/api/baseModelApi.refresh.test.js index 55724efe..4d4a29d5 100644 --- a/tests/frontend/api/baseModelApi.refresh.test.js +++ b/tests/frontend/api/baseModelApi.refresh.test.js @@ -153,8 +153,8 @@ describe('BaseModelApiClient.refreshModels scan progress', () => { }; } - async function startRefresh(client, fullRebuild = false) { - const promise = client.refreshModels(fullRebuild); + async function startRefresh(client, fullRebuild = false, options = {}) { + const promise = client.refreshModels(fullRebuild, options); await vi.waitFor(() => { expect(FakeWebSocket.instances.length).toBe(1); }); @@ -392,6 +392,75 @@ describe('BaseModelApiClient.refreshModels scan progress', () => { await promise; }); + it('requests a scoped scan when roots are passed', async () => { + global.fetch = vi.fn().mockResolvedValue({ + ok: true, + json: async () => ({ status: 'success' }), + }); + + const client = await createClient(); + await client.refreshModels(false, { roots: ['/mnt/a/loras', '/mnt/b/loras'] }); + + const [url] = global.fetch.mock.calls[0]; + expect(url.searchParams.getAll('roots')).toEqual(['/mnt/a/loras', '/mnt/b/loras']); + expect(url.searchParams.get('full_rebuild')).toBe('false'); + }); + + it('reports the scoped scan summary and the entries kept unreachable', async () => { + const fetchControl = mockFetchPending(); + const client = await createClient(); + const { promise } = await startRefresh(client, false, { roots: ['/mnt/a/loras'] }); + + fetchControl.resolveOk({ + status: 'success', + scanned_roots: ['a/loras'], + added: 2, + removed: 1, + kept_unreachable: 5, + unavailable_paths: [ + { path: '/mnt/g/loras', reason: 'root_unreachable', kept: 5 }, + ], + skipped_roots: [{ path: '/mnt/g/loras', label: 'g/loras' }], + }); + await promise; + + expect(showToastMock).toHaveBeenCalledWith( + 'toast.api.refreshCompleteScoped', + { scope: 'a/loras', added: 2, removed: 1 }, + 'success' + ); + expect(showToastMock).toHaveBeenCalledWith( + 'toast.api.refreshKeptUnreachable', + { count: '5', paths: 'g/loras' }, + 'info' + ); + }); + + it('keeps the generic completion toast for a full-library scan', async () => { + const fetchControl = mockFetchPending(); + const client = await createClient(); + const { promise } = await startRefresh(client); + + fetchControl.resolveOk({ + status: 'success', + scanned_roots: ['a/loras', 'b/loras'], + added: 0, + removed: 0, + }); + await promise; + + expect(showToastMock).toHaveBeenCalledWith( + 'toast.api.refreshComplete', + { action: 'Refresh' }, + 'success' + ); + expect(showToastMock).not.toHaveBeenCalledWith( + 'toast.api.refreshCompleteScoped', + expect.anything(), + expect.anything() + ); + }); + it('shows the cancelled toast when the server reports cancellation', async () => { const fetchControl = mockFetchPending(); const client = await createClient(); diff --git a/tests/frontend/components/controls/pageControls.scanScope.test.js b/tests/frontend/components/controls/pageControls.scanScope.test.js new file mode 100644 index 00000000..a636b975 --- /dev/null +++ b/tests/frontend/components/controls/pageControls.scanScope.test.js @@ -0,0 +1,236 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; +import { readFileSync } from 'fs'; +import path from 'path'; + +const { + PAGE_CONTROLS_MODULE, + STATE_MODULE, + STORAGE_MODULE, + UI_HELPERS_MODULE, + EVENT_MANAGER_MODULE, + UPDATE_CHECK_MODULE, + SIDEBAR_MODULE, + SORT_DROPDOWN_MODULE, + API_FACTORY_MODULE, + I18N_MODULE, +} = vi.hoisted(() => ({ + PAGE_CONTROLS_MODULE: new URL('../../../../static/js/components/controls/PageControls.js', import.meta.url).pathname, + STATE_MODULE: new URL('../../../../static/js/state/index.js', import.meta.url).pathname, + STORAGE_MODULE: new URL('../../../../static/js/utils/storageHelpers.js', import.meta.url).pathname, + UI_HELPERS_MODULE: new URL('../../../../static/js/utils/uiHelpers.js', import.meta.url).pathname, + EVENT_MANAGER_MODULE: new URL('../../../../static/js/utils/EventManager.js', import.meta.url).pathname, + UPDATE_CHECK_MODULE: new URL('../../../../static/js/utils/updateCheckHelpers.js', import.meta.url).pathname, + SIDEBAR_MODULE: new URL('../../../../static/js/components/SidebarManager.js', import.meta.url).pathname, + SORT_DROPDOWN_MODULE: new URL('../../../../static/js/components/controls/SortDropdown.js', import.meta.url).pathname, + API_FACTORY_MODULE: new URL('../../../../static/js/api/modelApiFactory.js', import.meta.url).pathname, + I18N_MODULE: new URL('../../../../static/js/utils/i18nHelpers.js', import.meta.url).pathname, +})); + +const CONTROLS_TEMPLATE = path.resolve( + __dirname, + '../../../../templates/components/controls.html' +); + +const showToastMock = vi.fn(); +const modelClient = { + fetchModelRoots: vi.fn(), + refreshModels: vi.fn(), +}; + +vi.mock(STATE_MODULE, () => ({ + state: {}, + getCurrentPageState: vi.fn(() => ({ filters: {} })), + setCurrentPageType: vi.fn(), +})); + +vi.mock(STORAGE_MODULE, () => ({ + getStorageItem: vi.fn(), + setStorageItem: vi.fn(), + removeStorageItem: vi.fn(), + getSessionItem: vi.fn(), + setSessionItem: vi.fn(), + removeSessionItem: vi.fn(), +})); + +vi.mock(UI_HELPERS_MODULE, () => ({ + showToast: showToastMock, + openCivitaiByMetadata: vi.fn(), + isTypingContext: vi.fn(() => false), +})); + +vi.mock(EVENT_MANAGER_MODULE, () => ({ + eventManager: { on: vi.fn(), off: vi.fn(), emit: vi.fn() }, +})); + +vi.mock(UPDATE_CHECK_MODULE, () => ({ + performModelUpdateCheck: vi.fn(), +})); + +vi.mock(SIDEBAR_MODULE, () => ({ + sidebarManager: { + setHostPageControls: vi.fn(), + initialize: vi.fn(async () => {}), + refresh: vi.fn(async () => {}), + }, +})); + +vi.mock(SORT_DROPDOWN_MODULE, () => ({ + initSortDropdown: vi.fn(), + applySortToSelect: vi.fn(), + randomizeSortValue: vi.fn(), +})); + +vi.mock(API_FACTORY_MODULE, () => ({ + getModelApiClient: () => modelClient, +})); + +vi.mock(I18N_MODULE, () => ({ + translate: vi.fn((key, params, fallback) => { + if (!fallback) { + return key; + } + return Object.entries(params || {}).reduce( + (text, [name, value]) => text.replaceAll(`{${name}}`, value), + fallback + ); + }), +})); + +const { PageControls } = await import(PAGE_CONTROLS_MODULE); + +const ROOT_DETAILS = [ + { path: '/mnt/a/loras', label: 'a/loras', reachable: true, models: 12 }, + { path: '/mnt/b/loras', label: 'b/loras', reachable: false, models: 3 }, +]; + +function buildControlsDom() { + document.body.innerHTML = ` + `; +} + +function bareControls(api = {}) { + const controls = Object.create(PageControls.prototype); + controls.pageType = 'loras'; + controls.api = api; + controls._scanScopeDetails = null; + controls._scanScopeLoading = false; + controls.refreshModels = vi.fn(); + controls.sidebarManager = { + setHostPageControls: vi.fn(), + initialize: vi.fn(async () => {}), + }; + return controls; +} + +describe('PageControls refresh scope menu', () => { + beforeEach(() => { + showToastMock.mockReset(); + modelClient.fetchModelRoots.mockReset(); + modelClient.refreshModels.mockReset(); + modelClient.fetchModelRoots.mockResolvedValue({ root_details: ROOT_DETAILS }); + }); + + afterEach(() => { + document.body.innerHTML = ''; + }); + + it('keeps the scope container in the controls template (non-recipes pages)', () => { + const html = readFileSync(CONTROLS_TEMPLATE, 'utf-8'); + const rebuildIndex = html.indexOf('data-action="full-rebuild"'); + const scopeIndex = html.indexOf('id="refreshScopeMenu"'); + + expect(rebuildIndex).toBeGreaterThan(-1); + // Same dropdown, after the rebuild entry. + expect(scopeIndex).toBeGreaterThan(rebuildIndex); + expect(html).toContain('loras.controls.refresh.scopeSection'); + // The recipes page has no model roots, so the section is gated. + expect(html.slice(rebuildIndex, scopeIndex)).toContain("{% if page_id != 'recipes' %}"); + }); + + it('registers the scan-scope API for every page through registerAPI', async () => { + const controls = bareControls(); + controls.registerAPI({}); + + expect(typeof controls.api.fetchModelRoots).toBe('function'); + expect(typeof controls.api.refreshModels).toBe('function'); + + await controls.api.fetchModelRoots(); + expect(modelClient.fetchModelRoots).toHaveBeenCalled(); + + // The page facades used to drop the second argument, which silently turned + // a scoped scan into a full refresh. + await controls.api.refreshModels(false, { roots: ['/mnt/a/loras'] }); + expect(modelClient.refreshModels).toHaveBeenCalledWith(false, { roots: ['/mnt/a/loras'] }); + }); + + it('renders the root rows when the dropdown is opened', async () => { + buildControlsDom(); + const controls = bareControls({ + fetchModelRoots: vi.fn(async () => ({ root_details: ROOT_DETAILS })), + refreshModels: vi.fn(), + }); + + controls.initDropdowns(); + document.querySelector('.dropdown-toggle').click(); + + await vi.waitFor(() => { + expect(document.querySelectorAll('.scan-root-item').length).toBe(2); + }); + expect(document.querySelector('.scan-root-item').dataset.root).toBe('/mnt/a/loras'); + expect(document.querySelector('.scan-root-count').textContent).toBe('12 models'); + expect(document.querySelectorAll('.scan-root-item')[1].classList.contains('is-offline')).toBe(true); + }); + + it('scans the clicked root only', async () => { + buildControlsDom(); + const controls = bareControls({ + fetchModelRoots: vi.fn(async () => ({ root_details: ROOT_DETAILS })), + refreshModels: vi.fn(), + }); + + controls.initDropdowns(); + document.querySelector('.dropdown-toggle').click(); + await vi.waitFor(() => { + expect(document.querySelectorAll('.scan-root-item').length).toBe(2); + }); + + document.querySelectorAll('.scan-root-item')[0].click(); + + expect(controls.refreshModels).toHaveBeenCalledWith(false, { roots: ['/mnt/a/loras'] }); + expect(showToastMock).not.toHaveBeenCalled(); + }); + + it('explains an offline root instead of scanning it', async () => { + buildControlsDom(); + const controls = bareControls({ + fetchModelRoots: vi.fn(async () => ({ root_details: ROOT_DETAILS })), + refreshModels: vi.fn(), + }); + + controls.initDropdowns(); + document.querySelector('.dropdown-toggle').click(); + await vi.waitFor(() => { + expect(document.querySelectorAll('.scan-root-item').length).toBe(2); + }); + + document.querySelectorAll('.scan-root-item')[1].click(); + + expect(controls.refreshModels).not.toHaveBeenCalled(); + // Exactly three arguments: key, params, type. A 4th "fallback" argument + // pushed the sentence into the type slot and rendered an unstyled toast. + expect(showToastMock).toHaveBeenCalledWith( + 'toast.api.scanRootUnreachable', + { scope: 'b/loras' }, + 'info' + ); + }); +}); diff --git a/tests/frontend/components/controls/scanScopeMenu.test.js b/tests/frontend/components/controls/scanScopeMenu.test.js new file mode 100644 index 00000000..b5dac3a4 --- /dev/null +++ b/tests/frontend/components/controls/scanScopeMenu.test.js @@ -0,0 +1,77 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +const { I18N_MODULE, MODULE } = vi.hoisted(() => ({ + I18N_MODULE: new URL('../../../../static/js/utils/i18nHelpers.js', import.meta.url).pathname, + MODULE: new URL('../../../../static/js/components/controls/ScanScopeMenu.js', import.meta.url).pathname, +})); + +vi.mock(I18N_MODULE, () => ({ + translate: vi.fn((key, params, fallback) => { + if (!fallback) { + return key; + } + return Object.entries(params || {}).reduce( + (text, [name, value]) => text.replaceAll(`{${name}}`, value), + fallback + ); + }), +})); + +const { renderScanScopeMenu, resolveScanScopeTarget } = await import(MODULE); + +describe('ScanScopeMenu', () => { + let menu; + + beforeEach(() => { + document.body.innerHTML = '
'; + menu = document.getElementById('refreshScopeMenu'); + }); + + it('renders one row per root with label, count and offline state', () => { + renderScanScopeMenu(menu, [ + { path: '/mnt/a/loras', label: 'a/loras', reachable: true, models: 7 }, + { path: '/mnt/b/loras', label: 'b/loras', reachable: false, models: 0 }, + ]); + + const rows = menu.querySelectorAll('.scan-root-item'); + expect(rows.length).toBe(2); + + expect(rows[0].dataset.action).toBe('scan-root'); + expect(rows[0].dataset.root).toBe('/mnt/a/loras'); + expect(rows[0].title).toBe('/mnt/a/loras'); + expect(rows[0].querySelector('.scan-root-label').textContent).toBe('a/loras'); + expect(rows[0].querySelector('.scan-root-count').textContent).toBe('7 models'); + expect(rows[0].classList.contains('is-offline')).toBe(false); + + expect(rows[1].classList.contains('is-offline')).toBe(true); + expect(rows[1].querySelector('.scan-root-offline').textContent).toBe('Offline'); + // Offline rows stay clickable so the click can explain the state. + expect(rows[1].dataset.action).toBe('scan-root'); + }); + + it('resolveScanScopeTarget reads path, label and offline state', () => { + renderScanScopeMenu(menu, [ + { path: '/mnt/b/loras', label: 'b/loras', reachable: false, models: 0 }, + ]); + + expect(resolveScanScopeTarget(menu.querySelector('.scan-root-item'))).toEqual({ + rootPath: '/mnt/b/loras', + label: 'b/loras', + offline: true, + }); + }); + + it('skips malformed entries and replaces the previous list', () => { + renderScanScopeMenu(menu, [{ path: '/mnt/a' }, null, {}]); + expect(menu.querySelectorAll('.scan-root-item').length).toBe(1); + + renderScanScopeMenu(menu, []); + expect(menu.querySelectorAll('.scan-root-item').length).toBe(0); + }); + + it('falls back to the path when a root has no label', () => { + renderScanScopeMenu(menu, [{ path: '/mnt/a/loras', reachable: true, models: 0 }]); + + expect(menu.querySelector('.scan-root-label').textContent).toBe('/mnt/a/loras'); + }); +}); diff --git a/tests/frontend/utils/toastTypes.test.js b/tests/frontend/utils/toastTypes.test.js new file mode 100644 index 00000000..509c6257 --- /dev/null +++ b/tests/frontend/utils/toastTypes.test.js @@ -0,0 +1,36 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +const { UI_HELPERS_MODULE, I18N_MODULE } = vi.hoisted(() => ({ + UI_HELPERS_MODULE: new URL('../../../static/js/utils/uiHelpers.js', import.meta.url).pathname, + I18N_MODULE: new URL('../../../static/js/utils/i18nHelpers.js', import.meta.url).pathname, +})); + +vi.mock(I18N_MODULE, () => ({ + translate: vi.fn((key, params, fallback) => fallback || key), +})); + +const { showToast } = await import(UI_HELPERS_MODULE); + +describe('toast type styling', () => { + beforeEach(() => { + document.body.innerHTML = ''; + }); + + it('applies the requested type class', () => { + showToast('plain message', {}, 'success'); + expect(document.querySelector('.toast').className).toBe('toast toast-success'); + }); + + it('coerces an unknown type to info instead of rendering an unstyled toast', () => { + // Regression guard: a call site that passed its fallback sentence as the + // `type` produced `toast toast-drive-Z is not reachable ...`, which matched + // no rule at all (no icon, no accent border). + showToast('plain message', {}, 'drive-Z is not reachable right now'); + expect(document.querySelector('.toast').className).toBe('toast toast-info'); + }); + + it('defaults to info when no type is given', () => { + showToast('plain message'); + expect(document.querySelector('.toast').className).toBe('toast toast-info'); + }); +}); diff --git a/tests/routes/test_model_query_handler.py b/tests/routes/test_model_query_handler.py index d3cd27a3..f333eea4 100644 --- a/tests/routes/test_model_query_handler.py +++ b/tests/routes/test_model_query_handler.py @@ -196,3 +196,126 @@ async def test_get_folder_tree_threads_include_empty(): assert payload["success"] is True assert service.received_include_empty is True + + +class QueryParams(dict): + """Minimal stand-in for aiohttp's MultiDict query (supports getall).""" + + def getall(self, key, default=None): + value = self.get(key) + if value is None: + return list(default) if default else [] + return value if isinstance(value, list) else [value] + + +class ScopedScanService: + """Stub service recording the scan scope the handler resolves.""" + + model_type = "lora" + + def __init__(self, roots=None, summary=None, cancelled=False): + self._roots = list(roots or []) + self._summary = summary + self.cancelled = cancelled + self.scanner = SimpleNamespace(is_cancelled=lambda: self.cancelled) + self.received_scope = "not-called" + self.received_rebuild = None + + def get_model_roots(self): + return list(self._roots) + + def describe_model_roots(self): + return [ + {"path": root, "label": root.rsplit("/", 1)[-1], "reachable": True, "models": 7} + for root in self._roots + ] + + async def scan_models(self, force_refresh=False, rebuild_cache=False, scope=None): + self.received_scope = scope + self.received_rebuild = rebuild_cache + return self._summary + + +SUMMARY = { + "added": 3, + "removed": 0, + "repaired": 1, + "scanned_roots": ["a"], + "skipped_roots": [], + "unavailable_paths": [], + "unavailable_paths_total": 0, + "kept_unreachable": 0, +} + + +@pytest.mark.asyncio +async def test_scan_models_accepts_roots_param(): + service = ScopedScanService(roots=["/mnt/a", "/mnt/b"], summary=SUMMARY) + handler = ModelQueryHandler(service=service, logger=logging.getLogger(__name__)) + + response = await handler.scan_models( + SimpleNamespace(query=QueryParams({"roots": ["/mnt/a"]})) + ) + payload = json.loads(response.text) + + assert response.status == 200 + assert payload["status"] == "success" + assert payload["added"] == 3 + assert payload["scanned_roots"] == ["a"] + assert service.received_scope is not None + assert service.received_scope.roots == ("/mnt/a",) + assert service.received_rebuild is False + + +@pytest.mark.asyncio +async def test_scan_models_without_roots_scans_every_root(): + service = ScopedScanService(roots=["/mnt/a"], summary=SUMMARY) + handler = ModelQueryHandler(service=service, logger=logging.getLogger(__name__)) + + await handler.scan_models(SimpleNamespace(query=QueryParams({}))) + + assert service.received_scope is None + + +@pytest.mark.asyncio +async def test_scan_models_rejects_unknown_root(): + service = ScopedScanService(roots=["/mnt/a"], summary=SUMMARY) + handler = ModelQueryHandler(service=service, logger=logging.getLogger(__name__)) + + response = await handler.scan_models( + SimpleNamespace(query=QueryParams({"roots": ["/mnt/nope"]})) + ) + payload = json.loads(response.text) + + assert response.status == 400 + assert payload["roots"] == ["/mnt/nope"] + assert service.received_scope == "not-called" + + +@pytest.mark.asyncio +async def test_scan_models_rejects_roots_with_full_rebuild(): + service = ScopedScanService(roots=["/mnt/a"], summary=SUMMARY) + handler = ModelQueryHandler(service=service, logger=logging.getLogger(__name__)) + + response = await handler.scan_models( + SimpleNamespace( + query=QueryParams({"roots": ["/mnt/a"], "full_rebuild": "true"}) + ) + ) + + assert response.status == 400 + assert service.received_scope == "not-called" + + +@pytest.mark.asyncio +async def test_get_model_roots_reports_details(): + service = ScopedScanService(roots=["/mnt/a", "/mnt/b"]) + handler = ModelQueryHandler(service=service, logger=logging.getLogger(__name__)) + + response = await handler.get_model_roots(SimpleNamespace(query=QueryParams({}))) + payload = json.loads(response.text) + + # `roots` stays a plain list of paths for the existing callers. + assert payload["roots"] == ["/mnt/a", "/mnt/b"] + assert [detail["label"] for detail in payload["root_details"]] == ["a", "b"] + assert all(detail["models"] == 7 for detail in payload["root_details"]) diff --git a/tests/routes/test_preview_routes.py b/tests/routes/test_preview_routes.py index 6060cd21..569aff94 100644 --- a/tests/routes/test_preview_routes.py +++ b/tests/routes/test_preview_routes.py @@ -1,7 +1,8 @@ +import asyncio import os import urllib.parse from pathlib import Path -from unittest.mock import patch +from unittest.mock import AsyncMock, patch import pytest from aiohttp import web @@ -348,3 +349,67 @@ async def test_deep_symlink_discovered_on_first_access(tmp_path): # Deep symlink should now be in mappings assert normalized_external in config._path_mappings + + +async def test_preview_404_keeps_cache_when_parent_dir_missing(tmp_path): + """A preview whose directory is unreachable (drive off) must not clear the + cached preview_url: nothing was deleted, the file just cannot be read.""" + library_root = tmp_path / "library" + library_root.mkdir() + unreachable_parent = library_root / "gone" + preview_path = unreachable_parent / "model.webp" + + config = Config() + config.apply_library_settings( + { + "folder_paths": { + "loras": [str(library_root)], + "checkpoints": [], + "unet": [], + "embeddings": [], + } + } + ) + + handler = PreviewHandler(config=config) + encoded_path = urllib.parse.quote(str(preview_path), safe="") + request = make_mocked_request("GET", f"/api/lm/previews?path={encoded_path}") + + with patch.object(handler, "_cleanup_stale_preview_url", new=AsyncMock()) as cleanup: + with pytest.raises(web.HTTPNotFound): + await handler.serve_preview(request) + await asyncio.sleep(0) + + cleanup.assert_not_called() + + +async def test_preview_404_clears_cache_when_file_really_deleted(tmp_path): + """A preview deleted from a reachable directory is genuinely stale.""" + library_root = tmp_path / "library" + library_root.mkdir() + preview_path = library_root / "model.webp" + preview_path.write_bytes(b"preview") + preview_path.unlink() + + config = Config() + config.apply_library_settings( + { + "folder_paths": { + "loras": [str(library_root)], + "checkpoints": [], + "unet": [], + "embeddings": [], + } + } + ) + + handler = PreviewHandler(config=config) + encoded_path = urllib.parse.quote(str(preview_path), safe="") + request = make_mocked_request("GET", f"/api/lm/previews?path={encoded_path}") + + with patch.object(handler, "_cleanup_stale_preview_url", new=AsyncMock()) as cleanup: + with pytest.raises(web.HTTPNotFound): + await handler.serve_preview(request) + await asyncio.sleep(0) + + cleanup.assert_awaited_once() diff --git a/tests/services/test_model_scanner.py b/tests/services/test_model_scanner.py index 19cdbc64..ae6f63bf 100644 --- a/tests/services/test_model_scanner.py +++ b/tests/services/test_model_scanner.py @@ -20,9 +20,11 @@ from py.services.model_hash_index import ModelHashIndex from py.services.model_scanner import ( CacheBuildResult, ModelScanner, + ReconcileScope, _ReconcileWalkTracker, _count_cached_entries_per_root, _group_roots_by_device, + _root_display_labels, ) from py.services.pending_delete_service import ( PENDING_DELETE_DIR_NAME, @@ -2479,10 +2481,9 @@ async def test_reconcile_walk_broadcasts_per_root_progress(tmp_path: Path, monke @pytest.mark.asyncio -async def test_reconcile_prunes_entries_of_an_offline_root(tmp_path: Path): - """A root that is gone (drive switched off) is skipped by the walk, so its - cached entries are reported as missing. Locked in because it is the - documented consequence of refreshing with a drive powered down.""" +async def test_reconcile_keeps_entries_of_an_offline_root(tmp_path: Path): + """A configured root that is gone (drive switched off) must not be treated + as deleted: its entries are kept and reported as unreachable instead.""" root_a = tmp_path / "a" root_b = tmp_path / "b" root_a.mkdir() @@ -2498,9 +2499,217 @@ async def test_reconcile_prunes_entries_of_an_offline_root(tmp_path: Path): root_b.rename(tmp_path / "b_offline") - await scanner._reconcile_cache() + summary = await scanner._reconcile_cache() remaining = {item["file_path"] for item in scanner._cache.raw_data} - assert remaining == {_normalize_path(file_a)} - assert scanner._hash_index.get_path("hash-b0") is None + assert remaining == {_normalize_path(file_a), _normalize_path(file_b)} + assert scanner._hash_index.get_path("hash-b0") == _normalize_path(file_b) assert scanner._hash_index.get_path("hash-a0") == _normalize_path(file_a) + + assert summary is not None + assert summary["removed"] == 0 + assert summary["kept_unreachable"] == 1 + assert summary["unavailable_paths_total"] == 1 + assert [entry["path"] for entry in summary["unavailable_paths"]] == [ + _normalize_path(root_b) + ] + assert summary["unavailable_paths"][0]["reason"] == "root_unreachable" + assert [entry["path"] for entry in summary["skipped_roots"]] == [ + _normalize_path(root_b) + ] + + +# --- scoped scans (issue #1108) -------------------------------------------- + + +@pytest.mark.asyncio +async def test_reconcile_scoped_scan_leaves_other_roots_untouched(tmp_path: Path): + """Scanning one root must not add, remove or re-read anything in another.""" + root_a = tmp_path / "a" + root_b = tmp_path / "b" + root_a.mkdir() + root_b.mkdir() + (root_a / "a0.txt").write_text("a", encoding="utf-8") + (root_b / "b0.txt").write_text("b", encoding="utf-8") + + scanner = MultiRootDummyScanner([root_a, root_b]) + await scanner._initialize_cache() + + # Changes in B stay invisible while only A is scanned. + (root_b / "b0.txt").unlink() + (root_b / "b_new.txt").write_text("new", encoding="utf-8") + new_a = root_a / "a_new.txt" + new_a.write_text("new", encoding="utf-8") + + summary = await scanner._reconcile_cache(scope=ReconcileScope(roots=(str(root_a),))) + + assert summary is not None + assert summary["added"] == 1 + assert summary["removed"] == 0 + assert summary["scanned_roots"] == ["a"] + + cached_paths = {item["file_path"] for item in scanner._cache.raw_data} + assert cached_paths == { + _normalize_path(root_a / "a0.txt"), + _normalize_path(new_a), + _normalize_path(root_b / "b0.txt"), + } + + +@pytest.mark.asyncio +async def test_reconcile_scoped_scan_removes_deleted_files_in_scope(tmp_path: Path): + """Deletions inside the scanned root still reconcile normally.""" + root_a = tmp_path / "a" + root_b = tmp_path / "b" + root_a.mkdir() + root_b.mkdir() + deleted_a = root_a / "a0.txt" + deleted_a.write_text("a", encoding="utf-8") + (root_b / "b0.txt").write_text("b", encoding="utf-8") + + scanner = MultiRootDummyScanner([root_a, root_b]) + await scanner._initialize_cache() + + deleted_a.unlink() + + summary = await scanner._reconcile_cache(scope=ReconcileScope(roots=(str(root_a),))) + + assert summary is not None + assert summary["removed"] == 1 + assert scanner._hash_index.get_path("hash-a0") is None + assert {item["file_path"] for item in scanner._cache.raw_data} == { + _normalize_path(root_b / "b0.txt") + } + + +@pytest.mark.asyncio +async def test_reconcile_folder_scope_keeps_folder_tree_outside_scope(tmp_path: Path): + """A folder-scoped scan must not collapse the recorded folder list to the + scanned subtree (the sidebar would lose every other folder).""" + root = tmp_path / "loras" + (root / "alpha").mkdir(parents=True) + (root / "beta").mkdir() + (root / "alpha" / "m.txt").write_text("m", encoding="utf-8") + + scanner = DummyScanner(root) + await scanner._initialize_cache() + folders_before = set(scanner._cache.all_folders or []) + assert "beta" in folders_before + + (root / "alpha" / "nested").mkdir() + (root / "alpha" / "m.txt").unlink() + + summary = await scanner._reconcile_cache(scope=ReconcileScope(folder="alpha")) + + assert summary is not None + assert summary["removed"] == 1 + folders_after = set(scanner._cache.all_folders or []) + assert "beta" in folders_after + assert "alpha/nested" in folders_after + assert {item["file_path"] for item in scanner._cache.raw_data} == set() + + +@pytest.mark.asyncio +async def test_reconcile_keeps_entries_under_unreadable_dir(tmp_path: Path, monkeypatch): + """A directory os.walk cannot enter (permissions, I/O error, offline + junction) keeps its cached entries instead of losing them.""" + root = tmp_path / "loras" + sub = root / "sub" + sub.mkdir(parents=True) + (sub / "m.txt").write_text("m", encoding="utf-8") + (root / "top.txt").write_text("t", encoding="utf-8") + + scanner = DummyScanner(root) + await scanner._initialize_cache() + + real_scandir = os.scandir + + def _failing_scandir(path=".", *args, **kwargs): + if os.path.normpath(str(path)) == os.path.normpath(str(sub)): + raise PermissionError(13, "Permission denied", str(sub)) + return real_scandir(path, *args, **kwargs) + + monkeypatch.setattr(model_scanner.os, "scandir", _failing_scandir) + + summary = await scanner._reconcile_cache() + + assert summary is not None + assert summary["removed"] == 0 + assert summary["kept_unreachable"] == 1 + assert summary["unavailable_paths"][0]["reason"] == "unreadable_dir" + assert _normalize_path(sub / "m.txt") in { + item["file_path"] for item in scanner._cache.raw_data + } + + +@pytest.mark.asyncio +async def test_reconcile_keeps_entries_under_offline_first_level_symlink( + tmp_path: Path, monkeypatch +): + """A known first-level symlink whose target is gone protects the entries + that live under the link path (the linked drive is switched off).""" + root = tmp_path / "loras" + linked = root / "linked" + linked.mkdir(parents=True) + (linked / "m.txt").write_text("m", encoding="utf-8") + + scanner = DummyScanner(root) + await scanner._initialize_cache() + + # The linked folder goes away with its drive; config still knows the link. + linked.rename(tmp_path / "linked_offline") + monkeypatch.setattr( + model_scanner.config, + "iter_path_mappings", + lambda: [(str(tmp_path / "gone_target"), str(linked))], + ) + + summary = await scanner._reconcile_cache() + + assert summary is not None + assert summary["removed"] == 0 + assert summary["kept_unreachable"] == 1 + assert summary["unavailable_paths"][0]["reason"] == "symlink_target_offline" + assert _normalize_path(linked / "m.txt") in { + item["file_path"] for item in scanner._cache.raw_data + } + + +def test_root_display_labels_dedupe_by_parent_segments(monkeypatch): + labels = _root_display_labels(["/mnt/usb/loras", "/mnt/ssd/loras"]) + assert labels == {"/mnt/usb/loras": "usb/loras", "/mnt/ssd/loras": "ssd/loras"} + + # Still colliding after one parent segment: keep growing leftwards. + labels = _root_display_labels(["/mnt/a/models/loras", "/mnt/b/models/loras"]) + assert labels == { + "/mnt/a/models/loras": "a/models/loras", + "/mnt/b/models/loras": "b/models/loras", + } + + # A single root keeps the short form. + assert _root_display_labels(["/mnt/usb/loras"]) == {"/mnt/usb/loras": "loras"} + + # Windows drive prefixes disambiguate on their own. + def _fake_splitdrive(path): + for drive in ("G:", "H:"): + if path.startswith(drive): + return drive, path[len(drive):] + return "", path + + monkeypatch.setattr(model_scanner.os.path, "splitdrive", _fake_splitdrive) + labels = _root_display_labels(["G:\\x\\loras", "H:\\y\\loras"]) + assert labels == {"G:\\x\\loras": "G: loras", "H:\\y\\loras": "H: loras"} + + # Identical renderings (trailing separator / duplicated config entry) grow to + # the full path and then fall back to a deterministic suffix instead of + # silently sharing a label. + labels = _root_display_labels(["/mnt/x/loras", "/mnt/x/loras/"]) + assert labels == { + "/mnt/x/loras": "mnt/x/loras (1)", + "/mnt/x/loras/": "mnt/x/loras (2)", + } + + long_root = "/mnt/" + "d" * 60 + "/loras" + label = _root_display_labels([long_root])[long_root] + assert len(label) <= 40 + assert label.endswith("loras")