mirror of
https://github.com/willmiao/ComfyUI-Lora-Manager.git
synced 2026-10-08 02:22:12 -03:00
feat(scanner): report and re-admit model roots that are unavailable (#1108)
`Config._dedupe_existing_paths()` drops roots whose directory does not exist
when the root list is built, so a drive that is switched off while LM starts was
not "an offline root" — it was not a root at all. The Refresh ▾ menu had no row
for it in either state, a full refresh reported nothing (its cached entries
survived only because they fall outside every configured prefix), and plugging
the drive back in changed nothing until the next restart.
* `Config` now records the configured (existence-unfiltered) paths per model
type — from ComfyUI's list in plugin mode, from the active library snapshot in
standalone mode, where the host mock filters non-existent paths itself — and
exposes them through `configured_roots_for()`.
* `ModelScanner.describe_model_roots()` appends configured-but-unavailable roots
with a live `reachable`, an `available` flag and their cached entry count, so
`/roots` (and therefore the menu) offers a row in both states: greyed while the
directory is missing, normal and clickable once it is back.
* `_reconcile_cache()` reports them as `skipped_roots` / `unavailable_paths` with
reason `root_unavailable` and counts their cached entries in `kept_unreachable`,
which is what makes the "N models kept" toast appear in the startup-offline
case. Report-only: they stay outside the scan scope, so nothing about pruning
changes. A root-scoped scan stays quiet about roots it was not asked about, and
a folder-scoped scan only mentions a missing root that holds cached entries of
that folder.
* `Config.admit_configured_roots()` re-runs the per-type prepare helpers against
the configured paths and APPENDS what is readable now (plus the
checkpoint/unet/other side maps), then refreshes the preview allowlist, so a
drive plugged in mid-session can be scanned without restarting. Append-only and
order-preserving by design: removing a root mid-session would let a later
settings save persist the loss, and re-sorting would move `*_roots[0]`, which
derives the recipes directory and the usage-stats file location. `_roots[0]`
never moves.
* `/roots` and `/scan` admit first, so `describe_model_roots()` stays a pure read
and a caller that never opens the menu (the browser extension) still gets a
drive that came back.
Verified on the sandbox with drive-Z switched off before startup: `/roots`
reports `drive-Z reachable=false available=false models=60`, a full refresh
returns `skipped_roots=[drive-Z root_unavailable]`,
`unavailable_paths=[{... kept: 60}]`, `kept_unreachable=60`, and all 432 models
stay cached. Renaming the drive back while the server runs admits it
(`available=true`) and `GET /scan?roots=<drive-Z>` walks it with no restart.
3714 passed, 7 skipped; frontend 1495 passed (149 files); vue widgets 96 passed.
This commit is contained in:
@@ -2,9 +2,9 @@
|
|||||||
|
|
||||||
**Issue:** [#1108](https://github.com/willmiao/ComfyUI-Lora-Manager/issues/1108) — scan a single
|
**Issue:** [#1108](https://github.com/willmiao/ComfyUI-Lora-Manager/issues/1108) — scan a single
|
||||||
folder/root instead of the whole library.
|
folder/root instead of the whole library.
|
||||||
**Status:** **P1 and P2 implemented** (2026-10-07). P1 shipped in `470d85cc` (translations in
|
**Status:** **P1, P2 and Wave 6 implemented** (2026-10-07). P1 shipped in `470d85cc` (translations
|
||||||
`bd184559`); P2 (folder scope + the sidebar entry) is in the working tree. Supersedes the earlier
|
in `bd184559`), P2 in `12930ce7` (translations in `0941f992`). Supersedes the earlier draft's
|
||||||
draft's per-root *status panel* and persisted *unreachable-subtree* state (both dropped, see
|
per-root *status panel* and persisted *unreachable-subtree* state (both dropped, see
|
||||||
"Must NOT have").
|
"Must NOT have").
|
||||||
|
|
||||||
Implementation notes / deviations from the draft below:
|
Implementation notes / deviations from the draft below:
|
||||||
@@ -271,6 +271,85 @@ and in the exact pruning predicate; both get explicit tests.
|
|||||||
`scope_label=pack000`, keeps drive-Z's 6 entries under that folder (`kept_unreachable=6`) and
|
`scope_label=pack000`, keeps drive-Z's 6 entries under that folder (`kept_unreachable=6`) and
|
||||||
leaves all 420 models cached.
|
leaves all 420 models cached.
|
||||||
|
|
||||||
|
## Wave 6 — configured vs currently available roots (approved follow-up)
|
||||||
|
|
||||||
|
The P1/P2 work exposed an asymmetry: `Config._dedupe_existing_paths()` drops roots whose directory
|
||||||
|
does not exist **at the moment the root list is built** (startup, or applying a library snapshot),
|
||||||
|
so a drive that is switched off while LM starts is not "an offline root" — it is not a root at all.
|
||||||
|
Consequences, all verified on the sandbox:
|
||||||
|
|
||||||
|
* `/roots` does not list it, so the Refresh ▾ menu shows no row for it in either state;
|
||||||
|
* a full refresh reports nothing (`kept_unreachable: 0`, no toast) even though its cached entries
|
||||||
|
are kept — they survive only because they fall outside every configured root prefix;
|
||||||
|
* plugging the drive back in does not bring the row back: the in-memory list is not re-validated;
|
||||||
|
* **plugin mode only:** `Config.save_folder_paths_to_settings()` (called from `Config.__init__`)
|
||||||
|
persists `target_folder_paths["loras"] = list(self.loras_roots)` through
|
||||||
|
`upsert_library(folder_paths=...)`, which **replaces** the library's paths. Starting ComfyUI with
|
||||||
|
a drive switched off therefore *erases that path from `settings.json`* — a configuration loss,
|
||||||
|
not just a display gap. Extra paths are unaffected (that call reuses the stored
|
||||||
|
`extra_folder_paths`), and the same pattern applies to checkpoints/unet/embeddings/other-model
|
||||||
|
primary paths.
|
||||||
|
|
||||||
|
### Must have
|
||||||
|
|
||||||
|
1. **Never erase an unavailable root from the library config.** `Config` records the *configured*
|
||||||
|
(existence-unfiltered) primary paths per model type while building the live lists, and
|
||||||
|
`save_folder_paths_to_settings()` persists those instead of the filtered ones.
|
||||||
|
`_resolve_valid_default_root()` receives the configured paths in `allowed_paths` too, so a
|
||||||
|
`default_*_root` that sits on a switched-off drive is not "repaired" away.
|
||||||
|
2. **Report configured-but-unavailable roots.** `Config.configured_roots_for(model_type)` feeds
|
||||||
|
`ModelScanner.describe_model_roots()`, which appends them with `available: false`, a live
|
||||||
|
`reachable` and their cached entry count — so the Refresh ▾ menu has a row in **both** states
|
||||||
|
(greyed while the directory is missing, normal and clickable once it is back). The reconcile
|
||||||
|
summary reports them as `skipped_roots` / `unavailable_paths` with reason `root_unavailable` and
|
||||||
|
counts their cached entries in `kept_unreachable`, which is what makes the "N models kept" toast
|
||||||
|
appear in the startup-offline case.
|
||||||
|
3. **Admit them again, append-only.** `Config.admit_configured_roots()` re-runs the per-type
|
||||||
|
prepare helpers against the configured paths and **appends** what exists now
|
||||||
|
(plus the checkpoint/unet/other side maps) to the live lists, then refreshes the preview
|
||||||
|
allowlist. Called from `/roots` and `/scan`, so a drive plugged in mid-session can be scanned
|
||||||
|
without restarting. Appending (never re-sorting) keeps `loras_roots[0]` — which derives the
|
||||||
|
recipes directory and the usage-stats file location — stable for the whole session.
|
||||||
|
|
||||||
|
### Must NOT have
|
||||||
|
|
||||||
|
* NO removal of a root from the live lists mid-session (that is what would let a later settings
|
||||||
|
save persist a reduced configuration, and it would move `*_roots[0]`).
|
||||||
|
* NO change to the scan/prune scope: a root that is still unavailable stays out of scope, so its
|
||||||
|
entries remain "kept because out of scope" exactly as today.
|
||||||
|
* NO persisted unavailable-state, no DB schema change, no per-entry filesystem probe.
|
||||||
|
|
||||||
|
### Where "configured" comes from
|
||||||
|
|
||||||
|
`Config` records the existence-unfiltered paths while building the live lists, but the *source*
|
||||||
|
differs per mode and that matters:
|
||||||
|
|
||||||
|
* **Plugin mode:** `folder_paths.get_folder_paths()` is ComfyUI's own list, unfiltered — a path the
|
||||||
|
user removed there must be forgotten, one that is merely missing must be kept. The host list wins.
|
||||||
|
* **Standalone mode:** `standalone.MockFolderPaths.get_folder_paths()` already filters
|
||||||
|
`os.path.exists` out of `settings.json`, so the host list cannot answer "what did the user
|
||||||
|
configure". The active **library snapshot** (`libraries[<active>].folder_paths`, falling back to
|
||||||
|
the top-level `folder_paths`) is the record, and it wins there.
|
||||||
|
(`_remember_library_configured_paths()` implements the split.)
|
||||||
|
|
||||||
|
### Verification
|
||||||
|
|
||||||
|
* Config: a configured path that does not exist is still written back by
|
||||||
|
`save_folder_paths_to_settings()`; a path the host no longer configures is still dropped.
|
||||||
|
* Promotion: create the directory after the lists were built → `admit_configured_roots()` adds it
|
||||||
|
**at the end**, keeps every existing root in place, and adds nothing twice.
|
||||||
|
* Scanner: `describe_model_roots()` reports the missing root (`available: false`, cached count);
|
||||||
|
a full refresh lists it in `skipped_roots`/`unavailable_paths` and keeps its entries; `/scan`
|
||||||
|
with that root works once the directory is back.
|
||||||
|
* Sandbox, drive-Z gone **before** startup: `/roots` reports `drive-Z` with
|
||||||
|
`reachable=false available=false models=60`; a full refresh returns
|
||||||
|
`skipped_roots=[drive-Z root_unavailable]`, `unavailable_paths=[{... kept: 60}]`,
|
||||||
|
`kept_unreachable=60` and leaves all 432 models cached. *(Both measured; before this wave the
|
||||||
|
menu had no row and the refresh reported nothing.)*
|
||||||
|
* Sandbox, drive renamed back **while the server runs**: `/roots` admits it
|
||||||
|
(`available=true reachable=true`) and `GET /scan?roots=<drive-Z>` walks it
|
||||||
|
(`scanned_roots=['drive-Z']`, 0 added / 0 removed) — no restart required. *(Measured.)*
|
||||||
|
|
||||||
## Known limitations (accepted)
|
## Known limitations (accepted)
|
||||||
|
|
||||||
- **Nested symlinks** (a symlink *below* a root pointing at another drive) are only covered when
|
- **Nested symlinks** (a symlink *below* a root pointing at another drive) are only covered when
|
||||||
|
|||||||
@@ -1679,6 +1679,105 @@ class Config:
|
|||||||
"""
|
"""
|
||||||
return list((getattr(self, "_configured_root_paths", None) or {}).get(model_type, []))
|
return list((getattr(self, "_configured_root_paths", None) or {}).get(model_type, []))
|
||||||
|
|
||||||
|
def admit_configured_roots(self) -> List[str]:
|
||||||
|
"""Append configured roots that became readable again, never removing any.
|
||||||
|
|
||||||
|
The live lists are built once at startup (and when a library snapshot is
|
||||||
|
applied), so a drive plugged in later is invisible until the process is
|
||||||
|
restarted. Re-running the per-type prepare helpers against the configured
|
||||||
|
paths and appending only the new entries is what makes "plug the drive
|
||||||
|
in, scan it" work without a restart. Nothing is ever dropped or
|
||||||
|
re-ordered — see ``_append_new_paths`` for why order matters.
|
||||||
|
|
||||||
|
Returns the paths that were admitted by this call.
|
||||||
|
"""
|
||||||
|
admitted: List[str] = []
|
||||||
|
|
||||||
|
def _extend(current, candidates) -> List[str]:
|
||||||
|
merged = _append_new_paths(list(current or []), candidates)
|
||||||
|
admitted.extend(
|
||||||
|
path for path in merged if path not in (current or [])
|
||||||
|
)
|
||||||
|
return merged
|
||||||
|
|
||||||
|
try:
|
||||||
|
lora_configured = self.configured_roots_for("lora")
|
||||||
|
if lora_configured:
|
||||||
|
self.loras_roots = _extend(
|
||||||
|
self.loras_roots, self._prepare_lora_paths(lora_configured)
|
||||||
|
)
|
||||||
|
|
||||||
|
checkpoint_configured = self.configured_roots_for("checkpoint")
|
||||||
|
unet_configured = self.configured_roots_for("unet")
|
||||||
|
if checkpoint_configured or unet_configured:
|
||||||
|
(
|
||||||
|
all_roots,
|
||||||
|
checkpoint_roots,
|
||||||
|
unet_roots,
|
||||||
|
) = self._prepare_checkpoint_paths(
|
||||||
|
checkpoint_configured, unet_configured
|
||||||
|
)
|
||||||
|
self.base_models_roots = _extend(self.base_models_roots, all_roots)
|
||||||
|
self.checkpoints_roots = _extend(self.checkpoints_roots, checkpoint_roots)
|
||||||
|
self.unet_roots = _extend(self.unet_roots, unet_roots)
|
||||||
|
|
||||||
|
embedding_configured = self.configured_roots_for("embedding")
|
||||||
|
if embedding_configured:
|
||||||
|
self.embeddings_roots = _extend(
|
||||||
|
self.embeddings_roots,
|
||||||
|
self._prepare_embedding_paths(embedding_configured),
|
||||||
|
)
|
||||||
|
|
||||||
|
other_configured = self.configured_roots_for("other")
|
||||||
|
if other_configured:
|
||||||
|
(
|
||||||
|
other_roots,
|
||||||
|
other_subtypes,
|
||||||
|
other_per_key,
|
||||||
|
) = self._prepare_other_paths(
|
||||||
|
{
|
||||||
|
key: [
|
||||||
|
path
|
||||||
|
for path in self._configured_other_paths_for_key(key)
|
||||||
|
]
|
||||||
|
for key in self._get_enabled_other_folder_keys()
|
||||||
|
}
|
||||||
|
)
|
||||||
|
self.other_roots = _extend(self.other_roots, other_roots)
|
||||||
|
for root, sub_type in other_subtypes.items():
|
||||||
|
self.other_root_subtypes.setdefault(root, sub_type)
|
||||||
|
for key, roots in other_per_key.items():
|
||||||
|
self.other_folder_roots[key] = _append_new_paths(
|
||||||
|
self.other_folder_roots.get(key, []), roots
|
||||||
|
)
|
||||||
|
except Exception as exc: # pragma: no cover - defensive logging
|
||||||
|
logger.warning("Failed to admit configured roots: %s", exc)
|
||||||
|
return admitted
|
||||||
|
|
||||||
|
if admitted:
|
||||||
|
logger.info(
|
||||||
|
"Admitted %d previously unavailable model root(s): %s",
|
||||||
|
len(admitted),
|
||||||
|
", ".join(admitted),
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
# Previews of the newly admitted drive must be servable again.
|
||||||
|
self.refresh_preview_roots()
|
||||||
|
except Exception as exc: # pragma: no cover - defensive logging
|
||||||
|
logger.debug("Failed to refresh preview roots: %s", exc)
|
||||||
|
return admitted
|
||||||
|
|
||||||
|
def _configured_other_paths_for_key(self, key: str) -> List[str]:
|
||||||
|
"""Configured other-model paths for one folder_paths key (may not exist)."""
|
||||||
|
try:
|
||||||
|
return [
|
||||||
|
path.strip()
|
||||||
|
for path in folder_paths.get_folder_paths(key) or []
|
||||||
|
if isinstance(path, str) and path.strip()
|
||||||
|
]
|
||||||
|
except Exception:
|
||||||
|
return []
|
||||||
|
|
||||||
def refresh_other_roots(self) -> None:
|
def refresh_other_roots(self) -> None:
|
||||||
"""Rebuild other-model roots after the management toggles changed.
|
"""Rebuild other-model roots after the management toggles changed.
|
||||||
|
|
||||||
|
|||||||
@@ -1163,6 +1163,11 @@ class ModelQueryHandler:
|
|||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
return web.json_response({"error": str(exc)}, status=400)
|
return web.json_response({"error": str(exc)}, status=400)
|
||||||
|
|
||||||
|
# A drive that was plugged in after startup must be scannable without
|
||||||
|
# a restart, and callers that never open the Refresh menu (the browser
|
||||||
|
# extension) have no other chance to admit it.
|
||||||
|
self._service.refresh_model_roots()
|
||||||
|
|
||||||
if requested_roots:
|
if requested_roots:
|
||||||
configured = self._service.get_model_roots()
|
configured = self._service.get_model_roots()
|
||||||
unknown = [root for root in requested_roots if root not in configured]
|
unknown = [root for root in requested_roots if root not in configured]
|
||||||
@@ -1206,6 +1211,9 @@ class ModelQueryHandler:
|
|||||||
|
|
||||||
async def get_model_roots(self, request: web.Request) -> web.Response:
|
async def get_model_roots(self, request: web.Request) -> web.Response:
|
||||||
try:
|
try:
|
||||||
|
# A drive plugged in after startup is admitted here, so the menu shows
|
||||||
|
# it as a normal row instead of "unavailable" until the next restart.
|
||||||
|
self._service.refresh_model_roots()
|
||||||
roots = self._service.get_model_roots()
|
roots = self._service.get_model_roots()
|
||||||
try:
|
try:
|
||||||
root_details = self._service.describe_model_roots()
|
root_details = self._service.describe_model_roots()
|
||||||
|
|||||||
@@ -954,9 +954,17 @@ class BaseModelService(ABC):
|
|||||||
return self.scanner.last_reconcile_summary
|
return self.scanner.last_reconcile_summary
|
||||||
|
|
||||||
def describe_model_roots(self) -> List[Dict[str, Any]]:
|
def describe_model_roots(self) -> List[Dict[str, Any]]:
|
||||||
"""Describe configured roots (label / reachability / cached count)."""
|
"""Describe configured roots (label / reachability / cached count).
|
||||||
|
|
||||||
|
Pure read: callers that want a drive which was plugged in after startup to
|
||||||
|
appear as a normal row call :meth:`refresh_model_roots` first.
|
||||||
|
"""
|
||||||
return self.scanner.describe_model_roots()
|
return self.scanner.describe_model_roots()
|
||||||
|
|
||||||
|
def refresh_model_roots(self) -> List[str]:
|
||||||
|
"""Admit configured roots that became readable again (append-only)."""
|
||||||
|
return self.scanner.refresh_model_roots()
|
||||||
|
|
||||||
async def get_model_info_by_name(self, name: str):
|
async def get_model_info_by_name(self, name: str):
|
||||||
"""Get model information by name"""
|
"""Get model information by name"""
|
||||||
return await self.scanner.get_model_info_by_name(name)
|
return await self.scanner.get_model_info_by_name(name)
|
||||||
|
|||||||
@@ -257,7 +257,13 @@ class _UnreachablePaths:
|
|||||||
self._prefixes: List[str] = []
|
self._prefixes: List[str] = []
|
||||||
self._details: Dict[str, Dict[str, Any]] = {}
|
self._details: Dict[str, Dict[str, Any]] = {}
|
||||||
|
|
||||||
def add(self, path: str, reason: str) -> None:
|
def add(self, path: str, reason: str, kept: int = 0) -> None:
|
||||||
|
"""Register an unreadable prefix.
|
||||||
|
|
||||||
|
``kept`` seeds the count for a root whose entries never reach the prune
|
||||||
|
phase because they are outside the scan scope entirely (a configured root
|
||||||
|
that was not even part of the live root list).
|
||||||
|
"""
|
||||||
prefix = _normalized_root_prefix(path)
|
prefix = _normalized_root_prefix(path)
|
||||||
with self._lock:
|
with self._lock:
|
||||||
if prefix in self._details:
|
if prefix in self._details:
|
||||||
@@ -265,7 +271,7 @@ class _UnreachablePaths:
|
|||||||
self._details[prefix] = {
|
self._details[prefix] = {
|
||||||
'path': path.replace(os.sep, '/'),
|
'path': path.replace(os.sep, '/'),
|
||||||
'reason': reason,
|
'reason': reason,
|
||||||
'kept': 0,
|
'kept': kept,
|
||||||
}
|
}
|
||||||
self._prefixes.append(prefix)
|
self._prefixes.append(prefix)
|
||||||
|
|
||||||
@@ -1576,6 +1582,44 @@ class ModelScanner:
|
|||||||
"""
|
"""
|
||||||
return self._last_reconcile_summary
|
return self._last_reconcile_summary
|
||||||
|
|
||||||
|
def get_configured_model_roots(self) -> List[str]:
|
||||||
|
"""Roots the user configured for this type, including unavailable ones.
|
||||||
|
|
||||||
|
``get_model_roots()` answers "what can be walked right now"; this answers
|
||||||
|
"what did the user configure", which is what the refresh menu reports and
|
||||||
|
what must survive a switched-off drive.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from ..config import config
|
||||||
|
except Exception: # pragma: no cover - import guard
|
||||||
|
return []
|
||||||
|
|
||||||
|
# The checkpoint page scans two configured keys.
|
||||||
|
keys = ('checkpoint', 'unet') if self.model_type == 'checkpoint' else (self.model_type,)
|
||||||
|
roots: List[str] = []
|
||||||
|
seen: Set[str] = set()
|
||||||
|
for key in keys:
|
||||||
|
for root in config.configured_roots_for(key):
|
||||||
|
if root and root not in seen:
|
||||||
|
seen.add(root)
|
||||||
|
roots.append(root)
|
||||||
|
return roots
|
||||||
|
|
||||||
|
def refresh_model_roots(self) -> List[str]:
|
||||||
|
"""Admit configured roots whose directory is readable again.
|
||||||
|
|
||||||
|
Append-only and idempotent: a drive plugged in while the process runs
|
||||||
|
becomes scannable without a restart. Called before describing the roots
|
||||||
|
(`/roots`) and before a scan, so both see the same list.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from ..config import config
|
||||||
|
|
||||||
|
return config.admit_configured_roots()
|
||||||
|
except Exception as exc: # pragma: no cover - defensive logging
|
||||||
|
logger.debug("Failed to refresh model roots: %s", exc)
|
||||||
|
return []
|
||||||
|
|
||||||
def describe_model_roots(self) -> List[Dict[str, Any]]:
|
def describe_model_roots(self) -> List[Dict[str, Any]]:
|
||||||
"""Describe the configured roots for the refresh scope menu.
|
"""Describe the configured roots for the refresh scope menu.
|
||||||
|
|
||||||
@@ -1591,21 +1635,30 @@ class ModelScanner:
|
|||||||
seen.add(root)
|
seen.add(root)
|
||||||
roots.append(root)
|
roots.append(root)
|
||||||
|
|
||||||
labels = _root_display_labels(roots)
|
# Configured roots that are missing from the live list: their directory
|
||||||
|
# did not exist when it was built (a drive switched off at startup).
|
||||||
|
# They are reported so the menu offers a row in both states instead of
|
||||||
|
# hiding the root until the process is restarted.
|
||||||
|
unavailable = [
|
||||||
|
root for root in self.get_configured_model_roots() if root not in seen
|
||||||
|
]
|
||||||
|
|
||||||
|
labels = _root_display_labels(roots + unavailable)
|
||||||
cached_paths = (
|
cached_paths = (
|
||||||
{item.get('file_path', '') for item in self._cache.raw_data}
|
{item.get('file_path', '') for item in self._cache.raw_data}
|
||||||
if self._cache is not None
|
if self._cache is not None
|
||||||
else set()
|
else set()
|
||||||
)
|
)
|
||||||
counts = _count_cached_entries_per_root(cached_paths, roots)
|
counts = _count_cached_entries_per_root(cached_paths, roots + unavailable)
|
||||||
return [
|
return [
|
||||||
{
|
{
|
||||||
'path': root.replace(os.sep, '/'),
|
'path': root.replace(os.sep, '/'),
|
||||||
'label': labels[root],
|
'label': labels[root],
|
||||||
'reachable': os.path.exists(root),
|
'reachable': os.path.exists(root),
|
||||||
|
'available': root in seen,
|
||||||
'models': counts.get(root, 0),
|
'models': counts.get(root, 0),
|
||||||
}
|
}
|
||||||
for root in roots
|
for root in roots + unavailable
|
||||||
]
|
]
|
||||||
|
|
||||||
async def _initialize_cache(self) -> None:
|
async def _initialize_cache(self) -> None:
|
||||||
@@ -1732,7 +1785,17 @@ class ModelScanner:
|
|||||||
seen_roots.add(root_path)
|
seen_roots.add(root_path)
|
||||||
configured_roots.append(root_path)
|
configured_roots.append(root_path)
|
||||||
|
|
||||||
all_labels = _root_display_labels(configured_roots)
|
# Configured roots that never made it into the live list (their
|
||||||
|
# directory was missing when it was built). They are outside the scan
|
||||||
|
# scope entirely — which is exactly why their cached entries survive
|
||||||
|
# — so they only need reporting, and their labels must not collide
|
||||||
|
# with the live ones.
|
||||||
|
configured_only = [
|
||||||
|
root for root in self.get_configured_model_roots()
|
||||||
|
if root and root not in seen_roots
|
||||||
|
]
|
||||||
|
|
||||||
|
all_labels = _root_display_labels(configured_roots + configured_only)
|
||||||
scope_roots = [
|
scope_roots = [
|
||||||
root for root in configured_roots
|
root for root in configured_roots
|
||||||
if scope is None or scope.roots is None or root in scope.roots
|
if scope is None or scope.roots is None or root in scope.roots
|
||||||
@@ -1755,6 +1818,29 @@ class ModelScanner:
|
|||||||
continue
|
continue
|
||||||
roots.append(root_path)
|
roots.append(root_path)
|
||||||
|
|
||||||
|
# Report the roots that were filtered out of the live list, so a
|
||||||
|
# switched-off drive no longer keeps its models in silence. A scoped
|
||||||
|
# scan stays quiet about them: with explicit roots the caller asked
|
||||||
|
# for specific ones, and for a folder only the roots that actually
|
||||||
|
# hold cached entries of that folder are relevant.
|
||||||
|
if configured_only and (scope is None or scope.roots is None):
|
||||||
|
folder_candidates = [
|
||||||
|
(root, _scope_prefix(root, scope_folder)) for root in configured_only
|
||||||
|
]
|
||||||
|
folder_counts = _count_cached_entries_for_prefixes(
|
||||||
|
cached_paths, folder_candidates
|
||||||
|
)
|
||||||
|
for root_path, _prefix in folder_candidates:
|
||||||
|
kept = folder_counts.get(root_path, 0)
|
||||||
|
if scope_folder is not None and not kept:
|
||||||
|
continue
|
||||||
|
unreachable.add(root_path, 'root_unavailable', kept=kept)
|
||||||
|
skipped_roots.append({
|
||||||
|
'path': root_path.replace(os.sep, '/'),
|
||||||
|
'label': all_labels.get(root_path, root_path),
|
||||||
|
'reason': 'root_unavailable',
|
||||||
|
})
|
||||||
|
|
||||||
self._collect_offline_symlink_prefixes(roots, unreachable)
|
self._collect_offline_symlink_prefixes(roots, unreachable)
|
||||||
|
|
||||||
# Scope prefixes come from every root the caller asked for (even the
|
# Scope prefixes come from every root the caller asked for (even the
|
||||||
|
|||||||
@@ -0,0 +1,132 @@
|
|||||||
|
"""Wave 6: configured roots that are unavailable must be reportable and
|
||||||
|
re-admitted append-only, never dropped from the live lists."""
|
||||||
|
|
||||||
|
import os
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from py import config as config_module
|
||||||
|
|
||||||
|
|
||||||
|
def _make_config(**overrides) -> config_module.Config:
|
||||||
|
config = config_module.Config.__new__(config_module.Config)
|
||||||
|
config._path_mappings = {}
|
||||||
|
config._preview_root_paths = set()
|
||||||
|
config._cached_fingerprint = None
|
||||||
|
config.loras_roots = []
|
||||||
|
config.extra_loras_roots = []
|
||||||
|
config.base_models_roots = []
|
||||||
|
config.checkpoints_roots = []
|
||||||
|
config.unet_roots = []
|
||||||
|
config.embeddings_roots = []
|
||||||
|
config.extra_checkpoints_roots = []
|
||||||
|
config.extra_unet_roots = []
|
||||||
|
config.extra_embeddings_roots = []
|
||||||
|
config.other_roots = []
|
||||||
|
config.recipes_path = ""
|
||||||
|
config.other_root_subtypes = {}
|
||||||
|
config.other_folder_roots = {}
|
||||||
|
config._configured_root_paths = {}
|
||||||
|
for key, value in overrides.items():
|
||||||
|
setattr(config, key, value)
|
||||||
|
return config
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize(path) -> str:
|
||||||
|
return os.path.normpath(str(path)).replace(os.sep, "/")
|
||||||
|
|
||||||
|
|
||||||
|
class TestAppendNewPaths:
|
||||||
|
def test_appends_without_reordering_or_duplicating(self):
|
||||||
|
merged = config_module._append_new_paths(
|
||||||
|
["/b/loras", "/a/loras"], ["/a/loras", "/c/loras", "", None]
|
||||||
|
)
|
||||||
|
# Existing order is preserved (it derives *_roots[0] consumers) and the
|
||||||
|
# new root lands at the end.
|
||||||
|
assert merged == ["/b/loras", "/a/loras", "/c/loras"]
|
||||||
|
|
||||||
|
def test_matches_case_insensitively(self):
|
||||||
|
merged = config_module._append_new_paths(["C:/Loras"], ["c:/loras"])
|
||||||
|
assert merged == ["C:/Loras"]
|
||||||
|
|
||||||
|
|
||||||
|
class TestAdmitConfiguredRoots:
|
||||||
|
def test_admits_a_root_that_became_readable(self, tmp_path):
|
||||||
|
primary = tmp_path / "primary"
|
||||||
|
primary.mkdir()
|
||||||
|
drive = tmp_path / "drive-Z" / "loras"
|
||||||
|
|
||||||
|
config = _make_config(
|
||||||
|
loras_roots=[str(primary)],
|
||||||
|
_configured_root_paths={"lora": [str(primary), str(drive)]},
|
||||||
|
)
|
||||||
|
assert config.configured_roots_for("lora") == [str(primary), str(drive)]
|
||||||
|
|
||||||
|
# Still switched off: nothing to admit.
|
||||||
|
assert config.admit_configured_roots() == []
|
||||||
|
assert config.loras_roots == [str(primary)]
|
||||||
|
|
||||||
|
drive.mkdir(parents=True)
|
||||||
|
admitted = config.admit_configured_roots()
|
||||||
|
|
||||||
|
assert admitted == [str(drive)]
|
||||||
|
assert config.loras_roots == [str(primary), str(drive)]
|
||||||
|
# Idempotent: a second call adds nothing.
|
||||||
|
assert config.admit_configured_roots() == []
|
||||||
|
|
||||||
|
def test_never_removes_a_root_that_disappeared(self, tmp_path):
|
||||||
|
primary = tmp_path / "primary"
|
||||||
|
primary.mkdir()
|
||||||
|
drive = tmp_path / "drive-Z" / "loras"
|
||||||
|
drive.mkdir(parents=True)
|
||||||
|
|
||||||
|
config = _make_config(
|
||||||
|
loras_roots=[str(primary), str(drive)],
|
||||||
|
_configured_root_paths={"lora": [str(drive), str(primary)]},
|
||||||
|
)
|
||||||
|
|
||||||
|
# The drive is switched off after the lists were built: the live list
|
||||||
|
# keeps it (reporting and pruning decide separately) and admission must
|
||||||
|
# not "clean it up" — a later settings save would persist the loss.
|
||||||
|
drive.rmdir()
|
||||||
|
config.admit_configured_roots()
|
||||||
|
|
||||||
|
assert config.loras_roots == [str(primary), str(drive)]
|
||||||
|
|
||||||
|
def test_admits_every_model_type_and_its_side_maps(self, tmp_path):
|
||||||
|
checkpoint_dir = tmp_path / "checkpoints"
|
||||||
|
unet_dir = tmp_path / "unet"
|
||||||
|
embedding_dir = tmp_path / "embeddings"
|
||||||
|
other_dir = tmp_path / "vae"
|
||||||
|
for directory in (checkpoint_dir, unet_dir, embedding_dir, other_dir):
|
||||||
|
directory.mkdir()
|
||||||
|
|
||||||
|
config = _make_config(
|
||||||
|
_configured_root_paths={
|
||||||
|
"checkpoint": [str(checkpoint_dir)],
|
||||||
|
"unet": [str(unet_dir)],
|
||||||
|
"embedding": [str(embedding_dir)],
|
||||||
|
"other": [str(other_dir)],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
config._get_enabled_other_folder_keys = lambda: ["vae"]
|
||||||
|
config._configured_other_paths_for_key = lambda key: [str(other_dir)]
|
||||||
|
|
||||||
|
admitted = config.admit_configured_roots()
|
||||||
|
|
||||||
|
assert set(admitted) == {
|
||||||
|
_normalize(checkpoint_dir),
|
||||||
|
_normalize(unet_dir),
|
||||||
|
_normalize(embedding_dir),
|
||||||
|
_normalize(other_dir),
|
||||||
|
}
|
||||||
|
assert config.base_models_roots == [
|
||||||
|
_normalize(checkpoint_dir),
|
||||||
|
_normalize(unet_dir),
|
||||||
|
]
|
||||||
|
assert config.checkpoints_roots == [_normalize(checkpoint_dir)]
|
||||||
|
assert config.unet_roots == [_normalize(unet_dir)]
|
||||||
|
assert config.embeddings_roots == [_normalize(embedding_dir)]
|
||||||
|
assert config.other_roots == [_normalize(other_dir)]
|
||||||
|
assert config.other_root_subtypes[_normalize(other_dir)] == "vae"
|
||||||
|
assert config.other_folder_roots["vae"] == [_normalize(other_dir)]
|
||||||
@@ -213,21 +213,43 @@ class ScopedScanService:
|
|||||||
|
|
||||||
model_type = "lora"
|
model_type = "lora"
|
||||||
|
|
||||||
def __init__(self, roots=None, summary=None, cancelled=False):
|
def __init__(self, roots=None, summary=None, cancelled=False, unavailable=None):
|
||||||
self._roots = list(roots or [])
|
self._roots = list(roots or [])
|
||||||
|
self._unavailable = list(unavailable or [])
|
||||||
self._summary = summary
|
self._summary = summary
|
||||||
self.cancelled = cancelled
|
self.cancelled = cancelled
|
||||||
self.scanner = SimpleNamespace(is_cancelled=lambda: self.cancelled)
|
self.scanner = SimpleNamespace(is_cancelled=lambda: self.cancelled)
|
||||||
self.received_scope = "not-called"
|
self.received_scope = "not-called"
|
||||||
self.received_rebuild = None
|
self.received_rebuild = None
|
||||||
|
self.root_refreshes = 0
|
||||||
|
|
||||||
def get_model_roots(self):
|
def get_model_roots(self):
|
||||||
return list(self._roots)
|
return list(self._roots)
|
||||||
|
|
||||||
|
def refresh_model_roots(self):
|
||||||
|
"""Mimic the scanner admitting roots that became readable again."""
|
||||||
|
self.root_refreshes += 1
|
||||||
|
return []
|
||||||
|
|
||||||
def describe_model_roots(self):
|
def describe_model_roots(self):
|
||||||
return [
|
return [
|
||||||
{"path": root, "label": root.rsplit("/", 1)[-1], "reachable": True, "models": 7}
|
{
|
||||||
|
"path": root,
|
||||||
|
"label": root.rsplit("/", 1)[-1],
|
||||||
|
"reachable": True,
|
||||||
|
"available": True,
|
||||||
|
"models": 7,
|
||||||
|
}
|
||||||
for root in self._roots
|
for root in self._roots
|
||||||
|
] + [
|
||||||
|
{
|
||||||
|
"path": root,
|
||||||
|
"label": root.rsplit("/", 1)[-1],
|
||||||
|
"reachable": False,
|
||||||
|
"available": False,
|
||||||
|
"models": 3,
|
||||||
|
}
|
||||||
|
for root in self._unavailable
|
||||||
]
|
]
|
||||||
|
|
||||||
async def scan_models(self, force_refresh=False, rebuild_cache=False, scope=None):
|
async def scan_models(self, force_refresh=False, rebuild_cache=False, scope=None):
|
||||||
@@ -265,6 +287,8 @@ async def test_scan_models_accepts_roots_param():
|
|||||||
assert service.received_scope is not None
|
assert service.received_scope is not None
|
||||||
assert service.received_scope.roots == ("/mnt/a",)
|
assert service.received_scope.roots == ("/mnt/a",)
|
||||||
assert service.received_rebuild is False
|
assert service.received_rebuild is False
|
||||||
|
# A drive plugged in after startup is admitted before the roots are validated.
|
||||||
|
assert service.root_refreshes == 1
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
@@ -307,6 +331,28 @@ async def test_scan_models_rejects_roots_with_full_rebuild():
|
|||||||
assert service.received_scope == "not-called"
|
assert service.received_scope == "not-called"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_get_model_roots_reports_configured_but_unavailable_roots():
|
||||||
|
"""Wave 6: a root whose directory was missing at startup is still reported."""
|
||||||
|
service = ScopedScanService(
|
||||||
|
roots=["/mnt/a"], unavailable=["/mnt/drive-Z/loras"]
|
||||||
|
)
|
||||||
|
handler = ModelQueryHandler(service=service, logger=logging.getLogger(__name__))
|
||||||
|
|
||||||
|
response = await handler.get_model_roots(SimpleNamespace(query=QueryParams({})))
|
||||||
|
payload = json.loads(response.text)
|
||||||
|
|
||||||
|
assert response.status == 200
|
||||||
|
assert payload["roots"] == ["/mnt/a"]
|
||||||
|
details = {entry["path"]: entry for entry in payload["root_details"]}
|
||||||
|
assert details["/mnt/a"]["available"] is True
|
||||||
|
assert details["/mnt/drive-Z/loras"]["available"] is False
|
||||||
|
assert details["/mnt/drive-Z/loras"]["reachable"] is False
|
||||||
|
assert details["/mnt/drive-Z/loras"]["models"] == 3
|
||||||
|
# Describing the roots admits anything that became readable again.
|
||||||
|
assert service.root_refreshes == 1
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_scan_models_accepts_folder_param():
|
async def test_scan_models_accepts_folder_param():
|
||||||
service = ScopedScanService(roots=["/mnt/a", "/mnt/b"], summary=SUMMARY)
|
service = ScopedScanService(roots=["/mnt/a", "/mnt/b"], summary=SUMMARY)
|
||||||
|
|||||||
@@ -2649,6 +2649,112 @@ async def test_reconcile_folder_scope_covers_every_root_holding_it(tmp_path: Pat
|
|||||||
assert _normalize_path(primary / "only-primary" / "c.txt") in paths
|
assert _normalize_path(primary / "only-primary" / "c.txt") in paths
|
||||||
|
|
||||||
|
|
||||||
|
async def _startup_filtered_scanner(tmp_path: Path, monkeypatch, configured_missing: Path):
|
||||||
|
"""Scanner whose live root list lacks a root that the config still holds.
|
||||||
|
|
||||||
|
Models the startup case: the directory did not exist when the live list was
|
||||||
|
built (drive switched off), so `Config` filtered it out while its cached
|
||||||
|
entries are still there.
|
||||||
|
"""
|
||||||
|
primary = tmp_path / "primary"
|
||||||
|
(primary / "pack").mkdir(parents=True)
|
||||||
|
(primary / "pack" / "a.txt").write_text("a", encoding="utf-8")
|
||||||
|
configured_missing.mkdir(parents=True)
|
||||||
|
(configured_missing / "pack").mkdir()
|
||||||
|
(configured_missing / "pack" / "b.txt").write_text("b", encoding="utf-8")
|
||||||
|
|
||||||
|
scanner = MultiRootDummyScanner([primary, configured_missing])
|
||||||
|
await scanner._initialize_cache()
|
||||||
|
# The drive goes away before the process starts, as far as the live list is
|
||||||
|
# concerned: keep the roots the filter would have kept.
|
||||||
|
shutil.rmtree(configured_missing)
|
||||||
|
scanner._roots = [str(primary)]
|
||||||
|
monkeypatch.setattr(
|
||||||
|
scanner,
|
||||||
|
"get_configured_model_roots",
|
||||||
|
lambda: [str(primary), str(configured_missing)],
|
||||||
|
)
|
||||||
|
return scanner, primary, configured_missing
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_describe_model_roots_reports_unavailable_configured_root(
|
||||||
|
tmp_path: Path, monkeypatch
|
||||||
|
):
|
||||||
|
configured_missing = tmp_path / "drive-Z" / "loras"
|
||||||
|
scanner, primary, missing = await _startup_filtered_scanner(
|
||||||
|
tmp_path, monkeypatch, configured_missing
|
||||||
|
)
|
||||||
|
|
||||||
|
described = scanner.describe_model_roots()
|
||||||
|
|
||||||
|
assert [entry["path"] for entry in described] == [
|
||||||
|
_normalize_path(primary),
|
||||||
|
_normalize_path(missing),
|
||||||
|
]
|
||||||
|
available, unavailable = described
|
||||||
|
assert available["reachable"] is True
|
||||||
|
assert available["available"] is True
|
||||||
|
# The missing root is reported with its cached count so the refresh menu can
|
||||||
|
# offer it as an offline row instead of hiding it until a restart.
|
||||||
|
assert unavailable["reachable"] is False
|
||||||
|
assert unavailable["available"] is False
|
||||||
|
assert unavailable["models"] == 1
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_reconcile_reports_root_filtered_out_at_startup(tmp_path: Path, monkeypatch):
|
||||||
|
configured_missing = tmp_path / "drive-Z" / "loras"
|
||||||
|
scanner, _primary, missing = await _startup_filtered_scanner(
|
||||||
|
tmp_path, monkeypatch, configured_missing
|
||||||
|
)
|
||||||
|
kept_path = _normalize_path(missing / "pack" / "b.txt")
|
||||||
|
|
||||||
|
summary = await scanner._reconcile_cache()
|
||||||
|
|
||||||
|
assert summary is not None
|
||||||
|
# Nothing is added or removed...
|
||||||
|
assert summary["added"] == 0
|
||||||
|
assert summary["removed"] == 0
|
||||||
|
# ...but the switched-off root is no longer silent: it is reported with the
|
||||||
|
# entries that were kept for it.
|
||||||
|
assert summary["skipped_roots"] == [
|
||||||
|
{
|
||||||
|
"path": _normalize_path(missing),
|
||||||
|
"label": missing.name,
|
||||||
|
"reason": "root_unavailable",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
assert summary["kept_unreachable"] == 1
|
||||||
|
assert summary["unavailable_paths"] == [
|
||||||
|
{"path": _normalize_path(missing), "reason": "root_unavailable", "kept": 1}
|
||||||
|
]
|
||||||
|
assert kept_path in {item["file_path"] for item in scanner._cache.raw_data}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_scoped_scan_stays_quiet_about_filtered_out_roots(tmp_path: Path, monkeypatch):
|
||||||
|
configured_missing = tmp_path / "drive-Z" / "loras"
|
||||||
|
scanner, primary, missing = await _startup_filtered_scanner(
|
||||||
|
tmp_path, monkeypatch, configured_missing
|
||||||
|
)
|
||||||
|
|
||||||
|
summary = await scanner._reconcile_cache(
|
||||||
|
scope=ReconcileScope(roots=(str(primary),))
|
||||||
|
)
|
||||||
|
|
||||||
|
assert summary is not None
|
||||||
|
# The caller asked for one root: a root it did not ask about is not its
|
||||||
|
# business, but its entries are still kept.
|
||||||
|
assert summary["scanned_roots"] == [primary.name]
|
||||||
|
assert "root_unavailable" not in {
|
||||||
|
entry["reason"] for entry in summary["skipped_roots"]
|
||||||
|
}
|
||||||
|
assert _normalize_path(missing / "pack" / "b.txt") in {
|
||||||
|
item["file_path"] for item in scanner._cache.raw_data
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_reconcile_keeps_entries_under_unreadable_dir(tmp_path: Path, monkeypatch):
|
async def test_reconcile_keeps_entries_under_unreadable_dir(tmp_path: Path, monkeypatch):
|
||||||
"""A directory os.walk cannot enter (permissions, I/O error, offline
|
"""A directory os.walk cannot enter (permissions, I/O error, offline
|
||||||
|
|||||||
Reference in New Issue
Block a user