mirror of
https://github.com/willmiao/ComfyUI-Lora-Manager.git
synced 2026-10-07 18:12:12 -03:00
feat(scanner): report and re-admit model roots that are unavailable (#1108)
`Config._dedupe_existing_paths()` drops roots whose directory does not exist
when the root list is built, so a drive that is switched off while LM starts was
not "an offline root" — it was not a root at all. The Refresh ▾ menu had no row
for it in either state, a full refresh reported nothing (its cached entries
survived only because they fall outside every configured prefix), and plugging
the drive back in changed nothing until the next restart.
* `Config` now records the configured (existence-unfiltered) paths per model
type — from ComfyUI's list in plugin mode, from the active library snapshot in
standalone mode, where the host mock filters non-existent paths itself — and
exposes them through `configured_roots_for()`.
* `ModelScanner.describe_model_roots()` appends configured-but-unavailable roots
with a live `reachable`, an `available` flag and their cached entry count, so
`/roots` (and therefore the menu) offers a row in both states: greyed while the
directory is missing, normal and clickable once it is back.
* `_reconcile_cache()` reports them as `skipped_roots` / `unavailable_paths` with
reason `root_unavailable` and counts their cached entries in `kept_unreachable`,
which is what makes the "N models kept" toast appear in the startup-offline
case. Report-only: they stay outside the scan scope, so nothing about pruning
changes. A root-scoped scan stays quiet about roots it was not asked about, and
a folder-scoped scan only mentions a missing root that holds cached entries of
that folder.
* `Config.admit_configured_roots()` re-runs the per-type prepare helpers against
the configured paths and APPENDS what is readable now (plus the
checkpoint/unet/other side maps), then refreshes the preview allowlist, so a
drive plugged in mid-session can be scanned without restarting. Append-only and
order-preserving by design: removing a root mid-session would let a later
settings save persist the loss, and re-sorting would move `*_roots[0]`, which
derives the recipes directory and the usage-stats file location. `_roots[0]`
never moves.
* `/roots` and `/scan` admit first, so `describe_model_roots()` stays a pure read
and a caller that never opens the menu (the browser extension) still gets a
drive that came back.
Verified on the sandbox with drive-Z switched off before startup: `/roots`
reports `drive-Z reachable=false available=false models=60`, a full refresh
returns `skipped_roots=[drive-Z root_unavailable]`,
`unavailable_paths=[{... kept: 60}]`, `kept_unreachable=60`, and all 432 models
stay cached. Renaming the drive back while the server runs admits it
(`available=true`) and `GET /scan?roots=<drive-Z>` walks it with no restart.
3714 passed, 7 skipped; frontend 1495 passed (149 files); vue widgets 96 passed.
This commit is contained in:
@@ -2,9 +2,9 @@
|
||||
|
||||
**Issue:** [#1108](https://github.com/willmiao/ComfyUI-Lora-Manager/issues/1108) — scan a single
|
||||
folder/root instead of the whole library.
|
||||
**Status:** **P1 and P2 implemented** (2026-10-07). P1 shipped in `470d85cc` (translations in
|
||||
`bd184559`); P2 (folder scope + the sidebar entry) is in the working tree. Supersedes the earlier
|
||||
draft's per-root *status panel* and persisted *unreachable-subtree* state (both dropped, see
|
||||
**Status:** **P1, P2 and Wave 6 implemented** (2026-10-07). P1 shipped in `470d85cc` (translations
|
||||
in `bd184559`), P2 in `12930ce7` (translations in `0941f992`). Supersedes the earlier draft's
|
||||
per-root *status panel* and persisted *unreachable-subtree* state (both dropped, see
|
||||
"Must NOT have").
|
||||
|
||||
Implementation notes / deviations from the draft below:
|
||||
@@ -271,6 +271,85 @@ and in the exact pruning predicate; both get explicit tests.
|
||||
`scope_label=pack000`, keeps drive-Z's 6 entries under that folder (`kept_unreachable=6`) and
|
||||
leaves all 420 models cached.
|
||||
|
||||
## Wave 6 — configured vs currently available roots (approved follow-up)
|
||||
|
||||
The P1/P2 work exposed an asymmetry: `Config._dedupe_existing_paths()` drops roots whose directory
|
||||
does not exist **at the moment the root list is built** (startup, or applying a library snapshot),
|
||||
so a drive that is switched off while LM starts is not "an offline root" — it is not a root at all.
|
||||
Consequences, all verified on the sandbox:
|
||||
|
||||
* `/roots` does not list it, so the Refresh ▾ menu shows no row for it in either state;
|
||||
* a full refresh reports nothing (`kept_unreachable: 0`, no toast) even though its cached entries
|
||||
are kept — they survive only because they fall outside every configured root prefix;
|
||||
* plugging the drive back in does not bring the row back: the in-memory list is not re-validated;
|
||||
* **plugin mode only:** `Config.save_folder_paths_to_settings()` (called from `Config.__init__`)
|
||||
persists `target_folder_paths["loras"] = list(self.loras_roots)` through
|
||||
`upsert_library(folder_paths=...)`, which **replaces** the library's paths. Starting ComfyUI with
|
||||
a drive switched off therefore *erases that path from `settings.json`* — a configuration loss,
|
||||
not just a display gap. Extra paths are unaffected (that call reuses the stored
|
||||
`extra_folder_paths`), and the same pattern applies to checkpoints/unet/embeddings/other-model
|
||||
primary paths.
|
||||
|
||||
### Must have
|
||||
|
||||
1. **Never erase an unavailable root from the library config.** `Config` records the *configured*
|
||||
(existence-unfiltered) primary paths per model type while building the live lists, and
|
||||
`save_folder_paths_to_settings()` persists those instead of the filtered ones.
|
||||
`_resolve_valid_default_root()` receives the configured paths in `allowed_paths` too, so a
|
||||
`default_*_root` that sits on a switched-off drive is not "repaired" away.
|
||||
2. **Report configured-but-unavailable roots.** `Config.configured_roots_for(model_type)` feeds
|
||||
`ModelScanner.describe_model_roots()`, which appends them with `available: false`, a live
|
||||
`reachable` and their cached entry count — so the Refresh ▾ menu has a row in **both** states
|
||||
(greyed while the directory is missing, normal and clickable once it is back). The reconcile
|
||||
summary reports them as `skipped_roots` / `unavailable_paths` with reason `root_unavailable` and
|
||||
counts their cached entries in `kept_unreachable`, which is what makes the "N models kept" toast
|
||||
appear in the startup-offline case.
|
||||
3. **Admit them again, append-only.** `Config.admit_configured_roots()` re-runs the per-type
|
||||
prepare helpers against the configured paths and **appends** what exists now
|
||||
(plus the checkpoint/unet/other side maps) to the live lists, then refreshes the preview
|
||||
allowlist. Called from `/roots` and `/scan`, so a drive plugged in mid-session can be scanned
|
||||
without restarting. Appending (never re-sorting) keeps `loras_roots[0]` — which derives the
|
||||
recipes directory and the usage-stats file location — stable for the whole session.
|
||||
|
||||
### Must NOT have
|
||||
|
||||
* NO removal of a root from the live lists mid-session (that is what would let a later settings
|
||||
save persist a reduced configuration, and it would move `*_roots[0]`).
|
||||
* NO change to the scan/prune scope: a root that is still unavailable stays out of scope, so its
|
||||
entries remain "kept because out of scope" exactly as today.
|
||||
* NO persisted unavailable-state, no DB schema change, no per-entry filesystem probe.
|
||||
|
||||
### Where "configured" comes from
|
||||
|
||||
`Config` records the existence-unfiltered paths while building the live lists, but the *source*
|
||||
differs per mode and that matters:
|
||||
|
||||
* **Plugin mode:** `folder_paths.get_folder_paths()` is ComfyUI's own list, unfiltered — a path the
|
||||
user removed there must be forgotten, one that is merely missing must be kept. The host list wins.
|
||||
* **Standalone mode:** `standalone.MockFolderPaths.get_folder_paths()` already filters
|
||||
`os.path.exists` out of `settings.json`, so the host list cannot answer "what did the user
|
||||
configure". The active **library snapshot** (`libraries[<active>].folder_paths`, falling back to
|
||||
the top-level `folder_paths`) is the record, and it wins there.
|
||||
(`_remember_library_configured_paths()` implements the split.)
|
||||
|
||||
### Verification
|
||||
|
||||
* Config: a configured path that does not exist is still written back by
|
||||
`save_folder_paths_to_settings()`; a path the host no longer configures is still dropped.
|
||||
* Promotion: create the directory after the lists were built → `admit_configured_roots()` adds it
|
||||
**at the end**, keeps every existing root in place, and adds nothing twice.
|
||||
* Scanner: `describe_model_roots()` reports the missing root (`available: false`, cached count);
|
||||
a full refresh lists it in `skipped_roots`/`unavailable_paths` and keeps its entries; `/scan`
|
||||
with that root works once the directory is back.
|
||||
* Sandbox, drive-Z gone **before** startup: `/roots` reports `drive-Z` with
|
||||
`reachable=false available=false models=60`; a full refresh returns
|
||||
`skipped_roots=[drive-Z root_unavailable]`, `unavailable_paths=[{... kept: 60}]`,
|
||||
`kept_unreachable=60` and leaves all 432 models cached. *(Both measured; before this wave the
|
||||
menu had no row and the refresh reported nothing.)*
|
||||
* Sandbox, drive renamed back **while the server runs**: `/roots` admits it
|
||||
(`available=true reachable=true`) and `GET /scan?roots=<drive-Z>` walks it
|
||||
(`scanned_roots=['drive-Z']`, 0 added / 0 removed) — no restart required. *(Measured.)*
|
||||
|
||||
## Known limitations (accepted)
|
||||
|
||||
- **Nested symlinks** (a symlink *below* a root pointing at another drive) are only covered when
|
||||
|
||||
@@ -1679,6 +1679,105 @@ class Config:
|
||||
"""
|
||||
return list((getattr(self, "_configured_root_paths", None) or {}).get(model_type, []))
|
||||
|
||||
def admit_configured_roots(self) -> List[str]:
|
||||
"""Append configured roots that became readable again, never removing any.
|
||||
|
||||
The live lists are built once at startup (and when a library snapshot is
|
||||
applied), so a drive plugged in later is invisible until the process is
|
||||
restarted. Re-running the per-type prepare helpers against the configured
|
||||
paths and appending only the new entries is what makes "plug the drive
|
||||
in, scan it" work without a restart. Nothing is ever dropped or
|
||||
re-ordered — see ``_append_new_paths`` for why order matters.
|
||||
|
||||
Returns the paths that were admitted by this call.
|
||||
"""
|
||||
admitted: List[str] = []
|
||||
|
||||
def _extend(current, candidates) -> List[str]:
|
||||
merged = _append_new_paths(list(current or []), candidates)
|
||||
admitted.extend(
|
||||
path for path in merged if path not in (current or [])
|
||||
)
|
||||
return merged
|
||||
|
||||
try:
|
||||
lora_configured = self.configured_roots_for("lora")
|
||||
if lora_configured:
|
||||
self.loras_roots = _extend(
|
||||
self.loras_roots, self._prepare_lora_paths(lora_configured)
|
||||
)
|
||||
|
||||
checkpoint_configured = self.configured_roots_for("checkpoint")
|
||||
unet_configured = self.configured_roots_for("unet")
|
||||
if checkpoint_configured or unet_configured:
|
||||
(
|
||||
all_roots,
|
||||
checkpoint_roots,
|
||||
unet_roots,
|
||||
) = self._prepare_checkpoint_paths(
|
||||
checkpoint_configured, unet_configured
|
||||
)
|
||||
self.base_models_roots = _extend(self.base_models_roots, all_roots)
|
||||
self.checkpoints_roots = _extend(self.checkpoints_roots, checkpoint_roots)
|
||||
self.unet_roots = _extend(self.unet_roots, unet_roots)
|
||||
|
||||
embedding_configured = self.configured_roots_for("embedding")
|
||||
if embedding_configured:
|
||||
self.embeddings_roots = _extend(
|
||||
self.embeddings_roots,
|
||||
self._prepare_embedding_paths(embedding_configured),
|
||||
)
|
||||
|
||||
other_configured = self.configured_roots_for("other")
|
||||
if other_configured:
|
||||
(
|
||||
other_roots,
|
||||
other_subtypes,
|
||||
other_per_key,
|
||||
) = self._prepare_other_paths(
|
||||
{
|
||||
key: [
|
||||
path
|
||||
for path in self._configured_other_paths_for_key(key)
|
||||
]
|
||||
for key in self._get_enabled_other_folder_keys()
|
||||
}
|
||||
)
|
||||
self.other_roots = _extend(self.other_roots, other_roots)
|
||||
for root, sub_type in other_subtypes.items():
|
||||
self.other_root_subtypes.setdefault(root, sub_type)
|
||||
for key, roots in other_per_key.items():
|
||||
self.other_folder_roots[key] = _append_new_paths(
|
||||
self.other_folder_roots.get(key, []), roots
|
||||
)
|
||||
except Exception as exc: # pragma: no cover - defensive logging
|
||||
logger.warning("Failed to admit configured roots: %s", exc)
|
||||
return admitted
|
||||
|
||||
if admitted:
|
||||
logger.info(
|
||||
"Admitted %d previously unavailable model root(s): %s",
|
||||
len(admitted),
|
||||
", ".join(admitted),
|
||||
)
|
||||
try:
|
||||
# Previews of the newly admitted drive must be servable again.
|
||||
self.refresh_preview_roots()
|
||||
except Exception as exc: # pragma: no cover - defensive logging
|
||||
logger.debug("Failed to refresh preview roots: %s", exc)
|
||||
return admitted
|
||||
|
||||
def _configured_other_paths_for_key(self, key: str) -> List[str]:
|
||||
"""Configured other-model paths for one folder_paths key (may not exist)."""
|
||||
try:
|
||||
return [
|
||||
path.strip()
|
||||
for path in folder_paths.get_folder_paths(key) or []
|
||||
if isinstance(path, str) and path.strip()
|
||||
]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
def refresh_other_roots(self) -> None:
|
||||
"""Rebuild other-model roots after the management toggles changed.
|
||||
|
||||
|
||||
@@ -1163,6 +1163,11 @@ class ModelQueryHandler:
|
||||
except ValueError as exc:
|
||||
return web.json_response({"error": str(exc)}, status=400)
|
||||
|
||||
# A drive that was plugged in after startup must be scannable without
|
||||
# a restart, and callers that never open the Refresh menu (the browser
|
||||
# extension) have no other chance to admit it.
|
||||
self._service.refresh_model_roots()
|
||||
|
||||
if requested_roots:
|
||||
configured = self._service.get_model_roots()
|
||||
unknown = [root for root in requested_roots if root not in configured]
|
||||
@@ -1206,6 +1211,9 @@ class ModelQueryHandler:
|
||||
|
||||
async def get_model_roots(self, request: web.Request) -> web.Response:
|
||||
try:
|
||||
# A drive plugged in after startup is admitted here, so the menu shows
|
||||
# it as a normal row instead of "unavailable" until the next restart.
|
||||
self._service.refresh_model_roots()
|
||||
roots = self._service.get_model_roots()
|
||||
try:
|
||||
root_details = self._service.describe_model_roots()
|
||||
|
||||
@@ -954,9 +954,17 @@ class BaseModelService(ABC):
|
||||
return self.scanner.last_reconcile_summary
|
||||
|
||||
def describe_model_roots(self) -> List[Dict[str, Any]]:
|
||||
"""Describe configured roots (label / reachability / cached count)."""
|
||||
"""Describe configured roots (label / reachability / cached count).
|
||||
|
||||
Pure read: callers that want a drive which was plugged in after startup to
|
||||
appear as a normal row call :meth:`refresh_model_roots` first.
|
||||
"""
|
||||
return self.scanner.describe_model_roots()
|
||||
|
||||
def refresh_model_roots(self) -> List[str]:
|
||||
"""Admit configured roots that became readable again (append-only)."""
|
||||
return self.scanner.refresh_model_roots()
|
||||
|
||||
async def get_model_info_by_name(self, name: str):
|
||||
"""Get model information by name"""
|
||||
return await self.scanner.get_model_info_by_name(name)
|
||||
|
||||
@@ -257,7 +257,13 @@ class _UnreachablePaths:
|
||||
self._prefixes: List[str] = []
|
||||
self._details: Dict[str, Dict[str, Any]] = {}
|
||||
|
||||
def add(self, path: str, reason: str) -> None:
|
||||
def add(self, path: str, reason: str, kept: int = 0) -> None:
|
||||
"""Register an unreadable prefix.
|
||||
|
||||
``kept`` seeds the count for a root whose entries never reach the prune
|
||||
phase because they are outside the scan scope entirely (a configured root
|
||||
that was not even part of the live root list).
|
||||
"""
|
||||
prefix = _normalized_root_prefix(path)
|
||||
with self._lock:
|
||||
if prefix in self._details:
|
||||
@@ -265,7 +271,7 @@ class _UnreachablePaths:
|
||||
self._details[prefix] = {
|
||||
'path': path.replace(os.sep, '/'),
|
||||
'reason': reason,
|
||||
'kept': 0,
|
||||
'kept': kept,
|
||||
}
|
||||
self._prefixes.append(prefix)
|
||||
|
||||
@@ -1576,6 +1582,44 @@ class ModelScanner:
|
||||
"""
|
||||
return self._last_reconcile_summary
|
||||
|
||||
def get_configured_model_roots(self) -> List[str]:
|
||||
"""Roots the user configured for this type, including unavailable ones.
|
||||
|
||||
``get_model_roots()` answers "what can be walked right now"; this answers
|
||||
"what did the user configure", which is what the refresh menu reports and
|
||||
what must survive a switched-off drive.
|
||||
"""
|
||||
try:
|
||||
from ..config import config
|
||||
except Exception: # pragma: no cover - import guard
|
||||
return []
|
||||
|
||||
# The checkpoint page scans two configured keys.
|
||||
keys = ('checkpoint', 'unet') if self.model_type == 'checkpoint' else (self.model_type,)
|
||||
roots: List[str] = []
|
||||
seen: Set[str] = set()
|
||||
for key in keys:
|
||||
for root in config.configured_roots_for(key):
|
||||
if root and root not in seen:
|
||||
seen.add(root)
|
||||
roots.append(root)
|
||||
return roots
|
||||
|
||||
def refresh_model_roots(self) -> List[str]:
|
||||
"""Admit configured roots whose directory is readable again.
|
||||
|
||||
Append-only and idempotent: a drive plugged in while the process runs
|
||||
becomes scannable without a restart. Called before describing the roots
|
||||
(`/roots`) and before a scan, so both see the same list.
|
||||
"""
|
||||
try:
|
||||
from ..config import config
|
||||
|
||||
return config.admit_configured_roots()
|
||||
except Exception as exc: # pragma: no cover - defensive logging
|
||||
logger.debug("Failed to refresh model roots: %s", exc)
|
||||
return []
|
||||
|
||||
def describe_model_roots(self) -> List[Dict[str, Any]]:
|
||||
"""Describe the configured roots for the refresh scope menu.
|
||||
|
||||
@@ -1591,21 +1635,30 @@ class ModelScanner:
|
||||
seen.add(root)
|
||||
roots.append(root)
|
||||
|
||||
labels = _root_display_labels(roots)
|
||||
# Configured roots that are missing from the live list: their directory
|
||||
# did not exist when it was built (a drive switched off at startup).
|
||||
# They are reported so the menu offers a row in both states instead of
|
||||
# hiding the root until the process is restarted.
|
||||
unavailable = [
|
||||
root for root in self.get_configured_model_roots() if root not in seen
|
||||
]
|
||||
|
||||
labels = _root_display_labels(roots + unavailable)
|
||||
cached_paths = (
|
||||
{item.get('file_path', '') for item in self._cache.raw_data}
|
||||
if self._cache is not None
|
||||
else set()
|
||||
)
|
||||
counts = _count_cached_entries_per_root(cached_paths, roots)
|
||||
counts = _count_cached_entries_per_root(cached_paths, roots + unavailable)
|
||||
return [
|
||||
{
|
||||
'path': root.replace(os.sep, '/'),
|
||||
'label': labels[root],
|
||||
'reachable': os.path.exists(root),
|
||||
'available': root in seen,
|
||||
'models': counts.get(root, 0),
|
||||
}
|
||||
for root in roots
|
||||
for root in roots + unavailable
|
||||
]
|
||||
|
||||
async def _initialize_cache(self) -> None:
|
||||
@@ -1732,7 +1785,17 @@ class ModelScanner:
|
||||
seen_roots.add(root_path)
|
||||
configured_roots.append(root_path)
|
||||
|
||||
all_labels = _root_display_labels(configured_roots)
|
||||
# Configured roots that never made it into the live list (their
|
||||
# directory was missing when it was built). They are outside the scan
|
||||
# scope entirely — which is exactly why their cached entries survive
|
||||
# — so they only need reporting, and their labels must not collide
|
||||
# with the live ones.
|
||||
configured_only = [
|
||||
root for root in self.get_configured_model_roots()
|
||||
if root and root not in seen_roots
|
||||
]
|
||||
|
||||
all_labels = _root_display_labels(configured_roots + configured_only)
|
||||
scope_roots = [
|
||||
root for root in configured_roots
|
||||
if scope is None or scope.roots is None or root in scope.roots
|
||||
@@ -1755,6 +1818,29 @@ class ModelScanner:
|
||||
continue
|
||||
roots.append(root_path)
|
||||
|
||||
# Report the roots that were filtered out of the live list, so a
|
||||
# switched-off drive no longer keeps its models in silence. A scoped
|
||||
# scan stays quiet about them: with explicit roots the caller asked
|
||||
# for specific ones, and for a folder only the roots that actually
|
||||
# hold cached entries of that folder are relevant.
|
||||
if configured_only and (scope is None or scope.roots is None):
|
||||
folder_candidates = [
|
||||
(root, _scope_prefix(root, scope_folder)) for root in configured_only
|
||||
]
|
||||
folder_counts = _count_cached_entries_for_prefixes(
|
||||
cached_paths, folder_candidates
|
||||
)
|
||||
for root_path, _prefix in folder_candidates:
|
||||
kept = folder_counts.get(root_path, 0)
|
||||
if scope_folder is not None and not kept:
|
||||
continue
|
||||
unreachable.add(root_path, 'root_unavailable', kept=kept)
|
||||
skipped_roots.append({
|
||||
'path': root_path.replace(os.sep, '/'),
|
||||
'label': all_labels.get(root_path, root_path),
|
||||
'reason': 'root_unavailable',
|
||||
})
|
||||
|
||||
self._collect_offline_symlink_prefixes(roots, unreachable)
|
||||
|
||||
# Scope prefixes come from every root the caller asked for (even the
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
"""Wave 6: configured roots that are unavailable must be reportable and
|
||||
re-admitted append-only, never dropped from the live lists."""
|
||||
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
from py import config as config_module
|
||||
|
||||
|
||||
def _make_config(**overrides) -> config_module.Config:
|
||||
config = config_module.Config.__new__(config_module.Config)
|
||||
config._path_mappings = {}
|
||||
config._preview_root_paths = set()
|
||||
config._cached_fingerprint = None
|
||||
config.loras_roots = []
|
||||
config.extra_loras_roots = []
|
||||
config.base_models_roots = []
|
||||
config.checkpoints_roots = []
|
||||
config.unet_roots = []
|
||||
config.embeddings_roots = []
|
||||
config.extra_checkpoints_roots = []
|
||||
config.extra_unet_roots = []
|
||||
config.extra_embeddings_roots = []
|
||||
config.other_roots = []
|
||||
config.recipes_path = ""
|
||||
config.other_root_subtypes = {}
|
||||
config.other_folder_roots = {}
|
||||
config._configured_root_paths = {}
|
||||
for key, value in overrides.items():
|
||||
setattr(config, key, value)
|
||||
return config
|
||||
|
||||
|
||||
def _normalize(path) -> str:
|
||||
return os.path.normpath(str(path)).replace(os.sep, "/")
|
||||
|
||||
|
||||
class TestAppendNewPaths:
|
||||
def test_appends_without_reordering_or_duplicating(self):
|
||||
merged = config_module._append_new_paths(
|
||||
["/b/loras", "/a/loras"], ["/a/loras", "/c/loras", "", None]
|
||||
)
|
||||
# Existing order is preserved (it derives *_roots[0] consumers) and the
|
||||
# new root lands at the end.
|
||||
assert merged == ["/b/loras", "/a/loras", "/c/loras"]
|
||||
|
||||
def test_matches_case_insensitively(self):
|
||||
merged = config_module._append_new_paths(["C:/Loras"], ["c:/loras"])
|
||||
assert merged == ["C:/Loras"]
|
||||
|
||||
|
||||
class TestAdmitConfiguredRoots:
|
||||
def test_admits_a_root_that_became_readable(self, tmp_path):
|
||||
primary = tmp_path / "primary"
|
||||
primary.mkdir()
|
||||
drive = tmp_path / "drive-Z" / "loras"
|
||||
|
||||
config = _make_config(
|
||||
loras_roots=[str(primary)],
|
||||
_configured_root_paths={"lora": [str(primary), str(drive)]},
|
||||
)
|
||||
assert config.configured_roots_for("lora") == [str(primary), str(drive)]
|
||||
|
||||
# Still switched off: nothing to admit.
|
||||
assert config.admit_configured_roots() == []
|
||||
assert config.loras_roots == [str(primary)]
|
||||
|
||||
drive.mkdir(parents=True)
|
||||
admitted = config.admit_configured_roots()
|
||||
|
||||
assert admitted == [str(drive)]
|
||||
assert config.loras_roots == [str(primary), str(drive)]
|
||||
# Idempotent: a second call adds nothing.
|
||||
assert config.admit_configured_roots() == []
|
||||
|
||||
def test_never_removes_a_root_that_disappeared(self, tmp_path):
|
||||
primary = tmp_path / "primary"
|
||||
primary.mkdir()
|
||||
drive = tmp_path / "drive-Z" / "loras"
|
||||
drive.mkdir(parents=True)
|
||||
|
||||
config = _make_config(
|
||||
loras_roots=[str(primary), str(drive)],
|
||||
_configured_root_paths={"lora": [str(drive), str(primary)]},
|
||||
)
|
||||
|
||||
# The drive is switched off after the lists were built: the live list
|
||||
# keeps it (reporting and pruning decide separately) and admission must
|
||||
# not "clean it up" — a later settings save would persist the loss.
|
||||
drive.rmdir()
|
||||
config.admit_configured_roots()
|
||||
|
||||
assert config.loras_roots == [str(primary), str(drive)]
|
||||
|
||||
def test_admits_every_model_type_and_its_side_maps(self, tmp_path):
|
||||
checkpoint_dir = tmp_path / "checkpoints"
|
||||
unet_dir = tmp_path / "unet"
|
||||
embedding_dir = tmp_path / "embeddings"
|
||||
other_dir = tmp_path / "vae"
|
||||
for directory in (checkpoint_dir, unet_dir, embedding_dir, other_dir):
|
||||
directory.mkdir()
|
||||
|
||||
config = _make_config(
|
||||
_configured_root_paths={
|
||||
"checkpoint": [str(checkpoint_dir)],
|
||||
"unet": [str(unet_dir)],
|
||||
"embedding": [str(embedding_dir)],
|
||||
"other": [str(other_dir)],
|
||||
},
|
||||
)
|
||||
config._get_enabled_other_folder_keys = lambda: ["vae"]
|
||||
config._configured_other_paths_for_key = lambda key: [str(other_dir)]
|
||||
|
||||
admitted = config.admit_configured_roots()
|
||||
|
||||
assert set(admitted) == {
|
||||
_normalize(checkpoint_dir),
|
||||
_normalize(unet_dir),
|
||||
_normalize(embedding_dir),
|
||||
_normalize(other_dir),
|
||||
}
|
||||
assert config.base_models_roots == [
|
||||
_normalize(checkpoint_dir),
|
||||
_normalize(unet_dir),
|
||||
]
|
||||
assert config.checkpoints_roots == [_normalize(checkpoint_dir)]
|
||||
assert config.unet_roots == [_normalize(unet_dir)]
|
||||
assert config.embeddings_roots == [_normalize(embedding_dir)]
|
||||
assert config.other_roots == [_normalize(other_dir)]
|
||||
assert config.other_root_subtypes[_normalize(other_dir)] == "vae"
|
||||
assert config.other_folder_roots["vae"] == [_normalize(other_dir)]
|
||||
@@ -213,21 +213,43 @@ class ScopedScanService:
|
||||
|
||||
model_type = "lora"
|
||||
|
||||
def __init__(self, roots=None, summary=None, cancelled=False):
|
||||
def __init__(self, roots=None, summary=None, cancelled=False, unavailable=None):
|
||||
self._roots = list(roots or [])
|
||||
self._unavailable = list(unavailable or [])
|
||||
self._summary = summary
|
||||
self.cancelled = cancelled
|
||||
self.scanner = SimpleNamespace(is_cancelled=lambda: self.cancelled)
|
||||
self.received_scope = "not-called"
|
||||
self.received_rebuild = None
|
||||
self.root_refreshes = 0
|
||||
|
||||
def get_model_roots(self):
|
||||
return list(self._roots)
|
||||
|
||||
def refresh_model_roots(self):
|
||||
"""Mimic the scanner admitting roots that became readable again."""
|
||||
self.root_refreshes += 1
|
||||
return []
|
||||
|
||||
def describe_model_roots(self):
|
||||
return [
|
||||
{"path": root, "label": root.rsplit("/", 1)[-1], "reachable": True, "models": 7}
|
||||
{
|
||||
"path": root,
|
||||
"label": root.rsplit("/", 1)[-1],
|
||||
"reachable": True,
|
||||
"available": True,
|
||||
"models": 7,
|
||||
}
|
||||
for root in self._roots
|
||||
] + [
|
||||
{
|
||||
"path": root,
|
||||
"label": root.rsplit("/", 1)[-1],
|
||||
"reachable": False,
|
||||
"available": False,
|
||||
"models": 3,
|
||||
}
|
||||
for root in self._unavailable
|
||||
]
|
||||
|
||||
async def scan_models(self, force_refresh=False, rebuild_cache=False, scope=None):
|
||||
@@ -265,6 +287,8 @@ async def test_scan_models_accepts_roots_param():
|
||||
assert service.received_scope is not None
|
||||
assert service.received_scope.roots == ("/mnt/a",)
|
||||
assert service.received_rebuild is False
|
||||
# A drive plugged in after startup is admitted before the roots are validated.
|
||||
assert service.root_refreshes == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -307,6 +331,28 @@ async def test_scan_models_rejects_roots_with_full_rebuild():
|
||||
assert service.received_scope == "not-called"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_model_roots_reports_configured_but_unavailable_roots():
|
||||
"""Wave 6: a root whose directory was missing at startup is still reported."""
|
||||
service = ScopedScanService(
|
||||
roots=["/mnt/a"], unavailable=["/mnt/drive-Z/loras"]
|
||||
)
|
||||
handler = ModelQueryHandler(service=service, logger=logging.getLogger(__name__))
|
||||
|
||||
response = await handler.get_model_roots(SimpleNamespace(query=QueryParams({})))
|
||||
payload = json.loads(response.text)
|
||||
|
||||
assert response.status == 200
|
||||
assert payload["roots"] == ["/mnt/a"]
|
||||
details = {entry["path"]: entry for entry in payload["root_details"]}
|
||||
assert details["/mnt/a"]["available"] is True
|
||||
assert details["/mnt/drive-Z/loras"]["available"] is False
|
||||
assert details["/mnt/drive-Z/loras"]["reachable"] is False
|
||||
assert details["/mnt/drive-Z/loras"]["models"] == 3
|
||||
# Describing the roots admits anything that became readable again.
|
||||
assert service.root_refreshes == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_scan_models_accepts_folder_param():
|
||||
service = ScopedScanService(roots=["/mnt/a", "/mnt/b"], summary=SUMMARY)
|
||||
|
||||
@@ -2649,6 +2649,112 @@ async def test_reconcile_folder_scope_covers_every_root_holding_it(tmp_path: Pat
|
||||
assert _normalize_path(primary / "only-primary" / "c.txt") in paths
|
||||
|
||||
|
||||
async def _startup_filtered_scanner(tmp_path: Path, monkeypatch, configured_missing: Path):
|
||||
"""Scanner whose live root list lacks a root that the config still holds.
|
||||
|
||||
Models the startup case: the directory did not exist when the live list was
|
||||
built (drive switched off), so `Config` filtered it out while its cached
|
||||
entries are still there.
|
||||
"""
|
||||
primary = tmp_path / "primary"
|
||||
(primary / "pack").mkdir(parents=True)
|
||||
(primary / "pack" / "a.txt").write_text("a", encoding="utf-8")
|
||||
configured_missing.mkdir(parents=True)
|
||||
(configured_missing / "pack").mkdir()
|
||||
(configured_missing / "pack" / "b.txt").write_text("b", encoding="utf-8")
|
||||
|
||||
scanner = MultiRootDummyScanner([primary, configured_missing])
|
||||
await scanner._initialize_cache()
|
||||
# The drive goes away before the process starts, as far as the live list is
|
||||
# concerned: keep the roots the filter would have kept.
|
||||
shutil.rmtree(configured_missing)
|
||||
scanner._roots = [str(primary)]
|
||||
monkeypatch.setattr(
|
||||
scanner,
|
||||
"get_configured_model_roots",
|
||||
lambda: [str(primary), str(configured_missing)],
|
||||
)
|
||||
return scanner, primary, configured_missing
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_describe_model_roots_reports_unavailable_configured_root(
|
||||
tmp_path: Path, monkeypatch
|
||||
):
|
||||
configured_missing = tmp_path / "drive-Z" / "loras"
|
||||
scanner, primary, missing = await _startup_filtered_scanner(
|
||||
tmp_path, monkeypatch, configured_missing
|
||||
)
|
||||
|
||||
described = scanner.describe_model_roots()
|
||||
|
||||
assert [entry["path"] for entry in described] == [
|
||||
_normalize_path(primary),
|
||||
_normalize_path(missing),
|
||||
]
|
||||
available, unavailable = described
|
||||
assert available["reachable"] is True
|
||||
assert available["available"] is True
|
||||
# The missing root is reported with its cached count so the refresh menu can
|
||||
# offer it as an offline row instead of hiding it until a restart.
|
||||
assert unavailable["reachable"] is False
|
||||
assert unavailable["available"] is False
|
||||
assert unavailable["models"] == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reconcile_reports_root_filtered_out_at_startup(tmp_path: Path, monkeypatch):
|
||||
configured_missing = tmp_path / "drive-Z" / "loras"
|
||||
scanner, _primary, missing = await _startup_filtered_scanner(
|
||||
tmp_path, monkeypatch, configured_missing
|
||||
)
|
||||
kept_path = _normalize_path(missing / "pack" / "b.txt")
|
||||
|
||||
summary = await scanner._reconcile_cache()
|
||||
|
||||
assert summary is not None
|
||||
# Nothing is added or removed...
|
||||
assert summary["added"] == 0
|
||||
assert summary["removed"] == 0
|
||||
# ...but the switched-off root is no longer silent: it is reported with the
|
||||
# entries that were kept for it.
|
||||
assert summary["skipped_roots"] == [
|
||||
{
|
||||
"path": _normalize_path(missing),
|
||||
"label": missing.name,
|
||||
"reason": "root_unavailable",
|
||||
}
|
||||
]
|
||||
assert summary["kept_unreachable"] == 1
|
||||
assert summary["unavailable_paths"] == [
|
||||
{"path": _normalize_path(missing), "reason": "root_unavailable", "kept": 1}
|
||||
]
|
||||
assert kept_path in {item["file_path"] for item in scanner._cache.raw_data}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_scoped_scan_stays_quiet_about_filtered_out_roots(tmp_path: Path, monkeypatch):
|
||||
configured_missing = tmp_path / "drive-Z" / "loras"
|
||||
scanner, primary, missing = await _startup_filtered_scanner(
|
||||
tmp_path, monkeypatch, configured_missing
|
||||
)
|
||||
|
||||
summary = await scanner._reconcile_cache(
|
||||
scope=ReconcileScope(roots=(str(primary),))
|
||||
)
|
||||
|
||||
assert summary is not None
|
||||
# The caller asked for one root: a root it did not ask about is not its
|
||||
# business, but its entries are still kept.
|
||||
assert summary["scanned_roots"] == [primary.name]
|
||||
assert "root_unavailable" not in {
|
||||
entry["reason"] for entry in summary["skipped_roots"]
|
||||
}
|
||||
assert _normalize_path(missing / "pack" / "b.txt") in {
|
||||
item["file_path"] for item in scanner._cache.raw_data
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reconcile_keeps_entries_under_unreadable_dir(tmp_path: Path, monkeypatch):
|
||||
"""A directory os.walk cannot enter (permissions, I/O error, offline
|
||||
|
||||
Reference in New Issue
Block a user