fix(scanner): compare model roots by identity, not by spelling (#1146)

ComfyUI hands its model folders over in the host's own spelling --
`utils/extra_config.py` runs every configured path through `os.path.normpath()`,
which is all backslashes on Windows, and `main.py` registers `<output>/loras`
unconditionally for its save nodes. `Config` kept that spelling verbatim while
`_prepare_*_paths()` normalized the live root lists to forward slashes, and the
scanner compared the two as raw strings. On Windows every configured root
therefore looked like a root that was not in the live list: the Refresh menu
listed each directory twice (the twin showing "0 models"), the completion toast
claimed reachable roots were "currently inaccessible" and kept their whole cache
count as `kept_unreachable`, and the settings sync persisted host-registered
directories that were never created.

The duplication was not only cosmetic. `_UnreachablePaths.add()` normalizes the
prefix it is given, so the configured twin of a live root registered exactly the
live root's prefix as unreadable: in the prune phase every cached-but-missing
entry under it matched and was "kept", so models deleted from disk were never
removed from the cache again.

* `py/utils/root_paths.py` holds the single `normalize_root_identity()`
  (separator style, redundant/trailing separators, and case where the filesystem
  folds it: whole paths on Windows, drive letters everywhere). `config.py` and
  `settings_manager.py` each carried a private copy and `model_scanner.py`
  compared raw strings.
* `ModelScanner` compares root identities in `get_configured_model_roots()`,
  `describe_model_roots()` (dedupe + `available`), the reconcile's
  `configured_only` list and the `ReconcileScope.roots` membership test, so one
  directory is one root whatever the host called it. `_normalized_root_prefix()`
  is built on the same identity, and an empty prefix no longer matches every
  path.
* A configured root that never made it into the live list is reported only while
  it still holds cached entries -- in the menu and in `skipped_roots` /
  `unavailable_paths` alike. A directory that neither exists nor was ever scanned
  has nothing to keep, so `<output>/loras` stops being an unavailable root.
* The startup sync drops saved roots that neither exist nor hold cached entries
  (`PersistentModelCache.roots_with_cached_entries()`, one scan per model type,
  separator-agnostic and wildcard-escaped). A root named by the library's
  `default_*_root` is kept even when empty: dropping it would let the next boot
  "repair" the default onto another root. When the cache cannot be consulted,
  every root is kept.
* `GET /scan?roots=` validates requested roots by identity too, so the browser
  extension or a script passing the spelling settings.json holds selects the root
  instead of answering 400 "Unknown model root(s)".

Verified: 8 of the new tests fail against the pre-fix source (checked with the
fix stashed); 3798 passed, 7 skipped, 11 deselected. The probe costs 71ms for a
100k-model library with seven missing roots (357ms when probing root by root).
This commit is contained in:
Will Miao
2026-10-12 08:43:06 +08:00
parent e056142669
commit 6d0d7cb15b
11 changed files with 822 additions and 57 deletions
+73 -1
View File
@@ -4,16 +4,33 @@ import os
import sqlite3
import threading
from dataclasses import dataclass, field
from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple
from typing import Any, Dict, List, Mapping, Optional, Sequence, Set, Tuple
from ..utils.cache_db import connect_cache_db
from ..utils.cache_paths import CacheType, resolve_cache_path_with_migration
from ..utils.file_lock import exclusive_lock
from ..utils.root_paths import normalize_root_identity
from .model_sources import normalize_metadata_source
logger = logging.getLogger(__name__)
def _like_prefix_pattern(identity: str) -> str:
"""SQLite LIKE pattern matching every path inside *identity*.
``%``/``_`` are legal characters in a path and ``\\`` is the escape character,
so all three are escaped. The separator is added only when the identity does not
already end in one, so a filesystem root (``/``) still matches everything below
it instead of only itself.
"""
escaped = (
identity.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
)
prefix = escaped if escaped.endswith("/") else f"{escaped}/"
return f"{prefix}%"
@dataclass
class PersistedCacheData:
"""Lightweight structure returned by the persistent cache."""
@@ -254,6 +271,61 @@ class PersistentModelCache:
all_folders=all_folders,
)
def roots_with_cached_entries(
self, model_type: str, roots: Sequence[str]
) -> Set[str]:
"""Return the subset of *roots* that hold at least one cached entry.
Used by the settings sync to tell a root that was registered but never
scanned -- ComfyUI registers ``<output>/loras`` for its save nodes whether
or not that directory exists -- from a real root whose drive happens to be
switched off: only the latter must keep its place in the saved library
paths (issue #1146). Paths are matched on their forward-slash identity, so
the host's own slash style does not matter.
Raises when the answer is unknown (a disabled cache, a missing database, an
unreadable file): the caller keeps every root in that case.
"""
if not self.is_enabled():
raise RuntimeError("persistent model cache is disabled")
candidates: Dict[str, str] = {}
for root in roots:
identity = normalize_root_identity(root)
if identity:
candidates.setdefault(identity, root)
if not candidates:
return set()
matched: Set[str] = set()
# One scan per model type rather than one per root: the probe runs at
# startup for every configured root that is missing right now, which on
# Windows includes all of ComfyUI's `<output>/<type>` folders.
clause = "replace(file_path, '\\', '/') LIKE ? ESCAPE '\\'"
sql = (
"SELECT DISTINCT replace(file_path, '\\', '/') FROM models "
f"WHERE model_type = ? AND ({' OR '.join([clause] * len(candidates))})"
)
params: List[Any] = [model_type]
params.extend(_like_prefix_pattern(identity) for identity in candidates)
with self._db_lock:
conn = self._connect(readonly=True)
try:
rows = conn.execute(sql, params).fetchall()
finally:
conn.close()
for (cached_path,) in rows:
folded = normalize_root_identity(cached_path)
for identity, root in candidates.items():
prefix = identity if identity.endswith("/") else f"{identity}/"
if folded.startswith(prefix):
matched.add(root)
break
return matched
def save_cache(self, model_type: str, raw_data: Sequence[Dict[str, Any]], hash_index: Dict[str, List[str]], excluded_models: Sequence[str], autov3_hash_index: Optional[Dict[str, List[str]]] = None, all_folders: Optional[Sequence[str]] = None) -> None:
if not self.is_enabled():
return