fix(scanner): serve folder tree from scan-recorded, persisted directory list (#1110)

The include_empty folder tree (download/move modals) walked every model
root synchronously on the event loop via get_all_folders(). On network
(NAS) roots this froze the whole server for the duration of the walk —
blocking WebSocket progress, aria2 RPC and the download queue — and the
5s TTL re-triggered the walk on nearly every modal interaction.

The scanners already visit every directory during cache scans, so record
the full directory list (including empty folders) there instead:

- _gather_model_data/_reconcile_cache collect directories during the
  existing walks; reconcile refreshes and persists the list even when no
  model files changed.
- ModelCache gains an all_folders field (None = never recorded).
- PersistentModelCache stores the list in a new folders table, with a
  cache_meta flag distinguishing 'recorded empty' from legacy snapshots.
- get_all_folders() is now a pure in-memory read. A legacy snapshot
  triggers a one-shot backfill walk in a worker thread (never on the
  event loop) that records and persists the list.
- Moves add the destination folder (and parents) incrementally instead
  of invalidating a TTL cache.
This commit is contained in:
Will Miao
2026-09-11 23:03:24 +08:00
parent 91b2735dad
commit 6d3f82976f
5 changed files with 278 additions and 73 deletions
+5
View File
@@ -33,6 +33,11 @@ class ModelCache:
raw_data: List[Dict[str, Any]]
folders: List[str]
# Every directory under the model roots (including empty ones), as
# recorded by the last scan/hydration. ``None`` means "never recorded"
# (e.g. a persisted snapshot predating this field) and triggers a
# background filesystem backfill in the scanner.
all_folders: Optional[List[str]] = None
version_index: Dict[int, Dict[str, Any]] = field(default_factory=dict)
model_id_index: Dict[int, List[Dict[str, Any]]] = field(default_factory=dict)
# Multi-valued companion to version_index: every local file entry of a