perf(rename): make bulk filename-template apply O(n) instead of O(n^2)

Applying a filename template to a large library re-did O(library) work for
every renamed file: a full natsort resort plus whole-table SQLite rewrite and
download-history resync after each rename, and a full scan plus resort of the
entire recipe collection per renamed LoRA. On a 20k-model library with 300k
recipes on a HDD this pushed "Apply to Library" into multi-day runs.

- ModelScanner.defer_cache_persist(): bulk loops update the in-memory entry
  and indexes only; resort + persist + download-history sync run once at
  context exit, forced even on cancellation/error since files are already
  renamed on disk. Single-rename callers keep immediate per-call behavior.
- RecipeScanner.build_lora_hash_index(): one-shot hash -> recipes index so
  per-file lookups are O(1); update_lora_filename_by_hash gains hash_index /
  defer_maintenance params, with a single finalize_bulk_filename_updates()
  resort at the end of a bulk session.
- ModelLifecycleService.bulk_rename_session() / BulkRenameContext wire the
  deferred path through rename_model (hash index built lazily on first
  recipe-touching rename).
- Blocking os.rename sequence offloaded via asyncio.to_thread so one file's
  HDD I/O no longer stalls the event loop (no cross-file parallelism).
- Skip logic, per-batch WebSocket progress, cancellation, and result
  counters unchanged.
This commit is contained in:
Will Miao
2026-10-03 14:31:10 +08:00
parent 515469054c
commit 896ce5eddb
8 changed files with 1080 additions and 63 deletions
@@ -33,7 +33,9 @@ class FilenameTemplateUseCase:
An empty template restores the recorded original filename instead of
rendering a template. Shares the auto-organize lock (and its in-progress
error) so a bulk rename never runs concurrently with an auto-organize
operation.
operation. The whole loop runs inside a bulk rename session so cache
persist/resort and recipe maintenance happen once at the end instead of
per renamed file.
"""
def __init__(
@@ -106,23 +108,24 @@ class FilenameTemplateUseCase:
await self._emit_progress(progress_callback, result, "started")
for index in range(0, result.total, AUTO_ORGANIZE_BATCH_SIZE):
if self._scanner.is_cancelled():
logger.info(
"Filename template apply cancelled for %s", self._model_type
)
break
batch = models[index : index + AUTO_ORGANIZE_BATCH_SIZE]
for model in batch:
async with self._lifecycle_service.bulk_rename_session() as bulk_context:
for index in range(0, result.total, AUTO_ORGANIZE_BATCH_SIZE):
if self._scanner.is_cancelled():
logger.info(
"Filename template apply cancelled for %s", self._model_type
)
break
await self._process_model(model, template, result)
result.processed += 1
await self._emit_progress(progress_callback, result, "processing")
# Yield between batches so the server stays responsive.
await asyncio.sleep(0.1)
batch = models[index : index + AUTO_ORGANIZE_BATCH_SIZE]
for model in batch:
if self._scanner.is_cancelled():
break
await self._process_model(model, template, result, bulk_context)
result.processed += 1
await self._emit_progress(progress_callback, result, "processing")
# Yield between batches so the server stays responsive.
await asyncio.sleep(0.1)
if self._scanner.is_cancelled():
result.status = "cancelled"
@@ -150,6 +153,7 @@ class FilenameTemplateUseCase:
model: Dict[str, Any],
template: str,
result: AutoOrganizeResult,
bulk_context: Any = None,
) -> None:
model_name = model.get("model_name", "Unknown")
try:
@@ -177,7 +181,7 @@ class FilenameTemplateUseCase:
return
await self._lifecycle_service.rename_model(
file_path=file_path, new_file_name=new_stem
file_path=file_path, new_file_name=new_stem, bulk_context=bulk_context
)
result.success_count += 1