# undo-delete-staging - Work Plan

## TL;DR (For humans)

**What you'll get:** Deleting a model or recipe becomes recoverable: the file disappears right away, but a toast with an "Undo" button (30-second countdown) lets you bring it back — the file is only physically erased after the window closes. You can also turn this off in settings to delete instantly as before.

**Why this approach:** Files are moved (models) or copied (recipes) to a hidden staging folder on the same drive instead of being deleted — a rename is instant even for multi-gigabyte files, so freeing disk space still happens within 30 seconds. Deleting then looks and feels the same as today; only the accident-recovery net is added.

**What it will NOT do:** No typing-to-confirm, no system recycle bin, no "trash folder" page to manage, no changes to how anything else works (exclude, downloads, metadata).

**Effort:** Medium
**Risk:** Medium - the riskiest parts are Windows file-locking during the staged purge and keeping the library cache consistent across restore; both have explicit fallback and test coverage.
**Decisions to sanity-check:** (1) 30-second undo window — long enough to catch a misclick, short enough that disk space is barely delayed; (2) undo for bulk/duplicate deletes restores everything with one click, not per-file; (3) if moving files to staging fails for any reason, the app silently falls back to the old permanent delete rather than blocking you.

Your next move: approve, or run a high-accuracy review first.

---

> TL;DR (machine): Medium effort/risk; 12 todos in 5 waves + final verification; same-volume rename staging [updated 2026-08: now guaranteed by SIBLING staging inside the model's own folder — see addendum at end] + 30s undo toast + triple-trigger purge + skip-toggle; no new deps, no DB changes.

## Scope
### Must have
- Backend staging service: model files renamed to same-volume per-root staging dir (`.lm-pending-delete/` under each model root) [updated 2026-08: staging moved to a SIBLING dir inside the deleted model's own folder — see addendum at end]; recipe JSON + preview image copied to global staging (`{settings_dir}/.lm-pending-delete/`). Manifest JSON (batch_id, kind, expires_at, entries[staged->original], model cached_entry snapshot) is the only state.
- Delete responses (single model, single recipe, bulk models, bulk recipes — hence duplicates flows) gain an optional `batch_id` field when staged.
- `POST /api/lm/undo-delete` endpoint: restores staged files to original paths + restores cache entries exactly (model: raw_data append + resort + bump_cache_version + _persist_current_cache + hash_index.add_entry; recipe: recipe_scanner.add_recipe) + `_broadcast_models_changed()`. Expired batch -> 404 `{"success": false, "error": ...}`.
- Purge triple-trigger: per-batch asyncio timer task (30s TTL), on_startup sweep inside `_initialize_services` (covers both ComfyUI + standalone since StandaloneLoraManager reuses it), opportunistic purge at each stage/undo call. Purge only batches with `expires_at < now`.
- Settings toggle `delete_undo_enabled` (default `true`) in DEFAULT_SETTINGS + checkbox in settings_modal.html; when disabled -> current hard-delete behavior, no batch_id, no undo toast.
- Staging failure (rename/copy OSError) -> fall back to existing hard delete (delete_model_artifacts / os.remove path), no batch_id.
- Frontend: new `showActionToast` (showToast signature untouched; extract shared createToastElement/appendToast internals) with action button + 30s countdown text; all 5 delete flows consume batch_id and show Undo toast; shared `handleUndoDelete(batchId, refreshFn)`; full list refresh after undo (recipes: `window.recipeManager.loadRecipes(true)`; models: `resetAndReload(true)` from modelApiFactory).
- C-friction (NO type-to-confirm): delete-confirm buttons delay-activate 1.5s after modal opens; initial focus on Cancel; model delete modal gains "will be permanently deleted from disk" warning + freed-size display (card.dataset.file_size + formatFileSize).
- i18n keys in locales/en.json + `python scripts/sync_translation_keys.py` run.
- Backend pytest + frontend vitest coverage for all of the above.

### Must NOT have (guardrails, anti-slop, scope boundaries)
- NO type-to-confirm / hold-to-confirm friction anywhere (user vetoed).
- NO OS trash integration (send2trash) — no new dependencies at all.
- NO persistent recycle-bin UI (no trash browsing page, no trash listing endpoint).
- NO changes to exclude/unexclude flows, download flows, metadata editing, or recipe-JSON rewriting on model undo (recipe refs are hash-based and re-resolve on restore).
- NO DB migrations, NO schema changes to existing SQLite caches.
- NO `os.path.realpath` for staging/undo routing — business paths only (AGENTS.md rule); realpath only where the scanner itself already uses it.
- NO changes to `showToast` signature or its existing call sites' behavior (no regressions in the 48 files using it).
- NO silent semantics: when undo is skipped (setting off or staging failure), the delete must behave exactly as today — no partial staging, no orphaned files.
- NO persistent undo affordance across browser refresh (the toast is ephemeral; a page reload within the window loses the Undo button — the staged batch still expires after 30s).
- NO per-file undo within a bulk/duplicates action (one batch = one undo for the whole action).
- NO partial-batch purge: purge and undo operate on whole batches only; a batch is never half-restored (except the mid-undo-failure retry case defined in todo 1, where per-entry restored flags make the remainder retry-able).
- NO multi-instance support: a single server instance is assumed (two servers sharing the same model roots/settings dir is out of scope; cross-process coordination is not implemented).
- NO deletion of manifest-less or corrupted staging batches — quarantine only (`<batch_id>.orphaned`), never rmtree past per-file errors.
- NO ghost library entries: the model AND checkpoint scanners must never index files under `.lm-pending-delete/` (exclusion implemented in todo 1 for model_scanner.py walk sites + checkpoint_scanner._find_pending_models_from_filesystem).
- NO merge data-loss: merge_batches moves (never drops) staged files; loser batch dirs are removed only when empty; merge failure aborts with all batches intact (falls back to a `batch_ids` array in the response).
- UNDO-BLIND DELETE FLOWS (documented scope edges, unchanged behavior): the versions-tab delete (ModelVersionsTab.js:1136-1144) is staged server-side but shows NO undo toast (own success toast, out of scope); the delete-by-version endpoint (py/routes/handlers/misc_handlers.py:2456) hard-deletes via delete_model_artifacts bypassing staging entirely (pre-existing behavior, unchanged).

## Verification strategy
> Zero human intervention - all verification is agent-executed.
- Test decision: TDD for backend (pytest — service + endpoint tests written with the implementation); tests-after for frontend flows (vitest) where flows are integration-heavy.
- Commands: `pytest tests/services/test_pending_delete_service.py tests/services/test_model_lifecycle_service.py tests/services/test_recipe_persistence.py tests/routes/test_pending_delete_routes.py -q`; `npm test`; `python scripts/sync_translation_keys.py --dry-run` clean.
- Evidence: .omo/evidence/undo-delete-staging/task-<N>-undo-delete-staging.<ext> per todo (attemptDir = currentAttemptDir from 'omo ulw-loop status --json'; outside ulw-loop use .omo/evidence/)

## Execution strategy
### Parallel execution waves
- Wave 1: todo 1 (staging service core + tests) — everything depends on it.
- Wave 2: todos 2, 3, 4 in parallel (model single / model bulk / recipe single+bulk wiring — all depend only on 1; different files: model_lifecycle_service.py / model_scanner.py / persistence_service.py).
- Wave 3: todos 5, 7, 10 in parallel, then 6 AFTER 5 (todo 5 and todo 6 both edit py/lora_manager.py — same-file edits must be serialized: 5 first for route registration, 6 second for the startup sweep hook; 6 additionally depends on 1).
- Wave 4: todo 8, then 9 AFTER 8 (todo 8 and todo 9 both edit static/js/api/baseModelApi.js — same-file edits must be serialized: 8 (deleteModel) first, 9 (bulkDeleteModels) second; 9 additionally depends on 3,4,5,7,10).
- Wave 5: todo 11, then 12 AFTER 11 (todo 11 and todo 12 both edit locales/en.json — 12's final sync step must run after 11's strings exist; 11 depends on 8, 9 because it edits the SAME frontend files: modalUtils.js, RecipeCard.js, BulkManager.js, DuplicatesManager.js, ModelDuplicatesManager.js — so 11 runs after Wave 4 completes; 12 additionally depends on 8,9,10,11 strings).
- Wave 6: Final verification wave (F1-F4).

### Dependency matrix
| Todo | Depends on | Blocks | Can parallelize with |
| --- | --- | --- | --- |
| 1. PendingDeleteService + tests | — | 2,3,4,5,6,7 | — |
| 2. Single model delete staging + tests | 1 | 5, 8 | 3, 4 |
| 3. Bulk model delete staging + tests | 1 | 5, 9 | 2, 4 |
| 4. Recipe delete staging (single+bulk) + tests | 1 | 5, 8, 9 | 2, 3 |
| 5. Undo endpoint + cache restore + tests | 1, 2, 3, 4 | 6, 8, 9 | 7, 10 |
| 6. Purge scheduling (timer+startup+opportunistic) + tests | 1, 5 (same file lora_manager.py) | — | 7, 10 |
| 7. Settings toggle + tests | 1 | 9 | 5, 6, 10 |
| 8. Frontend single-delete undo flows + tests | 2, 4, 5, 10 | 9, 11, 12 | 7 |
| 9. Frontend bulk+duplicates undo flows + tests | 3, 4, 5, 7, 8 (same file baseModelApi.js), 10 | 11, 12 | — |
| 10. showActionToast + CSS + tests | — | 8, 9 | 1..7 |
| 11. C-friction modal changes + tests | 8, 9 (same files: modalUtils.js, RecipeCard.js, BulkManager.js, DuplicatesManager.js, ModelDuplicatesManager.js) | 12 | — |
| 12. i18n keys + sync + tests | 8, 9, 10, 11 (12's sync step runs AFTER 11) | — | — |

## Todos
> Implementation + Test = ONE todo. Never separate.
<!-- APPEND TASK BATCHES BELOW THIS LINE WITH edit/apply_patch - never rewrite the headers above. -->
- [x] 1. Create py/services/pending_delete_service.py — staging service (stage/undo/purge/manifest) + TTL constant + settings default + unit tests
  What to do: New module `py/services/pending_delete_service.py`. Class `PendingDeleteService` following the house singleton+asyncio.Lock pattern (see py/services/model_scanner.py:40-63). Service-level `asyncio.Lock` serializing stage/merge/undo/purge_batch operations (prevents purge-timer vs undo races AND merge-vs-purge races). LOCK HIERARCHY (critical — asyncio.Lock is NOT re-entrant, a deadlock here breaks first use): the lock is acquired ONLY by stage_model_delete, stage_recipe_delete, merge_batches, undo, and purge_batch; `purge_expired()` itself NEVER acquires the lock — it enumerates staging dirs and delegates each batch to purge_batch (which locks); therefore the opportunistic `await self.purge_expired()` at the start of stage_*/undo MUST be called BEFORE those methods acquire the lock (lock-free section), never while holding it. Public async methods: `stage_model_delete(*, scanner, target_dir, file_name, main_extension, original_file_path, cached_entry) -> str | None` (returns batch_id or None when undo disabled / staging impossible); `stage_recipe_delete(*, recipe_json_path, image_path, recipe_data) -> str | None`; `merge_batches(batch_ids) -> str | None` — CRITICAL SEMANTICS (never drops files, never orphans from purge): merges several batches into one manifest, RE-ANCHORS `expires_at = now + PENDING_DELETE_TTL_SECONDS` at merge time, winner = first batch_id; the staged FILES of the losing batches are MOVED into the winner's batch dir (os.rename per file) and each entry's `staged` path is REWRITTEN in the merged manifest BEFORE any loser batch dir is removed; loser batch dirs are removed ONLY after they are empty; the merged manifest is written atomically (temp + os.replace); after a successful merge, ARM A FRESH PURGE TIMER for the winner with the RE-ANCHORED expiry (the winner's original timer fires at the old expiry, re-reads the later re-anchored expires_at, no-ops — WITHOUT a fresh timer the merged batch would never be purged on an idle server; the fresh timer guarantees the "30s purge" contract holds for merged batches); on PARTIAL-MOVE FAILURE (some files moved, then a move fails): MOVE the already-moved files BACK to their original batch dirs and RESTORE the per-batch manifests (revert staged paths), then return None — all original batch dirs and files intact, each constituent batch fully undoable (callers fall back to the `batch_ids` array contract: todos 3/4); `undo(batch_id) -> dict` (renames/copies files back per-entry, removes batch dir + manifest when fully restored); `purge_expired() -> int` — lock-free (see LOCK HIERARCHY); MUST enumerate ALL model roots across ALL scanner types (lora/checkpoint/embedding — iterate the per-type scanners like the route registrars do, collecting every `get_model_roots()`) PLUS the global recipe staging dir, or cross-type batches would leak; SKIPS dirs whose name ends with `.orphaned` (never re-quarantined, never re-renamed, never deleted — the quarantine is terminal); delegates each found batch to purge_batch; `purge_batch(batch_id)` — acquires the lock; treats MISSING staged files (already-restored / partially-restored batches) as already-purged: FileNotFoundError per file is a silent no-op, NOT a failure (the batch dir is removed when no staged files remain); re-reads the manifest's current expires_at and no-ops when the batch is missing / already restored / not yet expired (stale timers from merged-away or undone batches are harmless). Constants: `PENDING_DELETE_TTL_SECONDS = 30`, `PENDING_DELETE_DIR_NAME = ".lm-pending-delete"`. Staging dir computation: for models, root = `scanner._find_root_for_file(original_file_path)` (py/services/model_scanner.py:1108-1124) then `<root>/.lm-pending-delete/<batch_id>/`; for recipes, `<settings_dir>/.lm-pending-delete/<batch_id>/` (settings dir via get_settings_manager() — see py/services/settings_manager.py:2215). Manifest `manifest.json` per batch dir — write atomically via temp file + os.replace: `{"batch_id", "kind": "model"|"recipe", "model_type": <"loras"|"checkpoints"|"embeddings" — ONLY for kind=model; captured from scanner.model_type at stage time>, "state": "staged", "expires_at": int(epoch+30), "entries": [{"staged": <abs path>, "original": <abs business path>, "restored": false}], "model_snapshot": <cached_entry dict or null — kind=model>, "recipe_snapshot": <recipe_data dict or null — kind=recipe>}`. The per-batch staging dir guarantees NO staged-name collisions (same filename in concurrent batches lives in different batch dirs). Model staging = os.rename per artifact (same volume guaranteed because staging dir is under the containing root) [CORRECTED 2026-08: this guarantee held only for plain directories — a nested symlinked subdir could resolve to another volume and break the rename with EXDEV; the symlink fix moved staging to `<model_dir>/.lm-pending-delete/<batch_id>/`, a SIBLING of the model artifacts inside the model's own folder, so stage/undo renames are same-device BY CONSTRUCTION — see addendum at end]; SKIP artifacts that do not exist (missing metadata sidecar / preview must not trip rollback — mirror delete_model_artifacts' tolerance, enumerate then filter by exists); recipe staging = shutil.copy2 of the JSON and — ONLY IF IT EXISTS — the image (missing/shared previews are skipped; manifest entries reflect actual files); then the caller removes originals. Enumerate model artifacts EXACTLY like delete_model_artifacts (py/services/model_lifecycle_service.py:19-48): main file + `{file_name}.metadata.json` + PREVIEW_EXTENSIONS (py/utils/constants.py:22-37). Use `os.path.abspath`, never realpath, for all stored paths (AGENTS.md business-path rule — a symlinked root must not silently route staging to another volume). NOTE: merging batches from DIFFERENT roots on DIFFERENT volumes will fail os.rename with EXDEV — that is EXPECTED and fine: the merge aborts, callers fall back to the `batch_ids` array, and the frontend undoes sequentially; this fallback is the NORMAL path for cross-volume bulks, not a rare edge. [annotated 2026-08: with sibling staging, single-file stage/undo renames can no longer hit EXDEV at all; EXDEV remains possible ONLY for cross-volume MERGES (moving loser files into the winner's batch dir) — see addendum at end.] Guard: skip staging entirely when `get_settings_manager().get("delete_undo_enabled", True)` is falsy (return None). ALSO add `"delete_undo_enabled": True` to DEFAULT_SETTINGS in py/services/settings_manager.py:57-119 as part of this todo (this makes the toggle exist before any wiring todo reads it). On any OSError during staging: log warning, roll back any already-staged files of that batch (rename back), return None (caller falls back to hard delete). SCANNER EXCLUSION (critical — without it staged files appear in the library as ghost model entries): every directory walk under model roots MUST skip directories whose name equals `.lm-pending-delete`: (a) the walk sites in py/services/model_scanner.py (~:706 count_recursive, ~:867 os.walk, ~:1404 scan_recursive, plus `_process_model_file` at :1128), (b) py/services/checkpoint_scanner.py `_find_pending_models_from_filesystem` (~:331 — checkpoint metadata.json discovery), and (c) py/utils/usage_stats.py `_find_checkpoint_file_on_disk` (~:424 — walks checkpoint roots for usage-tracking lookups; staged files must not be matched there either; cosmetic but keeps the predicate uniform). Add ONE shared predicate (e.g. `_is_excluded_dir(name)`) used by all sites. `undo()` must: hold the service lock; reject expired batches (ValueError "Undo window expired"); pre-check ALL target paths EXCEPT entries already marked `restored: true` (skip those) — if any other original path exists on disk (occupied — e.g. user re-downloaded), refuse with ValueError "Target path occupied" and leave the whole batch intact (protects new files; batch stays staged until TTL); restore files PER ENTRY and write the manifest through after each successful entry restore (`restored: true`) so a mid-undo failure (locked file, permissions) leaves a retry-able state — a subsequent undo skips entries whose staged file is already gone (already restored) and finishes the rest; remove the batch dir + manifest only after ALL entries restored; after success mark manifest state "restored". `purge_expired()` scans: every model root from ALL scanner types' `get_model_roots()` (py/services/model_scanner.py:1073, impls lora_scanner.py:31-45, checkpoint_scanner.py:428-441, embedding_scanner.py:24-36) for `.lm-pending-delete/` dirs, plus the global recipe staging dir; deletes entries with expires_at < now (os.remove + os.rmdir); per-file purge failure (locked file, e.g. Windows/antivirus): skip THAT file and leave the batch dir intact — NEVER delete a batch dir or rmtree past per-file errors (the batch is retried by the next opportunistic purge; failure direction is disk-leak, not data-loss); TOLERATES malformed manifests (quarantine: rename dir to `<batch_id>.orphaned` and skip, do not crash the sweep); MANIFEST-LESS batch dirs (crash between rename and manifest write): treat exactly like corrupted manifests — rename to `<batch_id>.orphaned`, NEVER delete the staged files (they may be the only copy of the user's data). Purge tasks must re-read the manifest's current `expires_at` at fire time (not a precomputed sleep value) and no-op when the batch is missing / already restored / not yet expired — this makes stale timers from merged-away or undone batches harmless. Must NOT do: no DB writes; no new dependencies; no realpath; no deletion of non-expired batches; no websocket broadcasts (caller's job); do NOT touch the recipe FTS index or recipe cache here (undo does that via recipe_scanner.add_recipe in todo 5); do NOT delete manifest-less or corrupted batches — quarantine only; merge_batches must NEVER drop or delete a staged file (any move failure aborts the whole merge) and MUST arm the fresh purge timer after a successful merge.
  Parallelization: Wave 1 | Blocked by: — | Blocks: 2,3,4,5,6,7
  References: py/services/model_lifecycle_service.py:19-48 (artifact patterns), :101-154 (delete flow), py/utils/constants.py:22-37 (PREVIEW_EXTENSIONS), py/services/model_scanner.py:40-63 (singleton pattern), :1073-1075 (get_model_roots base), :1108-1124 (_find_root_for_file), :706/:867/:1404 + _process_model_file at :1128 (walk sites needing staging-dir exclusion), py/services/checkpoint_scanner.py:323-350 (_find_pending_models_from_filesystem — 4th exclusion site, os.walk body ~:331), py/services/settings_manager.py:57-119 (DEFAULT_SETTINGS), :1390-1392 (get), :2215-2228 (get_settings_manager), tests/services/test_model_lifecycle_service.py (inline tmp_path test style), tests/conftest.py:134-212 (MockScanner/MockCache/MockHashIndex), :336-368 (singleton reset)
  Acceptance criteria (agent-executable): `pytest tests/services/test_pending_delete_service.py -q` passes with tests covering: (a) stage_model_delete renames all EXISTING artifact patterns into `<root>/.lm-pending-delete/<batch_id>/` and writes manifest (incl. model_type + model_snapshot) with expires_at = now+30; (b) undo() restores all files to original paths and removes batch dir; (c) undo() on expired batch raises ValueError; (d) undo() when original path occupied raises ValueError and leaves batch dir + manifest intact; (e) PARTIAL-UNDO RETRY: monkeypatch os.rename to fail on the 2nd of 3 entries -> undo raises, manifest shows entry 1 restored:true / entries 2-3 restored:false; second undo (no monkeypatch) completes the rest and removes the dir; (f) purge_expired() removes only expired batches (manually rewrite expires_at in manifest to test); (g) MANIFEST-LESS dir: create batch dir with staged files but no manifest -> sweep renames to `<batch_id>.orphaned`, files still present, sweep completes; (h) CORRUPTED manifest: garbage manifest.json -> dir quarantined, no crash; (i) PURGE LOCKED FILE: monkeypatch os.remove to fail for one file -> sweep skips that file, batch dir remains, no exception; (j) STALE TIMER: call purge_batch on an undone/merged-away/missing batch id -> silent no-op; (k) MERGE: merge_batches of two batches -> single manifest, expires_at = now+TTL (not the earlier of the two), ALL staged files EXIST under the winner batch dir (assert each file present, byte-compare content), loser batch dirs are EMPTY before removal and removed, no file dropped; (k2) MERGE THEN UNDO: merge two batches then undo(winner) -> EVERY file (incl. the losers' files) is restored to its ORIGINAL path (assert each original path exists with matching bytes); (k3) MERGE THEN PURGE: merge two batches, rewrite expires_at to the past, purge_expired() -> winner batch dir is EMPTY (all files incl. moved ones removed) and removed; (l) MERGE MOVE FAILURE: monkeypatch os.rename to fail during merge (after 1 file already moved) -> merge returns None, already-moved file is moved BACK, ALL original batch dirs + manifests + files intact, and a subsequent sequential undo of each constituent batch still restores every file to its original path; (m) delete_undo_enabled=false -> stage returns None and nothing is created; (n) simulated OSError during staging (monkeypatch os.rename to fail on 2nd file) -> returns None, first file renamed back, no orphaned batch dir; (o) DEFAULT_SETTINGS contains delete_undo_enabled=True; (p) SCANNER EXCLUSION: build a fake tree `<root>/.lm-pending-delete/x/model.safetensors` + `<root>/.lm-pending-delete/x/model.metadata.json` and run the model scanner's directory walk against the root -> no cache/raw_data entry whose path contains `.lm-pending-delete`; run `_find_pending_models_from_filesystem` (checkpoint scanner) against the root -> staged metadata.json paths are NOT returned; (q) MERGE TIMER: merge two batches, spy on asyncio.create_task -> a FRESH purge task is spawned for the winner id; then simulate time passing (advance the re-anchored expires_at to the past) and call purge_batch(winner) -> files purged (proves the fresh timer contract); (r) CROSS-TYPE PURGE ENUMERATION: create an expired batch under a LORA root, an expired batch under a CHECKPOINT root, an expired batch under an EMBEDDINGS root, and an expired recipe batch under the recipe staging dir -> one purge_expired() call removes ALL FOUR (proves enumeration across scanner types + recipe dir); (s) PARTIALLY-RESTORED PURGE: batch with 2 entries where entry 1 is restored:true (staged file absent) and entry 2's staged file present -> purge_batch removes entry 2 + the batch dir with NO exception (missing file treated as already-purged); (t) QUARANTINE IS TERMINAL: after a quarantine (dir renamed to `<batch_id>.orphaned`), run purge_expired() a SECOND time -> the dir name is UNCHANGED (no re-rename), files still present, no exception; (u) LOCK NO-DEADLOCK: call stage_model_delete while a purge_expired() (with an expired batch) is in flight, and call undo while purge_batch is in flight -> both complete without deadlock or exception, and the expired batch is either fully purged or fully restored, never partially (exercises the LOCK HIERARCHY: purge_expired is lock-free, purge_batch locks).
  QA scenarios (name the exact tool + invocation): happy: `pytest tests/services/test_pending_delete_service.py -q` -> all pass; failure: monkeypatch-based tests (e), (i), (l), (n) -> expect rollback/retry/skip/abort assertions pass; scanner exclusion (p) -> assert zero ghost entries in both scanners; merge lifecycle (k)/(k2)/(k3)/(q) -> assert no file dropped, original paths restored, purge empties merged dir, fresh timer armed. Evidence .omo/evidence/undo-delete-staging/task-1-undo-delete-staging.md
  Commit: Y | `feat(delete): add pending-delete staging service with undo/purge`

- [x] 2. Wire single model delete into staging — ModelLifecycleService.delete_model + handler passthrough + tests
  What to do: In `py/services/model_lifecycle_service.py` `delete_model` (lines 101-154): replace the direct `delete_model_artifacts` call (line 132-134) with: (1) build staging via a module-level `get_pending_delete_service()` (lazy singleton — do NOT add to __init__ signature; construct inside the method or via a small accessor to keep the existing constructor contract); (2) `batch_id = await pending_delete_service.stage_model_delete(scanner=self._scanner, target_dir=target_dir, file_name=file_name, main_extension=main_extension, original_file_path=file_path, cached_entry=cached_entry)`; (3) if batch_id is None -> call existing `delete_model_artifacts` (fallback, unchanged behavior); (4) if staged -> the artifacts are already renamed away, so SKIP delete_model_artifacts; cache/hash/sync/persist logic (lines 136-152) runs identically (files are gone from original location, cache removal is still correct). Return dict becomes `{"success": True, "deleted_files": deleted_files, "batch_id": batch_id}` (batch_id null when not staged). Keep `_require_path_in_library_roots` and `_sync_update_for_model` untouched. `py/routes/handlers/model_handlers.py` delete_model (478-492) needs NO change (result passthrough already returns the dict). Must NOT do: do not change delete_model_artifacts itself (bulk todo 3 also uses it); do not touch bulk paths in this todo; do not add batch_id to the 400/500 error responses.
  Parallelization: Wave 2 | Blocked by: 1 | Blocks: 5, 8
  References: py/services/model_lifecycle_service.py:101-154 (delete_model full body), :19-48 (delete_model_artifacts), py/routes/handlers/model_handlers.py:478-492 (handler), :57-74 (_broadcast_models_changed — no change needed here), tests/services/test_model_lifecycle_service.py (existing delete tests: test_delete_model_removes_gguf_file ~:470-508)
  Acceptance criteria (agent-executable): `pytest tests/services/test_model_lifecycle_service.py tests/services/test_pending_delete_service.py -q` passes. New tests: delete_model with undo enabled -> file renamed to staging, response contains batch_id, cache entry removed, `_persist_calls` incremented (existing ScannerForDelete tracks it); delete_model with delete_undo_enabled=false (monkeypatch settings) -> old behavior (os.remove), batch_id is None; delete_model with staging failure -> files deleted via fallback, batch_id None, no staging dir left.
  QA scenarios: happy: `pytest tests/services/test_model_lifecycle_service.py -q` -> green; failure: staging-failure test -> asserts fallback deleted files AND no `.lm-pending-delete` leftover. Evidence .omo/evidence/undo-delete-staging/task-2-undo-delete-staging.md
  Commit: Y | `feat(delete): stage single model deletes for undo`

- [x] 3. Wire bulk model delete into staging — ModelScanner.bulk_delete_models + response batch_id + tests
  What to do: In `py/services/model_scanner.py` `bulk_delete_models` (lines 2181-2269): inside the per-file loop where `delete_model_artifacts` is called (line 2221), attempt staging per file BUT aggregate into ONE batch for the whole bulk action: capture each file's `cached_entry` snapshot BEFORE the cache mutation (`_batch_update_cache_for_deleted_models` runs after the loop, 2271-2335 — snapshots must be taken pre-mutation); call `pending_delete_service.stage_model_delete(...)` per file, collecting non-null batch_ids; at the end of the loop call `merge_batches(batch_ids)` (todo 1 — re-anchors expires_at to now+TTL at merge, so the merged batch outlives the frontend toast countdown; stale timers of merged-away ids are harmless per todo 1 fire-time re-check). NO-MERGE FALLBACK CONTRACT (merge_batches can return None on move failure): response then contains `"batch_ids": [constituent ids]` instead of a single `"batch_id"` (frontend todo 9 undoes them sequentially); when merge succeeds response contains `"batch_id": merged_id` and no batch_ids array. When staging returns None for a file -> existing hard delete fallback for that file only. Add the batch field(s) to the success return dict (lines 2254-2269). If the operation was cancelled mid-way (is_cancelled, line 2255): still merge whatever was staged and include the batch field(s) in the SAME success dict (status='cancelled' + batch) — the frontend (todo 9) shows the undo toast for the staged subset. Must NOT do: do not change the cancellation semantics; do not stage when the batch is cancelled before any file was processed; do not touch `_batch_update_cache_for_deleted_models`; no changes to the except-block failure return dict (2261-2263).
  Parallelization: Wave 2 | Blocked by: 1 | Blocks: 5, 9
  References: py/services/model_scanner.py:2181-2269 (bulk_delete_models), :2221 (delete_model_artifacts call), :2254-2269 (return dicts incl. status='cancelled' path), :2271-2335 (_batch_update_cache_for_deleted_models), py/services/model_lifecycle_service.py:308-318 (lifecycle bulk entry — passthrough), py/services/pending_delete_service.py (todo 1 module — merge_batches + fire-time re-check + no-merge fallback)
  Acceptance criteria (agent-executable): `pytest tests/services/test_model_scanner.py tests/services/test_model_lifecycle_service.py -q` passes. New tests (put the NEW bulk-staging tests in `tests/services/test_model_scanner.py`, NOT in test_model_lifecycle_service.py — todo 2 also edits that file in the same wave and the two test suites must not collide): bulk_delete_models with 2 files -> response contains ONE batch_id, both files present in that batch's staging dir, cache updated; merged manifest expires_at >= staging completion time + TTL (re-anchor assertion); merge-failure fallback (monkeypatch os.rename to fail during merge) -> response contains batch_ids array of length 2, both batches intact; with delete_undo_enabled=false -> batch_id null, files os.removed; cancelled mid-way after 1 file staged -> status='cancelled' AND batch_id present, that file undoable.
  QA scenarios: happy: run the new bulk test -> green; failure: assert zero staging dirs exist when undo disabled. Evidence .omo/evidence/undo-delete-staging/task-3-undo-delete-staging.md
  Commit: Y | `feat(delete): stage bulk model deletes for undo`

- [x] 4. Wire recipe deletes into staging — persistence_service delete_recipe + bulk_delete + batch_id + tests
  What to do: In `py/services/recipes/persistence_service.py`: `delete_recipe` (lines 193-209) — before the two os.remove calls (204-206), call `batch_id = await pending_delete_service.stage_recipe_delete(recipe_json_path=recipe_json_path, image_path=image_path, recipe_data=recipe_data)` (stage only existing files — missing preview image is skipped; recipe_data is stored as recipe_snapshot in the manifest); if batch_id is None -> existing behavior; if staged -> remove originals via os.remove as today (files exist in staging copy). Include `"batch_id": batch_id` in the PersistenceResult payload (line 209). `bulk_delete` (lines 439-482) — same per id, then merge into ONE batch via `merge_batches(...)` with the SAME no-merge fallback contract as todo 3 (merge failure -> `"batch_ids"` array in the payload); include the batch field(s) in the result payload (474-482). Must NOT do: do not stage the image only when the JSON stage failed (stage_recipe_delete must be atomic internally — copy JSON first, image second, roll back copies on failure, per todo 1); do not change `recipe_scanner.remove_recipe` / `bulk_remove` calls.
  Parallelization: Wave 2 | Blocked by: 1 | Blocks: 5, 8, 9
  References: py/services/recipes/persistence_service.py:20-25 (PersistenceResult), :193-209 (delete_recipe), :439-482 (bulk_delete), :464-466 (per-id os.remove), py/routes/handlers/recipe_handlers.py:1422-1438 (single handler passthrough — no change), :1554-1573 (bulk handler passthrough — no change), py/services/pending_delete_service.py (todo 1 module — merge_batches + no-merge fallback), tests/services/test_recipe_persistence.py (existing persistence tests; create if missing, follow tests/services/test_recipe_scanner.py style)
  Acceptance criteria (agent-executable): `pytest tests/services/test_recipe_persistence.py tests/services/test_pending_delete_service.py -q` passes. New tests: delete_recipe with undo enabled -> JSON + image exist in global staging dir, originals removed, payload batch_id set, manifest recipe_snapshot present; recipe with missing preview image -> only JSON staged, no crash; with undo disabled -> no staging, payload batch_id None; bulk_delete with 2 ids -> single batch_id, both recipes in one batch dir with re-anchored expires_at; merge-failure fallback -> batch_ids array of length 2.
  QA scenarios: happy: run new tests -> green; failure: assert staged copies match original bytes (read both files). Evidence .omo/evidence/undo-delete-staging/task-4-undo-delete-staging.md
  Commit: Y | `feat(delete): stage recipe deletes for undo`

- [x] 5. Undo endpoint POST /api/lm/undo-delete + cache restoration + broadcast + tests
  What to do: New handler file `py/routes/handlers/pending_delete_handler.py` with `class PendingDeleteHandler`: `async def undo_delete(self, request) -> web.Response` — body `{"batch_id": str}`; call `pending_delete_service.undo(batch_id)`; after success: if kind == "model", resolve the CORRECT scanner by `manifest["model_type"]` (the manifest carries it — resolve via the same per-type service/scanner resolution the model route registrars use; see py/routes/base_model_routes.py:227 per-type registrar construction and ModelServiceFactory; do NOT use a hardcoded lora scanner); restore cache entry from manifest `model_snapshot`: `cache = await scanner.get_cached_data()`; remove ALL existing raw_data items whose file_path equals the snapshot's file_path (a rescan between delete and undo may have re-added a stale entry); `cache.raw_data.append(snapshot)`; `await cache.resort()`; `scanner.bump_cache_version()`; follow the bulk-delete cache-update pattern (py/services/model_scanner.py:2271-2335) INCLUDING any version-index rebuild (rebuild_version_index at :2324) so the version index does not go stale (the single-delete path does not do this today — the undo path must); RESTORE TAG COUNTS from the snapshot's tag data (mirror the `_tags_count` decrement logic in `_batch_update_cache_for_deleted_models` in reverse — the bulk path decrements counts, undo must re-increment; guard missing tag field); `scanner._hash_index.add_entry(snapshot.get("sha256", ""), snapshot["file_path"], snapshot.get("autov3"))` (py/services/model_hash_index.py:16 — the service guards missing sha256 internally, still guard defensively); `await scanner._persist_current_cache()`. If kind == "recipe": load restored JSON file, `await recipe_scanner.add_recipe(recipe_data)` (py/services/recipe_scanner.py:2136 — recipe JSON embeds the full recipe_data incl. id/file_path; NOTE add_recipe only READS `_json_path_map` so full consistency of path mapping is restored by the forced frontend refresh (window.recipeManager.loadRecipes(true)) — covered by test (i) below; the SQLite row may transiently carry an empty json_path until that refresh, self-healing). Then `_broadcast_models_changed()` (import from py/routes/handlers/model_handlers.py:57-74) ONLY for kind == "model" — recipe undo is client-refresh only (documented limitation: other open tabs' recipe lists refresh on their next interaction). Return `{"success": True, "restored": [<original paths>], "kind": kind}`. ValueError (expired/unknown batch/target occupied) -> 404 `{"success": False, "error": str}`; Exception -> 500. ROUTE REGISTRATION (use a shared routes class, NOT the per-model-type registrar — the ModelRouteRegistrar is instantiated per model type and its route table is registered 3x; adding a non-prefixed route there would register it 3 times): create `py/routes/pending_delete_routes.py` with `class PendingDeleteRoutes: @staticmethod def setup_routes(app)` registering `POST /api/lm/undo-delete` once, mirroring the existing shared-route classes (py/routes/misc_routes.py MiscRoutes / py/routes/update_routes.py UpdateRoutes); register it ONCE per mode next to the other shared routes: in py/lora_manager.py:170-172 (ComfyUI mode) and standalone.py:356-358 (standalone mode). Must NOT do: do not add a per-prefix undo endpoint; do not register through the per-model-type registrar; do not rescan the whole library on undo (snapshot restore only); do not purge the batch before cache restoration; do not touch FTS index directly (add_recipe handles it).
  Parallelization: Wave 3 | Blocked by: 1, 2, 3, 4 | Blocks: 6, 8, 9
  References: py/services/model_hash_index.py:16 (add_entry), py/services/recipe_scanner.py:2136 (add_recipe), :2163 (_json_path_map read-only in add_recipe), py/services/model_cache.py (raw_data/resort), py/services/model_scanner.py:673-684 (_persist_current_cache), :113-119 (bump_cache_version), :2271-2335 (bulk cache-update pattern), :2324 (rebuild_version_index), py/routes/handlers/model_handlers.py:57-74 (_broadcast_models_changed), py/routes/base_model_routes.py:227 (per-type registrar construction — why NOT to use it), py/routes/misc_routes.py (shared routes class pattern to mirror), py/lora_manager.py:170-172 (shared route registration in plugin mode), standalone.py:356-358 (shared route registration in standalone mode), tests/routes/test_lora_manager_lifecycle.py (route registration test style)
  Acceptance criteria (agent-executable): `pytest tests/routes/test_pending_delete_routes.py -q` passes. Tests: (a) undo of a staged model batch (loras) -> files restored + cache raw_data contains entry again + hash index has path + broadcast called (mock ws_manager) + response restored list; (b) undo of a staged CHECKPOINT batch -> resolves the checkpoint scanner (assert the checkpoint cache was updated, not the lora cache); (c) undo of staged recipe batch -> files restored + recipe_scanner.add_recipe called; (d) undo with unknown/expired batch_id -> 404 response; (e) route registered exactly ONCE in both modes (assert `len([r for r in app.router.routes() if r.method == 'POST' and r.path == '/api/lm/undo-delete']) == 1` after both plugin-mode and standalone-mode setup); (f) RESTART-SAFE UNDO (T8): write a staged batch + manifest to disk WITHOUT spawning the in-process timer (simulate post-restart state), call the undo handler -> files restored + cache restored (proves undo does not depend on the timer task); (g) RESCAN STALENESS (T7): pre-insert a stale raw_data entry with the same file_path as the snapshot, run undo -> exactly one entry remains and it equals the snapshot; (h) RECIPE RE-DELETE AFTER UNDO (T9): after undo + forced refresh (simulate loadRecipes(true) by rebuilding recipe scanner state), immediately delete the same recipe again -> succeeds (path map rebuilt); (i) recipe undo without prior refresh still restores files (path-map gap is masked by refresh in the flow, but the endpoint itself does not crash); (j) TAG COUNTS: bulk-delete 2 tagged models, capture `_tags_count`, undo -> `_tags_count` matches pre-delete counts; (k) EMBEDDINGS UNDO: undo of a staged embeddings batch -> resolves the embedding scanner (assert the embeddings cache was updated, not the lora cache) — embeddings shares the loras/checkpoints code path, this test pins the per-type resolution for ALL model types.
  QA scenarios: happy: full route test suite -> green; failure: (d) expired batch -> assert 404 body contains "expired"; (e) assert no duplicate route error at startup in both modes. Evidence .omo/evidence/undo-delete-staging/task-5-undo-delete-staging.md
  Commit: Y | `feat(delete): add undo-delete endpoint with cache restoration`

- [x] 6. Purge scheduling — per-batch timer + on_startup sweep + opportunistic purge + tests
  What to do: (a) In PendingDeleteService.stage_* methods: after successful staging, spawn `asyncio.create_task(self._purge_batch_after_ttl(batch_id), name=f"pending_delete_purge_{batch_id}")` where the task sleeps a computed duration then calls purge_batch — CRITICAL: purge_batch RE-READS the manifest's current expires_at at fire time (per todo 1 spec) so merged-away/undone/not-yet-expired batches are silent no-ops; stale timers from merged constituent batches or undone batches are harmless. (The merge fresh-timer is armed inside merge_batches per todo 1 — do not double-arm.) KNOWN RESIDUAL GAP (document, do not fix): a non-expired batch whose in-process timer died with the process is purged only by the next stage/undo call (opportunistic purge) or next restart (startup sweep skips non-expired); on a fully idle server it lingers past expiry until then — bounded, self-healing, no data loss. (b) Startup sweep: in `py/lora_manager.py` `_initialize_services` (lines 189-255) — after scanners initialize (in `_run_post_initialization_tasks` at :257 or right after the init_tasks gather), add `asyncio.create_task(pending_delete_service.purge_expired(), name="pending_delete_startup_sweep")`. This single hook covers both modes because StandaloneLoraManager reuses `cls._initialize_services()` (standalone.py:370-374). (c) Opportunistic: call `await self.purge_expired()` at the start of stage_model_delete/stage_recipe_delete/undo — MUST be called BEFORE those methods acquire the service lock, and purge_expired itself takes NO lock (LOCK HIERARCHY per todo 1 — asyncio.Lock is not re-entrant; calling it while holding the lock deadlocks). Fire-and-forget is NOT acceptable — await it; it is cheap when empty. Must NOT do: do not purge non-expired batches on startup (undo must survive restart with live browser tab); do not delete manifest-less/corrupted batches on sweep (quarantine only, per todo 1); do not rmtree past per-file purge errors; do not block startup on the sweep (create_task, not await); do not let purge_expired acquire the lock; no changes to on_shutdown.
  Parallelization: Wave 3 | Blocked by: 1, 5 (same file lora_manager.py — run AFTER 5) | Blocks: —
  References: py/lora_manager.py:183-187 (on_startup append), :189-255 (_initialize_services), :257+ (_run_post_initialization_tasks), standalone.py:370-374 (reuses cls._initialize_services), py/services/model_scanner.py:40-63 (task/singleton idiom), py/services/pending_delete_service.py (todo 1 module — purge semantics + fire-time re-check)
  Acceptance criteria (agent-executable): `pytest tests/services/test_pending_delete_service.py -q` passes. Tests: (a) staging schedules a purge — monkeypatch asyncio.create_task capture, assert task created with name prefix "pending_delete_purge_"; (b) purge_expired called at stage/undo entry (spy on method); (c) `tests/routes/test_lora_manager_lifecycle.py` still passes (startup hook wiring) and a new assertion: after LoraManager.add_routes + on_startup invocation, purge_expired task was spawned (mock or spy via monkeypatched create_task); (d) stale-timer no-op covered by todo 1 acceptance (j).
  QA scenarios: happy: run pytest on both files -> green; failure: assert non-expired batch survives startup sweep call. Evidence .omo/evidence/undo-delete-staging/task-6-undo-delete-staging.md
  Commit: Y | `feat(delete): schedule pending-delete purges (timer + startup sweep)`

- [x] 7. Settings toggle delete_undo_enabled — backend default + settings modal checkbox + tests
  What to do: Backend: the `delete_undo_enabled` DEFAULT_SETTINGS key is added in todo 1 — do NOT re-add it here; only verify it exists (skip if already present). Frontend: in `templates/components/modals/settings_modal.html` add a checkbox labeled with the new i18n key `settings.deleteUndoEnabled` (find an existing boolean setting checkbox in the same file, e.g. any `type="checkbox"` bound setting, and copy its exact markup + wiring pattern); wire the change handler in `static/js/managers/SettingsManager.js` following the existing pattern used by sibling checkboxes (search that file for the sibling setting key). Must NOT do: do not create a new settings section; do not change how settings are persisted; do not touch other settings; do not modify DEFAULT_SETTINGS (already handled in todo 1).
  Parallelization: Wave 3 | Blocked by: 1 | Blocks: 9
  References: py/services/settings_manager.py:57-119 (DEFAULT_SETTINGS), :1390-1392 (get), templates/components/modals/settings_modal.html (existing checkbox patterns), static/js/managers/SettingsManager.js (existing checkbox wiring; ~3300 lines — grep for a sibling boolean setting key), locales/en.json (key added in todo 12 — add key here or in 12, whichever lands first; keep the key name `settings.deleteUndoEnabled`)
  Acceptance criteria (agent-executable): `pytest tests/services/test_settings_manager.py -q` passes (add assertion: DEFAULT_SETTINGS contains delete_undo_enabled=True; and staging respects it — covered in todo 1/2 tests). Frontend: `npm run test:js` passes; the settings modal renders the checkbox (covered by existing settings page test file if present, else a jsdom assertion in the settings manager test).
  QA scenarios: happy: pytest + npm test green; failure: toggle checkbox -> settings saved -> server delete skips staging (covered by todo 2 test with monkeypatched setting). Evidence .omo/evidence/undo-delete-staging/task-7-undo-delete-staging.md
  Commit: Y | `feat(settings): add delete_undo_enabled toggle`

- [x] 8. Frontend single-delete undo flows — baseModelApi.deleteModel + modalUtils.confirmDelete + RecipeCard.confirmDeleteRecipe + tests
  What to do: (a) `static/js/api/baseModelApi.js` `deleteModel` (184-216): return `{ success: true, batch_id: data.batch_id || null }` instead of `true`; when `data.batch_id` present, do NOT show `toast.api.deleteSuccess` — the caller shows the undo toast; when absent, keep existing toast + `removeItemByFilePath` as today. Keep `removeItemByFilePath` called in both cases (file is gone). UNDO-BLIND CALLER NOTE: `ModelVersionsTab.js:1136-1144` (version delete) calls this same method — its server-side deletes WILL be staged (undoable server-side) but the versions-tab shows its own success toast and NO undo toast; that flow is intentionally out of scope (documented in Scope OUT) — only ensure the new return object does not break its `if (!result)`-style checks (verify at lines 1136-1144 and adapt ONLY the type check, nothing else). (b) `static/js/utils/modalUtils.js` `confirmDelete` (27-42): capture `const result = await getModelApiClient().deleteModel(pendingDeletePath)`; if `result?.batch_id` -> `showActionToast('toast.undo.deleted', { name: <model name> }, 'success', { actionText: t('toast.undo.action'), onAction: () => handleUndoDelete(result.batch_id, refreshModels) })`; keep existing duplicate-badge refresh. (c) `static/js/components/RecipeCard.js` `confirmDeleteRecipe` (405-449): after the raw fetch, read `data.batch_id`; when present show action toast with `handleUndoDelete(batch_id, () => window.recipeManager.loadRecipes(true))`; when absent keep existing success toast. (d) New shared util `static/js/utils/undoHelpers.js` (or add to uiHelpers.js — pick ONE location and state it in the PR; do not split across files): `export async function handleUndoDelete(batchId, refreshFn, options = {})` with `options = { showToast = true, refresh = true }` — POST `/api/lm/undo-delete` body `{batch_id}`; on 200: if `options.refresh` -> `refreshFn()`, if `options.showToast` -> `showToast('toast.undo.restored', {}, 'success')`; on 404: read the error BODY text (parse `{error}` — distinguishes "expired" from unknown/occupied) and show `toast.undo.expired` when the message contains "expired", else `toast.undo.failed` with the server message; on other error -> `showToast('toast.undo.failed', { error }, 'error')`. The suppression options exist SOLELY for todo 9's sequential batch_ids loop (each iteration calls `handleUndoDelete(id, null, { showToast: false, refresh: false })`, then the loop does ONE final `refreshFn()` + ONE `toast.undo.restored` toast). refreshModels for model pages: reuse the pattern from ModelDuplicatesManager.js:740 — `resetAndReload(true)` imported from modelApiFactory. Must NOT do: do not change showToast signature; do not alter ModelVersionsTab.js behavior beyond the return-type compatibility check; do not let handleUndoDelete show N restored toasts or fire N refreshes for a multi-batch undo.
  Parallelization: Wave 4 | Blocked by: 2, 4, 5, 10 | Blocks: 9, 11, 12
  References: static/js/api/baseModelApi.js:184-216 (deleteModel), static/js/utils/modalUtils.js:27-42 (confirmDelete), static/js/components/RecipeCard.js:405-449 (confirmDeleteRecipe), static/js/components/shared/ModelVersionsTab.js:1136-1144 (other caller), static/js/components/ModelDuplicatesManager.js:740 (resetAndReload usage), static/js/managers/BulkManager.js:657-659 (removeItemByFilePath loop pattern), static/js/utils/uiHelpers.js:136-193 (showToast — from todo 10), tests/frontend/utils/uiHelpers.dom.test.js, tests/frontend/pages/lorasPage.test.js + recipesPage.test.js (existing delete-flow tests to extend)
  Acceptance criteria (agent-executable): `npm run test:js` passes with new tests: baseModelApi.deleteModel returns batch_id and suppresses success toast when present; modalUtils.confirmDelete shows action toast and undo triggers handleUndoDelete POST + refresh; RecipeCard.confirmDeleteRecipe same; handleUndoDelete 404 path shows expired toast.
  QA scenarios: happy: vitest for the three flows -> green; failure: mock fetch 404 -> assert expired toast key called. Evidence .omo/evidence/undo-delete-staging/task-8-undo-delete-staging.md
  Commit: Y | `feat(delete): undo toasts for single model/recipe deletes`

- [x] 9. Frontend bulk + duplicates undo flows — BulkManager + both API clients + both DuplicatesManager + tests
  What to do: (a) `static/js/api/baseModelApi.js` `bulkDeleteModels` (1591-1642): include `batch_id: result.batch_id || null` AND `batch_ids: result.batch_ids || null` in the success return object (incl. the status='cancelled' path, which uses the same success dict). (b) `static/js/api/recipeApi.js` `RecipeSidebarApiClient.bulkDeleteModels` (623-664): include `batch_id: result.batch_id || null` + `batch_ids: result.batch_ids || null`. (c) `static/js/managers/BulkManager.js` `confirmBulkDelete` (633-672): after success, if `result.batch_id` -> single action toast for the whole action: `showActionToast('toast.undo.deletedBulk', { count: result.deleted_count }, 'success', { actionText: t('toast.undo.action'), onAction: () => handleUndoDelete(result.batch_id, refreshFn) })` — one batch covers the whole bulk action (backend merges per todo 3); MERGE-FAILURE FALLBACK: if `result.batch_id` is null but `result.batch_ids?.length` — same single toast, onAction loops `handleUndoDelete(id, null, { showToast: false, refresh: false })` for each id sequentially (suppression per todo 8's options contract), then after ALL succeed does ONE final `refreshFn()` + ONE `showToast('toast.undo.restored', {}, 'success')`; if any iteration fails, stop the loop and show its error toast (no final refresh). refreshFn = recipes ? `window.recipeManager.loadRecipes(true)` : `resetAndReload(true)`. Keep the existing `toast.models.deletedSuccessfully` when both batch fields are null. CANCELLED BULK with a batch field (status='cancelled' + staged subset): show the SAME action toast (the staged subset is undoable) INSTEAD of the plain cancelled toast; cancelled without batch field keeps the existing cancelled toast. (d) `static/js/components/DuplicatesManager.js` `confirmDeleteDuplicates` (457-494) and `static/js/components/ModelDuplicatesManager.js` `confirmDeleteDuplicates` (710-776): read `data.batch_id` (and fall back to `data.batch_ids` array) from the raw fetch response; when non-empty show action toast (undo restores the whole selected group) before/instead of the current success toast, with EXPLICIT refreshFn: DuplicatesManager (recipes) -> `() => window.recipeManager.loadRecipes(true)`; ModelDuplicatesManager (models) -> `resetAndReload(true)` from modelApiFactory (same as todo 8's refreshModels). Keep exitDuplicateMode / resetAndReload + find-duplicates re-check logic unchanged (the undo click triggers its own full refresh via refreshFn). Must NOT do: do not add per-file undo toasts for bulk (single group toast only); do not change abort/cancel handling; do not alter the failed-count error toasts.
  Parallelization: Wave 4 | Blocked by: 3, 4, 5, 7, 8 (same file baseModelApi.js — run AFTER 8), 10 | Blocks: 11, 12
  References: static/js/api/baseModelApi.js:1591-1642 (bulkDeleteModels), static/js/api/recipeApi.js:623-664 (recipe bulk), static/js/managers/BulkManager.js:633-672 (confirmBulkDelete), :134-142 (getActiveApiClient), static/js/components/DuplicatesManager.js:457-494, static/js/components/ModelDuplicatesManager.js:710-776, tests/frontend/api/recipeApi.bulk.test.js, tests/frontend/components/duplicatesManager.test.js, tests/frontend/components/modelDuplicatesManager.test.js
  Acceptance criteria (agent-executable): `npm run test:js` passes with new tests: bulkDeleteModels passthrough of batch_id AND batch_ids in both clients; BulkManager shows action toast with batch_id and falls back to success toast when both are null; BATCH_IDS FALLBACK PATH (first-class — cross-volume bulks hit it via EXDEV during MERGE [annotated 2026-08: sibling staging removed EXDEV from stage/undo renames; only cross-volume merges can still produce it]): response with batch_id null + batch_ids [id1, id2] -> action toast shown, clicking undo calls handleUndoDelete(id1, null, {showToast:false, refresh:false}) then handleUndoDelete(id2, null, {showToast:false, refresh:false}) with exactly ONE final refresh and exactly ONE 'toast.undo.restored' toast (spy on refreshFn and showToast); fallback loop failure mid-way (mock id2 undo to 404) -> loop stops, error toast shown, NO final refresh; handleUndoDelete 404 with body {"error":"...expired..."} -> expired toast, with body {"error":"Target path occupied"} -> failed toast with server message; cancelled bulk with batch_id shows the action toast (not the cancelled toast); cancelled with batch_ids shows the action toast; both DuplicatesManager confirm methods read batch_id (and batch_ids fallback) and trigger undo.
  QA scenarios: happy: vitest for the four flows -> green; failure: response without batch_ids -> assert legacy success toast still fires. Evidence .omo/evidence/undo-delete-staging/task-9-undo-delete-staging.md
  Commit: Y | `feat(delete): undo toasts for bulk and duplicates deletes`

- [x] 10. showActionToast — extract toast internals + action button + 30s countdown + CSS + tests
  What to do: In `static/js/utils/uiHelpers.js` (showToast at 136-193): extract two internal helpers (NOT exported): `createToastElement(message, type)` (builds the div, returns it) and `appendToast(toast, durationMs)` (append + show class + setTimeout dismiss + transitionend removal + reposition logic). Rewrite showToast to use them with its existing durations (2000/5000). Add NEW exported `showActionToast(key, params = {}, type = 'info', options = {})` where options = `{ actionText, onAction, durationMs = 30000, countdown = true }`: builds toast via createToastElement; appends a `<button class="toast-action-btn">` with actionText; clicking it calls onAction once and dismisses the toast immediately; countdown: when enabled, append a `<span class="toast-countdown">` and run `setInterval` 1s updating `(N)s` text until 0 then dismiss (clear interval on manual dismiss via button click or normal timeout path). CSS in `static/css/components/toast.css`: `.toast-action-btn` (bordered/ghost button, hover state), `.toast-countdown` (muted small text). Must NOT do: do not change showToast's signature or any existing call site; do not change toast durations for existing toasts; do not add a dependency.
  Parallelization: Wave 3 | Blocked by: — | Blocks: 8, 9
  References: static/js/utils/uiHelpers.js:136-193 (showToast), static/css/components/toast.css (.toast flex gap:12px — button slot), static/js/managers/BannerService.js (actions array + onRegister as house pattern reference only), tests/frontend/utils/uiHelpers.dom.test.js (existing toast DOM tests)
  Acceptance criteria (agent-executable): `npm run test:js` passes. New tests: showActionToast renders button + countdown span; clicking button invokes onAction once and dismisses; countdown reaches 0 and dismisses (use fake timers); showToast behavior unchanged (existing uiHelpers.dom.test.js still green).
  QA scenarios: happy: vitest with fake timers -> green; failure: assert double-click calls onAction exactly once. Evidence .omo/evidence/undo-delete-staging/task-10-undo-delete-staging.md
  Commit: Y | `feat(toast): add action toasts with countdown`

- [x] 11. C-friction modal changes — delay-activate delete buttons + cancel focus + warning copy + size display + tests
  What to do: (a) Delay-activate: in the shared delete-confirm modals (`templates/components/modals/confirm_modals.html` — deleteModal, duplicateDeleteModal, modelDuplicateDeleteModal, bulkDeleteModal; and the inline RecipeCard modal content) — implement via a shared JS helper `static/js/utils/modalUtils.js` (or uiHelpers): `armDeleteButton(modalElement)` — disables all `.delete-btn` in the modal on open, `setTimeout(1500)` then enables; call it in `showDeleteModal` (modalUtils.js:7-25) after showModal, in `RecipeCard.showDeleteConfirmation` (339-403) after building content, in `BulkManager.showBulkDeleteModal` (605-631) and both `deleteSelectedDuplicates` methods (DuplicatesManager.js:435-454, ModelDuplicatesManager.js:688-707). Handle the RecipeCard onClose reset (RecipeCard.js:378-384) — reset also clears the pending timer. (b) Cancel focus: in ModalManager.showModal (static/js/managers/ModalManager.js:397-437) — after showing a modal with class `delete-modal`, focus the first `.cancel-btn` (guard: only for delete-type modals; `modal.element.querySelector('.cancel-btn')?.focus()`), and on close restore focus to the previously focused element (store in showModal, restore in closeModal 439+). (c) Warning copy + size: in `showDeleteModal` (modalUtils.js:18-22) change `.delete-model-info` to include a CONDITIONAL warning line — when the client-side settings value `delete_undo_enabled` is falsy (read from the settings manager singleton used by the settings page; if the setting is unavailable, use the neutral copy): `modals.deleteModel.permanentWarning` ("This will permanently delete the file from disk.") — when undo IS enabled use `modals.deleteModel.recoverableWarning` ("This will permanently delete the file after 30 seconds unless you undo."). Also, when `card.dataset.file_size` is present, a line "Frees {size}" using `formatFileSize(parseInt(card.dataset.file_size))` (import formatFileSize — locate its export, e.g. from static/js/utils/i18nHelpers.js or wherever ModelModal.js:615 imports it from; verify at implementation). Must NOT do: no type-to-confirm; do not delay-activate the exclude modal or other modals; do not change focus behavior for non-delete modals; no changes to modal animations.
  Parallelization: Wave 5 | Blocked by: 8, 9 (same files: modalUtils.js, RecipeCard.js, BulkManager.js, DuplicatesManager.js, ModelDuplicatesManager.js) | Blocks: 12
  References: templates/components/modals/confirm_modals.html:1-70 (delete modals), static/js/utils/modalUtils.js:7-25 (showDeleteModal), :44-47 (closeDeleteModal), static/js/components/RecipeCard.js:339-403 (showDeleteConfirmation), :378-384 (onClose reset), static/js/managers/BulkManager.js:605-631 (showBulkDeleteModal), static/js/components/DuplicatesManager.js:435-454, static/js/components/ModelDuplicatesManager.js:688-707, static/js/managers/ModalManager.js:397-437 (showModal), :439-460 (closeModal), static/js/components/shared/ModelCard.js:467 (card.dataset.file_size), static/js/components/shared/ModelModal.js:615 (formatFileSize usage — find its import source), tests/frontend/pages/*Page.test.js (modal open/close tests)
  Acceptance criteria (agent-executable): `npm run test:js` passes. New tests: delete modal opens with disabled delete button, enabled after 1500ms (fake timers); cancel button receives focus on open; closeModal restores focus; showDeleteModal content includes permanentWarning and size line when dataset present; recipe/bulk/duplicates delete modals also delay-activate.
  QA scenarios: happy: vitest fake-timer tests -> green; failure: assert clicking disabled delete button fires nothing (dispatch click, expect no API call). Evidence .omo/evidence/undo-delete-staging/task-11-undo-delete-staging.md
  Commit: Y | `feat(delete): harden delete confirmation modals`

- [x] 12. i18n keys + sync script + validation
  What to do: Add to `locales/en.json` (keep alphabetical nesting): `toast.undo.deleted` = "Deleted {name}" ; `toast.undo.deletedBulk` = "Deleted {count} item(s)"; `toast.undo.action` = "Undo"; `toast.undo.restored` = "Item restored"; `toast.undo.expired` = "Undo window expired. The item was permanently deleted."; `toast.undo.failed` = "Undo failed: {error}"; `modals.deleteModel.permanentWarning` = "This will permanently delete the file from disk."; `modals.deleteModel.recoverableWarning` = "This will permanently delete the file after 30 seconds unless you undo."; `modals.deleteModel.freesSpace` = "Frees {size}"; `settings.deleteUndoEnabled` = "Keep deleted items recoverable for 30 seconds (undo)" (label for todo 7 checkbox). KEY PLACEMENT RULE: keys may be added incrementally as their consuming todos (8-11) land, so tests in those waves can reference them — the sync run below happens LAST in this todo. Then run `python scripts/sync_translation_keys.py` (commits the [TODO: Translate] placeholders into the other 9 locales) and verify exit code 0. Must NOT do: do not translate the other locales by hand; do not remove obsolete keys; do not modify scripts/sync_translation_keys.py.
  Parallelization: Wave 5 | Blocked by: 8, 9, 10, 11 (all string keys final) | Blocks: —
  References: locales/en.json (delete keys near 1303-1314, 1945-1948, 1987-1991, 2124-2130, 2166-2170 for placement style), scripts/sync_translation_keys.py (en reference locale, [TODO: Translate] placeholders), tests/i18n/test_i18n.py (structural validation), AGENTS.md ("Run python scripts/sync_translation_keys.py after adding UI strings to locales/en.json")
  Acceptance criteria (agent-executable): `python scripts/sync_translation_keys.py --dry-run` reports the new keys as missing in other locales (before run) then clean after the run; `pytest tests/i18n/test_i18n.py -q` passes; all new keys resolve via `translate('toast.undo.deleted', {name: 'x'})` in a jsdom test.
  QA scenarios: happy: sync + pytest green; failure: assert en.json valid JSON (json.load) and keys present. Evidence .omo/evidence/undo-delete-staging/task-12-undo-delete-staging.md
  Commit: Y | `chore(i18n): add undo-delete and delete-confirmation strings`

## Final verification wave
> Runs in parallel after ALL todos. ALL must APPROVE. Surface results and wait for the user's explicit okay before declaring complete.
- [x] F1. Plan compliance audit
- [x] F2. Code quality review
- [x] F3. Real manual QA
- [x] F4. Scope fidelity

## Commit strategy
- One conventional commit per todo (Y on each todo row), style per AGENTS.md: `feat(delete): ...`, `feat(settings): ...`, `chore(i18n): ...`. Stage only files relevant to the todo (never `git add .`). Do not commit unless the user explicitly asks; commits are prepared per-todo for review.
- No new dependencies, no lockfile changes expected (verify with `git status` — if a lockfile changes, it is a red flag).

## Success criteria
- Deleting a model/recipe (single, bulk, or duplicates, models or recipes): item disappears from UI immediately; an Undo toast with a 30s countdown appears when the server staged the delete (one Undo for the whole bulk/duplicates action).
- Clicking Undo within the window restores the file(s) AND the library entry exactly (model metadata/previews intact; recipe JSON + preview back; list shows the item after refresh).
- After 30s: files are physically removed; disk space freed; Undo attempt returns "expired" toast.
- With `delete_undo_enabled` off: behavior identical to today (hard delete, success toast, no undo).
- Server restart: startup sweep purges expired batches only; a live browser tab can still undo a non-expired batch. Known limitation (accepted): a browser-page refresh within the window loses the Undo button; recipe undo refreshes only the clicking client's list (other tabs catch up on their next refresh).
- Staging failure never blocks deletion (falls back to hard delete), never orphans files. ACCEPTED TRADE-OFF (documented): the modal warning copy is chosen from the SETTING (recoverable when undo is on), so a rare per-file staging failure hard-deletes instantly despite the "recoverable" copy — the delete itself still works and the fallback is the pre-existing behavior (see decisions (3)).
- Bulk undo count mismatch (accepted): `toast.undo.deletedBulk` count reflects ALL deleted files (incl. hard-fallback ones); undo restores only the staged subset — a rare per-file staging-failure can make the count slightly higher than what undo restores; cosmetic, documented.
- Undo refused when the original path is occupied again (re-downloaded file is protected from overwrite); the batch stays staged until TTL — the user's only notice is the "undo failed" toast; this loss-after-refusal is accepted-intent and stated in the modal copy (recoverable wording applies only when undo is on).
- Staged files NEVER appear in the library as ghost entries (model + checkpoint scanners exclude `.lm-pending-delete/`); a mid-undo failure is retry-able via per-entry manifest state; a locked file during purge never blocks the sweep (skipped, retried next round); merge never drops files (move-into-winner + empty-before-remove + abort-on-failure).
- Bulk-delete + undo round-trips restore tag counts exactly (no tag-filter drift until next rescan).
- Delete-confirm modals: delete button unusable for 1.5s, Cancel focused, model modal warns with copy matching the actual undo state (recoverable vs permanent) and shows freed size when known.
- `pytest` (backend) and `npm test` (frontend) fully green; no regressions in the existing 48 showToast call sites; i18n sync clean (exit 0).
- Addendum (2026-08 symlink fix): model stage/undo renames are same-device BY CONSTRUCTION (staging lives inside the deleted model's own folder), so EXDEV on stage/undo is impossible — even when the business path traverses nested symlinks to other volumes. Accepted edge: manually deleting the model's whole FOLDER during the 30s window destroys that batch (it lived inside the folder); a later undo attempt returns 404. See "Symlink fix (2026-08)" below.

## Addendum — Symlink fix (2026-08)

> Post-execution correction (plan `.omo/plans/undo-delete-symlink-fix.md`, commits 5fd4946b / 0c00ee22). The executed todo structure above is unchanged history; this section records what actually shipped for the staging location and updates the same-volume claims annotated inline.

- **What changed:** model undo batches moved from `<model_root>/.lm-pending-delete/<batch_id>/` to `os.path.join(os.path.abspath(model_dir), ".lm-pending-delete", batch_id)` — a sibling of the model artifacts, INSIDE the deleted model's own folder. Business-path (abspath, never realpath) semantics preserved.
- **(a) Same-device guarantee:** because the staging dir and the artifacts share the same parent directory, stage/undo renames never cross devices — EXDEV is impossible by construction, even when the business path traverses nested symlinks to other volumes (the old "under the containing root" argument only covered plain directories; a symlinked subdir between root and model could silently resolve to another volume). EXDEV remains possible only for cross-volume MERGES (loser files moved into the winner's batch dir), which still falls back to the `batch_ids` array contract.
- **(b) Accepted edge:** if the user manually deletes the model's whole FOLDER during the 30s window, that batch is destroyed with it (it lived inside the folder) — a subsequent undo returns 404. Rare and accepted.
- **(c) Batch registry:** batch discovery uses an in-memory registry (`_known_batch_dirs`) instead of walking every root; the startup reconciliation scan (`purge_expired(scan_roots=True)`) re-populates it after restarts and catches crash leftovers, so restart behavior is unchanged.
- **(d) Recipes unchanged:** recipe batches keep the old fixed settings-dir staging (copy-based, still cross-volume-capable via the `_restore_file` EXDEV fallback).
