diff --git a/.omo/drafts/undo-delete-staging.md b/.omo/drafts/undo-delete-staging.md new file mode 100644 index 00000000..8d954ab5 --- /dev/null +++ b/.omo/drafts/undo-delete-staging.md @@ -0,0 +1,202 @@ +--- +slug: undo-delete-staging +status: drafting +intent: clear +review_required: false +pending-action: write .omo/plans/undo-delete-staging.md +approach: "Option B: delayed physical deletion with Undo. Backend: same-volume rename to per-root staging dir (.lm-pending-delete/) [updated 2026-08: model staging moved to a SIBLING dir inside each deleted model's own folder — see 'Symlink fix (2026-08)' under Decisions] + manifest JSON (batch_id, expires_at, staged->original map) + purge (30s TTL timer + startup sweep + opportunistic) + undo-delete endpoint + settings toggle 'skip undo'. Small files (recipes: JSON+preview) copy to global staging under settings dir instead of rename. Frontend: extend toast system with action button + 30s countdown; delete flows (single model / recipe / bulk / duplicates) consume batch_id from delete response and show Undo toast; expired undo -> 'undo expired' toast. Plus confirm-modal friction (C-friction, NO type-to-confirm): delete button delay-activation 1.5s + modal shows file size 'will free X GB' + Cancel gets initial focus. i18n keys + sync_translation_keys.py." +--- + +# Draft: undo-delete-staging + +## Components (topology ledger) + + +- backend staging module (stage/purge/undo + manifest + per-volume dir resolution) | new module, active | pending exploration: model_lifecycle_service.py delete_model / delete_model_artifacts +- delete endpoints return batch_id (model/recipe/bulk/duplicates) | active | pending exploration: handlers + response shapes +- undo-delete HTTP endpoint + route registration | active | pending exploration: route registrar pattern +- purge scheduling (30s timer + startup sweep + opportunistic) | active | pending exploration: app on_startup hooks +- settings toggle "skip undo window" | active | pending exploration: settings service read pattern +- frontend toast extension (action button + countdown) | active | pending exploration: showToast impl +- frontend delete flows consume batch_id + Undo toast | active | pending exploration: call sites +- confirm-modal friction (delay-activate + size display + cancel focus) | active | pending exploration: modal focus behavior +- i18n keys + sync_translation_keys.py | active | known + +## Open assumptions (announced defaults) + + +- Undo window TTL = 30s | 30s balances space-freeing intent vs accident recovery | yes (constant) +- Staging dir name: `.lm-pending-delete/` under each model root; recipes: `{settings_dir}/.lm-pending-delete/` | hidden, same-volume [updated 2026-08: same-volume is now guaranteed by sibling staging inside the model's own folder, not by the root location], consistent | yes +- Staging failure falls back to existing hard delete | user intent is delete; staging is best-effort; hard delete likely fails identically under same conditions | yes +- Purge on startup uses expires_at (not purge-all) so a <30s restart with live tab can still undo | robust, matches client-side timer | yes +- Settings toggle label: "Delete permanently immediately (skip undo window)" | power users freeing space | yes +- C-friction: delete button enabled after 1.5s + modal shows freed size; NO type-to-confirm (user vetoed) | user explicitly rejected type-to-confirm | n/a +- Bulk/duplicates delete: one batch id for whole action, one undo restores all | simplest consistent semantics | yes + +## Findings (cited - path:lines) + +### Backend +- `delete_model_artifacts` (py/services/model_lifecycle_service.py:19-48) = physical delete via os.remove; patterns: main file + `{name}.metadata.json` + PREVIEW_EXTENSIONS (py/utils/constants.py:22-37). ALSO called by ModelScanner.bulk_delete_models (py/services/model_scanner.py:2221) - single swap point covers bulk models. +- `ModelLifecycleService.delete_model` (model_lifecycle_service.py:101-154): fetches `cached_entry` (111-116) - SNAPSHOT available for cache restore; after delete: cache.raw_data removal + resort + bump_cache_version (136-143), `_hash_index.remove_by_path` (145-146), `_sync_update_for_model` (148; update-service only, no recipe JSON rewrites - recipe refs are hash-based, re-resolve on restore), `_persist_current_cache` (150-152), returns `{"success": True, "deleted_files": [...]}` (154). +- Handler `delete_model` (py/routes/handlers/model_handlers.py:478-492): POST /api/lm/{prefix}/delete; response passthrough; `_broadcast_models_changed()` (57-74) after success; 400 `{"success":false,"error"}`; 500 plain text. +- Recipe delete: handler (recipe_handlers.py:1422-1438) DELETE /api/lm/recipe/{recipe_id} -> persistence_service.delete_recipe (py/services/recipes/persistence_service.py:193-209): os.remove(recipe_json_path) + os.remove(image_path) (204-206), recipe_scanner.remove_recipe (208), returns `{"success": true, "message": ...}`. PersistenceResult dataclass (20-25). +- Bulk models: POST /api/lm/{prefix}/bulk-delete (model_route_registrar.py:39) -> handler (model_handlers.py:974-994) -> lifecycle_service.bulk_delete_models (model_lifecycle_service.py:308-318) -> scanner.bulk_delete_models (model_scanner.py:2181-2269) which calls delete_model_artifacts per file (2221) + `_batch_update_cache_for_deleted_models` (2271-2335); response `{"success","status","total_deleted","total_attempted","cache_updated","results"}` (2254-2269). +- Bulk recipes: POST /api/lm/recipes/bulk-delete (recipe_route_registrar.py:50) -> handler (recipe_handlers.py:1554-1573) -> persistence_service.bulk_delete (persistence_service.py:439-482): per-id os.remove x2 (464-466), recipe_scanner.bulk_remove (472); response `{"success","deleted","failed","total_deleted","total_failed"}` (474-482). +- Duplicates: NO dedicated delete endpoints (find-only: GET /api/lm/{prefix}/find-duplicates model_route_registrar.py:59, GET /api/lm/recipes/find-duplicates recipe_route_registrar.py:49). Duplicate deletion reuses bulk-delete endpoints. +- Startup hooks: lora_manager.py:183-187 `app.on_startup.append(lambda app: cls._initialize_services())` (ComfyUI mode, app = PromptServer.instance.app at :78); standalone.py:370-374 same (StandaloneLoraManager.add_routes). Background tasks: `asyncio.create_task(name=...)` (lora_manager.py:224-239; recipe_handlers.py:793). Singleton+asyncio.Lock pattern: model_scanner.py:40-63. +- Settings: DEFAULT_SETTINGS (py/services/settings_manager.py:57-119), `get(key, default)` (1390-1392), get_settings_manager() (2215-2228), reset_settings_manager() (2231). Typed-bool getter example: get_skip_previously_downloaded_model_versions (1253-1262). Handlers: base_model_routes.py:70, base_recipe_routes.py:54. +- Model roots: ModelScanner.get_model_roots base NotImplementedError (model_scanner.py:1073-1075); impls lora_scanner.py:31-45, checkpoint_scanner.py:428-441, embedding_scanner.py:24-36. `_find_root_for_file(file_path)` (model_scanner.py:1108-1124) returns containing root - for per-root staging dir computation [updated 2026-08: staging no longer uses the containing root; batches are siblings inside the model's own folder]. Business-path rule (AGENTS.md): use os.path.abspath, never realpath, for staging/undo routing. +- Cache restore methods: ModelCache has raw_data + resort (conftest mocks: tests/conftest.py:144-154); ModelHashIndex.add_entry(sha256, file_path, autov3) (py/services/model_hash_index.py:16); RecipeScanner.add_recipe(recipe_data) (recipe_scanner.py:2136) -> recipe_cache.add_recipe (recipe_cache.py:64). No single-file incremental model rescan - use snapshot restore instead of rescan. +- Route registrar: model_route_registrar.py:177 add_route(method, path, handler), :180 add_prefixed_route - undo endpoint can be a non-prefixed route via add_route. +- Tests: tests/services/test_model_lifecycle_service.py (inline tmp_path files, per-test stub scanners ScannerForDelete/VersionAwareScanner etc); conftest MockScanner/MockCache/MockHashIndex (tests/conftest.py:134-212); integration fixtures tests/integration/conftest.py; lifecycle hook tests tests/routes/test_lora_manager_lifecycle.py:177-178, tests/standalone/test_standalone_server.py:83-84. + +### Frontend +- 5 delete call sites: + a) Single model: static/js/utils/modalUtils.js confirmDelete (27-42) -> getModelApiClient().deleteModel(path); ignores return. + b) Recipe single: static/js/components/RecipeCard.js confirmDeleteRecipe (405-449) - RAW fetch DELETE /api/lm/recipe/{id}, checks only response.ok, showToast toast.recipes.deletedSuccessfully, state.virtualScroller.removeItemByFilePath. + c) Bulk: static/js/managers/BulkManager.js confirmBulkDelete (633-672) -> getActiveApiClient() (134-142) -> bulkDeleteModels(filePaths); reads result.cancelled/success/deleted_count/error. + d) Recipe duplicates: static/js/components/DuplicatesManager.js confirmDeleteDuplicates (457-494) - RAW fetch POST /api/lm/recipes/bulk-delete, reads data.success/data.total_deleted, exitDuplicateMode(). + e) Model duplicates: static/js/components/ModelDuplicatesManager.js confirmDeleteDuplicates (710-776) - RAW fetch POST /api/lm/{type}/bulk-delete, reads data.total_deleted, then resetAndReload(true) + find-duplicates re-check. + Bonus: static/js/components/shared/ModelVersionsTab.js:1136-1144 client.deleteModel (ignores return). +- API clients: BaseModelApiClient.deleteModel (static/js/api/baseModelApi.js:184-216) returns true/false, shows its own toasts, does removeItemByFilePath inside; bulkDeleteModels (1591-1642) returns {success, deleted_count, failed_count, errors} or {success:false, cancelled:true}; RecipeSidebarApiClient.bulkDeleteModels (recipeApi.js:623-664) returns {success, deleted_count: total_deleted, ...}. Endpoint map apiConfig.js:56,64. +- Toast: showToast(key, params={}, type='info', fallback=null) (static/js/utils/uiHelpers.js:136-193) - textContent only, NO action/button support; durations 2000/5000ms; CSS static/css/components/toast.css (.toast flex gap:12px - button can be added). Closest action pattern: bannerService.registerBanner actions array + onRegister (static/js/managers/BannerService.js; used uiHelpers.js:18-57). +- i18n: locales/en.json delete keys (1303-1314 bulkDelete, 1945-1948 recipes, 1987-1991 models, 2124-2130 duplicates, 2166-2170 toast.api); t()/interpolate (static/js/i18n/index.js:193-248); translate wrapper (utils/i18nHelpers.js:13-23); sync script scripts/sync_translation_keys.py (en reference, [TODO: Translate] placeholders). +- Refresh after undo: recipes -> window.recipeManager.loadRecipes(true) (recipes.js:359; used by FilterManager.js:752 etc) or refreshRecipes (recipeApi.js:308); models -> resetAndReload(true) from modelApiFactory (used by ModelDuplicatesManager.js:740). +- Size for modal: card.dataset.file_size (ModelCard.js:467), formatFileSize (ModelModal.js:615). +- Tests: tests/frontend/utils/uiHelpers.dom.test.js (toast), api/recipeApi.bulk.test.js, components/duplicatesManager.test.js, components/modelDuplicatesManager.test.js, pages/*Page.test.js, i18n tests tests/i18n/test_i18n.py. + +## Decisions (with rationale) + +1. Same-volume rename staging for model files (atomic, no copy cost for multi-GB files); cross-volume rename forbidden. [CORRECTED 2026-08: "same-volume because under the containing root" was only true for plain directories — nested symlinked subdirs could cross volumes. Superseded by sibling staging: `.lm-pending-delete//` inside the deleted model's own folder makes stage/undo same-device by construction; see "Symlink fix (2026-08)" below.] +2. Copy-to-global-staging for recipes (small files; avoids recipe JSON vs preview image cross-volume problem). +3. Manifest JSON files are the only state - no DB changes. Manifest includes model cached_entry snapshot for exact cache restore (no rescan needed). +4. Undo endpoint returns restored paths; expired batch -> 404-style error -> frontend 'undo expired' toast. +5. Skip-undo setting honored server-side (no batch_id in response -> no undo toast client-side). +6. Staging failure falls back to existing hard delete (best-effort undo, never blocks delete). +7. Undo window TTL = 30s constant (PENDING_DELETE_TTL_SECONDS); startup sweep uses expires_at (survives restart; browser-tab timer survives). +8. Purge triple-trigger: per-batch asyncio timer task + on_startup sweep + opportunistic purge at each stage/undo. +9. Frontend: new showActionToast (keep showToast signature untouched; extract shared createToastElement/appendToast internals); undo click -> shared handleUndoDelete(batchId, refreshFn); full list refresh after undo (recipes: window.recipeManager.loadRecipes(true); models: resetAndReload(true)). +10. C-friction wave (NO type-to-confirm - user vetoed): delete buttons delay-activate 1.5s after modal open, initial focus on Cancel, model delete modal gains "permanently deleted from disk" warning + file size display (card.dataset.file_size + formatFileSize). +11. Model cache restore on undo: append snapshot to cache.raw_data (dedupe by file_path) + resort + bump_cache_version + _persist_current_cache + _hash_index.add_entry + _broadcast_models_changed. Recipe restore: copy back files + recipe_scanner.add_recipe(recipe_data loaded from restored JSON). + +### Symlink fix (2026-08) + +Post-execution addendum (plan `.omo/plans/undo-delete-symlink-fix.md`, commits 5fd4946b / 0c00ee22): + +12. Model staging moved from `/.lm-pending-delete//` to `/.lm-pending-delete//` (sibling of the model artifacts, inside the deleted model's own folder). Stage/undo renames are same-device BY CONSTRUCTION — EXDEV is impossible even when the business path traverses nested symlinks to other volumes (the decision-1 "containing root" guarantee covered only plain directories). EXDEV remains possible only for cross-volume merges, which keep the batch_ids-array fallback. Accepted edge: deleting the model's whole FOLDER during the 30s window destroys that batch (undo returns 404). Batch discovery uses an in-memory registry (`_known_batch_dirs`) with a startup reconciliation scan (`purge_expired(scan_roots=True)`) covering restarts and crash leftovers. Recipe batches unchanged (copy-based settings-dir staging with the `_restore_file` EXDEV fallback). + +## Scope IN + +- Model single delete (model_handlers delete_model / model_lifecycle_service) +- Recipe delete (recipe_handlers delete_recipe / persistence_service) +- Bulk delete (models scanner + recipes persistence) + duplicates (reuse bulk endpoints) +- Undo endpoint POST /api/lm/undo-delete (models + recipes, one batch space) +- Purge: timer + startup sweep + opportunistic +- Settings toggle delete_undo_enabled + settings page checkbox +- Frontend: showActionToast + all 5 delete flows + shared undo handler +- C-friction modal changes (delay-activate + cancel focus + warning copy + size display) +- i18n keys + sync_translation_keys.py +- Backend + frontend tests + +## Scope OUT (Must NOT have) + +- NO type-to-confirm / hold-to-confirm friction (user vetoed) +- NO OS trash integration (send2trash) in this iteration +- NO persistent recycle-bin UI (no trash browsing page) +- NO changes to exclude/unexclude flow +- NO DB migrations +- NO new dependencies (no send2trash) +- NO changes to download flows +- NO recipe-JSON rewriting on model undo (hash-based refs re-resolve themselves) + +## Open questions + +None - all implementation details resolved by exploration. Design decisions settled in conversation (B+C, no type-to-confirm). + +## Approval gate +status: approved + + +## Review round state (ulw-plan-review-round-state-contract) +```json +{ + "transition": "replace", + "phase": "review_round_initialized", + "applies_when": ["retry_after_plan_change"], + "atomic": true, + "review_required": true, + "plan_path": ".omo/plans/undo-delete-staging.md", + "plan_sha256": "8cf7c9be38a76d8ef1fb832aba043d6d7e82b60465b1bf28c6eafa7045117adc", + "review_round_id": "rr-undo-del-20260811-006", + "round_status": "active", + "pending-action": "review .omo/plans/undo-delete-staging.md", + "review": { + "momus": { "status": "pending", "workspace_root": "/mnt/data/reinstall-backup-2026-04-12/data/workspace/ComfyUI/custom_nodes/ComfyUI-Lora-Manager", "runtime_home": null, "target": ".omo/plans/undo-delete-staging.md", "round_id": "rr-undo-del-20260811-006", "plan_sha256": "8cf7c9be38a76d8ef1fb832aba043d6d7e82b60465b1bf28c6eafa7045117adc", "launch_id": null, "session": null, "result": null }, + "independent": { "status": "pending", "workspace_root": "/mnt/data/reinstall-backup-2026-04-12/data/workspace/ComfyUI/custom_nodes/ComfyUI-Lora-Manager", "runtime_home": null, "target": ".omo/plans/undo-delete-staging.md", "round_id": "rr-undo-del-20260811-006", "plan_sha256": "8cf7c9be38a76d8ef1fb832aba043d6d7e82b60465b1bf28c6eafa7045117adc", "launch_id": null, "session": null, "result": null } + } +} +``` + +## Review results + fix/retry ledger + +### Round 1 (rr-undo-del-20260811-001, plan sha256 6c52bf99...) +- momus: APPROVE (non-blocking notes: todo1+7 duplicate DEFAULT_SETTINGS key -> fixed todo 7 to verify-only; "batch_ids" plural in todos 8/9 acceptance -> fixed; purge OSError note -> folded into todo 1 purge semantics) +- independent (oracle): CHANGES_REQUESTED + - BLOCKING S1: scanner walks would index .lm-pending-delete staged files as ghost entries -> fixed: todo 1 now mandates scanner walk exclusion at model_scanner.py:706/:867/:1404/_process_model_file + acceptance (o) scanner-visibility test + - BLOCKING S2: manifest lacks model_type, undo could restore into wrong cache/hash index -> fixed: manifest now carries model_type + todo 5 resolves per-type scanner via registrar pattern + acceptance (b) checkpoint-batch test + - S3 merged-batch expires_at re-anchor -> fixed: merge_batches re-anchors now+TTL in todo 1 + todo 3/4 assertions + - S4 manifest-less dir policy -> fixed: quarantine to .orphaned, never delete (todo 1 + acceptance g) + - S5 partial-undo retry semantics -> fixed: per-entry restored flag write-through + retry test (acceptance e) + - S6 purge locked-file failure semantics -> fixed: skip file, keep batch, never rmtree past errors (todo 1 + acceptance i) + - T8 undo-after-restart test -> fixed: todo 5 acceptance (f) + - T9 recipe undo -> re-delete test -> fixed: todo 5 acceptance (h) + - T7 rescan-stale-entry test -> fixed: todo 5 acceptance (g) + - Route registration pinned to shared routes class per mode (NOT per-model-type registrar which registers 3x) -> fixed: todo 5 now creates py/routes/pending_delete_routes.py registered once in lora_manager.py:170-172 + standalone.py:356-358 + duplicate-route test (e) + - Version-index staleness on single-delete undo -> fixed: todo 5 follows bulk cache-update pattern incl. rebuild_version_index (model_scanner.py:2324) + - Cancelled-bulk batch_id frontend handling -> fixed: todo 9 shows action toast on cancelled+staged-subset + - Single-instance assumption -> added to Scope OUT + - Occupied-refusal loss UX -> accepted-intent documented in success criteria + modal copy + +### Round 2 (rr-undo-del-20260811-002, plan sha256 f3d52235...) +- momus: APPROVE (all 12 round-1 fixes verified present; zero dead references; non-blocking nits only) +- independent (oracle): CHANGES_REQUESTED + - BLOCK-1: merge_batches file-movement semantics unspecified (silent data-loss vector) -> fixed: todo 1 now specifies move-into-winner-dir + entry re-point + loser-dirs-removed-only-when-empty + abort-on-move-failure (all batches intact) + merge inside service lock + acceptance (k) file-survival assertions + acceptance (l) merge-failure abort test + - BLOCK-2: same-file parallel edits within waves (todo 5 vs 6 on lora_manager.py; todo 8 vs 9 on baseModelApi.js) -> fixed: waves/matrix now serialize 5->6 and 8->9 with explicit reasons; matrix updated + - Recommended: checkpoint_scanner.py:331 exclusion -> fixed (todo 1 + acceptance p); S5 pre-check skips restored:true entries -> fixed (todo 1); _tags_count restore on undo -> fixed (todo 5 + acceptance j); undo-blind flows documented (ModelVersionsTab + misc_handlers:2456) -> fixed (todo 8 note + Scope OUT); merge-failure no-merge fallback contract (batch_ids array) -> fixed (todos 3/4/9) + +### Round 3 (rr-undo-del-20260811-003, plan sha256 8f2dfd46...) +- momus: APPROVE (all round-2 fixes verified present + spot-checked refs; no new contradictions) +- independent (oracle): CHANGES_REQUESTED + - BLOCKING A: merged batches never timer-purged after re-anchor (winner's original timer no-ops at old expiry; no fresh timer for re-anchored expiry; idle server -> merged batch lingers, violating "30s purge" success criterion; affects EVERY bulk delete) -> fixed: todo 1 merge_batches now ARMS A FRESH PURGE TIMER for the winner with re-anchored expiry + acceptance (q) fresh-timer test + purge_expired must enumerate ALL scanner types' roots (explicit in todo 1) + - BLOCKING B: dependency matrix contradicted same-file policy for todos 8/9<->11 (5 shared files) and 12<->11 -> fixed: todo 11 now "Blocked by: 8, 9 (same files...)"; todo 12 blocked by 11 (sync after 11); wave text updated (11, then 12 AFTER 11); "Can parallelize with" columns corrected + - BLOCKING C: frontend batch_ids sequential-undo fallback has NO test + merge->undo loser-restore + merge->purge assertions missing -> fixed: todo 9 acceptance now tests the batch_ids fallback path; todo 1 acceptance now has (k2)/(k3) + - Notes folded: sub-second toast-tail expiry race accepted; EXDEV fallback = NORMAL path for cross-volume bulks [annotated 2026-08: after the sibling-staging fix, EXDEV can only arise during cross-volume MERGES, never during single stage/undo renames] + +### Round 4 (rr-undo-del-20260811-004, plan sha256 179e7ff7...) +- momus: APPROVE (round-3 fixes verified; one non-blocking nit: todo 11 inline "Blocked by: —" stale -> fixed to "8, 9") +- independent (oracle): CHANGES_REQUESTED + - BLOCKING GAP-1 (NEW, introduced by round-3 fix): todo 8 handleUndoDelete always-refresh/always-toast contract contradicted todo 9's sequential loop "exactly ONE final refresh" -> fixed: handleUndoDelete(batchId, refreshFn, {showToast, refresh}) suppression options; todo 9 loop uses suppressed calls + one final refresh/toast; acceptance extended (loop failure mid-way -> stop + error toast + no final refresh; 404 body discrimination expired vs occupied) + - BLOCKING GAP-2: no cross-type purge enumeration test -> fixed: todo 1 acceptance (r) purges expired batches across lora root + checkpoint root + recipe staging dir in one call + - Non-blocking folded: GAP-3 404-copy discrimination -> fixed in todo 8 (d); GAP-4 merge partial-failure rollback direction (move back + restore manifests, extended (l) asserts sequential constituent undo still restores everything) -> fixed in todo 1; GAP-5 post-restart timer-loss residual gap documented -> fixed in todo 6; GAP-6 usage_stats.py:424 walk added to exclusion mandate + todo 5 acceptance (k) embeddings undo test + +### Round 5 (rr-undo-del-20260811-005, plan sha256 dfaa39ea...) +- momus: APPROVE (all round-4 fixes verified; no new contradictions) +- independent (oracle): CHANGES_REQUESTED + - BLOCK-1: lock-ordering deadlock ambiguity (asyncio.Lock not re-entrant: opportunistic purge_expired called while stage/undo hold the lock would deadlock on first use) -> fixed: todo 1 now has explicit LOCK HIERARCHY (lock acquired ONLY by stage/merge/undo/purge_batch; purge_expired is lock-free and must be called BEFORE lock acquisition); todo 6 (c) updated with the same rule + acceptance (u) lock-no-deadlock test + - BLOCK-2: purge edge semantics unspecified -> fixed: purge_batch treats missing staged files (partially-restored batches) as already-purged (FileNotFoundError silent no-op); sweep skips `.orphaned`-suffixed dirs (quarantine is terminal); acceptance (s) partially-restored purge + (t) quarantine-terminal tests + - Non-blocking folded: todo 2/3 test-file collision -> todo 3's bulk tests moved to tests/services/test_model_scanner.py; todo 9 (d) DuplicatesManager refreshFn stated explicitly (recipes loadRecipes / models resetAndReload); modal-copy + bulk-count trade-offs acknowledged in success criteria; acceptance (r) extended with embeddings root + +### Round 6 (rr-undo-del-20260811-006, plan sha256 8cf7c9be...) +- momus: APPROVE (all round-5 fixes verified; no new contradictions; references verified) +- independent (oracle): APPROVE — no blocking issues; all round-5 items fixed with working, tested solutions; no new race/data-loss/consistency defects +- Deferred optional improvements (non-blocking, recorded for executor awareness; plan file left untouched to preserve the approved digest): + 1. Tag-count asymmetry: single delete_model never decrements _tags_count (lifecycle 101-154), bulk does (scanner 2297-2303); undo re-increment is exact for bulk, over-counts for single until rescan (cosmetic, self-healing). Optional fix riding in todo 2: decrement tags in the single-delete path to mirror bulk. + 2. Todo 5 factual nit: ModelCache.resort() already rebuilds the version index — explicit rebuild in undo is belt-and-braces, no action needed. + 3. Todo 8 premise nit: ModelVersionsTab call ignores deleteModel's return entirely — nothing breaks, no adaptation needed. + 4. Todo 3's pytest command includes test_model_lifecycle_service.py which todo 2 edits in the same wave — run that file's tests after todo 2 lands. + 5. merge_batches with a missing/quarantined constituent id: any sane fallback (abort -> batch_ids, or skip missing) acceptable — files stay staged either way. + +## Review lifecycle +- rounds: 6 (rr-undo-del-20260811-001..006); final round both lanes APPROVE +- final live-plan validation: sha256 = 8cf7c9be38a76d8ef1fb832aba043d6d7e82b60465b1bf28c6eafa7045117adc — MATCHES approved round-6 digest +- status: APPROVED — ready for execution handoff ($start-work undo-delete-staging) diff --git a/.omo/plans/undo-delete-staging.md b/.omo/plans/undo-delete-staging.md new file mode 100644 index 00000000..b2a4fc7d --- /dev/null +++ b/.omo/plans/undo-delete-staging.md @@ -0,0 +1,216 @@ +# undo-delete-staging - Work Plan + +## TL;DR (For humans) + +**What you'll get:** Deleting a model or recipe becomes recoverable: the file disappears right away, but a toast with an "Undo" button (30-second countdown) lets you bring it back — the file is only physically erased after the window closes. You can also turn this off in settings to delete instantly as before. + +**Why this approach:** Files are moved (models) or copied (recipes) to a hidden staging folder on the same drive instead of being deleted — a rename is instant even for multi-gigabyte files, so freeing disk space still happens within 30 seconds. Deleting then looks and feels the same as today; only the accident-recovery net is added. + +**What it will NOT do:** No typing-to-confirm, no system recycle bin, no "trash folder" page to manage, no changes to how anything else works (exclude, downloads, metadata). + +**Effort:** Medium +**Risk:** Medium - the riskiest parts are Windows file-locking during the staged purge and keeping the library cache consistent across restore; both have explicit fallback and test coverage. +**Decisions to sanity-check:** (1) 30-second undo window — long enough to catch a misclick, short enough that disk space is barely delayed; (2) undo for bulk/duplicate deletes restores everything with one click, not per-file; (3) if moving files to staging fails for any reason, the app silently falls back to the old permanent delete rather than blocking you. + +Your next move: approve, or run a high-accuracy review first. + +--- + +> TL;DR (machine): Medium effort/risk; 12 todos in 5 waves + final verification; same-volume rename staging [updated 2026-08: now guaranteed by SIBLING staging inside the model's own folder — see addendum at end] + 30s undo toast + triple-trigger purge + skip-toggle; no new deps, no DB changes. + +## Scope +### Must have +- Backend staging service: model files renamed to same-volume per-root staging dir (`.lm-pending-delete/` under each model root) [updated 2026-08: staging moved to a SIBLING dir inside the deleted model's own folder — see addendum at end]; recipe JSON + preview image copied to global staging (`{settings_dir}/.lm-pending-delete/`). Manifest JSON (batch_id, kind, expires_at, entries[staged->original], model cached_entry snapshot) is the only state. +- Delete responses (single model, single recipe, bulk models, bulk recipes — hence duplicates flows) gain an optional `batch_id` field when staged. +- `POST /api/lm/undo-delete` endpoint: restores staged files to original paths + restores cache entries exactly (model: raw_data append + resort + bump_cache_version + _persist_current_cache + hash_index.add_entry; recipe: recipe_scanner.add_recipe) + `_broadcast_models_changed()`. Expired batch -> 404 `{"success": false, "error": ...}`. +- Purge triple-trigger: per-batch asyncio timer task (30s TTL), on_startup sweep inside `_initialize_services` (covers both ComfyUI + standalone since StandaloneLoraManager reuses it), opportunistic purge at each stage/undo call. Purge only batches with `expires_at < now`. +- Settings toggle `delete_undo_enabled` (default `true`) in DEFAULT_SETTINGS + checkbox in settings_modal.html; when disabled -> current hard-delete behavior, no batch_id, no undo toast. +- Staging failure (rename/copy OSError) -> fall back to existing hard delete (delete_model_artifacts / os.remove path), no batch_id. +- Frontend: new `showActionToast` (showToast signature untouched; extract shared createToastElement/appendToast internals) with action button + 30s countdown text; all 5 delete flows consume batch_id and show Undo toast; shared `handleUndoDelete(batchId, refreshFn)`; full list refresh after undo (recipes: `window.recipeManager.loadRecipes(true)`; models: `resetAndReload(true)` from modelApiFactory). +- C-friction (NO type-to-confirm): delete-confirm buttons delay-activate 1.5s after modal opens; initial focus on Cancel; model delete modal gains "will be permanently deleted from disk" warning + freed-size display (card.dataset.file_size + formatFileSize). +- i18n keys in locales/en.json + `python scripts/sync_translation_keys.py` run. +- Backend pytest + frontend vitest coverage for all of the above. + +### Must NOT have (guardrails, anti-slop, scope boundaries) +- NO type-to-confirm / hold-to-confirm friction anywhere (user vetoed). +- NO OS trash integration (send2trash) — no new dependencies at all. +- NO persistent recycle-bin UI (no trash browsing page, no trash listing endpoint). +- NO changes to exclude/unexclude flows, download flows, metadata editing, or recipe-JSON rewriting on model undo (recipe refs are hash-based and re-resolve on restore). +- NO DB migrations, NO schema changes to existing SQLite caches. +- NO `os.path.realpath` for staging/undo routing — business paths only (AGENTS.md rule); realpath only where the scanner itself already uses it. +- NO changes to `showToast` signature or its existing call sites' behavior (no regressions in the 48 files using it). +- NO silent semantics: when undo is skipped (setting off or staging failure), the delete must behave exactly as today — no partial staging, no orphaned files. +- NO persistent undo affordance across browser refresh (the toast is ephemeral; a page reload within the window loses the Undo button — the staged batch still expires after 30s). +- NO per-file undo within a bulk/duplicates action (one batch = one undo for the whole action). +- NO partial-batch purge: purge and undo operate on whole batches only; a batch is never half-restored (except the mid-undo-failure retry case defined in todo 1, where per-entry restored flags make the remainder retry-able). +- NO multi-instance support: a single server instance is assumed (two servers sharing the same model roots/settings dir is out of scope; cross-process coordination is not implemented). +- NO deletion of manifest-less or corrupted staging batches — quarantine only (`.orphaned`), never rmtree past per-file errors. +- NO ghost library entries: the model AND checkpoint scanners must never index files under `.lm-pending-delete/` (exclusion implemented in todo 1 for model_scanner.py walk sites + checkpoint_scanner._find_pending_models_from_filesystem). +- NO merge data-loss: merge_batches moves (never drops) staged files; loser batch dirs are removed only when empty; merge failure aborts with all batches intact (falls back to a `batch_ids` array in the response). +- UNDO-BLIND DELETE FLOWS (documented scope edges, unchanged behavior): the versions-tab delete (ModelVersionsTab.js:1136-1144) is staged server-side but shows NO undo toast (own success toast, out of scope); the delete-by-version endpoint (py/routes/handlers/misc_handlers.py:2456) hard-deletes via delete_model_artifacts bypassing staging entirely (pre-existing behavior, unchanged). + +## Verification strategy +> Zero human intervention - all verification is agent-executed. +- Test decision: TDD for backend (pytest — service + endpoint tests written with the implementation); tests-after for frontend flows (vitest) where flows are integration-heavy. +- Commands: `pytest tests/services/test_pending_delete_service.py tests/services/test_model_lifecycle_service.py tests/services/test_recipe_persistence.py tests/routes/test_pending_delete_routes.py -q`; `npm test`; `python scripts/sync_translation_keys.py --dry-run` clean. +- Evidence: .omo/evidence/undo-delete-staging/task--undo-delete-staging. per todo (attemptDir = currentAttemptDir from 'omo ulw-loop status --json'; outside ulw-loop use .omo/evidence/) + +## Execution strategy +### Parallel execution waves +- Wave 1: todo 1 (staging service core + tests) — everything depends on it. +- Wave 2: todos 2, 3, 4 in parallel (model single / model bulk / recipe single+bulk wiring — all depend only on 1; different files: model_lifecycle_service.py / model_scanner.py / persistence_service.py). +- Wave 3: todos 5, 7, 10 in parallel, then 6 AFTER 5 (todo 5 and todo 6 both edit py/lora_manager.py — same-file edits must be serialized: 5 first for route registration, 6 second for the startup sweep hook; 6 additionally depends on 1). +- Wave 4: todo 8, then 9 AFTER 8 (todo 8 and todo 9 both edit static/js/api/baseModelApi.js — same-file edits must be serialized: 8 (deleteModel) first, 9 (bulkDeleteModels) second; 9 additionally depends on 3,4,5,7,10). +- Wave 5: todo 11, then 12 AFTER 11 (todo 11 and todo 12 both edit locales/en.json — 12's final sync step must run after 11's strings exist; 11 depends on 8, 9 because it edits the SAME frontend files: modalUtils.js, RecipeCard.js, BulkManager.js, DuplicatesManager.js, ModelDuplicatesManager.js — so 11 runs after Wave 4 completes; 12 additionally depends on 8,9,10,11 strings). +- Wave 6: Final verification wave (F1-F4). + +### Dependency matrix +| Todo | Depends on | Blocks | Can parallelize with | +| --- | --- | --- | --- | +| 1. PendingDeleteService + tests | — | 2,3,4,5,6,7 | — | +| 2. Single model delete staging + tests | 1 | 5, 8 | 3, 4 | +| 3. Bulk model delete staging + tests | 1 | 5, 9 | 2, 4 | +| 4. Recipe delete staging (single+bulk) + tests | 1 | 5, 8, 9 | 2, 3 | +| 5. Undo endpoint + cache restore + tests | 1, 2, 3, 4 | 6, 8, 9 | 7, 10 | +| 6. Purge scheduling (timer+startup+opportunistic) + tests | 1, 5 (same file lora_manager.py) | — | 7, 10 | +| 7. Settings toggle + tests | 1 | 9 | 5, 6, 10 | +| 8. Frontend single-delete undo flows + tests | 2, 4, 5, 10 | 9, 11, 12 | 7 | +| 9. Frontend bulk+duplicates undo flows + tests | 3, 4, 5, 7, 8 (same file baseModelApi.js), 10 | 11, 12 | — | +| 10. showActionToast + CSS + tests | — | 8, 9 | 1..7 | +| 11. C-friction modal changes + tests | 8, 9 (same files: modalUtils.js, RecipeCard.js, BulkManager.js, DuplicatesManager.js, ModelDuplicatesManager.js) | 12 | — | +| 12. i18n keys + sync + tests | 8, 9, 10, 11 (12's sync step runs AFTER 11) | — | — | + +## Todos +> Implementation + Test = ONE todo. Never separate. + +- [x] 1. Create py/services/pending_delete_service.py — staging service (stage/undo/purge/manifest) + TTL constant + settings default + unit tests + What to do: New module `py/services/pending_delete_service.py`. Class `PendingDeleteService` following the house singleton+asyncio.Lock pattern (see py/services/model_scanner.py:40-63). Service-level `asyncio.Lock` serializing stage/merge/undo/purge_batch operations (prevents purge-timer vs undo races AND merge-vs-purge races). LOCK HIERARCHY (critical — asyncio.Lock is NOT re-entrant, a deadlock here breaks first use): the lock is acquired ONLY by stage_model_delete, stage_recipe_delete, merge_batches, undo, and purge_batch; `purge_expired()` itself NEVER acquires the lock — it enumerates staging dirs and delegates each batch to purge_batch (which locks); therefore the opportunistic `await self.purge_expired()` at the start of stage_*/undo MUST be called BEFORE those methods acquire the lock (lock-free section), never while holding it. Public async methods: `stage_model_delete(*, scanner, target_dir, file_name, main_extension, original_file_path, cached_entry) -> str | None` (returns batch_id or None when undo disabled / staging impossible); `stage_recipe_delete(*, recipe_json_path, image_path, recipe_data) -> str | None`; `merge_batches(batch_ids) -> str | None` — CRITICAL SEMANTICS (never drops files, never orphans from purge): merges several batches into one manifest, RE-ANCHORS `expires_at = now + PENDING_DELETE_TTL_SECONDS` at merge time, winner = first batch_id; the staged FILES of the losing batches are MOVED into the winner's batch dir (os.rename per file) and each entry's `staged` path is REWRITTEN in the merged manifest BEFORE any loser batch dir is removed; loser batch dirs are removed ONLY after they are empty; the merged manifest is written atomically (temp + os.replace); after a successful merge, ARM A FRESH PURGE TIMER for the winner with the RE-ANCHORED expiry (the winner's original timer fires at the old expiry, re-reads the later re-anchored expires_at, no-ops — WITHOUT a fresh timer the merged batch would never be purged on an idle server; the fresh timer guarantees the "30s purge" contract holds for merged batches); on PARTIAL-MOVE FAILURE (some files moved, then a move fails): MOVE the already-moved files BACK to their original batch dirs and RESTORE the per-batch manifests (revert staged paths), then return None — all original batch dirs and files intact, each constituent batch fully undoable (callers fall back to the `batch_ids` array contract: todos 3/4); `undo(batch_id) -> dict` (renames/copies files back per-entry, removes batch dir + manifest when fully restored); `purge_expired() -> int` — lock-free (see LOCK HIERARCHY); MUST enumerate ALL model roots across ALL scanner types (lora/checkpoint/embedding — iterate the per-type scanners like the route registrars do, collecting every `get_model_roots()`) PLUS the global recipe staging dir, or cross-type batches would leak; SKIPS dirs whose name ends with `.orphaned` (never re-quarantined, never re-renamed, never deleted — the quarantine is terminal); delegates each found batch to purge_batch; `purge_batch(batch_id)` — acquires the lock; treats MISSING staged files (already-restored / partially-restored batches) as already-purged: FileNotFoundError per file is a silent no-op, NOT a failure (the batch dir is removed when no staged files remain); re-reads the manifest's current expires_at and no-ops when the batch is missing / already restored / not yet expired (stale timers from merged-away or undone batches are harmless). Constants: `PENDING_DELETE_TTL_SECONDS = 30`, `PENDING_DELETE_DIR_NAME = ".lm-pending-delete"`. Staging dir computation: for models, root = `scanner._find_root_for_file(original_file_path)` (py/services/model_scanner.py:1108-1124) then `/.lm-pending-delete//`; for recipes, `/.lm-pending-delete//` (settings dir via get_settings_manager() — see py/services/settings_manager.py:2215). Manifest `manifest.json` per batch dir — write atomically via temp file + os.replace: `{"batch_id", "kind": "model"|"recipe", "model_type": <"loras"|"checkpoints"|"embeddings" — ONLY for kind=model; captured from scanner.model_type at stage time>, "state": "staged", "expires_at": int(epoch+30), "entries": [{"staged": , "original": , "restored": false}], "model_snapshot": , "recipe_snapshot": }`. The per-batch staging dir guarantees NO staged-name collisions (same filename in concurrent batches lives in different batch dirs). Model staging = os.rename per artifact (same volume guaranteed because staging dir is under the containing root) [CORRECTED 2026-08: this guarantee held only for plain directories — a nested symlinked subdir could resolve to another volume and break the rename with EXDEV; the symlink fix moved staging to `/.lm-pending-delete//`, a SIBLING of the model artifacts inside the model's own folder, so stage/undo renames are same-device BY CONSTRUCTION — see addendum at end]; SKIP artifacts that do not exist (missing metadata sidecar / preview must not trip rollback — mirror delete_model_artifacts' tolerance, enumerate then filter by exists); recipe staging = shutil.copy2 of the JSON and — ONLY IF IT EXISTS — the image (missing/shared previews are skipped; manifest entries reflect actual files); then the caller removes originals. Enumerate model artifacts EXACTLY like delete_model_artifacts (py/services/model_lifecycle_service.py:19-48): main file + `{file_name}.metadata.json` + PREVIEW_EXTENSIONS (py/utils/constants.py:22-37). Use `os.path.abspath`, never realpath, for all stored paths (AGENTS.md business-path rule — a symlinked root must not silently route staging to another volume). NOTE: merging batches from DIFFERENT roots on DIFFERENT volumes will fail os.rename with EXDEV — that is EXPECTED and fine: the merge aborts, callers fall back to the `batch_ids` array, and the frontend undoes sequentially; this fallback is the NORMAL path for cross-volume bulks, not a rare edge. [annotated 2026-08: with sibling staging, single-file stage/undo renames can no longer hit EXDEV at all; EXDEV remains possible ONLY for cross-volume MERGES (moving loser files into the winner's batch dir) — see addendum at end.] Guard: skip staging entirely when `get_settings_manager().get("delete_undo_enabled", True)` is falsy (return None). ALSO add `"delete_undo_enabled": True` to DEFAULT_SETTINGS in py/services/settings_manager.py:57-119 as part of this todo (this makes the toggle exist before any wiring todo reads it). On any OSError during staging: log warning, roll back any already-staged files of that batch (rename back), return None (caller falls back to hard delete). SCANNER EXCLUSION (critical — without it staged files appear in the library as ghost model entries): every directory walk under model roots MUST skip directories whose name equals `.lm-pending-delete`: (a) the walk sites in py/services/model_scanner.py (~:706 count_recursive, ~:867 os.walk, ~:1404 scan_recursive, plus `_process_model_file` at :1128), (b) py/services/checkpoint_scanner.py `_find_pending_models_from_filesystem` (~:331 — checkpoint metadata.json discovery), and (c) py/utils/usage_stats.py `_find_checkpoint_file_on_disk` (~:424 — walks checkpoint roots for usage-tracking lookups; staged files must not be matched there either; cosmetic but keeps the predicate uniform). Add ONE shared predicate (e.g. `_is_excluded_dir(name)`) used by all sites. `undo()` must: hold the service lock; reject expired batches (ValueError "Undo window expired"); pre-check ALL target paths EXCEPT entries already marked `restored: true` (skip those) — if any other original path exists on disk (occupied — e.g. user re-downloaded), refuse with ValueError "Target path occupied" and leave the whole batch intact (protects new files; batch stays staged until TTL); restore files PER ENTRY and write the manifest through after each successful entry restore (`restored: true`) so a mid-undo failure (locked file, permissions) leaves a retry-able state — a subsequent undo skips entries whose staged file is already gone (already restored) and finishes the rest; remove the batch dir + manifest only after ALL entries restored; after success mark manifest state "restored". `purge_expired()` scans: every model root from ALL scanner types' `get_model_roots()` (py/services/model_scanner.py:1073, impls lora_scanner.py:31-45, checkpoint_scanner.py:428-441, embedding_scanner.py:24-36) for `.lm-pending-delete/` dirs, plus the global recipe staging dir; deletes entries with expires_at < now (os.remove + os.rmdir); per-file purge failure (locked file, e.g. Windows/antivirus): skip THAT file and leave the batch dir intact — NEVER delete a batch dir or rmtree past per-file errors (the batch is retried by the next opportunistic purge; failure direction is disk-leak, not data-loss); TOLERATES malformed manifests (quarantine: rename dir to `.orphaned` and skip, do not crash the sweep); MANIFEST-LESS batch dirs (crash between rename and manifest write): treat exactly like corrupted manifests — rename to `.orphaned`, NEVER delete the staged files (they may be the only copy of the user's data). Purge tasks must re-read the manifest's current `expires_at` at fire time (not a precomputed sleep value) and no-op when the batch is missing / already restored / not yet expired — this makes stale timers from merged-away or undone batches harmless. Must NOT do: no DB writes; no new dependencies; no realpath; no deletion of non-expired batches; no websocket broadcasts (caller's job); do NOT touch the recipe FTS index or recipe cache here (undo does that via recipe_scanner.add_recipe in todo 5); do NOT delete manifest-less or corrupted batches — quarantine only; merge_batches must NEVER drop or delete a staged file (any move failure aborts the whole merge) and MUST arm the fresh purge timer after a successful merge. + Parallelization: Wave 1 | Blocked by: — | Blocks: 2,3,4,5,6,7 + References: py/services/model_lifecycle_service.py:19-48 (artifact patterns), :101-154 (delete flow), py/utils/constants.py:22-37 (PREVIEW_EXTENSIONS), py/services/model_scanner.py:40-63 (singleton pattern), :1073-1075 (get_model_roots base), :1108-1124 (_find_root_for_file), :706/:867/:1404 + _process_model_file at :1128 (walk sites needing staging-dir exclusion), py/services/checkpoint_scanner.py:323-350 (_find_pending_models_from_filesystem — 4th exclusion site, os.walk body ~:331), py/services/settings_manager.py:57-119 (DEFAULT_SETTINGS), :1390-1392 (get), :2215-2228 (get_settings_manager), tests/services/test_model_lifecycle_service.py (inline tmp_path test style), tests/conftest.py:134-212 (MockScanner/MockCache/MockHashIndex), :336-368 (singleton reset) + Acceptance criteria (agent-executable): `pytest tests/services/test_pending_delete_service.py -q` passes with tests covering: (a) stage_model_delete renames all EXISTING artifact patterns into `/.lm-pending-delete//` and writes manifest (incl. model_type + model_snapshot) with expires_at = now+30; (b) undo() restores all files to original paths and removes batch dir; (c) undo() on expired batch raises ValueError; (d) undo() when original path occupied raises ValueError and leaves batch dir + manifest intact; (e) PARTIAL-UNDO RETRY: monkeypatch os.rename to fail on the 2nd of 3 entries -> undo raises, manifest shows entry 1 restored:true / entries 2-3 restored:false; second undo (no monkeypatch) completes the rest and removes the dir; (f) purge_expired() removes only expired batches (manually rewrite expires_at in manifest to test); (g) MANIFEST-LESS dir: create batch dir with staged files but no manifest -> sweep renames to `.orphaned`, files still present, sweep completes; (h) CORRUPTED manifest: garbage manifest.json -> dir quarantined, no crash; (i) PURGE LOCKED FILE: monkeypatch os.remove to fail for one file -> sweep skips that file, batch dir remains, no exception; (j) STALE TIMER: call purge_batch on an undone/merged-away/missing batch id -> silent no-op; (k) MERGE: merge_batches of two batches -> single manifest, expires_at = now+TTL (not the earlier of the two), ALL staged files EXIST under the winner batch dir (assert each file present, byte-compare content), loser batch dirs are EMPTY before removal and removed, no file dropped; (k2) MERGE THEN UNDO: merge two batches then undo(winner) -> EVERY file (incl. the losers' files) is restored to its ORIGINAL path (assert each original path exists with matching bytes); (k3) MERGE THEN PURGE: merge two batches, rewrite expires_at to the past, purge_expired() -> winner batch dir is EMPTY (all files incl. moved ones removed) and removed; (l) MERGE MOVE FAILURE: monkeypatch os.rename to fail during merge (after 1 file already moved) -> merge returns None, already-moved file is moved BACK, ALL original batch dirs + manifests + files intact, and a subsequent sequential undo of each constituent batch still restores every file to its original path; (m) delete_undo_enabled=false -> stage returns None and nothing is created; (n) simulated OSError during staging (monkeypatch os.rename to fail on 2nd file) -> returns None, first file renamed back, no orphaned batch dir; (o) DEFAULT_SETTINGS contains delete_undo_enabled=True; (p) SCANNER EXCLUSION: build a fake tree `/.lm-pending-delete/x/model.safetensors` + `/.lm-pending-delete/x/model.metadata.json` and run the model scanner's directory walk against the root -> no cache/raw_data entry whose path contains `.lm-pending-delete`; run `_find_pending_models_from_filesystem` (checkpoint scanner) against the root -> staged metadata.json paths are NOT returned; (q) MERGE TIMER: merge two batches, spy on asyncio.create_task -> a FRESH purge task is spawned for the winner id; then simulate time passing (advance the re-anchored expires_at to the past) and call purge_batch(winner) -> files purged (proves the fresh timer contract); (r) CROSS-TYPE PURGE ENUMERATION: create an expired batch under a LORA root, an expired batch under a CHECKPOINT root, an expired batch under an EMBEDDINGS root, and an expired recipe batch under the recipe staging dir -> one purge_expired() call removes ALL FOUR (proves enumeration across scanner types + recipe dir); (s) PARTIALLY-RESTORED PURGE: batch with 2 entries where entry 1 is restored:true (staged file absent) and entry 2's staged file present -> purge_batch removes entry 2 + the batch dir with NO exception (missing file treated as already-purged); (t) QUARANTINE IS TERMINAL: after a quarantine (dir renamed to `.orphaned`), run purge_expired() a SECOND time -> the dir name is UNCHANGED (no re-rename), files still present, no exception; (u) LOCK NO-DEADLOCK: call stage_model_delete while a purge_expired() (with an expired batch) is in flight, and call undo while purge_batch is in flight -> both complete without deadlock or exception, and the expired batch is either fully purged or fully restored, never partially (exercises the LOCK HIERARCHY: purge_expired is lock-free, purge_batch locks). + QA scenarios (name the exact tool + invocation): happy: `pytest tests/services/test_pending_delete_service.py -q` -> all pass; failure: monkeypatch-based tests (e), (i), (l), (n) -> expect rollback/retry/skip/abort assertions pass; scanner exclusion (p) -> assert zero ghost entries in both scanners; merge lifecycle (k)/(k2)/(k3)/(q) -> assert no file dropped, original paths restored, purge empties merged dir, fresh timer armed. Evidence .omo/evidence/undo-delete-staging/task-1-undo-delete-staging.md + Commit: Y | `feat(delete): add pending-delete staging service with undo/purge` + +- [x] 2. Wire single model delete into staging — ModelLifecycleService.delete_model + handler passthrough + tests + What to do: In `py/services/model_lifecycle_service.py` `delete_model` (lines 101-154): replace the direct `delete_model_artifacts` call (line 132-134) with: (1) build staging via a module-level `get_pending_delete_service()` (lazy singleton — do NOT add to __init__ signature; construct inside the method or via a small accessor to keep the existing constructor contract); (2) `batch_id = await pending_delete_service.stage_model_delete(scanner=self._scanner, target_dir=target_dir, file_name=file_name, main_extension=main_extension, original_file_path=file_path, cached_entry=cached_entry)`; (3) if batch_id is None -> call existing `delete_model_artifacts` (fallback, unchanged behavior); (4) if staged -> the artifacts are already renamed away, so SKIP delete_model_artifacts; cache/hash/sync/persist logic (lines 136-152) runs identically (files are gone from original location, cache removal is still correct). Return dict becomes `{"success": True, "deleted_files": deleted_files, "batch_id": batch_id}` (batch_id null when not staged). Keep `_require_path_in_library_roots` and `_sync_update_for_model` untouched. `py/routes/handlers/model_handlers.py` delete_model (478-492) needs NO change (result passthrough already returns the dict). Must NOT do: do not change delete_model_artifacts itself (bulk todo 3 also uses it); do not touch bulk paths in this todo; do not add batch_id to the 400/500 error responses. + Parallelization: Wave 2 | Blocked by: 1 | Blocks: 5, 8 + References: py/services/model_lifecycle_service.py:101-154 (delete_model full body), :19-48 (delete_model_artifacts), py/routes/handlers/model_handlers.py:478-492 (handler), :57-74 (_broadcast_models_changed — no change needed here), tests/services/test_model_lifecycle_service.py (existing delete tests: test_delete_model_removes_gguf_file ~:470-508) + Acceptance criteria (agent-executable): `pytest tests/services/test_model_lifecycle_service.py tests/services/test_pending_delete_service.py -q` passes. New tests: delete_model with undo enabled -> file renamed to staging, response contains batch_id, cache entry removed, `_persist_calls` incremented (existing ScannerForDelete tracks it); delete_model with delete_undo_enabled=false (monkeypatch settings) -> old behavior (os.remove), batch_id is None; delete_model with staging failure -> files deleted via fallback, batch_id None, no staging dir left. + QA scenarios: happy: `pytest tests/services/test_model_lifecycle_service.py -q` -> green; failure: staging-failure test -> asserts fallback deleted files AND no `.lm-pending-delete` leftover. Evidence .omo/evidence/undo-delete-staging/task-2-undo-delete-staging.md + Commit: Y | `feat(delete): stage single model deletes for undo` + +- [x] 3. Wire bulk model delete into staging — ModelScanner.bulk_delete_models + response batch_id + tests + What to do: In `py/services/model_scanner.py` `bulk_delete_models` (lines 2181-2269): inside the per-file loop where `delete_model_artifacts` is called (line 2221), attempt staging per file BUT aggregate into ONE batch for the whole bulk action: capture each file's `cached_entry` snapshot BEFORE the cache mutation (`_batch_update_cache_for_deleted_models` runs after the loop, 2271-2335 — snapshots must be taken pre-mutation); call `pending_delete_service.stage_model_delete(...)` per file, collecting non-null batch_ids; at the end of the loop call `merge_batches(batch_ids)` (todo 1 — re-anchors expires_at to now+TTL at merge, so the merged batch outlives the frontend toast countdown; stale timers of merged-away ids are harmless per todo 1 fire-time re-check). NO-MERGE FALLBACK CONTRACT (merge_batches can return None on move failure): response then contains `"batch_ids": [constituent ids]` instead of a single `"batch_id"` (frontend todo 9 undoes them sequentially); when merge succeeds response contains `"batch_id": merged_id` and no batch_ids array. When staging returns None for a file -> existing hard delete fallback for that file only. Add the batch field(s) to the success return dict (lines 2254-2269). If the operation was cancelled mid-way (is_cancelled, line 2255): still merge whatever was staged and include the batch field(s) in the SAME success dict (status='cancelled' + batch) — the frontend (todo 9) shows the undo toast for the staged subset. Must NOT do: do not change the cancellation semantics; do not stage when the batch is cancelled before any file was processed; do not touch `_batch_update_cache_for_deleted_models`; no changes to the except-block failure return dict (2261-2263). + Parallelization: Wave 2 | Blocked by: 1 | Blocks: 5, 9 + References: py/services/model_scanner.py:2181-2269 (bulk_delete_models), :2221 (delete_model_artifacts call), :2254-2269 (return dicts incl. status='cancelled' path), :2271-2335 (_batch_update_cache_for_deleted_models), py/services/model_lifecycle_service.py:308-318 (lifecycle bulk entry — passthrough), py/services/pending_delete_service.py (todo 1 module — merge_batches + fire-time re-check + no-merge fallback) + Acceptance criteria (agent-executable): `pytest tests/services/test_model_scanner.py tests/services/test_model_lifecycle_service.py -q` passes. New tests (put the NEW bulk-staging tests in `tests/services/test_model_scanner.py`, NOT in test_model_lifecycle_service.py — todo 2 also edits that file in the same wave and the two test suites must not collide): bulk_delete_models with 2 files -> response contains ONE batch_id, both files present in that batch's staging dir, cache updated; merged manifest expires_at >= staging completion time + TTL (re-anchor assertion); merge-failure fallback (monkeypatch os.rename to fail during merge) -> response contains batch_ids array of length 2, both batches intact; with delete_undo_enabled=false -> batch_id null, files os.removed; cancelled mid-way after 1 file staged -> status='cancelled' AND batch_id present, that file undoable. + QA scenarios: happy: run the new bulk test -> green; failure: assert zero staging dirs exist when undo disabled. Evidence .omo/evidence/undo-delete-staging/task-3-undo-delete-staging.md + Commit: Y | `feat(delete): stage bulk model deletes for undo` + +- [x] 4. Wire recipe deletes into staging — persistence_service delete_recipe + bulk_delete + batch_id + tests + What to do: In `py/services/recipes/persistence_service.py`: `delete_recipe` (lines 193-209) — before the two os.remove calls (204-206), call `batch_id = await pending_delete_service.stage_recipe_delete(recipe_json_path=recipe_json_path, image_path=image_path, recipe_data=recipe_data)` (stage only existing files — missing preview image is skipped; recipe_data is stored as recipe_snapshot in the manifest); if batch_id is None -> existing behavior; if staged -> remove originals via os.remove as today (files exist in staging copy). Include `"batch_id": batch_id` in the PersistenceResult payload (line 209). `bulk_delete` (lines 439-482) — same per id, then merge into ONE batch via `merge_batches(...)` with the SAME no-merge fallback contract as todo 3 (merge failure -> `"batch_ids"` array in the payload); include the batch field(s) in the result payload (474-482). Must NOT do: do not stage the image only when the JSON stage failed (stage_recipe_delete must be atomic internally — copy JSON first, image second, roll back copies on failure, per todo 1); do not change `recipe_scanner.remove_recipe` / `bulk_remove` calls. + Parallelization: Wave 2 | Blocked by: 1 | Blocks: 5, 8, 9 + References: py/services/recipes/persistence_service.py:20-25 (PersistenceResult), :193-209 (delete_recipe), :439-482 (bulk_delete), :464-466 (per-id os.remove), py/routes/handlers/recipe_handlers.py:1422-1438 (single handler passthrough — no change), :1554-1573 (bulk handler passthrough — no change), py/services/pending_delete_service.py (todo 1 module — merge_batches + no-merge fallback), tests/services/test_recipe_persistence.py (existing persistence tests; create if missing, follow tests/services/test_recipe_scanner.py style) + Acceptance criteria (agent-executable): `pytest tests/services/test_recipe_persistence.py tests/services/test_pending_delete_service.py -q` passes. New tests: delete_recipe with undo enabled -> JSON + image exist in global staging dir, originals removed, payload batch_id set, manifest recipe_snapshot present; recipe with missing preview image -> only JSON staged, no crash; with undo disabled -> no staging, payload batch_id None; bulk_delete with 2 ids -> single batch_id, both recipes in one batch dir with re-anchored expires_at; merge-failure fallback -> batch_ids array of length 2. + QA scenarios: happy: run new tests -> green; failure: assert staged copies match original bytes (read both files). Evidence .omo/evidence/undo-delete-staging/task-4-undo-delete-staging.md + Commit: Y | `feat(delete): stage recipe deletes for undo` + +- [x] 5. Undo endpoint POST /api/lm/undo-delete + cache restoration + broadcast + tests + What to do: New handler file `py/routes/handlers/pending_delete_handler.py` with `class PendingDeleteHandler`: `async def undo_delete(self, request) -> web.Response` — body `{"batch_id": str}`; call `pending_delete_service.undo(batch_id)`; after success: if kind == "model", resolve the CORRECT scanner by `manifest["model_type"]` (the manifest carries it — resolve via the same per-type service/scanner resolution the model route registrars use; see py/routes/base_model_routes.py:227 per-type registrar construction and ModelServiceFactory; do NOT use a hardcoded lora scanner); restore cache entry from manifest `model_snapshot`: `cache = await scanner.get_cached_data()`; remove ALL existing raw_data items whose file_path equals the snapshot's file_path (a rescan between delete and undo may have re-added a stale entry); `cache.raw_data.append(snapshot)`; `await cache.resort()`; `scanner.bump_cache_version()`; follow the bulk-delete cache-update pattern (py/services/model_scanner.py:2271-2335) INCLUDING any version-index rebuild (rebuild_version_index at :2324) so the version index does not go stale (the single-delete path does not do this today — the undo path must); RESTORE TAG COUNTS from the snapshot's tag data (mirror the `_tags_count` decrement logic in `_batch_update_cache_for_deleted_models` in reverse — the bulk path decrements counts, undo must re-increment; guard missing tag field); `scanner._hash_index.add_entry(snapshot.get("sha256", ""), snapshot["file_path"], snapshot.get("autov3"))` (py/services/model_hash_index.py:16 — the service guards missing sha256 internally, still guard defensively); `await scanner._persist_current_cache()`. If kind == "recipe": load restored JSON file, `await recipe_scanner.add_recipe(recipe_data)` (py/services/recipe_scanner.py:2136 — recipe JSON embeds the full recipe_data incl. id/file_path; NOTE add_recipe only READS `_json_path_map` so full consistency of path mapping is restored by the forced frontend refresh (window.recipeManager.loadRecipes(true)) — covered by test (i) below; the SQLite row may transiently carry an empty json_path until that refresh, self-healing). Then `_broadcast_models_changed()` (import from py/routes/handlers/model_handlers.py:57-74) ONLY for kind == "model" — recipe undo is client-refresh only (documented limitation: other open tabs' recipe lists refresh on their next interaction). Return `{"success": True, "restored": [], "kind": kind}`. ValueError (expired/unknown batch/target occupied) -> 404 `{"success": False, "error": str}`; Exception -> 500. ROUTE REGISTRATION (use a shared routes class, NOT the per-model-type registrar — the ModelRouteRegistrar is instantiated per model type and its route table is registered 3x; adding a non-prefixed route there would register it 3 times): create `py/routes/pending_delete_routes.py` with `class PendingDeleteRoutes: @staticmethod def setup_routes(app)` registering `POST /api/lm/undo-delete` once, mirroring the existing shared-route classes (py/routes/misc_routes.py MiscRoutes / py/routes/update_routes.py UpdateRoutes); register it ONCE per mode next to the other shared routes: in py/lora_manager.py:170-172 (ComfyUI mode) and standalone.py:356-358 (standalone mode). Must NOT do: do not add a per-prefix undo endpoint; do not register through the per-model-type registrar; do not rescan the whole library on undo (snapshot restore only); do not purge the batch before cache restoration; do not touch FTS index directly (add_recipe handles it). + Parallelization: Wave 3 | Blocked by: 1, 2, 3, 4 | Blocks: 6, 8, 9 + References: py/services/model_hash_index.py:16 (add_entry), py/services/recipe_scanner.py:2136 (add_recipe), :2163 (_json_path_map read-only in add_recipe), py/services/model_cache.py (raw_data/resort), py/services/model_scanner.py:673-684 (_persist_current_cache), :113-119 (bump_cache_version), :2271-2335 (bulk cache-update pattern), :2324 (rebuild_version_index), py/routes/handlers/model_handlers.py:57-74 (_broadcast_models_changed), py/routes/base_model_routes.py:227 (per-type registrar construction — why NOT to use it), py/routes/misc_routes.py (shared routes class pattern to mirror), py/lora_manager.py:170-172 (shared route registration in plugin mode), standalone.py:356-358 (shared route registration in standalone mode), tests/routes/test_lora_manager_lifecycle.py (route registration test style) + Acceptance criteria (agent-executable): `pytest tests/routes/test_pending_delete_routes.py -q` passes. Tests: (a) undo of a staged model batch (loras) -> files restored + cache raw_data contains entry again + hash index has path + broadcast called (mock ws_manager) + response restored list; (b) undo of a staged CHECKPOINT batch -> resolves the checkpoint scanner (assert the checkpoint cache was updated, not the lora cache); (c) undo of staged recipe batch -> files restored + recipe_scanner.add_recipe called; (d) undo with unknown/expired batch_id -> 404 response; (e) route registered exactly ONCE in both modes (assert `len([r for r in app.router.routes() if r.method == 'POST' and r.path == '/api/lm/undo-delete']) == 1` after both plugin-mode and standalone-mode setup); (f) RESTART-SAFE UNDO (T8): write a staged batch + manifest to disk WITHOUT spawning the in-process timer (simulate post-restart state), call the undo handler -> files restored + cache restored (proves undo does not depend on the timer task); (g) RESCAN STALENESS (T7): pre-insert a stale raw_data entry with the same file_path as the snapshot, run undo -> exactly one entry remains and it equals the snapshot; (h) RECIPE RE-DELETE AFTER UNDO (T9): after undo + forced refresh (simulate loadRecipes(true) by rebuilding recipe scanner state), immediately delete the same recipe again -> succeeds (path map rebuilt); (i) recipe undo without prior refresh still restores files (path-map gap is masked by refresh in the flow, but the endpoint itself does not crash); (j) TAG COUNTS: bulk-delete 2 tagged models, capture `_tags_count`, undo -> `_tags_count` matches pre-delete counts; (k) EMBEDDINGS UNDO: undo of a staged embeddings batch -> resolves the embedding scanner (assert the embeddings cache was updated, not the lora cache) — embeddings shares the loras/checkpoints code path, this test pins the per-type resolution for ALL model types. + QA scenarios: happy: full route test suite -> green; failure: (d) expired batch -> assert 404 body contains "expired"; (e) assert no duplicate route error at startup in both modes. Evidence .omo/evidence/undo-delete-staging/task-5-undo-delete-staging.md + Commit: Y | `feat(delete): add undo-delete endpoint with cache restoration` + +- [x] 6. Purge scheduling — per-batch timer + on_startup sweep + opportunistic purge + tests + What to do: (a) In PendingDeleteService.stage_* methods: after successful staging, spawn `asyncio.create_task(self._purge_batch_after_ttl(batch_id), name=f"pending_delete_purge_{batch_id}")` where the task sleeps a computed duration then calls purge_batch — CRITICAL: purge_batch RE-READS the manifest's current expires_at at fire time (per todo 1 spec) so merged-away/undone/not-yet-expired batches are silent no-ops; stale timers from merged constituent batches or undone batches are harmless. (The merge fresh-timer is armed inside merge_batches per todo 1 — do not double-arm.) KNOWN RESIDUAL GAP (document, do not fix): a non-expired batch whose in-process timer died with the process is purged only by the next stage/undo call (opportunistic purge) or next restart (startup sweep skips non-expired); on a fully idle server it lingers past expiry until then — bounded, self-healing, no data loss. (b) Startup sweep: in `py/lora_manager.py` `_initialize_services` (lines 189-255) — after scanners initialize (in `_run_post_initialization_tasks` at :257 or right after the init_tasks gather), add `asyncio.create_task(pending_delete_service.purge_expired(), name="pending_delete_startup_sweep")`. This single hook covers both modes because StandaloneLoraManager reuses `cls._initialize_services()` (standalone.py:370-374). (c) Opportunistic: call `await self.purge_expired()` at the start of stage_model_delete/stage_recipe_delete/undo — MUST be called BEFORE those methods acquire the service lock, and purge_expired itself takes NO lock (LOCK HIERARCHY per todo 1 — asyncio.Lock is not re-entrant; calling it while holding the lock deadlocks). Fire-and-forget is NOT acceptable — await it; it is cheap when empty. Must NOT do: do not purge non-expired batches on startup (undo must survive restart with live browser tab); do not delete manifest-less/corrupted batches on sweep (quarantine only, per todo 1); do not rmtree past per-file purge errors; do not block startup on the sweep (create_task, not await); do not let purge_expired acquire the lock; no changes to on_shutdown. + Parallelization: Wave 3 | Blocked by: 1, 5 (same file lora_manager.py — run AFTER 5) | Blocks: — + References: py/lora_manager.py:183-187 (on_startup append), :189-255 (_initialize_services), :257+ (_run_post_initialization_tasks), standalone.py:370-374 (reuses cls._initialize_services), py/services/model_scanner.py:40-63 (task/singleton idiom), py/services/pending_delete_service.py (todo 1 module — purge semantics + fire-time re-check) + Acceptance criteria (agent-executable): `pytest tests/services/test_pending_delete_service.py -q` passes. Tests: (a) staging schedules a purge — monkeypatch asyncio.create_task capture, assert task created with name prefix "pending_delete_purge_"; (b) purge_expired called at stage/undo entry (spy on method); (c) `tests/routes/test_lora_manager_lifecycle.py` still passes (startup hook wiring) and a new assertion: after LoraManager.add_routes + on_startup invocation, purge_expired task was spawned (mock or spy via monkeypatched create_task); (d) stale-timer no-op covered by todo 1 acceptance (j). + QA scenarios: happy: run pytest on both files -> green; failure: assert non-expired batch survives startup sweep call. Evidence .omo/evidence/undo-delete-staging/task-6-undo-delete-staging.md + Commit: Y | `feat(delete): schedule pending-delete purges (timer + startup sweep)` + +- [x] 7. Settings toggle delete_undo_enabled — backend default + settings modal checkbox + tests + What to do: Backend: the `delete_undo_enabled` DEFAULT_SETTINGS key is added in todo 1 — do NOT re-add it here; only verify it exists (skip if already present). Frontend: in `templates/components/modals/settings_modal.html` add a checkbox labeled with the new i18n key `settings.deleteUndoEnabled` (find an existing boolean setting checkbox in the same file, e.g. any `type="checkbox"` bound setting, and copy its exact markup + wiring pattern); wire the change handler in `static/js/managers/SettingsManager.js` following the existing pattern used by sibling checkboxes (search that file for the sibling setting key). Must NOT do: do not create a new settings section; do not change how settings are persisted; do not touch other settings; do not modify DEFAULT_SETTINGS (already handled in todo 1). + Parallelization: Wave 3 | Blocked by: 1 | Blocks: 9 + References: py/services/settings_manager.py:57-119 (DEFAULT_SETTINGS), :1390-1392 (get), templates/components/modals/settings_modal.html (existing checkbox patterns), static/js/managers/SettingsManager.js (existing checkbox wiring; ~3300 lines — grep for a sibling boolean setting key), locales/en.json (key added in todo 12 — add key here or in 12, whichever lands first; keep the key name `settings.deleteUndoEnabled`) + Acceptance criteria (agent-executable): `pytest tests/services/test_settings_manager.py -q` passes (add assertion: DEFAULT_SETTINGS contains delete_undo_enabled=True; and staging respects it — covered in todo 1/2 tests). Frontend: `npm run test:js` passes; the settings modal renders the checkbox (covered by existing settings page test file if present, else a jsdom assertion in the settings manager test). + QA scenarios: happy: pytest + npm test green; failure: toggle checkbox -> settings saved -> server delete skips staging (covered by todo 2 test with monkeypatched setting). Evidence .omo/evidence/undo-delete-staging/task-7-undo-delete-staging.md + Commit: Y | `feat(settings): add delete_undo_enabled toggle` + +- [x] 8. Frontend single-delete undo flows — baseModelApi.deleteModel + modalUtils.confirmDelete + RecipeCard.confirmDeleteRecipe + tests + What to do: (a) `static/js/api/baseModelApi.js` `deleteModel` (184-216): return `{ success: true, batch_id: data.batch_id || null }` instead of `true`; when `data.batch_id` present, do NOT show `toast.api.deleteSuccess` — the caller shows the undo toast; when absent, keep existing toast + `removeItemByFilePath` as today. Keep `removeItemByFilePath` called in both cases (file is gone). UNDO-BLIND CALLER NOTE: `ModelVersionsTab.js:1136-1144` (version delete) calls this same method — its server-side deletes WILL be staged (undoable server-side) but the versions-tab shows its own success toast and NO undo toast; that flow is intentionally out of scope (documented in Scope OUT) — only ensure the new return object does not break its `if (!result)`-style checks (verify at lines 1136-1144 and adapt ONLY the type check, nothing else). (b) `static/js/utils/modalUtils.js` `confirmDelete` (27-42): capture `const result = await getModelApiClient().deleteModel(pendingDeletePath)`; if `result?.batch_id` -> `showActionToast('toast.undo.deleted', { name: }, 'success', { actionText: t('toast.undo.action'), onAction: () => handleUndoDelete(result.batch_id, refreshModels) })`; keep existing duplicate-badge refresh. (c) `static/js/components/RecipeCard.js` `confirmDeleteRecipe` (405-449): after the raw fetch, read `data.batch_id`; when present show action toast with `handleUndoDelete(batch_id, () => window.recipeManager.loadRecipes(true))`; when absent keep existing success toast. (d) New shared util `static/js/utils/undoHelpers.js` (or add to uiHelpers.js — pick ONE location and state it in the PR; do not split across files): `export async function handleUndoDelete(batchId, refreshFn, options = {})` with `options = { showToast = true, refresh = true }` — POST `/api/lm/undo-delete` body `{batch_id}`; on 200: if `options.refresh` -> `refreshFn()`, if `options.showToast` -> `showToast('toast.undo.restored', {}, 'success')`; on 404: read the error BODY text (parse `{error}` — distinguishes "expired" from unknown/occupied) and show `toast.undo.expired` when the message contains "expired", else `toast.undo.failed` with the server message; on other error -> `showToast('toast.undo.failed', { error }, 'error')`. The suppression options exist SOLELY for todo 9's sequential batch_ids loop (each iteration calls `handleUndoDelete(id, null, { showToast: false, refresh: false })`, then the loop does ONE final `refreshFn()` + ONE `toast.undo.restored` toast). refreshModels for model pages: reuse the pattern from ModelDuplicatesManager.js:740 — `resetAndReload(true)` imported from modelApiFactory. Must NOT do: do not change showToast signature; do not alter ModelVersionsTab.js behavior beyond the return-type compatibility check; do not let handleUndoDelete show N restored toasts or fire N refreshes for a multi-batch undo. + Parallelization: Wave 4 | Blocked by: 2, 4, 5, 10 | Blocks: 9, 11, 12 + References: static/js/api/baseModelApi.js:184-216 (deleteModel), static/js/utils/modalUtils.js:27-42 (confirmDelete), static/js/components/RecipeCard.js:405-449 (confirmDeleteRecipe), static/js/components/shared/ModelVersionsTab.js:1136-1144 (other caller), static/js/components/ModelDuplicatesManager.js:740 (resetAndReload usage), static/js/managers/BulkManager.js:657-659 (removeItemByFilePath loop pattern), static/js/utils/uiHelpers.js:136-193 (showToast — from todo 10), tests/frontend/utils/uiHelpers.dom.test.js, tests/frontend/pages/lorasPage.test.js + recipesPage.test.js (existing delete-flow tests to extend) + Acceptance criteria (agent-executable): `npm run test:js` passes with new tests: baseModelApi.deleteModel returns batch_id and suppresses success toast when present; modalUtils.confirmDelete shows action toast and undo triggers handleUndoDelete POST + refresh; RecipeCard.confirmDeleteRecipe same; handleUndoDelete 404 path shows expired toast. + QA scenarios: happy: vitest for the three flows -> green; failure: mock fetch 404 -> assert expired toast key called. Evidence .omo/evidence/undo-delete-staging/task-8-undo-delete-staging.md + Commit: Y | `feat(delete): undo toasts for single model/recipe deletes` + +- [x] 9. Frontend bulk + duplicates undo flows — BulkManager + both API clients + both DuplicatesManager + tests + What to do: (a) `static/js/api/baseModelApi.js` `bulkDeleteModels` (1591-1642): include `batch_id: result.batch_id || null` AND `batch_ids: result.batch_ids || null` in the success return object (incl. the status='cancelled' path, which uses the same success dict). (b) `static/js/api/recipeApi.js` `RecipeSidebarApiClient.bulkDeleteModels` (623-664): include `batch_id: result.batch_id || null` + `batch_ids: result.batch_ids || null`. (c) `static/js/managers/BulkManager.js` `confirmBulkDelete` (633-672): after success, if `result.batch_id` -> single action toast for the whole action: `showActionToast('toast.undo.deletedBulk', { count: result.deleted_count }, 'success', { actionText: t('toast.undo.action'), onAction: () => handleUndoDelete(result.batch_id, refreshFn) })` — one batch covers the whole bulk action (backend merges per todo 3); MERGE-FAILURE FALLBACK: if `result.batch_id` is null but `result.batch_ids?.length` — same single toast, onAction loops `handleUndoDelete(id, null, { showToast: false, refresh: false })` for each id sequentially (suppression per todo 8's options contract), then after ALL succeed does ONE final `refreshFn()` + ONE `showToast('toast.undo.restored', {}, 'success')`; if any iteration fails, stop the loop and show its error toast (no final refresh). refreshFn = recipes ? `window.recipeManager.loadRecipes(true)` : `resetAndReload(true)`. Keep the existing `toast.models.deletedSuccessfully` when both batch fields are null. CANCELLED BULK with a batch field (status='cancelled' + staged subset): show the SAME action toast (the staged subset is undoable) INSTEAD of the plain cancelled toast; cancelled without batch field keeps the existing cancelled toast. (d) `static/js/components/DuplicatesManager.js` `confirmDeleteDuplicates` (457-494) and `static/js/components/ModelDuplicatesManager.js` `confirmDeleteDuplicates` (710-776): read `data.batch_id` (and fall back to `data.batch_ids` array) from the raw fetch response; when non-empty show action toast (undo restores the whole selected group) before/instead of the current success toast, with EXPLICIT refreshFn: DuplicatesManager (recipes) -> `() => window.recipeManager.loadRecipes(true)`; ModelDuplicatesManager (models) -> `resetAndReload(true)` from modelApiFactory (same as todo 8's refreshModels). Keep exitDuplicateMode / resetAndReload + find-duplicates re-check logic unchanged (the undo click triggers its own full refresh via refreshFn). Must NOT do: do not add per-file undo toasts for bulk (single group toast only); do not change abort/cancel handling; do not alter the failed-count error toasts. + Parallelization: Wave 4 | Blocked by: 3, 4, 5, 7, 8 (same file baseModelApi.js — run AFTER 8), 10 | Blocks: 11, 12 + References: static/js/api/baseModelApi.js:1591-1642 (bulkDeleteModels), static/js/api/recipeApi.js:623-664 (recipe bulk), static/js/managers/BulkManager.js:633-672 (confirmBulkDelete), :134-142 (getActiveApiClient), static/js/components/DuplicatesManager.js:457-494, static/js/components/ModelDuplicatesManager.js:710-776, tests/frontend/api/recipeApi.bulk.test.js, tests/frontend/components/duplicatesManager.test.js, tests/frontend/components/modelDuplicatesManager.test.js + Acceptance criteria (agent-executable): `npm run test:js` passes with new tests: bulkDeleteModels passthrough of batch_id AND batch_ids in both clients; BulkManager shows action toast with batch_id and falls back to success toast when both are null; BATCH_IDS FALLBACK PATH (first-class — cross-volume bulks hit it via EXDEV during MERGE [annotated 2026-08: sibling staging removed EXDEV from stage/undo renames; only cross-volume merges can still produce it]): response with batch_id null + batch_ids [id1, id2] -> action toast shown, clicking undo calls handleUndoDelete(id1, null, {showToast:false, refresh:false}) then handleUndoDelete(id2, null, {showToast:false, refresh:false}) with exactly ONE final refresh and exactly ONE 'toast.undo.restored' toast (spy on refreshFn and showToast); fallback loop failure mid-way (mock id2 undo to 404) -> loop stops, error toast shown, NO final refresh; handleUndoDelete 404 with body {"error":"...expired..."} -> expired toast, with body {"error":"Target path occupied"} -> failed toast with server message; cancelled bulk with batch_id shows the action toast (not the cancelled toast); cancelled with batch_ids shows the action toast; both DuplicatesManager confirm methods read batch_id (and batch_ids fallback) and trigger undo. + QA scenarios: happy: vitest for the four flows -> green; failure: response without batch_ids -> assert legacy success toast still fires. Evidence .omo/evidence/undo-delete-staging/task-9-undo-delete-staging.md + Commit: Y | `feat(delete): undo toasts for bulk and duplicates deletes` + +- [x] 10. showActionToast — extract toast internals + action button + 30s countdown + CSS + tests + What to do: In `static/js/utils/uiHelpers.js` (showToast at 136-193): extract two internal helpers (NOT exported): `createToastElement(message, type)` (builds the div, returns it) and `appendToast(toast, durationMs)` (append + show class + setTimeout dismiss + transitionend removal + reposition logic). Rewrite showToast to use them with its existing durations (2000/5000). Add NEW exported `showActionToast(key, params = {}, type = 'info', options = {})` where options = `{ actionText, onAction, durationMs = 30000, countdown = true }`: builds toast via createToastElement; appends a `