Commit Graph

558 Commits

Author SHA1 Message Date
Will Miao e0052cd237 fix(download): align location-step root selection with backend diffusion routing
The download modal's location step decided between checkpoint and unet
roots using only the CivitAI file-type signal, while the backend also
falls back to DIFFUSION_MODEL_BASE_MODELS. Models like Anima (file type
"Model") were offered checkpoint roots in the UI even though
use_default_paths would route them to the unet root.

- Extract the two-tier decision into py/services/download_routing.py and
  reuse it in DownloadManager._execute_download
- Add POST /api/lm/download/routing so the UI asks the backend for the
  routing decision; fall back to the local file-type check on failure
- ModelVersionsTab: search both checkpoint and unet roots when resolving
  an existing version's download path
2026-09-11 12:41:03 +08:00
Will Miao 1b5cbbbaa0 feat(recipes): add reconnect remediation paths for missing recipe LoRAs
- Snapshot pre-rematch entry state (reconnectSnapshot) so rematched
  entries can be undone via the existing restore flow
- Bulk missing-LoRA downloads mark unresolvable failures hash-invalid,
  flipping those entries from download to reconnect candidacy
- Recipe modal always offers a reconnect action next to download for
  missing LoRA entries
- Rematch runs collect an opt-in relaxed-matching choice (also reconnect
  missing models by file name) via a pre-run options dialog on the
  global, bulk and single-recipe entries
- L4 (filename-level) matches are listed in a results dialog with
  per-entry undo
2026-09-09 06:59:54 +08:00
Will Miao 82b34097fb refactor(metadata): remove vestigial top-level trainedWords field
The field dates back to a development-stage bug in the enrich-metadata
(agent) pipeline, which briefly wrote trigger words at the top level of
model metadata instead of the established civitai.trainedWords location.
The write path was fixed before the feature merged to main (PR #1013)
and never shipped in any release, so no writer has existed since.

Remove the leftover pieces:

- BaseModelMetadata.trainedWords field (py/utils/models.py); sidecars
  from that dev window now pass the key through _unknown_fields instead
- HF download handler's strip-empty-trainedWords special case, reverting
  to saving the metadata object directly (py/routes/handlers/hf_handlers.py)
- trainedWords in the LLM enrichment context (agent_service.py)
- matching fallbacks/fixtures in the enrich_hf_validation harness and
  post-processor test

Trigger words continue to live in civitai.trainedWords for all model
sources, which is what the UI, agent post-processor, and metadata sync
all read and write.
2026-09-07 16:24:15 +08:00
Will Miao a17399d667 feat(recipes): delegate CivitAI-image re-import to companion browser extension
Recipes imported from CivitAI image URLs can contain 0 LoRAs: the backend
only sees the REST image API + EXIF, while the complete generation data
lives in the image page's internal trpc payload (see
docs/recipe-civitai-image-no-metadata.md). When the companion
lm-civitai-extension is installed with a valid license, re-import (single
and bulk) of CivitAI-image-sourced recipes is now delegated to the
extension via DOM CustomEvents; the extension scrapes the image page with
the user's session and calls back into the reimport endpoint with the
full metadata payload. Without the extension (or with an invalid license)
the native path runs unchanged.

- POST /api/lm/recipe/{id}/reimport accepts optional payload params
  (image_url/name/resources/gen_params/base_model/tags); the payload path
  reuses the import-remote engine with reimport semantics (user-edit
  carryover, delete-after-save), and malformed/failed payloads fall back
  to the legacy URL import. Response gains loras_count.
- The endpoint also accepts GET: the extension is GET-only by convention
  (documented in AGENTS.md).
- New static/js/utils/extensionReimportBridge.js (probeExtension /
  delegateReimport / getCivitaiImageInfo) wired into RecipeContextMenu
  and BulkManager with silent native fallback.
- i18n: toast.recipes.reimportingViaExtension added and translated in
  all 9 locales.
2026-09-06 20:26:14 +08:00
Will Miao f86b7b55d6 feat(recipes): remove deprecated Repair Metadata feature
The recipe "Repair Metadata" action has been marked Deprecated in the UI
for a while and cannot reliably recover recipes imported from CivitAI URLs
whose REST meta has no resources/hashes and whose image has no embedded
metadata (e.g. CivitAI-only generation data). Drop the feature end to end.

Backend:
- remove repair routes (repair, cancel-repair, recipe/{id}/repair,
  repair-bulk, repair-progress) and their handler mappings/methods
- remove RecipeScanner repair_all_recipes / repair_recipe_by_id /
  _repair_single_recipe and REPAIR_VERSION
- remove WebSocketManager recipe-repair progress channel
- drop repair_version column from the persistent recipe cache
- rematch mutual-exclusion now only checks rematch

Frontend:
- remove repair entries from per-recipe, bulk and global context menus
- remove repairRecipe / repairSelectedRecipes / repairRecipes + cancelRepair
  and the repairBulk API client method/endpoint
- drop recipe-repair i18n keys (synced across locales; doctor keys kept)

Tests/docs: delete test_recipe_repair.py, update scaffolding/routes/ws/
persistent-cache/integration tests and i18n guideline examples.
2026-09-05 16:56:20 +08:00
Will Miao b309becdf9 fix(recipes): keep source_path empty on local-fallback re-import
Re-importing a file-imported recipe fell back to its own saved preview
image, then recorded that internal path as the new recipe's source_path.
Since the old preview is deleted with the old recipe, this left a
dangling source_path that showed up as a bogus source URL and blocked
any further re-import with 'no re-importable source'.

Only persist source_path when the re-import source is an accessible
external file; otherwise keep it empty. Also let a dangling non-URL
source_path fall back to the recipe's own image so existing affected
recipes can re-import again.
2026-09-03 17:13:24 +08:00
Will Miao 00095a5398 fix(autocomplete): sync active filters via server-side store (#1091)
The LoRA Manager page kept its active filters in localStorage, which the
ComfyUI-side autocomplete read directly. When the two run in different
browsers, origins, or the ComfyUI Desktop Electron shell, localStorage is
not shared and the active-filters search silently did nothing.

The manager page now mirrors its filter state to a server-side in-memory
store (PUT /api/lm/{prefix}/active-filters), pushed on every change via a
storage-listener hook and once on page load. The autocomplete widget sends
only use_active_filters=true, and the relative-paths endpoint injects the
stored filters into the search, with explicit query params taking
precedence.
2026-09-02 14:33:44 +08:00
Will Miao 9584fa85c9 feat(recipes): add location open and recipe ID copy to recipe modal
Add a de-emphasized meta footer to the recipe modal, mirroring the model
modal's hash footnote: a clickable file location on the left (opens the
recipe JSON via the generic open-file-location route, with the Docker
clipboard fallback) and a middle-truncated recipe ID with copy button on
the right.

The recipe detail API now exposes recipe_json_path so the frontend does
not have to guess the on-disk storage layout. Translations for the new
recipes.modal.* keys are filled in for all 9 locales, reusing the model
modal's openFileLocation wording per locale.
2026-09-01 21:58:47 +08:00
Will Miao 39e7c1376c Support re-import for recipes without a source URL
Recipes imported by drag & drop / file-picker record no source_path and
were rejected by re-import. Fall back to the recipe's own saved image,
which still carries the original embedded generation metadata.

Re-import now re-parses that original metadata instead of the appended
recipe JSON block, so parser upgrades produce fresh results. The
already-optimized preview image is kept verbatim: only its WebP EXIF
chunk is rewritten in place to replace the recipe metadata block, and
the recipe JSON is rewritten with the new analysis plus carried-over
user edits.
2026-08-31 10:01:18 +08:00
Will Miao 2a3c632dc5 feat(recipes): add Unknown base-model filter bucket for undetermined recipes
Normalize undetermined recipe base_model to None in RecipeFormatParser
(previously ''). get_base_models now reports an "Unknown" bucket backed
by a dedicated __unknown__ marker, and the listing filter matches it
against recipes whose base model is falsy. Frontend renders the bucket
label as "Unknown" while filtering via the marker.

Tests: handler, scanner, parser, and frontend filtering.
2026-08-31 09:09:53 +08:00
Will Miao c8b9db5bf4 feat(recipes): add manual checkpoint reconnect for broken recipe entries
Checkpoint entries that cannot be restored by download (deleted,
unresolvable hash, or name-only remnants with no CivitAI identifiers)
now get the same remediation chain LoRAs already had:

- scanner: parameterized reconnect-suggestion ranking, update/restore/
  set-hash-invalid for the checkpoint entry, and clear hashInvalid on
  rematch write-back (was only done for LoRAs)
- persistence/handlers/routes: reconnect/restore/reconnect-suggestions/
  mark-hash-invalid endpoints under /api/lm/recipe/checkpoint/*
- modal: checkpoint reconnect UI (deleted/hash-invalid badges, inline
  form with suggestions, undo for reconnected entries); download
  failures mark the hash invalid only on explicit unresolvable signals
  (not found/deleted/404/410), matching the LoRA rule
- css: checkpoint undo button shares the LoRA undo styles
- i18n: the 14 new keys translated in all 9 locales
2026-08-30 18:02:15 +08:00
Will Miao bccd494a56 feat(recipes): explain empty LoRA lists with collapsible "Why no LoRAs?" panel
Record import provenance on every recipe: a new import_info block
(channel, machine-readable no-LoRA reason, diagnostic details) built at
import time across all channels (batch import, single URL, local file,
upload, widget save, re-imports) and persisted in the recipe JSON plus
the SQLite persistent cache (new import_info_json column with ALTER
TABLE migration).

The recipe modal renders the empty LoRA list with a collapsed details
panel showing the import method, the reason (CivitAI API returned no
LoRA resource data, API meta missing, no embedded metadata, ComfyUI
workflow metadata, video, unparsable format), and recorded diagnostics.
Legacy recipes without import_info fall back to heuristics labeled as
inferred. Genuine no-LoRA generations show no panel.

CivitAI images are always classified by API meta shape: the onsite
generator writes A1111-style EXIF without LoRA references, so parsed
EXIF cannot prove "no LoRAs used".

Adds recipes.resources.noLoras* i18n keys (all 10 locales) plus
frontend vitest and backend pytest coverage.
2026-08-30 16:28:41 +08:00
Will Miao 3fd29f6943 remove(nodes): delete Random Checkpoint Loader and Random Unet Loader nodes
- Remove py/nodes/random_checkpoint_loader.py and random_unet_loader.py
- Remove their dedicated test file
- Clean up imports and NODE_CLASS_MAPPINGS in __init__.py
- Update loader-pool comments/docstrings to reference the remaining Checkpoint/Unet Loader nodes' control_after_generate feature
2026-08-30 11:38:01 +08:00
Will Miao 838a374a56 feat(recipes): reconnect suggestions, undo, and base-model family tolerance
Enhance the deleted-LoRA reconnect flow in the recipe modal:

- Suggest local reconnect candidates when the panel opens, ranked by
  identity (same hash / same CivitAI version) then filename/name
  similarity, with a hard filter on confident base-model mismatches;
  the input gets a Combobox backed by the same endpoint as you type.
- Snapshot the pre-reconnect entry and offer a permanent restore:
  reconnected entries show an undo icon at the right end of the info
  row, with the original filename in the tooltip.
- Relax the manual reconnect base-model guard to a three-tier check:
  exact/unknown labels pass silently, same-architecture families
  (e.g. Pony <-> Illustrious) pass with a warning toast, and only
  cross-architecture mismatches stay hard-rejected.
2026-08-30 08:17:35 +08:00
Will Miao 856c9a87ac fix(recipes): resolve stale LoRA hash on import and add hashInvalid state
- import: prefer A1111 Lora hashes (12-char AutoV3) over conflicting Hashes
  JSON values; recover the quote-wrapped AutoV3 from CivitAI image API meta;
  merge EXIF-parsed LoRAs when the API-only parse yields none (meta=null)
- rematch: treat entries whose hash failed CivitAI resolution (hashInvalid)
  as unresolved candidates; clear the flag on rematch/reconnect write-back
- download: persist hashInvalid and show a distinct toast when hash lookup
  returns "Model not found", so unresolvable entries become recoverable
- ui: add Unresolvable Hash badge styling and reconnect affordance
- i18n: translate the new keys across all 10 locales
2026-08-28 22:24:07 +08:00
Will Miao 641a61f804 feat(relink): accept CivitArchive URLs when linking models 2026-08-26 21:31:30 +08:00
Will Miao 3025c64fea fix(recipes): serve duplicate scan from cache and guard against re-entry 2026-08-26 20:34:50 +08:00
Will Miao 08895f77ff fix(update): align update-check summary count with Updates filter scope (#1083) 2026-08-25 20:36:46 +08:00
Will Miao a202c666bc fix(download): return 200 for missing queue items and quiet download-progress 404s
The browser extension's apiFetch treats any 404 as a missing endpoint and
retries the legacy non-/api/lm URL, producing two spurious
'error_middleware - WARNING - API GET ... 404' log lines per occurrence.

- complete_download_in_queue / update_download_queue_status /
  retry_download_from_history: 'not found' is a normal business outcome,
  return 200 + success:false instead of 404 (extension behavior unchanged;
  apiGet ignores the HTTP status)
- error_middleware: downgrade /api/lm/download-progress/ 404s to debug like
  previews - the 404 status itself stays (extension uses it for failure
  detection), only the log level is lowered
2026-08-25 09:59:33 +08:00
Will Miao 41ed03e5c6 fix(download): stop stale aria2 GIDs from spamming errors after queue clears
- Log expected "GID not found" tellStatus probes at DEBUG, and treat a
  forgotten GID as permanent so the poll loop recovers immediately
  instead of burning 4 retries x 3s of ERROR lines per cycle
- cancel_download tolerates a forgotten GID and always pops the
  in-memory transfer so concurrent polls cannot re-register a
  cancelled download
- Restore sweep deletes aria2 state records with no resolvable target
  path instead of skipping them forever
- Clearing the download queue now also cancels in-memory tasks, removes
  live aria2 transfers and drops persisted state for the cleared ids
  (partial files on disk are preserved)
2026-08-25 08:19:54 +08:00
Will Miao da071e8452 feat(versions): add file-variant badge and hide download button for in-library versions (#1058) 2026-08-24 23:17:10 +08:00
Will Miao 6f5c444ec5 feat(recipes): add lora availability filter to recipe filter panel 2026-08-24 17:00:02 +08:00
Will Miao 879745da53 fix(init): add missing /api/lm/init-status endpoint used by polling fallback
initialization.js falls back to polling /api/lm/init-status when the
/ws/init-progress WebSocket cannot be established, but no route ever
registered that path — each poll 404'd and the page never reloaded after
the scan completed. Report the aggregate status of all four scanners and
omit pageType so every initialization page accepts the update.
2026-08-24 14:17:06 +08:00
Will Miao 06c270a6e1 fix(recipes): show initialization screen and auto-reload during first scan
The recipes page always rendered with is_initializing=False, so a cold
start displayed an empty grid that never updated until a manual refresh.
Mirror the model pages: gate render_page on the scanner state, broadcast
init progress from RecipeScanner (including a completion message, and a
failure fallback so the page never stalls), and teach initialization.js
to detect the /loras/recipes page before the generic /loras match.
2026-08-24 14:17:06 +08:00
Will Miao c2360a35ad fix(ui): show empty folders as move and download destinations (#999) 2026-08-23 21:09:16 +08:00
Will Miao 030a32f8fa feat(ui): add hash search option and de-emphasized hash display in model modal 2026-08-23 10:09:55 +08:00
Will Miao 3ebf256c5d feat(recipe): send embedded recipe workflow to ComfyUI canvas 2026-08-21 21:09:58 +08:00
Will Miao 259e08e47c feat(download): expose per-file downloadedFiles in check-model-exists (#1058)
The version branch of check-model-exists now returns
downloadedFiles: [{fileId, fileName, filePath}] so clients (e.g. the
browser extension) can tell a partially downloaded version apart from a
fully downloaded one. Reuses ModelCivitaiHandler._match_downloaded_files
(D2 rule) against the local cache; unmatchable local files are reported
with fileId: None. No CivitAI API call added.
2026-08-20 20:59:19 +08:00
Will Miao e7c26bf722 feat(download): per-file download status and multi-file selection (#1058) 2026-08-19 17:51:31 +08:00
Will Miao cef4129fc9 fix(download): allow downloading additional files of an in-library model version (#1058) 2026-08-19 16:29:59 +08:00
Will Miao fc3f3f3bdb feat(loaders): add control_after_generate random model selection to checkpoint/unet loaders
The Checkpoint/Unet Loader (LoraManager) nodes now support ComfyUI's
built-in control_after_generate mechanism on the ckpt_name/unet_name combos,
letting users pick a random model on every queue with the selected model
written back into the widget (visible, and lockable via the 'fixed' mode).

A base_model input narrows the random pool: a front-end extension fetches
the name/base_model mapping from the new /api/lm/checkpoints/loader-pool
endpoint and filters the combo options, wired through the node callback,
the refreshComboInNodes extension hook, and a graph.onConfigure hook
installed from onAdded (onNodeCreated fires before the node is attached to
a graph, so the graph reference is unavailable there).
2026-08-19 05:13:51 +08:00
Will Miao ef3e7d7bf4 feat(update): detect CivitAI paidAccess versions and add hide paid updates (#1060)
CivitAI's PaidAccess cutover deprecated the availability=EarlyAccess and
earlyAccessEndsAt signals; gated versions now report availability=Public
with a paidAccess DTO that LoRA Manager previously ignored, so "Hide
Early Access Updates" missed paid/early-access models and downloads
failed with 401.

Parse and persist paidAccess from model-level, bulk, and by-hash
responses; treat timed paid gates as early access and permanent paid
versions as a distinct is_paid state; add a hide_paid_updates setting
with a "Paid" badge in the versions tab; warn before downloading gated
versions. Includes SQLite migration, i18n for all locales, and
backend/frontend tests.
2026-08-15 18:08:14 +08:00
Will Miao c85b6b64a1 feat(recipes): add recently opened sort with modal open tracking
Track recipe modal opens in a separate stats file (never touching recipe
JSON/EXIF), expose a fire-and-forget POST endpoint, and add an 'opened'
sort that hides never-opened recipes as a true recently-opened view.
Includes i18n for all locales and backend/frontend tests.
2026-08-15 11:37:46 +08:00
Will Miao 303cca0d85 fix(download): accept newer CivitAI file types for primary file selection
Downloads failed with "No suitable file found in metadata" for models whose
only file uses newer CivitAI file types (e.g. 'Enhancement LoRA' for
Anima/AIR image-editing LoRAs) because the primary-file allowlist only
covered legacy types.

- unify the weights-type allowlist as MODEL_WEIGHT_FILE_TYPES
  (py/utils/constants.py) and apply it across download, recipe and
  metadata-refresh lookups
- mirror CivitAI's getPrimaryFile() semantics: prefer weights-type primary,
  fall back to weights files, then trust CivitAI's primary flag (excluding
  non-downloadable artifacts like Config/Archive/Workflow)
- mirror the allowlist in the frontend via shared isModelWeightFile() helper
- add regression tests for the Enhancement LoRA primary-file download,
  primary-flag fallback and weights-over-non-weights-primary preference
2026-08-12 21:14:23 +08:00
Will Miao 1da2433bb2 feat(delete): add undo-delete endpoint and purge scheduling 2026-08-11 14:08:28 +08:00
Will Miao 6a259a14fa feat(nodes): flag missing local models at queue and load time (#1057) 2026-08-10 12:31:36 +08:00
Will Miao 95fb3c7fc9 feat(recipes): add prompt-aware duplicate detection toggle 2026-08-10 00:07:14 +08:00
Will Miao 3001f0f0ef feat(recipes): add recipe rematch API endpoints 2026-08-09 11:30:50 +08:00
Will Miao 27027c4497 refactor(recipes): reuse shared local hash cache in create-from-example 2026-08-08 22:45:29 +08:00
Will Miao 86c85c08ec feat(recipes): pass local hash cache to remote and url recipe imports 2026-08-08 22:13:19 +08:00
Will Miao 3e1216e9bc feat(recipes): add cache version counter to model scanners 2026-08-08 21:57:36 +08:00
Will Miao 8e724538bd fix(types): resolve pre-existing basedpyright errors in py/ and standalone.py
Fix ~950 basedpyright errors across the backend:
- Convert ineffective # type: ignore comments to # pyright: ignore[rule]
- Add missing generic type arguments (Dict[str, Any], list[Any], ...)
- Annotate dynamic dict literals and runtime-initialized attributes
- Widen CivitAI provider tuple signatures in recipe parsers
- Remove dead LoraRoutes handlers calling nonexistent LoraService methods
- Suppress unavoidable ServiceRegistry import cycles (basedpyright counts
  function-local imports as cycle edges)
2026-08-08 20:12:52 +08:00
Will Miao 97b9b1f62b feat(metadata): add CivitAI AutoV3 hash support across all storage layers
- Three-state autov3 field (not-checked / checked-unavailable / 12-hex value)
  in .metadata.json sidecars, in-memory ModelHashIndex, and SQLite
  (models.autov3 column + autov3_index table) with column-presence migration
- Background self-terminating backfill for legacy rows: per-model-type
  concurrency guard, executor-offloaded I/O, Civitai-first resolution
  (SHA256-matched version file) falling back to the embedded safetensors
  header hash
- Civitai-first propagation on metadata refresh, scan, and download paths;
  reject the empty-string SHA256 placeholder and strip OneTrainer 0x prefix
- List API hash filters and hash index lookups accept 12-char AutoV3
- Cap safetensors header reads at 64 MiB to prevent crafted-file allocation
- Prevent stale AutoV3 mappings on file replacement while preserving them on
  same-file re-registration (lazy-hash completion)
2026-08-08 14:30:34 +08:00
Will Miao 56acefbd6c feat(autocomplete): search loras within active filters of LoRA Manager page
Add /af and /noaf toggle commands (plus /activefilters aliases) to the
loras autocomplete widget. When enabled (default off), suggestions are
matched within the active filters (folder, base model, tags, auto-tags,
license, tag logic) persisted by the LoRA Manager page in localStorage,
keeping the match pool consistent with the list endpoint, including the
global show_only_sfw setting.

Backend: /lm/{prefix}/relative-paths accepts the filter query params and
pre-filters the scanner cache with ModelFilterSet. The presence of the
recursive param signals the filter pipeline to run even without concrete
filters so global settings stay in parity with the list endpoint.
2026-08-07 20:07:34 +08:00
Will Miao 7980ee77d0 perf(recipes): batch preview dimension reads via asyncio.gather 2026-08-07 15:31:13 +08:00
Will Miao 720fa6d909 feat(recipes): expose preview width/height in recipe listing API 2026-08-07 11:59:26 +08:00
Will Miao 75e63c758b feat(api): add cursor-based pagination to civitai user-models endpoint 2026-08-03 11:07:06 +08:00
Will Miao dd6bdbf297 fix(update): persist update_channel via settings.json instead of hasGit
After b464fdc3 (preserve .git on release switch), the hasGit-based
channel detection is unreliable — .git now exists for both release
and nightly installs, so page refresh always reset the channel.

- Add _resolveChannelFromSettings() with migration heuristic:
  !hasGit → release (ZIP), detached HEAD → release (on tag),
  on branch → nightly. Uses gitInfo.branch from check-updates.
- Persist resolved channel to settings.json on first load
  (one-time migration) and on explicit switchChannel.
- Add update_channel validation (release|nightly) in backend
  update_settings handler.
- Remove hasGit-based guessing from initialize(); defer to
  checkForUpdates where full gitInfo is available.
- Channel resolution runs before checkForUpdates early-returns
  to avoid null channelMode on reload-within-interval.

Tests: 361 passed.
2026-07-31 13:23:54 +08:00
Will Miao 99e65cccd8 fix(update): downgrade settings backup/restore logs from INFO to DEBUG 2026-07-30 20:32:00 +08:00
Will Miao b464fdc333 fix(update): preserve .git on release channel switch, use git checkout tag
Previously, switching to the release channel would delete .git/ and
fall back to a ZIP download. This broke update.bat, manual git
commands, and CM git-based update detection.

Now the release path uses git checkout <latest-tag> when .git exists,
and only falls back to ZIP when .git is absent (CM CNR installs).
.git is never deleted - the ZIP→nightly path remains a one-way
upgrade via _init_git_repo.

Also updates locale strings (en, zh-CN, zh-TW, ja) to remove the
now-inaccurate "remove the Git repository" wording.
2026-07-29 21:23:39 +08:00