Compare commits

..
318 Commits
Author SHA1 Message Date
Will Miao 8b344ea39f feat(ui): add View on Hugging Face button, plumb hf_url through full cache pipeline 2026-07-01 08:38:16 +08:00
Will Miao 8348a0cef8 fix(download): harden HF download path validation, fix WebSocket leak, add URL detection tests (#965, #977)
Security hardening:
- Validate repo format with strict regex (reject .. traversal)
- Validate filename rejects path separators and ..
- Validate relative_path rejects absolute paths and ..
- Verify model_root is within configured scanner roots using
  realpath + os.sep guard to prevent prefix-match bypass
- Add realpath-based escape detection for final dest_path

Bug fixes:
- Fix WebSocket leak in _downloadHfSingle: wrap ws.close() in
  try/finally so it closes even if downloadHfModel() throws
- Same fix for batch HF download per-file WebSocket loop

Frontend hardening:
- Tighten HF repo regex: require huggingface.co for full URLs,
  reject bare .. patterns
- Add 12 unit tests for detectUrlType() covering HF resolve,
  HF repo, CivitAI, CivArchive, direct HTTP, edge cases
2026-07-01 05:51:58 +08:00
Will Miao 7cf785b72f fix(ui): unify HF file selection UI, remove cloud icon, add select-all, cleanup dead code (#965, #977)
- Unify single-URL and multi-URL HF repo flows to use the same batch
  preview interface (remove separate repoFileStep)
- Remove unnecessary cloud icon from HF batch preview items
- Use formatFileSize() instead of hardcoded MB text
- Change default selection to unchecked (no preselected files)
- Add select all / deselect all checkbox with dynamic Next button
- Clean up dead CSS, HTML template, and JS methods from removed
  repoFileStep
- Add selectAll i18n key with translations for all 10 locales
- Fix batch progress bar name fallback for HF items
2026-06-30 23:28:35 +08:00
Will Miao e8913f4481 feat(ui): dynamically populate base model dropdown from CivitAI API, add Krea 2 constants (#1001) 2026-06-30 22:41:17 +08:00
Will Miao f9c3d8dc97 fix(metadata): demote CivArchive hash lookup failure from ERROR to DEBUG
A model not being found on CivArchive by hash is a routine case (the
model simply isn't published there), not an error. The callers already
log the outcome at WARNING (bulk_metadata_refresh) or DEBUG
(metadata_sync_service) with full context, making this ERROR-level log
both misleading and redundant.
2026-06-30 19:42:30 +08:00
Will Miao 09ca91fc0e feat(download): add Hugging Face model download to standalone UI wizard (#965, #977)
Integrate HF model downloading into the existing CivitAI-style wizard flow:
- URL type detection (civitai / hf-resolve / hf-repo / direct-http)
- Repo file explorer with checkbox-based file selection
- Batch/queue download with per-file WebSocket progress
- Aria2 backend support (respects download_backend setting)
- Scanner cache integration via create_default_metadata + add_model_to_cache
- i18n updates for all 10 locales
2026-06-30 19:36:12 +08:00
Will Miao 16f5222efd fix(cache): prevent corrupted cache rows from breaking model listings (#730)
Cache corruption (NULL model_name/file_name from legacy DB rows or partial
writes) caused format_response to raise KeyError/AttributeError, failing the
entire /loras/list request and showing no models in the UI.

Fix across three layers:
- format_response (lora/checkpoint/embedding): replace direct dict[] access
  with .get() fallbacks; return None for entries missing file_path
- handlers: filter None entries from list/excluded/fetch/duplicate/conflict
  endpoints instead of letting them crash or appear as null in responses
- model_scanner: always use validate_batch repaired copies (previously
  discarded when no invalid entries, leaving None values in raw_data)
- persistent_model_cache: add or-empty-string guards on read and write for
  nullable TEXT columns (model_name, file_name, folder, base_model, etc.)
2026-06-30 09:02:42 +08:00
Will Miao 28e7c04b37 fix(settings): migrate all settings subdirectories on portable mode switch 2026-06-29 21:40:37 +08:00
Will Miao 28f99c46d3 fix(update): preserve user data dirs during Git-based update via git clean -e excludes
git clean -fd in _perform_git_update deleted untracked, non-ignored
directories (wildcards, stats, backups, civitai, caches, logs) during
portable-mode updates, since released tags do not list them in .gitignore.
Add -e excludes for all user-managed paths to both nightly and stable
update branches. Add regression tests for both paths.
2026-06-29 21:10:38 +08:00
Will Miao 205194f4e6 chore: add stats, wildcards, backups, and logs dirs to .gitignore 2026-06-29 19:46:04 +08:00
willmiao 402d8b07cf docs: auto-update supporters list in README 2026-06-28 14:17:19 +00:00
Will Miao 3e303ab316 chore(release): bump version to v1.1.6 2026-06-28 22:17:02 +08:00
Will Miao e9e8c31ad1 fix(registry): store nodes per-client to prevent multi-tab race condition
Move NodeRegistry from a single global _nodes dict to a per-client
(_tab_nodes) structure so that multiple ComfyUI browser tabs no
longer overwrite each other's workflow node data during a
lora_registry_refresh cycle.  The merged result is a union of all
known tabs' target nodes, eliminating the non-deterministic failure
where send-to-workflow could randomly target a tab lacking valid
targets.

- NodeRegistry.register_nodes(sid, nodes) replaces per-tab data
  without affecting other tabs.
- NodeRegistry.get_merged_registry() returns the union across all
  connected clients, together with tab_count / per-tab metadata.
- prepare_for_refresh() snapshots the current active sockets; caller
  re-reads before merging so that newly-connected tabs are not pruned.
- workflow_registry.js sends api.clientId in the POST body so the
  backend can identify which tab is registering.
2026-06-28 17:57:58 +08:00
Will Miao 703a6a4ea0 fix(import): request withMeta=true from CivitAI API, fix checkpoint type guard and CivArchive version lookup
- Add &withMeta=true to image info URL so API returns full generation
  metadata (resources with hash/type) instead of null meta
- Fix checkpoint assignment guard: check modelId instead of id so non-
  checkpoint types (upscaler) are not wrongly set as recipe checkpoint
- Skip modelVersionIds loop when resources/civitaiResources already
  provided LoRAs, preventing hash-resolved duplicates
- Fix int/str type comparison in CivArchive get_model_version so
  version ID matching works correctly
2026-06-27 22:22:48 +08:00
Will Miao 283730cf38 fix(import): discover LoRA + checkpoint from modelVersionIds when API meta is null
When CivitAI image API returns meta=null and modelVersionIds at root
level, the import flow now:

- Injects modelVersionIds + browsingLevel into a minimal metadata dict
  so the parser can discover LoRAs and checkpoints (both import-from-url
  and analyze-image paths)
- Adds checkpoint dedup + fallback in the parser's modelVersionIds
  handler to avoid duplicate API calls
- Runs EXIF extraction unconditionally in analyze-image path, then
  merges with API metadata (fixes gen params loss)
- Propagates preview_nsfw_level through all three import paths:
  import-from-url, analyze-image (UI Import), and batch-import,
  plus the frontend save flow
2026-06-27 17:05:38 +08:00
Will Miao 20417797e8 fix(download): accept UNet and Diffusion Model file types from CivitAI
- Prefer file type (UNet/Diffusion Model) over baseModel name when
  deciding whether a checkpoint routes to the unet folder
- Add UNet to backend primary file type whitelist
- Add Krea 2 to DIFFUSION_MODEL_BASE_MODELS
- Include UNet/Diffusion Model files in frontend file selection UI
- Use actual file type from CivitAI in download params instead of
  hardcoded 'Model'
2026-06-27 08:56:11 +08:00
Will Miao 004c69b9ef fix(marquee): use document coordinates, add auto-scroll, support VirtualScroller off-screen cards
- Convert marquee selection from viewport to document coordinates so
  scrolling during a drag no longer deselects off-screen cards.
- Add RAF-based auto-scroll when dragging near viewport edges.
- Compute off-screen card positions from VirtualScroller layout
  parameters instead of relying on DOM queries.
2026-06-27 08:21:21 +08:00
Will Miao 47fe2d3783 chore: remove deprecated reference files from refs/ 2026-06-27 07:02:22 +08:00
Will Miao 36ef840a22 fix(parser): merge Lora hashes over empty Hashes JSON values and skip entries without hash 2026-06-26 22:31:36 +08:00
Will Miao 09c2445ac9 fix(ui): prevent scroll jump on model card click caused by sort dropdown focus
The document-level click handler in SortDropdown.js called trigger.focus()
unconditionally on every click outside the sort group. When a model card
was clicked to open the modal, focus() triggered scrollIntoView on the
.sort-trigger button, perturbing .page-content.scrollTop and causing the
card grid to jump up a few pixels.

The same interference also broke the back-to-top smooth-scroll animation:
frame-by-frame focus/scroll perturbations caused VirtualScroller to
schedule repeated re-renders, interrupting the compositor-thread scroll.

Fix: only return focus to the trigger when the dropdown was actually open,
so ordinary page clicks (e.g. clicking a model card) never force focus.
2026-06-26 19:40:12 +08:00
Will Miao 8a6d23f9c7 Revert "fix(ui): replace smooth scroll with instant for back-to-top to avoid VirtualScroller conflict"
This reverts commit a429e6b1c3.
2026-06-26 19:36:08 +08:00
Will Miao 3d207b6744 fix(updates): mark cross-folder versions as in-library during folder-filtered refresh (#997)
When refreshing updates with a folder filter, versions already present in
other folders were excluded from the is_in_library check, making them
appear as available updates. When the user tried to download, the global
check found the file already exists and returned 'model already exists'.

Fix by also collecting the cross-folder version set when folder_path is
provided, and using the union (folder-filtered + cross-folder) for
is_in_library in both _build_record_from_remote and
_merge_with_local_versions.
2026-06-26 17:40:41 +08:00
Will Miao b3edda62ad refactor(ui): persist sort per-mode with two storage keys, add recipes sort persistence 2026-06-26 17:07:17 +08:00
Will Miao a429e6b1c3 fix(ui): replace smooth scroll with instant for back-to-top to avoid VirtualScroller conflict
The back-to-top button used scrollTo({top:0, behavior:'smooth'}) which
conflicts with VirtualScroller's DOM manipulations during the smooth
scroll animation. Each animation frame triggered handleScroll() ->
scheduleRender() -> renderItems(), causing the browser to interrupt
the smooth scroll animation mid-way, resulting in only ~1 page of
upward scroll instead of reaching the top.

Root cause: commit 311e89e9 fixed VirtualScroller to listen on the
correct scroll container (.page-content), but this meant every scroll
event during smooth animation now triggers expensive DOM operations
that abort the browser's compositor-thread smooth scroll animation.

Fix: use instant scroll (scrollTop = 0) so the position is set
immediately without triggering frame-by-frame VirtualScroller
interference.
2026-06-26 16:31:31 +08:00
Will Miao c1bf9c6221 test(aria2): verify _wait_until_ready captures stderr on subprocess early exit
Regression test for the pipe-race bug where _drain_stderr consumed
aria2's error output before _wait_until_ready could read it.
2026-06-26 14:41:32 +08:00
Will Miao 75fffc1e25 fix(aria2): move stderr drain after _wait_until_ready to avoid swallowing startup errors
_drain_stderr and _wait_until_ready both read from the same stderr pipe.
Starting the drain task before _wait_until_ready creates a race where the
drain task consumes aria2's early-exit error message before the startup
waiter can read it, resulting in an empty error message in the logs.

Also confirmed that --fsync does not exist as an aria2 option (exit code
28 = Invalid argument).
2026-06-26 14:32:43 +08:00
Will Miao f264bab65c fix(aria2): remove --fsync=false to avoid crash on older aria2c versions
Exit code 28 (Invalid argument) indicates this user's aria2c does not
support the --fsync option. Remove it unconditionally; the stderr drain,
relaxed RPC timeouts, and increased retry coverage remain in place.
2026-06-26 14:24:46 +08:00
Will Miao 154fcd803b fix(aria2): disable fsync and relax RPC timeouts to prevent aria2 freeze on large files
aria2 default --fsync=true calls fsync() after each write, which blocks
the entire single-threaded process on large files under Docker overlay.
Add --fsync=false to eliminate this blocking source.

Relax aiohttp session timeout: total=30 → sock_connect=10, sock_read=60
so that transient I/O delays don't cut off legitimate tellStatus RPCs.

Increase retry params (4 attempts, 3s delay) to give aria2 more recovery
time when blocked on synchronous I/O.
2026-06-26 14:19:37 +08:00
Will Miao 4ef32d3a96 fix(ui): prevent bulk-mode highlight from being clipped on edge cards 2026-06-26 11:59:28 +08:00
Will Miao d2d109a69c feat(ui): replace native sort select with custom dropdown sized to selected text 2026-06-26 09:53:04 +08:00
Will Miao 3a2941d751 fix(aria2): drain stderr pipe to prevent aria2 freeze, retry RPC status on transient failure
Root cause: aria2c subprocess stderr pipe (64 KB buffer) was never
drained. When enough error/warning output accumulated, aria2's write()
blocked, freezing the entire process including its RPC handler. The
tellStatus call then timed out after 30s with asyncio.TimeoutError(),
producing the empty error message in 'Failed to query aria2 download
status: '.

Fixes:
- Drain stderr in a background task so pipe never fills up
- Retry get_status() RPC calls up to 3 times on transient failure
- In the failure path, preserve .safetensors when .aria2 is absent
  (the download was likely complete on disk)
2026-06-26 08:25:05 +08:00
Will Miao 0ac10dfd42 fix(ui): prevent Launch LoRA Manager button from disappearing when opening properties panel in subgraph (#996) 2026-06-25 20:47:29 +08:00
Will Miao 9c95856b2f fix(trigger-wheel): prevent Vue render mode from intercepting strength wheel events
In Vue render mode, ComfyUI's TransformPane uses a capture-phase wheel
handler (@wheel.capture) that fires before the tag element's bubble-phase
strength-adjustment listener. It checks wheelCapturedByFocusedElement(),
which requires data-capture-wheel on a focused element. The tag divs had
data-capture-wheel but were not focusable, so the check failed, causing
the capture handler to forward the event to the canvas (triggering zoom)
and stopPropagation() which prevented the strength handler from running.

Fix: move data-capture-wheel from individual tags to the container, make
it focusable (tabIndex=-1), and add a window-level capture-phase wheel
listener that focuses the container before TransformPane checks it.
2026-06-25 14:58:20 +08:00
Will Miao 5ce4667d32 feat(node-marker): add 🎯 emoji prefix to Mark as context menu item 2026-06-24 22:36:45 +08:00
willmiao be53fda6df docs: auto-update supporters list in README 2026-06-24 14:11:36 +00:00
Will Miao f48de05102 chore(release): bump version to v1.1.5 2026-06-24 22:11:17 +08:00
Will Miao 93ad81ed87 fix(ui): replace full-page loading overlay with grid-scoped loader to eliminate flicker
- Add .grid-loading-overlay CSS: position:absolute inside card grid,
  semi-transparent dark background, z-index 100, pointer-events:none
- Add showGridLoading() / hideGridLoading() to VirtualScroller:
  creates/removes the scoped overlay inside the card grid only
- Modify loadMoreWithVirtualScroll(): replace full-page
  state.loadingManager overlay with grid-scoped loading, defer
  hide via requestAnimationFrame to eliminate blank-frame gap
- Clean up gridLoadingOverlay in dispose() to prevent DOM leak
2026-06-24 21:11:13 +08:00
Will Miao ea14d211be refactor(ui): unify search bar placeholder to i18n key header.search.placeholder
- Replace page-specific header.search.placeholders.* keys with a single
  header.search.placeholder key (value: "Search", no ellipsis)
- Keep header.search.notAvailable for the statistics page
- Remove unused placeholder/placeholders/notAvailable entries from all
  10 locale files; preserve options and searchIn keys
- Update Jinja template and JS header to use the new unified key
2026-06-24 20:30:38 +08:00
Will Miao 8052cefd46 feat(ui): add keyboard shortcut cue in search bar, fix clear button positioning 2026-06-24 20:21:15 +08:00
Will Miao 845815b9b7 fix(flash): fix text widget flash in Vue mode, add fade and hover dismissal
- Fix Vue mode: text widgets (CLIPTextEncode, Prompt LM) had no
  [data-testid=widget-layout-field-label], so findRowEl never matched.
  Added fallback strategies: bare <label> text match and widget index match.
- Fix Vue mode: flash background pulse was never applied — @keyframes was
  defined but no rule bound it to .lm-flash. Replaced with CSS transition
  on .lm-flash-host class for value text color fade-in/fade-out.
- Fix Vue mode: -webkit-text-fill-color set by ComfyUI overrode
  even with !important. Added -webkit-text-fill-color override to .lm-flash.
- Fix canvas mode: highlight rect was double-offset because onDrawForeground
  ctx is pre-translated to node.pos. Removed background rect entirely per
  design decision; kept text_color + inline color only.
- Add fade-in (250ms) / fade-out (400ms) for text color in both modes.
  Canvas-drawn widgets use rAF color interpolation; DOM widgets use CSS
  transition. Fixed hexToRgb to handle 3-digit hex shorthand (#DDD).
- Add hover dismissal to canvas mode via app.canvas.getWidgetAtCursor().
  Vue mode already had it via mouseover listener.
- Replace 60fps rAF poll with 100ms setInterval for hover detection.
- Fix batch cleanup closure bug: isDomWidget evaluated per-widget instead
  of per-call; fade rAF cancellers tracked per-widget in _lmFadeCancels map.
- Unify flash color from #66B3FF to LM brand accent #4299E0.
- Fix Vue fade-out: keep .lm-flash-host 300ms after removing .lm-flash so
  CSS transition persists. Canvas DOM widgets: keep inline transition 300ms
  after clearing color.
2026-06-24 19:35:30 +08:00
Will Miao 609dc5d783 feat(sort): enable versions_count sort in non-grouped mode
Sort by Most/Fewest versions first now works when Group by model is off.

- Backend: group items by modelId (respecting version_grouping setting),
  count versions per group, sort groups by count, expand groups with
  versions sorted by version id descending
- CSS: remove rule that hid the sort option in non-grouped mode
- Tests: add 3 tests covering desc, asc, and same_base variants
2026-06-24 17:14:39 +08:00
Will Miao 7a71b34b54 feat(vlm): sort versions by newest first in VLM view, with disabled sort dropdown
When viewing all versions of a model (VLM mode via 'x versions' button):
- Backend always sorts by version ID descending, ignoring current sort_by
- A temporary 'Newest version first' option is injected into the sort
  dropdown (removed on exit, not a permanent option)
- The sort dropdown is disabled (greyed out) while VLM is active
- On clearing VLM, the previous sort preference is restored and the
  dropdown re-enabled
- Handles stale VLM state (e.g. after page reload with leftover session)
- Covers all three model page types: loras, checkpoints, embeddings

Also fixes review nits:
- Correct i18n call pattern (defaultValue in options object)
- Shared _restoreSortAfterVlm() helper to avoid triple duplication
2026-06-24 16:25:14 +08:00
Will Miao 71a459422f feat: send gen params to workflow with visual cues
- Add genParamsMapper.js: sampler/scheduler display→internal mapping,
  combined-name parsing, widget matching
- Add sendGenParamsToWorkflow() in uiHelpers.js: resolves sampler,
  fetches registry by send_gen_params marker, sends via update-node-widget
- Add send-params-btn UI in showcase hover panel and recipe modal
- Add flashWidget() in workflow_registry.js: text-color visual cue
  on updated widget values (Vue: inline style + CSS, canvas: property shadow)
- Add silent option to sendWidgetValueToNodes for consolidated toast
- Normalize param display labels (cfg_scale→CFG, etc.) in recipe modal
- Add 33 tests for genParamsMapper; update existing test assertions
2026-06-24 15:39:57 +08:00
Will Miao cd2628a0ee feat(ui): add send-prompt-to-workflow button for prompt and negative prompt
- Add sendPromptToWorkflow() and stripLoraTags() exports to uiHelpers.js
- Add send button (paper-plane icon) to recipe modal and showcase hover panel
- Restructure showcase metadata panel layout to match recipe modal style
- Respect strip <lora:> setting before sending
- Uses 'replace' mode (not append) on text-capable workflow nodes
- Add translations for all 10 locales
2026-06-23 21:36:24 +08:00
Will Miao 85da7175bc feat: add Node Marker system with right-click marking 2026-06-23 20:54:32 +08:00
Will Miao d3bf0a164b fix(gitignore): add .reasonix/ to ignore list 2026-06-23 07:06:15 +08:00
Will Miao afb6ca1b8d refactor(settings): rename update_flag_strategy to version_grouping with migration 2026-06-22 16:59:32 +08:00
Will Miao 94f43426d7 feat(ui): show version count in group-by-model cards, add versions_count sort, no-reload VLM
- group_by_model dedup now counts versions per group and attaches
  version_count; respects update_flag_strategy (same_base) by
  sub-grouping on base_model
- Card footer shows clickable 'x versions' link instead of version
  name when grouped (hides HIGH/LOW badges); clicking triggers
  View Local Versions without page reload
- Added 'Local Versions' sort option (versions_count), auto-hidden
  when group_by_model is off
- Sort preference is saved/restored separately for normal and
  grouped modes
- VLM flow (triggerVlmView, clearCustomFilter) uses resetAndReload()
  via API instead of window.location.reload()
- Fixed cache mutation bug: version_count is now set on a shallow
  copy, not the cached dict, preventing stale version_count leaking
  into VLM responses
- i18n: all 9 locale files translated
2026-06-22 16:02:12 +08:00
Will Miao 2b361f4f5d feat(ui): add group-by-model toggle to global context menu
Adds a 'Group by Model' toggle entry to the right-click global context
menu for quick access, complementing the existing setting in
Settings → Layout Settings. The menu item shows a checkmark indicator
reflecting the current state and immediately reloads the view on toggle.

Also fixes he.json translation that was mojibake (garbled characters).

Includes:
- Context menu HTML item with check-indicator
- JS toggle logic via settingsManager
- i18n for all 10 locales
- Hebrew translation fix
2026-06-22 11:31:15 +08:00
Will Miao 7438072f8c feat(save-image): add %batch_num% support in batch loop 2026-06-22 09:11:38 +08:00
Will Miao 26c54fd358 fix(versions): scope VLM custom filter per-page to prevent cross-page leak
Store the originating page type alongside VLM data in sessionStorage;
validate it on every page load before applying the filter or showing
the indicator. Stale data is auto-cleaned on mismatch.

This prevents the 'View all local versions' custom filter from leaking
into the checkpoints (or embeddings) page, which caused an empty grid.
2026-06-21 12:02:06 +08:00
Will Miao 7cb6b04c63 chore: remove duplicate _truncateText from LorasControls/CheckpointsControls, add backend test for civitai_model_id filter 2026-06-21 11:19:54 +08:00
Will Miao fc29cde82a feat(versions): add View all local versions button to model versions tab
Clicking the button closes the modal, writes filter params to sessionStorage,
and reloads the page to show all local versions of the model as individual
cards (bypassing group-by-model dedup). The filter respects the update flag
strategy and the versions-filter-toggle state (same-base vs all versions).

Supporting changes:
- sessionStorage keys vlm_model_id / vlm_model_name / vlm_base_model
- BaseModelApiClient._addModelSpecificParams adds civitai_model_id param
- LoraApiClient calls super._addModelSpecificParams for VLM detection
- LorasControls / CheckpointsControls clearCustomFilter checks VLM first
- PageControls.checkVlmFilter shows customFilterIndicator with label
- Backend parses civitai_model_id, filters before group_by_model dedup
2026-06-21 11:13:53 +08:00
Will Miao 559ca946dc feat(models): add group-by-model option to collapse multiple versions into one card
Adds a 'Group by Model' toggle in Layout Settings. When enabled, only the
latest version (highest civitai.id) of each Civitai model is shown as a
single card — older versions sharing the same modelId are hidden.

Backend dedup runs in BaseModelService.get_paginated_data() before
filtering/pagination, ensuring correct paginated results. The setting
is persisted via the existing settings pipeline and passed as a query
parameter to the listing endpoint.

Includes:
- Backend: dedup logic, route param parsing, settings default
- Frontend: API param, SettingsManager wiring, toggle UI
- i18n: translations for all 10 locales
- Tests: unit test covering dedup on/off and standalone items
2026-06-21 08:48:42 +08:00
Will Miao 2b8e7c7504 fix(tests): update recipes page tests for unified controls template
- Inject #customFilterIndicator DOM in beforeEach (raw template
  renderer doesn't process Jinja2 {% include %} tags)
- Fix selector from #customFilterText to .customFilterText
2026-06-20 06:55:47 +08:00
Will Miao 6816d75933 refactor(recipes): unify controls and breadcrumb UI with model pages
- Replace inline controls+breadcrumb in recipes.html with shared includes
- Add page_id conditionals in controls.html to adapt buttons per page type
- Unify customFilterText selector to class-based in recipes.js
- Add [data-action="find-duplicates"] event listener for unified button
- Fix i18n keys to use recipes-specific translations on recipes page
2026-06-19 22:41:50 +08:00
willmiao b58abbad7c docs: auto-update supporters list in README 2026-06-19 10:31:18 +00:00
Will Miao 999814ca87 chore(release): bump version to v1.1.4 2026-06-19 18:31:03 +08:00
Will Miao 3c2760a803 fix(stats): sort Base Model Distribution X-axis labels alphabetically (#796) 2026-06-19 17:29:33 +08:00
Will Miao 0edbd7bcca fix(metadata): add LoraTextLoaderLM extractor so SaveImageLM records its loras (#801) 2026-06-19 17:13:48 +08:00
Will Miao 21e89fa7de fix(tags): normalize tag case on save and make filtering case-insensitive (#727)
- save_metadata_updates now trims/lowercases/dedupes tags on write
- ModelFilterSet tag matching is now case-insensitive (both include/exclude)
- Removed redundant .lower() calls in tag_update_service.py
2026-06-19 16:42:09 +08:00
Will Miao 968d6d1d1f feat(tags): unify recipe modal tag UI with model modal
- Replace recipe modal's custom tag display/edit with shared
  renderCompactTags/setupTagEditMode from ModelTags and utils
- Remove 300+ lines of duplicated tag display and editing code
- Parameterize setupTagEditMode with saveHandler/onSaved/showSuggestions
  options for recipe-specific save flow (updateRecipeMetadata + dirty state)
- Scope all DOM queries in ModelTags.js via options.container / this.closest
  to prevent cross-modal element conflicts
- Fix edit button alignment (justify-content: flex-start)
- Fix tag tooltip selector scoping in setupTagTooltip
- Add width: 100% to #recipeTagsContainer for edit container full width
2026-06-19 16:31:27 +08:00
Will Miao cf0fd0e0ad feat(i18n): internationalize dynamic insights content with key/params architecture (#489) 2026-06-19 13:49:03 +08:00
Will Miao 16e5dcf7b2 feat(i18n): internationalize statistics page strings across all locales 2026-06-19 13:37:01 +08:00
Will Miao ab6bb25d46 fix(example-images): skip hidden files in path validation, show offending items on failure (#807) 2026-06-19 11:54:55 +08:00
Will Miao 07f49559be fix(virtual-scroll): avoid full reload on move-to-folder, scroll to top on filter/page reset
- MoveManager/SidebarManager: replace resetAndReload with in-place
  VirtualScroller update after move operations (remove non-visible,
  update visible items' file_path). Preserves scroll position and
  avoids empty grid.
- VirtualScroller: add removeMultipleItemsByFilePath for efficient
  batch removal with Array.isArray guard.
- baseModelApi: scroll to top on loadMoreWithVirtualScroll(true),
  covering filter/sort/search/folder/views changes.
- SidebarManager selectFolder: scroll now handled centrally.
2026-06-19 09:18:49 +08:00
Will Miao b24b1a7e57 feat(settings): hide API key from frontend, use status+edit instead of password field
Backend changes:
- Add civitai_api_key to _NO_SYNC_KEYS, return only boolean civitai_api_key_set
- Clean up known template placeholder on load to prevent false positive

Frontend changes:
- Replace type=password with type=text + CSS masking (-webkit-text-security)
- Replace pre-filled input with status display (Configured/Not configured)
- Add inline edit view with Save/Cancel buttons
- Re-add eye toggle via CSS class toggle (not type switching)
- Use CSS transitions for smooth status/edit view switching

This prevents Chromium/Vivaldi password manager from triggering
'save password' prompts when opening the settings modal.
2026-06-19 08:05:04 +08:00
Will Miao faf64f8986 fix(css): migrate duplicates component to canonical color tokens
Replace undefined --lora-accent-l/c/h and --lora-warning-l/c/h with
canonical --color-accent-l/c/h and --color-warning-l/c/h from the
design token system. Fix 5 border-color declarations missing oklch()
wrapper, fix var() space syntax error in .group-toggle-btn:hover,
and replace hardcoded green with --color-success token.
2026-06-18 22:41:46 +08:00
Will Miao a617487a43 fix(ui): lift theme popover out of header stacking context to appear above modals 2026-06-18 22:19:36 +08:00
Will Miao 3012a7aef3 fix(settings): prevent Firefox save-password prompt from API key input
- Remove server-side value='...' from password field in settings modal template
  so the API key is never baked into the DOM at page load time
- Populate the input dynamically via loadSettingsToUI() when modal opens
- Clear both API key and proxy password fields on modal close to prevent
  Firefox from detecting pre-filled password fields on page navigation
2026-06-18 21:57:03 +08:00
Will Miao 499e19de34 fix(modals): tone down batch summary modal styling - remove icons, flatten gradients, lock to design tokens
- Metadata Fetch Summary: remove per-card icons, demote total/duration cards
  to neutral border, drop title icon, fix table header border width
- Batch Import Summary: replace 3em centered hero with inline left-aligned
  layout, flatten progress bar gradient, simplify circular badges to plain
  colored icons, unify border widths to 4px and token namespace to --color-
- Lock all off-scale em typography to --text-{xs,lg} design tokens
2026-06-18 21:56:58 +08:00
Will Miao 9161762ca9 fix(sidebar): align hidden indicator height (48px) and icon size with sidebar header 2026-06-18 21:14:35 +08:00
Will Miao 9bbd26efe6 feat(license-icons): add second set of license icons matching current CivitAI design
- Add 5 new Tabler SVG icons (currency-dollar, brush, user, git-merge, license)
- Implement Set 2 rendering in ModelModal.js (standalone UI) with green/red
  permission indicators and preview_tooltip.js (ComfyUI widget)
- Add use_new_license_icons setting (default: true) with toggle in settings UI
- ComfyUI tooltip reads setting directly from preview-url API response to
  eliminate race conditions and respect standalone settings changes
- Remove the now-unused separate ComfyUI setting loramanager.license_icon_style
- Add CSS for both standalone (lora-modal.css) and widget (lm_styles.css)
- i18n: translate licenseIcons keys into all 10 supported languages
- Fix test to use classic style explicitly for continued coverage
2026-06-18 21:07:44 +08:00
Will Miao 258b2622d5 fix(sidebar): align restore indicator with sidebar header and add first-use breathing animation (#990) 2026-06-18 19:22:38 +08:00
Will Miao 80ec9085dd fix(theme): replace Gruvbox with Midnight, fix accent/info hue collisions and hardcoded colors
- Replace Gruvbox preset with Midnight (deep blue-purple, violet accent)
- Fix accent/info hue collisions in Nord, Monokai, Dracula, Solarized
- Fix Solarized error/warning collision (error-h 25->5) and WCAG contrast
- Make --color-skip-refresh-* follow --color-warning-h dynamically
- Replace hardcoded rgba(24,144,255) in onboarding.css with --color-accent
- Replace hardcoded #00B87A in import modals with --color-success
2026-06-18 18:57:53 +08:00
Will Miao c5c7373e10 feat(theme): add 5 preset color themes (Nord/Gruvbox/Monokai/Dracula/Solarized) with popover selector
Implements Approach C (dual-attribute: data-theme + data-theme-preset),
keeping all 106 existing [data-theme="dark"] overrides unchanged.

- Colors: 5 professionally designed oklch palettes in tokens/colors.css
- UI: popover theme selector with mode (Light/Dark/Auto) + preset grid
- JS: cycleTheme(), setPreset(), localStorage persistence
- Locale: 12 new translation keys across 10 languages
- Polish: solid accent swatches matching flat token-driven aesthetic
2026-06-18 09:53:40 +08:00
Will Miao b7721866e5 fix(stats): implement Model Types chart in Collection tab with correct type distribution 2026-06-18 06:48:46 +08:00
Will Miao 8314b9bedb feat(downloads): add /downloads/queue/status endpoint and integrate queue lifecycle
- New GET /api/lm/downloads/queue/status handler for non-terminal status
  transitions (queued -> downloading, downloading -> paused, etc.)
- Queue lifecycle auto-integration in DownloadManager._download_with_semaphore:
  downloading -> SQLite update_status('downloading') on semaphore acquire
  completed -> complete_download('completed') on success
  canceled -> complete_download('canceled') on CancelledError
  failed -> complete_download('failed') on Exception
- All queue operations wrapped in try/except to never break the download flow
2026-06-17 23:04:30 +08:00
Will Miao 75298a402f chore(release): bump version to v1.1.3 2026-06-17 17:52:56 +08:00
Will Miao 92b5efd414 fix: guard posix_fadvise on non-Linux platforms to prevent AttributeError on Windows (#988) 2026-06-17 17:22:10 +08:00
Will Miao 33ee392b7b feat(settings): redesign Card Overlay Blur range slider to match settings UI style 2026-06-17 15:24:14 +08:00
Will Miao 5237f8b7dc chore: remove keyboard navigation UI elements and related code
- Delete static/css/components/keyboard-nav.css entirely
- Remove @import of keyboard-nav.css from style.css
- Remove keyboard-nav-hint divs from controls.html and recipes.html
- Clean up all keyboard.* translation keys from 10 locale files

The actual keyboard scrolling handlers (PageUp/PageDown in infiniteScroll.js
and VirtualScroller.js) are kept as they provide core scroll functionality.
2026-06-17 15:07:34 +08:00
Will Miao 5107313fd1 revert: restore &logo=github parameter to release-date badge
This reverts commit 95bbc669efb1aa0c23b94be6f0a5e7a188f1c019.

The real issue was shields.io GitHub API token pool exhaustion (intermittent),
not the &logo=github parameter. All 3 badges (Discord, Release, Release Date)
were affected at various times due to the same root cause: shields.io
temporarily unable to query GitHub API.
2026-06-17 11:24:40 +08:00
Will Miao 95bbc66919 fix: remove broken logo parameter from release-date badge URL 2026-06-17 11:21:26 +08:00
Will Miao e268e59419 chore: stop tracking .docs/ and add to .gitignore
.docs/ is now excluded from git tracking so working/research notes
can live there without being committed.
2026-06-17 11:20:19 +08:00
willmiao 547e1f9498 docs: auto-update supporters list in README 2026-06-17 01:57:52 +00:00
Will Miao bf32d8b6fd chore(release): bump version to v1.1.2 2026-06-17 09:57:37 +08:00
Will Miao 8299881024 refactor(sidebar): remove pin/unpin and global hide, use per-page hide only
- Remove pin/unpin and auto-hide hover mechanism (isPinned, isHovering,
  hoverTimeout, showSidebar/hideSidebar, updateAutoHideState, etc.)
- Remove global show_folder_sidebar setting (SettingsManager,
  PageControls, recipes, backend default)
- Simplify sidebar visibility to a single per-page toggle:
  · Dedicated chevron-left button in header to hide sidebar
  · Edge indicator (chevron-right) to restore when hidden
  · No dropdown, no hover area, no pin button
- Add _migrateOldSettings() to convert old sidebarPinned and
  show_folder_sidebar states to per-page sidebarDisabled
- Fix sidebar flicker on page load: CSS defaults to off-screen,
  JS explicitly sets .visible or .hidden-by-setting
- Remove obsolete CSS classes: auto-hide, hover-active, collapsed
- Remove i18n keys: pinSidebar, unpinSidebar, moreOptions
- Update test mocks for the new initialize() interface
2026-06-17 09:49:24 +08:00
Will Miao da02268196 fix(css): add top margin to stat-cards container for consistent spacing 2026-06-17 08:24:03 +08:00
Will Miao 8c4b9a1e70 fix(metadata-sync): persist not-found flags to SQLite cache on deleted-provider path
When a model is already classified as civitai_deleted=True via
.metadata.json but re-enters the failure block through the
civarchive/sqlite provider path (not the default provider),
needs_save was never set to True because civitai_api_not_found
and sqlite_attempted were both False. The flags were never
persisted to SQLite, causing the model to be re-fetched on
every restart.

Also demoted duplicate INFO/ERROR logging in fetch_and_update_model
to DEBUG (the use case already logs at WARNING), and added
exc_info=True to the fetch_all_civitai error handler.
2026-06-17 08:22:24 +08:00
Will Miao 0906c484e9 fix: actually halt bulk operations on cancel — frontend AbortController + backend guards (#986) 2026-06-17 07:20:32 +08:00
Will Miao 4199c30fec fix(metadata-sync): downgrade "Model not found" to INFO and replace model_name with file+sha256 in log 2026-06-17 00:06:43 +08:00
Will Miao 4a8084cdbc feat(save-image): support %NodeTitle.WidgetName% placeholders and fix %seed% None fallback (#314) 2026-06-16 23:48:44 +08:00
Will Miao 6263e6848c fix: move posix_fadvise(DONTNEED) after read loop so it actually evicts pages (#985) 2026-06-16 23:12:02 +08:00
Will Miao 58c266ad07 fix(scanner): respect lazy hash for checkpoints, add posix_fadvise, cancel on shutdown (#985) 2026-06-16 23:00:23 +08:00
Will Miao 2939813e1a feat(metadata-fetch): add result summary modal with i18n, fix contrast and counting bugs (#38) 2026-06-16 22:38:50 +08:00
Will Miao a9e5ee7e79 fix: follow-up nits for AVIF/JXL brotli support
- Fix JXL container ftyp size check (==20 → >=16) to accept
  wider range of valid JXL files
- Add brotli decompression size limit (2 MB) to prevent OOM
- Add trailing newline to requirements.txt
- Add unit tests for new ISOBMFF/brotli extraction paths:
  JXL/AVIF happy paths, missing brob, corrupt payload,
  non-ISOBMFF fallthrough, write-skip on AVIF/JXL,
  JSON dict/list fields, and oversized decompression
2026-06-16 16:27:56 +08:00
Will Miao a17b0e9901 Merge pull request #982 from koloved/main
Add AVIF and JXL image support with brotli metadata decompression
2026-06-16 16:24:30 +08:00
s.ivanov 8f23d966bf Update requirements.txt 2026-06-16 07:27:32 +02:00
Will Miao 7a76fc72d0 fix(rate-limit): continue to next provider on CivArchive 429 to prevent bulk refresh from freezing (#983)
When CivArchive returns HTTP 429 with a large retry_after, the bulk
metadata refresh would block for hours because:

1. FallbackMetadataProvider raised RateLimitError instead of continuing
   to the next provider (e.g., SQLite archive was never reached).

2. _RateLimitRetryHelper retried long-rate-limit 429s 3 times — all
   futile since the hourly cap hasn't reset.

3. The batch loop had no awareness of persistent rate-limiting,
   causing 192+ models to each hammer the same rate-limited endpoint.

Changes:
- FallbackMetadataProvider: all 6 methods now continue to next provider
  on RateLimitError instead of raising (model_metadata_provider.py)
- fetch_and_update_model: deleted-model path also continues on
  RateLimitError so sqlite provider gets a chance (metadata_sync_service.py)
- _RateLimitRetryHelper: when retry_after >= 120s, only 1 attempt is
  made — retries are futile for hour-scale rate limits
- BulkMetadataRefreshUseCase: tracks consecutive rate-limit failures
  and aborts early after 3 (bulk_metadata_refresh_use_case.py)

Tests: updated test_fallback_respects_retry_limit for new continue
behavior; added tests for large/small retry_after thresholds.
2026-06-16 13:08:34 +08:00
Will Miao 518a4dd5ee chore: add reasonix.toml and .codegraph/ to .gitignore 2026-06-16 13:05:11 +08:00
s.ivanov 2b6d4e5d8b Add AVIF and JXL image support with brotli metadata decompression 2026-06-15 09:28:49 +02:00
Will Miao 1f4edbeb9d chore(release): bump version to v1.1.1 2026-06-14 23:49:44 +08:00
Will Miao a256558a0e fix(downloads): delete history entries on retry and add dedup for bug #980
- retry_from_history() and retry_all_failed() now DELETE the original
  history entry after re-queuing it. Previously the old entry stayed
  in history causing exponential growth on repeated retry→cancel→retry
  cycles.
- Add deduplicate() called once on singleton creation to clean up
  existing duplicate queue/history entries left by the bug:
  1. In-status dedup (keep highest id per model+version+status)
  2. Cross-status dedup (prefer completed > failed > canceled)
  3. Queue dedup (keep highest rowid per model+version)
  4. Orphan queue cleanup (source='retry' entries obsoleted by
     terminal history entries)
2026-06-14 22:52:44 +08:00
Will Miao 818b9113f0 fix(preview): add Cache-Control header to FileResponse for browser caching (#975)
Chrome does not cache 206 Partial Content responses for <video> elements
without an explicit Cache-Control header. When VirtualScroller recycles
cards and creates new <video> elements with the same URL, Chrome
re-downloads the full video (several MB each) instead of using the cache.

Verified via Chrome DevTools: same .mp4 URL appears 2-3 times in network
trace as separate requests with no cache hit, each returning 206. With
Cache-Control: max-age=86400, the browser will reuse the cached response
for 24 hours across scroll cycles.

Video preview files are ~3.5MB while image previews are ~50-100KB (due
to WebP optimization), making caching especially impactful for videos.
2026-06-14 17:36:59 +08:00
Will Miao 6a4fd020dc fix(api): return JSON error responses for all /api/* routes — prevent JSON.parse crashes on 404/500 2026-06-14 13:13:01 +08:00
Will Miao 7a23040452 fix(save-image): sanitize invalid filename chars from %pprompt%, %nprompt%, %model% patterns (#978) 2026-06-14 09:33:12 +08:00
Will Miao 138024aefe fix(preview): revert to FileResponse as default for all platforms (#975)
The previous commit (a19ddc14) restored Linux sendfile but kept the
manual streaming path for Windows via sys.platform guard. A Windows
user reports performance is still worse than v1.0.5.

Switch back to web.FileResponse for all files on all platforms as the
default. The IOCP crash is an edge case (fast scrolling through many
video previews) that affects few users, while the Python chunked I/O
performance penalty affects everyone.

_stream_file() is kept as an unused fallback for a future compat
setting toggle.
2026-06-13 21:43:44 +08:00
Will Miao a19ddc14f6 perf(preview): restore Linux sendfile, add cache headers, increase chunk size (#975)
- Restrict manual video streaming to Windows only (sys.platform == 'win32');
  Linux/macOS now uses kernel sendfile (zero-copy DMA) via aiohttp FileResponse
- Add Cache-Control: public, max-age=86400 to streaming responses so browsers
  cache video previews across scroll cycles
- Increase chunk size from 256KB to 1MB to reduce async iteration overhead on
  Windows where streaming is still required
2026-06-13 20:06:58 +08:00
Will Miao 7001ced694 fix(rate-limit): respect server retry_after instead of capping at 30s 2026-06-13 18:01:13 +08:00
pixelpaws a5c861646c Merge pull request #974 from itkitteh/fix/socks-proxy-support
fix: support SOCKS proxies for outbound requests
2026-06-13 14:15:02 +08:00
Artem YakimenkoandClaude Opus 4.8 3e0bb73793 fix: support SOCKS proxies for outbound requests
The proxy settings allow selecting a SOCKS proxy type, but the SOCKS
URL was passed to aiohttp's per-request `proxy=` argument, which only
supports http(s) proxies. With a SOCKS proxy this opens a plain TCP
connection to the proxy port and sends an HTTP request; the SOCKS
server replies with its handshake bytes (e.g. b"\x05\xff") and aiohttp
fails with "Bad status line ... Expected HTTP/, RTSP/ or ICE/".

Route SOCKS proxy types through an aiohttp-socks ProxyConnector on the
session instead, leaving the `proxy=` kwarg for http(s) proxies only.
trust_env now keys off whether an app-level proxy is active. Adds
aiohttp-socks to requirements.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 14:05:15 +10:00
Will Miao ac51f6a2f6 feat(settings): add adjustable card overlay blur setting (#973) 2026-06-13 09:43:49 +08:00
Will Miao bef222c77d perf(recipe): precompute image_id_map for O(1) CivitAI image existence checks
Build a civitai_image_id → recipe_id mapping once during cache
initialization instead of scanning all recipes on every
check_image_exists and import_from_url call.

- RecipeCache gains an image_id_map field populated by
  _build_image_id_map() during cache init
- check_image_exists and import_from_url duplicate detection
  now use the precomputed map (O(k) / O(1) vs O(n))
- Map is persisted in SQLite cache_metadata for fast startup
- Incrementally updated on add/remove/bulk_remove paths
- Fix: conn.close() before cache_metadata query (dead connection)
2026-06-13 08:32:03 +08:00
Will Miao 7cd6a53447 fix(downloads): accept optional completed_at in complete_download to preserve original timestamps 2026-06-13 07:06:59 +08:00
willmiao 6850b35770 docs: auto-update supporters list in README 2026-06-12 15:38:33 +00:00
Will Miao 237a015cde chore(release): bump version to v1.1.0 2026-06-12 23:38:16 +08:00
Will Miao 1ae2778baa feat(sidebar): add per-page hide toggle with more options dropdown
- Add ``` button in sidebar header with dropdown menu
- Add "Hide sidebar on this page" option with per-page localStorage state
- Show edge indicator (14px chevron) on left when hidden per-page
- Show brief toast notification when hiding
- Fix container margin not resetting when sidebar is per-page hidden
- Add i18n translations for all 10 locales
2026-06-12 18:27:54 +08:00
Will Miao 84fcdb5f20 fix(recipe): compute folder field on save to prevent reimported recipes disappearing from subfolder grid 2026-06-12 16:49:57 +08:00
Will Miao 8a0b368b44 feat(downloads): add persistent download queue/history with REST API 2026-06-12 15:00:21 +08:00
Will Miao 3990535505 fix(i18n): align bulk reimport label with single context menu, drop 'Metadata' for clarity 2026-06-12 10:19:33 +08:00
Will Miao 3e961a9860 fix(stats): load embeddings from saved stats on startup
_load_stats() was missing the embeddings section, so on every restart
the embeddings usage tracking hash would start from an empty dict.
This caused all previously saved embedding usage data to appear reset.

Added the missing load path for the 'embeddings' key, parallel to the
existing checkpoints and loras loading logic.
2026-06-12 08:57:25 +08:00
Will Miao d6669f1d04 fix(ui): stabilize node selector ordering by type then ID 2026-06-12 08:47:11 +08:00
Will Miao 519bafebc8 fix(i18n): add missing embedding translation keys, sync locales, clean up dead replaceMode branch 2026-06-11 23:03:14 +08:00
Will Miao d87863b423 feat(embedding): send embedding to workflow + fix copy button format
- Fix copy button on embedding cards to copy 'embedding:folder/name' format
- Add send-embedding-to-workflow for Prompt (LoraManager), Text (LoraManager),
  and CLIPTextEncode nodes, appending embedding code to text content
- Extend workflow registry to register text-capable nodes by comfyClass
  (not generic widget name 'text') to avoid false matches
- Add mode parameter to update_node_widget API/event for append support
- Fix single/bulk context menus: single shows plain 'Send to Workflow',
  bulk collapses submenu into direct action for embeddings (append-only)
2026-06-11 22:41:42 +08:00
Will Miao 84e9fe2dfb fix(import): defer git import to module-level to prevent startup crash when git executable missing (#971) 2026-06-11 21:47:55 +08:00
Will Miao 46cbcf94c8 fix(recipe): reimport data loss, local file support, and scroll bugs
- Add local file reimport support via _do_reimport_from_local
- Validate source_path BEFORE deleting old recipe (prevent data loss)
- Move delete_recipe after save_recipe (safe ordering)
- Preserve folder location, NSFW level, and carry over user edits
- Remove old timestamp preservation (use current time)
- Add scrollTop reset in resetAndReloadWithVirtualScroll
- Only reload on successful bulk reimport (avoid empty grid)
- Disable preserveScroll for both single and bulk reimport
2026-06-11 21:31:30 +08:00
Will Miao 05f3018495 refactor(stats): move lora_manager_stats.json from loras root to settings_dir/stats/
- Change _get_stats_file_path() to use get_settings_dir()/stats/ instead of
  first loras root directory
- Add _migrate_from_old_location() to copy existing stats from loras root
  to new location on first access, then clean up old file
- Add 'stats' to update protection skip lists (clean, extract, tracking)
  to prevent data loss during ZIP/git upgrades in portable mode
- Add usage_stats entry to backup targets and restore resolver so stats
  are included in automatic snapshots
2026-06-11 18:03:29 +08:00
Will Miao f565cc35ca feat(stats): track embedding usage from prompt text — Plan A + hybrid approach docs 2026-06-11 17:12:34 +08:00
Will Miao dd1cdce16d fix(ui): unify context menu ordering and add visual section separators across all menus 2026-06-10 22:18:43 +08:00
Will Miao a9e0e7dc8d feat(recipe): add reimport UI with context menus, progress display, and i18n
- Single recipe right-click menu: Re-import from Source
- Bulk context menu: Re-import Metadata for Selected
- Progress overlay with LoadingManager for single and bulk operations
- Virtual scroller data lookup (replaces fragile DOM querySelector)
- Fix dynamic import path for resetAndReload on recipe pages
- Add translation keys for all 9 supported languages
2026-06-10 21:51:04 +08:00
Will Miao b302d1db7d feat(recipe): add reimport endpoint to re-import recipe from source URL
Adds POST /api/lm/recipe/{recipe_id}/reimport that atomically:
1. Reads the existing recipe to extract source_url and user edits
2. Deletes the old recipe files and cache entries
3. Re-downloads the image from CivitAI, re-parses EXIF metadata
4. Carries over user edits (title, tags, favorite) and timestamps
2026-06-10 21:50:43 +08:00
Will Miao 7cbddd9cf7 fix(recipe): fall back to original image for metadata extraction when optimized lacks embedded data (#968)
When CivitAI API returns meta=null and the optimized CDN image has no
embedded generation parameters (e.g. PNG tEXt chunks stripped by
Cloudflare Images), download the original image as fallback to recover
full recipe metadata (prompt, seed, LoRAs, etc.).

Also fixes Chrome password manager popping up on recipe save by adding
autocomplete="new-password" to the settings API key and proxy password
fields.
2026-06-10 15:06:56 +08:00
Will Miao cb8c699224 chore(template): update template workflow 2026-06-10 15:01:48 +08:00
Will Miao 451f74b874 fix(ui): return minWidth/minHeight from autocomplete text widget factory for proper node initial sizing 2026-06-09 15:21:45 +08:00
pixelpaws a1d248baa6 Merge pull request #966 from willmiao/design-token-system-phase4
Design token system phase4
2026-06-09 14:37:02 +08:00
Will Miao 18577fa336 refactor(phase-4): standardize remaining transitions and box-shadows
- Replace all remaining 'transition: all' with specific token-based transitions
- Replace 80+ hardcoded box-shadow rgba values with semantic tokens
- Add new tokens: --shadow-side, --shadow-elevated, --shadow-dialog, --shadow-inset-top
- Update dark theme overrides for new shadow tokens
- 32 files changed, net +8 lines (more consistent, less duplication)
2026-06-09 14:27:53 +08:00
Will Miao 5797ce9408 feat(phase-4): visual polish — font stack, shadow system, transitions, micro-interactions
Phase 4: Visual Polish

4.1 Font Stack Upgrade:
- Add --font-display token for headings
- Replace all hardcoded font-family: monospace with var(--font-mono)
- Replace hardcoded 'Segoe UI' stack with var(--font-body)

4.2 Shadow Elevation System:
- Add --shadow-2xl, --shadow-card/dropdown/modal/toast/header/dark-lg tokens
- Replace hardcoded shadows in header, menu, banner, shared, recipe-modal,
  progress-panel, import-modal, alphabet-bar with semantic tokens
- Add dark theme shadow overrides with increased opacity

4.3 Transitions & Micro-interactions:
- Replace transition: all with specified properties (performance)
- Use --transition-fast/base/slow tokens instead of hardcoded 0.2s/0.3s
- Add :active scale feedback to modal buttons
- Enhance card hover with box-shadow + border-color lift

4.4 Dark Theme Refinement:
- Elevated shadow opacity for dark theme visibility

4.5 Density:
- Standardize container padding with --space-2 token

21 files changed
2026-06-09 14:07:36 +08:00
pixelpaws 826f06255a Merge pull request #964 from willmiao/design-token-system
Design token system phase1
2026-06-09 11:38:31 +08:00
Will Miao 84e16b5c5b refactor(css): remove hardcoded background/border from modal sections - use design tokens instead 2026-06-09 09:52:11 +08:00
Will Miao eb22054580 fix: add --surface-subtle token, restore info grouping, and apply theme-aware favorite color
- Add --surface-subtle (oklch 3% opacity) to replace rgba(0,0,0,0.03)
- Fix info items, creator-info, civitai-view, modal-send-btn, header-actions
  to use --surface-subtle instead of --surface-hover
- Keep true hover states on --surface-hover
- Use light #d4a017 / dark #ffc107 for --favorite-color based on theme
- Replace hardcoded #ffc107 and #d4a017 with var(--favorite-color)
2026-06-09 09:27:11 +08:00
Will Miao 08afb05ece refactor: normalize components in Phase 2
- Unify button styles (padding, gap, border-radius, hover states) in _base.css
- Fix .secondary-btn syntax error (extra space in var())
- Remove duplicated .card-actions in card.css
- Replace hardcoded #f0f0f0 with --surface-hover token
- Replace #ffc107 with accessible #d4a017 for favorite stars
- Replace hardcoded rgba shadows with semantic --shadow-* tokens in layout.css
- Replace hardcoded rgba(0,0,0,0.03)/rgba(255,255,255,0.03) with --surface-hover
- Remove redundant [data-theme=dark] overrides by using theme-aware tokens
- Replace .dropdown-main hardcoded border with --border-color token
2026-06-09 09:26:28 +08:00
Will Miao f51f125cf1 feat: introduce design token system foundation
- Add semantic OKLch color tokens with light/dark themes
- Add typography, spacing, effects, breakpoints, z-index tokens
- Refactor base.css with backward-compatible aliases
- Add prefers-reduced-motion support
- Add MIGRATION.md for Phase 2 component audit
2026-06-09 09:26:28 +08:00
Will Miao 24b2078f21 fix: batch URL download UI polish - hint text, label, and i18n (#936)
- Add .input-hint helper text below textarea guiding multi-URL input
- Update label to CivitAI URL(s): for batch-agnostic hint
- Add urlHint locale key across all 10 languages
- Remove unused url locale key
2026-06-09 07:57:33 +08:00
Will Miao 130fb5d2d5 fix: batch URL download dedup by modelId+modelVersionId composite key (#936)
When batch-downloading different versions of the same model, dedup by
modelId alone discards the second URL. Use modelId:modelVersionId as
the dedup key so users can download, e.g., latest + a specific version.
2026-06-09 07:02:56 +08:00
Will Miao 23c6863a3a fix: batch URL download i18n and CSS polish (#936)
- Add common.actions.remove/change translation keys across all locales
- Remove hardcoded #e74c3c error colors, use --lora-error CSS variable
2026-06-08 21:28:24 +08:00
Will Miao c0e2578640 feat(ui): add adaptive expand/collapse for Additional Notes section (#962) 2026-06-08 20:52:41 +08:00
Will Miao e3c812367e fix(ui): cap lora widget height and enable wheel scroll in Node 2.0 mode (#959)
- Add 'Node 2.0: Maximum visible LoRA entries' setting (default 12)
- Apply max-height to loras container in Vue mode to prevent unbounded growth
- Add enableListWheelScroll: window capture-phase wheel hook so scroll
  inside the widget scrolls the list instead of zooming the canvas
2026-06-08 16:19:08 +08:00
Will Miao 4d239008a6 fix(update): respect hide_early_access_updates in refresh toast count
The refresh_model_updates handler was calling record.has_update() with
default hide_early_access=False, causing the toast to report early-access
updates that the Updates filter (which uses the user's hide_early_access
setting) would then hide. This resulted in misleading "Found N updates"
toasts followed by an empty Updates view.

Now the handler reads hide_early_access_updates from settings and passes
it to has_update(), matching the behavior of _serialize_record and
_annotate_update_flags.
2026-06-08 13:58:21 +08:00
Will Miao 00177a06d0 fix(ui): keep autocomplete text widget at max-height on node resize in Vue mode 2026-06-08 10:49:04 +08:00
Will Miao 568daa351e Revert "Merge pull request #959 from id-fa/fix/lora-loader-list-scroll-nodes2"
This reverts commit 01dac57c35, reversing
changes made to 62f9e3f44a.
2026-06-07 17:25:30 +08:00
Will Miao 5a4664fa12 Merge pull request #936 from 1756141021/feat/batch-url-download
feat: batch URL download for LoRA models
2026-06-06 20:22:52 +08:00
Will Miao dd5b213adc fix(ui): make autocomplete text widget scrollable in Nodes 2.0 mode
In Vue/Node 2.0 mode, the AutocompleteTextWidget's textarea wheel events were intercepted by TransformPane @wheel.capture before reaching the @wheel handler, causing canvas zoom instead of text scrolling.

- Add lm-wheel-scrollable class in Vue mode to hook into the window capture-phase handler (enableListWheelScroll) which scrolls the textarea manually before TransformPane can react.
- Add maxHeight prop and container max-height for Lora Loader/Stacker/WanVideo nodes (modelType === 'loras'), matching canvas mode's height cap. Prompt/Text nodes remain uncapped.
2026-06-06 08:12:09 +08:00
Will Miao d9ee9b3155 fix(utils): catch MemoryError in read_safetensors_metadata for non-safetensors files 2026-06-06 07:35:36 +08:00
pixelpaws 01dac57c35 Merge pull request #959 from id-fa/fix/lora-loader-list-scroll-nodes2
fix(ui): make Lora Loader list scrollable in Nodes 2.0 mode
2026-06-06 07:33:19 +08:00
id-faandClaude Opus 4.8 7f92d09239 fix(ui): make Lora Loader list scrollable in Nodes 2.0 mode
In Nodes 2.0 / Vue node mode the Lora Loader list could not be capped
and the node grew to show every row, unlike classic mode which fixes the
list area to 12 rows. The Vue layout engine measures the rendered DOM, so
CSS variables and computeLayoutSize alone were ignored.

- Physically cap the container via max-height so the rendered element is
  bounded to the 12-row height; extra rows scroll (overflow: auto).
- Report the capped height through computeSize / computeLayoutSize /
  getHeight / getMinHeight so the node background matches the list.
- Add enableListWheelScroll: a window capture-phase wheel hook that scrolls
  the hovered list instead of letting ComfyUI zoom the canvas, which fires
  on the document/canvas in capture and beat a container-level listener.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 20:29:01 +09:00
Will Miao 62f9e3f44a fix(scripts): use platformdirs for cross-platform settings path resolution
Both restore_suffixed_filenames.py and migrate_legacy_metadata.py
hardcoded Path.home() / '.config' / APP_NAME for finding settings.json,
which only works on Linux. On Windows this resolves to the wrong path
(~/.config/ instead of %LOCALAPPDATA%).

Replace the hand-rolled fallback with platformdirs.user_config_dir(),
which correctly resolves to the OS-appropriate config directory on all
platforms (Windows: %%LOCALAPPDATA%%, macOS: ~/Library/Application Support,
Linux: ~/.config). The portable mode check (settings.json in repo root
with use_portable_settings: true) is preserved unchanged.
2026-06-04 07:17:53 +08:00
willmiao e55895786d docs: auto-update supporters list in README 2026-06-03 14:30:44 +00:00
Will Miao 82b77bf593 chore(release): bump version to v1.0.11 2026-06-03 22:30:21 +08:00
Will Miao 1beef5dea9 fix(ui): show title tooltips on disabled showcase media control buttons 2026-06-03 20:33:58 +08:00
Will Miao c8beaa64e1 feat(scripts): add restore_suffixed_filenames script to revert leftover hash suffixes 2026-06-03 20:06:42 +08:00
Will Miao fb443ed6ae perf(recipe): skip CivitAI API calls for locally-known models in create-from-example (#945)
Build a local_cache from the scanner cache before calling the metadata
parser. When a resource hash is found in the cache, populate the entry
directly from cached civitai metadata instead of calling CivitAI's
/model-versions/by-hash endpoint.

This eliminates redundant API calls and retries for the common case
where the example image only uses the parent model plus a checkpoint.
2026-06-03 19:16:52 +08:00
Will Miao 151a467598 feat(recipe): add Create As Recipe from example images with import dedup check (#945) 2026-06-03 19:16:52 +08:00
Will Miao 98e1d168b0 feat(utils): add AutoV2 and AutoV3 hash calculation functions 2026-06-03 19:16:35 +08:00
Will Miao 716f18e0ed chore: remove 'Describe alternatives' section from feature request template 2026-06-02 20:45:43 +08:00
Will Miao b060dc99fc feat(download): add skip-download endpoint that cancels in-memory tracking while preserving partial files on disk 2026-06-02 20:38:47 +08:00
Will Miao 54bcdfab38 fix(test): add folder_path param to DummyUpdateService to match updated interface 2026-06-02 19:02:18 +08:00
Will Miao 2e7532eecc feat(update): add per-folder update check via sidebar context menu (#944) 2026-06-02 18:34:01 +08:00
Will Miao 7e5e3b1ec7 feat(download): support multi-precision file selection for CivitAI model downloads (#956) 2026-06-02 15:41:42 +08:00
Will Miao df67bd396a fix(recipe): re-export syncChanges and add show mock to fix test 2026-06-02 11:02:20 +08:00
Will Miao dd5d9cfcb2 fix(recipe): align refresh split button behavior with models page
- refreshRecipes() now accepts fullRebuild param and passes it to scan endpoint
- Use consistent toast.api.refreshComplete / toast.api.refreshFailed keys
- Use loadingManager.show() with progress bar (matching models page style)
- Both Refresh and Rebuild Cache now hit the real /api/lm/recipes/scan endpoint
- Add sidebarManager.refresh() after recipe scan completes
- Backend scan_recipes handler reads full_rebuild query param
2026-06-02 09:50:59 +08:00
Will Miao d9fd60bec1 fix(recipe): use VirtualScroller pageSize in reload helpers to prevent pagination offset gap 2026-06-02 08:43:30 +08:00
Will Miao b633b22779 fix(recipe): prevent empty grid by removing preserveScroll from refresh triggers
Bug: when scrolling down on recipes page, any operation with
preserveScroll: true would fetch only page 1 data then restore
scroll position to beyond the loaded items, leaving the grid empty.

Fix:
- Remove preserveScroll: true from all 7 must-refresh trigger
  paths (filter, search, sort, import, settings reload, sync,
  rebuild cache, sidebar folder nav)
- Replace full list refresh with updateSingleItem() for repair
  and bulk missing-LoRA download operations
- Update tests to match new scroll-free behavior
2026-06-02 08:15:29 +08:00
Will Miao 1ffa543160 fix(recipe): set dataset.favorite on recipe cards for correct bulk favorite menu 2026-06-02 07:06:58 +08:00
Will Miao cdc940586e fix(civarchive): infer metadata.format from extension and prioritize safetensors in file list 2026-06-01 22:07:55 +08:00
Will Miao ccf1c6f2ae fix(recipe): resolve base_model from parser and prevent empty checkpoint save on CivitAI import
- Apply CivitaiApiMetadataParser's base_model result to metadata in
  _do_import_remote_recipe and _do_import_from_url (was previously discarded)
- Extract baseModel from raw civitai_info before populate_checkpoint_from_civitai
  so it's not lost when the type check rejects non-checkpoint model versions
- Only format and save checkpoint entry when it has real data (modelId, versionId,
  name, or version), preventing empty {'type': 'checkpoint'} stubs
2026-06-01 17:58:08 +08:00
Will Miao bfe7b5e1c7 fix(constants): add missing diffusion model base models (Flux, DiT, video, etc.) 2026-05-31 17:12:09 +08:00
Will Miao 85c020cd12 fix(update): preserve wildcards, backups dirs during ZIP upgrade, add log rotation
- Add wildcards and backups to skip_files in all three ZIP upgrade
  skip locations: _clean_plugin_folder, copy loop, .tracking generation
- Remove logs from skip_files (logs are transient and rotate automatically)
- Add _prune_old_logs() to session_logging.py: keeps only the 3 newest
  session log files, deletes older ones on each standalone startup
2026-05-31 15:56:56 +08:00
Will Miao 1b202f8ec7 fix(autocomplete): escape parentheses in prompt tag insertion (#951) 2026-05-31 15:40:19 +08:00
Will Miao d02a0611d3 fix(update): close SQLite connection and protect cache dir during ZIP update
On Windows, shutil.rmtree() fails when deleting a directory that contains
an open SQLite database file. The ZIP update path in _download_and_replace_zip()
calls _clean_plugin_folder() which tries to delete the cache/ directory,
but downloaded_versions.sqlite is held open by DownloadedVersionHistoryService.

Fix:
- Add close() method to DownloadedVersionHistoryService to release
  the persistent SQLite connection
- Call close() before _clean_plugin_folder() in the ZIP update flow
- Add 'cache' to the skip_files list so the runtime cache directory is
  never deleted during plugin updates
2026-05-31 15:06:15 +08:00
pixelpaws 92166a161a Update Portable Package link to version 1.0.10 2026-05-31 10:08:28 +08:00
Will Miao b509f27cb7 chore(release): bump version to v1.0.10 2026-05-31 09:39:26 +08:00
Will Miao 5c2ef48917 fix(aria2): apply certifi CA bundle to aria2c via --ca-certificate
When certifi is available, pass its CA bundle path as --ca-certificate
to the aria2c subprocess so that aria2 downloads use the same
certificate store as Python aiohttp downloads. Graceful fallback when
certifi is not installed.
2026-05-30 21:47:13 +08:00
Will Miao ad2bd82c67 fix(downloader): use certifi CA bundle as SSL fallback and log SSL error diagnostics
- Prefer certifi's CA bundle in aiohttp SSL context with graceful
  fallback to system default when certifi is unavailable
- Add is_ssl_cert_verify_error() helper for SSL cert failure detection
- Log actionable error message (pip install --upgrade certifi /
  pip install pip-system-certs) when SSL certificate verification fails
- Apply same diagnostic logging to aria2 redirect resolution path
2026-05-30 21:28:18 +08:00
willmiao 17ba350153 docs: auto-update supporters list in README 2026-05-28 13:47:09 +00:00
Will Miao 60175334b5 chore(release): bump version to v1.0.9 2026-05-28 21:46:46 +08:00
Will Miao f65a01df00 feat(recipe): add bulk Repair Metadata for Selected operation to recipes page
Adds a new bulk operation in the recipes page that allows users to select
multiple recipes and repair their metadata in batch.

Backend:
- New POST /api/lm/recipes/repair-bulk endpoint accepting recipe_ids array
- repair_recipes_bulk handler iterates repair_recipe_by_id for each recipe
- Response includes per-recipe updated data for frontend card refresh

Frontend:
- Bulk context menu: new 'Repair Metadata for Selected' item in Metadata section
- BulkManager.repairSelectedRecipes() with loading/toast flow
- Uses VirtualScroller.updateSingleItem() per repaired recipe (no full reload)
- Visibility controlled via repairMetadata actionConfig flag

Locales:
- Added repairMetadata, repairBulkComplete, repairBulkSkipped, repairBulkFailed
- Translated across all 9 supported languages
2026-05-28 20:16:59 +08:00
Will Miao 430e24d70b fix(ui): hide skip-metadata-refresh bulk menu items for recipes 2026-05-28 19:11:49 +08:00
Will Miao 14f0c48fdd fix(recipe): detect and repair corrupted checkpoints in repair flow
Add corruption detection to _repair_single_recipe: if checkpoint.modelVersionId matches any LoRA's modelVersionId, the checkpoint is corrupted (a LoRA was saved as checkpoint). Clear the checkpoint and remove the matching LoRA entry, then let enrichment re-resolve the correct checkpoint from CivitAI metadata.

This fixes the retroactive repair path for the modelVersionIds[0] fallback bug.
2026-05-28 17:19:27 +08:00
Will Miao 34791c2ad7 fix(recipe): use resources type field to identify checkpoint instead of modelVersionIds[0]
When importing a CivitAI image as a recipe, modelVersionIds[0] was blindly used as the checkpoint version ID. This array mixes checkpoints and LoRAs without ordering guarantees, causing LoRAs to be saved as the recipe checkpoint.

Fix by:
1. Removing the modelVersionIds[0] fallback in _download_remote_media
2. Parsing resources entries with type:"model" as the checkpoint
3. Adding model type validation in populate_checkpoint_from_civitai

Also add 2 tests for the new behavior and fix 3 tests whose mocks lacked the required model.type field.
2026-05-28 15:46:38 +08:00
Will MiaoandClaude Opus 4.7 3f6824eef6 fix(example-images): exclude failed_models from check_pending_models pending count
Previously check_pending_models() only skipped models already in
processed_models, so models that had permanently failed (no CivitAI
images available, download errors) were forever reported as "pending".
This caused repeated auto-download cycles with no actual work to do.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-28 12:00:25 +08:00
Will MiaoandClaude Opus 4.7 3919dfa3f4 fix(metadata): suppress rate-limit propagation when model already confirmed deleted
When CivitAI returns 404 (ResourceNotFoundError) and a fallback provider
like CivArchive subsequently rate-limits, the ChainedMetadataProvider
now suppresses the RateLimitError instead of propagating it. Previously,
the rate-limit error would bubble up through _refresh_single_model and
cause the outer retry loop to re-process the same model repeatedly,
producing dozens of duplicate "Model X is no longer available" log
messages and wasting API quota.

The model is NOT permanently marked as ignored — its last_checked_at
timestamp is preserved, so it will be retried on the next refresh cycle
when the rate limit has cleared and CivArchive may still have the data.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-28 11:56:22 +08:00
Will Miao 7124b5293f chore(settings): remove unused example_images config, add unet folder_paths example 2026-05-27 19:58:56 +08:00
Will Miao d2a04f8993 fix(model-hash-index): clean up AutoV2 entry in remove_by_hash 2026-05-27 19:38:08 +08:00
pixelpaws 7027a7c270 Merge pull request #946 from 1756141021/fix/autov2-hash-matching
fix: match local LoRAs by AutoV2 hash when Civitai model is deleted
2026-05-27 19:20:31 +08:00
heinandClaude Opus 4.6 0a1d7dfd4c fix: match local LoRAs by AutoV2 hash when Civitai model is deleted
When recipe metadata contains AutoV2 hashes (10-char short hash from
image metadata) and the Civitai API cannot resolve them to SHA256
(model deleted, API offline), the local hash index failed to match
because it only stored full SHA256 hashes.

AutoV2 is simply SHA256[:10], so we derive it automatically in
add_entry() — no extra file I/O or schema changes needed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-27 14:15:01 +08:00
Will Miao 3962b1a96d fix(civitai): fall back to direct version fetch when modelVersions is empty for newly published models 2026-05-27 06:40:13 +08:00
Will Miao 8b856276bf fix(ui): escape HTML entities in parseMarkdown to prevent swallowed angle brackets 2026-05-27 06:40:13 +08:00
willmiao c97c802956 docs: auto-update supporters list in README 2026-05-26 13:27:45 +00:00
Will Miao 24e2909627 chore(release): bump version to v1.0.8 2026-05-26 21:27:29 +08:00
Will Miao b768f1368f fix(i18n): update aria2 annotation from experimental to recommended across all locales 2026-05-26 20:22:25 +08:00
Will Miao 37ccd29fc0 feat(modal): make version name editable in model modal (#931) 2026-05-26 20:16:35 +08:00
Will MiaoandSisyphus 7416080cfb fix(civitai): retry transient server errors and cache version info to reduce 504 timeouts
CivitaiClient._make_request now retries 5xx/524/network errors up to 3 times with exponential backoff (1s, 2s) before giving up to the fallback provider chain.

get_model_version_info gains an in-memory OrderedDict cache (LRU, max 500 entries) so duplicate lookups of the same version ID within a single import/scan flow return instantly without a redundant API call.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-05-26 16:09:08 +08:00
Will MiaoandSisyphus 26be187d42 fix(i18n): translate remaining loraSyntaxFormat TODO keys across all locales
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-05-26 06:15:57 +08:00
Will Miao d7caa1fa47 fix(license): remove cascading commercial-use bit encoding, clarify Allow Selling label (#941)
- _resolve_commercial_bits() no longer has Sell-implies-Image
  cascading; each CommercialUse value sets only its own bit,
  matching CivitAI's modern array-format API.
- Keep filter tag label as 'Allow Selling' for brevity; add
  title/tooltip 'Allow selling generated images' on hover.
- Same tooltip treatment for 'No Credit Required'.
- Add i18n keys for both tooltips across all 10 locales.
2026-05-26 06:02:17 +08:00
Will MiaoandSisyphus 2629fcce23 fix(doctor): add i18n translations for check items, action buttons, and labels
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-05-25 22:35:48 +08:00
Will MiaoandSisyphus 438e7d07b9 fix(i18n): add missing conflictConfirm.detail and conflictConfirm.impact keys to all locales
These keys are referenced in DoctorManager.js via translate() calls but were never added to any locale file, causing the i18n regression test to fail.

Added to all 10 locales: en, zh-CN, zh-TW, ja, ko, ru, de, fr, es, he.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-05-25 22:25:13 +08:00
Will Miao e9932ea870 feat(tags): add right-click context menu with copy for trigger word tags
- Add showTagContextMenu() with Copy option for all tags,
  plus Edit Group for multi-item group tags
- Attach contextmenu listener to simple tags
- Move group tag contextmenu outside items.length > 1 guard so
  single-child groups also get the context menu (bugfix)
- Clean up hanging context menu on re-render
2026-05-25 22:16:54 +08:00
Will Miao 5dd8b96422 fix(autocomplete): reactively refresh lora syntax format cache on settings change (#917)
The autocomplete module cached the lora_syntax_format value at module load
but never updated it when the setting changed, causing autocomplete to
always insert legacy A1111 format even when 'full path' was configured.

- Expose refreshLoraSyntaxFormat() to re-fetch the setting from the API
- Listen for cross-tab 'storage' events to react to settings saved in
  the standalone web UI
- Listen for 'visibilitychange' to refresh when the user switches back
  to the ComfyUI tab
- Wire SettingsManager.saveSetting() to set a localStorage key when
  lora_syntax_format changes, triggering the storage event
2026-05-25 22:03:56 +08:00
Will Miao 5e1cf68bbd fix(settings): sync loraSyntaxFormat select value from state on modal open (#917)
was missing the line to set the
select element's value from ,
causing the dropdown to always show the first option ("Full Path")
when reopening the settings modal, regardless of the persisted value.
Runtime behavior was unaffected since  reads from
the state directly.
2026-05-25 21:35:15 +08:00
Will Miao 1044fa3c83 feat(doctor): improve duplicate filename conflict UX with confirm modal, syntax-format nav, and i18n
- Remove [LoRAs] prefix noise from conflict detail display
- Limit inline conflict groups to 5, show remainder count
- Add 'Switch to Full Path Syntax' action in conflict card
- Add confirmation modal before resolving conflicts (shows rename strategy)
- Register resolveFilenameConflictsModal in ModalManager (fix no-op showModal)
- Switch to Interface section and add highlight animation on syntax-format nav
- Sync and translate conflictConfirm strings across all 10 locales
2026-05-25 21:25:35 +08:00
Will Miao 397892bb7f fix(recipe): treat transient server errors (524/5xx) as non-fatal in image info fetch
Extend _is_transient_server_error() check introduced in 15dfaed4 to
get_image_info(), so Cloudflare 524 and generic 5xx errors during
remote recipe import are logged as info instead of error and do not
produce scary tracebacks.

Same pattern as get_model_versions() - transient upstream failures
return None gracefully rather than being logged as errors.
2026-05-25 08:35:35 +08:00
Will Miao f105500740 feat(doctor): suppress duplicate filename warnings when full path syntax is active (#917) 2026-05-22 22:35:06 +08:00
Will Miao 806555cf06 fix(test): update autocomplete test expectations for legacy lora syntax format (#917) 2026-05-22 21:56:38 +08:00
Will Miao 5cd7204101 fix(autocomplete): prevent blur-on-click race condition causing dropped selection (#939)
Add mousedown(e.preventDefault()) on dropdown items to prevent the textarea blur event from firing before click. Without this, the blur handler's formatAutocompleteTextOnBlur() modifies text with unmatched commas (e.g. "<lora:X:1>,search") and triggers hide() via suppressAutocompleteOnce, removing the item from the DOM before the click handler can execute.

Fixes #939
2026-05-22 21:50:26 +08:00
Will MiaoandSisyphus 3b602a3698 feat(lora): add lora_syntax_format setting for syntax version toggle (#917)
Adds lora_syntax_format setting (full/legacy) that controls whether <lora:...> syntax uses relative paths (full) or filename only (legacy). Default is legacy for backward compatibility with A1111 convention. The full path format (<lora:relative/path/filename:strength>) enables lossless model resolution across subfolders.

Ultraworked with Sisyphus (https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-05-22 21:03:29 +08:00
Will MiaoandSisyphus 15dfaed462 fix(api): treat transient server errors (524/5xx) as non-fatal in model updates (#935)
Teach CivitaiClient.get_model_versions() to recognise Cloudflare 524, generic
5xx, and connection-level errors as transient failures and return None
instead of raising RuntimeError, so a single upstream glitch does not
block the entire batch update or produce a scary traceback.

Also downgrade the generic except Exception log level in
ModelUpdateService._refresh_single_model() from error (with exc_info)
to warning (message only), since the full traceback is already logged
upstream in CivitaiClient.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-05-22 07:05:06 +08:00
Will MiaoandSisyphus 0e51851025 fix(preview): stream video files manually to avoid Windows sendfile crash
aiohttp's FileResponse uses _sendfile_native on Windows (IOCP-based), which crashes with ov.getresult() when the client disconnects mid-transfer. This happens constantly when users scroll through a gallery of animated previews (video files like .mp4/.webm).

Detect video extensions and stream manually via StreamResponse + chunked reads instead, gracefully handling ConnectionResetError. Images continue using FileResponse (small files, sendfile works fine).

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-05-21 09:12:10 +08:00
Will Miao 0d0f4defca feat(recipes): enable bulk Add Tags to Selected for recipes (#934)
- Set addTags: true in recipes bulk action config
- Add _saveRecipeTags() helper using recipe API endpoint
- Replace mode: saves tags array directly via PUT recipe/update
- Append mode: merges with existing tags from virtual scroller
- Shows bulk Add Tags modal & target menu item on recipes page
2026-05-20 23:14:38 +08:00
Will Miao 818fa34a48 fix(ui): auto-focus tag input and flush uncommitted text on save (#934)
- ModelModal (ModelTags.js): auto-focus input on entering tag edit mode
- ModelModal (ModelTags.js): flush uncommitted input text as tag on Save
- Bulk Add Tags (BulkManager.js): same two fixes
- RecipeModal already handled both cases correctly
2026-05-20 23:06:40 +08:00
Will Miao 78303b2a5e feat(ui): merge user tags into auto-tag badges and refresh on tag edit (#918)
- Layer 2 fallback: user tags overlapping with auto-tag categories
  (HIGH/LOW/I2V/T2V/TI2V/Lightning/Turbo) are merged into auto_tags,
  providing manual override when filename-based detection fails.
  Matching is case-insensitive so "high"/"High"/"HIGH" all work.
- Refresh on tag edit: save_metadata and add_tags handlers now return
  recalculated auto_tags in the response; the frontend passes them to
  VirtualScroller.updateSingleItem so badges update immediately without
  requiring a page reload.
- 8 new test cases for Layer 2 fallback and case-insensitive matching.
2026-05-20 22:48:44 +08:00
Will Miao 9ce56dd40c feat(lora): support relative paths in <lora:folder/name:strength> syntax (#917)
Autocomplete, copy/send-to-workflow, and recipe syntax now emit
<lora:folder/name:strength> instead of <lora:name:strength>, using
relative paths to disambiguate identically-named loras in different
subfolders without requiring file renames.

Backend: 3-tier hybrid resolution (path → bare → basename fallback)
across get_lora_info, get_lora_info_absolute, get_model_preview_url,
get_model_civitai_url, get_model_info_by_name, get_lora_metadata_by_filename,
and get_hash_by_filename. Also fix get_random_loras and get_cycler_list
to return path-prefixed names for randomizer/cycler consistency.

Frontend: autocomplete, copyLoraSyntax, handleSendToWorkflow emit
folder-prefixed syntax. extract_lora_name preserves relative paths.

Saved image metadata (<lora:...> in EXIF) intentionally keeps basename-only
for compatibility with A1111/Forge ecosystem.
2026-05-20 19:39:12 +08:00
heinandClaude Opus 4.6 4e3ede23b7 feat: batch URL download for LoRA models
Add multi-URL batch download support to the download modal.
Users can paste multiple CivitAI URLs (one per line) in a textarea,
preview all parsed models in a compact list, optionally change versions
per model, select a unified download path, and batch download sequentially.

Single URL behavior is preserved unchanged.

Changes:
- Replace single-line input with textarea for multi-URL input
- Add batch preview step with compact list (thumbnail, version, size)
- Per-item version editing via existing version selector
- Batch download with WebSocket progress tracking (reuses existing infra)
- URL deduplication by model ID, preserving paste order
- Invalid URLs shown inline with remove option
- Fix: prevent click listener accumulation in showVersionStep

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-20 11:37:36 +08:00
Will Miao 33e5f3d85d fix(#933): compute SHA256 locally when CivitAI API returns empty hashes 2026-05-18 18:30:33 +08:00
Will Miao 031d5e4f40 fix(doctor): exclude checkpoints/embeddings from duplicate filename detection (#934)
Duplicate filename detection is only relevant for LoRAs, which use
basename-only syntax (<lora:name:strength>). Checkpoints and diffusion
models reference files via relative paths with extensions, so filename
conflicts there are false positives — there is no resolution ambiguity.

Both _log_duplicate_filename_summary() and DoctorHandler's
_check_filename_conflicts() now skip scanners with model_type != 'lora'.
2026-05-18 13:57:28 +08:00
willmiao 4ff5774e34 docs: auto-update supporters list in README 2026-05-17 12:40:26 +00:00
Will Miao 94e1a8ac7b chore(release): bump version to v1.0.7 2026-05-17 20:40:13 +08:00
Will Miao cc20d3b992 feat(ui): auto-detect HIGH/LOW badges and auto-tag filters (#918)
- Backend auto-tag extraction service: detect HIGH/LOW (Wan-only), I2V/T2V/TI2V,
  Lightning/Turbo from filename, base_model, and CivitAI version name
- HIGH/LOW badge in card footer (inline before version name), color-coded:
  blue for HIGH, teal for LOW; abbreviated to H/L in medium/compact density
- Auto-tag filter panel (I2V, T2V, TI2V, Lightning, Turbo) with tri-state
  include/exclude filtering
- Full filter pipeline: FilterCriteria → ModelFilterSet → baseModelApi params
- AUTO_TAG_GROUPS exported for frontend use
- 19 unit tests for auto-tag extraction edge cases
2026-05-17 17:45:12 +08:00
Will Miao a74cbe7aa2 fix(test): sync civitai bulk test with nsfw param 2026-05-16 22:15:55 +08:00
Will Miao 94edfaa190 fix(import): discover all resources from CivitAI modelVersionIds
CivitAI image API returns modelVersionIds at the root level of the
response (not inside meta), containing ALL model version IDs across
all resources (checkpoint + LoRAs). Two bugs prevented LoRAs from
being discovered:

1. _download_remote_media only extracted the first modelVersionId for
   enrichment, dropping the rest.
2. CivitAI API meta parsing only ran as an EXIF fallback, but most
   images have embedded EXIF metadata (prompt, steps, etc.), so the
   fallback was never triggered.
3. When civitai_meta_raw itself has a nested 'meta' key, unwrapping
   it stripped the injected modelVersionIds.

Also fixed gen_params merge: API gen_params now overlays EXIF at the
field level instead of full replacement, preserving EXIF-only fields
like detailed generation parameters.
2026-05-16 22:12:30 +08:00
Will Miao 31c54ff068 fix(civitai): add nsfw param to user-models and batch-ids queries (#930)
The CivitAI /api/v1/models endpoint defaults to filtering out NSFW
content when the nsfw query parameter is omitted. Both get_user_models()
and get_model_versions_bulk() hit this endpoint without passing nsfw=true,
causing models whose nsfwLevel doesn't include the PG bit to be silently
dropped from results.

Add nsfw=true to both call sites so all browsing levels are returned.
2026-05-16 20:15:03 +08:00
Will Miao 21872a8e9e fix(ui): default_active in group mode should not propagate to children; hide group badge/edit for single-child groups (#929) 2026-05-16 16:52:06 +08:00
Will Miao 612612f1c7 feat(ui): add Open Source URL action to recipe modal header, align header styles with model modal 2026-05-16 16:11:14 +08:00
Will Miao ff240db5b1 chore: reduce remote recipe import log verbosity, demote detail fields to debug 2026-05-15 21:04:09 +08:00
Will Miao bcfed4b874 feat(ui): use recipes terminology in bulk delete confirmation for recipes page
The bulk delete confirmation modal always displayed "models" in its
text (title, message, countMessage) regardless of the current page
type. On the recipes page this is misleading since users are managing
recipes, not models.

- Add bulkDeleteRecipes i18n keys to all 10 locale files
- Update showBulkDeleteModal() to detect currentPageType and use
  recipes-specific wording when on the recipes page
2026-05-15 20:55:02 +08:00
Will Miao 1352c6ecbe fix(recipes): fall back to Civitai API meta when EXIF is empty, enrich checkpoint in analyze_remote_image
- When downloaded Civitai image has no embedded EXIF, parse the
  already-fetched Civitai API meta (resources, hashes) directly
  instead of skipping parser altogether.
- Extract loras and model from parser output to fill metadata gaps
  when the primary import path doesn't provide them.
- Read modelVersionIds[0] as fallback when modelVersionId is None
  (Civitai API returns both but the singular form can be absent).
- Run RecipeEnricher in analyze_remote_image before returning, so
  the LM UI receives complete metadata including checkpoint with
  zero additional API calls (reuses the image_info already fetched).
2026-05-15 20:31:34 +08:00
Will Miao 30b01b8a92 fix(recipes): offload EXIF to thread pool, throttle concurrent imports, eliminate duplicate Civitai API call
- Wrap ExifUtils.extract_image_metadata() with asyncio.to_thread() in
  both import handlers and analysis_service to prevent Pillow/piexif
  from blocking ComfyUI's event loop during batch imports.
- Add asyncio.Semaphore(2) to import_remote_recipe and import_from_url
  endpoints to cap concurrent heavy work and prevent event loop starvation.
- Pre-fetch Civitai image_info during download and pass it to the recipe
  enricher, eliminating a redundant get_image_info() API round-trip.
2026-05-15 18:29:54 +08:00
Will Miao a105cb322b fix(metadata): prune stale example-image entries when files are deleted on disk (#927) 2026-05-14 20:51:33 +08:00
Will Miao 3bf396d003 feat(recipes): add toggle to strip <lora:> tags when copying prompt/negative_prompt
Adds a compact inline toggle in the Generation Parameters section of the
Recipe Modal that, when enabled, strips <lora:name:weight> tags and
cleans up residual punctuation before copying to clipboard. The setting
persists across sessions via localStorage.
2026-05-13 11:47:02 +08:00
Will Miao 60cfb3b8e0 chore: add .sisyphus/ to .gitignore 2026-05-13 09:30:26 +08:00
Will Miao 6763abb83c fix(test): update test recipes to use source_path instead of source_url
Follow-up to 86118d06 which consolidated on source_path but missed updating these two tests.
2026-05-13 09:27:05 +08:00
Will Miao 5c53968caa refactor(download-history): rename mark_not_downloaded to mark_as_deleted
The method mark_not_downloaded() was misleading — it doesn't negate
'downloaded' history (the model was indeed downloaded before), but
rather sets is_deleted_override = 1 to indicate the version was
downloaded and subsequently deleted. This flag allows re-download when
the 'skip previously downloaded' setting is enabled.

Rename to mark_as_deleted() to accurately reflect its semantics.
2026-05-12 22:50:30 +08:00
Will Miao b4f7dd75af fix(persistent-cache): persist scanner cache after model deletion
After deleting a model, the in-memory scanner cache was updated but the
SQLite persistent cache was not. On server restart, the stale persistent
cache caused check_model_version_exists() to return True, blocking
re-download with 'Model version already exists'.

Add _persist_current_cache() calls in both deletion paths:
- ModelLifecycleService.delete_model() (used by versions tab delete)
- delete_model_version handler in MiscHandlers
2026-05-12 22:50:10 +08:00
Will Miao 86118d0654 fix(recipes): persist source_path in SQLite cache and eliminate source_url redundancy
- Add source_path column to PersistentRecipeCache SQLite schema with
  migration for existing databases (ALTER TABLE ADD COLUMN)
- Backfill source_path from recipe JSON files on first startup after
  migration to avoid requiring manual cache rebuild
- Remove all source_url recipe field references (import_remote_recipe,
  import_from_url, check_image_exists, enrichment, batch_import)
  and consolidate on source_path as the single source of truth
- Add civitai.green to supported Civitai page hosts
- Register check-image-exists and import-from-url recipe endpoints
2026-05-12 20:39:09 +08:00
Will Miao df1410535e fix(ui): remove redundant Quick Refresh from Refresh split button dropdown
The main Refresh button and Quick Refresh dropdown item both called refreshModels(false). Split button dropdowns should only contain alternative actions (Hick's Law). Dropdown now has only Rebuild Cache (fullRebuild=true). Removed from 2 templates, 2 JS files, 1 test fixture, and 10 locale files.
2026-05-12 07:50:54 +08:00
Will MiaoandClaude Opus 4.7 75f74d54d8 feat(bulk): reorganize context menu with sections and submenu for workflow actions
Group 15 flat menu items into 5 logical sections (Workflow, Metadata,
Attributes, Organize, Download) with section headers to reduce cognitive
load. Nest the three workflow-related actions (Append, Replace, Copy
Syntax) into a single "Send to Workflow" hover-triggered submenu.

Add submenu infrastructure to BaseContextMenu with mouseover/mouseout
boundary detection, 250ms close delay, and viewport-aware positioning.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-11 21:06:47 +08:00
Will MiaoandClaude Opus 4.7 ab6100f596 feat(bulk): add "Download Example Images" to bulk select context menu (#923)
Allows downloading example images only for selected models instead of
the entire library. Reuses the existing /api/lm/force-download-example-images
endpoint which already accepts an array of model hashes.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-11 18:05:00 +08:00
Will Miao 5d3ab3bbf8 feat(showcase): click-to-view full-size image/video in recipe and model modals (#926)
- Add MediaViewer overlay for full-size image/video display with prev/next
  navigation, direction keys, counter, and adjacent preloading
- Recipe modal: click preview image/video opens full-size viewer
- Model showcase: click any example image/video opens viewer with full
  gallery navigation; blurred NSFW content opens directly to clear view
- Use Map<Element, number> for DOM-index mapping instead of URL comparison
  to avoid index mismatch from lazy-loaded vs data-attribute URLs
2026-05-10 22:22:24 +08:00
Will Miao d9dc0dba8d perf(startup): load extra model paths during Config init to avoid double symlink scan
Move extra folder path resolution from _initialize_services (app.on_startup)
into Config.__init__ via new _load_extra_paths_from_settings() method.
This eliminates a redundant second symlink scan and consolidates all
'Found roots' / 'Found extra roots' logs into one contiguous block
during custom node import, before the ComfyUI server starts.
2026-05-08 14:55:53 +08:00
Will Miao 3631c5eb10 chore: bump version to 1.0.6 2026-05-07 18:59:00 +08:00
Will Miao 6d5b4b7312 fix(test): update drag interaction test to match 454210a4's renderFunction→setValue change
Commit 454210a4 replaced renderFunction() with widget.value setter +
widget.callback() in endDrag, so the test assertion should verify
callback invocation instead of the removed renderSpy call.
2026-05-07 11:03:38 +08:00
Will Miao 7803bd542d feat(base-models): add Ernie, Ernie Turbo, Nucleus base model types (#922)
- Ernie & Anima: auto-fetched via CivitaiBaseModelService from Civitai API
- Ernie Turbo & Nucleus: pre-added as hardcoded constants (not yet in Civitai API)
- Added abbreviations (ERNI, ETRB, NUCL) and category entries across all layers
2026-05-07 10:49:01 +08:00
Will Miao f0a86dbbc0 feat(bulk): add bulk favorite/unfavorite toggle with context-sensitive single menu item
Replaces two separate menu items with a single smart item that dynamically
switches between 'Set as Favorite' and 'Remove from Favorites' based on
whether all selected items are already favorited. Shows a count badge
'(3/5)' when only some items are favorited in a mixed selection.

Supports all model types (LoRA, Checkpoint, Embedding) and recipes via
existing per-item save/update API — no backend changes needed.
2026-05-07 09:51:23 +08:00
Will Miao 682e964f89 fix(usage-control): enrich usageControl from CivitAI by-hash API for all model types
The model-level API (GET /api/v1/models/{id}) does not include usageControl
on version entries, causing generation-only models to show as downloadable.

Backend changes:
- Add get_model_versions_by_hashes() to CivitaiClient (POST by-hash batch)
- Propagate through all provider classes including RateLimitRetryingProvider
- Add _enrich_version_entries() pipeline: extract SHA256 from files[].hashes,
  batch-call by-hash endpoint, inject usageControl+earlyAccessEndsAt in-place
- Wire enrichment into both bulk (_fetch_model_versions_bulk) and individual
  (_refresh_single_model) refresh paths
- Fix _build_record_from_remote dropping usage_control field
- Fix POST by-hash request format (plain JSON array, not {hashes:[...]} object)

Frontend changes:
- Fix disabled download button tooltip: wrap in <span> since HTML title
  attribute does not fire on disabled elements
2026-05-07 08:56:19 +08:00
Will Miao 908464bc0a docs: remove inline release notes from README (now maintained via GitHub Releases) 2026-05-06 22:40:06 +08:00
willmiao 0ffee3a854 docs: auto-update supporters list in README 2026-05-06 10:29:43 +00:00
Will Miao 8aa9739c44 data: refresh supporters from license server (739 supporters, includes Patreon data) 2026-05-06 18:29:21 +08:00
Will Miao 50739bbb43 fix(css): remove dead CSS properties causing Biome errors
- batch-import-modal.css: add generic font family fallback to Font Awesome
- card.css: remove dead margin-left overridden by shorthand margin: 0
- shared.css: remove duplicate position: absolute overridden by position: fixed
2026-05-06 09:33:15 +08:00
Will Miao e849303763 fix(header): eliminate search input focus layout shift and reduce focus ring size
- Remove transform: translateY(-1px) that caused layout shift on focus
- Reduce box-shadow focus ring from 2px to 1px for subtler appearance
- Tone down drop-shadow from 4px/16px to 2px/8px (matches base state)
2026-05-06 09:33:04 +08:00
Will Miao 241b2e15d2 docs: update extension image URL 2026-05-05 22:26:40 +08:00
Will Miao 88da754504 docs: migrate wiki-images to wiki repo, remove stale docs
Moved wiki-images to the wiki repo (willmiao/ComfyUI-Lora-Manager.wiki). Updated README.md image reference to use wiki raw URL. Removed docs/LM-Extension-Wiki.md (superseded by wiki pages).
2026-05-05 22:20:19 +08:00
Will Miao b4a706651f feat(delete-model-version): add GET endpoint to delete a model version by version ID 2026-05-05 21:25:08 +08:00
pixelpaws ff7cc6d9bb Merge pull request #921 from 1756141021/fix/drag-strength-notify-setValue
fix: commit dragged strength through options.setValue at drag end
2026-05-05 16:20:48 +08:00
hein 454210a47c fix: commit dragged strength through options.setValue at drag end
During drag, handleStrengthDrag is called with updateWidget=false, which
mutates widgetValue in-place via parseLoraValue's direct array reference,
bypassing widget.value setter and options.setValue entirely.

endDrag only called renderFunction for a DOM refresh, but never flushed the
mutation through options.setValue. Any external observer that wraps
options.setValue (e.g. ComfyUI Mirror Panel's bidirectional sync) would
therefore never see the dragged value and would treat the widget as unchanged.

Fix: replace the explicit renderFunction call with widget.value = widget.value.
This flushes the in-place mutation through the setter (options.setValue), which
re-renders the DOM internally AND notifies all setValue wrappers. Also fire
widget.callback for parity with the updateWidget=true path in handleStrengthDrag.

Applies the same fix to initHeaderDrag (proportional all-LoRA header drag).
2026-05-04 22:40:30 +08:00
Will Miao 2d7c404ebb fix(recipes): preserve scroll position on filter, search, and folder-driven reloads
Five entry points that trigger recipe page reloads were not passing
preserveScroll: true, causing the page to snap back to top after
filtering, searching, or navigating folders — especially painful with
hundreds of recipes.

- RecipePageControls.resetAndReload() → refreshVirtualScroll() now
  passes { preserveScroll: true } (sidebar folder clicks/drag moves)
- FilterManager applyFilters/clearAllFilters → loadRecipes(true)
  changed to loadRecipes({ preserveScroll: true })
- SearchManager performSearch → loadRecipes(true) changed to
  loadRecipes({ preserveScroll: true })
- SettingsManager reloadContent → loadRecipes() changed to
  loadRecipes({ preserveScroll: true })

The normalizeLoadRecipesOptions boolean path always forces
preserveScroll: false — the object form is required to pass it.
2026-05-04 20:26:13 +08:00
Will Miao e23d803ecf fix(layout): ensure refresh split-button dropdown renders above breadcrumb nav 2026-05-03 18:14:54 +08:00
Will Miao 0cc640cfaa fix(recipe): support ComfyUI-Easy-Use nodes in runtime metadata extraction (#920)
- Add EasyComfyLoaderExtractor for comfyLoader (easy comfyLoader):
  extracts checkpoint, optional_lora_stack as LoRA apply node,
  prompt text, clip_skip, and latent dimensions
- Add EasyPreSamplingExtractor for samplerSettings (easy preSampling):
  extracts steps, cfg, sampler_name, scheduler, denoise, seed
- Add EasySeedExtractor for easySeed
- Fix clip_skip hardcoded to '1' — now searched from SAMPLING metadata
- Lora Stacker nodes intentionally excluded from extraction to
  prevent double-counting; LoRAs only recorded at apply nodes
2026-05-02 23:21:51 +08:00
Will Miao 2ac0eb0f9d fix(wanvideo): resolve lora path resolution and name truncation for extra folder paths
- Use get_lora_info_absolute to obtain correct absolute paths for loras
  in LM extra folder paths, instead of folder_paths.get_full_path which
  only searches ComfyUI's standard loras directories (returned None)
- Fix name field truncation: str.split('.')[0] stopped at the first dot,
  replaced with os.path.splitext to only strip the file extension
- Add _relpath_within_loras helper to preserve subdirectory info in the
  name field, matching WanVideoWrapper's os.path.splitext(lora)[0] format
2026-05-02 14:55:12 +08:00
Will Miao f028625ce9 feat(check-models-exist): add batch endpoint for checking multiple model IDs
New endpoint: GET /api/lm/check-models-exist?modelIds=1,2,3,...

Accepts comma-separated modelIds, returns a results array with one
entry per modelId. Uses a single scanner lookup batch - three
service-registry calls total, regardless of model count. Skips
history checks entirely (same rationale as the singleton endpoint:
when models exist locally, history is redundant).

Expected: reduces 231 HTTP round-trips to 1 for the browser
extension's model-card indicator flow. Combined with the prior
SQLite-connection and history-skip fixes, total wall-clock time
for a 175K-lora user's page load drops from ~9.4s to <10ms.
2026-05-02 13:43:53 +08:00
Will Miao 06acc7f576 fix(trigger-word-toggle): default group children to active regardless of default_active 2026-05-02 13:33:42 +08:00
Will Miao d324b57274 perf(check-model-exists): eliminate SQLite connection-per-query overhead and skip redundant history checks
Root cause: 231 concurrent /check-model-exists requests on 175K-lora library
caused ~9.4s wall clock time. The bottleneck was two-fold:

1. DownloadedVersionHistoryService opened a new sqlite3.connect() for every
   query under asyncio.Lock. With a large WAL from 175K entries, each
   connect() took ~8ms. Serialized by the lock across 231 requests, the
   230th request waited ~1848ms just for lock acquisition.

2. check_model_exists always queried download history even when the model
   was found locally. The history result (hasBeenDownloaded /
   downloadedVersionIds) is only used by the UI when the model is NOT
   found locally; when found, the 'in library' indicator takes priority.

Changes:
- downloaded_version_history_service.py: added persistent _get_conn() that
  creates the SQLite connection once and reuses it across all queries
- misc_handlers.py: early-return from check_model_exists when the model
  exists locally, bypassing the history service entirely (lock skipped)

Expected: per-request wait time drops from ~1912ms to <3ms, wall clock
from ~9.4s to <0.3s for the 175K-lora user's 231-card page.
2026-05-02 13:31:20 +08:00
Will Miao 502b7eab31 fix(layout): correct breadcrumb sticky behavior and controls wrapping overflow
- Extract breadcrumb from controls template into sibling component
- Fix breadcrumb sticky positioning (top: 0, z-index: calc(--z-header - 1))
- Add 1500px breakpoint to wrap controls-right and prevent overflow
- Adjust breadcrumb padding-bottom to cover controls-right area when sticky
2026-05-01 22:53:40 +08:00
Will Miao be75ad930e feat(layout): implement responsive edge-to-edge card grid with density-aware column calculation
- Add dynamic column calculation based on container width and min card width
- Prevent tiny cards on narrow windows by respecting density-based minimums:
  - Default: 240px, Medium: 200px, Compact: 170px
- Fix edge-to-edge layout with proper CSS selector (.virtual-scroll-item.model-card)
- Add hamburger menu for mobile/small screens with proper translations
- Update all locale files with 'common.actions.menu' key

Fixes: Cards becoming too small/overlapping on narrow window widths (e.g., 1156px)
Changes: 15 files, +569/-114 lines
2026-05-01 21:34:31 +08:00
Will Miao 763c4f4dad feat(usage-control): add support for Civitai usageControl field
Handle models that are only available for on-site generation (usageControl:
"Generation" or "InternalGeneration") rather than downloadable.

Backend changes:
- Add usage_control field to ModelVersionRecord dataclass
- Extract usageControl from Civitai API responses
- Filter non-downloadable versions from update availability checks
- Add database schema migration for usage_control column
- Include usageControl in version response JSON

Frontend changes:
- Add isDownloadAllowed() helper function
- Show disabled download button for non-downloadable versions
- Add "On-Site Only" badge for restricted versions
- Update resolveUpdateAvailability() to filter non-downloadable versions
- Add CSS styling for disabled action button

Internationalization:
- Add translations for onSiteOnly badge and downloadNotAllowedTooltip
- Complete translations for all 10 supported languages
2026-05-01 13:10:15 +08:00
Will Miao d32c492bdb feat(scripts): add legacy metadata migration tool
Add script to migrate metadata from legacy sidecar JSON files to
LoRA Manager's metadata.json format.

Features:
- Auto-discovers model folders from settings.json
- Supports LoRA and Checkpoint model types
- Migrates activation text, preferred weight (LoRA only), and notes
- Dry-run mode for safe preview
- Idempotent migration (won't duplicate existing data)
2026-05-01 08:56:00 +08:00
Will MiaoandClaude Opus 4.7 5dcfde36ea feat(doctor): add duplicate filename conflict detection and one-click resolution
Detects when multiple model files share the same basename (causing
ambiguity in LoRA resolution), logs warnings during scanning, and
provides a "Resolve Conflicts" button in the Doctor panel. Resolution
renames duplicates with hash-prefixed unique filenames, migrates all
sidecar and preview files, and updates the cache and frontend scroller
in-place so the model modal immediately reflects the new filename.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-30 15:21:26 +08:00
Will MiaoandClaude Opus 4.7 1d035361a4 fix(download): accept Diffusion Model file type when selecting primary file from CivitAI metadata
CivitAI returns file type "Diffusion Model" for checkpoint files (e.g., Anima
models), but the file selection logic only accepted "Model" and "Negative",
causing "No suitable file found in metadata" errors.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-30 11:54:14 +08:00
Will Miao 25605c5e78 feat(ui): add setting to toggle version name display on model cards (#916) 2026-04-29 20:04:40 +08:00
Will MiaoandClaude Opus 4.7 f3268a6179 fix(autocomplete): prevent migrateWidgetsValues from dropping text widget values (#915)
shouldBypassAutocompleteWidgetMigration only matched inputs by widget name,
but ComfyUI's migrateWidgetsValues also matches forceInput inputs (like "seed").
This discrepancy meant the bypass never triggered for TextLM/PromptLM nodes,
causing migrateWidgetsValues to filter out real widget values by incorrectly
mapping forceInput flags onto saved autocomplete values.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-29 16:44:08 +08:00
Will MiaoandClaude Opus 4.7 055e94d77b fix(updates): chunk bulk queries to avoid SQLite variable limit (#914)
_split _get_records_bulk into 500-id batches so the WHERE IN clause
never exceeds SQLite's 999-parameter ceiling.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-28 19:15:44 +08:00
Will Miao 47fcd530a0 feat(settings): add aria2 wiki help link to download backend setting 2026-04-28 18:37:59 +08:00
Will MiaoandClaude Opus 4.7 3c32b9e088 feat(example-images): add wiki help link and i18n keys for remote open mode
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-27 19:45:16 +08:00
Will Miao ffe0670a27 feat(example-images): add remote open mode support 2026-04-27 14:05:21 +08:00
Will Miao cc147a1795 fix(metadata): preserve workflow when recipe images convert to webp 2026-04-25 07:50:51 +08:00
Will Miao e81409bea4 fix(i18n): shorten bulk delete labels 2026-04-25 07:21:42 +08:00
Will Miao b31fae4e51 fix(widgets): isolate autocomplete text cleanup 2026-04-23 20:07:11 +08:00
Will Miao c6e5467907 fix(metadata): add MyOriginalWaifu prompt extractors 2026-04-23 16:05:40 +08:00
Will Miao df0e5797d0 fix(nodes): save recipes synchronously from save image 2026-04-23 15:46:57 +08:00
Will Miao ebdbb36271 fix(metadata): trace conditioning provenance for prompts 2026-04-23 14:41:54 +08:00
Will Miao 2eef629821 fix(checkpoints): singleflight pending hash calculation 2026-04-23 11:36:32 +08:00
Will Miao 658a04736d fix(recipes): save widget checkpoint metadata as dict 2026-04-23 11:20:20 +08:00
Will Miao ef7f677933 chore(skills): add lora manager runtime context 2026-04-23 09:42:47 +08:00
Will Miao 63f0942452 fix(models): classify Anima as diffusion model 2026-04-23 07:35:34 +08:00
Will Miao a1dff6dd47 fix(download): auto fetch example images after model download 2026-04-21 22:48:06 +08:00
Will Miao 7fa40023b0 fix(trigger-words): edit tag on double click 2026-04-21 22:31:56 +08:00
Will Miao 3c8acdb65e fix(trigger-words): support stable inline editing 2026-04-21 22:18:35 +08:00
Will Miao 1e9a7812d6 fix(model-modal): allow resizing notes editor 2026-04-21 21:42:06 +08:00
Will Miao 37f0e8f213 fix(trigger-words): raise group word limit 2026-04-21 16:35:25 +08:00
Will Miao ecf7ea21e4 fix(duplicates): clear stale hash mismatch state (#900) 2026-04-21 16:22:04 +08:00
Will Miao 79dd9a1b29 fix(trigger-word-toggle): compact group editing for #907 2026-04-21 10:44:05 +08:00
Will Miao ef4923fd94 fix(settings): normalize default root path comparisons 2026-04-21 09:43:37 +08:00
Will Miao 1eeba666f5 fix(network): restore destination-scoped memory download guard 2026-04-20 18:27:38 +08:00
pixelpaws 89e26d9292 Merge pull request #906 from willmiao/codex/github-mention-fixnetwork-add-connectivityguard-to-short
fix(network): return friendly offline message for memory downloads
2026-04-20 16:07:06 +08:00
pixelpaws fc19a145ff Merge branch 'main' into codex/github-mention-fixnetwork-add-connectivityguard-to-short 2026-04-20 15:54:30 +08:00
Will Miao 34f03d6495 fix(settings): preserve extra default roots in comfyui sync 2026-04-20 15:48:30 +08:00
pixelpaws 9443175abc fix(network): return friendly offline message for memory downloads 2026-04-20 15:42:03 +08:00
pixelpaws dc5072628f Merge pull request #905 from willmiao/codex/task-title
fix(network): add ConnectivityGuard to short‑circuit offline requests and reduce log spam
2026-04-20 15:41:38 +08:00
pixelpaws ff4b8ec849 test(network): align cooldown short-circuit test with per-host guard 2026-04-20 15:30:50 +08:00
pixelpaws 7ab271c752 fix(network): scope connectivity cooldown by destination 2026-04-20 15:20:57 +08:00
pixelpaws 5a7f4dc88b fix(network): add offline cooldown guard for remote metadata requests 2026-04-20 15:04:04 +08:00
Will Miao 761108bfd1 fix(download): restore aria2 resume lifecycle 2026-04-20 09:52:48 +08:00
Will Miao 24dd3a777c fix(settings): align modal form control widths 2026-04-19 21:59:33 +08:00
Will Miao 1c530ea013 feat(download): add experimental aria2 backend 2026-04-19 21:46:09 +08:00
mudknight 0ced53c059 Use flex gap for header spacing (#901)
* Use flex gap for header spacing

* Remove extra margin
2026-04-18 19:33:39 +08:00
Will Miao 67ad68a23f fix(filters): apply preset base models from full list 2026-04-18 07:00:24 +08:00
pixelpaws d9ec9c512e Merge pull request #899 from Phinease/fix/resumable-download-retries
fix: preserve resumable downloads across retries
2026-04-17 20:46:22 +08:00
Will Miao 0bcd8e09a9 fix(filters): improve base model filtering UX 2026-04-17 20:27:48 +08:00
Shuangrui CHEN fa049a28c8 fix: preserve resumable downloads across retries 2026-04-17 03:35:41 +08:00
337 changed files with 54241 additions and 23018 deletions
@@ -0,0 +1,69 @@
---
name: lora-manager-runtime-context
description: Inspect ComfyUI LoRA Manager runtime configuration and local diagnostic state. Use when debugging LoRA Manager issues that require locating or reading settings.json, active library paths, model metadata JSON sidecars, recipe metadata JSON files, example image folders, SQLite caches, symlink maps, download history, aria2 state, or other cache files under the LoRA Manager user config directory.
---
# LoRA Manager Runtime Context
## Core Rules
- Treat runtime state as local user data. Prefer read-only inspection unless the user explicitly asks for mutation.
- Never print secret-like settings values. Redact keys containing `key`, `token`, `secret`, `password`, `auth`, or `credential`, including `civitai_api_key`.
- Resolve paths from the runtime configuration before guessing. In this environment the settings file is normally `/home/miao/.config/ComfyUI-LoRA-Manager/settings.json`, but portable settings can override this through the repository `settings.json`.
- Use the active library when selecting per-library caches and paths. Read `active_library` from settings; fall back to `default` if missing.
- Normalize and expand `~` before comparing paths. Symlinks are common in this repo.
## Quick Start
Use the bundled helper for a safe first pass:
```bash
python .agents/skills/lora-manager-runtime-context/scripts/inspect_runtime_context.py summary
python .agents/skills/lora-manager-runtime-context/scripts/inspect_runtime_context.py caches
```
The script redacts sensitive settings, opens SQLite databases read-only, and reports inaccessible or locked databases as warnings.
For focused checks:
```bash
python .agents/skills/lora-manager-runtime-context/scripts/inspect_runtime_context.py recipes
python .agents/skills/lora-manager-runtime-context/scripts/inspect_runtime_context.py model --path /path/to/model.safetensors
python .agents/skills/lora-manager-runtime-context/scripts/inspect_runtime_context.py sqlite --db /path/to/cache.sqlite --limit 3
```
## Runtime Path Rules
- Settings directory: use `py/utils/settings_paths.py`. Default platform path is `platformdirs.user_config_dir("ComfyUI-LoRA-Manager", appauthor=False)`.
- Settings file: `<settings_dir>/settings.json`.
- Cache root: `<settings_dir>/cache`.
- Canonical cache files:
- Model cache: `cache/model/<active_library>.sqlite`.
- Recipe cache: `cache/recipe/<active_library>.sqlite`.
- Model update cache: `cache/model_update/<active_library>.sqlite`.
- Recipe FTS: `cache/fts/recipe_fts.sqlite`.
- Tag FTS: `cache/fts/tag_fts.sqlite`.
- Symlink map: `cache/symlink/symlink_map.json`.
- Download history: `cache/download_history/downloaded_versions.sqlite`.
- aria2 state: `cache/aria2/downloads.json`.
- Legacy cache locations may exist; prefer canonical paths unless diagnosing migrations.
## Data Location Rules
- Model roots come from `settings.folder_paths` and the active library payload under `settings.libraries[active_library]`.
- Model metadata JSON sidecars live next to the model file as `<model basename>.metadata.json`.
- Recipes root is `settings.recipes_path` when it is a non-empty string. If empty, use the first configured LoRA root plus `/recipes`.
- Recipe JSON files are named `*.recipe.json` under the recipes root and may be nested in folders.
- Example image root is `settings.example_images_path`.
- If multiple libraries are configured, example images are stored under `<example_images_path>/<sanitized_library>/<sha256>/`; otherwise they are under `<example_images_path>/<sha256>/`.
## Useful Cache Tables
- Model cache: `models`, `model_tags`, `hash_index`, `excluded_models`.
- Recipe cache: `recipes`, `cache_metadata`.
- Model update cache: `model_update_status`, `model_update_versions`.
- Tag FTS cache: `tags`, `fts_metadata`, plus FTS internal tables.
- Recipe FTS cache: `recipe_rowid`, `fts_metadata`, plus FTS internal tables.
- Download history: `downloaded_model_versions`.
Prefer querying only counts, schema, and a few sample rows unless the user asks for full output.
@@ -0,0 +1,4 @@
interface:
display_name: "LoRA Manager Runtime Context"
short_description: "Inspect LoRA Manager runtime state"
default_prompt: "Use $lora-manager-runtime-context to inspect LoRA Manager settings, metadata paths, and caches for debugging."
@@ -0,0 +1,381 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import json
import os
import re
import shutil
import sqlite3
import sys
import tempfile
from pathlib import Path
from typing import Any
SECRET_PATTERN = re.compile(r"(key|token|secret|password|auth|credential)", re.IGNORECASE)
APP_NAME = "ComfyUI-LoRA-Manager"
CACHE_SQLITE = {
"model": ("model", "{library}.sqlite"),
"recipe": ("recipe", "{library}.sqlite"),
"model_update": ("model_update", "{library}.sqlite"),
"recipe_fts": ("fts", "recipe_fts.sqlite"),
"tag_fts": ("fts", "tag_fts.sqlite"),
"download_history": ("download_history", "downloaded_versions.sqlite"),
}
CACHE_JSON = {
"symlink": ("symlink", "symlink_map.json"),
"aria2": ("aria2", "downloads.json"),
}
def main() -> int:
parser = argparse.ArgumentParser(description="Inspect LoRA Manager runtime state read-only.")
subparsers = parser.add_subparsers(dest="command", required=True)
subparsers.add_parser("summary", help="Print redacted settings and resolved paths.")
subparsers.add_parser("caches", help="Print cache paths and SQLite table summaries.")
subparsers.add_parser("recipes", help="Print resolved recipes root and recipe JSON count.")
model_parser = subparsers.add_parser("model", help="Inspect a model metadata sidecar path.")
model_parser.add_argument("--path", required=True, help="Path to a model file or metadata JSON file.")
sqlite_parser = subparsers.add_parser("sqlite", help="Inspect a SQLite database read-only.")
sqlite_parser.add_argument("--db", required=True, help="Path to the SQLite database.")
sqlite_parser.add_argument("--limit", type=int, default=3, help="Rows to sample from each user table.")
args = parser.parse_args()
context = build_context()
if args.command == "summary":
print_json(summary_payload(context))
elif args.command == "caches":
print_json(caches_payload(context))
elif args.command == "recipes":
print_json(recipes_payload(context))
elif args.command == "model":
print_json(model_payload(args.path))
elif args.command == "sqlite":
print_json(sqlite_payload(Path(args.db).expanduser(), args.limit))
return 0
def build_context() -> dict[str, Any]:
settings_path = resolve_settings_path()
settings = load_json(settings_path)
settings_dir = settings_path.parent
active_library = settings.get("active_library") or "default"
safe_library = sanitize_library_name(str(active_library))
cache_root = settings_dir / "cache"
return {
"settings_path": str(settings_path),
"settings_dir": str(settings_dir),
"settings": settings,
"active_library": active_library,
"safe_library": safe_library,
"cache_root": str(cache_root),
"cache_paths": resolve_cache_paths(cache_root, safe_library),
}
def resolve_settings_path() -> Path:
repo_root = find_repo_root()
portable = repo_root / "settings.json"
if portable.exists():
payload = load_json(portable)
if isinstance(payload, dict) and payload.get("use_portable_settings") is True:
return portable
config_home = os.environ.get("XDG_CONFIG_HOME")
if config_home:
return Path(config_home).expanduser() / APP_NAME / "settings.json"
return Path.home() / ".config" / APP_NAME / "settings.json"
def find_repo_root() -> Path:
current = Path(__file__).resolve()
for parent in current.parents:
if (parent / "py").is_dir() and (parent / "standalone.py").exists():
return parent
return Path.cwd()
def load_json(path: Path) -> dict[str, Any]:
try:
with path.open("r", encoding="utf-8") as handle:
payload = json.load(handle)
except FileNotFoundError:
return {}
except json.JSONDecodeError as exc:
return {"_error": f"invalid JSON: {exc}"}
except OSError as exc:
return {"_error": f"unreadable: {exc}"}
return payload if isinstance(payload, dict) else {"_error": "JSON root is not an object"}
def resolve_cache_paths(cache_root: Path, library: str) -> dict[str, str]:
paths: dict[str, str] = {}
for name, (subdir, filename) in CACHE_SQLITE.items():
paths[name] = str(cache_root / subdir / filename.format(library=library))
for name, (subdir, filename) in CACHE_JSON.items():
paths[name] = str(cache_root / subdir / filename)
return paths
def summary_payload(context: dict[str, Any]) -> dict[str, Any]:
settings = context["settings"]
return {
"settings_path": context["settings_path"],
"settings_dir": context["settings_dir"],
"active_library": context["active_library"],
"settings": redact(settings),
"model_roots": model_roots(settings, context["active_library"]),
"recipes_root": str(resolve_recipes_root(settings, context["active_library"]) or ""),
"example_images": example_images_payload(settings, context["active_library"]),
"cache_root": context["cache_root"],
"cache_paths": context["cache_paths"],
}
def caches_payload(context: dict[str, Any]) -> dict[str, Any]:
caches: dict[str, Any] = {}
for name, path_string in context["cache_paths"].items():
path = Path(path_string)
item: dict[str, Any] = {
"path": str(path),
"exists": path.exists(),
"size": path.stat().st_size if path.exists() else None,
}
if path.suffix == ".sqlite":
item["sqlite"] = sqlite_payload(path, limit=0)
elif path.suffix == ".json":
item["json"] = json_file_summary(path)
caches[name] = item
return {"active_library": context["active_library"], "caches": caches}
def recipes_payload(context: dict[str, Any]) -> dict[str, Any]:
root = resolve_recipes_root(context["settings"], context["active_library"])
files: list[str] = []
if root and root.exists():
files = [str(path) for path in sorted(root.rglob("*.recipe.json"))[:20]]
return {
"recipes_root": str(root or ""),
"exists": bool(root and root.exists()),
"recipe_json_count": count_recipe_files(root),
"sample_recipe_json": files,
"recipe_cache": context["cache_paths"].get("recipe"),
}
def model_payload(raw_path: str) -> dict[str, Any]:
path = Path(raw_path).expanduser()
metadata_path = path if path.name.endswith(".metadata.json") else path.with_suffix(".metadata.json")
payload = {
"input_path": str(path),
"metadata_path": str(metadata_path),
"model_exists": path.exists(),
"metadata_exists": metadata_path.exists(),
}
if metadata_path.exists():
data = load_json(metadata_path)
payload["metadata_summary"] = redact(summarize_value(data))
return payload
def sqlite_payload(path: Path, limit: int = 3, allow_copy: bool = True) -> dict[str, Any]:
result: dict[str, Any] = {"path": str(path), "exists": path.exists(), "tables": {}}
if not path.exists():
return result
try:
conn = connect_sqlite_readonly(path)
except sqlite3.Error as exc:
result["error"] = str(exc)
return result
try:
table_rows = conn.execute(
"SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"
).fetchall()
for table_row in table_rows:
table = table_row["name"]
columns = [
row["name"]
for row in conn.execute(f"PRAGMA table_info({quote_identifier(table)})").fetchall()
]
table_info: dict[str, Any] = {"columns": columns}
try:
table_info["count"] = conn.execute(
f"SELECT COUNT(*) FROM {quote_identifier(table)}"
).fetchone()[0]
except sqlite3.Error as exc:
table_info["count_error"] = str(exc)
if limit > 0 and columns and not is_internal_sqlite_table(table):
try:
rows = conn.execute(
f"SELECT * FROM {quote_identifier(table)} LIMIT ?", (limit,)
).fetchall()
table_info["sample"] = [redact(dict(row)) for row in rows]
except sqlite3.Error as exc:
table_info["sample_error"] = str(exc)
result["tables"][table] = table_info
except sqlite3.Error as exc:
fallback = sqlite_copy_payload(path, limit, str(exc)) if allow_copy else None
if fallback is not None:
result.update(fallback)
else:
result["error"] = str(exc)
finally:
conn.close()
return result
def connect_sqlite_readonly(path: Path) -> sqlite3.Connection:
errors: list[str] = []
for query in ("mode=ro", "mode=ro&immutable=1"):
try:
conn = sqlite3.connect(f"file:{path}?{query}", uri=True)
conn.row_factory = sqlite3.Row
return conn
except sqlite3.Error as exc:
errors.append(f"{query}: {exc}")
raise sqlite3.OperationalError("; ".join(errors))
def sqlite_copy_payload(path: Path, limit: int, original_error: str) -> dict[str, Any] | None:
try:
with tempfile.TemporaryDirectory(prefix="lm-cache-inspect-") as temp_dir:
copy_path = Path(temp_dir) / path.name
shutil.copy2(path, copy_path)
payload = sqlite_payload(copy_path, limit, allow_copy=False)
payload["path"] = str(path)
payload["inspected_copy"] = True
payload["original_error"] = original_error
return payload
except Exception:
return None
def json_file_summary(path: Path) -> dict[str, Any]:
if not path.exists():
return {"exists": False}
data = load_json(path)
return {"exists": True, "summary": redact(summarize_value(data))}
def model_roots(settings: dict[str, Any], active_library: str) -> dict[str, list[str]]:
roots: dict[str, list[str]] = {}
sources = [settings]
library = settings.get("libraries", {}).get(active_library)
if isinstance(library, dict):
sources.insert(0, library)
for source in sources:
folder_paths = source.get("folder_paths")
if isinstance(folder_paths, dict):
for key, value in folder_paths.items():
roots.setdefault(key, []).extend(normalize_path_list(value))
for default_key, folder_key in (
("default_lora_root", "loras"),
("default_checkpoint_root", "checkpoints"),
("default_embedding_root", "embeddings"),
("default_unet_root", "unet"),
):
value = settings.get(default_key)
if isinstance(value, str) and value:
roots.setdefault(folder_key, []).append(expand_path(value))
return {key: dedupe(values) for key, values in roots.items()}
def resolve_recipes_root(settings: dict[str, Any], active_library: str) -> Path | None:
recipes_path = settings.get("recipes_path")
library = settings.get("libraries", {}).get(active_library)
if isinstance(library, dict) and isinstance(library.get("recipes_path"), str):
recipes_path = library["recipes_path"] or recipes_path
if isinstance(recipes_path, str) and recipes_path.strip():
return Path(expand_path(recipes_path.strip()))
lora_roots = model_roots(settings, active_library).get("loras") or []
return Path(lora_roots[0]) / "recipes" if lora_roots else None
def example_images_payload(settings: dict[str, Any], active_library: str) -> dict[str, Any]:
root = settings.get("example_images_path") or ""
libraries = settings.get("libraries")
library_count = len(libraries) if isinstance(libraries, dict) else 0
scoped = library_count > 1
root_path = Path(expand_path(root)) if isinstance(root, str) and root else None
library_root = root_path / sanitize_library_name(active_library) if root_path and scoped else root_path
return {
"root": str(root_path or ""),
"uses_library_scoped_folders": scoped,
"library_root": str(library_root or ""),
}
def count_recipe_files(root: Path | None) -> int:
if not root or not root.exists():
return 0
return sum(1 for _ in root.rglob("*.recipe.json"))
def normalize_path_list(value: Any) -> list[str]:
if isinstance(value, str):
return [expand_path(value)] if value else []
if isinstance(value, list):
return [expand_path(item) for item in value if isinstance(item, str) and item]
return []
def expand_path(value: str) -> str:
return str(Path(value).expanduser().resolve(strict=False))
def sanitize_library_name(name: str) -> str:
safe = re.sub(r"[^A-Za-z0-9_.-]", "_", name or "default")
return safe or "default"
def dedupe(values: list[str]) -> list[str]:
seen: set[str] = set()
result: list[str] = []
for value in values:
if value not in seen:
result.append(value)
seen.add(value)
return result
def redact(value: Any, key: str = "") -> Any:
if key and SECRET_PATTERN.search(key):
return "<redacted>"
if isinstance(value, dict):
return {str(k): redact(v, str(k)) for k, v in value.items()}
if isinstance(value, list):
return [redact(item) for item in value]
return value
def summarize_value(value: Any) -> Any:
if isinstance(value, dict):
return {key: summarize_value(item) for key, item in value.items()}
if isinstance(value, list):
return {
"type": "array",
"length": len(value),
"first": summarize_value(value[0]) if value else None,
}
return value
def quote_identifier(identifier: str) -> str:
return '"' + identifier.replace('"', '""') + '"'
def is_internal_sqlite_table(table: str) -> bool:
return table.startswith("sqlite_") or table.endswith(("_data", "_idx", "_docsize", "_config", "_content"))
def print_json(payload: Any) -> None:
json.dump(payload, sys.stdout, indent=2, ensure_ascii=False)
sys.stdout.write("\n")
if __name__ == "__main__":
raise SystemExit(main())
-153
View File
@@ -1,153 +0,0 @@
# Recipe Batch Import Feature Design
## Overview
Enable users to import multiple images as recipes in a single operation, rather than processing them individually. This feature addresses the need for efficient bulk recipe creation from existing image collections.
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ Frontend │
├─────────────────────────────────────────────────────────────────┤
│ BatchImportManager.js │
│ ├── InputCollector (收集URL列表/目录路径) │
│ ├── ConcurrencyController (自适应并发控制) │
│ ├── ProgressTracker (进度追踪) │
│ └── ResultAggregator (结果汇总) │
├─────────────────────────────────────────────────────────────────┤
│ batch_import_modal.html │
│ └── 批量导入UI组件 │
├─────────────────────────────────────────────────────────────────┤
│ batch_import_progress.css │
│ └── 进度显示样式 │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Backend │
├─────────────────────────────────────────────────────────────────┤
│ py/routes/handlers/recipe_handlers.py │
│ ├── start_batch_import() - 启动批量导入 │
│ ├── get_batch_import_progress() - 查询进度 │
│ └── cancel_batch_import() - 取消导入 │
├─────────────────────────────────────────────────────────────────┤
│ py/services/batch_import_service.py │
│ ├── 自适应并发执行 │
│ ├── 结果汇总 │
│ └── WebSocket进度广播 │
└─────────────────────────────────────────────────────────────────┘
```
## API Endpoints
| 端点 | 方法 | 说明 |
|------|------|------|
| `/api/lm/recipes/batch-import/start` | POST | 启动批量导入,返回 operation_id |
| `/api/lm/recipes/batch-import/progress` | GET | 查询进度状态 |
| `/api/lm/recipes/batch-import/cancel` | POST | 取消导入 |
## Backend Implementation Details
### BatchImportService
Location: `py/services/batch_import_service.py`
Key classes:
- `BatchImportItem`: Dataclass for individual import item
- `BatchImportProgress`: Dataclass for tracking progress
- `BatchImportService`: Main service class
Features:
- Adaptive concurrency control (adjusts based on success/failure rate)
- WebSocket progress broadcasting
- Graceful error handling (individual failures don't stop the batch)
- Result aggregation
### WebSocket Message Format
```json
{
"type": "batch_import_progress",
"operation_id": "xxx",
"total": 50,
"completed": 23,
"success": 21,
"failed": 2,
"skipped": 0,
"current_item": "image_024.png",
"status": "running"
}
```
### Input Types
1. **URL List**: Array of URLs (http/https)
2. **Local Paths**: Array of local file paths
3. **Directory**: Path to directory with optional recursive flag
### Error Handling
- Invalid URLs/paths: Skip and record error
- Download failures: Record error, continue
- Metadata extraction failures: Mark as "no metadata"
- Duplicate detection: Option to skip duplicates
## Frontend Implementation Details (TODO)
### UI Components
1. **BatchImportModal**: Main modal with tabs for URLs/Directory input
2. **ProgressDisplay**: Real-time progress bar and status
3. **ResultsSummary**: Final results with success/failure breakdown
### Adaptive Concurrency Controller
```javascript
class AdaptiveConcurrencyController {
constructor(options = {}) {
this.minConcurrency = options.minConcurrency || 1;
this.maxConcurrency = options.maxConcurrency || 5;
this.currentConcurrency = options.initialConcurrency || 3;
}
adjustConcurrency(taskDuration, success) {
if (success && taskDuration < 1000 && this.currentConcurrency < this.maxConcurrency) {
this.currentConcurrency = Math.min(this.currentConcurrency + 1, this.maxConcurrency);
}
if (!success || taskDuration > 10000) {
this.currentConcurrency = Math.max(this.currentConcurrency - 1, this.minConcurrency);
}
return this.currentConcurrency;
}
}
```
## File Structure
```
Backend (implemented):
├── py/services/batch_import_service.py # 后端服务
├── py/routes/handlers/batch_import_handler.py # API处理器 (added to recipe_handlers.py)
├── tests/services/test_batch_import_service.py # 单元测试
└── tests/routes/test_batch_import_routes.py # API集成测试
Frontend (TODO):
├── static/js/managers/BatchImportManager.js # 主管理器
├── static/js/managers/batch/ # 子模块
│ ├── ConcurrencyController.js # 并发控制
│ ├── ProgressTracker.js # 进度追踪
│ └── ResultAggregator.js # 结果汇总
├── static/css/components/batch-import-modal.css # 样式
└── templates/components/batch_import_modal.html # Modal模板
```
## Implementation Status
- [x] Backend BatchImportService
- [x] Backend API handlers
- [x] WebSocket progress broadcasting
- [x] Unit tests
- [x] Integration tests
- [ ] Frontend BatchImportManager
- [ ] Frontend UI components
- [ ] E2E tests
@@ -13,8 +13,5 @@ A clear and concise description of what the problem is. Ex. I'm always frustrate
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
+13 -1
View File
@@ -7,15 +7,24 @@ py/run_test.py
.vscode/
cache/
civitai/
stats/
wildcards/
backups/
logs/
node_modules/
coverage/
.coverage
model_cache/
# agent
# agent / dev tooling
.opencode/
.claude/
.sisyphus/
.codex
.omo
reasonix.toml
.reasonix/
.codegraph/
# Vue widgets development cache (but keep build output)
vue-widgets/node_modules/
@@ -24,3 +33,6 @@ vue-widgets/dist/
# Hypothesis test cache
.hypothesis/
# Working/research notes (not committed)
.docs/
+181
View File
@@ -0,0 +1,181 @@
# Embeddings Usage Tracking — Hybrid Approach (Plan C)
> **Status**: Reference document for future implementation
> **Current implementation**: Plan A (prompt text parsing only, see `usage_stats.py:_process_embeddings`)
> **Next step**: Add Plan B as a supplement when edge-case coverage is needed
## Problem
Embeddings in ComfyUI are not loaded through dedicated ComfyUI nodes like LoRAs or
Checkpoints. They are resolved during CLIP tokenization when the prompt text contains
`embedding:<name>` syntax (see `comfy/sd1_clip.py:SDTokenizer.tokenize_with_weights`).
This means the existing metadata_collector hook (which intercepts node execution via
`_map_node_over_list`) cannot capture embeddings the same way it captures LoRAs and
checkpoints — there is no "EmbeddingLoader" node to intercept.
## Solution Architecture
The hybrid approach combines **two complementary mechanisms** to capture embedding
usage from all possible paths.
```
┌─────────────────────────────────────────────────────────┐
│ Plan A (已实现) │
│ │
│ MetadataRegistry.prompt_metadata["prompts"] │
│ │ │
│ ▼ │
│ _process_embeddings() │
│ │ │
│ ├─ Iterate all prompt node texts │
│ ├─ regex extract "embedding:<name>" │
│ ├─ resolve name → sha256 via EmbeddingScanner │
│ └─ UsageStats.stats["embeddings"][sha256]++ │
│ │
│ Coverage: ~95% — all CLIPTextEncode/Flux/etc nodes │
│ │
│ Gap: Custom nodes that load embeddings programmatically │
│ without putting embedding:name in prompt text │
└─────────────────────────────────────────────────────────┘
+
↓ (future: enable Plan B when needed)
┌─────────────────────────────────────────────────────────┐
│ Plan B (未来 — monkey-patch) │
│ │
│ comfy/sd1_clip.py:load_embed() │
│ │ │
│ ▼ │
│ Monkey-patch intercepts EVERY embedding file load │
│ │ │
│ ├─ Records embedding_name + success/failure │
│ ├─ Associates with current prompt_id (via registry)│
│ └─ Feeds into UsageStats same as Plan A │
│ │
│ Coverage: 100% — catches ALL embedding loads │
│ │
│ Cost: Requires patching into ComfyUI internals │
│ (sd1_clip.py, sdxl_clip.py, some text_encoders) │
└─────────────────────────────────────────────────────────┘
```
## Plan B Detail — Monkey-patch `load_embed`
### Target Function
**`comfy.sd1_clip.load_embed(embedding_name, embedding_directory, embedding_size, embed_key=None)`**
at line 415 of `sd1_clip.py`.
This is the **single choke point** for all embedding file loads in ComfyUI. Every
CLIP variant (SD1, SDXL, SD3, Flux) calls this same function.
### Implementation Sketch
```python
# In metadata_collector/metadata_hook.py (or a new module)
import comfy.sd1_clip as sd1_clip
_original_load_embed = sd1_clip.load_embed
def _patched_load_embed(embedding_name, embedding_directory, embedding_size, embed_key=None):
result = _original_load_embed(
embedding_name, embedding_directory, embedding_size, embed_key
)
if result is not None:
_record_embedding_usage(embedding_name)
return result
sd1_clip.load_embed = _patched_load_embed
```
### Prompt ID Association
The challenge is associating the `load_embed` call with the current `prompt_id`.
Options:
1. **Thread-local / contextvar**: Store current `prompt_id` in a `contextvars.ContextVar`
that the metadata_collector sets at the start of each prompt execution.
2. **MetadataRegistry singleton**: The MetadataRegistry already has `current_prompt_id`.
The patch can read it directly since both run in the same thread.
3. **Lazy aggregation**: Instead of associating with prompt_id at load time, collect
all loaded embedding names in a global set during execution, then flush to
UsageStats after the prompt completes.
### Files to Patch
| File | Function | Coverage |
|------|----------|----------|
| `comfy/sd1_clip.py:415` | `load_embed()` | Primary — SD1.x, SDXL, SD3, Flux |
| `comfy/sdxl_clip.py` | Not needed (calls `sd1_clip.SDTokenizer`) | — |
| `comfy/text_encoders/sd3_clip.py` | Not needed (calls `sd1_clip.SDTokenizer`) | — |
| `comfy/text_encoders/flux.py` | Not needed (calls `sd1_clip.SDTokenizer`) | — |
The SD1 tokenizer is the base class for all CLIP variants' tokenizers, so patching
`load_embed` covers them all.
### Edge Cases
| Edge Case | Plan A | Plan B |
|-----------|--------|--------|
| `embedding:name` in CLIPTextEncode | ✅ | ✅ |
| `embedding:name` in CLIPTextEncodeFlux | ✅ | ✅ |
| `embedding:name` in PromptLM (LoRA Manager) | ✅ | ✅ |
| `embedding:name` in WAS_Text_to_Conditioning | ✅ | ✅ |
| Custom node that loads embedding programmatically | ❌ | ✅ |
| Embedding loaded multiple times in same prompt | ✅ (dedup via set) | ✅ (dedup via set) |
| Embedding file not found | N/A | ✅ (can log) |
| Embedding dimension mismatch | N/A | ✅ (can log) |
| Text encoder with non-standard tokenizer (LLaMA, T5...) | Partial | ✅ (if it calls load_embed) |
## Migration Path: Standalone → Hybrid
### Phase 1 — Plan A (当前状态)
- Prompt text parsing only
- No monkey-patching required
- Covers all standard workflows
### Phase 2 — Enable Plan B (未来工作)
1. Add monkey-patch of `load_embed` in `metadata_collector/metadata_hook.py` (alongside
the existing `_map_node_over_list` hook)
2. Collect loaded embedding names in a `set()` on the registry
3. In `UsageStats._process_embeddings()`, merge the Plan A results (from prompt text)
with the Plan B results (from the patch)
4. Add `prompt_data` field on MetadataRegistry to store loaded embeddings per prompt
### Deduplication
```python
# Merge Plan A + Plan B results in _process_embeddings
plan_a_names = extract_from_prompt_texts(prompts_data)
plan_b_names = registry.get_loaded_embeddings(prompt_id)
all_names = plan_a_names | plan_b_names
```
## Testing the Hybrid
| Scenario | What to verify |
|----------|---------------|
| Standard `embedding:name` in prompt | Plan A captures it |
| Embedding loaded by custom node script | Plan B captures it |
| Both paths fire for same embedding | No double-counting (dedup) |
| Embedding name resolves to hash | EmbeddingScanner.get_hash_by_filename works |
| No embedding scanner available | Graceful skip, no crash |
| Missing embedding file | Plan B logs warning, Plan A skips gracefully |
| Empty prompt | No crash, no entries |
| Standalone mode | Both plans disabled gracefully |
## Key Files Reference
| File | Role |
|------|------|
| `py/utils/usage_stats.py` | Core — `_process_embeddings()` for Plan A |
| `py/metadata_collector/constants.py` | `EMBEDDINGS` category constant |
| `py/metadata_collector/metadata_hook.py` | Future — monkey-patch for Plan B |
| `py/services/embedding_scanner.py` | Hash resolution service |
| `py/routes/stats_routes.py` | Already handles `usage_data.get('embeddings', {})` |
| `comfy/sd1_clip.py` (ComfyUI) | `load_embed()` — Plan B target |
+35 -135
View File
File diff suppressed because one or more lines are too long
+464 -291
View File
File diff suppressed because it is too large Load Diff
-183
View File
@@ -1,183 +0,0 @@
## Overview
The **LoRA Manager Civitai Extension** is a Browser extension designed to work seamlessly with [LoRA Manager](https://github.com/willmiao/ComfyUI-Lora-Manager) to significantly enhance your browsing experience on [Civitai](https://civitai.com). With this extension, you can:
✅ Instantly see which models are already present in your local library
✅ Download new models with a single click
✅ Manage downloads efficiently with queue and parallel download support
✅ Keep your downloaded models automatically organized according to your custom settings
![Civitai Models page](https://github.com/willmiao/ComfyUI-Lora-Manager/blob/main/wiki-images/civitai-models-page.png)
**Update:** It now also supports browsing on [CivArchive](https://civarchive.com/) (formerly CivitaiArchive).
![CivArchive Models page](https://github.com/willmiao/ComfyUI-Lora-Manager/blob/main/wiki-images/civarchive-models-page.png)
---
## Why Supporter Access?
LoRA Manager is built with love for the Stable Diffusion and ComfyUI communities. Your support makes it possible for me to keep improving and maintaining the tool full-time.
Supporter-exclusive features help ensure the long-term sustainability of LoRA Manager, allowing continuous updates, new features, and better performance for everyone.
Every contribution directly fuels development and keeps the core LoRA Manager free and open-source. In addition to monthly supporters, one-time donation supporters will also receive a license key, with the duration scaling according to the contribution amount. Thank you for helping keep this project alive and growing. ❤️
---
## Installation
### Supported Browsers & Installation Methods
| Browser | Installation Method |
|--------------------|-------------------------------------------------------------------------------------|
| **Google Chrome** | [Chrome Web Store link](https://chromewebstore.google.com/detail/capigligggeijgmocnaflanlbghnamgm?utm_source=item-share-cb) |
| **Microsoft Edge** | Install via Chrome Web Store (compatible) |
| **Brave Browser** | Install via Chrome Web Store (compatible) |
| **Opera** | Install via Chrome Web Store (compatible) |
| **Firefox** | <div id="firefox-install" class="install-ok"><a href="https://github.com/willmiao/lm-civitai-extension-firefox/releases/latest/download/extension.xpi">📦 Install Firefox Extension (reviewed and verified by Mozilla)</a></div> |
For non-Chrome browsers (e.g., Microsoft Edge), you can typically install extensions from the Chrome Web Store by following these steps: open the extension’s Chrome Web Store page, click 'Get extension', then click 'Allow' when prompted to enable installations from other stores, and finally click 'Add extension' to complete the installation.
---
## Privacy & Security
I understand concerns around browser extensions and privacy, and I want to be fully transparent about how the **LM Civitai Extension** works:
- **Reviewed and Verified**
This extension has been **manually reviewed and approved by the Chrome Web Store**. The Firefox version uses the **exact same code** (only the packaging format differs) and has passed **Mozilla’s Add-on review**.
- **Minimal Network Access**
The only external server this extension connects to is:
**`https://willmiao.shop`** — used solely for **license validation**.
It does **not collect, transmit, or store any personal or usage data**.
No browsing history, no user IDs, no analytics, no hidden trackers.
- **Local-Only Model Detection**
Model detection and LoRA Manager communication all happen **locally** within your browser, directly interacting with your local LoRA Manager backend.
I value your trust and are committed to keeping your local setup private and secure. If you have any questions, feel free to reach out!
---
## How to Use
After installing the extension, you'll automatically receive a **7-day trial** to explore all features.
When the extension is correctly installed and your license is valid:
- Open **Civitai**, and you'll see visual indicators added by the extension on model cards, showing:
- ✅ Models already present in your local library
- ⬇️ A download button for models not in your library
Clicking the download button adds the corresponding model version to the download queue, waiting to be downloaded. You can set up to **5 models to download simultaneously**.
### Visual Indicators Appear On:
- **Home Page** — Featured models
- **Models Page**
- **Creator Profiles** — If the creator has set their models to be visible
- **Recommended Resources** — On individual model pages
### Version Buttons on Model Pages
On a specific model page, visual indicators also appear on version buttons, showing which versions are already in your local library.
**Starting from v0.4.8**, model pages use a dedicated download button for better compatibility. When switching to a specific version by clicking a version button:
- The new **dedicated download button** directly triggers download via **LoRA Manager**
- The **original download button** remains unchanged for standard browser downloads
![Civitai Model Page](https://github.com/willmiao/ComfyUI-Lora-Manager/blob/main/wiki-images/civitai-model-page.png)
### Hide Models Already in Library (Beta)
**New in v0.4.8**: A new **Hide models already in library (Beta)** option makes it easier to focus on models you haven't added yet. It can be enabled from Settings, or toggled quickly using **Ctrl + Shift + H** (macOS: **Command + Shift + H**).
### Resources on Image Pages — now shows in-library indicators for image resources plus one-click recipe import
- **One-Click Import Civitai Image as Recipe** — Import any Civitai image as a recipe with a single click in the Resources Used panel.
- **Auto-Queue Missing Assets** — In Settings you can decide if LoRAs or checkpoints referenced by that image should automatically be added to your download queue.
- **More Accurate Metadata** — Importing directly from the page is faster than copying inside LM and keeps on-site tags and other metadata perfectly aligned.
![Civitai Image Page](https://github.com/willmiao/ComfyUI-Lora-Manager/blob/main/wiki-images/civitai-image-page.jpg)
[![alt](url)](https://github.com/user-attachments/assets/41fd4240-c949-4f83-bde7-8f3124c09494)
---
## Model Download Location & LoRA Manager Settings
To use the **one-click download function**, you must first set:
- Your **Default LoRAs Root**
- Your **Default Checkpoints Root**
These are set within LoRA Manager's settings.
When everything is configured, downloaded model files will be placed in:
`<Default_Models_Root>/<Base_Model_of_the_Model>/<First_Tag_of_the_Model>`
### Update: Default Path Customization (2025-07-21)
A new setting to customize the default download path has been added in the nightly version. You can now personalize where models are saved when downloading via the LM Civitai Extension.
![Default Path Customization](https://github.com/willmiao/ComfyUI-Lora-Manager/blob/main/wiki-images/default-path-customization.png)
The previous YAML path mapping file will be deprecated—settings will now be unified in settings.json to simplify configuration.
---
## Backend Port Configuration
If your **ComfyUI** or **LoRA Manager** backend is running on a port **other than the default 8188**, you must configure the backend port in the extension's settings.
After correctly setting and saving the port, you'll see in the extension's header area:
- A **Healthy** status with the tooltip: `Connected to LoRA Manager on port xxxx`
---
## Advanced Usage
### Connecting to a Remote LoRA Manager
If your LoRA Manager is running on another computer, you can still connect from your browser using port forwarding.
> **Why can't you set a remote IP directly?**
>
> For privacy and security, the extension only requests access to `http://127.0.0.1/*`. Supporting remote IPs would require much broader permissions, which may be rejected by browser stores and could raise user concerns.
**Solution: Port Forwarding with `socat`**
On your browser computer, run:
`socat TCP-LISTEN:8188,bind=127.0.0.1,fork TCP:REMOTE.IP.ADDRESS.HERE:8188`
- Replace `REMOTE.IP.ADDRESS.HERE` with the IP of the machine running LoRA Manager.
- Adjust the port if needed.
This lets the extension connect to `127.0.0.1:8188` as usual, with traffic forwarded to your remote server.
_Thanks to user **Temikus** for sharing this solution!_
---
## Roadmap
The extension will evolve alongside **LoRA Manager** improvements. Planned features include:
- [x] Support for **additional model types** (e.g., embeddings)
- [x] One-click **Recipe Import**
- [x] Display of in-library status for all resources in the **Resources Used** section of the image page
- [x] One-click **Auto-organize Models**
- [x] **Hide models already in library (Beta)** - Focus on models you haven't added yet
**Stay tuned — and thank you for your support!**
---
File diff suppressed because one or more lines are too long
+2140 -1866
View File
File diff suppressed because it is too large Load Diff
+322 -48
View File
@@ -15,10 +15,14 @@
"settings": "Settings",
"help": "Help",
"add": "Add",
"close": "Close"
"close": "Close",
"menu": "Menu",
"remove": "Remove",
"change": "Change"
},
"status": {
"loading": "Loading...",
"cancelling": "Cancelling...",
"unknown": "Unknown",
"date": "Date",
"version": "Version",
@@ -101,6 +105,7 @@
"removeFromFavorites": "Remove from favorites",
"viewOnCivitai": "View on Civitai",
"notAvailableFromCivitai": "Not available from Civitai",
"viewOnHuggingFace": "View on Hugging Face",
"sendToWorkflow": "Send to ComfyUI (Click: Append, Shift+Click: Replace)",
"copyLoRASyntax": "Copy LoRA Syntax",
"checkpointNameCopied": "Checkpoint name copied",
@@ -110,6 +115,7 @@
"replacePreview": "Replace Preview",
"copyCheckpointName": "Copy checkpoint name",
"copyEmbeddingName": "Copy embedding name",
"embeddingNameCopied": "Embedding syntax copied",
"sendCheckpointToWorkflow": "Send to ComfyUI",
"sendEmbeddingToWorkflow": "Send to ComfyUI"
},
@@ -140,6 +146,10 @@
},
"usage": {
"timesUsed": "Times used"
},
"footer": {
"versionCount": "{count} versions",
"viewAllVersions": "View all local versions"
}
},
"globalContextMenu": {
@@ -178,6 +188,9 @@
},
"manageExcludedModels": {
"label": "Manage Excluded Models"
},
"groupByModel": {
"label": "Group by Model"
}
},
"header": {
@@ -190,13 +203,7 @@
"statistics": "Stats"
},
"search": {
"placeholder": "Search...",
"placeholders": {
"loras": "Search LoRAs...",
"recipes": "Search recipes...",
"checkpoints": "Search checkpoints...",
"embeddings": "Search embeddings..."
},
"placeholder": "Search",
"options": "Search Options",
"searchIn": "Search In:",
"notAvailable": "Search not available on statistics page",
@@ -225,12 +232,17 @@
"presetOverwriteConfirm": "Preset \"{name}\" already exists. Overwrite?",
"presetNamePlaceholder": "Preset name...",
"baseModel": "Base Model",
"baseModelSearchPlaceholder": "Search base models...",
"modelTags": "Tags (Top 20)",
"modelTypes": "Model Types",
"license": "License",
"noCreditRequired": "No Credit Required",
"allowSellingGeneratedContent": "Allow Selling",
"allowSellingGeneratedContentTooltip": "Allow selling generated images",
"noCreditRequiredTooltip": "Use the model without crediting the creator",
"noTags": "No tags",
"autoTags": "Auto Tags",
"noBaseModelMatches": "No base models match the current search.",
"clearAll": "Clear All Filters",
"any": "Any",
"all": "All",
@@ -241,7 +253,18 @@
"toggle": "Toggle theme",
"switchToLight": "Switch to light theme",
"switchToDark": "Switch to dark theme",
"switchToAuto": "Switch to auto theme"
"switchToAuto": "Switch to auto theme",
"presets": "Theme Presets",
"default": "Default",
"nord": "Nord",
"midnight": "Midnight",
"monokai": "Monokai",
"dracula": "Dracula",
"solarized": "Solarized",
"mode": "Mode",
"light": "Light",
"dark": "Dark",
"auto": "Auto"
},
"actions": {
"checkUpdates": "Check Updates",
@@ -253,6 +276,9 @@
"civitaiApiKey": "Civitai API Key",
"civitaiApiKeyPlaceholder": "Enter your Civitai API key",
"civitaiApiKeyHelp": "Used for authentication when downloading models from Civitai",
"civitaiApiKeyConfigured": "Configured",
"civitaiApiKeyNotConfigured": "Not configured",
"civitaiApiKeySet": "Set up",
"civitaiHost": {
"label": "Civitai host",
"help": "Choose which Civitai site opens when using View on Civitai links.",
@@ -261,6 +287,20 @@
"red": "civitai.red (unrestricted)"
}
},
"downloadBackend": {
"label": "Download backend",
"help": "Choose how model files are downloaded. Python uses the built-in downloader. aria2 uses the recommended external downloader process.",
"options": {
"python": "Python (built-in)",
"aria2": "aria2 (recommended)"
}
},
"aria2cPath": {
"label": "aria2c path",
"help": "Optional path to the aria2c executable. Leave empty to use aria2c from your system PATH.",
"placeholder": "Leave empty to use aria2c from PATH"
},
"aria2HelpLink": "Learn how to set up the aria2 download backend",
"civitaiHostBanner": {
"title": "Civitai host preference available",
"content": "Civitai now uses civitai.com for SFW content and civitai.red for unrestricted content. You can change which site opens by default in Settings.",
@@ -276,8 +316,10 @@
},
"sections": {
"contentFiltering": "Content Filtering",
"downloads": "Downloads",
"videoSettings": "Video Settings",
"layoutSettings": "Layout Settings",
"licenseIcons": "License Icons",
"misc": "Miscellaneous",
"backup": "Backups",
"folderSettings": "Default Roots",
@@ -285,7 +327,7 @@
"extraFolderPaths": "Extra Folder Paths",
"downloadPathTemplates": "Download Path Templates",
"priorityTags": "Priority Tags",
"updateFlags": "Update Flags",
"versionScope": "Version Scope",
"exampleImages": "Example Images",
"autoOrganize": "Auto-organize",
"metadata": "Metadata",
@@ -390,6 +432,8 @@
"help": "When enabled, versions downloaded before will be skipped."
},
"layoutSettings": {
"groupByModel": "Group by Model",
"groupByModelHelp": "When enabled, only the latest version of each Civitai model is shown as a single card. Older versions are hidden.",
"displayDensity": "Display Density",
"displayDensityOptions": {
"default": "Default",
@@ -411,6 +455,8 @@
"hover": "Reveal on Hover"
},
"cardInfoDisplayHelp": "Choose when to display model information and action buttons",
"showVersionOnCard": "Show Version on Card",
"showVersionOnCardHelp": "Show or hide the version name on model cards",
"modelCardFooterAction": "Model Card Button Action",
"modelCardFooterActionOptions": {
"exampleImages": "Open Example Images",
@@ -422,7 +468,9 @@
"modelName": "Model Name",
"fileName": "File Name"
},
"modelNameDisplayHelp": "Choose what to display in the model card footer"
"modelNameDisplayHelp": "Choose what to display in the model card footer",
"cardBlurAmount": "Card Overlay Blur",
"cardBlurAmountHelp": "Adjust the blur intensity of the header and footer overlays on model and recipe cards (0 = no blur, 20 = maximum blur)."
},
"folderSettings": {
"activeLibrary": "Active Library",
@@ -522,26 +570,51 @@
"downloadLocationHelp": "Enter the folder path where example images from Civitai will be saved",
"autoDownload": "Auto Download Example Images",
"autoDownloadHelp": "Automatically download example images for models that don't have them (requires download location to be set)",
"openMode": "Open Example Images Action",
"openModeHelp": "Choose whether the action opens on the server, copies a mapped local path, or launches a custom URI.",
"openModeOptions": {
"system": "Open on server",
"clipboard": "Copy local path",
"uriTemplate": "Open custom URI"
},
"localRoot": "Local Example Images Root",
"localRootHelp": "Optional local or mounted root that mirrors the server example images directory. If blank, the server path is reused.",
"localRootPlaceholder": "Example: /Volumes/ComfyUI/example_images",
"uriTemplate": "Open URI Template",
"uriTemplateHelp": "Use a custom deep link such as a file URI or a Shortcuts link.",
"uriTemplatePlaceholder": "Example: shortcuts://run-shortcut?name=Open%20Finder&input=text&text={{encoded_local_path}}",
"uriTemplatePlaceholders": "Available placeholders: {{local_path}}, {{encoded_local_path}}, {{relative_path}}, {{encoded_relative_path}}, {{file_uri}}, {{encoded_file_uri}}",
"openModeWikiLink": "Learn more about remote open modes",
"optimizeImages": "Optimize Downloaded Images",
"optimizeImagesHelp": "Optimize example images to reduce file size and improve loading speed (metadata will be preserved)",
"download": "Download",
"restartRequired": "Requires restart"
},
"updateFlagStrategy": {
"label": "Update Flag Strategy",
"help": "Decide whether update badges should only appear when a new release shares the same base model as your local files or whenever any newer version exists for that model.",
"versionGrouping": {
"label": "Version Grouping",
"help": "Decide how versions are grouped for display: by base model or all together. Also controls update badge logic and the VLM version list filtering.",
"options": {
"sameBase": "Match updates by base model",
"any": "Flag any available update"
"sameBase": "Group by base model (same_base)",
"any": "Show all versions (any)"
}
},
"hideEarlyAccessUpdates": {
"label": "Hide Early Access Updates",
"help": "When enabled, models with only early access updates will not show 'Update available' badge"
},
"licenseIcons": {
"useNewStyle": "Use updated license icons",
"useNewStyleHelp": "Display license permissions with colored indicators (new style) or restriction-only icons (classic style). Mirroring the current CivitAI design."
},
"misc": {
"includeTriggerWords": "Include Trigger Words in LoRA Syntax",
"includeTriggerWordsHelp": "Include trained trigger words when copying LoRA syntax to clipboard"
"includeTriggerWordsHelp": "Include trained trigger words when copying LoRA syntax to clipboard",
"loraSyntaxFormat": "LoRA Syntax Format",
"loraSyntaxFormatHelp": "LoRA syntax format. Full includes subfolder path (<lora:style/anime/x:1.0>) for lossless model resolution. Legacy uses filename only (<lora:x:1.0>) — A1111 convention, may be ambiguous with duplicate filenames across folders.",
"loraSyntaxFormatOptions": {
"full": "Full path (subfolder/name)",
"legacy": "Legacy A1111 (name only)"
}
},
"metadataArchive": {
"enableArchiveDb": "Enable Metadata Archive Database",
@@ -601,12 +674,14 @@
"sizeAsc": "Smallest",
"usage": "Use Count",
"usageDesc": "Most",
"usageAsc": "Least"
"usageAsc": "Least",
"versionsCount": "Local Versions",
"versionsCountDesc": "Most versions first",
"versionsCountAsc": "Fewest versions first",
"versionIdDesc": "Newest version first"
},
"refresh": {
"title": "Refresh model list",
"quick": "Sync Changes",
"quickTooltip": "Scan for new or missing model files so the list stays current.",
"full": "Rebuild Cache",
"fullTooltip": "Reload all model details from metadata files—use if the library looks out of date or after manual edits."
},
@@ -647,16 +722,30 @@
"setContentRating": "Set Content Rating for Selected",
"copyAll": "Copy Selected Syntax",
"refreshAll": "Refresh Selected Metadata",
"repairMetadata": "Repair Metadata for Selected",
"reimportMetadata": "Re-import from Source",
"checkUpdates": "Check Updates for Selected",
"moveAll": "Move Selected to Folder",
"autoOrganize": "Auto-Organize Selected",
"skipMetadataRefresh": "Skip Metadata Refresh for Selected",
"resumeMetadataRefresh": "Resume Metadata Refresh for Selected",
"deleteAll": "Delete Selected Models",
"setFavorite": "Set as Favorite",
"setFavoriteCount": "Set as Favorite ({favorited}/{total})",
"unfavorite": "Remove from Favorites",
"deleteAll": "Delete Selected",
"downloadMissingLoras": "Download Missing LoRAs",
"downloadExamples": "Download Example Images",
"clear": "Clear Selection",
"skipMetadataRefreshCount": "Skip ({count} models)",
"resumeMetadataRefreshCount": "Resume ({count} models)",
"sendToWorkflow": "Send to Workflow",
"sections": {
"workflow": "Workflow",
"metadata": "Metadata",
"attributes": "Attributes",
"organize": "Organize",
"download": "Download"
},
"autoOrganizeProgress": {
"initializing": "Initializing auto-organize...",
"starting": "Starting auto-organize for {type}...",
@@ -682,6 +771,7 @@
"setContentRating": "Set Content Rating",
"moveToFolder": "Move to Folder",
"repairMetadata": "Repair metadata",
"reimportMetadata": "Re-import from Source",
"excludeModel": "Exclude Model",
"restoreModel": "Restore Model",
"deleteModel": "Delete Model",
@@ -769,8 +859,6 @@
},
"refresh": {
"title": "Refresh recipe list",
"quick": "Sync Changes",
"quickTooltip": "Sync changes - quick refresh without rebuilding cache",
"full": "Rebuild Cache",
"fullTooltip": "Rebuild cache - full rescan of all recipe files"
},
@@ -811,6 +899,13 @@
"skipped": "Recipe already at latest version, no repair needed",
"failed": "Failed to repair recipe: {message}",
"missingId": "Cannot repair recipe: Missing recipe ID"
},
"reimport": {
"starting": "Re-importing recipe from source...",
"success": "Recipe re-imported successfully",
"noSourceUrl": "Recipe has no source URL, cannot re-import",
"failed": "Failed to re-import recipe: {message}",
"missingId": "Cannot re-import recipe: Missing recipe ID"
}
},
"batchImport": {
@@ -889,8 +984,9 @@
"sidebar": {
"modelRoot": "Root",
"collapseAll": "Collapse All Folders",
"pinSidebar": "Pin Sidebar",
"unpinSidebar": "Unpin Sidebar",
"hideOnThisPage": "Hide sidebar on this page",
"showSidebar": "Show sidebar",
"sidebarHiddenNotification": "Folder sidebar hidden on {page} page",
"switchToListView": "Switch to List View",
"switchToTreeView": "Switch to Tree View",
"recursiveOn": "Include subfolders",
@@ -910,6 +1006,13 @@
"empty": {
"noFolders": "No folders found",
"dragHint": "Drag items here to create folders"
},
"folderUpdateCheck": {
"label": "Check for updates in this folder",
"loading": "Checking {type} updates for this folder...",
"success": "Found {count} update(s) for {type}s in this folder",
"none": "All {type}s in this folder are up to date",
"error": "Failed to check folder for {type} updates: {message}"
}
},
"statistics": {
@@ -921,6 +1024,18 @@
"storage": "Storage",
"insights": "Insights"
},
"metrics": {
"totalModels": "Total Models",
"totalStorage": "Total Storage",
"totalGenerations": "Total Generations",
"usageRate": "Usage Rate",
"loras": "LoRAs",
"checkpoints": "Checkpoints",
"embeddings": "Embeddings",
"uniqueTags": "Unique Tags",
"unusedModels": "Unused Models",
"avgUsesPerModel": "Avg. Uses/Model"
},
"usage": {
"mostUsedLoras": "Most Used LoRAs",
"mostUsedCheckpoints": "Most Used Checkpoints",
@@ -938,13 +1053,77 @@
},
"insights": {
"smartInsights": "Smart Insights",
"recommendations": "Recommendations"
"recommendations": "Recommendations",
"noInsights": "No insights available",
"unusedLoras": {
"high": {
"title": "High Number of Unused LoRAs",
"description": "{percent}% of your LoRAs ({count}/{total}) have never been used.",
"suggestion": "Consider organizing or archiving unused models to free up storage space."
}
},
"unusedCheckpoints": {
"detected": {
"title": "Unused Checkpoints Detected",
"description": "{percent}% of your checkpoints ({count}/{total}) have never been used.",
"suggestion": "Review and consider removing checkpoints you no longer need."
}
},
"unusedEmbeddings": {
"high": {
"title": "High Number of Unused Embeddings",
"description": "{percent}% of your embeddings ({count}/{total}) have never been used.",
"suggestion": "Consider organizing or archiving unused embeddings to optimize your collection."
}
},
"collection": {
"large": {
"title": "Large Collection Detected",
"description": "Your model collection is using {size} of storage.",
"suggestion": "Consider using external storage or cloud solutions for better organization."
}
},
"activity": {
"active": {
"title": "Active User",
"description": "You've completed {count} generations so far!",
"suggestion": "Keep exploring and creating amazing content with your models."
}
}
},
"charts": {
"collectionOverview": "Collection Overview",
"baseModelDistribution": "Base Model Distribution",
"usageTrends": "Usage Trends (Last 30 Days)",
"usageDistribution": "Usage Distribution"
"usageDistribution": "Usage Distribution",
"date": "Date",
"usageCount": "Usage Count",
"fileSizeBytes": "File Size (bytes)",
"models": "Models",
"loraUsage": "LoRA Usage",
"checkpointUsage": "Checkpoint Usage",
"embeddingUsage": "Embedding Usage"
},
"modelTypes": {
"lora": "LoRA",
"locon": "LyCORIS",
"dora": "DoRA",
"checkpoint": "Checkpoint",
"diffusion_model": "Diffusion Model",
"embedding": "Embeddings"
},
"placeholders": {
"loading": "Loading...",
"noModels": "No models found",
"errorLoading": "Error loading data",
"noStorageData": "No storage data available",
"rootFolder": "Root",
"chartLibraryMissing": "Chart requires Chart.js library"
},
"tooltips": {
"tagCount": "{tag}: {count} models",
"chartUsage": "{name}: {size}, {count} uses",
"chartPercentage": "{label}: {value} ({pct}%)"
}
},
"modals": {
@@ -954,9 +1133,12 @@
"download": {
"title": "Download Model from URL",
"titleWithType": "Download {type} from URL",
"url": "Civitai URL",
"civitaiUrl": "Civitai URL:",
"civitaiUrl": "Civitai URL(s):",
"placeholder": "https://civitai.com/models/...",
"urlHint": "Enter one CivitAI, CivArchive, or Hugging Face URL per line. Supports multiple URLs for batch download.",
"selectHfFiles": "Select file(s) to download from this repository:",
"selectAll": "Select All",
"fetchingRepoFiles": "Fetching repository files...",
"locationPreview": "Download Location Preview",
"useDefaultPath": "Use Default Path",
"useDefaultPathTooltip": "When enabled, files are automatically organized using configured path templates",
@@ -978,9 +1160,16 @@
"downloadedTooltip": "Previously downloaded, but it is not currently in your library.",
"alreadyInLibrary": "Already in Library",
"autoOrganizedPath": "[Auto-organized by path template]",
"fileSelection": {
"title": "Select File Format",
"files": "files",
"select": "Select File"
},
"errors": {
"invalidUrl": "Invalid Civitai URL format",
"noVersions": "No versions available for this model"
"noVersions": "No versions available for this model",
"mixedSources": "Cannot mix CivitAI and Hugging Face URLs in the same batch.",
"noModelFiles": "No model files found in this repository."
},
"status": {
"preparing": "Preparing download...",
@@ -1042,6 +1231,12 @@
"countMessage": "models will be permanently deleted.",
"action": "Delete All"
},
"bulkDeleteRecipes": {
"title": "Delete Multiple Recipes",
"message": "Are you sure you want to delete all selected recipes and their associated files?",
"countMessage": "recipes will be permanently deleted.",
"action": "Delete All"
},
"checkUpdates": {
"title": "Check updates for all {typePlural}?",
"message": "This checks every {typePlural} in your library for updates. Large collections may take a little longer.",
@@ -1122,8 +1317,11 @@
"editModelName": "Edit model name",
"editFileName": "Edit file name",
"editBaseModel": "Edit base model",
"editVersionName": "Edit version name",
"viewOnCivitai": "View on Civitai",
"viewOnCivitaiText": "View on Civitai",
"viewOnHuggingFace": "View on Hugging Face",
"viewOnHuggingFaceText": "View on Hugging Face",
"viewCreatorProfile": "View Creator Profile",
"openFileLocation": "Open File Location",
"sendToWorkflow": "Send to ComfyUI",
@@ -1153,7 +1351,9 @@
},
"notes": {
"saved": "Notes saved successfully",
"saveFailed": "Failed to save notes"
"saveFailed": "Failed to save notes",
"showMore": "Show more",
"showLess": "Show less"
},
"usageTips": {
"addPresetParameter": "Add preset parameter...",
@@ -1174,6 +1374,8 @@
"cancel": "Cancel editing",
"save": "Save changes",
"addPlaceholder": "Type to add or click suggestions below",
"editWord": "Edit trigger word",
"editPlaceholder": "Edit trigger word",
"copyWord": "Copy trigger word",
"deleteWord": "Delete trigger word",
"suggestions": {
@@ -1256,12 +1458,15 @@
"earlyAccess": "Early Access",
"earlyAccessTooltip": "This version currently requires Civitai early access",
"ignored": "Ignored",
"ignoredTooltip": "Update notifications are disabled for this version"
"ignoredTooltip": "Update notifications are disabled for this version",
"onSiteOnly": "On-Site Only",
"onSiteOnlyTooltip": "This version is only available for on-site generation on Civitai"
},
"actions": {
"download": "Download",
"downloadTooltip": "Download this version",
"downloadEarlyAccessTooltip": "Download this early access version from Civitai",
"downloadNotAllowedTooltip": "This version is only available for on-site generation on Civitai",
"delete": "Delete",
"deleteTooltip": "Delete this local version",
"ignore": "Ignore",
@@ -1273,7 +1478,7 @@
"resumeModelUpdates": "Resume updates for this model",
"ignoreModelUpdates": "Ignore updates for this model",
"viewLocalVersions": "View all local versions",
"viewLocalTooltip": "Coming soon"
"viewLocalTooltip": "Show all local versions of this model on the main page"
},
"filters": {
"label": "Base filter",
@@ -1301,6 +1506,21 @@
"versionDeleted": "Version deleted"
}
}
},
"metadataFetchSummary": {
"title": "Metadata Fetch Summary",
"statSuccess": "Success",
"statFailed": "Failed",
"statSkipped": "Skipped",
"statTotal": "Total Scanned",
"statDuration": "Duration",
"successMessage": "All {count} {type}s updated successfully!",
"failedItems": "Failed Items ({count})",
"close": "Close",
"copyReport": "Copy Report",
"downloadCsv": "Download CSV",
"columnModelName": "Model Name",
"columnError": "Error"
}
},
"modelTags": {
@@ -1314,15 +1534,6 @@
"duplicate": "This tag already exists"
}
},
"keyboard": {
"navigation": "Keyboard Navigation:",
"shortcuts": {
"pageUp": "Scroll up one page",
"pageDown": "Scroll down one page",
"home": "Jump to top",
"end": "Jump to bottom"
}
},
"initialization": {
"title": "Initializing",
"message": "Preparing your workspace...",
@@ -1410,11 +1621,17 @@
"noMatchingNodes": "No compatible nodes available in the current workflow",
"noTargetNodeSelected": "No target node selected",
"modelUpdated": "Model updated in workflow",
"modelFailed": "Failed to update model node"
"modelFailed": "Failed to update model node",
"embeddingAdded": "Embedding added to workflow",
"embeddingFailed": "Failed to add embedding",
"promptSent": "Prompt sent to workflow",
"promptFailed": "Failed to send prompt"
},
"nodeSelector": {
"recipe": "Recipe",
"lora": "LoRA",
"embedding": "Embedding",
"prompt": "Prompt",
"replace": "Replace",
"append": "Append",
"selectTargetNode": "Select target node",
@@ -1424,6 +1641,10 @@
"opened": "Example images folder opened",
"openingFolder": "Opening example images folder",
"failedToOpen": "Failed to open example images folder",
"copiedPath": "Path copied to clipboard: {{path}}",
"clipboardFallback": "Path: {{path}}",
"copiedUri": "Link copied to clipboard: {{uri}}",
"uriClipboardFallback": "Link: {{uri}}",
"setupRequired": "Example Images Storage",
"setupDescription": "To add custom example images, you need to set a download location first.",
"setupUsage": "This path is used for both downloaded and custom example images.",
@@ -1587,12 +1808,17 @@
"noRecipeId": "No recipe ID available",
"sendToWorkflowFailed": "Failed to send recipe to workflow: {message}",
"copyFailed": "Error copying recipe syntax: {message}",
"createError": "Error creating recipe: {message}",
"createFailed": "Failed to create recipe: {error}",
"createMissingData": "Missing required data to create recipe",
"created": "Recipe created successfully",
"noMissingLoras": "No missing LoRAs to download",
"missingLorasInfoFailed": "Failed to get information for missing LoRAs",
"preparingForDownloadFailed": "Error preparing LoRAs for download",
"enterLoraName": "Please enter a LoRA name or syntax",
"reconnectedSuccessfully": "LoRA reconnected successfully",
"reconnectFailed": "Error reconnecting LoRA: {message}",
"noPromptToSend": "No prompt to send",
"cannotSend": "Cannot send recipe: Missing recipe ID",
"sendFailed": "Failed to send recipe to workflow",
"sendError": "Error sending recipe to workflow",
@@ -1625,6 +1851,13 @@
"batchImportBrowseFailed": "Failed to browse directory: {message}",
"batchImportDirectorySelected": "Directory selected: {path}",
"noRecipesSelected": "No recipes selected",
"repairBulkComplete": "Repair complete: {repaired} repaired, {skipped} skipped (of {total})",
"repairBulkSkipped": "No repair needed for any of the {total} selected recipes",
"repairBulkFailed": "Failed to repair selected recipes: {message}",
"reimporting": "Re-importing recipe from source...",
"reimportSuccess": "Recipe re-imported successfully",
"reimportBulkComplete": "Re-import complete: {completed} re-imported, {failed} failed (of {total})",
"reimportBulkFailed": "Failed to re-import some recipes",
"noMissingLorasInSelection": "No missing LoRAs found in selected recipes",
"noLoraRootConfigured": "No LoRA root directory configured. Please set a default LoRA root in settings."
},
@@ -1655,6 +1888,11 @@
"bulkContentRatingSet": "Set content rating to {level} for {count} model(s)",
"bulkContentRatingPartial": "Set content rating to {level} for {success} model(s), {failed} failed",
"bulkContentRatingFailed": "Failed to update content rating for selected models",
"bulkFavoriteUpdating": "Adding {count} model(s) to favorites...",
"bulkUnfavoriteUpdating": "Removing {count} model(s) from favorites...",
"bulkFavoritePartialAdded": "Added {success} model(s) to favorites, {failed} failed",
"bulkFavoritePartialRemoved": "Removed {success} model(s) from favorites, {failed} failed",
"bulkFavoriteFailed": "Failed to update favorite status for selected models",
"bulkUpdatesChecking": "Checking selected {type}(s) for updates...",
"bulkUpdatesSuccess": "Updates available for {count} selected {type}(s)",
"bulkUpdatesNone": "No updates found for selected {type}(s)",
@@ -1745,8 +1983,8 @@
},
"triggerWords": {
"loadFailed": "Could not load trained words",
"tooLong": "Trigger word should not exceed 100 words",
"tooMany": "Maximum 30 trigger words allowed",
"tooLong": "Trigger word should not exceed 500 words",
"tooMany": "Maximum 100 trigger words allowed",
"alreadyExists": "This trigger word already exists",
"updateSuccess": "Trigger words updated successfully",
"updateFailed": "Failed to update trigger words",
@@ -1837,7 +2075,9 @@
"bulkMoveSuccess": "Successfully moved {successCount} {type}s",
"exampleImagesDownloadSuccess": "Successfully downloaded example images!",
"exampleImagesDownloadFailed": "Failed to download example images: {message}",
"moveFailed": "Failed to move item: {message}"
"moveFailed": "Failed to move item: {message}",
"copiedToClipboard": "Copied to clipboard",
"downloadStarted": "Download started"
}
},
"doctor": {
@@ -1857,18 +2097,52 @@
"warning": "Needs Attention",
"error": "Action Required"
},
"issues": {
"civitai_api_key": {
"title": "Civitai API Key"
},
"cache_health": {
"title": "Model Cache Health"
},
"filename_conflicts": {
"title": "Duplicate Filename Conflicts"
},
"ui_version": {
"title": "UI Version"
}
},
"actions": {
"runAgain": "Run Again",
"exportBundle": "Export Bundle"
"exportBundle": "Export Bundle",
"open-settings": "Open Settings",
"open-settings-syntax-format": "Switch to Full Path Syntax",
"repair-cache": "Rebuild Cache",
"resolve-filename-conflicts": "Resolve Conflicts",
"reload-page": "Reload UI"
},
"labels": {
"conflicts": "Conflicts",
"version": "Version"
},
"toast": {
"loadFailed": "Failed to load diagnostics: {message}",
"repairSuccess": "Cache rebuild completed.",
"repairFailed": "Cache rebuild failed: {message}",
"exportSuccess": "Diagnostics bundle exported.",
"exportFailed": "Failed to export diagnostics bundle: {message}"
"exportFailed": "Failed to export diagnostics bundle: {message}",
"conflictsResolved": "{count} filename conflict(s) resolved.",
"conflictsResolveFailed": "Failed to resolve filename conflicts: {message}"
}
},
"conflictConfirm": {
"title": "Resolve Filename Conflicts",
"message": "Renaming by appending a 4-character hash to each duplicate filename.",
"note": "This operation renames files on disk. Model references in existing workflows may need updating if you use the A1111 syntax format.",
"detail": "Example: <code>filename_v1.2</code> → <code>filename_v1.2-ab3c</code>",
"impact": "Will rename <strong>{count}</strong> file(s) across <strong>{groups}</strong> duplicate group(s).",
"confirm": "Rename Files",
"cancel": "Cancel"
},
"banners": {
"versionMismatch": {
"title": "Application Update Detected",
+2140 -1866
View File
File diff suppressed because it is too large Load Diff
+2140 -1866
View File
File diff suppressed because it is too large Load Diff
+2140 -1866
View File
File diff suppressed because it is too large Load Diff
+2140 -1866
View File
File diff suppressed because it is too large Load Diff
+2140 -1866
View File
File diff suppressed because it is too large Load Diff
+2140 -1866
View File
File diff suppressed because it is too large Load Diff
+2140 -1866
View File
File diff suppressed because it is too large Load Diff
+2140 -1866
View File
File diff suppressed because it is too large Load Diff
+178 -8
View File
@@ -1,5 +1,6 @@
import os
import platform
import posixpath
import threading
from pathlib import Path
import folder_paths # type: ignore
@@ -25,21 +26,57 @@ standalone_mode = (
logger = logging.getLogger(__name__)
def _normalize_root_identity(path: str) -> str:
"""Normalize a root path for comparisons across slash styles."""
normalized = posixpath.normpath(path.strip().replace("\\", "/"))
if len(normalized) >= 2 and normalized[1] == ":":
return normalized.lower()
return normalized
def _resolve_valid_default_root(
current: str, primary_paths: List[str], name: str
current: str, primary_paths: List[str], allowed_paths: List[str], name: str
) -> str:
"""Return a valid default root from the current primary path set."""
"""Return a valid default root from the current primary/extra path set."""
valid_paths = [path for path in primary_paths if isinstance(path, str) and path.strip()]
if not valid_paths:
return ""
fallback_paths: List[str] = []
seen: Set[str] = set()
for path in allowed_paths:
if not isinstance(path, str):
continue
stripped = path.strip()
if not stripped:
continue
identity = _normalize_root_identity(stripped)
if identity in seen:
continue
seen.add(identity)
fallback_paths.append(stripped)
if current in valid_paths:
allowed = {_normalize_root_identity(path) for path in fallback_paths}
if current and _normalize_root_identity(current) in allowed:
return current
if not valid_paths:
if not fallback_paths:
return ""
if current:
logger.info(
"Repaired stale %s from '%s' to '%s' because it is not present in primary or extra roots",
name,
current,
fallback_paths[0],
)
else:
logger.info("Auto-setting %s to '%s'", name, fallback_paths[0])
return fallback_paths[0]
if current:
logger.info(
"Repaired stale %s from '%s' to '%s'",
"Repaired stale %s from '%s' to '%s' because it is not present in primary or extra roots",
name,
current,
valid_paths[0],
@@ -135,6 +172,12 @@ class Config:
self.extra_unet_roots: List[str] = []
self.extra_embeddings_roots: List[str] = []
self.recipes_path: str = ""
# Load extra folder paths from active library settings before symlink scan
# so both primary and extra paths are discovered in a single pass.
if not standalone_mode:
self._load_extra_paths_from_settings()
# Scan symbolic links during initialization
self._initialize_symlink_mappings()
@@ -142,6 +185,96 @@ class Config:
# Save the paths to settings.json when running in ComfyUI mode
self.save_folder_paths_to_settings()
def _load_extra_paths_from_settings(self) -> None:
"""Read extra folder paths from the active library and apply them.
Called during ``Config.__init__`` before the symlink scan so both primary and
extra paths are discovered in a single pass. Mirrors the extra-path
portion of ``_apply_library_paths`` without replacing the primary roots
that were already resolved from ComfyUI's ``folder_paths``.
"""
try:
from .services.settings_manager import get_settings_manager
settings_manager = get_settings_manager()
library_name = settings_manager.get_active_library_name()
libraries = settings_manager.get_libraries()
if not library_name or library_name not in libraries:
return
library_config = libraries[library_name]
if not isinstance(library_config, dict):
return
extra_folder_paths = library_config.get("extra_folder_paths")
if not isinstance(extra_folder_paths, dict):
return
extra_lora = extra_folder_paths.get("loras", []) or []
extra_checkpoint = extra_folder_paths.get("checkpoints", []) or []
extra_unet = extra_folder_paths.get("unet", []) or []
extra_embedding = extra_folder_paths.get("embeddings", []) or []
if not any([extra_lora, extra_checkpoint, extra_unet, extra_embedding]):
return
filtered_extra_lora = self._filter_overlapping_extra_lora_paths(
self.loras_roots, extra_lora
)
self.extra_loras_roots = self._prepare_lora_paths(filtered_extra_lora)
(
_,
self.extra_checkpoints_roots,
self.extra_unet_roots,
) = self._prepare_checkpoint_paths(extra_checkpoint, extra_unet)
self.extra_embeddings_roots = self._prepare_embedding_paths(
extra_embedding
)
recipes_path = library_config.get("recipes_path", "")
if isinstance(recipes_path, str) and recipes_path:
self.recipes_path = recipes_path
if self.extra_loras_roots:
logger.info(
"Found extra LoRA roots:"
+ "\n - "
+ "\n - ".join(self.extra_loras_roots)
)
if self.extra_checkpoints_roots:
logger.info(
"Found extra checkpoint roots:"
+ "\n - "
+ "\n - ".join(self.extra_checkpoints_roots)
)
if self.extra_unet_roots:
logger.info(
"Found extra diffusion model roots:"
+ "\n - "
+ "\n - ".join(self.extra_unet_roots)
)
if self.extra_embeddings_roots:
logger.info(
"Found extra embedding roots:"
+ "\n - "
+ "\n - ".join(self.extra_embeddings_roots)
)
logger.info(
"Applied library settings for '%s' with extra paths: loras=%s, "
"checkpoints=%s, embeddings=%s",
library_name,
extra_lora,
extra_checkpoint,
extra_embedding,
)
except Exception as exc:
logger.debug(
"Could not load extra paths from library settings: %s", exc
)
def save_folder_paths_to_settings(self):
"""Persist ComfyUI-derived folder paths to the multi-library settings."""
try:
@@ -226,39 +359,76 @@ class Config:
default_lora_root = _resolve_valid_default_root(
comfy_library.get("default_lora_root", ""),
list(self.loras_roots or []),
list(self.loras_roots or [])
+ list(comfy_library.get("extra_folder_paths", {}).get("loras", []) or []),
"default_lora_root",
)
default_checkpoint_root = _resolve_valid_default_root(
comfy_library.get("default_checkpoint_root", ""),
list(self.checkpoints_roots or []),
list(self.checkpoints_roots or [])
+ list(comfy_library.get("extra_folder_paths", {}).get("checkpoints", []) or []),
"default_checkpoint_root",
)
default_embedding_root = _resolve_valid_default_root(
comfy_library.get("default_embedding_root", ""),
list(self.embeddings_roots or []),
list(self.embeddings_roots or [])
+ list(comfy_library.get("extra_folder_paths", {}).get("embeddings", []) or []),
"default_embedding_root",
)
metadata = dict(comfy_library.get("metadata", {}))
metadata.setdefault("display_name", "ComfyUI")
metadata["source"] = "comfyui"
extra_folder_paths = {}
if isinstance(comfy_library, Mapping):
existing_extra_paths = comfy_library.get("extra_folder_paths", {})
if isinstance(existing_extra_paths, Mapping):
extra_folder_paths = {
key: list(value) if isinstance(value, list) else []
for key, value in existing_extra_paths.items()
}
active_library_name = settings_service.get_active_library_name()
should_activate = (
active_library_name == "comfyui"
or self._should_activate_comfy_library(libraries, libraries_changed)
)
settings_service.upsert_library(
"comfyui",
folder_paths=target_folder_paths,
extra_folder_paths=extra_folder_paths,
default_lora_root=default_lora_root,
default_checkpoint_root=default_checkpoint_root,
default_embedding_root=default_embedding_root,
metadata=metadata,
activate=True,
activate=should_activate,
)
logger.info("Updated 'comfyui' library with current folder paths")
if should_activate:
logger.info("Updated 'comfyui' library with current folder paths")
else:
logger.info(
"Updated 'comfyui' library with current folder paths without activating it"
)
except Exception as e:
logger.warning(f"Failed to save folder paths: {e}")
def _should_activate_comfy_library(
self, libraries: Mapping[str, Any], libraries_changed: bool
) -> bool:
"""Return whether startup sync should make the ComfyUI library active."""
if libraries_changed:
return True
if not libraries:
return True
return "comfyui" in libraries and len(libraries) == 1
def _is_link(self, path: str) -> bool:
try:
if os.path.islink(path):
+19 -33
View File
@@ -33,6 +33,7 @@ from .utils.example_images_migration import ExampleImagesMigration
from .services.websocket_manager import ws_manager
from .services.example_images_cleanup_service import ExampleImagesCleanupService
from .middleware.csp_middleware import relax_csp_for_remote_media
from .middleware.error_middleware import api_json_error
logger = logging.getLogger(__name__)
@@ -76,6 +77,11 @@ class LoraManager:
"""Initialize and register all routes using the new refactored architecture"""
app = PromptServer.instance.app
# Register JSON error middleware for /api/* routes as the outermost
# middleware so it catches errors from all other middlewares.
if api_json_error not in app.middlewares:
app.middlewares.insert(0, api_json_error)
if relax_csp_for_remote_media not in app.middlewares:
# Ensure CSP relaxer executes after ComfyUI's block_external_middleware so it can
# see and extend the restrictive header instead of being overwritten by it.
@@ -184,44 +190,15 @@ class LoraManager:
async def _initialize_services(cls):
"""Initialize all services using the ServiceRegistry"""
try:
# Apply library settings to load extra folder paths before scanning
# Only apply if extra paths haven't been loaded yet (preserves test mocks)
try:
from .services.settings_manager import get_settings_manager
settings_manager = get_settings_manager()
library_name = settings_manager.get_active_library_name()
libraries = settings_manager.get_libraries()
if library_name and library_name in libraries:
library_config = libraries[library_name]
# Only apply settings if extra paths are not already configured
# This preserves values set by tests via monkeypatch
extra_paths = library_config.get("extra_folder_paths", {})
has_extra_paths = (
config.extra_loras_roots
or config.extra_checkpoints_roots
or config.extra_unet_roots
or config.extra_embeddings_roots
)
if not has_extra_paths and any(extra_paths.values()):
config.apply_library_settings(library_config)
logger.info(
"Applied library settings for '%s' with extra paths: loras=%s, checkpoints=%s, embeddings=%s",
library_name,
extra_paths.get("loras", []),
extra_paths.get("checkpoints", []),
extra_paths.get("embeddings", []),
)
except Exception as exc:
logger.warning(
"Failed to apply library settings during initialization: %s", exc
)
# Initialize CivitaiClient first to ensure it's ready for other services
await ServiceRegistry.get_civitai_client()
# Register DownloadManager with ServiceRegistry
await ServiceRegistry.get_download_manager()
# Initialize DownloadQueueService for persistent queue/history
await ServiceRegistry.get_download_queue_service()
await ServiceRegistry.get_backup_service()
from .services.metadata_service import initialize_metadata_providers
@@ -459,5 +436,14 @@ class LoraManager:
try:
logger.info("LoRA Manager: Cleaning up services")
# Cancel any in-flight scanner initialization tasks so thread-pool
# workers (e.g. _initialize_cache_sync) can break out of their loops
# when the server shuts down (e.g. Ctrl+C on WSL).
for name in ("lora_scanner", "checkpoint_scanner", "embedding_scanner"):
scanner = ServiceRegistry.get_service_sync(name)
if scanner is not None and hasattr(scanner, "cancel_task"):
scanner.cancel_task()
logger.debug("LoRA Manager: Cancelled %s", name)
except Exception as e:
logger.error(f"Error during cleanup: {e}", exc_info=True)
+2 -1
View File
@@ -5,9 +5,10 @@ MODELS = "models"
PROMPTS = "prompts"
SAMPLING = "sampling"
LORAS = "loras"
EMBEDDINGS = "embeddings"
SIZE = "size"
IMAGES = "images"
IS_SAMPLER = "is_sampler" # New constant to mark sampler nodes
# Complete list of categories to track
METADATA_CATEGORIES = [MODELS, PROMPTS, SAMPLING, LORAS, SIZE, IMAGES]
METADATA_CATEGORIES = [MODELS, PROMPTS, SAMPLING, LORAS, EMBEDDINGS, SIZE, IMAGES]
+97 -40
View File
@@ -352,50 +352,101 @@ class MetadataProcessor:
# Check if we have stored conditioning objects for this sampler
if sampler_id in metadata.get(PROMPTS, {}) and (
"pos_conditioning" in metadata[PROMPTS][sampler_id] or
"neg_conditioning" in metadata[PROMPTS][sampler_id]):
"pos_conditioning" in metadata[PROMPTS][sampler_id] or
"neg_conditioning" in metadata[PROMPTS][sampler_id]
):
pos_conditioning = metadata[PROMPTS][sampler_id].get("pos_conditioning")
neg_conditioning = metadata[PROMPTS][sampler_id].get("neg_conditioning")
# Helper function to recursively find prompt text for a conditioning object
def find_prompt_text_for_conditioning(conditioning_obj, is_positive=True):
def extend_unique(target, values):
for value in values:
if value and value not in target:
target.append(value)
# Helper function to recursively find prompt texts for a conditioning object.
# Transform nodes can map one output conditioning to multiple source conditionings.
def find_prompt_texts_for_conditioning(
conditioning_obj, is_positive=True, visited=None
):
if conditioning_obj is None:
return ""
return []
if visited is None:
visited = set()
conditioning_id = id(conditioning_obj)
if conditioning_id in visited:
return []
visited.add(conditioning_id)
prompt_texts = []
# Try to match conditioning objects with those stored by extractors
for prompt_node_id, prompt_data in metadata[PROMPTS].items():
# For nodes with single conditioning output
if "conditioning" in prompt_data:
if id(prompt_data["conditioning"]) == id(conditioning_obj):
return prompt_data.get("text", "")
# For nodes with separate pos_conditioning and neg_conditioning outputs (like TSC_EfficientLoader)
if is_positive and "positive_encoded" in prompt_data:
if id(prompt_data["positive_encoded"]) == id(conditioning_obj):
if "positive_text" in prompt_data:
return prompt_data["positive_text"]
else:
orig_conditioning = prompt_data.get("orig_pos_cond", None)
if orig_conditioning is not None:
# Recursively find the prompt text for the original conditioning
return find_prompt_text_for_conditioning(orig_conditioning, is_positive=True)
if not is_positive and "negative_encoded" in prompt_data:
if id(prompt_data["negative_encoded"]) == id(conditioning_obj):
if "negative_text" in prompt_data:
return prompt_data["negative_text"]
else:
orig_conditioning = prompt_data.get("orig_neg_cond", None)
if orig_conditioning is not None:
# Recursively find the prompt text for the original conditioning
return find_prompt_text_for_conditioning(orig_conditioning, is_positive=False)
return ""
if not isinstance(prompt_data, dict):
continue
# For CLIP text nodes with a single conditioning output.
if id(prompt_data.get("conditioning")) == conditioning_id:
text = prompt_data.get("text", "")
if text:
extend_unique(prompt_texts, [text])
# Generic provenance for passthrough/transform/combine nodes.
for source in prompt_data.get("conditioning_sources", []):
if id(source.get("output")) != conditioning_id:
continue
for input_conditioning in source.get("inputs", []):
extend_unique(
prompt_texts,
find_prompt_texts_for_conditioning(
input_conditioning, is_positive, visited
),
)
# For nodes with separate pos_conditioning and neg_conditioning outputs
# like TSC_EfficientLoader and existing ControlNet-style metadata.
if (
is_positive
and id(prompt_data.get("positive_encoded")) == conditioning_id
):
if prompt_data.get("positive_text"):
extend_unique(prompt_texts, [prompt_data["positive_text"]])
else:
extend_unique(
prompt_texts,
find_prompt_texts_for_conditioning(
prompt_data.get("orig_pos_cond"),
is_positive=True,
visited=visited,
),
)
if (
not is_positive
and id(prompt_data.get("negative_encoded")) == conditioning_id
):
if prompt_data.get("negative_text"):
extend_unique(prompt_texts, [prompt_data["negative_text"]])
else:
extend_unique(
prompt_texts,
find_prompt_texts_for_conditioning(
prompt_data.get("orig_neg_cond"),
is_positive=False,
visited=visited,
),
)
return prompt_texts
# Find prompt texts using the helper function
result["prompt"] = find_prompt_text_for_conditioning(pos_conditioning, is_positive=True)
result["negative_prompt"] = find_prompt_text_for_conditioning(neg_conditioning, is_positive=False)
result["prompt"] = ", ".join(
find_prompt_texts_for_conditioning(pos_conditioning, is_positive=True)
)
result["negative_prompt"] = ", ".join(
find_prompt_texts_for_conditioning(neg_conditioning, is_positive=False)
)
return result
@@ -509,8 +560,14 @@ class MetadataProcessor:
params["loras"] = " ".join(lora_parts)
# Set default clip_skip value
params["clip_skip"] = "1" # Common default
# Extract clip_skip from any SAMPLING node that provides it
for sampler_info in metadata.get(SAMPLING, {}).values():
clip_skip = sampler_info.get("parameters", {}).get("clip_skip")
if clip_skip is not None:
params["clip_skip"] = clip_skip
break
if params["clip_skip"] is None:
params["clip_skip"] = "1"
return params
+419 -1
View File
@@ -144,6 +144,118 @@ class TSCCheckpointLoaderExtractor(NodeMetadataExtractor):
metadata[PROMPTS][node_id]["positive_encoded"] = positive_conditioning
metadata[PROMPTS][node_id]["negative_encoded"] = negative_conditioning
class EasyComfyLoaderExtractor(NodeMetadataExtractor):
@staticmethod
def extract(node_id, inputs, outputs, metadata):
if not inputs:
return
if "ckpt_name" in inputs:
_store_checkpoint_metadata(metadata, node_id, inputs["ckpt_name"])
# Only extract from optional_lora_stack — skip the single lora_name to
# avoid double-counting LoRAs that come through the LORA_STACK path.
active_loras = []
optional_lora_stack = inputs.get("optional_lora_stack")
if optional_lora_stack is not None and isinstance(optional_lora_stack, (list, tuple)):
for item in optional_lora_stack:
if isinstance(item, (list, tuple)) and len(item) >= 2:
lora_path = item[0]
model_strength = item[1]
lora_name = os.path.splitext(os.path.basename(lora_path))[0]
active_loras.append({
"name": lora_name,
"strength": model_strength
})
if active_loras:
metadata[LORAS][node_id] = {
"lora_list": active_loras,
"node_id": node_id
}
positive_text = inputs.get("positive", "")
negative_text = inputs.get("negative", "")
if positive_text or negative_text:
if node_id not in metadata[PROMPTS]:
metadata[PROMPTS][node_id] = {"node_id": node_id}
metadata[PROMPTS][node_id]["positive_text"] = positive_text
metadata[PROMPTS][node_id]["negative_text"] = negative_text
if "clip_skip" in inputs:
clip_skip = inputs["clip_skip"]
if node_id not in metadata[SAMPLING]:
metadata[SAMPLING][node_id] = {"parameters": {}, "node_id": node_id}
metadata[SAMPLING][node_id]["parameters"]["clip_skip"] = clip_skip
width = inputs.get("empty_latent_width")
height = inputs.get("empty_latent_height")
if width is not None and height is not None:
if SIZE not in metadata:
metadata[SIZE] = {}
metadata[SIZE][node_id] = {
"width": int(width),
"height": int(height),
"node_id": node_id
}
@staticmethod
def update(node_id, outputs, metadata):
# outputs: [(pipe_dict, model, vae), ...]
if not outputs or not isinstance(outputs, list) or len(outputs) == 0:
return
first_output = outputs[0]
if not isinstance(first_output, tuple) or len(first_output) < 1:
return
pipe = first_output[0]
if not isinstance(pipe, dict):
return
positive_conditioning = pipe.get("positive")
negative_conditioning = pipe.get("negative")
if positive_conditioning is not None or negative_conditioning is not None:
if node_id not in metadata[PROMPTS]:
metadata[PROMPTS][node_id] = {"node_id": node_id}
if positive_conditioning is not None:
metadata[PROMPTS][node_id]["positive_encoded"] = positive_conditioning
if negative_conditioning is not None:
metadata[PROMPTS][node_id]["negative_encoded"] = negative_conditioning
class EasyPreSamplingExtractor(NodeMetadataExtractor):
@staticmethod
def extract(node_id, inputs, outputs, metadata):
if not inputs:
return
sampling_params = {}
for key in ("steps", "cfg", "sampler_name", "scheduler", "denoise", "seed"):
if key in inputs:
sampling_params[key] = inputs[key]
metadata[SAMPLING][node_id] = {
"parameters": sampling_params,
"node_id": node_id,
IS_SAMPLER: True
}
class EasySeedExtractor(NodeMetadataExtractor):
@staticmethod
def extract(node_id, inputs, outputs, metadata):
if not inputs or "seed" not in inputs:
return
metadata[SAMPLING][node_id] = {
"parameters": {"seed": inputs["seed"]},
"node_id": node_id,
IS_SAMPLER: False
}
class CLIPTextEncodeExtractor(NodeMetadataExtractor):
@staticmethod
def extract(node_id, inputs, outputs, metadata):
@@ -163,6 +275,251 @@ class CLIPTextEncodeExtractor(NodeMetadataExtractor):
conditioning = outputs[0][0]
metadata[PROMPTS][node_id]["conditioning"] = conditioning
class MyOriginalWaifuTextExtractor(NodeMetadataExtractor):
"""Extractor for ComfyUI-MyOriginalWaifu TextProvider nodes."""
@staticmethod
def extract(node_id, inputs, outputs, metadata):
if not inputs:
return
positive_text = inputs.get("positive", "")
negative_text = inputs.get("negative", "")
if positive_text or negative_text:
metadata[PROMPTS][node_id] = {
"positive_text": positive_text,
"negative_text": negative_text,
"node_id": node_id,
}
@staticmethod
def update(node_id, outputs, metadata):
output_tuple = _first_output_tuple(outputs)
if not output_tuple or len(output_tuple) < 2:
return
prompt_metadata = _ensure_prompt_metadata(metadata, node_id)
prompt_metadata["positive_text"] = output_tuple[0]
prompt_metadata["negative_text"] = output_tuple[1]
class MyOriginalWaifuClipExtractor(NodeMetadataExtractor):
"""Extractor for ComfyUI-MyOriginalWaifu ClipProvider nodes."""
@staticmethod
def extract(node_id, inputs, outputs, metadata):
if not inputs:
return
positive_text = inputs.get("positive", "")
negative_text = inputs.get("negative", "")
if positive_text or negative_text:
metadata[PROMPTS][node_id] = {
"positive_text": positive_text,
"negative_text": negative_text,
"node_id": node_id,
}
@staticmethod
def update(node_id, outputs, metadata):
output_tuple = _first_output_tuple(outputs)
if not output_tuple or len(output_tuple) < 2:
return
prompt_metadata = _ensure_prompt_metadata(metadata, node_id)
prompt_metadata["positive_encoded"] = output_tuple[0]
prompt_metadata["negative_encoded"] = output_tuple[1]
def _ensure_prompt_metadata(metadata, node_id):
if node_id not in metadata[PROMPTS]:
metadata[PROMPTS][node_id] = {"node_id": node_id}
return metadata[PROMPTS][node_id]
def _first_output_tuple(outputs):
if not outputs or not isinstance(outputs, list) or len(outputs) == 0:
return None
first_output = outputs[0]
if isinstance(first_output, tuple):
return first_output
return None
def _record_conditioning_source(
metadata, node_id, output_conditioning, input_conditionings
):
if output_conditioning is None:
return
sources = [
conditioning for conditioning in input_conditionings if conditioning is not None
]
if not sources:
return
prompt_metadata = _ensure_prompt_metadata(metadata, node_id)
prompt_metadata.setdefault("conditioning_sources", []).append(
{
"output": output_conditioning,
"inputs": sources,
}
)
def _get_variable_name(inputs):
for key in ("key", "name", "variable_name", "tag", "text"):
value = inputs.get(key)
if isinstance(value, str) and value:
return value
return None
def _get_node_variable_name(metadata, node_id, inputs):
variable_name = _get_variable_name(inputs)
if variable_name:
return variable_name
prompt = metadata.get("current_prompt")
original_prompt = getattr(prompt, "original_prompt", None)
if not original_prompt or node_id not in original_prompt:
return None
node_data = original_prompt[node_id]
variable_name = _get_variable_name(node_data.get("inputs", {}))
if variable_name:
return variable_name
widgets_values = node_data.get("widgets_values", [])
if widgets_values and isinstance(widgets_values[0], str):
return widgets_values[0]
return None
class ControlNetApplyAdvancedExtractor(NodeMetadataExtractor):
@staticmethod
def extract(node_id, inputs, outputs, metadata):
if not inputs:
return
prompt_metadata = _ensure_prompt_metadata(metadata, node_id)
if inputs.get("positive") is not None:
prompt_metadata["orig_pos_cond"] = inputs["positive"]
if inputs.get("negative") is not None:
prompt_metadata["orig_neg_cond"] = inputs["negative"]
@staticmethod
def update(node_id, outputs, metadata):
output_tuple = _first_output_tuple(outputs)
if not output_tuple:
return
prompt_metadata = _ensure_prompt_metadata(metadata, node_id)
positive_input = prompt_metadata.get("orig_pos_cond")
negative_input = prompt_metadata.get("orig_neg_cond")
if len(output_tuple) >= 1:
prompt_metadata["positive_encoded"] = output_tuple[0]
_record_conditioning_source(
metadata, node_id, output_tuple[0], [positive_input]
)
if len(output_tuple) >= 2:
prompt_metadata["negative_encoded"] = output_tuple[1]
_record_conditioning_source(
metadata, node_id, output_tuple[1], [negative_input]
)
class ConditioningCombineExtractor(NodeMetadataExtractor):
@staticmethod
def extract(node_id, inputs, outputs, metadata):
if not inputs:
return
input_conditionings = []
for input_name in inputs:
if (
input_name.startswith("conditioning")
and inputs[input_name] is not None
):
input_conditionings.append(inputs[input_name])
if input_conditionings:
prompt_metadata = _ensure_prompt_metadata(metadata, node_id)
prompt_metadata["orig_conditionings"] = input_conditionings
@staticmethod
def update(node_id, outputs, metadata):
output_tuple = _first_output_tuple(outputs)
if not output_tuple or len(output_tuple) < 1:
return
prompt_metadata = _ensure_prompt_metadata(metadata, node_id)
output_conditioning = output_tuple[0]
prompt_metadata["conditioning"] = output_conditioning
_record_conditioning_source(
metadata,
node_id,
output_conditioning,
prompt_metadata.get("orig_conditionings", []),
)
class SetNodeExtractor(NodeMetadataExtractor):
@staticmethod
def extract(node_id, inputs, outputs, metadata):
if not inputs:
return
variable_name = _get_node_variable_name(metadata, node_id, inputs)
conditioning = inputs.get("CONDITIONING")
if conditioning is None:
conditioning = inputs.get("conditioning")
if conditioning is None:
return
prompt_metadata = _ensure_prompt_metadata(metadata, node_id)
prompt_metadata["conditioning"] = conditioning
if variable_name:
prompt_metadata["variable_name"] = variable_name
metadata[PROMPTS].setdefault("__conditioning_variables__", {})[
variable_name
] = conditioning
class GetNodeExtractor(NodeMetadataExtractor):
@staticmethod
def extract(node_id, inputs, outputs, metadata):
variable_name = _get_node_variable_name(metadata, node_id, inputs or {})
if variable_name:
prompt_metadata = _ensure_prompt_metadata(metadata, node_id)
prompt_metadata["variable_name"] = variable_name
@staticmethod
def update(node_id, outputs, metadata):
output_tuple = _first_output_tuple(outputs)
if not output_tuple or len(output_tuple) < 1:
return
prompt_metadata = _ensure_prompt_metadata(metadata, node_id)
output_conditioning = output_tuple[0]
prompt_metadata["conditioning"] = output_conditioning
variable_name = prompt_metadata.get("variable_name")
if not variable_name:
return
input_conditioning = metadata[PROMPTS].get("__conditioning_variables__", {}).get(
variable_name
)
_record_conditioning_source(
metadata, node_id, output_conditioning, [input_conditioning]
)
# Base Sampler Extractor to reduce code redundancy
class BaseSamplerExtractor(NodeMetadataExtractor):
"""Base extractor for sampler nodes with common functionality"""
@@ -544,6 +901,55 @@ class LoraLoaderManagerExtractor(NodeMetadataExtractor):
"node_id": node_id
}
class LoraTextLoaderManagerExtractor(NodeMetadataExtractor):
"""Extract LoRA metadata from LoraTextLoaderLM (LoRA Text Loader).
The node accepts a `lora_syntax` STRING containing <lora:name:strength> tags
(same format as the ComfyUI prompt), plus an optional `lora_stack`.
This extractor parses the syntax string using the same regex as the node.
"""
@staticmethod
def extract(node_id, inputs, outputs, metadata):
if not inputs:
return
active_loras = []
# Process lora_stack if available (optional input)
if "lora_stack" in inputs:
lora_stack = inputs.get("lora_stack", [])
for item in lora_stack:
# lora_stack entries are (path, model_strength, clip_strength) tuples
if isinstance(item, (list, tuple)) and len(item) >= 2:
lora_path = item[0]
model_strength = item[1]
lora_name = os.path.splitext(os.path.basename(lora_path))[0]
active_loras.append({
"name": lora_name,
"strength": round(float(model_strength), 2)
})
# Process lora_syntax string input
if "lora_syntax" in inputs:
lora_syntax = inputs.get("lora_syntax", "")
if lora_syntax and isinstance(lora_syntax, str):
pattern = r"<lora:([^:>]+):([^:>]+)(?::([^:>]+))?>"
matches = re.findall(pattern, lora_syntax, re.IGNORECASE)
for match in matches:
lora_name = match[0]
model_strength = float(match[1])
active_loras.append({
"name": lora_name,
"strength": round(model_strength, 2)
})
if active_loras:
metadata[LORAS][node_id] = {
"lora_list": active_loras,
"node_id": node_id
}
class FluxGuidanceExtractor(NodeMetadataExtractor):
@staticmethod
def extract(node_id, inputs, outputs, metadata):
@@ -768,9 +1174,12 @@ NODE_EXTRACTORS = {
"KSamplerSelect": KSamplerSelectExtractor, # Add KSamplerSelect
"BasicScheduler": BasicSchedulerExtractor, # Add BasicScheduler
"AlignYourStepsScheduler": BasicSchedulerExtractor, # Add AlignYourStepsScheduler
# ComfyUI-Easy-Use pre-sampling / seed
"samplerSettings": EasyPreSamplingExtractor, # easy preSampling
"easySeed": EasySeedExtractor, # easy seed
# Loaders
"CheckpointLoaderSimple": CheckpointLoaderExtractor,
"comfyLoader": CheckpointLoaderExtractor, # easy comfyLoader
"comfyLoader": EasyComfyLoaderExtractor, # ComfyUI-Easy-Use easy comfyLoader
"CheckpointLoaderSimpleWithImages": CheckpointLoaderExtractor, # CheckpointLoader|pysssss
"TSC_EfficientLoader": TSCCheckpointLoaderExtractor, # Efficient Nodes
"NunchakuFluxDiTLoader": NunchakuFluxDiTLoaderExtractor, # ComfyUI-Nunchaku
@@ -780,10 +1189,13 @@ NODE_EXTRACTORS = {
"GGUFLoaderKJ": KJNodesModelLoaderExtractor, # KJNodes
"DiffusionModelLoaderKJ": KJNodesModelLoaderExtractor, # KJNodes
"CheckpointLoaderKJ": CheckpointLoaderExtractor, # KJNodes
"CheckpointLoaderLM": CheckpointLoaderExtractor, # LoRA Manager
"UNETLoader": UNETLoaderExtractor, # Updated to use dedicated extractor
"UnetLoaderGGUF": UNETLoaderExtractor, # Updated to use dedicated extractor
"UNETLoaderLM": UNETLoaderExtractor, # LoRA Manager
"LoraLoader": LoraLoaderExtractor,
"LoraLoaderLM": LoraLoaderManagerExtractor,
"LoraTextLoaderLM": LoraTextLoaderManagerExtractor,
"RgthreePowerLoraLoader": RgthreePowerLoraLoaderExtractor,
"TensorRTLoader": TensorRTLoaderExtractor,
# Conditioning
@@ -796,6 +1208,12 @@ NODE_EXTRACTORS = {
"smZ_CLIPTextEncode": CLIPTextEncodeExtractor, # From https://github.com/shiimizu/ComfyUI_smZNodes
"CR_ApplyControlNetStack": CR_ApplyControlNetStackExtractor, # Add CR_ApplyControlNetStack
"PCTextEncode": CLIPTextEncodeExtractor, # From https://github.com/asagi4/comfyui-prompt-control
"TextProvider": MyOriginalWaifuTextExtractor, # ComfyUI-MyOriginalWaifu
"ClipProvider": MyOriginalWaifuClipExtractor, # ComfyUI-MyOriginalWaifu
"ControlNetApplyAdvanced": ControlNetApplyAdvancedExtractor,
"ConditioningCombine": ConditioningCombineExtractor,
"SetNode": SetNodeExtractor,
"GetNode": GetNodeExtractor,
# Latent
"EmptyLatentImage": ImageSizeExtractor,
# Flux
+2
View File
@@ -16,6 +16,8 @@ IMG_EXTENSIONS = (
".tif",
".tiff",
".webp",
".avif",
".jxl",
".mp4"
)
+71
View File
@@ -0,0 +1,71 @@
"""JSON error middleware for API routes.
Ensures all responses to /api/* requests return valid JSON that the
browser-extension frontend can JSON.parse() without crashing, even when
the route does not exist (404) or the handler raises an exception (500).
Extension consumers call response.json() unconditionally — an HTML error
page causes ``SyntaxError: unexpected end of data`` that leaks into the
popup UI as a toast notification.
"""
from __future__ import annotations
import logging
from typing import Awaitable, Callable
from aiohttp import web
logger = logging.getLogger(__name__)
@web.middleware
async def api_json_error(
request: web.Request,
handler: Callable[[web.Request], Awaitable[web.Response]],
) -> web.Response:
"""Return JSON ``{"success": false, "error": "..."}`` for API errors.
Only intercepts paths starting with ``/api/`` — all other routes
(frontend pages, static files, WebSocket upgrades) pass through
unchanged.
"""
if not request.path.startswith("/api/"):
return await handler(request)
try:
response = await handler(request)
return response
except web.HTTPException as exc:
# Let redirects (301, 302, 307, 308) propagate — they are not errors.
if exc.status < 400:
raise
logger.warning(
"API %s %s returned HTTP %d: %s",
request.method,
request.path,
exc.status,
exc.reason,
)
return web.json_response(
{"success": False, "error": f"{exc.status}: {exc.reason}"},
status=exc.status,
)
except Exception as exc:
logger.error(
"API %s %s raised unhandled exception: %s",
request.method,
request.path,
exc,
exc_info=True,
)
return web.json_response(
{
"success": False,
"error": f"500: Internal Server Error ({type(exc).__name__})",
},
status=500,
)
+2 -1
View File
@@ -9,6 +9,7 @@ from ..utils.utils import get_lora_info_absolute
from .utils import (
FlexibleOptionalInputType,
any_type,
apply_lora_syntax_format,
detect_nunchaku_model_kind,
extract_lora_name,
get_loras_list,
@@ -52,7 +53,7 @@ def _collect_widget_entries(kwargs):
for lora in get_loras_list(kwargs):
if not lora.get("active", False):
continue
lora_name = lora["name"]
lora_name = apply_lora_syntax_format(lora["name"])
model_strength = float(lora["strength"])
clip_strength = float(lora.get("clipStrength", model_strength))
lora_path, trigger_words = get_lora_info_absolute(lora_name)
+2 -2
View File
@@ -1,6 +1,6 @@
import os
from ..utils.utils import get_lora_info
from .utils import FlexibleOptionalInputType, any_type, extract_lora_name, get_loras_list
from .utils import FlexibleOptionalInputType, any_type, apply_lora_syntax_format, extract_lora_name, get_loras_list
import logging
@@ -48,7 +48,7 @@ class LoraStackerLM:
if not lora.get('active', False):
continue
lora_name = lora['name']
lora_name = apply_lora_syntax_format(lora['name'])
model_strength = float(lora['strength'])
# Get clip strength - use model strength as default if not specified
clip_strength = float(lora.get('clipStrength', model_strength))
+230 -2
View File
@@ -1,12 +1,17 @@
import json
import os
import re
import time
import uuid
from typing import Any, Dict, Optional
import numpy as np
import folder_paths # type: ignore
from ..services.service_registry import ServiceRegistry
from ..metadata_collector.metadata_processor import MetadataProcessor
from ..metadata_collector import get_metadata
from ..utils.constants import CARD_PREVIEW_WIDTH
from ..utils.exif_utils import ExifUtils
from ..utils.utils import calculate_recipe_fingerprint, sanitize_folder_name
from PIL import Image, PngImagePlugin
import piexif
import logging
@@ -86,6 +91,13 @@ class SaveImageLM:
"tooltip": "Adds an incremental counter to filenames to prevent overwriting previous images.",
},
),
"save_as_recipe": (
"BOOLEAN",
{
"default": False,
"tooltip": "Also saves each generated image as a LoRA Manager recipe.",
},
),
},
"hidden": {
"id": "UNIQUE_ID",
@@ -286,7 +298,12 @@ class SaveImageLM:
key = parts[0]
if key == "seed" and "seed" in metadata_dict:
filename = filename.replace(segment, str(metadata_dict.get("seed", "")))
seed_value = metadata_dict.get("seed")
if seed_value is not None:
filename = filename.replace(segment, str(seed_value))
else:
# Fallback if seed was not captured by metadata collector
filename = filename.replace(segment, "0")
elif key == "width" and "size" in metadata_dict:
size = metadata_dict.get("size", "x")
w = size.split("x")[0] if isinstance(size, str) else size[0]
@@ -297,12 +314,14 @@ class SaveImageLM:
filename = filename.replace(segment, str(h))
elif key == "pprompt" and "prompt" in metadata_dict:
prompt = metadata_dict.get("prompt", "").replace("\n", " ")
prompt = sanitize_folder_name(prompt)
if len(parts) >= 2:
length = int(parts[1])
prompt = prompt[:length]
filename = filename.replace(segment, prompt.strip())
elif key == "nprompt" and "negative_prompt" in metadata_dict:
prompt = metadata_dict.get("negative_prompt", "").replace("\n", " ")
prompt = sanitize_folder_name(prompt)
if len(parts) >= 2:
length = int(parts[1])
prompt = prompt[:length]
@@ -316,6 +335,7 @@ class SaveImageLM:
model = "model_unavailable"
else:
model = os.path.splitext(os.path.basename(model_value))[0]
model = sanitize_folder_name(model)
if len(parts) >= 2:
length = int(parts[1])
model = model[:length]
@@ -346,6 +366,203 @@ class SaveImageLM:
return filename
@staticmethod
def _get_cached_model_by_name(scanner, name):
cache = getattr(scanner, "_cache", None)
if cache is None or not name:
return None
candidates = [
name,
os.path.basename(name),
os.path.splitext(os.path.basename(name))[0],
]
for model in getattr(cache, "raw_data", []):
file_name = model.get("file_name")
if file_name in candidates:
return model
return None
def _build_recipe_loras(self, recipe_scanner, lora_stack):
lora_matches = re.findall(r"<lora:([^:]+):([^>]+)>", lora_stack or "")
lora_scanner = getattr(recipe_scanner, "_lora_scanner", None)
loras_data = []
base_model_counts = {}
for name, strength in lora_matches:
lora_info = self._get_cached_model_by_name(lora_scanner, name)
civitai = (lora_info or {}).get("civitai") or {}
civitai_model = civitai.get("model") or {}
try:
parsed_strength = float(strength)
except (TypeError, ValueError):
parsed_strength = 1.0
loras_data.append(
{
"file_name": name,
"strength": parsed_strength,
"hash": ((lora_info or {}).get("sha256") or "").lower(),
"modelVersionId": civitai.get("id", 0),
"modelName": civitai_model.get("name", name) if lora_info else "",
"modelVersionName": civitai.get("name", "") if lora_info else "",
"isDeleted": False,
"exclude": False,
}
)
base_model = (lora_info or {}).get("base_model")
if base_model:
base_model_counts[base_model] = base_model_counts.get(base_model, 0) + 1
return lora_matches, loras_data, base_model_counts
def _build_recipe_checkpoint(self, recipe_scanner, checkpoint_raw):
if not isinstance(checkpoint_raw, str) or not checkpoint_raw.strip():
return None
checkpoint_name = checkpoint_raw.strip()
file_name = os.path.splitext(os.path.basename(checkpoint_name))[0]
checkpoint_scanner = getattr(recipe_scanner, "_checkpoint_scanner", None)
checkpoint_info = self._get_cached_model_by_name(
checkpoint_scanner, checkpoint_name
)
if not checkpoint_info:
return {
"type": "checkpoint",
"name": checkpoint_name,
"file_name": file_name,
"hash": self.get_checkpoint_hash(checkpoint_name) or "",
}
civitai = checkpoint_info.get("civitai") or {}
civitai_model = civitai.get("model") or {}
file_path = checkpoint_info.get("file_path") or checkpoint_info.get("path") or ""
cached_file_name = (
checkpoint_info.get("file_name")
or (os.path.splitext(os.path.basename(file_path))[0] if file_path else "")
or file_name
)
return {
"type": "checkpoint",
"modelId": civitai_model.get("id", 0),
"modelVersionId": civitai.get("id", 0),
"name": civitai_model.get("name")
or checkpoint_info.get("model_name")
or checkpoint_name,
"version": civitai.get("name", ""),
"hash": (
checkpoint_info.get("sha256") or checkpoint_info.get("hash") or ""
).lower(),
"file_name": cached_file_name,
"modelName": civitai_model.get("name", ""),
"modelVersionName": civitai.get("name", ""),
"baseModel": checkpoint_info.get("base_model")
or civitai.get("baseModel", ""),
}
@staticmethod
def _derive_recipe_name(lora_matches):
recipe_name_parts = [
f"{name.strip()}-{float(strength):.2f}" for name, strength in lora_matches[:3]
]
return "_".join(recipe_name_parts) or "recipe"
@staticmethod
def _sync_recipe_cache(recipe_scanner, recipe_data, json_path):
cache = getattr(recipe_scanner, "_cache", None)
if cache is not None:
cache.raw_data.append(recipe_data)
cache.sorted_by_name = sorted(
cache.raw_data, key=lambda item: item.get("title", "").lower()
)
cache.sorted_by_date = sorted(
cache.raw_data,
key=lambda item: (
item.get("modified", item.get("created_date", 0)),
item.get("file_path", ""),
),
reverse=True,
)
recipe_scanner._update_folder_metadata(cache)
recipe_scanner._update_fts_index_for_recipe(recipe_data, "add")
recipe_id = str(recipe_data.get("id", ""))
if recipe_id:
recipe_scanner._json_path_map[recipe_id] = json_path
persistent_cache = getattr(recipe_scanner, "_persistent_cache", None)
if persistent_cache:
persistent_cache.update_recipe(recipe_data, json_path)
def _save_image_as_recipe(self, file_path, metadata_dict):
if not metadata_dict:
raise ValueError("No generation metadata found")
recipe_scanner = ServiceRegistry.get_service_sync("recipe_scanner")
if recipe_scanner is None:
raise RuntimeError("Recipe scanner unavailable")
recipes_dir = recipe_scanner.recipes_dir
if not recipes_dir:
raise RuntimeError("Recipes directory unavailable")
os.makedirs(recipes_dir, exist_ok=True)
recipe_id = str(uuid.uuid4())
optimized_image, extension = ExifUtils.optimize_image(
image_data=file_path,
target_width=CARD_PREVIEW_WIDTH,
format="webp",
quality=85,
preserve_metadata=True,
)
image_path = os.path.normpath(os.path.join(recipes_dir, f"{recipe_id}{extension}"))
with open(image_path, "wb") as file_obj:
file_obj.write(optimized_image)
lora_stack = metadata_dict.get("loras", "")
lora_matches, loras_data, base_model_counts = self._build_recipe_loras(
recipe_scanner, lora_stack
)
checkpoint_entry = self._build_recipe_checkpoint(
recipe_scanner, metadata_dict.get("checkpoint")
)
most_common_base_model = (
max(base_model_counts.items(), key=lambda item: item[1])[0]
if base_model_counts
else ""
)
current_time = time.time()
recipe_data = {
"id": recipe_id,
"file_path": image_path,
"title": self._derive_recipe_name(lora_matches),
"modified": current_time,
"created_date": current_time,
"base_model": most_common_base_model
or (checkpoint_entry or {}).get("baseModel", ""),
"loras": loras_data,
"gen_params": {
key: value
for key, value in metadata_dict.items()
if key not in ["checkpoint", "loras"]
},
"loras_stack": lora_stack,
"fingerprint": calculate_recipe_fingerprint(loras_data),
}
if checkpoint_entry:
recipe_data["checkpoint"] = checkpoint_entry
json_path = os.path.normpath(
os.path.join(recipes_dir, f"{recipe_id}.recipe.json")
)
with open(json_path, "w", encoding="utf-8") as file_obj:
json.dump(recipe_data, file_obj, indent=4, ensure_ascii=False)
ExifUtils.append_recipe_metadata(image_path, recipe_data)
self._sync_recipe_cache(recipe_scanner, recipe_data, json_path)
def save_images(
self,
images,
@@ -359,6 +576,7 @@ class SaveImageLM:
embed_workflow=False,
save_with_metadata=True,
add_counter_to_filename=True,
save_as_recipe=False,
):
"""Save images with metadata"""
results = []
@@ -390,7 +608,7 @@ class SaveImageLM:
img = Image.fromarray(np.clip(img, 0, 255).astype(np.uint8))
# Generate filename with counter if needed
base_filename = filename
base_filename = filename.replace("%batch_num%", str(i))
if add_counter_to_filename:
# Use counter + i to ensure unique filenames for all images in batch
current_counter = counter + i
@@ -477,6 +695,14 @@ class SaveImageLM:
img.save(file_path, format="WEBP", **save_kwargs)
if save_as_recipe:
try:
self._save_image_as_recipe(file_path, metadata_dict)
except Exception as e:
logger.warning(
"Failed to save image as recipe: %s", e, exc_info=True
)
results.append(
{"filename": file, "subfolder": subfolder, "type": self.type}
)
@@ -499,6 +725,7 @@ class SaveImageLM:
embed_workflow=False,
save_with_metadata=True,
add_counter_to_filename=True,
save_as_recipe=False,
):
"""Process and save image with metadata"""
# Make sure the output directory exists
@@ -527,6 +754,7 @@ class SaveImageLM:
embed_workflow,
save_with_metadata,
add_counter_to_filename,
save_as_recipe,
)
return {
+43 -1
View File
@@ -76,6 +76,9 @@ class TriggerWordToggleLM:
# Filter out empty strings and return as set
return set(word for word in words if word)
def _group_has_child_items(self, item):
return isinstance(item, dict) and isinstance(item.get("items"), list)
def process_trigger_words(
self,
id,
@@ -112,7 +115,11 @@ class TriggerWordToggleLM:
if isinstance(trigger_data, list):
if group_mode:
if allow_strength_adjustment:
if any(self._group_has_child_items(item) for item in trigger_data):
filtered_groups = self._process_group_items(
trigger_data, allow_strength_adjustment
)
elif allow_strength_adjustment:
parsed_items = [
self._parse_trigger_item(
item, allow_strength_adjustment
@@ -174,6 +181,41 @@ class TriggerWordToggleLM:
return (filtered_triggers,)
def _process_group_items(self, trigger_data, allow_strength_adjustment):
filtered_groups = []
for item in trigger_data:
group = self._parse_trigger_item(item, allow_strength_adjustment)
if not group["text"] or not group["active"]:
continue
raw_items = item.get("items") if isinstance(item, dict) else None
if isinstance(raw_items, list):
active_items = []
for raw_item in raw_items:
child = self._parse_trigger_item(
raw_item, allow_strength_adjustment=False
)
if child["text"] and child["active"]:
active_items.append(child["text"])
if not active_items:
continue
group_text = ", ".join(active_items)
else:
group_text = group["text"]
filtered_groups.append(
self._format_word_output(
group_text,
group["strength"],
allow_strength_adjustment,
)
)
return filtered_groups
def _parse_trigger_item(self, item, allow_strength_adjustment):
text = (item.get("text") or "").strip()
active = bool(item.get("active", False))
+22 -4
View File
@@ -44,11 +44,29 @@ import folder_paths # type: ignore
logger = logging.getLogger(__name__)
def get_lora_syntax_format():
try:
from ..services.settings_manager import get_settings_manager
return get_settings_manager().get("lora_syntax_format", "legacy")
except Exception:
return "legacy"
def apply_lora_syntax_format(name):
fmt = get_lora_syntax_format()
if fmt == "legacy":
return name.replace("\\", "/").rstrip("/").split("/")[-1]
return name
def extract_lora_name(lora_path):
"""Extract the lora name from a lora path (e.g., 'IL\\aorunIllstrious.safetensors' -> 'aorunIllstrious')"""
# Get the basename without extension
basename = os.path.basename(lora_path)
return os.path.splitext(basename)[0]
normalized = lora_path.replace("\\", "/")
basename = os.path.basename(normalized)
name_no_ext = os.path.splitext(basename)[0]
dirname = os.path.dirname(normalized)
if dirname and dirname not in (".", "/") and not normalized.startswith("/"):
return apply_lora_syntax_format(f"{dirname}/{name_no_ext}")
return apply_lora_syntax_format(name_no_ext)
def get_loras_list(kwargs):
+17 -5
View File
@@ -1,10 +1,22 @@
import folder_paths # type: ignore
from ..utils.utils import get_lora_info
import os
from ..utils.utils import get_lora_info_absolute
from ..config import config
from .utils import FlexibleOptionalInputType, any_type, get_loras_list
import logging
logger = logging.getLogger(__name__)
def _relpath_within_loras(abs_path):
"""Return abs_path relative to the first matching lora root, or basename as fallback."""
all_roots = list(config.loras_roots or []) + list(config.extra_loras_roots or [])
for root in all_roots:
try:
return os.path.relpath(abs_path, root)
except ValueError:
continue
return os.path.basename(abs_path)
class WanVideoLoraSelectLM:
NAME = "WanVideo Lora Select (LoraManager)"
CATEGORY = "Lora Manager/stackers"
@@ -56,13 +68,13 @@ class WanVideoLoraSelectLM:
clip_strength = float(lora.get('clipStrength', model_strength))
# Get lora path and trigger words
lora_path, trigger_words = get_lora_info(lora_name)
lora_path, trigger_words = get_lora_info_absolute(lora_name)
# Create lora item for WanVideo format
lora_item = {
"path": folder_paths.get_full_path("loras", lora_path),
"path": lora_path,
"strength": model_strength,
"name": lora_path.split(".")[0],
"name": os.path.splitext(_relpath_within_loras(lora_path))[0],
"blocks": selected_blocks,
"layer_filter": layer_filter,
"low_mem_load": low_mem_load,
+17 -5
View File
@@ -1,11 +1,23 @@
import folder_paths # type: ignore
from ..utils.utils import get_lora_info
import os
from ..utils.utils import get_lora_info_absolute
from ..config import config
from .utils import any_type
import logging
# 初始化日志记录器
logger = logging.getLogger(__name__)
def _relpath_within_loras(abs_path):
"""Return abs_path relative to the first matching lora root, or basename as fallback."""
all_roots = list(config.loras_roots or []) + list(config.extra_loras_roots or [])
for root in all_roots:
try:
return os.path.relpath(abs_path, root)
except ValueError:
continue
return os.path.basename(abs_path)
# 定义新节点的类
class WanVideoLoraTextSelectLM:
# 节点在UI中显示的名称
@@ -87,12 +99,12 @@ class WanVideoLoraTextSelectLM:
else:
continue
lora_path, trigger_words = get_lora_info(lora_name_raw)
lora_path, trigger_words = get_lora_info_absolute(lora_name_raw)
lora_item = {
"path": folder_paths.get_full_path("loras", lora_path),
"path": lora_path,
"strength": model_strength,
"name": lora_path.split(".")[0],
"name": os.path.splitext(_relpath_within_loras(lora_path))[0],
"blocks": selected_blocks,
"layer_filter": layer_filter,
"low_mem_load": low_mem_load,
+61 -4
View File
@@ -7,7 +7,7 @@ import re
from typing import Dict, List, Any, Optional, Tuple
from abc import ABC, abstractmethod
from ..config import config
from ..utils.constants import VALID_LORA_TYPES
from ..utils.constants import VALID_LORA_TYPES, VALID_CHECKPOINT_SUB_TYPES
from ..utils.civitai_utils import rewrite_preview_url
logger = logging.getLogger(__name__)
@@ -58,9 +58,52 @@ class RecipeMetadataParser(ABC):
civitai_info, error_msg = civitai_info_tuple if isinstance(civitai_info_tuple, tuple) else (civitai_info_tuple, None)
if not civitai_info or error_msg == "Model not found":
# Model not found or deleted
lora_entry['isDeleted'] = True
lora_entry['thumbnailUrl'] = '/loras_static/images/no-preview.png'
# CivitAI may fail to resolve a hash that is still being
# computed (known CivitAI issue). Before marking as deleted,
# try to reconcile with a local model that has the same
# filename and matching AutoV3 hash.
reconciled = False
file_name = lora_entry.get("file_name")
if file_name and recipe_scanner and hash_value:
lora_scanner = getattr(recipe_scanner, "_lora_scanner", None)
if lora_scanner:
try:
# Local import to avoid circular dependency:
# base.py → file_utils → settings_manager → ...
# → recipe_scanner → enrichment → base.py
from ..utils.file_utils import calculate_autov3 # fmt: skip
cache = await lora_scanner.get_cached_data()
for item in getattr(cache, "raw_data", []):
if item.get("file_name") == file_name:
local_path = item.get("file_path")
if local_path and os.path.exists(local_path):
local_autov3 = calculate_autov3(local_path)
if local_autov3 and local_autov3 == hash_value:
lora_entry["existsLocally"] = True
lora_entry["localPath"] = local_path
lora_entry["hash"] = item.get("sha256", hash_value)
if "preview_url" in item:
lora_entry["thumbnailUrl"] = config.get_preview_static_url(item["preview_url"])
civ = item.get("civitai") or {}
if isinstance(civ, dict):
if civ.get("id") is not None:
lora_entry["id"] = civ["id"]
if civ.get("modelId") is not None:
lora_entry["modelId"] = civ["modelId"]
if civ.get("name"):
lora_entry["version"] = civ["name"]
# model_name is the CivitAI model display
# name stored directly in the cache column.
cached_model_name = item.get("model_name")
if cached_model_name:
lora_entry["name"] = cached_model_name
reconciled = True
break
except Exception:
pass
if not reconciled:
lora_entry['isDeleted'] = True
lora_entry['thumbnailUrl'] = '/loras_static/images/no-preview.png'
return lora_entry
# Get model type and validate
@@ -173,6 +216,20 @@ class RecipeMetadataParser(ABC):
checkpoint['isDeleted'] = True
return checkpoint
# Validate that the model type is actually a checkpoint.
# Unlike populate_lora_from_civitai which has this check,
# this function was missing type validation — allowing LoRA
# version data to be saved as the recipe's checkpoint when the
# wrong version ID was passed downstream (fixed in v2.7+).
model_type = civitai_data.get('model', {}).get('type', '').lower()
if model_type not in VALID_CHECKPOINT_SUB_TYPES:
logger.warning(
f"Cannot populate checkpoint: model version {civitai_data.get('id')} "
f"has type '{model_type}', expected one of {VALID_CHECKPOINT_SUB_TYPES}. "
f"Skipping checkpoint enrichment."
)
return checkpoint
if 'model' in civitai_data and 'name' in civitai_data['model']:
checkpoint['name'] = civitai_data['model']['name']
+76 -51
View File
@@ -16,55 +16,65 @@ class RecipeEnricher:
async def enrich_recipe(
recipe: Dict[str, Any],
civitai_client: Any,
request_params: Optional[Dict[str, Any]] = None
request_params: Optional[Dict[str, Any]] = None,
prefetched_civitai_meta_raw: Optional[Dict[str, Any]] = None,
prefetched_model_version_id: Optional[int] = None,
) -> bool:
"""
Enrich a recipe dictionary in-place with metadata from Civitai and embedded params.
Args:
recipe: The recipe dictionary to enrich. Must have 'gen_params' initialized.
civitai_client: Authenticated Civitai client instance.
request_params: (Optional) Parameters from a user request (e.g. import).
prefetched_civitai_meta_raw: (Optional) Pre-fetched raw meta from Civitai
get_image_info, avoiding a duplicate API call.
prefetched_model_version_id: (Optional) Pre-fetched model version ID.
Returns:
bool: True if the recipe was modified, False otherwise.
"""
updated = False
gen_params = recipe.get("gen_params", {})
# 1. Fetch Civitai Info if available
# 1. Obtain Civitai metadata
civitai_meta = None
model_version_id = None
source_url = recipe.get("source_url") or recipe.get("source_path", "")
# Check if it's a Civitai image URL
image_id = extract_civitai_image_id(str(source_url))
if image_id:
try:
image_info = await civitai_client.get_image_info(
image_id, source_url=str(source_url)
)
if image_info:
# Handle nested meta often found in Civitai API responses
raw_meta = image_info.get("meta")
if isinstance(raw_meta, dict):
if "meta" in raw_meta and isinstance(raw_meta["meta"], dict):
civitai_meta = raw_meta["meta"]
else:
civitai_meta = raw_meta
model_version_id = image_info.get("modelVersionId")
# If not at top level, check resources in meta
if not model_version_id and civitai_meta:
resources = civitai_meta.get("civitaiResources", [])
for res in resources:
if res.get("type") == "checkpoint":
model_version_id = res.get("modelVersionId")
break
except Exception as e:
logger.warning(f"Failed to fetch Civitai image info: {e}")
model_version_id = prefetched_model_version_id
source_path = recipe.get("source_path", "")
if prefetched_civitai_meta_raw is not None:
raw_meta = prefetched_civitai_meta_raw
if isinstance(raw_meta, dict):
if "meta" in raw_meta and isinstance(raw_meta["meta"], dict):
civitai_meta = raw_meta["meta"]
else:
civitai_meta = raw_meta
else:
image_id = extract_civitai_image_id(str(source_path))
if image_id:
try:
image_info = await civitai_client.get_image_info(
image_id, source_url=str(source_path)
)
if image_info:
raw_meta = image_info.get("meta")
if isinstance(raw_meta, dict):
if "meta" in raw_meta and isinstance(raw_meta["meta"], dict):
civitai_meta = raw_meta["meta"]
else:
civitai_meta = raw_meta
model_version_id = image_info.get("modelVersionId")
except Exception as e:
logger.warning(f"Failed to fetch Civitai image info: {e}")
if not model_version_id and civitai_meta:
resources = civitai_meta.get("civitaiResources", [])
for res in resources:
if res.get("type") == "checkpoint":
model_version_id = res.get("modelVersionId")
break
# 2. Merge Parameters
# Priority: request_params > civitai_meta > embedded (existing gen_params)
@@ -180,27 +190,42 @@ class RecipeEnricher:
existing_cp = recipe.get("checkpoint")
if existing_cp is None:
existing_cp = {}
# Extract baseModel from raw civitai_info before populate_checkpoint_from_civitai
# (populate may reject non-checkpoint types and lose this data)
base_model_from_civitai: str = ""
if isinstance(civitai_info, dict):
base_model_from_civitai = civitai_info.get("baseModel", "") or ""
elif isinstance(civitai_info, tuple) and len(civitai_info) > 0 and isinstance(civitai_info[0], dict):
base_model_from_civitai = civitai_info[0].get("baseModel", "") or ""
checkpoint_data = await RecipeMetadataParser.populate_checkpoint_from_civitai(existing_cp, civitai_info)
# 1. First, resolve base_model using full data before we format it away
# 1. Resolve base_model from checkpoint_data first, then fall back to raw civitai_info
current_base_model = recipe.get("base_model")
resolved_base_model = checkpoint_data.get("baseModel")
resolved_base_model = checkpoint_data.get("baseModel") or base_model_from_civitai
if resolved_base_model:
# Update if empty OR if it matches our generic prefix but is less specific
is_generic = not current_base_model or current_base_model.lower() in ["flux", "sdxl", "sd15"]
if is_generic and resolved_base_model != current_base_model:
recipe["base_model"] = resolved_base_model
# 2. Format according to requirements: type, modelId, modelVersionId, modelName, modelVersionName
formatted_checkpoint = {
"type": "checkpoint",
"modelId": checkpoint_data.get("modelId"),
"modelVersionId": checkpoint_data.get("id") or checkpoint_data.get("modelVersionId"),
"modelName": checkpoint_data.get("name"), # In base.py, 'name' is populated from civitai_data['model']['name']
"modelVersionName": checkpoint_data.get("version") # In base.py, 'version' is populated from civitai_data['name']
}
# Remove None values
recipe["checkpoint"] = {k: v for k, v in formatted_checkpoint.items() if v is not None}
# 2. Only format and save checkpoint if it has real data (not just type after type rejection)
has_checkpoint_data = any([
checkpoint_data.get("modelId"),
checkpoint_data.get("id") or checkpoint_data.get("modelVersionId"),
checkpoint_data.get("name"),
checkpoint_data.get("version"),
])
if has_checkpoint_data:
formatted_checkpoint = {
"type": "checkpoint",
"modelId": checkpoint_data.get("modelId"),
"modelVersionId": checkpoint_data.get("id") or checkpoint_data.get("modelVersionId"),
"modelName": checkpoint_data.get("name"),
"modelVersionName": checkpoint_data.get("version"),
}
recipe["checkpoint"] = {k: v for k, v in formatted_checkpoint.items() if v is not None}
return True
else:
# Fallback to name extraction if we don't already have one
+32 -15
View File
@@ -123,24 +123,39 @@ class AutomaticMetadataParser(RecipeMetadataParser):
if model_hash_from_hashes:
metadata["model_hash"] = model_hash_from_hashes
# Extract Lora hashes in alternative format
# Extract Lora hashes in alternative format.
# Run unconditionally (not just as fallback) so that
# non-empty hashes from Lora hashes fill in the gaps left
# by empty values in the Hashes JSON dict. Some WebUI
# builds write real hash values only to Lora hashes and
# leave the Hashes JSON values empty.
lora_hashes_match = re.search(self.LORA_HASHES_REGEX, params_section)
if not hashes_match and lora_hashes_match:
if lora_hashes_match:
try:
lora_hashes_str = lora_hashes_match.group(1)
lora_hash_entries = lora_hashes_str.split(', ')
# Initialize hashes dict if it doesn't exist
if "hashes" not in metadata:
metadata["hashes"] = {}
# Parse each lora hash entry (format: "name: hash")
for entry in lora_hash_entries:
if ': ' in entry:
lora_name, lora_hash = entry.split(': ', 1)
# Add as lora type in the same format as regular hashes
metadata["hashes"][f"lora:{lora_name}"] = lora_hash.strip()
lora_hash = lora_hash.strip()
if not lora_hash:
# Skip entries without a hash value
continue
# Initialize hashes dict if it doesn't exist
if "hashes" not in metadata:
metadata["hashes"] = {}
# Add as lora type in the same format as
# regular hashes. Only override an
# existing entry if its value is empty
# (Lora hashes is the more reliable
# source when Hashes JSON has blanks).
key = f"lora:{lora_name}"
existing = metadata["hashes"].get(key, "")
if not existing:
metadata["hashes"][key] = lora_hash
# Remove lora hashes from params section
params_section = params_section.replace(lora_hashes_match.group(0), '')
except Exception as e:
@@ -362,6 +377,12 @@ class AutomaticMetadataParser(RecipeMetadataParser):
# Only process lora or hypernet types
if not hash_key.startswith(("lora:", "hypernet:")):
continue
# Skip entries without a hash value — they can't be
# resolved via CivitAI and would only produce a
# useless "Deleted" entry in the recipe.
if not lora_hash:
continue
lora_type, lora_name = hash_key.split(':', 1)
@@ -387,11 +408,7 @@ class AutomaticMetadataParser(RecipeMetadataParser):
# Try to get info from Civitai
if metadata_provider:
try:
if lora_hash:
# If we have hash, use it for lookup
civitai_info = await metadata_provider.get_model_by_hash(lora_hash)
else:
civitai_info = None
civitai_info = await metadata_provider.get_model_by_hash(lora_hash)
populated_entry = await self.populate_lora_from_civitai(
lora_entry,
+201 -31
View File
@@ -6,6 +6,7 @@ from typing import Dict, Any, Union
from ..base import RecipeMetadataParser
from ..constants import GEN_PARAM_KEYS
from ...services.metadata_service import get_default_metadata_provider
from ...config import config
logger = logging.getLogger(__name__)
@@ -73,7 +74,8 @@ class CivitaiApiMetadataParser(RecipeMetadataParser):
return False
async def parse_metadata( # type: ignore[override]
self, user_comment, recipe_scanner=None, civitai_client=None
self, user_comment, recipe_scanner=None, civitai_client=None,
local_cache: dict[str, Any] | None = None,
) -> Dict[str, Any]:
"""Parse metadata from Civitai image format
@@ -81,6 +83,8 @@ class CivitaiApiMetadataParser(RecipeMetadataParser):
user_comment: The metadata from the image (dict)
recipe_scanner: Optional recipe scanner service
civitai_client: Optional Civitai API client (deprecated, use metadata_provider instead)
local_cache: Optional dict mapping sha256/autov3 hash → scanner cache item.
When provided, matching models skip CivitAI API calls.
Returns:
Dict containing parsed recipe data
@@ -185,8 +189,77 @@ class CivitaiApiMetadataParser(RecipeMetadataParser):
# Process standard resources array
if "resources" in metadata and isinstance(metadata["resources"], list):
for resource in metadata["resources"]:
resource_type = resource.get("type", "lora")
# Track resources with type "model" — these are checkpoint models.
# The resources array is the most reliable source for checkpoint
# identification because it has an explicit type field and hash,
# unlike modelVersionIds which is a flat list with no type info.
if resource_type == "model":
checkpoint_entry = {
"id": 0,
"modelId": 0,
"name": resource.get("name", "Unknown Model"),
"version": "",
"type": resource.get("type", "model"),
"existsLocally": False,
"localPath": None,
"file_name": resource.get("name", ""),
"hash": resource.get("hash", "") or "",
"thumbnailUrl": "/loras_static/images/no-preview.png",
"baseModel": "",
"size": 0,
"downloadUrl": "",
"isDeleted": False,
}
# Try to look up base model from the checkpoint hash
cp_hash = checkpoint_entry.get("hash")
if cp_hash and metadata_provider:
local_cached = local_cache.get(cp_hash) if local_cache else None
if local_cached:
self._populate_entry_from_cache(
checkpoint_entry, local_cached
)
bm = checkpoint_entry.get("baseModel", "")
if bm and not result["base_model"]:
result["base_model"] = bm
else:
try:
civitai_info = (
await metadata_provider.get_model_by_hash(
cp_hash
)
)
civitai_data, error_msg = (
(civitai_info, None)
if not isinstance(civitai_info, tuple)
else civitai_info
)
if civitai_data and error_msg != "Model not found":
if 'model' in civitai_data and 'name' in civitai_data['model']:
checkpoint_entry['name'] = civitai_data['model']['name']
checkpoint_entry['id'] = civitai_data.get('id', 0)
checkpoint_entry['modelId'] = civitai_data.get('modelId', 0)
if 'name' in civitai_data:
checkpoint_entry['version'] = civitai_data['name']
base_model = civitai_data.get('baseModel', '')
if base_model:
checkpoint_entry['baseModel'] = base_model
if not result['base_model']:
result['base_model'] = base_model
except Exception as e:
logger.error(
f"Error fetching checkpoint info for hash "
f"{cp_hash}: {e}"
)
if result["model"] is None:
result["model"] = checkpoint_entry
continue
# Modified to process resources without a type field as potential LoRAs
if resource.get("type", "lora") == "lora":
if resource_type == "lora":
lora_hash = resource.get("hash", "")
# Try to get hash from the hashes field if not present in resource
@@ -220,34 +293,45 @@ class CivitaiApiMetadataParser(RecipeMetadataParser):
}
# Try to get info from Civitai if hash is available
if lora_entry["hash"] and metadata_provider:
try:
civitai_info = (
await metadata_provider.get_model_by_hash(lora_hash)
if lora_hash and metadata_provider:
local_cached = local_cache.get(lora_hash) if local_cache else None
if local_cached:
self._populate_entry_from_cache(
lora_entry, local_cached
)
populated_entry = await self.populate_lora_from_civitai(
lora_entry,
civitai_info,
recipe_scanner,
base_model_counts,
lora_hash,
)
if populated_entry is None:
continue # Skip invalid LoRA types
lora_entry = populated_entry
# If we have a version ID from Civitai, track it for deduplication
if "id" in lora_entry and lora_entry["id"]:
# Track by version ID for deduplication
if lora_entry.get("id"):
added_loras[str(lora_entry["id"])] = len(
result["loras"]
)
except Exception as e:
logger.error(
f"Error fetching Civitai info for LoRA hash {lora_entry['hash']}: {e}"
)
else:
try:
civitai_info = (
await metadata_provider.get_model_by_hash(lora_hash)
)
populated_entry = await self.populate_lora_from_civitai(
lora_entry,
civitai_info,
recipe_scanner,
base_model_counts,
lora_hash,
)
if populated_entry is None:
continue # Skip invalid LoRA types
lora_entry = populated_entry
# If we have a version ID from Civitai, track it for deduplication
if "id" in lora_entry and lora_entry["id"]:
added_loras[str(lora_entry["id"])] = len(
result["loras"]
)
except Exception as e:
logger.error(
f"Error fetching Civitai info for LoRA hash {lora_entry['hash']}: {e}"
)
# Track by hash if we have it
if lora_hash:
@@ -430,11 +514,21 @@ class CivitaiApiMetadataParser(RecipeMetadataParser):
result["loras"].append(lora_entry)
# Process modelVersionIds from Civitai image API
# These are model version IDs returned at root level when meta doesn't contain resources
if "modelVersionIds" in metadata and isinstance(
metadata["modelVersionIds"], list
# Process modelVersionIds from Civitai image API.
# These are version IDs returned at root level of the API response.
# When resources or civitaiResources are already present in metadata
# (which they are when ?withMeta=true is passed), those sections have
# complete hash/type information — modelVersionIds is a fallback for
# when meta is null and only the flat ID list is available. Skipping
# it here avoids duplicates: the same file hash often resolves to
# different version IDs via hash lookup (resources) vs the original
# version ID in modelVersionIds, and both paths would create entries.
if (
"modelVersionIds" in metadata
and isinstance(metadata["modelVersionIds"], list)
and not result.get("loras")
):
for version_id in metadata["modelVersionIds"]:
version_id_str = str(version_id)
@@ -442,6 +536,13 @@ class CivitaiApiMetadataParser(RecipeMetadataParser):
if version_id_str in added_loras:
continue
# Skip if this version ID is already the recipe's checkpoint
# (resolved earlier from embedded resources/Model hash,
# avoiding a duplicate CivitAI API call).
existing_model = result.get("model")
if existing_model and str(existing_model.get("id")) == version_id_str:
continue
# Initialize lora entry with version ID
lora_entry = {
"id": version_id,
@@ -475,9 +576,40 @@ class CivitaiApiMetadataParser(RecipeMetadataParser):
)
if populated_entry is None:
continue # Skip invalid LoRA types
# Not a LoRA — try as checkpoint (only if we
# don't already have one). Reuses the same
# civitai_info from the API call above so no
# extra query is made.
if result["model"] is None:
checkpoint_entry = {
"id": version_id,
"modelId": 0,
"name": "Unknown Model",
"version": "",
"type": "checkpoint",
"existsLocally": False,
"localPath": None,
"file_name": "",
"hash": "",
"thumbnailUrl": (
"/loras_static/images/no-preview.png"
),
"baseModel": "",
"size": 0,
"downloadUrl": "",
"isDeleted": False,
}
cp_populated = await (
self.populate_checkpoint_from_civitai(
checkpoint_entry, civitai_info
)
)
if cp_populated.get("modelId"):
result["model"] = cp_populated
continue # Not a LoRA, don't add to loras
lora_entry = populated_entry
except Exception as e:
logger.error(
f"Error fetching Civitai info for model version {version_id}: {e}"
@@ -625,3 +757,41 @@ class CivitaiApiMetadataParser(RecipeMetadataParser):
except Exception as e:
logger.error(f"Error parsing Civitai image metadata: {e}", exc_info=True)
return {"error": str(e), "loras": []}
@staticmethod
def _populate_entry_from_cache(
entry: dict[str, Any],
cache_item: dict[str, Any],
) -> None:
"""Fill a lora/checkpoint entry from a scanner cache item.
Avoids CivitAI API calls for models that exist locally.
Mirrors the population logic in
``RecipeMetadataParser.populate_lora_from_civitai()`` but operates
entirely on cached data.
"""
civ = cache_item.get("civitai") or {}
if isinstance(civ, dict):
if civ.get("id") is not None:
entry["id"] = civ["id"]
if civ.get("modelId") is not None:
entry["modelId"] = civ["modelId"]
if civ.get("name"):
entry["version"] = civ["name"]
cached_name = cache_item.get("model_name")
if cached_name:
entry["name"] = cached_name
entry["existsLocally"] = True
local_path = cache_item.get("file_path")
if local_path:
entry["localPath"] = local_path
sha256 = cache_item.get("sha256")
if sha256:
entry["hash"] = sha256
if "preview_url" in cache_item:
entry["thumbnailUrl"] = config.get_preview_static_url(
cache_item["preview_url"]
)
base_model = cache_item.get("base_model", "")
if base_model:
entry["baseModel"] = base_model
+1 -1
View File
@@ -251,7 +251,7 @@ class BaseModelRoutes(ABC):
def _find_model_file(self, files):
"""Find the appropriate model file from the files list - can be overridden by subclasses."""
return next((file for file in files if file.get("type") == "Model" and file.get("primary") is True), None)
return next((file for file in files if file.get("type") in ("Model", "Diffusion Model") and file.get("primary") is True), None)
def get_handler(self, name: str) -> Callable[[web.Request], web.StreamResponse]:
"""Expose handlers for subclasses or tests."""
+409
View File
@@ -0,0 +1,409 @@
"""Handlers for Hugging Face model listing and download.
Minimal MVP implementation — uses direct HTTP to the HF API for file
listing and the project's existing aiohttp-based Downloader for
downloading. No huggingface_hub dependency required.
"""
from __future__ import annotations
import json
import logging
import os
import re
from typing import Any
import aiohttp
from aiohttp import web
from ...config import config
from ...services.downloader import (
DownloadProgress,
get_downloader,
)
from ...services.aria2_downloader import Aria2Downloader
from ...services.settings_manager import get_settings_manager
from ...services.service_registry import ServiceRegistry
from ...services.websocket_manager import ws_manager
from ...utils.constants import MODEL_FILE_EXTENSIONS
from ...utils.metadata_manager import MetadataManager
from ...utils.models import LoraMetadata, CheckpointMetadata, EmbeddingMetadata
logger = logging.getLogger(__name__)
_DEFAULT_MODEL_CLASS = LoraMetadata
_DEFAULT_SCANNER_GETTER = "get_lora_scanner"
# Shared aiohttp session for HF API calls (created on first use)
_hf_api_session: aiohttp.ClientSession | None = None
async def _get_hf_api_session() -> aiohttp.ClientSession:
"""Get or create the shared aiohttp session for HF API calls."""
global _hf_api_session # needed because we reassign the module-level name
if _hf_api_session is None or _hf_api_session.closed:
_hf_api_session = aiohttp.ClientSession(
headers={"User-Agent": "ComfyUI-LoRA-Manager/1.0"},
timeout=aiohttp.ClientTimeout(total=30),
)
return _hf_api_session
def _infer_model_type(model_root: str) -> tuple[Any, str]:
"""Determine model class and scanner by matching ``model_root`` against the
configured root paths for each model type (from ``Config``).
The ``model_root`` value comes from the frontend's model-root dropdown,
which is populated from the current page's scanner roots. By checking
which scanner's root list it belongs to, we avoid fragile heuristics
like substring-matching path names.
"""
norm = os.path.normpath(model_root).replace(os.sep, "/")
# LoRA roots
for p in (config.loras_roots or []) + (config.extra_loras_roots or []):
if os.path.normpath(p).replace(os.sep, "/") == norm:
return LoraMetadata, "get_lora_scanner"
# Checkpoint / UNet roots
for p in (
(config.checkpoints_roots or [])
+ (config.extra_checkpoints_roots or [])
+ (config.unet_roots or [])
+ (config.extra_unet_roots or [])
):
if os.path.normpath(p).replace(os.sep, "/") == norm:
return CheckpointMetadata, "get_checkpoint_scanner"
# Embedding roots
for p in (config.embeddings_roots or []) + (config.extra_embeddings_roots or []):
if os.path.normpath(p).replace(os.sep, "/") == norm:
return EmbeddingMetadata, "get_embedding_scanner"
# Fallback — should not happen in normal use
logger.warning(
"Could not determine model type for root '%s'; defaulting to LoRA",
model_root,
)
return _DEFAULT_MODEL_CLASS, _DEFAULT_SCANNER_GETTER
async def _save_hf_metadata(dest_path: str, repo: str, model_root: str) -> None:
"""Create a proper .metadata.json and add the model to the scanner cache.
Uses ``MetadataManager.create_default_metadata()`` which computes the
SHA256 hash, extracts safetensors header metadata (base_model), and
produces a fully-populated ``LoraMetadata`` (or ``CheckpointMetadata`` /
``EmbeddingMetadata``) object. We then overlay HF-specific fields and
register the model in the in-memory scanner cache so it appears
immediately without a full filesystem walk.
"""
try:
hf_url = f"https://huggingface.co/{repo}"
model_class, scanner_getter_name = _infer_model_type(model_root)
# 1. Create proper metadata (computes SHA256, reads safetensors headers)
metadata = await MetadataManager.create_default_metadata(
dest_path, model_class=model_class
)
if metadata is None:
logger.warning("create_default_metadata returned None for %s", dest_path)
return
# 2. Overlay HF-specific fields
metadata._unknown_fields["hf_url"] = hf_url
metadata.from_civitai = False # HF models are not from CivitAI
# 3. Save metadata atomically
await MetadataManager.save_metadata(dest_path, metadata)
logger.info("Saved HF metadata (with hf_url) for %s", dest_path)
# 4. Determine relative folder path for cache
# model_root is an absolute path; dest_path is under it
folder = ""
if os.path.isabs(model_root) and dest_path.startswith(model_root):
rel = os.path.relpath(os.path.dirname(dest_path), model_root)
folder = rel.replace(os.sep, "/") if rel != "." else ""
# 5. Add to scanner cache (same as CivitAI's _execute_download does)
scanner_getter = getattr(ServiceRegistry, scanner_getter_name, None)
if scanner_getter is not None:
scanner = await scanner_getter()
if scanner is not None:
metadata_dict = metadata.to_dict()
metadata_dict["hf_url"] = hf_url
await scanner.add_model_to_cache(metadata_dict, folder)
logger.info("Added %s to scanner cache (folder=%s)", dest_path, folder)
except Exception as exc:
logger.warning("Failed to save HF metadata for %s: %s", dest_path, exc)
class HfHandler:
"""Handle Hugging Face model browsing and download."""
async def get_hf_repo_files(self, request: web.Request) -> web.Response:
"""List model-weight files from a HF repo with real file sizes.
Uses the HF tree API endpoint which returns accurate file sizes
(including LFS-tracked files), unlike the model info endpoint.
"""
repo = request.query.get("repo", "").strip()
if not repo or "/" not in repo:
return web.json_response(
{"error": "Missing or invalid 'repo' parameter (expected user/repo)"},
status=400,
)
url = f"https://huggingface.co/api/models/{repo}/tree/main"
try:
session = await _get_hf_api_session()
async with session.get(url) as resp:
if resp.status == 404:
return web.json_response(
{"error": f"Repo '{repo}' not found"}, status=404
)
if resp.status != 200:
text = await resp.text()
return web.json_response(
{"error": f"HF API error {resp.status}: {text[:200]}"},
status=resp.status,
)
tree: list[dict[str, Any]] = await resp.json()
except Exception as exc:
logger.error("Failed to fetch HF repo files: %s", exc)
return web.json_response({"error": str(exc)}, status=502)
files: list[dict[str, Any]] = []
for entry in tree:
path: str = entry.get("path", "")
ext = os.path.splitext(path)[1].lower()
if ext not in MODEL_FILE_EXTENSIONS:
continue
size = entry.get("size", 0) or 0
if size == 0 and "lfs" in entry:
size = entry["lfs"].get("size", 0) or 0
files.append({
"filename": path,
"size": size,
})
files.sort(key=lambda f: f["size"], reverse=True)
return web.json_response(files)
async def download_hf_model(self, request: web.Request) -> web.Response:
"""Download a single file from Hugging Face into the model directory.
POST JSON body::
{
"repo": "dx8152/Flux2-Klein-9B-Consistency",
"filename": "Flux2-Klein-9B-consistency-V2.safetensors",
"revision": "main",
"model_root": "loras",
"relative_path": "",
"use_default_paths": false,
"download_id": "optional-batch-id"
}
If ``download_id`` is provided, real-time progress (bytes, speed,
percentage) is broadcast via the WebSocket progress system, matching
the CivitAI download experience.
Respects the ``download_backend`` setting (``aria2`` or ``default``).
"""
try:
payload: dict[str, Any] = await request.json()
except json.JSONDecodeError:
return web.json_response({"error": "Invalid JSON"}, status=400)
repo = (payload.get("repo") or "").strip()
filename = (payload.get("filename") or "").strip()
revision = (payload.get("revision") or "main").strip()
model_root = (payload.get("model_root") or "").strip()
relative_path = (payload.get("relative_path") or "").strip()
use_default_paths = bool(payload.get("use_default_paths", False))
download_id: str | None = payload.get("download_id")
logger.info(
"download_hf_model: repo=%s file=%s root=%s download_id=%s",
repo, filename, model_root, download_id,
)
if not repo or not filename:
return web.json_response(
{"error": "Missing required fields: 'repo' and 'filename'"}, status=400
)
# Validate repo format — must be user/repo_name
if repo.count("/") != 1 or not re.match(r"^[a-zA-Z0-9_.-]+/[a-zA-Z0-9_.-]+$", repo):
return web.json_response({"error": f"Invalid repo format: {repo}"}, status=400)
author, repo_name = repo.split("/", 1)
if ".." in (author, repo_name) or "." in (author, repo_name):
return web.json_response({"error": f"Invalid repo format: {repo}"}, status=400)
# Validate filename — must not contain path separators or ..
if "/" in filename or "\\" in filename or ".." in filename:
return web.json_response({"error": "Invalid filename"}, status=400)
# Validate relative_path — must not be absolute or escape base directory
if relative_path:
if os.path.isabs(relative_path):
return web.json_response({"error": "relative_path must not be absolute"}, status=400)
if ".." in relative_path.split("/") or "\\" in relative_path:
return web.json_response({"error": "Invalid relative_path"}, status=400)
# Validate model_root — must not contain path traversal
if not os.path.isabs(model_root):
# For relative model_root, check it doesn't escape
resolved_model_root = os.path.realpath(
os.path.join(os.getcwd(), "models", model_root)
)
else:
resolved_model_root = os.path.realpath(model_root)
# Verify model_root is within a configured scanner root
allowed_roots = set()
for root_list in (
config.loras_roots or [],
config.extra_loras_roots or [],
config.checkpoints_roots or [],
config.extra_checkpoints_roots or [],
config.unet_roots or [],
config.extra_unet_roots or [],
config.embeddings_roots or [],
config.extra_embeddings_roots or [],
):
for r in root_list:
allowed_roots.add(os.path.realpath(r))
if not any(resolved_model_root == root or resolved_model_root.startswith(root + os.sep) for root in allowed_roots):
logger.warning("Invalid model_root rejected: %s", model_root)
return web.json_response({"error": f"Invalid model_root: {model_root}"}, status=400)
base_dir = resolved_model_root
if use_default_paths:
target_dir = os.path.join(base_dir, "huggingface", author, repo_name)
elif relative_path:
target_dir = os.path.join(base_dir, relative_path)
else:
target_dir = base_dir
os.makedirs(target_dir, exist_ok=True)
dest_path = os.path.join(target_dir, filename)
# Resolve symlinks and check for path traversal escape
real_dest = os.path.realpath(dest_path)
real_base = os.path.realpath(target_dir)
if not real_dest.startswith(real_base + os.sep):
logger.warning("Path traversal blocked: %s -> %s", dest_path, real_dest)
return web.json_response({"error": "Path traversal detected"}, status=400)
# Check if already exists (simple skip)
if os.path.exists(dest_path) and os.path.getsize(dest_path) > 0:
logger.info("download_hf_model: file already exists, skipping — %s", dest_path)
return web.json_response({
"success": True,
"message": f"File already exists: {dest_path}",
"path": dest_path,
})
# Build HF resolve URL
resolve_url = (
f"https://huggingface.co/{repo}/resolve/{revision}/{filename}"
)
# Set up progress callback if download_id is provided
progress_callback = None
if download_id:
async def _progress_callback(
progress: float | DownloadProgress,
snapshot: DownloadProgress | None = None,
) -> None:
percent = 0.0
metrics = snapshot if isinstance(snapshot, DownloadProgress) else None
if isinstance(progress, DownloadProgress):
percent = progress.percent_complete
metrics = progress
elif isinstance(snapshot, DownloadProgress):
percent = snapshot.percent_complete
else:
percent = float(progress)
broadcast: dict[str, Any] = {
"status": "progress",
"progress": round(percent),
}
if metrics:
broadcast["bytes_downloaded"] = metrics.bytes_downloaded
broadcast["total_bytes"] = metrics.total_bytes
broadcast["bytes_per_second"] = metrics.bytes_per_second
await ws_manager.broadcast_download_progress(download_id, broadcast)
progress_callback = _progress_callback
# Respect download backend setting (aria2 vs default)
download_backend = (
get_settings_manager().get("download_backend", "default")
)
if download_backend == "aria2":
aria2 = await Aria2Downloader.get_instance()
aid = download_id or f"hf_{repo}_{filename}"
try:
hf_success, hf_result = await aria2.download_file(
url=resolve_url,
save_path=dest_path,
download_id=aid,
progress_callback=progress_callback,
)
if hf_success:
await _save_hf_metadata(dest_path, repo, model_root)
return web.json_response({
"success": True,
"message": f"Downloaded to {dest_path}",
"path": dest_path,
})
else:
return web.json_response(
{"success": False, "error": hf_result or "aria2 download failed"},
status=500,
)
except Exception as exc:
logger.error("HF download (aria2) failed: %s", exc)
return web.json_response(
{"success": False, "error": str(exc)}, status=500
)
# Default: use built-in aiohttp Downloader
downloader = await get_downloader()
try:
success, result = await downloader.download_file(
url=resolve_url,
save_path=dest_path,
use_auth=False,
allow_resume=True,
progress_callback=progress_callback,
)
if success:
await _save_hf_metadata(dest_path, repo, model_root)
return web.json_response({
"success": True,
"message": f"Downloaded to {result}",
"path": result,
})
else:
return web.json_response(
{"success": False, "error": result or "Download failed"},
status=500,
)
except Exception as exc:
logger.error("HF download failed: %s", exc)
return web.json_response(
{"success": False, "error": str(exc)}, status=500
)
File diff suppressed because it is too large Load Diff
+450 -28
View File
@@ -16,6 +16,10 @@ import jinja2
from ...config import config
from ...services.download_coordinator import DownloadCoordinator
from ...services.connectivity_guard import (
OFFLINE_FRIENDLY_MESSAGE,
is_expected_offline_error,
)
from ...services.metadata_sync_service import MetadataSyncService
from ...services.model_file_service import ModelMoveService
from ...services.preview_asset_service import PreviewAssetService
@@ -33,6 +37,7 @@ from ...services.use_cases import (
)
from ...services.websocket_manager import WebSocketManager
from ...services.websocket_progress_callback import WebSocketProgressCallback
from ...services.download_queue_service import DownloadQueueService
from ...services.errors import RateLimitError, ResourceNotFoundError
from ...utils.civitai_utils import resolve_license_payload
from ...utils.file_utils import calculate_sha256
@@ -198,11 +203,17 @@ class ModelListingHandler:
result = await self._service.get_paginated_data(**params)
format_start = time.perf_counter()
formatted_raw = [
await self._service.format_response(entry)
for entry in result["items"]
]
# Filter out None entries returned for corrupted cache rows (issue #730).
# Note: "total" intentionally remains the pre-filter count to reflect
# the true number of models in the cache; corrupted entries are rare
# and adjusting total would cause pagination drift on every page.
formatted_items = [item for item in formatted_raw if item is not None]
formatted_result = {
"items": [
await self._service.format_response(item)
for item in result["items"]
],
"items": formatted_items,
"total": result["total"],
"page": result["page"],
"page_size": result["page_size"],
@@ -228,14 +239,20 @@ class ModelListingHandler:
start_time = time.perf_counter()
try:
params = self._parse_common_params(request)
# group_by_model is meaningless for excluded view; strip it
params.pop("group_by_model", None)
result = await self._service.get_excluded_paginated_data(**params)
format_start = time.perf_counter()
formatted_raw = [
await self._service.format_response(entry)
for entry in result["items"]
]
# Filter out None entries returned for corrupted cache rows (issue #730).
# "total" stays at the pre-filter count; see get_models for rationale.
formatted_items = [item for item in formatted_raw if item is not None]
formatted_result = {
"items": [
await self._service.format_response(item)
for item in result["items"]
],
"items": formatted_items,
"total": result["total"],
"page": result["page"],
"page_size": result["page_size"],
@@ -297,6 +314,15 @@ class ModelListingHandler:
for tag in exclude_tags:
if tag:
tag_filters[tag] = "exclude"
auto_tag_filters: Dict[str, str] = {}
for tag in request.query.getall("auto_tag_include", []):
if tag:
auto_tag_filters[tag] = "include"
for tag in request.query.getall("auto_tag_exclude", []):
if tag:
auto_tag_filters[tag] = "exclude"
favorites_only = request.query.get("favorites_only", "false").lower() == "true"
search_options = {
@@ -352,6 +378,19 @@ class ModelListingHandler:
request.query.get("name_pattern_use_regex", "false").lower() == "true"
)
# Group-by-model flag: deduplicate versions sharing the same civitai modelId
group_by_model = (
request.query.get("group_by_model", "false").lower() == "true"
)
# View-local-versions filter: show all local versions of a specific model
civitai_model_id = request.query.get("civitai_model_id")
if civitai_model_id is not None:
try:
civitai_model_id = int(civitai_model_id)
except (TypeError, ValueError):
civitai_model_id = None
return {
"page": page,
"page_size": page_size,
@@ -363,6 +402,7 @@ class ModelListingHandler:
"fuzzy_search": fuzzy_search,
"base_models": base_models,
"tags": tag_filters,
"auto_tags": auto_tag_filters,
"tag_logic": tag_logic,
"search_options": search_options,
"hash_filters": hash_filters,
@@ -374,6 +414,8 @@ class ModelListingHandler:
"name_pattern_include": name_pattern_include,
"name_pattern_exclude": name_pattern_exclude,
"name_pattern_use_regex": name_pattern_use_regex,
"group_by_model": group_by_model,
"civitai_model_id": civitai_model_id,
**self._parse_specific_params(request),
}
@@ -501,9 +543,19 @@ class ModelManagementHandler:
if not success:
return web.json_response({"success": False, "error": error})
formatted_metadata = await self._service.format_response(model_data)
return web.json_response({"success": True, "metadata": formatted_metadata})
formatted = await self._service.format_response(model_data)
if formatted is None:
return web.json_response(
{"success": False, "error": "Model entry is corrupted (missing file_path)"},
status=500,
)
return web.json_response({"success": True, "metadata": formatted})
except Exception as exc:
if is_expected_offline_error(str(exc)):
return web.json_response(
{"success": False, "error": OFFLINE_FRIENDLY_MESSAGE},
status=503,
)
self._logger.error("Error fetching from CivitAI: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
@@ -550,6 +602,11 @@ class ModelManagementHandler:
}
)
except Exception as exc:
if is_expected_offline_error(str(exc)):
return web.json_response(
{"success": False, "error": OFFLINE_FRIENDLY_MESSAGE},
status=503,
)
self._logger.error("Error re-linking to CivitAI: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
@@ -764,7 +821,7 @@ class ModelManagementHandler:
metadata_updates = {k: v for k, v in data.items() if k != "file_path"}
await self._metadata_sync.save_metadata_updates(
updated_metadata = await self._metadata_sync.save_metadata_updates(
file_path=file_path,
updates=metadata_updates,
metadata_loader=self._metadata_sync.load_local_metadata,
@@ -775,7 +832,12 @@ class ModelManagementHandler:
cache = await self._service.scanner.get_cached_data()
await cache.resort()
return web.json_response({"success": True})
from ...services.auto_tag_service import extract_auto_tags
auto_tags = extract_auto_tags(updated_metadata)
return web.json_response(
{"success": True, "auto_tags": auto_tags}
)
except Exception as exc:
self._logger.error("Error saving metadata: %s", exc, exc_info=True)
return web.Response(text=str(exc), status=500)
@@ -792,14 +854,16 @@ class ModelManagementHandler:
if not isinstance(new_tags, list):
return web.Response(text="Tags must be a list", status=400)
tags = await self._tag_update_service.add_tags(
tags, auto_tags = await self._tag_update_service.add_tags(
file_path=file_path,
new_tags=new_tags,
metadata_loader=self._metadata_sync.load_local_metadata,
update_cache=self._service.scanner.update_single_model_cache,
)
return web.json_response({"success": True, "tags": tags})
return web.json_response(
{"success": True, "tags": tags, "auto_tags": auto_tags}
)
except Exception as exc:
self._logger.error("Error adding tags: %s", exc, exc_info=True)
return web.Response(text=str(exc), status=500)
@@ -910,7 +974,7 @@ class ModelQueryHandler:
async def get_base_models(self, request: web.Request) -> web.Response:
try:
limit = int(request.query.get("limit", "20"))
if limit < 1 or limit > 100:
if limit < 0 or limit > 100:
limit = 20
base_models = await self._service.get_base_models(limit)
return web.json_response({"success": True, "base_models": base_models})
@@ -1042,10 +1106,12 @@ class ModelQueryHandler:
# Sort: originals first, copies last
sorted_models = self._sort_duplicate_group(filtered)
# Format response
# Format response, filtering out corrupted entries (issue #730)
group = {"hash": sha256, "models": []}
for model in sorted_models:
group["models"].append(await self._service.format_response(model))
formatted = await self._service.format_response(model)
if formatted is not None:
group["models"].append(formatted)
# Only include groups with 2+ models after filtering
if len(group["models"]) > 1:
@@ -1146,6 +1212,12 @@ class ModelQueryHandler:
async def find_filename_conflicts(self, request: web.Request) -> web.Response:
try:
settings = get_settings_manager()
if settings.get("lora_syntax_format", "legacy") == "full":
return web.json_response(
{"success": True, "conflicts": [], "count": 0}
)
duplicates = self._service.find_duplicate_filenames()
result = []
cache = await self._service.scanner.get_cached_data()
@@ -1156,9 +1228,9 @@ class ModelQueryHandler:
(m for m in cache.raw_data if m["file_path"] == path), None
)
if model:
group["models"].append(
await self._service.format_response(model)
)
formatted = await self._service.format_response(model)
if formatted is not None:
group["models"].append(formatted)
hash_val = self._service.scanner.get_hash_by_filename(filename)
if hash_val:
main_path = self._service.get_path_by_hash(hash_val)
@@ -1168,9 +1240,9 @@ class ModelQueryHandler:
None,
)
if main_model:
group["models"].insert(
0, await self._service.format_response(main_model)
)
formatted = await self._service.format_response(main_model)
if formatted is not None:
group["models"].insert(0, formatted)
if group["models"]:
result.append(group)
return web.json_response(
@@ -1234,6 +1306,14 @@ class ModelQueryHandler:
license_flags = (model_data or {}).get("license_flags")
if license_flags is not None:
response_payload["license_flags"] = int(license_flags)
# Include the user's license icon style preference so the
# ComfyUI tooltip can pick the right set without a separate
# API call.
try:
settings = get_settings_manager()
response_payload["use_new_license_icons"] = settings.get("use_new_license_icons", True)
except Exception:
pass
return web.json_response(response_payload)
return web.json_response(
{
@@ -1435,6 +1515,21 @@ class ModelDownloadHandler:
)
return web.Response(status=500, text=str(exc))
async def skip_download_get(self, request: web.Request) -> web.Response:
try:
download_id = request.query.get("download_id")
if not download_id:
return web.json_response(
{"success": False, "error": "Download ID is required"}, status=400
)
result = await self._download_coordinator.skip_download(download_id)
return web.json_response(result)
except Exception as exc:
self._logger.error(
"Error skipping download via GET: %s", exc, exc_info=True
)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def cancel_download_get(self, request: web.Request) -> web.Response:
try:
download_id = request.query.get("download_id")
@@ -1515,6 +1610,291 @@ class ModelDownloadHandler:
)
return web.json_response({"success": False, "error": str(exc)}, status=500)
# ------------------------------------------------------------------
# Download queue / history handlers
# ------------------------------------------------------------------
async def get_download_queue(self, request: web.Request) -> web.Response:
try:
service = await DownloadQueueService.get_instance()
queue = await service.get_queue()
stats = await service.get_stats()
return web.json_response({"success": True, "queue": queue, "stats": stats})
except Exception as exc:
self._logger.error(
"Error getting download queue: %s", exc, exc_info=True
)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def add_to_download_queue(self, request: web.Request) -> web.Response:
try:
import uuid
download_id = request.query.get("download_id") or str(uuid.uuid4())
model_id_str = request.query.get("model_id")
model_version_id_str = request.query.get("model_version_id")
model_name = request.query.get("model_name", "")
version_name = request.query.get("version_name", "")
thumbnail_url = request.query.get("thumbnail_url", "")
source = request.query.get("source")
file_params_json = request.query.get("file_params")
model_id = int(model_id_str) if model_id_str else None
model_version_id = int(model_version_id_str) if model_version_id_str else None
file_params = json.loads(file_params_json) if file_params_json else None
service = await DownloadQueueService.get_instance()
item = await service.add_to_queue(
download_id=download_id,
model_id=model_id,
model_version_id=model_version_id,
model_name=model_name,
version_name=version_name,
thumbnail_url=thumbnail_url,
source=source,
file_params=file_params,
)
return web.json_response({"success": True, "item": item})
except Exception as exc:
self._logger.error(
"Error adding to download queue: %s", exc, exc_info=True
)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def remove_from_download_queue(self, request: web.Request) -> web.Response:
try:
download_id = request.query.get("download_id")
if not download_id:
return web.json_response(
{"success": False, "error": "download_id is required"}, status=400
)
service = await DownloadQueueService.get_instance()
removed = await service.remove_from_queue(download_id)
return web.json_response({"success": removed})
except Exception as exc:
self._logger.error(
"Error removing from download queue: %s", exc, exc_info=True
)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def move_queue_item_to_top(self, request: web.Request) -> web.Response:
try:
download_id = request.query.get("download_id")
if not download_id:
return web.json_response(
{"success": False, "error": "download_id is required"}, status=400
)
service = await DownloadQueueService.get_instance()
moved = await service.move_to_top(download_id)
return web.json_response({"success": moved})
except Exception as exc:
self._logger.error(
"Error moving queue item to top: %s", exc, exc_info=True
)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def move_queue_item_to_end(self, request: web.Request) -> web.Response:
try:
download_id = request.query.get("download_id")
if not download_id:
return web.json_response(
{"success": False, "error": "download_id is required"}, status=400
)
service = await DownloadQueueService.get_instance()
moved = await service.move_to_end(download_id)
return web.json_response({"success": moved})
except Exception as exc:
self._logger.error(
"Error moving queue item to end: %s", exc, exc_info=True
)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def clear_download_queue(self, request: web.Request) -> web.Response:
try:
status_filter = request.query.get("status") or None
service = await DownloadQueueService.get_instance()
cleared = await service.clear_queue(status_filter=status_filter)
return web.json_response({"success": True, "cleared": cleared})
except Exception as exc:
self._logger.error(
"Error clearing download queue: %s", exc, exc_info=True
)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def get_download_history(self, request: web.Request) -> web.Response:
try:
limit = min(int(request.query.get("limit", "50")), 500)
offset = int(request.query.get("offset", "0"))
status_filter = request.query.get("status") or None
service = await DownloadQueueService.get_instance()
result = await service.get_history(
limit=limit, offset=offset, status_filter=status_filter
)
return web.json_response(
{
"success": True,
"items": result["items"],
"total": result["total"],
"limit": result["limit"],
"offset": result["offset"],
}
)
except Exception as exc:
self._logger.error(
"Error getting download history: %s", exc, exc_info=True
)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def clear_download_history(self, request: web.Request) -> web.Response:
try:
status_filter = request.query.get("status") or None
service = await DownloadQueueService.get_instance()
cleared = await service.clear_history(status_filter=status_filter)
return web.json_response({"success": True, "cleared": cleared})
except Exception as exc:
self._logger.error(
"Error clearing download history: %s", exc, exc_info=True
)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def delete_download_history_item(self, request: web.Request) -> web.Response:
try:
item_id = int(request.query.get("id", "0"))
if not item_id:
return web.json_response(
{"success": False, "error": "id is required"}, status=400
)
service = await DownloadQueueService.get_instance()
deleted = await service.delete_history_item(item_id)
return web.json_response({"success": deleted})
except Exception as exc:
self._logger.error(
"Error deleting download history item: %s", exc, exc_info=True
)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def retry_download_from_history(self, request: web.Request) -> web.Response:
try:
item_id = int(request.query.get("id", "0"))
if not item_id:
return web.json_response(
{"success": False, "error": "id is required"}, status=400
)
service = await DownloadQueueService.get_instance()
item = await service.retry_from_history(item_id)
if item is None:
return web.json_response(
{"success": False, "error": "History item not found or not retryable"},
status=404,
)
return web.json_response({"success": True, "item": item})
except Exception as exc:
self._logger.error(
"Error retrying download from history: %s", exc, exc_info=True
)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def retry_all_failed_downloads(self, request: web.Request) -> web.Response:
try:
service = await DownloadQueueService.get_instance()
retry_count = await service.retry_all_failed()
return web.json_response({"success": True, "retry_count": retry_count})
except Exception as exc:
self._logger.error(
"Error retrying all failed downloads: %s", exc, exc_info=True
)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def complete_download_in_queue(self, request: web.Request) -> web.Response:
"""Atomically move a download from queue to history with terminal status."""
try:
download_id = request.query.get("download_id")
if not download_id:
return web.json_response(
{"success": False, "error": "download_id is required"}, status=400
)
status = request.query.get("status", "completed")
error = request.query.get("error")
file_path = request.query.get("file_path")
try:
bytes_downloaded = int(request.query.get("bytes_downloaded", "0"))
except (TypeError, ValueError):
bytes_downloaded = 0
total_bytes_raw = request.query.get("total_bytes")
total_bytes = int(total_bytes_raw) if total_bytes_raw else None
completed_at_raw = request.query.get("completed_at")
completed_at = float(completed_at_raw) if completed_at_raw else None
service = await DownloadQueueService.get_instance()
item = await service.complete_download(
download_id=download_id,
status=status,
error=error,
file_path=file_path,
bytes_downloaded=bytes_downloaded,
total_bytes=total_bytes,
completed_at=completed_at,
)
if item is None:
return web.json_response(
{"success": False, "error": "Download not found in queue"}, status=404
)
return web.json_response({"success": True, "item": item})
except Exception as exc:
self._logger.error(
"Error completing download: %s", exc, exc_info=True
)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def get_download_stats(self, request: web.Request) -> web.Response:
try:
service = await DownloadQueueService.get_instance()
stats = await service.get_stats()
return web.json_response({"success": True, "stats": stats})
except Exception as exc:
self._logger.error(
"Error getting download stats: %s", exc, exc_info=True
)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def update_download_queue_status(self, request: web.Request) -> web.Response:
"""Update the status of a queue item (non-terminal transitions).
Supported transitions include ``queued → downloading``,
``downloading → paused``, ``paused → downloading``, etc.
Terminal transitions (``completed``, ``failed``, ``canceled``)
should use ``complete_download_in_queue`` instead.
"""
try:
download_id = request.query.get("download_id")
status = request.query.get("status")
if not download_id or not status:
return web.json_response(
{
"success": False,
"error": "download_id and status are required",
},
status=400,
)
service = await DownloadQueueService.get_instance()
updated = await service.update_status(download_id, status)
if not updated:
return web.json_response(
{"success": False, "error": "Download not found in queue"},
status=404,
)
return web.json_response({"success": True})
except Exception as exc:
self._logger.error(
"Error updating download queue status: %s", exc, exc_info=True
)
return web.json_response({"success": False, "error": str(exc)}, status=500)
class ModelCivitaiHandler:
"""CivitAI integration endpoints."""
@@ -1556,7 +1936,9 @@ class ModelCivitaiHandler:
return web.json_response(result)
except Exception as exc:
self._logger.error(
"Error in fetch_all_civitai for %ss: %s", self._service.model_type, exc
"Error in fetch_all_civitai for %ss: %s",
self._service.model_type, exc,
exc_info=True,
)
return web.Response(text=str(exc), status=500)
@@ -1858,6 +2240,11 @@ class ModelUpdateHandler:
status=429,
)
except Exception as exc: # pragma: no cover - defensive log
if is_expected_offline_error(str(exc)):
return web.json_response(
{"success": False, "error": OFFLINE_FRIENDLY_MESSAGE},
status=503,
)
self._logger.error("Failed to fetch license info: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
@@ -1918,6 +2305,10 @@ class ModelUpdateHandler:
if target_model_ids:
target_model_ids = sorted(set(target_model_ids))
folder_path: Optional[str] = payload.get("folder_path")
if folder_path is not None and not isinstance(folder_path, str):
folder_path = None
provider = await self._get_civitai_provider()
if provider is None:
return web.json_response(
@@ -1932,6 +2323,7 @@ class ModelUpdateHandler:
provider,
force_refresh=force_refresh,
target_model_ids=target_model_ids or None,
folder_path=folder_path,
)
if self._service.scanner.is_cancelled():
return web.json_response(
@@ -1946,15 +2338,29 @@ class ModelUpdateHandler:
{"success": False, "error": str(exc) or "Rate limited"}, status=429
)
except Exception as exc: # pragma: no cover - defensive logging
self._logger.error(
"Failed to refresh model updates: %s", exc, exc_info=True
)
if is_expected_offline_error(str(exc)):
return web.json_response(
{"success": False, "error": OFFLINE_FRIENDLY_MESSAGE},
status=503,
)
self._logger.error("Failed to refresh model updates: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
hide_early_access = False
if self._settings is not None:
try:
hide_early_access = bool(
self._settings.get("hide_early_access_updates", False)
)
except Exception:
pass
serialized_records = []
for record in records.values():
has_update_fn = getattr(record, "has_update", None)
if callable(has_update_fn) and has_update_fn():
if callable(has_update_fn) and has_update_fn(
hide_early_access=hide_early_access
):
serialized_records.append(self._serialize_record(record))
return web.json_response(
@@ -2401,6 +2807,7 @@ class ModelUpdateHandler:
"shouldIgnore": version.should_ignore,
"earlyAccessEndsAt": version.early_access_ends_at,
"isEarlyAccess": is_early_access,
"usageControl": version.usage_control,
"filePath": context.get("file_path"),
"fileName": context.get("file_name"),
}
@@ -2515,9 +2922,24 @@ class ModelHandlerSet:
"download_model": self.download.download_model,
"download_model_get": self.download.download_model_get,
"cancel_download_get": self.download.cancel_download_get,
"skip_download_get": self.download.skip_download_get,
"pause_download_get": self.download.pause_download_get,
"resume_download_get": self.download.resume_download_get,
"get_download_progress": self.download.get_download_progress,
"get_download_queue": self.download.get_download_queue,
"add_to_download_queue": self.download.add_to_download_queue,
"remove_from_download_queue": self.download.remove_from_download_queue,
"move_queue_item_to_top": self.download.move_queue_item_to_top,
"move_queue_item_to_end": self.download.move_queue_item_to_end,
"clear_download_queue": self.download.clear_download_queue,
"get_download_history": self.download.get_download_history,
"clear_download_history": self.download.clear_download_history,
"delete_download_history_item": self.download.delete_download_history_item,
"retry_download_from_history": self.download.retry_download_from_history,
"retry_all_failed_downloads": self.download.retry_all_failed_downloads,
"complete_download_in_queue": self.download.complete_download_in_queue,
"get_download_stats": self.download.get_download_stats,
"update_download_queue_status": self.download.update_download_queue_status,
"get_civitai_versions": self.civitai.get_civitai_versions,
"get_civitai_model_by_version": self.civitai.get_civitai_model_by_version,
"get_civitai_model_by_hash": self.civitai.get_civitai_model_by_hash,
+59 -2
View File
@@ -3,6 +3,7 @@
from __future__ import annotations
import logging
import mimetypes
import urllib.parse
from pathlib import Path
@@ -12,6 +13,12 @@ from ...config import config as global_config
logger = logging.getLogger(__name__)
_CHUNK_SIZE = 1024 * 1024 # 1 MB — balance between streaming iteration overhead and per-chunk memory
# Video file extensions that bypass native sendfile on Windows
# to avoid IOCP/ProactorEventLoop crashes during client disconnect.
_VIDEO_EXTENSIONS = frozenset({".mp4", ".webm", ".mov", ".avi", ".mkv"})
class PreviewHandler:
"""Serve preview assets for the active library at request time."""
@@ -48,8 +55,58 @@ class PreviewHandler:
logger.debug("Preview file not found at %s", str(resolved))
raise web.HTTPNotFound(text="Preview file not found")
# aiohttp's FileResponse handles range requests and content headers for us.
return web.FileResponse(path=resolved, chunk_size=256 * 1024)
# aiohttp's FileResponse handles range requests, content headers, and
# uses kernel sendfile (zero-copy DMA) on Linux/macOS. On Windows it
# uses IOCP-based _sendfile_native which can crash when the client
# disconnects mid-transfer during fast scrolling. The _stream_file()
# fallback is kept for a future compat toggle.
#
# Set explicit Cache-Control so the browser can cache video (and image)
# previews across VirtualScroller recycling cycles. Without this,
# Chrome does not cache 206 Partial Content responses for <video>
# elements, causing the same video to be re-downloaded on every scroll.
resp = web.FileResponse(path=resolved, chunk_size=_CHUNK_SIZE)
resp.headers["Cache-Control"] = "public, max-age=86400"
return resp
async def _stream_file(
self, request: web.Request, path: Path
) -> web.StreamResponse:
"""Stream a file chunk-by-chunk, bypassing native sendfile.
This avoids the Windows IOCP ``_sendfile_native`` crash that occurs
when the client disconnects during a large file transfer.
"""
content_type, _ = mimetypes.guess_type(str(path))
if content_type is None:
content_type = "application/octet-stream"
file_size = path.stat().st_size
resp = web.StreamResponse()
resp.content_type = content_type
resp.content_length = file_size
# Allow browser caching: video previews rarely change during a session.
# The frontend already appends ?t={version} to bust cache on update.
resp.headers["Cache-Control"] = "public, max-age=86400"
await resp.prepare(request)
try:
with open(path, "rb") as f:
while True:
chunk = f.read(_CHUNK_SIZE)
if not chunk:
break
await resp.write(chunk)
except (ConnectionResetError, ConnectionAbortedError):
# Client disconnected during streaming — expected when scrolling
# rapidly through a library with animated previews.
pass
except OSError as exc:
logger.debug("I/O error streaming preview %s: %s", path, exc)
return resp
__all__ = ["PreviewHandler"]
File diff suppressed because it is too large Load Diff
+12
View File
@@ -24,6 +24,7 @@ MISC_ROUTE_DEFINITIONS: tuple[RouteDefinition, ...] = (
RouteDefinition("POST", "/api/lm/settings", "update_settings"),
RouteDefinition("GET", "/api/lm/doctor/diagnostics", "get_doctor_diagnostics"),
RouteDefinition("POST", "/api/lm/doctor/repair-cache", "repair_doctor_cache"),
RouteDefinition("POST", "/api/lm/doctor/resolve-filename-conflicts", "resolve_doctor_filename_conflicts"),
RouteDefinition("POST", "/api/lm/doctor/export-bundle", "export_doctor_bundle"),
RouteDefinition("GET", "/api/lm/priority-tags", "get_priority_tags"),
RouteDefinition("GET", "/api/lm/settings/libraries", "get_settings_libraries"),
@@ -42,6 +43,7 @@ MISC_ROUTE_DEFINITIONS: tuple[RouteDefinition, ...] = (
RouteDefinition("POST", "/api/lm/update-node-widget", "update_node_widget"),
RouteDefinition("GET", "/api/lm/get-registry", "get_registry"),
RouteDefinition("GET", "/api/lm/check-model-exists", "check_model_exists"),
RouteDefinition("GET", "/api/lm/check-models-exist", "check_models_exist"),
RouteDefinition(
"GET",
"/api/lm/model-version-download-status",
@@ -89,6 +91,16 @@ MISC_ROUTE_DEFINITIONS: tuple[RouteDefinition, ...] = (
RouteDefinition(
"GET", "/api/lm/base-models/cache-status", "get_base_model_cache_status"
),
RouteDefinition(
"GET", "/api/lm/delete-model-version", "delete_model_version"
),
# Hugging Face model endpoints
RouteDefinition(
"GET", "/api/lm/hf-repo-files", "get_hf_repo_files"
),
RouteDefinition(
"POST", "/api/lm/download-hf-model", "download_hf_model"
),
)
+3
View File
@@ -39,6 +39,7 @@ from .handlers.misc_handlers import (
build_service_registry_adapter,
)
from .handlers.base_model_handlers import BaseModelHandlerSet
from .handlers.hf_handlers import HfHandler
from .misc_route_registrar import MiscRouteRegistrar
logger = logging.getLogger(__name__)
@@ -136,6 +137,7 @@ class MiscRoutes:
doctor = DoctorHandler(settings_service=self._settings)
example_workflows = ExampleWorkflowsHandler()
base_model = BaseModelHandlerSet()
hf_handler = HfHandler()
return self._handler_set_factory(
health=health,
@@ -155,6 +157,7 @@ class MiscRoutes:
doctor=doctor,
example_workflows=example_workflows,
base_model=base_model,
hf_handler=hf_handler,
)
+35
View File
@@ -101,11 +101,46 @@ COMMON_ROUTE_DEFINITIONS: tuple[RouteDefinition, ...] = (
RouteDefinition("POST", "/api/lm/download-model", "download_model"),
RouteDefinition("GET", "/api/lm/download-model-get", "download_model_get"),
RouteDefinition("GET", "/api/lm/cancel-download-get", "cancel_download_get"),
RouteDefinition("GET", "/api/lm/skip-download", "skip_download_get"),
RouteDefinition("GET", "/api/lm/pause-download", "pause_download_get"),
RouteDefinition("GET", "/api/lm/resume-download", "resume_download_get"),
RouteDefinition(
"GET", "/api/lm/download-progress/{download_id}", "get_download_progress"
),
RouteDefinition("GET", "/api/lm/downloads/queue", "get_download_queue"),
RouteDefinition("GET", "/api/lm/downloads/queue/add", "add_to_download_queue"),
RouteDefinition(
"GET", "/api/lm/downloads/queue/remove", "remove_from_download_queue"
),
RouteDefinition(
"GET", "/api/lm/downloads/queue/move-to-top", "move_queue_item_to_top"
),
RouteDefinition(
"GET", "/api/lm/downloads/queue/move-to-end", "move_queue_item_to_end"
),
RouteDefinition(
"GET", "/api/lm/downloads/queue/clear", "clear_download_queue"
),
RouteDefinition("GET", "/api/lm/downloads/history", "get_download_history"),
RouteDefinition(
"GET", "/api/lm/downloads/history/clear", "clear_download_history"
),
RouteDefinition(
"GET", "/api/lm/downloads/history/delete", "delete_download_history_item"
),
RouteDefinition(
"GET", "/api/lm/downloads/history/retry", "retry_download_from_history"
),
RouteDefinition(
"GET", "/api/lm/downloads/history/retry-all", "retry_all_failed_downloads"
),
RouteDefinition("GET", "/api/lm/downloads/stats", "get_download_stats"),
RouteDefinition(
"GET", "/api/lm/downloads/queue/complete", "complete_download_in_queue"
),
RouteDefinition(
"GET", "/api/lm/downloads/queue/status", "update_download_queue_status"
),
RouteDefinition("POST", "/api/lm/{prefix}/cancel-task", "cancel_task"),
RouteDefinition("GET", "/{prefix}", "handle_models_page"),
)
+11
View File
@@ -58,6 +58,7 @@ ROUTE_DEFINITIONS: tuple[RouteDefinition, ...] = (
RouteDefinition("POST", "/api/lm/recipes/repair", "repair_recipes"),
RouteDefinition("POST", "/api/lm/recipes/cancel-repair", "cancel_repair"),
RouteDefinition("POST", "/api/lm/recipe/{recipe_id}/repair", "repair_recipe"),
RouteDefinition("POST", "/api/lm/recipes/repair-bulk", "repair_recipes_bulk"),
RouteDefinition("GET", "/api/lm/recipes/repair-progress", "get_repair_progress"),
RouteDefinition("POST", "/api/lm/recipes/batch-import/start", "start_batch_import"),
RouteDefinition(
@@ -70,6 +71,16 @@ ROUTE_DEFINITIONS: tuple[RouteDefinition, ...] = (
"POST", "/api/lm/recipes/batch-import/directory", "start_directory_import"
),
RouteDefinition("POST", "/api/lm/recipes/browse-directory", "browse_directory"),
RouteDefinition(
"GET", "/api/lm/recipes/check-image-exists", "check_image_exists"
),
RouteDefinition("GET", "/api/lm/recipes/import-from-url", "import_from_url"),
RouteDefinition(
"POST", "/api/lm/recipes/create-from-example", "create_from_example"
),
RouteDefinition(
"POST", "/api/lm/recipe/{recipe_id}/reimport", "reimport_recipe"
),
)
+45 -16
View File
@@ -11,6 +11,8 @@ from ..config import config
from ..services.settings_manager import get_settings_manager
from ..services.server_i18n import server_i18n
from ..services.service_registry import ServiceRegistry
from ..services.model_query import normalize_sub_type, resolve_sub_type
from ..utils.constants import VALID_LORA_SUB_TYPES, VALID_CHECKPOINT_SUB_TYPES
from ..utils.usage_stats import UsageStats
logger = logging.getLogger(__name__)
@@ -140,6 +142,21 @@ class StatsRoutes:
# Get usage statistics
usage_data = await self.usage_stats.get_stats()
# CivitAI model type distribution across all model types
# Use the same logic as the filter panel: normalize_sub_type(resolve_sub_type(entry))
# with sub-type validation per model type
model_types_counter: Counter[str] = Counter()
for entry in lora_cache.raw_data:
ntype = normalize_sub_type(resolve_sub_type(entry))
if ntype and ntype in VALID_LORA_SUB_TYPES:
model_types_counter[ntype] += 1
for entry in checkpoint_cache.raw_data:
ntype = normalize_sub_type(resolve_sub_type(entry))
if ntype and ntype in VALID_CHECKPOINT_SUB_TYPES:
model_types_counter[ntype] += 1
# Embeddings: always count as "embedding" regardless of CivitAI sub-type
model_types_counter['embedding'] = len(embedding_cache.raw_data)
return web.json_response({
'success': True,
'data': {
@@ -154,7 +171,8 @@ class StatsRoutes:
'total_generations': usage_data.get('total_executions', 0),
'unused_loras': self._count_unused_models(lora_cache.raw_data, usage_data.get('loras', {})),
'unused_checkpoints': self._count_unused_models(checkpoint_cache.raw_data, usage_data.get('checkpoints', {})),
'unused_embeddings': self._count_unused_models(embedding_cache.raw_data, usage_data.get('embeddings', {}))
'unused_embeddings': self._count_unused_models(embedding_cache.raw_data, usage_data.get('embeddings', {})),
'model_types_distribution': dict(model_types_counter.most_common())
}
})
@@ -459,9 +477,12 @@ class StatsRoutes:
if unused_lora_percent > 50:
insights.append({
'type': 'warning',
'title': 'High Number of Unused LoRAs',
'description': f'{unused_lora_percent:.1f}% of your LoRAs ({unused_loras}/{total_loras}) have never been used.',
'suggestion': 'Consider organizing or archiving unused models to free up storage space.'
'key': 'insights.unusedLoras.high',
'params': {
'percent': f'{unused_lora_percent:.1f}',
'count': str(unused_loras),
'total': str(total_loras)
}
})
if total_checkpoints > 0:
@@ -469,9 +490,12 @@ class StatsRoutes:
if unused_checkpoint_percent > 30:
insights.append({
'type': 'warning',
'title': 'Unused Checkpoints Detected',
'description': f'{unused_checkpoint_percent:.1f}% of your checkpoints ({unused_checkpoints}/{total_checkpoints}) have never been used.',
'suggestion': 'Review and consider removing checkpoints you no longer need.'
'key': 'insights.unusedCheckpoints.detected',
'params': {
'percent': f'{unused_checkpoint_percent:.1f}',
'count': str(unused_checkpoints),
'total': str(total_checkpoints)
}
})
if total_embeddings > 0:
@@ -479,9 +503,12 @@ class StatsRoutes:
if unused_embedding_percent > 50:
insights.append({
'type': 'warning',
'title': 'High Number of Unused Embeddings',
'description': f'{unused_embedding_percent:.1f}% of your embeddings ({unused_embeddings}/{total_embeddings}) have never been used.',
'suggestion': 'Consider organizing or archiving unused embeddings to optimize your collection.'
'key': 'insights.unusedEmbeddings.high',
'params': {
'percent': f'{unused_embedding_percent:.1f}',
'count': str(unused_embeddings),
'total': str(total_embeddings)
}
})
# Storage insights
@@ -492,18 +519,20 @@ class StatsRoutes:
if total_size > 100 * 1024 * 1024 * 1024: # 100GB
insights.append({
'type': 'info',
'title': 'Large Collection Detected',
'description': f'Your model collection is using {self._format_size(total_size)} of storage.',
'suggestion': 'Consider using external storage or cloud solutions for better organization.'
'key': 'insights.collection.large',
'params': {
'size': self._format_size(total_size)
}
})
# Recent activity insight
if usage_data.get('total_executions', 0) > 100:
insights.append({
'type': 'success',
'title': 'Active User',
'description': f'You\'ve completed {usage_data["total_executions"]} generations so far!',
'suggestion': 'Keep exploring and creating amazing content with your models.'
'key': 'insights.activity.active',
'params': {
'count': str(usage_data['total_executions'])
}
})
return web.json_response({
+63 -14
View File
@@ -1,7 +1,6 @@
import os
import logging
import toml
import git
import zipfile
import shutil
import tempfile
@@ -11,11 +10,33 @@ from typing import Dict, List
from ..utils.settings_paths import ensure_settings_file
from ..services.downloader import get_downloader
from ..services.service_registry import ServiceRegistry
logger = logging.getLogger(__name__)
NETWORK_EXCEPTIONS = (ClientError, OSError, asyncio.TimeoutError)
# User-managed directories that live inside the plugin folder (portable
# mode) and must survive a Git-based update. ``git clean -fd`` would
# otherwise delete them because they are untracked and, in released tags,
# not listed in ``.gitignore``. ``-e`` excludes a path from cleaning
# regardless of whether it is ignored.
_PRESERVE_DIRS = ('settings.json', 'civitai', 'wildcards', 'backups', 'stats', 'logs', 'cache', 'model_cache')
def _clean_excludes() -> List[str]:
"""Build the ``-e`` arguments for ``git clean`` from :data:`_PRESERVE_DIRS`."""
excludes: List[str] = []
for name in _PRESERVE_DIRS:
excludes.append('-e')
excludes.append(name)
# For directories, also exclude nested matches explicitly
# (``-e dir`` alone matches the dir entry; ``-e dir/**`` guards
# contents under all git versions as defense-in-depth).
excludes.append('-e')
excludes.append(f'{name}/**')
return excludes
class UpdateRoutes:
"""Routes for handling plugin update checks"""
@@ -212,8 +233,19 @@ class UpdateRoutes:
zip_path = tmp_zip_path
# Skip both settings.json, civitai and model cache folder
UpdateRoutes._clean_plugin_folder(plugin_root, skip_files=['settings.json', 'civitai', 'model_cache'])
# Close the downloaded-versions SQLite connection before cleaning,
# so that shutil.rmtree() does not fail on Windows (the process
# cannot delete a file with an outstanding open handle).
try:
history_svc = ServiceRegistry._services.get("downloaded_version_history_service")
if history_svc is not None:
history_svc.close()
logger.info("Closed downloaded-version history database connection")
except Exception:
logger.debug("Could not close downloaded-version history database", exc_info=True)
# Skip settings.json, civitai, model cache and runtime cache folders
UpdateRoutes._clean_plugin_folder(plugin_root, skip_files=['settings.json', 'civitai', 'model_cache', 'cache', 'wildcards', 'backups', 'stats'])
# Extract ZIP to temp dir
with tempfile.TemporaryDirectory() as tmp_dir:
@@ -222,16 +254,17 @@ class UpdateRoutes:
# Find extracted folder (GitHub ZIP contains a root folder)
extracted_root = next(os.scandir(tmp_dir)).path
# Copy files, skipping settings.json and civitai folder
# Copy files, skipping user data that should be preserved
skip_items = {'settings.json', 'civitai', 'wildcards', 'backups', 'stats'}
for item in os.listdir(extracted_root):
if item == 'settings.json' or item == 'civitai':
if item in skip_items:
continue
src = os.path.join(extracted_root, item)
dst = os.path.join(plugin_root, item)
if os.path.isdir(src):
if os.path.exists(dst):
shutil.rmtree(dst)
shutil.copytree(src, dst, ignore=shutil.ignore_patterns('settings.json', 'civitai'))
shutil.copytree(src, dst, ignore=shutil.ignore_patterns(*skip_items))
else:
shutil.copy2(src, dst)
@@ -239,15 +272,17 @@ class UpdateRoutes:
# for ComfyUI Manager to work properly
tracking_info_file = os.path.join(plugin_root, '.tracking')
tracking_files = []
skip_tracked = {'civitai', 'wildcards', 'backups', 'stats'}
for root, dirs, files in os.walk(extracted_root):
# Skip civitai folder and its contents
# Skip user data directories and their contents
rel_root = os.path.relpath(root, extracted_root)
if rel_root == 'civitai' or rel_root.startswith('civitai' + os.sep):
top_dir = rel_root.split(os.sep)[0] if rel_root != '.' else ''
if top_dir in skip_tracked:
continue
for file in files:
rel_path = os.path.relpath(os.path.join(root, file), extracted_root)
# Skip settings.json and any file under civitai
if rel_path == 'settings.json' or rel_path.startswith('civitai' + os.sep):
# Skip settings.json and any file under user data dirs
if rel_path == 'settings.json' or rel_path.split(os.sep)[0] in skip_tracked:
continue
tracking_files.append(rel_path.replace("\\", "/"))
with open(tracking_info_file, "w", encoding='utf-8') as file:
@@ -342,6 +377,17 @@ class UpdateRoutes:
Returns:
tuple: (success, new_version)
"""
try:
import git
except ImportError:
logger.error(
"GitPython is not available: the git executable was not found in PATH. "
"Install git or set $GIT_PYTHON_GIT_EXECUTABLE to the git binary path."
)
return False, ""
clean_excludes = _clean_excludes()
try:
# Open the Git repository
repo = git.Repo(plugin_root)
@@ -353,8 +399,9 @@ class UpdateRoutes:
if nightly:
# Reset to discard any local changes
repo.git.reset('--hard')
# Clean untracked files
repo.git.clean('-fd')
# Clean untracked files, but preserve user-managed directories
# (wildcards, backups, stats, civitai, caches, settings.json).
repo.git.clean('-fd', *clean_excludes)
# Switch to main branch and pull latest
main_branch = 'main'
@@ -371,8 +418,9 @@ class UpdateRoutes:
else:
# Reset to discard any local changes
repo.git.reset('--hard')
# Clean untracked files
repo.git.clean('-fd')
# Clean untracked files, but preserve user-managed directories
# (wildcards, backups, stats, civitai, caches, settings.json).
repo.git.clean('-fd', *clean_excludes)
# Get latest release tag
tags = sorted(repo.tags, key=lambda t: t.commit.committed_datetime, reverse=True)
@@ -438,6 +486,7 @@ class UpdateRoutes:
if not os.path.exists(os.path.join(plugin_root, '.git')):
return git_info
import git
repo = git.Repo(plugin_root)
commit = repo.head.commit
git_info['commit_hash'] = commit.hexsha
+672
View File
@@ -0,0 +1,672 @@
from __future__ import annotations
import asyncio
import json
import logging
import os
import secrets
import shutil
import socket
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, Optional, Tuple
import aiohttp
from .downloader import DownloadProgress, get_downloader, is_ssl_cert_verify_error
from .aria2_transfer_state import Aria2TransferStateStore
from .settings_manager import get_settings_manager
logger = logging.getLogger(__name__)
def _try_certifi_ca_path() -> str | None:
"""Return the certifi CA bundle path if available, else None."""
try:
import certifi # type: ignore[import-untyped]
path = certifi.where()
if os.path.isfile(path):
logger.debug(
"aria2 --ca-certificate: using certifi CA bundle at %s", path
)
return path
except ImportError:
pass
logger.debug("aria2 --ca-certificate: certifi not available")
return None
CIVITAI_DOWNLOAD_URL_PREFIXES = (
"https://civitai.com/api/download/",
"https://civitai.red/api/download/",
)
class Aria2Error(RuntimeError):
"""Raised when aria2 integration fails."""
@dataclass
class Aria2Transfer:
"""Track an aria2 download registered by the Python coordinator."""
gid: str
save_path: str
class Aria2Downloader:
"""Manage an aria2 RPC daemon for recommended model downloads."""
_instance = None
_lock = asyncio.Lock()
@classmethod
async def get_instance(cls) -> "Aria2Downloader":
async with cls._lock:
if cls._instance is None:
cls._instance = cls()
return cls._instance
def __init__(self) -> None:
if hasattr(self, "_initialized"):
return
self._initialized = True
self._process: Optional[asyncio.subprocess.Process] = None
self._rpc_port: Optional[int] = None
self._rpc_secret = ""
self._rpc_url = ""
self._rpc_session: Optional[aiohttp.ClientSession] = None
self._rpc_session_lock = asyncio.Lock()
self._process_lock = asyncio.Lock()
self._transfers: Dict[str, Aria2Transfer] = {}
self._poll_interval = 0.5
self._state_store = Aria2TransferStateStore()
self._stderr_reader_task: Optional[asyncio.Task] = None
@property
def is_running(self) -> bool:
return self._process is not None and self._process.returncode is None
async def download_file(
self,
url: str,
save_path: str,
*,
download_id: str,
progress_callback=None,
headers: Optional[Dict[str, str]] = None,
) -> Tuple[bool, str]:
"""Download a file using aria2 RPC and wait for completion."""
await self._ensure_process()
save_path = os.path.abspath(save_path)
transfer = self._transfers.get(download_id)
if transfer is None or os.path.abspath(transfer.save_path) != save_path:
gid = await self._schedule_download(
url,
save_path,
download_id=download_id,
headers=headers,
)
transfer = Aria2Transfer(gid=gid, save_path=save_path)
self._transfers[download_id] = transfer
try:
while True:
status = await self._get_status_with_retry(download_id)
if status is None:
return False, "aria2 download not found"
snapshot = self._build_progress_snapshot(status)
if progress_callback is not None:
await self._dispatch_progress(progress_callback, snapshot)
state = status.get("status", "")
if state == "complete":
completed_path = self._resolve_completed_path(status, save_path)
return True, completed_path
if state == "error":
return False, status.get("errorMessage") or "aria2 download failed"
if state == "removed":
return False, "Download was cancelled"
await asyncio.sleep(self._poll_interval)
finally:
self._transfers.pop(download_id, None)
async def _get_status_with_retry(
self, download_id: str, *, max_retries: int = 4, retry_delay: float = 3.0
) -> Optional[Dict[str, Any]]:
"""Call get_status with retry for transient RPC failures.
Only retries on :exc:`Aria2Error` (RPC-level failure). Returns
``None`` immediately when the download_id is not tracked (a missing
transfer is not a transient condition, so retrying is pointless).
A single failed RPC call should not immediately fail the download,
because aria2 may be temporarily busy (e.g. finalizing multiple
concurrent downloads) and a retry will often succeed.
"""
last_exc: Optional[Exception] = None
for attempt in range(max_retries):
try:
return await self.get_status(download_id)
except Aria2Error as exc:
last_exc = exc
if attempt < max_retries - 1:
logger.warning(
"aria2 get_status transient failure (attempt %d/%d) for %s: %s",
attempt + 1, max_retries, download_id, exc,
)
await asyncio.sleep(retry_delay)
raise Aria2Error(
f"Failed to query aria2 download status after {max_retries} attempts: {last_exc}"
) from last_exc
async def _schedule_download(
self,
url: str,
save_path: str,
*,
download_id: str,
headers: Optional[Dict[str, str]] = None,
) -> str:
save_dir = os.path.dirname(save_path)
out_name = os.path.basename(save_path)
Path(save_dir).mkdir(parents=True, exist_ok=True)
resolved_url = url
request_headers = headers
if headers and url.startswith(CIVITAI_DOWNLOAD_URL_PREFIXES):
resolved_url = await self._resolve_authenticated_redirect_url(url, headers)
if resolved_url != url:
request_headers = None
logger.debug(
"Resolved Civitai download %s to signed URL for aria2",
download_id,
)
options: Dict[str, str] = {
"dir": save_dir,
"out": out_name,
"continue": "true",
"max-connection-per-server": "4",
"split": "4",
"min-split-size": "1M",
"allow-overwrite": "true",
"auto-file-renaming": "false",
"file-allocation": "none",
}
if request_headers:
options["header"] = [
f"{key}: {value}" for key, value in request_headers.items()
]
logger.debug(
"Submitting aria2 download %s -> %s (auth=%s, civitai_signed=%s)",
download_id,
save_path,
bool(request_headers),
resolved_url != url,
)
try:
gid = await self._rpc_call("aria2.addUri", [[resolved_url], options])
except Exception as exc:
raise Aria2Error(f"Failed to schedule aria2 download: {exc}") from exc
logger.debug("aria2 accepted download %s with gid %s", download_id, gid)
await self._state_store.upsert(
download_id,
{
"gid": gid,
"save_path": save_path,
"status": "downloading",
"url": url,
},
)
return gid
async def get_status(self, download_id: str) -> Optional[Dict[str, Any]]:
"""Return the raw aria2 status payload for a known download."""
transfer = self._transfers.get(download_id)
if transfer is None:
return None
keys = [
"gid",
"status",
"totalLength",
"completedLength",
"downloadSpeed",
"errorMessage",
"files",
]
try:
status = await self._rpc_call("aria2.tellStatus", [transfer.gid, keys])
except Exception as exc:
raise Aria2Error(f"Failed to query aria2 download status: {exc}") from exc
if isinstance(status, dict):
return status
return None
async def get_status_by_gid(self, gid: str) -> Optional[Dict[str, Any]]:
keys = [
"gid",
"status",
"totalLength",
"completedLength",
"downloadSpeed",
"errorMessage",
"files",
]
try:
status = await self._rpc_call("aria2.tellStatus", [gid, keys])
except Exception as exc:
message = str(exc)
if "cannot be found" in message.lower() or "not found" in message.lower():
return None
raise Aria2Error(f"Failed to query aria2 download status: {exc}") from exc
if isinstance(status, dict):
return status
return None
async def restore_transfer(self, download_id: str, gid: str, save_path: str) -> None:
await self._ensure_process()
self._transfers[download_id] = Aria2Transfer(
gid=gid,
save_path=os.path.abspath(save_path),
)
async def reassign_transfer(
self, from_download_id: str, to_download_id: str
) -> Optional[Aria2Transfer]:
transfer = self._transfers.get(from_download_id)
if transfer is None:
return None
self._transfers[to_download_id] = transfer
if from_download_id != to_download_id:
self._transfers.pop(from_download_id, None)
return transfer
async def has_transfer(self, download_id: str) -> bool:
return download_id in self._transfers
async def pause_download(self, download_id: str) -> Dict[str, Any]:
transfer = self._transfers.get(download_id)
if transfer is None:
return {"success": False, "error": "Download task not found"}
try:
await self._rpc_call("aria2.forcePause", [transfer.gid])
except Exception as exc:
return {"success": False, "error": str(exc)}
await self._state_store.upsert(download_id, {"status": "paused"})
return {"success": True, "message": "Download paused successfully"}
async def resume_download(self, download_id: str) -> Dict[str, Any]:
transfer = self._transfers.get(download_id)
if transfer is None:
return {"success": False, "error": "Download task not found"}
try:
await self._rpc_call("aria2.unpause", [transfer.gid])
except Exception as exc:
return {"success": False, "error": str(exc)}
await self._state_store.upsert(download_id, {"status": "downloading"})
return {"success": True, "message": "Download resumed successfully"}
async def cancel_download(self, download_id: str) -> Dict[str, Any]:
transfer = self._transfers.get(download_id)
if transfer is None:
return {"success": False, "error": "Download task not found"}
try:
await self._rpc_call("aria2.forceRemove", [transfer.gid])
except Exception as exc:
return {"success": False, "error": str(exc)}
await self._state_store.remove(download_id)
return {"success": True, "message": "Download cancelled successfully"}
async def close(self) -> None:
"""Shut down the RPC process and session."""
# Cancel the background stderr reader first so it stops reading
# from the pipe before the subprocess is terminated.
if self._stderr_reader_task is not None:
self._stderr_reader_task.cancel()
try:
await asyncio.wait_for(self._stderr_reader_task, timeout=2.0)
except (asyncio.CancelledError, asyncio.TimeoutError):
pass
self._stderr_reader_task = None
if self._rpc_session is not None:
await self._rpc_session.close()
self._rpc_session = None
process = self._process
self._process = None
self._transfers.clear()
if process is None:
return
if process.returncode is None:
process.terminate()
try:
await asyncio.wait_for(process.wait(), timeout=5.0)
except asyncio.TimeoutError:
process.kill()
await process.wait()
async def _drain_stderr(self) -> None:
"""Continuously drain aria2's stderr pipe so it never blocks.
When the 64 KB pipe buffer fills up, aria2's ``write()`` to stderr
blocks, which freezes the entire ``aria2c`` process — including its
RPC handler. This background task reads lines from stderr as they
arrive and forwards them to Python's logger.
"""
try:
assert self._process is not None and self._process.stderr is not None
async for line in self._process.stderr:
text = line.decode("utf-8", errors="replace").rstrip()
if text:
logger.debug("aria2 stderr: %s", text)
except Exception:
pass
async def _dispatch_progress(self, callback, snapshot: DownloadProgress) -> None:
try:
result = callback(snapshot, snapshot)
except TypeError:
result = callback(snapshot.percent_complete)
if asyncio.iscoroutine(result):
await result
elif hasattr(result, "__await__"):
await result
def _build_progress_snapshot(self, status: Dict[str, Any]) -> DownloadProgress:
completed = self._parse_int(status.get("completedLength"))
total = self._parse_int(status.get("totalLength"))
speed = float(self._parse_int(status.get("downloadSpeed")))
percent = 0.0
if total > 0:
percent = (completed / total) * 100.0
return DownloadProgress(
percent_complete=max(0.0, min(percent, 100.0)),
bytes_downloaded=completed,
total_bytes=total or None,
bytes_per_second=speed,
timestamp=datetime.now().timestamp(),
)
def _resolve_completed_path(self, status: Dict[str, Any], default_path: str) -> str:
files = status.get("files")
if isinstance(files, list) and files:
first = files[0]
if isinstance(first, dict):
candidate = first.get("path")
if isinstance(candidate, str) and candidate:
return candidate
return default_path
@staticmethod
def _parse_int(value: Any) -> int:
try:
return int(value)
except (TypeError, ValueError):
return 0
async def _resolve_authenticated_redirect_url(
self,
url: str,
headers: Dict[str, str],
) -> str:
downloader = await get_downloader()
session = await downloader.session
request_headers = dict(downloader.default_headers)
request_headers.update(headers)
request_headers["Accept-Encoding"] = "identity"
try:
async with session.get(
url,
headers=request_headers,
allow_redirects=False,
proxy=downloader.proxy_url,
) as response:
if response.status in {301, 302, 303, 307, 308}:
location = response.headers.get("Location")
if location:
return location
raise Aria2Error(
"Authenticated Civitai redirect did not include a Location header"
)
if response.status == 200:
return url
body = await response.text()
raise Aria2Error(
f"Failed to resolve authenticated Civitai redirect: status={response.status} body={body[:300]}"
)
except aiohttp.ClientError as exc:
if is_ssl_cert_verify_error(exc):
logger.error(
"SSL certificate verification failed during Civitai redirect "
"resolution for %s. This is usually caused by an outdated CA "
"certificate bundle. Recommended fixes:\n"
" 1. pip install --upgrade certifi\n"
" 2. pip install pip-system-certs",
url,
)
raise Aria2Error(
f"Failed to resolve authenticated Civitai redirect: {exc}"
) from exc
async def _ensure_process(self) -> None:
async with self._process_lock:
if self.is_running and await self._ping():
return
await self.close()
executable = self._resolve_executable()
self._rpc_port = self._find_free_port()
self._rpc_secret = secrets.token_hex(16)
self._rpc_url = f"http://127.0.0.1:{self._rpc_port}/jsonrpc"
command = [
executable,
"--enable-rpc=true",
"--rpc-listen-all=false",
f"--rpc-listen-port={self._rpc_port}",
f"--rpc-secret={self._rpc_secret}",
"--check-certificate=true",
# Point aria2 at certifi's CA bundle when available so it uses
# the same certificate store as Python downloads.
*((
f"--ca-certificate={ca_cert}",
) if (ca_cert := _try_certifi_ca_path()) else ()),
"--allow-overwrite=true",
"--auto-file-renaming=false",
"--file-allocation=none",
"--max-concurrent-downloads=5",
"--continue=true",
"--daemon=false",
"--quiet=true",
f"--stop-with-process={os.getpid()}",
]
logger.info("Starting aria2 RPC daemon from %s", executable)
self._process = await asyncio.create_subprocess_exec(
*command,
stdout=asyncio.subprocess.DEVNULL,
stderr=asyncio.subprocess.PIPE,
)
await self._wait_until_ready()
# Drain aria2's stderr in a background task so the pipe buffer
# never fills up. If the pipe blocks, aria2 itself freezes and
# cannot respond to RPC — this was the root cause of the
# "Failed to query aria2 download status" timeout bug.
# Must start AFTER _wait_until_ready to avoid a race where the
# drain task consumes aria2's early-exit error message before
# _wait_until_ready can read it.
self._stderr_reader_task = asyncio.create_task(
self._drain_stderr()
)
def _resolve_executable(self) -> str:
settings = get_settings_manager()
configured_path = (settings.get("aria2c_path") or "").strip()
candidate = configured_path or "aria2c"
resolved = shutil.which(candidate)
if resolved:
return resolved
if configured_path and os.path.isfile(configured_path) and os.access(
configured_path, os.X_OK
):
return configured_path
raise Aria2Error(
"aria2c executable was not found. Install aria2 or configure aria2c_path."
)
async def _wait_until_ready(self) -> None:
assert self._process is not None
start_time = asyncio.get_running_loop().time()
last_error = ""
while asyncio.get_running_loop().time() - start_time < 10.0:
if self._process.returncode is not None:
stderr_output = ""
if self._process.stderr is not None:
try:
stderr_output = (
await asyncio.wait_for(self._process.stderr.read(), timeout=0.2)
).decode("utf-8", errors="replace")
except Exception:
stderr_output = ""
raise Aria2Error(
f"aria2 RPC process exited early with code {self._process.returncode}: {stderr_output.strip()}"
)
try:
if await self._ping():
return
except Exception as exc: # pragma: no cover - startup race
last_error = str(exc)
await asyncio.sleep(0.2)
raise Aria2Error(
f"Timed out waiting for aria2 RPC to become ready{': ' + last_error if last_error else ''}"
)
async def _ping(self) -> bool:
try:
result = await self._rpc_call("aria2.getVersion", [])
except Exception:
return False
return isinstance(result, dict)
async def _rpc_call(self, method: str, params: list[Any]) -> Any:
if not self._rpc_url:
raise Aria2Error("aria2 RPC endpoint is not initialized")
session = await self._get_rpc_session()
payload = {
"jsonrpc": "2.0",
"id": secrets.token_hex(8),
"method": method,
"params": [f"token:{self._rpc_secret}", *params],
}
async with session.post(self._rpc_url, json=payload) as response:
text = await response.text()
try:
body = json.loads(text)
except json.JSONDecodeError:
body = None
if body is None:
if response.status != 200:
raise Aria2Error(
f"aria2 RPC returned status {response.status} with non-JSON body: {text}"
)
raise Aria2Error(f"Invalid aria2 RPC response: {text}")
if "error" in body:
error = body["error"] or {}
code = error.get("code") if isinstance(error, dict) else None
message = error.get("message") if isinstance(error, dict) else str(error)
logger.error(
"aria2 RPC %s failed with HTTP %s, code=%s, message=%s",
method,
response.status,
code,
message,
)
status_message = (
f"aria2 RPC {method} failed with status {response.status}: {message}"
if response.status != 200
else message
)
raise Aria2Error(status_message or "Unknown aria2 RPC error")
if response.status != 200:
logger.error(
"aria2 RPC %s returned unexpected HTTP status %s without error payload: %s",
method,
response.status,
body,
)
raise Aria2Error(
f"aria2 RPC {method} returned unexpected status {response.status}"
)
return body.get("result")
async def _get_rpc_session(self) -> aiohttp.ClientSession:
if self._rpc_session is None or self._rpc_session.closed:
async with self._rpc_session_lock:
if self._rpc_session is None or self._rpc_session.closed:
timeout = aiohttp.ClientTimeout(
total=None, sock_connect=10, sock_read=60
)
self._rpc_session = aiohttp.ClientSession(timeout=timeout)
return self._rpc_session
@staticmethod
def _find_free_port() -> int:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.bind(("127.0.0.1", 0))
sock.listen(1)
return int(sock.getsockname()[1])
async def get_aria2_downloader() -> Aria2Downloader:
"""Get the singleton aria2 downloader."""
return await Aria2Downloader.get_instance()
+108
View File
@@ -0,0 +1,108 @@
from __future__ import annotations
import asyncio
import json
import os
from copy import deepcopy
from typing import Any, Dict, Optional
from ..utils.cache_paths import get_cache_base_dir
def get_aria2_state_path() -> str:
base_dir = get_cache_base_dir(create=True)
state_dir = os.path.join(base_dir, "aria2")
os.makedirs(state_dir, exist_ok=True)
return os.path.join(state_dir, "downloads.json")
class Aria2TransferStateStore:
"""Persist aria2 transfer metadata needed for restart recovery."""
_locks_by_path: Dict[str, asyncio.Lock] = {}
def __init__(self, state_path: Optional[str] = None) -> None:
self._state_path = os.path.abspath(state_path or get_aria2_state_path())
self._lock = self._locks_by_path.setdefault(self._state_path, asyncio.Lock())
def _read_all_unlocked(self) -> Dict[str, Dict[str, Any]]:
try:
with open(self._state_path, "r", encoding="utf-8") as handle:
data = json.load(handle)
except FileNotFoundError:
return {}
except json.JSONDecodeError:
return {}
if not isinstance(data, dict):
return {}
normalized: Dict[str, Dict[str, Any]] = {}
for download_id, entry in data.items():
if isinstance(download_id, str) and isinstance(entry, dict):
normalized[download_id] = entry
return normalized
def _write_all_unlocked(self, data: Dict[str, Dict[str, Any]]) -> None:
directory = os.path.dirname(self._state_path)
if directory:
os.makedirs(directory, exist_ok=True)
temp_path = f"{self._state_path}.tmp"
with open(temp_path, "w", encoding="utf-8") as handle:
json.dump(data, handle, ensure_ascii=True, indent=2, sort_keys=True)
os.replace(temp_path, self._state_path)
async def load_all(self) -> Dict[str, Dict[str, Any]]:
async with self._lock:
return deepcopy(self._read_all_unlocked())
async def get(self, download_id: str) -> Optional[Dict[str, Any]]:
async with self._lock:
return deepcopy(self._read_all_unlocked().get(download_id))
async def upsert(self, download_id: str, payload: Dict[str, Any]) -> Dict[str, Any]:
async with self._lock:
data = self._read_all_unlocked()
current = data.get(download_id, {})
current.update(payload)
data[download_id] = current
self._write_all_unlocked(data)
return deepcopy(current)
async def remove(self, download_id: str) -> None:
async with self._lock:
data = self._read_all_unlocked()
if download_id in data:
del data[download_id]
self._write_all_unlocked(data)
async def find_by_save_path(
self, save_path: str, *, exclude_download_id: Optional[str] = None
) -> Optional[Dict[str, Any]]:
normalized_target = os.path.abspath(save_path)
async with self._lock:
data = self._read_all_unlocked()
for download_id, entry in data.items():
if exclude_download_id and download_id == exclude_download_id:
continue
candidate = entry.get("save_path")
if isinstance(candidate, str) and os.path.abspath(candidate) == normalized_target:
result = dict(entry)
result["download_id"] = download_id
return result
return None
async def reassign(self, from_download_id: str, to_download_id: str) -> Optional[Dict[str, Any]]:
async with self._lock:
data = self._read_all_unlocked()
existing = data.get(from_download_id)
if existing is None:
return None
updated = dict(existing)
updated["download_id"] = to_download_id
data[to_download_id] = updated
if from_download_id != to_download_id:
data.pop(from_download_id, None)
self._write_all_unlocked(data)
return deepcopy(updated)
+139
View File
@@ -0,0 +1,139 @@
"""
Auto-tag extraction service for model cards.
Extracts implicit model attributes (HIGH/LOW, I2V/T2V/TI2V, Lightning, Turbo)
from filename, base_model, and CivitAI version name — no manual tagging required.
"""
from __future__ import annotations
import re
from typing import Dict, List, Set
# ── Tag category definitions ──────────────────────────────────────────
# Each category maps a display label to a regex pattern.
# Patterns are case-insensitive and matched against filename, base_model,
# and civitai version name.
# Use (?<![a-zA-Z0-9]) and (?![a-zA-Z0-9]) instead of \b because
# Python's \b treats underscore as a word character, so \bHIGH\b
# won't match '_HIGH_' in filenames.
_B = r"(?<![a-zA-Z0-9])" # left boundary
_E = r"(?![a-zA-Z0-9])" # right boundary
AUTO_TAG_CATEGORIES: Dict[str, str] = {
"HIGH": _B + r"HIGH" + _E,
"LOW": _B + r"(?<!F)LOW" + _E,
"I2V": _B + r"I2V" + _E,
"T2V": _B + r"T2V" + _E,
"TI2V": _B + r"TI2V" + _E,
"Lightning": _B + r"Lightning" + _E,
"Turbo": _B + r"Turbo" + _E,
}
# Tags that belong to the "mode" group (HIGH/LOW)
MODE_TAGS = {"HIGH", "LOW"}
# Tags that belong to the "video mode" group (I2V/T2V/TI2V)
VIDEO_MODE_TAGS = {"I2V", "T2V", "TI2V"}
# Tags that belong to the "speed/optimization" group
SPEED_TAGS = {"Lightning", "Turbo"}
# ── Display category groups (for settings UI) ─────────────────────────
AUTO_TAG_GROUPS = {
"mode": {"HIGH", "LOW"},
"video": {"I2V", "T2V", "TI2V"},
"speed": {"Lightning", "Turbo"},
}
# Default enabled categories
DEFAULT_ENABLED_GROUPS = {"mode", "video"}
def _collect_sources(model_data: Dict) -> List[str]:
"""Collect all text sources from model data for tag matching."""
sources: List[str] = []
file_name = model_data.get("file_name", "")
if file_name:
sources.append(file_name)
base_model = model_data.get("base_model", "")
if base_model:
sources.append(base_model)
civitai = model_data.get("civitai", {})
if isinstance(civitai, dict):
version_name = civitai.get("name", "")
if version_name:
sources.append(version_name)
return sources
def extract_auto_tags(model_data: Dict) -> List[str]:
"""Extract auto-detected tags from model metadata.
Uses a two-layer approach:
Layer 1 — Regex-based detection against filename, base_model, and
CivitAI version name.
Layer 2 — Merge in any user-defined tags that overlap with known
auto-tag categories. This provides a manual fallback when
auto-detection fails (e.g. "I2V HN" or unlabeled models).
HIGH/LOW tags are only returned when the base_model indicates a Wan
family model — no other model architecture uses this distinction.
Args:
model_data: Model metadata dict with keys:
file_name, base_model, civitai (with optional 'name' field),
tags (user-defined tag list, used as fallback).
Returns:
Sorted list of unique auto-tag strings (e.g. ["I2V"]).
"""
sources = _collect_sources(model_data)
base_model = model_data.get("base_model", "")
is_wan = "wan" in base_model.lower()
found: Set[str] = set()
# ── Layer 1: regex-based detection ────────────────────────────
if sources:
for label, pattern in AUTO_TAG_CATEGORIES.items():
# HIGH/LOW are Wan-specific — skip for non-Wan to avoid noise
if label in ("HIGH", "LOW"):
if not is_wan:
continue
# Use case-insensitive character class + case-sensitive boundary,
# so "HighNoise" (camelCase) matches but "highlight" doesn't.
# Boundary: not followed by lowercase letter (= word has ended).
ci = "".join(f"[{c.lower()}{c.upper()}]" for c in label)
if label == "LOW":
regex = re.compile(r"(?<![Ff])" + ci + r"(?![a-z])")
else:
regex = re.compile(ci + r"(?![a-z])")
else:
regex = re.compile(pattern, re.IGNORECASE)
for source in sources:
if regex.search(source):
found.add(label)
break
# ── Layer 2: user-defined tags as manual fallback ─────────────
# When auto-detection fails (abbreviated names like "Hi"/"Lo",
# "I2V HN", or unlabeled models), users can add canonical tags
# (HIGH, LOW, I2V, etc.) to the model's regular tags for correct
# badge display and filtering. Matching is case-insensitive so
# "high"/"High"/"HIGH" all resolve to the canonical label.
user_tags = model_data.get("tags")
if user_tags:
label_map = {label.lower(): label for label in AUTO_TAG_CATEGORIES}
for t in user_tags:
canonical = label_map.get(t.lower())
if canonical:
found.add(canonical)
return sorted(found)
+12
View File
@@ -141,6 +141,16 @@ class BackupService:
)
)
stats_path = os.path.join(get_settings_dir(create=True), "stats", "lora_manager_stats.json")
if os.path.exists(stats_path):
targets.append(
(
"usage_stats",
"stats/lora_manager_stats.json",
stats_path,
)
)
return targets
@staticmethod
@@ -348,6 +358,8 @@ class BackupService:
if kind == "model_update":
filename = os.path.basename(archive_member)
return str(Path(get_cache_file_path(CacheType.MODEL_UPDATE, create_dir=True)).parent / filename)
if kind == "usage_stats":
return os.path.join(get_settings_dir(create=True), "stats", "lora_manager_stats.json")
return None
async def create_auto_snapshot_if_due(self) -> Optional[dict[str, Any]]:
+199 -6
View File
@@ -77,6 +77,7 @@ class BaseModelService(ABC):
base_models: list = None,
model_types: list = None,
tags: Optional[Dict[str, str]] = None,
auto_tags: Optional[Dict[str, str]] = None,
search_options: dict = None,
hash_filters: dict = None,
favorites_only: bool = False,
@@ -95,9 +96,108 @@ class BaseModelService(ABC):
sorted_data = await self._fetch_with_usage_sort(sort_params)
else:
sorted_data = await self.cache_repository.fetch_sorted(sort_params)
# Pre-compute auto_tags for every item — needed for both filtering
# and display. Computation is cheap (string regex on 2-3 fields).
from .auto_tag_service import extract_auto_tags
for item in sorted_data:
item["auto_tags"] = extract_auto_tags(item)
fetch_duration = time.perf_counter() - t0
initial_count = len(sorted_data)
# Optionally filter by civitai model ID (shows all local versions of a specific model)
civitai_model_id = kwargs.get("civitai_model_id")
if civitai_model_id is not None:
sorted_data = [
item for item in sorted_data
if self._extract_model_id(item) == civitai_model_id
]
# VLM mode: always sort by version ID descending (newest version first),
# regardless of the current sort_by preference.
sorted_data.sort(
key=lambda x: self._extract_version_id(x) or 0,
reverse=True,
)
# Optionally group by civitai modelId, showing only the latest version per model
dedup_lost = 0
if kwargs.get("group_by_model") and civitai_model_id is None:
# Determine whether to further sub-group by base model
# When version_grouping is "same_base", versions with different
# base models are effectively different groups — the dedup key
# needs to include base_model so the version count and VLM flow
# stay consistent (card shows correct count for its base model).
ufs = self.settings.get("version_grouping", "same_base")
group_by_base = ufs == "same_base"
dedup_map = {} # (modelId [,base_model]) -> (item, version_id)
version_counter = {} # same-key -> count
standalone = []
for item in sorted_data:
mid = self._extract_model_id(item)
if mid is None:
standalone.append(item)
continue
key = (mid, item.get("base_model") or "") if group_by_base else mid
# Count all versions per key
version_counter[key] = version_counter.get(key, 0) + 1
vid = self._extract_version_id(item) or 0
if key not in dedup_map or vid > dedup_map[key][1]:
dedup_map[key] = (item, vid)
# Attach version_count to each surviving grouped item (shallow copy
# to avoid mutating cached dicts — the cache is shared across requests)
for key, (item, vid) in dedup_map.items():
item = dict(item)
item["version_count"] = version_counter[key]
dedup_map[key] = (item, vid)
dedup_lost = len(sorted_data) - (len(dedup_map) + len(standalone))
sorted_data = [entry[0] for entry in dedup_map.values()] + standalone
# Re-sort by version_count (grouped: after dedup; non-grouped: group internally, sort, expand)
if sort_params.key == "versions_count" and civitai_model_id is None:
reverse = sort_params.order == "desc"
if kwargs.get("group_by_model"):
# Grouped mode: items are already dedup'd with version_count attached
sorted_data.sort(
key=lambda x: (
x.get("version_count", 0),
(x.get("model_name") or x.get("file_name") or "").lower(),
x.get("file_path", "").lower(),
),
reverse=reverse,
)
else:
# Non-grouped mode: group internally, sort groups by count, expand
# Respect the version_grouping setting (same logic as grouped dedup)
ufs = self.settings.get("version_grouping", "same_base")
group_by_base = ufs == "same_base"
model_groups: Dict[Any, List[Dict]] = {}
ungrouped_standalone: List[Dict] = []
for item in sorted_data:
mid = self._extract_model_id(item)
if mid is None:
ungrouped_standalone.append(item)
continue
key = (mid, item.get("base_model") or "") if group_by_base else mid
model_groups.setdefault(key, []).append(item)
# Sort versions within each group by version id descending
for items in model_groups.values():
items.sort(
key=lambda x: self._extract_version_id(x) or 0,
reverse=True,
)
# Sort groups by version count
sorted_groups = sorted(
model_groups.values(),
key=lambda items: len(items),
reverse=reverse,
)
# Flatten: grouped items first, standalone items last
sorted_data = []
for items in sorted_groups:
sorted_data.extend(items)
sorted_data.extend(ungrouped_standalone)
t1 = time.perf_counter()
if hash_filters:
filtered_data = await self._apply_hash_filters(sorted_data, hash_filters)
@@ -110,6 +210,7 @@ class BaseModelService(ABC):
base_models=base_models,
model_types=model_types,
tags=tags,
auto_tags=auto_tags,
favorites_only=favorites_only,
search_options=search_options,
tag_logic=tag_logic,
@@ -165,7 +266,7 @@ class BaseModelService(ABC):
overall_duration = time.perf_counter() - overall_start
logger.debug(
"%s.get_paginated_data took %.3fs (fetch: %.3fs, filter: %.3fs, update_filter: %.3fs, pagination: %.3fs, annotate: %.3fs). "
"Counts: initial=%d, post_filter=%d, final=%d",
"Counts: initial=%d, dedup=%d, post_filter=%d, final=%d",
self.__class__.__name__,
overall_duration,
fetch_duration,
@@ -174,6 +275,7 @@ class BaseModelService(ABC):
pagination_duration,
annotate_duration,
initial_count,
dedup_lost,
post_filter_count,
final_count,
)
@@ -354,6 +456,7 @@ class BaseModelService(ABC):
base_models: list = None,
model_types: list = None,
tags: Optional[Dict[str, str]] = None,
auto_tags: Optional[Dict[str, str]] = None,
favorites_only: bool = False,
search_options: dict = None,
tag_logic: str = "any",
@@ -367,6 +470,7 @@ class BaseModelService(ABC):
base_models=base_models,
model_types=model_types,
tags=tags,
auto_tags=auto_tags,
favorites_only=favorites_only,
search_options=normalized_options,
tag_logic=tag_logic,
@@ -486,7 +590,7 @@ class BaseModelService(ABC):
if not ordered_ids:
return annotated
strategy_value = self.settings.get("update_flag_strategy")
strategy_value = self.settings.get("version_grouping")
if isinstance(strategy_value, str) and strategy_value.strip():
strategy = strategy_value.strip().lower()
else:
@@ -687,8 +791,12 @@ class BaseModelService(ABC):
}
@abstractmethod
async def format_response(self, model_data: Dict) -> Dict:
"""Format model data for API response - must be implemented by subclasses"""
async def format_response(self, model_data: Dict) -> Optional[Dict]:
"""Format model data for API response - must be implemented by subclasses.
Subclasses should return None for corrupted entries so the handler
layer can filter them out. See issue #730.
"""
pass
# Common service methods that delegate to scanner
@@ -861,22 +969,75 @@ class BaseModelService(ABC):
"""Get the static preview URL for a model file"""
cache = await self.scanner.get_cached_data()
name_normalized = model_name.replace("\\", "/")
name_no_ext = name_normalized
for ext in (".safetensors", ".ckpt", ".pt", ".bin"):
if name_no_ext.lower().endswith(ext):
name_no_ext = name_no_ext[: -len(ext)]
break
has_path = "/" in name_no_ext
basename = os.path.basename(name_no_ext) if has_path else name_no_ext
best_fallback = None
for model in cache.raw_data:
if model["file_name"] == model_name:
file_name = model.get("file_name", "")
folder = model.get("folder", "")
file_name_no_ext = file_name
for ext in (".safetensors", ".ckpt", ".pt", ".bin"):
if file_name_no_ext.lower().endswith(ext):
file_name_no_ext = file_name_no_ext[: -len(ext)]
break
path_name = f"{folder}/{file_name_no_ext}".replace("\\", "/") if folder else file_name_no_ext
if name_no_ext == file_name_no_ext or name_no_ext == path_name:
preview_url = model.get("preview_url")
if preview_url:
from ..config import config
return config.get_preview_static_url(preview_url)
if has_path and file_name_no_ext == basename:
if folder and name_no_ext.startswith(folder.replace("\\", "/") + "/"):
best_fallback = model
elif best_fallback is None:
best_fallback = model
if best_fallback:
preview_url = best_fallback.get("preview_url")
if preview_url:
from ..config import config
return config.get_preview_static_url(preview_url)
return "/loras_static/images/no-preview.png"
async def get_model_civitai_url(self, model_name: str) -> Dict[str, Optional[str]]:
"""Get the Civitai URL for a model file"""
cache = await self.scanner.get_cached_data()
name_normalized = model_name.replace("\\", "/")
name_no_ext = name_normalized
for ext in (".safetensors", ".ckpt", ".pt", ".bin"):
if name_no_ext.lower().endswith(ext):
name_no_ext = name_no_ext[: -len(ext)]
break
has_path = "/" in name_no_ext
basename = os.path.basename(name_no_ext) if has_path else name_no_ext
best_fallback = None
for model in cache.raw_data:
if model["file_name"] == model_name:
file_name = model.get("file_name", "")
folder = model.get("folder", "")
file_name_no_ext = file_name
for ext in (".safetensors", ".ckpt", ".pt", ".bin"):
if file_name_no_ext.lower().endswith(ext):
file_name_no_ext = file_name_no_ext[: -len(ext)]
break
path_name = f"{folder}/{file_name_no_ext}".replace("\\", "/") if folder else file_name_no_ext
if name_no_ext == file_name_no_ext or name_no_ext == path_name:
civitai_data = model.get("civitai", {})
model_id = civitai_data.get("modelId")
version_id = civitai_data.get("id")
@@ -895,6 +1056,27 @@ class BaseModelService(ABC):
"version_id": str(version_id) if version_id else None,
}
if has_path and file_name_no_ext == basename:
if folder and name_no_ext.startswith(folder.replace("\\", "/") + "/"):
best_fallback = model
elif best_fallback is None:
best_fallback = model
if best_fallback:
civitai_data = best_fallback.get("civitai", {})
model_id = civitai_data.get("modelId")
if model_id:
version_id = civitai_data.get("id")
civitai_host = self.settings.get("civitai_host", "civitai.com")
civitai_url = build_civitai_model_page_url(
model_id, version_id, host=civitai_host
)
return {
"civitai_url": civitai_url,
"model_id": str(model_id),
"version_id": str(version_id) if version_id else None,
}
return {"civitai_url": None, "model_id": None, "version_id": None}
async def get_model_metadata(self, file_path: str) -> Optional[Dict]:
@@ -908,6 +1090,17 @@ class BaseModelService(ABC):
)
if should_skip or metadata is None:
return None
# Prune stale example-image metadata entries whose files no longer
# exist on disk (e.g. a user deleted the files manually).
from ..utils.example_images_metadata import MetadataUpdater
was_modified = await MetadataUpdater.prune_stale_example_images(metadata)
if was_modified:
asyncio.create_task(
MetadataManager.save_metadata(file_path, metadata)
)
return self.filter_civitai_data(metadata.to_dict().get("civitai", {}))
async def get_model_description(self, file_path: str) -> Optional[str]:
+5 -1
View File
@@ -224,7 +224,7 @@ class BatchImportService:
return False
for recipe in getattr(cache, "raw_data", []):
source_path = recipe.get("source_path") or recipe.get("source_url")
source_path = recipe.get("source_path")
if source_path and source_path == source:
return True
return False
@@ -523,6 +523,10 @@ class BatchImportService:
if payload.get("checkpoint"):
metadata["checkpoint"] = payload["checkpoint"]
nsfw = payload.get("preview_nsfw_level")
if isinstance(nsfw, int) and nsfw > 0:
metadata["preview_nsfw_level"] = nsfw
image_bytes = None
image_base64 = payload.get("image_base64")
+58 -3
View File
@@ -1,3 +1,4 @@
import asyncio
import json
import logging
import os
@@ -36,6 +37,9 @@ class CheckpointScanner(ModelScanner):
file_extensions=file_extensions,
hash_index=ModelHashIndex(),
)
if not hasattr(self, "_hash_calculation_lock"):
self._hash_calculation_lock = asyncio.Lock()
self._hash_calculation_tasks: dict[str, asyncio.Task[Optional[str]]] = {}
async def _create_default_metadata(
self, file_path: str
@@ -88,7 +92,7 @@ class CheckpointScanner(ModelScanner):
return None
async def calculate_hash_for_model(self, file_path: str) -> Optional[str]:
"""Calculate hash for a checkpoint on-demand.
"""Calculate hash for a checkpoint on-demand with per-file singleflight.
Args:
file_path: Path to the model file
@@ -96,14 +100,65 @@ class CheckpointScanner(ModelScanner):
Returns:
SHA256 hash string, or None if calculation failed
"""
from ..utils.file_utils import calculate_sha256
try:
real_path = os.path.realpath(file_path)
if not os.path.exists(real_path):
logger.error(f"File not found for hash calculation: {file_path}")
return None
metadata, _ = await MetadataManager.load_metadata(
file_path, self.model_class
)
if (
metadata is not None
and metadata.hash_status == "completed"
and metadata.sha256
):
return metadata.sha256
async with self._hash_calculation_lock:
metadata, _ = await MetadataManager.load_metadata(
file_path, self.model_class
)
if (
metadata is not None
and metadata.hash_status == "completed"
and metadata.sha256
):
return metadata.sha256
task = self._hash_calculation_tasks.get(real_path)
if task is None:
task = asyncio.create_task(
self._run_hash_calculation_task(file_path, real_path)
)
self._hash_calculation_tasks[real_path] = task
return await asyncio.shield(task)
except Exception as e:
logger.error(f"Error calculating hash for {file_path}: {e}")
return None
async def _run_hash_calculation_task(
self, file_path: str, real_path: str
) -> Optional[str]:
"""Run a hash calculation task and remove it from the in-flight map."""
try:
return await self._calculate_hash_for_model_uncached(file_path, real_path)
finally:
task = asyncio.current_task()
async with self._hash_calculation_lock:
if self._hash_calculation_tasks.get(real_path) is task:
del self._hash_calculation_tasks[real_path]
async def _calculate_hash_for_model_uncached(
self, file_path: str, real_path: str
) -> Optional[str]:
"""Calculate hash for a checkpoint without checking in-flight tasks."""
from ..utils.file_utils import calculate_sha256
try:
# Load current metadata
metadata, should_skip = await MetadataManager.load_metadata(
file_path, self.model_class
+30 -9
View File
@@ -1,8 +1,9 @@
import os
import logging
from typing import Dict
from typing import Dict, Optional
from .base_model_service import BaseModelService
from .auto_tag_service import extract_auto_tags
from ..utils.models import CheckpointMetadata
from ..config import config
@@ -20,20 +21,37 @@ class CheckpointService(BaseModelService):
"""
super().__init__("checkpoint", scanner, CheckpointMetadata, update_service=update_service)
async def format_response(self, checkpoint_data: Dict) -> Dict:
"""Format Checkpoint data for API response"""
async def format_response(self, checkpoint_data: Dict) -> Optional[Dict]:
"""Format Checkpoint data for API response.
Returns None when the entry is missing critical fields (corrupted cache
row), so the handler layer can filter it out. See issue #730.
"""
# Guard against corrupted cache entries missing critical fields
file_path = checkpoint_data.get("file_path")
if not file_path or not isinstance(file_path, str):
logger.warning(
"Skipping corrupted checkpoint entry (missing file_path): %s",
checkpoint_data.get("file_name", "<unknown>"),
)
return None
# Get sub_type from cache entry (new canonical field)
sub_type = checkpoint_data.get("sub_type", "checkpoint")
file_name = checkpoint_data.get("file_name") or ""
model_name = checkpoint_data.get("model_name") or file_name
folder = checkpoint_data.get("folder") or ""
return {
"model_name": checkpoint_data["model_name"],
"file_name": checkpoint_data["file_name"],
"model_name": model_name,
"file_name": file_name,
"preview_url": config.get_preview_static_url(checkpoint_data.get("preview_url", "")),
"preview_nsfw_level": checkpoint_data.get("preview_nsfw_level", 0),
"base_model": checkpoint_data.get("base_model", ""),
"folder": checkpoint_data["folder"],
"folder": folder,
"sha256": checkpoint_data.get("sha256", ""),
"file_path": checkpoint_data["file_path"].replace(os.sep, "/"),
"file_path": file_path.replace(os.sep, "/"),
"file_size": checkpoint_data.get("size", 0),
"modified": checkpoint_data.get("modified", ""),
"tags": checkpoint_data.get("tags", []),
@@ -45,7 +63,10 @@ class CheckpointService(BaseModelService):
"exclude": bool(checkpoint_data.get("exclude", False)),
"update_available": bool(checkpoint_data.get("update_available", False)),
"skip_metadata_refresh": bool(checkpoint_data.get("skip_metadata_refresh", False)),
"civitai": self.filter_civitai_data(checkpoint_data.get("civitai", {}), minimal=True)
"civitai": self.filter_civitai_data(checkpoint_data.get("civitai", {}), minimal=True),
"auto_tags": checkpoint_data.get("auto_tags") or extract_auto_tags(checkpoint_data),
"version_count": checkpoint_data.get("version_count"),
"hf_url": checkpoint_data.get("hf_url", ""),
}
def find_duplicate_hashes(self) -> Dict:
+32 -2
View File
@@ -186,6 +186,22 @@ class CivArchiveClient:
if "metadata" in file_data:
transformed["metadata"] = file_data["metadata"]
# Infer metadata.format from filename extension
name = transformed.get("name")
if name and isinstance(name, str):
lower_name = name.lower()
if lower_name.endswith(".safetensors"):
inferred_format = "SafeTensor"
elif lower_name.endswith(".ckpt"):
inferred_format = "PickleTensor"
else:
inferred_format = None
if inferred_format:
if "metadata" not in transformed:
transformed["metadata"] = {}
if isinstance(transformed["metadata"], dict):
transformed["metadata"].setdefault("format", inferred_format)
if file_data.get("modelVersionId") is not None:
transformed["modelVersionId"] = file_data.get("modelVersionId")
elif file_data.get("model_version_id") is not None:
@@ -213,6 +229,20 @@ class CivArchiveClient:
for file_data in candidates:
if isinstance(file_data, dict):
transformed_files.append(self._transform_file_entry(file_data))
# Sort: .safetensors first, .ckpt second, others last
# so the backend fallback (no file_params) prefers safetensors
def _sort_key(f: Dict) -> int:
fname = f.get("name") or ""
if isinstance(fname, str):
lower = fname.lower()
if lower.endswith(".safetensors"):
return 0
elif lower.endswith(".ckpt"):
return 1
return 2
transformed_files.sort(key=_sort_key)
return transformed_files
def _transform_version(
@@ -297,7 +327,7 @@ class CivArchiveClient:
if resolved:
return resolved, None
logger.error("Error fetching version of CivArchive model by hash %s", model_hash[:10])
logger.debug("Error fetching version of CivArchive model by hash %s", model_hash[:10])
return None, "No version data found"
except RateLimitError:
@@ -387,7 +417,7 @@ class CivArchiveClient:
if version_id is not None:
raw_id = version_data.get("id")
if raw_id != version_id:
if raw_id is not None and str(raw_id) != str(version_id):
logger.warning(
"Requested version %s doesn't match default version %s for model %s",
version_id,
@@ -193,6 +193,10 @@ class CivitaiBaseModelService:
"zimageturbo": "ZIT",
"zimagebase": "ZIB",
"anima": "ANI",
"ernie": "ERNI",
"ernie turbo": "ETRB",
"nucleus": "NUCL",
"krea 2": "KR2",
"svd": "SVD",
"ltxv": "LTXV",
"ltxv2": "LTV2",
@@ -418,6 +422,10 @@ class CivitaiBaseModelService:
"Kolors",
"NoobAI",
"Anima",
"Ernie",
"Ernie Turbo",
"Nucleus",
"Krea 2",
],
}
+210 -16
View File
@@ -2,7 +2,13 @@ import asyncio
import copy
import logging
import os
from collections import OrderedDict
from typing import Any, Optional, Dict, Tuple, List, Sequence
from .connectivity_guard import (
OFFLINE_FRIENDLY_MESSAGE,
is_expected_offline_error,
is_offline_cooldown_error,
)
from .model_metadata_provider import (
CivitaiModelMetadataProvider,
ModelMetadataProviderManager,
@@ -40,9 +46,17 @@ class CivitaiClient:
self._initialized = True
self.base_url = "https://civitai.red/api/v1"
# In-memory cache to avoid redundant get_model_version_info calls
# within the same import/scan flow. Only successful results are cached.
# Uses OrderedDict with LRU eviction at MAX_CACHE_ENTRIES to prevent
# unbounded growth in long-running server processes.
self._version_info_cache: OrderedDict[
str, Tuple[Optional[Dict], Optional[str]]
] = OrderedDict()
self._MAX_CACHE_ENTRIES = 500
def _build_image_info_url(self, image_id: str) -> str:
return f"{self.base_url}/images?imageId={image_id}&nsfw=X"
return f"{self.base_url}/images?imageId={image_id}&nsfw=X&withMeta=true"
async def _make_request(
self,
@@ -52,20 +66,57 @@ class CivitaiClient:
use_auth: bool = False,
**kwargs,
) -> Tuple[bool, Dict | str]:
"""Wrapper around downloader.make_request that surfaces rate limits."""
"""Wrapper around downloader.make_request that surfaces rate limits,
with retry for transient server errors (5xx, Cloudflare 524, network flakiness)."""
downloader = await get_downloader()
success, result = await downloader.make_request(
method,
url,
use_auth=use_auth,
**kwargs,
)
if not success and isinstance(result, RateLimitError):
if result.provider is None:
result.provider = "civitai_api"
raise result
return success, result
max_retries = 3
for attempt in range(max_retries):
downloader = await get_downloader()
success, result = await downloader.make_request(
method,
url,
use_auth=use_auth,
**kwargs,
)
if success:
return True, result
if isinstance(result, RateLimitError):
if result.provider is None:
result.provider = "civitai_api"
raise result
if is_offline_cooldown_error(result):
return False, OFFLINE_FRIENDLY_MESSAGE
# Transient server error — retry with exponential backoff
if self._is_transient_server_error(str(result)):
if attempt < max_retries - 1:
wait = 2**attempt # 1s, 2s, 4s
logger.info(
"Transient error on %s %s, retrying in %ds "
"(attempt %d/%d): %s",
method,
url,
wait,
attempt + 1,
max_retries,
result,
)
await asyncio.sleep(wait)
continue
logger.warning(
"All %d retries exhausted for %s %s: %s",
max_retries,
method,
url,
result,
)
return False, result
return False, result
return False, "Unexpected error in _make_request"
@staticmethod
def _remove_comfy_metadata(model_version: Optional[Dict]) -> None:
@@ -124,6 +175,8 @@ class CivitaiClient:
)
if not success:
message = str(version)
if is_expected_offline_error(message):
return None, OFFLINE_FRIENDLY_MESSAGE
if "not found" in message.lower():
return None, "Model not found"
@@ -164,6 +217,9 @@ class CivitaiClient:
return True
return False
except Exception as e:
if is_expected_offline_error(str(e)):
logger.debug("Preview download skipped due to offline state.")
return False
logger.error(f"Download Error: {str(e)}")
return False
@@ -189,6 +245,29 @@ class CivitaiClient:
return _from_value(payload)
@staticmethod
def _is_transient_server_error(message: str) -> bool:
"""Return True when the message indicates a transient upstream failure.
Recognises Cloudflare 524, generic 5xx, and connectivity-level flakiness
that should not be treated as a permanent failure.
"""
normalized = message.lower()
if "status 5" in normalized or "status 524" in normalized:
return True
if any(
keyword in normalized
for keyword in (
"connection refused",
"connection reset",
"temporary failure",
"name resolution",
"connection closed",
)
):
return True
return False
async def get_model_versions(self, model_id: str) -> Optional[Dict]:
"""Get all versions of a model with local availability info"""
try:
@@ -207,7 +286,17 @@ class CivitaiClient:
message = self._extract_error_message(result)
if message and "not found" in message.lower():
raise ResourceNotFoundError(f"Resource not found for model {model_id}")
if is_expected_offline_error(message):
logger.info("Civitai request skipped: %s", OFFLINE_FRIENDLY_MESSAGE)
return None
if message:
if self._is_transient_server_error(message):
logger.info(
"Transient server error for model %s: %s",
model_id,
message,
)
return None
raise RuntimeError(message)
return None
except RateLimitError:
@@ -242,7 +331,7 @@ class CivitaiClient:
"GET",
f"{self.base_url}/models",
use_auth=True,
params={"ids": query},
params={"ids": query, "nsfw": "true"},
)
if not success:
return None
@@ -321,6 +410,25 @@ class CivitaiClient:
return None
target_version = self._select_target_version(model_data, model_id, version_id)
# If modelVersions is empty (e.g. CivitAI cache lag for newly published
# models) but a specific version_id is known, fall back to fetching the
# version directly via the individual model-versions endpoint, then
# enrich it with the model-level data we already have.
if target_version is None and version_id is not None:
logger.info(
"modelVersions empty for model %s; falling back to direct "
"version lookup for %s",
model_id,
version_id,
)
version = await self._fetch_version_by_id(version_id)
if version:
self._enrich_version_with_model_data(version, model_data)
self._remove_comfy_metadata(version)
return version
return None
if target_version is None:
return None
@@ -357,6 +465,8 @@ class CivitaiClient:
)
if success:
return data
if is_expected_offline_error(data):
return None
logger.warning(f"Failed to fetch model data for model {model_id}")
return None
@@ -371,6 +481,8 @@ class CivitaiClient:
)
if success:
return version
if is_expected_offline_error(version):
return None
logger.warning(f"Failed to fetch version by id {version_id}")
return None
@@ -386,6 +498,8 @@ class CivitaiClient:
)
if success:
return version
if is_expected_offline_error(version):
return None
logger.warning(f"Failed to fetch version by hash {model_hash}")
return None
@@ -461,6 +575,14 @@ class CivitaiClient:
- The model version data or None if not found
- An error message if there was an error, or None on success
"""
# In-memory cache avoids redundant API calls within the same
# import/scan flow (e.g. _resolve_base_model_from_checkpoint
# followed by _resolve_and_populate_checkpoint with the same id).
if version_id in self._version_info_cache:
logger.debug("Cache hit for model version info: %s", version_id)
self._version_info_cache.move_to_end(version_id) # LRU bump
return self._version_info_cache[version_id]
try:
url = f"{self.base_url}/model-versions/{version_id}"
@@ -470,9 +592,16 @@ class CivitaiClient:
if success:
logger.debug("Successfully fetched model version info for: %s", version_id)
self._remove_comfy_metadata(result)
self._version_info_cache[version_id] = (result, None)
self._version_info_cache.move_to_end(version_id)
# Evict oldest entry when over capacity
if len(self._version_info_cache) > self._MAX_CACHE_ENTRIES:
self._version_info_cache.popitem(last=False)
return result, None
# Handle specific error cases
if is_expected_offline_error(result):
return None, OFFLINE_FRIENDLY_MESSAGE
if "not found" in str(result):
error_msg = f"Model not found"
logger.warning(f"Model version not found: {version_id} - {error_msg}")
@@ -507,6 +636,15 @@ class CivitaiClient:
success, result = await self._make_request("GET", url, use_auth=True)
if not success:
if is_expected_offline_error(result):
return None
if self._is_transient_server_error(str(result)):
logger.info(
"Transient server error fetching image info for ID %s: %s",
image_id,
result,
)
return None
logger.error(
"Failed to fetch image info for ID %s from civitai.red: %s",
image_id,
@@ -552,6 +690,59 @@ class CivitaiClient:
logger.error(error_msg)
return None
async def get_model_versions_by_hashes(
self, hashes: List[str]
) -> Optional[List[Dict]]:
"""Fetch full version details for up to 100 SHA256 hashes via the batch endpoint.
Uses POST /api/v1/model-versions/by-hash which returns full version
details including ``usageControl`` and ``earlyAccessEndsAt`` that are
not available from the model-level API.
Args:
hashes: List of SHA256 hashes (max 100 per batch; auto-split).
Returns:
List of version dicts or None on failure.
"""
if not hashes:
return []
BATCH_SIZE = 100
all_versions: List[Dict] = []
for start in range(0, len(hashes), BATCH_SIZE):
batch = hashes[start : start + BATCH_SIZE]
try:
success, result = await self._make_request(
"POST",
f"{self.base_url}/model-versions/by-hash",
use_auth=True,
json=batch,
)
if not success:
logger.warning(
"Batch by-hash request failed for %d hashes: %s",
len(batch),
result,
)
continue
if isinstance(result, list):
all_versions.extend(result)
else:
logger.debug(
"Unexpected by-hash response type: %s", type(result)
)
except RateLimitError:
raise
except Exception as exc: # pragma: no cover - defensive logging
logger.error(
"Error fetching model versions by hashes: %s", exc
)
return all_versions if all_versions else None
async def get_user_models(self, username: str) -> Optional[List[Dict]]:
"""Fetch all models for a specific Civitai user."""
if not username:
@@ -562,10 +753,13 @@ class CivitaiClient:
"GET",
f"{self.base_url}/models",
use_auth=True,
params={"username": username},
params={"username": username, "nsfw": "true"},
)
if not success:
if is_expected_offline_error(result):
logger.info("User model fetch skipped: %s", OFFLINE_FRIENDLY_MESSAGE)
return None
logger.error("Failed to fetch models for %s: %s", username, result)
return None
+204
View File
@@ -0,0 +1,204 @@
"""In-memory connectivity guard to suppress repeated network retries when offline."""
from __future__ import annotations
import asyncio
import errno
import logging
import socket
from dataclasses import dataclass
from datetime import datetime, timedelta
from typing import Any
import aiohttp
logger = logging.getLogger(__name__)
OFFLINE_COOLDOWN_ERROR = "offline_cooldown"
OFFLINE_FRIENDLY_MESSAGE = "Network offline, will retry automatically later"
def is_offline_cooldown_error(value: Any) -> bool:
"""Return True when a response payload represents guard short-circuit."""
return isinstance(value, str) and value == OFFLINE_COOLDOWN_ERROR
def is_expected_offline_error(value: Any) -> bool:
"""Return True when payload is an expected offline-related result."""
if is_offline_cooldown_error(value):
return True
if not isinstance(value, str):
return False
normalized = value.lower()
return "network offline" in normalized or "offline" in normalized
class ConnectivityGuard:
"""Tracks network failures and gates outbound requests during cooldown."""
_instance: "ConnectivityGuard | None" = None
_instance_lock = asyncio.Lock()
@classmethod
async def get_instance(cls) -> "ConnectivityGuard":
async with cls._instance_lock:
if cls._instance is None:
cls._instance = cls()
return cls._instance
def __init__(self) -> None:
if hasattr(self, "_initialized"):
return
self._initialized = True
self._default_destination = "__global__"
self._destination_states: dict[str, _DestinationState] = {
self._default_destination: _DestinationState()
}
self.base_backoff_seconds = 30
self.max_backoff_seconds = 300
self.failure_threshold = 3
@property
def online(self) -> bool:
return self._state_for_destination(None).online
@online.setter
def online(self, value: bool) -> None:
self._state_for_destination(None).online = value
@property
def failure_count(self) -> int:
return self._state_for_destination(None).failure_count
@failure_count.setter
def failure_count(self, value: int) -> None:
self._state_for_destination(None).failure_count = value
@property
def cooldown_until(self) -> datetime | None:
return self._state_for_destination(None).cooldown_until
@cooldown_until.setter
def cooldown_until(self, value: datetime | None) -> None:
self._state_for_destination(None).cooldown_until = value
def _now(self) -> datetime:
return datetime.now()
def _normalize_destination(self, destination: str | None) -> str:
if destination is None or not destination.strip():
return self._default_destination
return destination.lower().strip()
def _state_for_destination(self, destination: str | None) -> "_DestinationState":
destination_key = self._normalize_destination(destination)
if destination_key not in self._destination_states:
self._destination_states[destination_key] = _DestinationState()
return self._destination_states[destination_key]
def in_cooldown(self, destination: str | None = None) -> bool:
state = self._state_for_destination(destination)
if state.cooldown_until is None:
return False
return self._now() < state.cooldown_until
def cooldown_remaining_seconds(self, destination: str | None = None) -> float:
state = self._state_for_destination(destination)
if state.cooldown_until is None:
return 0.0
return max(0.0, (state.cooldown_until - self._now()).total_seconds())
def should_block_request(self, destination: str | None = None) -> bool:
return self.in_cooldown(destination)
def register_success(self, destination: str | None = None) -> None:
destination_key = self._normalize_destination(destination)
state = self._state_for_destination(destination_key)
was_offline = (not state.online) or state.cooldown_until is not None
state.online = True
state.failure_count = 0
state.cooldown_until = None
if was_offline:
logger.info(
"Connectivity restored for destination '%s'; requests resumed.",
destination_key,
)
def register_network_failure(
self, exc: Exception, destination: str | None = None
) -> None:
destination_key = self._normalize_destination(destination)
state = self._state_for_destination(destination_key)
state.online = False
state.failure_count += 1
if state.failure_count < self.failure_threshold:
logger.debug(
"Network failure tracked for destination '%s' (%d/%d): %s",
destination_key,
state.failure_count,
self.failure_threshold,
exc,
)
return
retry_step = state.failure_count - self.failure_threshold
backoff = min(
self.max_backoff_seconds,
self.base_backoff_seconds * (2**retry_step),
)
should_log_warning = not self.in_cooldown(destination_key)
state.cooldown_until = self._now() + timedelta(seconds=backoff)
if should_log_warning:
logger.warning(
"Connectivity offline for destination '%s'; enter cooldown for %ss after %d network failures.",
destination_key,
int(backoff),
state.failure_count,
)
else:
logger.debug(
"Cooldown still active for destination '%s'; failure_count=%d, backoff=%ss.",
destination_key,
state.failure_count,
int(backoff),
)
@staticmethod
def is_network_unreachable_error(exc: Exception) -> bool:
"""Return whether the exception should count as connectivity failure."""
if isinstance(exc, asyncio.CancelledError):
return False
if isinstance(
exc,
(
asyncio.TimeoutError,
TimeoutError,
ConnectionRefusedError,
socket.gaierror,
aiohttp.ServerTimeoutError,
aiohttp.ConnectionTimeoutError,
aiohttp.ClientConnectorError,
aiohttp.ClientConnectionError,
),
):
return True
if isinstance(exc, OSError) and exc.errno in {
errno.ENETUNREACH,
errno.EHOSTUNREACH,
errno.ETIMEDOUT,
errno.ECONNREFUSED,
}:
return True
return False
@dataclass
class _DestinationState:
online: bool = True
failure_count: int = 0
cooldown_until: datetime | None = None
+17
View File
@@ -110,6 +110,23 @@ class DownloadCoordinator:
return result
async def skip_download(self, download_id: str) -> Dict[str, Any]:
"""Skip a download while preserving all partial files on disk."""
download_manager = await self._download_manager_factory()
result = await download_manager.skip_download(download_id)
await self._ws_manager.broadcast_download_progress(
download_id,
{
"status": "skipped",
"progress": 0,
"download_id": download_id,
"message": "Download skipped by user (partial files preserved)",
},
)
return result
async def pause_download(self, download_id: str) -> Dict[str, Any]:
"""Pause an active download and notify listeners."""
File diff suppressed because it is too large Load Diff
+871
View File
@@ -0,0 +1,871 @@
from __future__ import annotations
import asyncio
import json
import logging
import os
import sqlite3
import time
from typing import Any, Optional
from ..utils.cache_paths import get_cache_base_dir
logger = logging.getLogger(__name__)
def _resolve_database_path() -> str:
base_dir = get_cache_base_dir(create=True)
history_dir = os.path.join(base_dir, "download_history")
os.makedirs(history_dir, exist_ok=True)
return os.path.join(history_dir, "download_queue.sqlite")
class DownloadQueueService:
"""Persistent download queue and history manager backed by SQLite.
Provides a singleton interface for managing a download queue and
corresponding history table, both stored in a single SQLite database
under the cache directory.
"""
_instance: Optional[DownloadQueueService] = None
_class_lock: asyncio.Lock = asyncio.Lock()
_SCHEMA = """
CREATE TABLE IF NOT EXISTS download_queue (
download_id TEXT PRIMARY KEY,
model_id INTEGER,
model_version_id INTEGER,
model_name TEXT NOT NULL DEFAULT '',
version_name TEXT DEFAULT '',
thumbnail_url TEXT DEFAULT '',
source TEXT,
file_params TEXT,
status TEXT NOT NULL DEFAULT 'queued',
priority INTEGER DEFAULT 0,
progress INTEGER DEFAULT 0,
bytes_downloaded INTEGER DEFAULT 0,
total_bytes INTEGER,
bytes_per_second REAL DEFAULT 0.0,
error TEXT,
file_path TEXT,
added_at REAL NOT NULL,
started_at REAL,
completed_at REAL
);
CREATE INDEX IF NOT EXISTS idx_dq_status ON download_queue(status);
CREATE INDEX IF NOT EXISTS idx_dq_added ON download_queue(added_at);
CREATE TABLE IF NOT EXISTS download_history (
id INTEGER PRIMARY KEY AUTOINCREMENT,
download_id TEXT,
model_id INTEGER,
model_version_id INTEGER,
model_name TEXT NOT NULL DEFAULT '',
version_name TEXT DEFAULT '',
thumbnail_url TEXT DEFAULT '',
status TEXT NOT NULL,
error TEXT,
file_path TEXT,
bytes_downloaded INTEGER DEFAULT 0,
total_bytes INTEGER,
completed_at REAL NOT NULL,
is_already_exists INTEGER DEFAULT 0
);
CREATE INDEX IF NOT EXISTS idx_dh_completed ON download_history(completed_at DESC);
CREATE INDEX IF NOT EXISTS idx_dh_status ON download_history(status);
"""
@classmethod
async def get_instance(cls) -> DownloadQueueService:
"""Return the singleton instance, creating it if necessary."""
async with cls._class_lock:
if cls._instance is None:
cls._instance = cls()
await cls._instance.deduplicate()
return cls._instance
def __init__(self, db_path: Optional[str] = None) -> None:
self._db_path = db_path or _resolve_database_path()
self._lock = asyncio.Lock()
self._conn: Optional[sqlite3.Connection] = None
self._schema_initialized = False
self._ensure_directory()
self._initialize_schema()
def _ensure_directory(self) -> None:
directory = os.path.dirname(self._db_path)
if directory:
os.makedirs(directory, exist_ok=True)
def _connect(self) -> sqlite3.Connection:
conn = sqlite3.connect(self._db_path, check_same_thread=False)
conn.row_factory = sqlite3.Row
return conn
def _get_conn(self) -> sqlite3.Connection:
if self._conn is None:
self._conn = sqlite3.connect(self._db_path, check_same_thread=False)
self._conn.row_factory = sqlite3.Row
return self._conn
def _initialize_schema(self) -> None:
if self._schema_initialized:
return
with self._connect() as conn:
conn.executescript(self._SCHEMA)
conn.commit()
self._schema_initialized = True
def get_database_path(self) -> str:
"""Return the resolved database file path."""
return self._db_path
def close(self) -> None:
"""Close the persistent SQLite connection, if open.
This is called before plugin update operations to release the
database file lock on Windows, allowing ``shutil.rmtree()`` to
succeed when the cache resides inside the plugin directory.
"""
if self._conn is not None:
try:
self._conn.close()
except Exception:
pass
finally:
self._conn = None
# ------------------------------------------------------------------
# Queue methods
# ------------------------------------------------------------------
async def add_to_queue(
self,
download_id: str,
model_id: Optional[int] = None,
model_version_id: Optional[int] = None,
model_name: str = "",
version_name: str = "",
thumbnail_url: str = "",
source: Optional[str] = None,
file_params: Optional[dict[str, Any]] = None,
) -> dict[str, Any]:
"""Insert a new download into the queue.
Returns the inserted row as a dict (or an empty dict if the
download_id already exists).
"""
now = time.time()
file_params_json = json.dumps(file_params) if file_params is not None else None
async with self._lock:
conn = self._get_conn()
conn.execute(
"""
INSERT OR IGNORE INTO download_queue (
download_id, model_id, model_version_id, model_name,
version_name, thumbnail_url, source, file_params,
status, priority, added_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'queued', 0, ?)
""",
(
download_id,
model_id,
model_version_id,
model_name,
version_name,
thumbnail_url,
source,
file_params_json,
now,
),
)
conn.commit()
row = conn.execute(
"SELECT * FROM download_queue WHERE download_id = ?",
(download_id,),
).fetchone()
return dict(row) if row else {}
async def get_queue(self) -> list[dict[str, Any]]:
"""Return all items in the queue ordered by priority then added time."""
async with self._lock:
conn = self._get_conn()
rows = conn.execute(
"SELECT * FROM download_queue ORDER BY priority DESC, added_at ASC"
).fetchall()
return [dict(row) for row in rows]
async def get_queued_count(self) -> int:
"""Return the number of items with status ``'queued'``."""
async with self._lock:
conn = self._get_conn()
row = conn.execute(
"SELECT COUNT(*) AS cnt FROM download_queue WHERE status = 'queued'"
).fetchone()
return row["cnt"] if row else 0
async def update_status(
self,
download_id: str,
status: str,
**extra: Any,
) -> bool:
"""Update the status and/or extra fields of a queue item.
Accepted extra keyword arguments:
``progress``, ``error``, ``file_path``, ``bytes_downloaded``,
``total_bytes``, ``bytes_per_second``.
Returns ``True`` if a row was updated.
"""
allowed_extra = {
"progress",
"error",
"file_path",
"bytes_downloaded",
"total_bytes",
"bytes_per_second",
}
set_clauses: list[str] = ["status = ?"]
params: list[Any] = [status]
now = time.time()
if status in ("downloading",):
set_clauses.append("started_at = COALESCE(started_at, ?)")
params.append(now)
if status in ("completed", "failed", "canceled"):
set_clauses.append("completed_at = ?")
params.append(now)
for key, value in extra.items():
if key in allowed_extra:
set_clauses.append(f"{key} = ?")
params.append(value)
params.append(download_id)
async with self._lock:
conn = self._get_conn()
cursor = conn.execute(
f"UPDATE download_queue SET {', '.join(set_clauses)} "
"WHERE download_id = ?",
params,
)
conn.commit()
return cursor.rowcount > 0
async def remove_from_queue(self, download_id: str) -> bool:
"""Remove a single item from the queue by download_id.
Returns ``True`` if a row was deleted.
"""
async with self._lock:
conn = self._get_conn()
cursor = conn.execute(
"DELETE FROM download_queue WHERE download_id = ?",
(download_id,),
)
conn.commit()
return cursor.rowcount > 0
async def move_to_top(self, download_id: str) -> bool:
"""Move an item to the front of the queue (highest priority).
Returns ``True`` if the item was found and updated.
"""
async with self._lock:
conn = self._get_conn()
row = conn.execute(
"SELECT priority FROM download_queue WHERE download_id = ?",
(download_id,),
).fetchone()
if row is None:
return False
max_row = conn.execute(
"SELECT MAX(priority) AS mx FROM download_queue"
).fetchone()
max_priority: int = max_row["mx"] if max_row["mx"] is not None else 0
conn.execute(
"UPDATE download_queue SET priority = ? WHERE download_id = ?",
(max_priority + 1, download_id),
)
conn.commit()
return True
async def move_to_end(self, download_id: str) -> bool:
"""Move an item to the end of the queue (lowest priority).
Returns ``True`` if the item was found and updated.
"""
async with self._lock:
conn = self._get_conn()
row = conn.execute(
"SELECT priority FROM download_queue WHERE download_id = ?",
(download_id,),
).fetchone()
if row is None:
return False
min_row = conn.execute(
"SELECT MIN(priority) AS mn FROM download_queue"
).fetchone()
min_priority: int = min_row["mn"] if min_row["mn"] is not None else 0
conn.execute(
"UPDATE download_queue SET priority = ? WHERE download_id = ?",
(min_priority - 1, download_id),
)
conn.commit()
return True
async def clear_queue(self, status_filter: Optional[str] = None) -> int:
"""Remove items from the queue.
When *status_filter* is provided only items with that status are
deleted. Returns the number of deleted rows.
"""
async with self._lock:
conn = self._get_conn()
if status_filter is not None:
cursor = conn.execute(
"DELETE FROM download_queue WHERE status = ?",
(status_filter,),
)
else:
cursor = conn.execute("DELETE FROM download_queue")
conn.commit()
return cursor.rowcount
async def complete_download(
self,
download_id: str,
status: str = "completed",
error: Optional[str] = None,
file_path: Optional[str] = None,
bytes_downloaded: int = 0,
total_bytes: Optional[int] = None,
completed_at: Optional[float] = None,
) -> Optional[dict[str, Any]]:
"""Atomically move a download from the queue into the history table.
Looks up the queue record by ``download_id``, deletes it from the
queue, and inserts a corresponding history entry with the given
terminal status (``completed``, ``failed``, or ``canceled``).
When *completed_at* is provided it is used as the completion
timestamp; otherwise ``time.time()`` is used.
Returns the original queue record (before deletion) on success,
or ``None`` if the download was not found in the queue.
"""
async with self._lock:
conn = self._get_conn()
row = conn.execute(
"SELECT * FROM download_queue WHERE download_id = ?",
(download_id,),
).fetchone()
if row is None:
return None
now = completed_at if completed_at is not None else time.time()
conn.execute(
"DELETE FROM download_queue WHERE download_id = ?",
(download_id,),
)
conn.execute(
"""
INSERT INTO download_history (
download_id, model_id, model_version_id, model_name,
version_name, thumbnail_url, status, error, file_path,
bytes_downloaded, total_bytes, completed_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
row["download_id"],
row["model_id"],
row["model_version_id"],
row["model_name"],
row["version_name"],
row["thumbnail_url"],
status,
error,
file_path,
bytes_downloaded,
total_bytes,
now,
),
)
conn.commit()
return dict(row)
async def pop_next_download(self) -> Optional[dict[str, Any]]:
"""Atomically fetch and mark the next queued item as ``downloading``.
The item with the highest priority (and earliest ``added_at``
among ties) whose status is ``'queued'`` is selected, set to
``'downloading'``, and returned as a dict. Returns ``None`` if
the queue is empty.
"""
async with self._lock:
conn = self._get_conn()
row = conn.execute(
"""
SELECT * FROM download_queue
WHERE status = 'queued'
ORDER BY priority DESC, added_at ASC
LIMIT 1
"""
).fetchone()
if row is None:
return None
download_id = row["download_id"]
now = time.time()
conn.execute(
"UPDATE download_queue SET status = 'downloading', "
"started_at = COALESCE(started_at, ?) "
"WHERE download_id = ?",
(now, download_id),
)
conn.commit()
updated = conn.execute(
"SELECT * FROM download_queue WHERE download_id = ?",
(download_id,),
).fetchone()
return dict(updated) if updated else None
# ------------------------------------------------------------------
# History methods
# ------------------------------------------------------------------
async def add_to_history(
self,
download_id: Optional[str] = None,
model_id: Optional[int] = None,
model_version_id: Optional[int] = None,
model_name: str = "",
version_name: str = "",
thumbnail_url: str = "",
status: str = "completed",
error: Optional[str] = None,
file_path: Optional[str] = None,
bytes_downloaded: int = 0,
total_bytes: Optional[int] = None,
is_already_exists: int = 0,
) -> int:
"""Insert a record into the download history.
Returns the ``id`` (AUTOINCREMENT primary key) of the newly
inserted row.
"""
now = time.time()
async with self._lock:
conn = self._get_conn()
cursor = conn.execute(
"""
INSERT INTO download_history (
download_id, model_id, model_version_id, model_name,
version_name, thumbnail_url, status, error, file_path,
bytes_downloaded, total_bytes, completed_at, is_already_exists
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
download_id,
model_id,
model_version_id,
model_name,
version_name,
thumbnail_url,
status,
error,
file_path,
bytes_downloaded,
total_bytes,
now,
is_already_exists,
),
)
conn.commit()
return cursor.lastrowid or 0
async def get_history(
self,
limit: int = 50,
offset: int = 0,
status_filter: Optional[str] = None,
) -> dict[str, Any]:
"""Return a page of download history entries.
Returns a dict with keys ``items``, ``total``, ``limit``, and
``offset``.
"""
async with self._lock:
conn = self._get_conn()
if status_filter is not None:
count_row = conn.execute(
"SELECT COUNT(*) AS cnt FROM download_history WHERE status = ?",
(status_filter,),
).fetchone()
rows = conn.execute(
"SELECT * FROM download_history WHERE status = ? "
"ORDER BY completed_at DESC LIMIT ? OFFSET ?",
(status_filter, limit, offset),
).fetchall()
else:
count_row = conn.execute(
"SELECT COUNT(*) AS cnt FROM download_history"
).fetchone()
rows = conn.execute(
"SELECT * FROM download_history "
"ORDER BY completed_at DESC LIMIT ? OFFSET ?",
(limit, offset),
).fetchall()
return {
"items": [dict(row) for row in rows],
"total": count_row["cnt"] if count_row else 0,
"limit": limit,
"offset": offset,
}
async def delete_history_item(self, id: int) -> bool:
"""Delete a single history entry by its *id*.
Returns ``True`` if a row was deleted.
"""
async with self._lock:
conn = self._get_conn()
cursor = conn.execute(
"DELETE FROM download_history WHERE id = ?",
(id,),
)
conn.commit()
return cursor.rowcount > 0
async def clear_history(
self,
status_filter: Optional[str] = None,
before_timestamp: Optional[float] = None,
) -> int:
"""Remove history entries matching the optional filters.
Both ``status_filter`` and ``before_timestamp`` can be combined
(AND logic). Returns the number of deleted rows.
"""
async with self._lock:
conn = self._get_conn()
clauses: list[str] = []
params: list[Any] = []
if status_filter is not None:
clauses.append("status = ?")
params.append(status_filter)
if before_timestamp is not None:
clauses.append("completed_at < ?")
params.append(before_timestamp)
where = ""
if clauses:
where = " WHERE " + " AND ".join(clauses)
cursor = conn.execute(
f"DELETE FROM download_history{where}",
params,
)
conn.commit()
return cursor.rowcount
async def get_history_count(self, status_filter: Optional[str] = None) -> int:
"""Return the number of history entries, optionally filtered by status."""
async with self._lock:
conn = self._get_conn()
if status_filter is not None:
row = conn.execute(
"SELECT COUNT(*) AS cnt FROM download_history WHERE status = ?",
(status_filter,),
).fetchone()
else:
row = conn.execute(
"SELECT COUNT(*) AS cnt FROM download_history"
).fetchone()
return row["cnt"] if row else 0
# ------------------------------------------------------------------
# Retry
# ------------------------------------------------------------------
async def retry_from_history(self, item_id: int) -> Optional[dict[str, Any]]:
"""Re-queue a failed or canceled download from history.
Looks up the history record by its primary key. If the status is
``failed`` or ``canceled`` a new queue entry is created with the
same model metadata and a fresh download id, and the original
history entry is **deleted** to prevent exponential growth when
the retried item is later canceled or fails again and re-retried.
"""
async with self._lock:
conn = self._get_conn()
row = conn.execute(
"SELECT * FROM download_history WHERE id = ?",
(item_id,),
).fetchone()
if row is None:
return None
status = str(row["status"])
if status not in ("failed", "canceled"):
return None
import uuid
new_id = str(uuid.uuid4())
now = time.time()
conn.execute(
"""
INSERT INTO download_queue (
download_id, model_id, model_version_id, model_name,
version_name, thumbnail_url, source, file_params,
status, priority, added_at
) VALUES (?, ?, ?, ?, ?, ?, ?, NULL, 'queued', 0, ?)
""",
(
new_id,
row["model_id"],
row["model_version_id"],
row["model_name"],
row["version_name"],
row["thumbnail_url"],
"retry",
now,
),
)
conn.execute(
"DELETE FROM download_history WHERE id = ?",
(item_id,),
)
conn.commit()
queued = conn.execute(
"SELECT * FROM download_queue WHERE download_id = ?",
(new_id,),
).fetchone()
return dict(queued) if queued else None
async def retry_all_failed(self) -> int:
"""Re-queue all failed and canceled downloads from history.
Each history entry is **deleted** after being re-queued so that
repeated retry-all calls do not cause exponential growth.
Returns the number of items that were re-queued.
"""
async with self._lock:
conn = self._get_conn()
rows = conn.execute(
"SELECT * FROM download_history WHERE status IN ('failed', 'canceled')"
).fetchall()
if not rows:
return 0
import uuid
now = time.time()
count = 0
for row in rows:
new_id = str(uuid.uuid4())
conn.execute(
"""
INSERT INTO download_queue (
download_id, model_id, model_version_id, model_name,
version_name, thumbnail_url, source, file_params,
status, priority, added_at
) VALUES (?, ?, ?, ?, ?, ?, ?, NULL, 'queued', 0, ?)
""",
(
new_id,
row["model_id"],
row["model_version_id"],
row["model_name"],
row["version_name"],
row["thumbnail_url"],
"retry",
now,
),
)
conn.execute(
"DELETE FROM download_history WHERE id = ?",
(row["id"],),
)
count += 1
conn.commit()
return count
# ------------------------------------------------------------------
# Stats
# ------------------------------------------------------------------
async def get_stats(self) -> dict[str, int]:
"""Return aggregate counts across both tables.
Returns a dict with keys ``queued``, ``downloading``, ``paused``
(all from the queue table) and ``completed``, ``failed``,
``canceled`` (all from the history table).
"""
async with self._lock:
conn = self._get_conn()
queue_rows = conn.execute(
"SELECT status, COUNT(*) AS cnt FROM download_queue GROUP BY status"
).fetchall()
queue_stats: dict[str, int] = {}
for row in queue_rows:
queue_stats[str(row["status"])] = row["cnt"]
history_rows = conn.execute(
"SELECT status, COUNT(*) AS cnt FROM download_history GROUP BY status"
).fetchall()
history_stats: dict[str, int] = {}
for row in history_rows:
history_stats[str(row["status"])] = row["cnt"]
return {
"queued": queue_stats.get("queued", 0),
"downloading": queue_stats.get("downloading", 0),
"paused": queue_stats.get("paused", 0),
"completed": history_stats.get("completed", 0),
"failed": history_stats.get("failed", 0),
"canceled": history_stats.get("canceled", 0),
}
# ------------------------------------------------------------------
# Deduplication (one-time cleanup for bug #980)
# ------------------------------------------------------------------
async def deduplicate(self) -> dict[str, int]:
"""Remove duplicate entries caused by the retry-amplification bug.
The bug (issue #980) caused the same download to appear N times in
both the queue and history tables when ``retry_all_failed`` was
called repeatedly without deleting the original history rows.
This method is called **once** when the singleton is first created.
It is idempotent — after the first run there will be no duplicates
to remove, so subsequent calls are a no-op.
Returns a dict with the count of removed rows per table.
"""
result: dict[str, int] = {
"removed_history": 0,
"removed_queue": 0,
"removed_orphan_queue": 0,
}
async with self._lock:
conn = self._get_conn()
# 1. History: for each (model_id, model_version_id, status) triplet
# keep only the row with the highest id (most recently inserted).
conn.execute("""
DELETE FROM download_history
WHERE id NOT IN (
SELECT MAX(id)
FROM download_history
GROUP BY model_id, model_version_id, status
)
""")
result["removed_history"] = conn.execute(
"SELECT changes()"
).fetchone()[0]
# 2. Cross-status dedup: for each (model_id, model_version_id),
# keep only the entry with the highest-priority terminal status.
# Priority: completed (3) > failed (2) > canceled (1).
# This prevents the same model version from having both a
# 'failed' and a 'canceled' entry (or a 'completed' alongside
# either) after the bug-created duplicates are removed.
conn.execute("""
DELETE FROM download_history
WHERE id NOT IN (
SELECT dh.id
FROM download_history dh
INNER JOIN (
SELECT model_id, model_version_id,
MAX(CASE status
WHEN 'completed' THEN 3
WHEN 'failed' THEN 2
WHEN 'canceled' THEN 1
ELSE 0
END) AS best_prio
FROM download_history
GROUP BY model_id, model_version_id
) best
ON dh.model_id = best.model_id
AND dh.model_version_id = best.model_version_id
AND CASE dh.status
WHEN 'completed' THEN 3
WHEN 'failed' THEN 2
WHEN 'canceled' THEN 1
ELSE 0
END = best.best_prio
GROUP BY dh.model_id, dh.model_version_id
HAVING dh.id = MAX(dh.id)
)
""")
result["removed_history"] += conn.execute(
"SELECT changes()"
).fetchone()[0]
# 3. Queue: for each (model_id, model_version_id) keep only the
# row with the latest added_at (most recently enqueued).
conn.execute("""
DELETE FROM download_queue
WHERE rowid NOT IN (
SELECT MAX(rowid)
FROM download_queue
WHERE status IN ('queued', 'downloading', 'paused', 'waiting')
GROUP BY model_id, model_version_id
)
AND status IN ('queued', 'downloading', 'paused', 'waiting')
""")
result["removed_queue"] = conn.execute(
"SELECT changes()"
).fetchone()[0]
# 4. Remove orphaned queue entries — items that were re-queued
# (source='retry') but whose model version already has a
# terminal history entry. These are artifacts of the buggy
# retry cycle that were never cleaned up.
conn.execute("""
DELETE FROM download_queue
WHERE source = 'retry'
AND (model_id, model_version_id) IN (
SELECT model_id, model_version_id
FROM download_history
WHERE status IN ('failed', 'canceled')
)
AND status IN ('queued', 'waiting')
""")
result["removed_orphan_queue"] = conn.execute(
"SELECT changes()"
).fetchone()[0]
conn.commit()
logger.info(
"Deduplicate: removed %s history rows, %s queue rows, "
"%s orphaned queue rows",
result["removed_history"],
result["removed_queue"],
result["removed_orphan_queue"],
)
return result
+120 -98
View File
@@ -64,6 +64,7 @@ class DownloadedVersionHistoryService:
self._db_path = db_path or _resolve_database_path()
self._settings = settings_manager or get_settings_manager()
self._lock = asyncio.Lock()
self._conn: sqlite3.Connection | None = None
self._schema_initialized = False
self._ensure_directory()
self._initialize_schema()
@@ -78,6 +79,12 @@ class DownloadedVersionHistoryService:
conn.row_factory = sqlite3.Row
return conn
def _get_conn(self) -> sqlite3.Connection:
if self._conn is None:
self._conn = sqlite3.connect(self._db_path, check_same_thread=False)
self._conn.row_factory = sqlite3.Row
return self._conn
def _initialize_schema(self) -> None:
if self._schema_initialized:
return
@@ -89,6 +96,21 @@ class DownloadedVersionHistoryService:
def get_database_path(self) -> str:
return self._db_path
def close(self) -> None:
"""Close the persistent SQLite connection, if open.
This is called before plugin update operations to release the
database file lock on Windows, allowing ``shutil.rmtree()`` to
succeed when the cache resides inside the plugin directory.
"""
if self._conn is not None:
try:
self._conn.close()
except Exception:
pass
finally:
self._conn = None
def _get_active_library_name(self) -> str | None:
try:
value = self._settings.get_active_library_name()
@@ -116,33 +138,33 @@ class DownloadedVersionHistoryService:
timestamp = time.time()
async with self._lock:
with self._connect() as conn:
conn.execute(
"""
INSERT INTO downloaded_model_versions (
model_type, version_id, model_id, first_seen_at, last_seen_at,
source, last_file_path, last_library_name, is_deleted_override
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, 0)
ON CONFLICT(model_type, version_id) DO UPDATE SET
model_id = COALESCE(excluded.model_id, downloaded_model_versions.model_id),
last_seen_at = excluded.last_seen_at,
source = excluded.source,
last_file_path = COALESCE(excluded.last_file_path, downloaded_model_versions.last_file_path),
last_library_name = COALESCE(excluded.last_library_name, downloaded_model_versions.last_library_name),
is_deleted_override = 0
""",
(
normalized_type,
normalized_version_id,
normalized_model_id,
timestamp,
timestamp,
source,
file_path,
active_library_name,
),
)
conn.commit()
conn = self._get_conn()
conn.execute(
"""
INSERT INTO downloaded_model_versions (
model_type, version_id, model_id, first_seen_at, last_seen_at,
source, last_file_path, last_library_name, is_deleted_override
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, 0)
ON CONFLICT(model_type, version_id) DO UPDATE SET
model_id = COALESCE(excluded.model_id, downloaded_model_versions.model_id),
last_seen_at = excluded.last_seen_at,
source = excluded.source,
last_file_path = COALESCE(excluded.last_file_path, downloaded_model_versions.last_file_path),
last_library_name = COALESCE(excluded.last_library_name, downloaded_model_versions.last_library_name),
is_deleted_override = 0
""",
(
normalized_type,
normalized_version_id,
normalized_model_id,
timestamp,
timestamp,
source,
file_path,
active_library_name,
),
)
conn.commit()
async def mark_downloaded_bulk(
self,
@@ -180,26 +202,26 @@ class DownloadedVersionHistoryService:
return
async with self._lock:
with self._connect() as conn:
conn.executemany(
"""
INSERT INTO downloaded_model_versions (
model_type, version_id, model_id, first_seen_at, last_seen_at,
source, last_file_path, last_library_name, is_deleted_override
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, 0)
ON CONFLICT(model_type, version_id) DO UPDATE SET
model_id = COALESCE(excluded.model_id, downloaded_model_versions.model_id),
last_seen_at = excluded.last_seen_at,
source = excluded.source,
last_file_path = COALESCE(excluded.last_file_path, downloaded_model_versions.last_file_path),
last_library_name = COALESCE(excluded.last_library_name, downloaded_model_versions.last_library_name),
is_deleted_override = 0
""",
payload,
)
conn.commit()
conn = self._get_conn()
conn.executemany(
"""
INSERT INTO downloaded_model_versions (
model_type, version_id, model_id, first_seen_at, last_seen_at,
source, last_file_path, last_library_name, is_deleted_override
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, 0)
ON CONFLICT(model_type, version_id) DO UPDATE SET
model_id = COALESCE(excluded.model_id, downloaded_model_versions.model_id),
last_seen_at = excluded.last_seen_at,
source = excluded.source,
last_file_path = COALESCE(excluded.last_file_path, downloaded_model_versions.last_file_path),
last_library_name = COALESCE(excluded.last_library_name, downloaded_model_versions.last_library_name),
is_deleted_override = 0
""",
payload,
)
conn.commit()
async def mark_not_downloaded(self, model_type: str, version_id: int) -> None:
async def mark_as_deleted(self, model_type: str, version_id: int) -> None:
normalized_type = _normalize_model_type(model_type)
normalized_version_id = _normalize_int(version_id)
if normalized_type is None or normalized_version_id is None:
@@ -208,28 +230,28 @@ class DownloadedVersionHistoryService:
timestamp = time.time()
async with self._lock:
with self._connect() as conn:
conn.execute(
"""
INSERT INTO downloaded_model_versions (
model_type, version_id, model_id, first_seen_at, last_seen_at,
source, last_file_path, last_library_name, is_deleted_override
) VALUES (?, ?, NULL, ?, ?, 'manual', NULL, ?, 1)
ON CONFLICT(model_type, version_id) DO UPDATE SET
last_seen_at = excluded.last_seen_at,
source = excluded.source,
last_library_name = COALESCE(excluded.last_library_name, downloaded_model_versions.last_library_name),
is_deleted_override = 1
""",
(
normalized_type,
normalized_version_id,
timestamp,
timestamp,
self._get_active_library_name(),
),
)
conn.commit()
conn = self._get_conn()
conn.execute(
"""
INSERT INTO downloaded_model_versions (
model_type, version_id, model_id, first_seen_at, last_seen_at,
source, last_file_path, last_library_name, is_deleted_override
) VALUES (?, ?, NULL, ?, ?, 'manual', NULL, ?, 1)
ON CONFLICT(model_type, version_id) DO UPDATE SET
last_seen_at = excluded.last_seen_at,
source = excluded.source,
last_library_name = COALESCE(excluded.last_library_name, downloaded_model_versions.last_library_name),
is_deleted_override = 1
""",
(
normalized_type,
normalized_version_id,
timestamp,
timestamp,
self._get_active_library_name(),
),
)
conn.commit()
async def has_been_downloaded(self, model_type: str, version_id: int) -> bool:
normalized_type = _normalize_model_type(model_type)
@@ -238,15 +260,15 @@ class DownloadedVersionHistoryService:
return False
async with self._lock:
with self._connect() as conn:
row = conn.execute(
"""
SELECT is_deleted_override
FROM downloaded_model_versions
WHERE model_type = ? AND version_id = ?
""",
(normalized_type, normalized_version_id),
).fetchone()
conn = self._get_conn()
row = conn.execute(
"""
SELECT is_deleted_override
FROM downloaded_model_versions
WHERE model_type = ? AND version_id = ?
""",
(normalized_type, normalized_version_id),
).fetchone()
return bool(row) and not bool(row["is_deleted_override"])
async def get_downloaded_version_ids(
@@ -258,16 +280,16 @@ class DownloadedVersionHistoryService:
return []
async with self._lock:
with self._connect() as conn:
rows = conn.execute(
"""
SELECT version_id
FROM downloaded_model_versions
WHERE model_type = ? AND model_id = ? AND is_deleted_override = 0
ORDER BY version_id ASC
""",
(normalized_type, normalized_model_id),
).fetchall()
conn = self._get_conn()
rows = conn.execute(
"""
SELECT version_id
FROM downloaded_model_versions
WHERE model_type = ? AND model_id = ? AND is_deleted_override = 0
ORDER BY version_id ASC
""",
(normalized_type, normalized_model_id),
).fetchall()
return [int(row["version_id"]) for row in rows]
async def get_downloaded_version_ids_bulk(
@@ -291,17 +313,17 @@ class DownloadedVersionHistoryService:
params: list[object] = [normalized_type, *normalized_model_ids]
async with self._lock:
with self._connect() as conn:
rows = conn.execute(
f"""
SELECT model_id, version_id
FROM downloaded_model_versions
WHERE model_type = ?
AND model_id IN ({placeholders})
AND is_deleted_override = 0
""",
params,
).fetchall()
conn = self._get_conn()
rows = conn.execute(
f"""
SELECT model_id, version_id
FROM downloaded_model_versions
WHERE model_type = ?
AND model_id IN ({placeholders})
AND is_deleted_override = 0
""",
params,
).fetchall()
result: dict[int, set[int]] = {}
for row in rows:
+202 -31
View File
@@ -13,18 +13,39 @@ This module provides a centralized download service with:
import os
import logging
import asyncio
import ssl
import aiohttp
from collections import deque
from dataclasses import dataclass
from datetime import datetime, timedelta
from email.utils import parsedate_to_datetime
from urllib.parse import urlparse
from typing import Optional, Dict, Tuple, Callable, Union, Awaitable
from ..services.settings_manager import get_settings_manager
from .connectivity_guard import (
OFFLINE_COOLDOWN_ERROR,
OFFLINE_FRIENDLY_MESSAGE,
ConnectivityGuard,
)
from .errors import RateLimitError
logger = logging.getLogger(__name__)
def is_ssl_cert_verify_error(exc: BaseException) -> bool:
"""Check if an exception represents an SSL certificate verification failure.
Matches ``ssl.SSLCertVerificationError``, ``aiohttp.ClientConnectorCertificateError``
(which wraps the former), and falls back to the standard OpenSSL error text.
"""
if isinstance(exc, ssl.SSLCertVerificationError):
return True
cert_error = getattr(exc, "certificate_error", None)
if isinstance(cert_error, ssl.SSLCertVerificationError):
return True
return "CERTIFICATE_VERIFY_FAILED" in str(exc)
@dataclass(frozen=True)
class DownloadProgress:
"""Snapshot of a download transfer at a moment in time."""
@@ -138,7 +159,7 @@ class Downloader:
self.chunk_size = (
16 * 1024 * 1024
) # 16MB chunks to balance I/O reduction and memory usage
self.max_retries = 5
self.max_retries = self._resolve_max_retries()
self.base_delay = 2.0 # Base delay for exponential backoff
self.session_timeout = 300 # 5 minutes
self.stall_timeout = self._resolve_stall_timeout()
@@ -192,6 +213,18 @@ class Downloader:
return max(30.0, timeout_value)
def _resolve_max_retries(self) -> int:
"""Determine max retry count from environment while preserving defaults."""
default_retries = 5
raw_value = os.environ.get("COMFYUI_DOWNLOAD_MAX_RETRIES")
try:
retries = int(raw_value)
except (TypeError, ValueError):
retries = default_retries
return max(0, retries)
def _should_refresh_session(self) -> bool:
"""Check if session should be refreshed"""
if self._session is None:
@@ -223,7 +256,9 @@ class Downloader:
self._session = None
# Check for app-level proxy settings
proxy_url = None
proxy_url = None # http(s) proxy, passed via the per-request `proxy=` kwarg
socks_proxy_url = None # SOCKS proxy, handled via aiohttp-socks connector
app_proxy_active = False
settings_manager = get_settings_manager()
if settings_manager.get("proxy_enabled", False):
proxy_host = settings_manager.get("proxy_host", "").strip()
@@ -235,9 +270,19 @@ class Downloader:
if proxy_host and proxy_port:
# Build proxy URL
if proxy_username and proxy_password:
proxy_url = f"{proxy_type}://{proxy_username}:{proxy_password}@{proxy_host}:{proxy_port}"
full_proxy_url = f"{proxy_type}://{proxy_username}:{proxy_password}@{proxy_host}:{proxy_port}"
else:
proxy_url = f"{proxy_type}://{proxy_host}:{proxy_port}"
full_proxy_url = f"{proxy_type}://{proxy_host}:{proxy_port}"
app_proxy_active = True
# aiohttp cannot tunnel SOCKS via the per-request `proxy=` kwarg
# (it would send HTTP to the SOCKS port and fail parsing the
# SOCKS handshake reply). SOCKS must be handled by an
# aiohttp-socks ProxyConnector instead.
if proxy_type.startswith("socks"):
socks_proxy_url = full_proxy_url
else:
proxy_url = full_proxy_url
logger.debug(
f"Using app-level proxy: {proxy_type}://{proxy_host}:{proxy_port}"
@@ -247,14 +292,41 @@ class Downloader:
logger.debug(
"Proxy mode: system-level proxy (trust_env) will be used if configured in environment."
)
# Build SSL context: prefer certifi's CA bundle for broader
# CA coverage across different Python environments (especially
# embedded/compatibility Python builds).
try:
import certifi # type: ignore[import-untyped]
ca_path = certifi.where()
ssl_context = ssl.create_default_context(cafile=ca_path)
logger.debug("SSL: using certifi CA bundle at %s", ca_path)
except (ImportError, FileNotFoundError, ValueError, OSError):
ssl_context = ssl.create_default_context()
logger.debug("SSL: certifi unavailable; using system default CA bundle")
# Optimize TCP connection parameters
connector = aiohttp.TCPConnector(
ssl=True,
connector_kwargs = dict(
ssl=ssl_context,
limit=8, # Concurrent connections
ttl_dns_cache=300, # DNS cache timeout
force_close=False, # Keep connections for reuse
enable_cleanup_closed=True,
)
if socks_proxy_url:
# Route all traffic through the SOCKS proxy via aiohttp-socks. The
# connector tunnels every connection, so no per-request `proxy=` is
# used (and must not be — see self._proxy_url below).
try:
from aiohttp_socks import ProxyConnector
except ImportError as e: # pragma: no cover
raise RuntimeError(
"A SOCKS proxy is configured but the 'aiohttp-socks' package "
"is not installed. Install it with: pip install aiohttp-socks"
) from e
connector = ProxyConnector.from_url(socks_proxy_url, **connector_kwargs)
else:
connector = aiohttp.TCPConnector(**connector_kwargs)
# Configure timeout parameters
timeout = aiohttp.ClientTimeout(
@@ -265,12 +337,14 @@ class Downloader:
self._session = aiohttp.ClientSession(
connector=connector,
trust_env=proxy_url
is None, # Only use system proxy if no app-level proxy is set
# Only fall back to system/env proxy when no app-level proxy is active
trust_env=not app_proxy_active,
timeout=timeout,
)
# Store proxy URL for use in requests
# Store proxy URL for per-request use. Stays None for SOCKS because the
# ProxyConnector already tunnels everything; passing proxy= for SOCKS
# would re-trigger the original aiohttp parse error.
self._proxy_url = proxy_url
self._session_created_at = datetime.now()
@@ -334,6 +408,7 @@ class Downloader:
logger.info(f"Resuming download from offset {resume_offset} bytes")
total_size = 0
range_redirect_retry_urls: set[str] = set()
while retry_count <= self.max_retries:
try:
@@ -372,6 +447,23 @@ class Downloader:
if response.status == 200:
# Full content response
if resume_offset > 0:
redirected_url = str(response.url)
if (
allow_resume
and response.history
and redirected_url
and redirected_url != url
and redirected_url not in range_redirect_retry_urls
):
range_redirect_retry_urls.add(redirected_url)
logger.info(
"Range request was not honored after redirect; retrying final URL directly: %s",
redirected_url,
)
url = redirected_url
response.release()
continue
# Server doesn't support ranges, restart from beginning
logger.warning(
"Server doesn't support range requests, restarting download"
@@ -571,37 +663,53 @@ class Downloader:
expected_size = total_size if total_size > 0 else None
integrity_error: Optional[str] = None
resumable_incomplete = False
if final_size <= 0:
integrity_error = "Downloaded file is empty"
elif expected_size is not None and final_size != expected_size:
integrity_error = f"File size mismatch. Expected: {expected_size}, Got: {final_size}"
resumable_incomplete = (
allow_resume
and part_path != save_path
and final_size > 0
and final_size < expected_size
)
if integrity_error is not None:
logger.error(
log_fn = logger.warning if resumable_incomplete else logger.error
log_fn(
"Download integrity check failed for %s: %s",
save_path,
integrity_error,
)
# Remove the corrupted payload so future attempts start fresh
if os.path.exists(part_path):
try:
os.remove(part_path)
except OSError as remove_error:
logger.warning(
"Failed to delete corrupted download %s: %s",
part_path,
remove_error,
)
if part_path != save_path and os.path.exists(save_path):
try:
os.remove(save_path)
except OSError as remove_error:
logger.warning(
"Failed to delete target file %s after integrity error: %s",
save_path,
remove_error,
)
if resumable_incomplete:
logger.info(
"Preserving incomplete download for resume: %s (%s/%s bytes)",
part_path,
final_size,
expected_size,
)
else:
# Remove corrupted payloads that cannot be safely resumed.
if os.path.exists(part_path):
try:
os.remove(part_path)
except OSError as remove_error:
logger.warning(
"Failed to delete corrupted download %s: %s",
part_path,
remove_error,
)
if part_path != save_path and os.path.exists(save_path):
try:
os.remove(save_path)
except OSError as remove_error:
logger.warning(
"Failed to delete target file %s after integrity error: %s",
save_path,
remove_error,
)
retry_count += 1
if retry_count <= self.max_retries:
@@ -611,8 +719,16 @@ class Downloader:
delay,
)
await asyncio.sleep(delay)
resume_offset = 0
total_size = 0
if resumable_incomplete and os.path.exists(part_path):
resume_offset = os.path.getsize(part_path)
total_size = expected_size or 0
logger.info(
"Will resume incomplete download from byte %s",
resume_offset,
)
else:
resume_offset = 0
total_size = 0
await self._create_session()
continue
@@ -676,6 +792,17 @@ class Downloader:
DownloadRestartRequested,
) as e:
retry_count += 1
if is_ssl_cert_verify_error(e):
logger.error(
"SSL certificate verification failed when connecting to %s. "
"This is usually caused by an outdated CA certificate bundle "
"in the Python environment. Recommended fixes:\n"
" 1. pip install --upgrade certifi\n"
" 2. pip install pip-system-certs",
url,
)
logger.warning(
f"Network error during download (attempt {retry_count}/{self.max_retries + 1}): {e}"
)
@@ -743,6 +870,11 @@ class Downloader:
Returns:
Tuple[bool, Union[bytes, str], Optional[Dict]]: (success, content or error message, response headers if requested)
"""
guard = await ConnectivityGuard.get_instance()
destination = self._guard_destination(url)
if guard.should_block_request(destination):
return False, OFFLINE_FRIENDLY_MESSAGE, None
try:
session = await self.session
# Debug log for proxy mode at request time
@@ -765,6 +897,7 @@ class Downloader:
) as response:
if response.status == 200:
content = await response.read()
guard.register_success(destination)
if return_headers:
return True, content, dict(response.headers)
else:
@@ -783,6 +916,12 @@ class Downloader:
return False, error_msg, None
except Exception as e:
if guard.is_network_unreachable_error(e):
guard.register_network_failure(e, destination)
if guard.should_block_request(destination):
return False, OFFLINE_FRIENDLY_MESSAGE, None
logger.debug("Network unavailable during memory download: %s", e)
return False, str(e), None
logger.error(f"Error downloading to memory from {url}: {e}")
return False, str(e), None
@@ -803,6 +942,11 @@ class Downloader:
Returns:
Tuple[bool, Union[Dict, str]]: (success, headers dict or error message)
"""
guard = await ConnectivityGuard.get_instance()
destination = self._guard_destination(url)
if guard.should_block_request(destination):
return False, OFFLINE_COOLDOWN_ERROR
try:
session = await self.session
# Debug log for proxy mode at request time
@@ -824,11 +968,18 @@ class Downloader:
url, headers=headers, proxy=self.proxy_url
) as response:
if response.status == 200:
guard.register_success(destination)
return True, dict(response.headers)
else:
return False, f"Head request failed with status {response.status}"
except Exception as e:
if guard.is_network_unreachable_error(e):
guard.register_network_failure(e, destination)
if guard.should_block_request(destination):
return False, OFFLINE_COOLDOWN_ERROR
logger.debug("Network unavailable during header probe: %s", e)
return False, str(e)
logger.error(f"Error getting headers from {url}: {e}")
return False, str(e)
@@ -853,6 +1004,11 @@ class Downloader:
Returns:
Tuple[bool, Union[Dict, str]]: (success, response data or error message)
"""
guard = await ConnectivityGuard.get_instance()
destination = self._guard_destination(url)
if guard.should_block_request(destination):
return False, OFFLINE_COOLDOWN_ERROR
try:
session = await self.session
# Debug log for proxy mode at request time
@@ -876,6 +1032,7 @@ class Downloader:
method, url, headers=headers, **kwargs
) as response:
if response.status == 200:
guard.register_success(destination)
# Try to parse as JSON, fall back to text
try:
data = await response.json()
@@ -906,6 +1063,12 @@ class Downloader:
return False, f"Request failed with status {response.status}"
except Exception as e:
if guard.is_network_unreachable_error(e):
guard.register_network_failure(e, destination)
if guard.should_block_request(destination):
return False, OFFLINE_COOLDOWN_ERROR
logger.debug("Network unavailable for %s %s: %s", method, url, e)
return False, str(e)
logger.error(f"Error making {method} request to {url}: {e}")
return False, str(e)
@@ -956,6 +1119,14 @@ class Downloader:
delta = retry_datetime - datetime.now(tz=retry_datetime.tzinfo)
return max(0.0, delta.total_seconds())
@staticmethod
def _guard_destination(url: str) -> str:
"""Build per-destination connectivity guard scope from request URL."""
parsed_url = urlparse(url)
if parsed_url.hostname:
return parsed_url.hostname.lower()
return "unknown"
# Global instance accessor
async def get_downloader() -> Downloader:
+30 -9
View File
@@ -1,8 +1,9 @@
import os
import logging
from typing import Dict
from typing import Dict, Optional
from .base_model_service import BaseModelService
from .auto_tag_service import extract_auto_tags
from ..utils.models import EmbeddingMetadata
from ..config import config
@@ -20,20 +21,37 @@ class EmbeddingService(BaseModelService):
"""
super().__init__("embedding", scanner, EmbeddingMetadata, update_service=update_service)
async def format_response(self, embedding_data: Dict) -> Dict:
"""Format Embedding data for API response"""
async def format_response(self, embedding_data: Dict) -> Optional[Dict]:
"""Format Embedding data for API response.
Returns None when the entry is missing critical fields (corrupted cache
row), so the handler layer can filter it out. See issue #730.
"""
# Guard against corrupted cache entries missing critical fields
file_path = embedding_data.get("file_path")
if not file_path or not isinstance(file_path, str):
logger.warning(
"Skipping corrupted embedding entry (missing file_path): %s",
embedding_data.get("file_name", "<unknown>"),
)
return None
# Get sub_type from cache entry (new canonical field)
sub_type = embedding_data.get("sub_type", "embedding")
file_name = embedding_data.get("file_name") or ""
model_name = embedding_data.get("model_name") or file_name
folder = embedding_data.get("folder") or ""
return {
"model_name": embedding_data["model_name"],
"file_name": embedding_data["file_name"],
"model_name": model_name,
"file_name": file_name,
"preview_url": config.get_preview_static_url(embedding_data.get("preview_url", "")),
"preview_nsfw_level": embedding_data.get("preview_nsfw_level", 0),
"base_model": embedding_data.get("base_model", ""),
"folder": embedding_data["folder"],
"folder": folder,
"sha256": embedding_data.get("sha256", ""),
"file_path": embedding_data["file_path"].replace(os.sep, "/"),
"file_path": file_path.replace(os.sep, "/"),
"file_size": embedding_data.get("size", 0),
"modified": embedding_data.get("modified", ""),
"tags": embedding_data.get("tags", []),
@@ -45,7 +63,10 @@ class EmbeddingService(BaseModelService):
"exclude": bool(embedding_data.get("exclude", False)),
"update_available": bool(embedding_data.get("update_available", False)),
"skip_metadata_refresh": bool(embedding_data.get("skip_metadata_refresh", False)),
"civitai": self.filter_civitai_data(embedding_data.get("civitai", {}), minimal=True)
"civitai": self.filter_civitai_data(embedding_data.get("civitai", {}), minimal=True),
"auto_tags": embedding_data.get("auto_tags") or extract_auto_tags(embedding_data),
"version_count": embedding_data.get("version_count"),
"hf_url": embedding_data.get("hf_url", ""),
}
def find_duplicate_hashes(self) -> Dict:
+51 -10
View File
@@ -5,6 +5,7 @@ from typing import Dict, List, Optional
from .base_model_service import BaseModelService
from .model_query import resolve_sub_type
from .auto_tag_service import extract_auto_tags
from ..utils.models import LoraMetadata
from ..config import config
@@ -23,23 +24,41 @@ class LoraService(BaseModelService):
"""
super().__init__("lora", scanner, LoraMetadata, update_service=update_service)
async def format_response(self, lora_data: Dict) -> Dict:
"""Format LoRA data for API response"""
async def format_response(self, lora_data: Dict) -> Optional[Dict]:
"""Format LoRA data for API response.
Returns None when the entry is missing critical fields (corrupted cache
row), so the handler layer can filter it out instead of crashing the
whole listing request. See issue #730.
"""
# Guard against corrupted cache entries missing critical fields
file_path = lora_data.get("file_path")
if not file_path or not isinstance(file_path, str):
logger.warning(
"Skipping corrupted LoRA entry (missing file_path): %s",
lora_data.get("file_name", "<unknown>"),
)
return None
# Resolve sub_type using priority: sub_type > model_type > civitai.model.type > default
# Normalize to lowercase for consistent API responses
sub_type = resolve_sub_type(lora_data).lower()
file_name = lora_data.get("file_name") or ""
model_name = lora_data.get("model_name") or file_name
folder = lora_data.get("folder") or ""
return {
"model_name": lora_data["model_name"],
"file_name": lora_data["file_name"],
"model_name": model_name,
"file_name": file_name,
"preview_url": config.get_preview_static_url(
lora_data.get("preview_url", "")
),
"preview_nsfw_level": lora_data.get("preview_nsfw_level", 0),
"base_model": lora_data.get("base_model", ""),
"folder": lora_data["folder"],
"folder": folder,
"sha256": lora_data.get("sha256", ""),
"file_path": lora_data["file_path"].replace(os.sep, "/"),
"file_path": file_path.replace(os.sep, "/"),
"file_size": lora_data.get("size", 0),
"modified": lora_data.get("modified", ""),
"tags": lora_data.get("tags", []),
@@ -57,6 +76,9 @@ class LoraService(BaseModelService):
"civitai": self.filter_civitai_data(
lora_data.get("civitai", {}), minimal=True
),
"auto_tags": lora_data.get("auto_tags") or extract_auto_tags(lora_data),
"version_count": lora_data.get("version_count"),
"hf_url": lora_data.get("hf_url", ""),
}
async def _apply_specific_filters(self, data: List[Dict], **kwargs) -> List[Dict]:
@@ -310,8 +332,23 @@ class LoraService(BaseModelService):
"""Return cached raw metadata for a LoRA matching the given filename."""
cache = await self.scanner.get_cached_data(force_refresh=False)
fn_normalized = filename.replace("\\", "/")
fn_no_ext = fn_normalized
for ext in (".safetensors", ".ckpt", ".pt", ".bin"):
if fn_no_ext.lower().endswith(ext):
fn_no_ext = fn_no_ext[: -len(ext)]
break
for lora in cache.raw_data if cache else []:
if lora.get("file_name") == filename:
file_name = lora.get("file_name", "")
folder = lora.get("folder", "")
file_name_no_ext = file_name
for ext in (".safetensors", ".ckpt", ".pt", ".bin"):
if file_name_no_ext.lower().endswith(ext):
file_name_no_ext = file_name_no_ext[: -len(ext)]
break
path_name = f"{folder}/{file_name_no_ext}".replace("\\", "/") if folder else file_name_no_ext
if fn_no_ext in (file_name_no_ext, path_name):
return lora
return None
@@ -399,7 +436,10 @@ class LoraService(BaseModelService):
locked_loras = locked_loras[:target_count]
# Filter out locked LoRAs from available pool
locked_names = {lora["name"] for lora in locked_loras}
locked_names = {
os.path.basename(lora["name"]) if "/" in str(lora.get("name", "")) else lora["name"]
for lora in locked_loras
}
available_pool = [
l for l in available_loras if l["file_name"] not in locked_names
]
@@ -454,7 +494,7 @@ class LoraService(BaseModelService):
result_loras.append(
{
"name": lora["file_name"],
"name": f"{lora['folder']}/{lora['file_name']}" if lora.get("folder") else lora["file_name"],
"strength": model_str,
"clipStrength": clip_str,
"active": True,
@@ -670,8 +710,9 @@ class LoraService(BaseModelService):
# Return minimal data needed for cycling
return [
{
"file_name": lora["file_name"],
"file_name": f"{lora['folder']}/{lora['file_name']}" if lora.get("folder") else lora["file_name"],
"model_name": lora.get("model_name", lora["file_name"]),
"folder": lora.get("folder", ""),
}
for lora in available_loras
]
+44 -6
View File
@@ -11,6 +11,7 @@ from typing import Any, Awaitable, Callable, Dict, Iterable, Optional
from ..services.settings_manager import SettingsManager
from ..utils.civitai_utils import resolve_license_payload
from ..utils.model_utils import determine_base_model
from .connectivity_guard import OFFLINE_FRIENDLY_MESSAGE, is_expected_offline_error
from .errors import RateLimitError
logger = logging.getLogger(__name__)
@@ -215,13 +216,19 @@ class MetadataSyncService:
provider_used: Optional[str] = None
last_error: Optional[str] = None
civitai_api_not_found = False
any_rate_limited = False
for provider_name, provider in provider_attempts:
try:
civitai_metadata_candidate, error = await provider.get_model_by_hash(sha256)
except RateLimitError as exc:
exc.provider = exc.provider or (provider_name or provider.__class__.__name__)
raise
logger.warning(
"Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
provider_name or provider.__class__.__name__,
exc.retry_after or 0,
)
any_rate_limited = True
continue
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Provider %s failed for hash %s: %s", provider_name, sha256, exc)
civitai_metadata_candidate, error = None, str(exc)
@@ -257,6 +264,14 @@ class MetadataSyncService:
model_data["last_checked_at"] = datetime.now().timestamp()
needs_save = True
# When the model was already classified as "not on CivitAI" via
# .metadata.json (civitai_deleted=True) but the SQLite cache is
# stale (because the pre-fix code never persisted these flags),
# ensure the flags are written to the scanner cache + SQLite.
if not needs_save and model_data.get("civitai_deleted") is True:
model_data["last_checked_at"] = datetime.now().timestamp()
needs_save = True
# Save metadata if any state was updated
if needs_save:
data_to_save = model_data.copy()
@@ -265,6 +280,7 @@ class MetadataSyncService:
if "last_checked_at" not in data_to_save:
data_to_save["last_checked_at"] = datetime.now().timestamp()
await self._metadata_manager.save_metadata(file_path, data_to_save)
await update_cache_func(file_path, file_path, data_to_save)
default_error = (
"CivitAI model is deleted and metadata archive DB is not enabled"
@@ -274,11 +290,19 @@ class MetadataSyncService:
else "No provider returned metadata"
)
resolved_error = last_error or default_error
if any_rate_limited and "Rate limited" not in resolved_error:
resolved_error = "Rate limited"
if is_expected_offline_error(resolved_error):
resolved_error = OFFLINE_FRIENDLY_MESSAGE
error_msg = (
f"Error fetching metadata: {last_error or default_error} "
f"(model_name={model_data.get('model_name', '')})"
f"Error fetching metadata: {resolved_error} "
f"(file={os.path.basename(file_path)}, sha256={sha256})"
)
logger.error(error_msg)
# Use case layer (BulkMetadataRefreshUseCase) logs failed models at WARNING level,
# so this level is demoted to DEBUG to avoid duplicate user-visible logging.
logger.debug(error_msg)
return False, error_msg
model_data["from_civitai"] = True
@@ -347,6 +371,9 @@ class MetadataSyncService:
return False, error_msg
except Exception as exc: # pragma: no cover - error path
error_msg = f"Error fetching metadata: {exc}"
if is_expected_offline_error(str(exc)):
logger.info(OFFLINE_FRIENDLY_MESSAGE)
return False, OFFLINE_FRIENDLY_MESSAGE
logger.error(error_msg, exc_info=True)
return False, error_msg
@@ -400,7 +427,18 @@ class MetadataSyncService:
metadata = await metadata_loader(metadata_path)
for key, value in updates.items():
if isinstance(value, dict) and isinstance(metadata.get(key), dict):
if key == "tags" and isinstance(value, list):
# Normalize tags: trim, lowercase, deduplicate
normalized = []
seen = set()
for tag in value:
if isinstance(tag, str):
t = tag.strip().lower()
if t and t not in seen:
normalized.append(t)
seen.add(t)
metadata[key] = normalized
elif isinstance(value, dict) and isinstance(metadata.get(key), dict):
metadata[key].update(value)
else:
metadata[key] = value
+13
View File
@@ -18,6 +18,8 @@ SUPPORTED_SORT_MODES = [
('size', 'desc'),
('usage', 'asc'),
('usage', 'desc'),
('versions_count', 'asc'),
('versions_count', 'desc'),
]
# Is this in use?
@@ -263,6 +265,17 @@ class ModelCache:
),
reverse=reverse
)
elif sort_key == 'versions_count':
# Pre-dedup sort: fall back to name sort.
# Actual re-sort by version_count happens in get_paginated_data after dedup.
result = natsorted(
data,
key=lambda x: (
self._get_display_name(x).lower(),
x.get('file_path', '').lower()
),
reverse=reverse
)
else:
# Fallback: no sort
result = list(data)
+42 -9
View File
@@ -7,6 +7,7 @@ class ModelHashIndex:
def __init__(self):
self._hash_to_path: Dict[str, str] = {}
self._filename_to_hash: Dict[str, str] = {}
self._autov2_to_path: Dict[str, str] = {}
# New data structures for tracking duplicates
self._duplicate_hashes: Dict[str, List[str]] = {} # sha256 -> list of paths
self._duplicate_filenames: Dict[str, List[str]] = {} # filename -> list of paths
@@ -63,6 +64,9 @@ class ModelHashIndex:
# Add new mappings
self._hash_to_path[sha256] = file_path
self._filename_to_hash[filename] = sha256
# AutoV2 = first 10 chars of SHA256
if len(sha256) >= 10:
self._autov2_to_path[sha256[:10]] = file_path
def _get_filename_from_path(self, file_path: str) -> str:
"""Extract filename without extension from path"""
@@ -79,6 +83,12 @@ class ModelHashIndex:
hash_val = h
break
if hash_val is None:
for h, paths in self._duplicate_hashes.items():
if file_path in paths:
hash_val = h
break
# If we didn't find a hash, nothing to do
if not hash_val:
return
@@ -151,7 +161,12 @@ class ModelHashIndex:
del self._duplicate_filenames[filename]
if filename in self._filename_to_hash:
del self._filename_to_hash[filename]
# Remove from AutoV2 index
autov2_keys_to_remove = [k for k, v in self._autov2_to_path.items() if v == file_path]
for k in autov2_keys_to_remove:
del self._autov2_to_path[k]
def remove_by_hash(self, sha256: str) -> None:
"""Remove entry by hash"""
sha256 = sha256.lower()
@@ -171,6 +186,10 @@ class ModelHashIndex:
# Remove hash-to-path mapping
del self._hash_to_path[sha256]
autov2_key = sha256[:10]
if autov2_key in self._autov2_to_path:
del self._autov2_to_path[autov2_key]
# Update filename-to-hash and duplicate filenames for all paths
for path_to_remove in paths_to_remove:
fname = self._get_filename_from_path(path_to_remove)
@@ -189,13 +208,24 @@ class ModelHashIndex:
# If only one entry remains, it's no longer a duplicate
del self._duplicate_filenames[fname]
def has_hash(self, sha256: str) -> bool:
"""Check if hash exists in index"""
return sha256.lower() in self._hash_to_path
def get_path(self, sha256: str) -> Optional[str]:
"""Get file path for a hash"""
return self._hash_to_path.get(sha256.lower())
def has_hash(self, hash_value: str) -> bool:
"""Check if hash exists in index (SHA256 or AutoV2)"""
normalized = hash_value.lower()
if normalized in self._hash_to_path:
return True
if len(normalized) == 10:
return normalized in self._autov2_to_path
return False
def get_path(self, hash_value: str) -> Optional[str]:
"""Get file path for a hash (SHA256 or AutoV2)"""
normalized = hash_value.lower()
path = self._hash_to_path.get(normalized)
if path is not None:
return path
if len(normalized) == 10:
return self._autov2_to_path.get(normalized)
return None
def get_hash(self, file_path: str) -> Optional[str]:
"""Get hash for a file path"""
@@ -203,13 +233,16 @@ class ModelHashIndex:
return self._filename_to_hash.get(filename)
def get_hash_by_filename(self, filename: str) -> Optional[str]:
"""Get hash for a filename without extension"""
"""Get hash for a filename (bare basename or path-prefixed name)"""
if "/" in filename or "\\" in filename:
filename = os.path.splitext(os.path.basename(filename.replace("\\", "/")))[0]
return self._filename_to_hash.get(filename)
def clear(self) -> None:
"""Clear all entries"""
self._hash_to_path.clear()
self._filename_to_hash.clear()
self._autov2_to_path.clear()
self._duplicate_hashes.clear()
self._duplicate_filenames.clear()
+5
View File
@@ -111,6 +111,11 @@ class ModelLifecycleService:
self._scanner._hash_index.remove_by_path(file_path)
await self._sync_update_for_model(model_id)
persist_current_cache = getattr(self._scanner, "_persist_current_cache", None)
if callable(persist_current_cache):
await persist_current_cache()
return {"success": True, "deleted_files": deleted_files}
@staticmethod
+120 -13
View File
@@ -5,7 +5,7 @@ import logging
import random
from typing import Optional, Dict, Tuple, Any, List, Sequence
from .downloader import get_downloader
from .errors import RateLimitError
from .errors import RateLimitError, ResourceNotFoundError
try:
from bs4 import BeautifulSoup
@@ -65,7 +65,14 @@ class _RateLimitRetryHelper:
return await func(*args, **kwargs)
except RateLimitError as exc:
attempt += 1
if attempt >= self._retry_limit:
# Determine effective retry limit based on rate-limit magnitude
effective_retry_limit = self._retry_limit # default: 3
if exc.retry_after is not None and exc.retry_after >= 120.0:
# Long rate-limit window (>=2 min) — retries are futile
effective_retry_limit = 1 # total 1 attempt = 0 retries
if attempt >= effective_retry_limit:
exc.provider = exc.provider or label
raise
@@ -81,7 +88,11 @@ class _RateLimitRetryHelper:
def _calculate_delay(self, retry_after: Optional[float], attempt: int) -> float:
if retry_after is not None:
return min(self._max_delay, max(0.0, retry_after))
# Cap at 1800s (30 min) as a safety ceiling. The old 30s cap was
# too low — CivArchive can return retry_after ~1500s, causing all
# retries to fail. A generous ceiling protects against pathological
# server values while still respecting the server's guidance.
return min(1800.0, max(0.0, retry_after))
base_delay = self._base_delay * (2 ** max(0, attempt - 1))
jitter_span = base_delay * self._jitter_ratio
@@ -108,6 +119,18 @@ class ModelMetadataProvider(ABC):
) -> Optional[Dict[int, Dict]]:
"""Fetch model versions for multiple model ids when supported."""
raise NotImplementedError
async def get_model_versions_by_hashes(
self, hashes: List[str]
) -> Optional[List[Dict]]:
"""Fetch full version details for multiple SHA256 hashes.
Used specifically to retrieve ``usageControl`` which is only
available from the per-version / by-hash API, not from model-level
responses. Providers that cannot resolve hashes should let the
default ``NotImplementedError`` propagate.
"""
raise NotImplementedError
@abstractmethod
async def get_model_version(self, model_id: int = None, version_id: int = None) -> Optional[Dict]:
@@ -140,6 +163,11 @@ class CivitaiModelMetadataProvider(ModelMetadataProvider):
self, model_ids: Sequence[int]
) -> Optional[Dict[int, Dict]]:
return await self.client.get_model_versions_bulk(model_ids)
async def get_model_versions_by_hashes(
self, hashes: List[str]
) -> Optional[List[Dict]]:
return await self.client.get_model_versions_by_hashes(hashes)
async def get_model_version(self, model_id: int = None, version_id: int = None) -> Optional[Dict]:
return await self.client.get_model_version(model_id, version_id)
@@ -457,14 +485,19 @@ class FallbackMetadataProvider(ModelMetadataProvider):
if result:
return result, error
except RateLimitError as exc:
exc.provider = exc.provider or label
raise exc
logger.warning(
"Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
label,
exc.retry_after or 0,
)
continue
except Exception as e:
logger.debug("Provider %s failed for get_model_by_hash: %s", label, e)
continue
return None, "Model not found"
async def get_model_versions(self, model_id: str) -> Optional[Dict]:
not_found_confirmed = False
for provider, label in self._iter_providers():
try:
result = await self._call_with_rate_limit(
@@ -475,8 +508,20 @@ class FallbackMetadataProvider(ModelMetadataProvider):
if result:
return result
except RateLimitError as exc:
exc.provider = exc.provider or label
raise exc
logger.warning(
"Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
label,
exc.retry_after or 0,
)
continue
except ResourceNotFoundError:
not_found_confirmed = True
logger.debug(
"Provider %s reports model %s as not found",
label,
model_id,
)
continue
except Exception as e:
logger.debug("Provider %s failed for get_model_versions: %s", label, e)
continue
@@ -494,8 +539,12 @@ class FallbackMetadataProvider(ModelMetadataProvider):
if result:
return result
except RateLimitError as exc:
exc.provider = exc.provider or label
raise exc
logger.warning(
"Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
label,
exc.retry_after or 0,
)
continue
except Exception as e:
logger.debug("Provider %s failed for get_model_version: %s", label, e)
continue
@@ -512,13 +561,47 @@ class FallbackMetadataProvider(ModelMetadataProvider):
if result:
return result, error
except RateLimitError as exc:
exc.provider = exc.provider or label
raise exc
logger.warning(
"Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
label,
exc.retry_after or 0,
)
continue
except Exception as e:
logger.debug("Provider %s failed for get_model_version_info: %s", label, e)
continue
return None, "No provider could retrieve the data"
async def get_model_versions_by_hashes(
self, hashes: List[str]
) -> Optional[List[Dict]]:
for provider, label in self._iter_providers():
try:
result = await self._call_with_rate_limit(
label,
provider.get_model_versions_by_hashes,
hashes,
)
if result is not None:
return result
except NotImplementedError:
continue
except RateLimitError as exc:
logger.warning(
"Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
label,
exc.retry_after or 0,
)
continue
except Exception as e:
logger.debug(
"Provider %s failed for get_model_versions_by_hashes: %s",
label,
e,
)
continue
return None
async def get_user_models(self, username: str) -> Optional[List[Dict]]:
for provider, label in self._iter_providers():
try:
@@ -530,8 +613,12 @@ class FallbackMetadataProvider(ModelMetadataProvider):
if result is not None:
return result
except RateLimitError as exc:
exc.provider = exc.provider or label
raise exc
logger.warning(
"Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
label,
exc.retry_after or 0,
)
continue
except Exception as e:
logger.debug("Provider %s failed for get_user_models: %s", label, e)
continue
@@ -593,6 +680,15 @@ class RateLimitRetryingProvider(ModelMetadataProvider):
model_ids,
)
async def get_model_versions_by_hashes(
self, hashes: List[str]
) -> Optional[List[Dict]]:
return await self._rate_limit_helper.run(
self._label,
self._provider.get_model_versions_by_hashes,
hashes,
)
async def get_model_version(self, model_id: int = None, version_id: int = None) -> Optional[Dict]:
return await self._rate_limit_helper.run(
self._label,
@@ -669,6 +765,17 @@ class ModelMetadataProviderManager:
provider = self._get_provider(provider_name)
return await provider.get_model_version_info(version_id)
async def get_model_versions_by_hashes(
self,
hashes: List[str],
provider_name: str = None,
) -> Optional[List[Dict]]:
provider = self._get_provider(provider_name)
try:
return await provider.get_model_versions_by_hashes(hashes)
except NotImplementedError:
return None
async def get_user_models(self, username: str, provider_name: str = None) -> Optional[List[Dict]]:
"""Fetch models owned by the specified user"""
provider = self._get_provider(provider_name)
+47 -10
View File
@@ -96,6 +96,7 @@ class FilterCriteria:
folder_exclude: Optional[Sequence[str]] = None
base_models: Optional[Sequence[str]] = None
tags: Optional[Dict[str, str]] = None
auto_tags: Optional[Dict[str, str]] = None
favorites_only: bool = False
search_options: Optional[Dict[str, Any]] = None
model_types: Optional[Sequence[str]] = None
@@ -293,12 +294,14 @@ class ModelFilterSet:
for tag, state in tag_filters.items():
if not tag:
continue
# Normalize to lowercase for case-insensitive matching
normalized = tag.strip().lower()
if state == "exclude":
exclude_tags.add(tag)
exclude_tags.add(normalized)
else:
include_tags.add(tag)
include_tags.add(normalized)
else:
include_tags = {tag for tag in tag_filters if tag}
include_tags = {tag.strip().lower() for tag in tag_filters if tag}
if include_tags:
tag_logic = criteria.tag_logic.lower() if criteria.tag_logic else "any"
@@ -317,13 +320,17 @@ class ModelFilterSet:
return True
# Otherwise, check if all non-special tags match
if non_special_tags:
return all(tag in (item_tags or []) for tag in non_special_tags)
# Case-insensitive: normalize item tags too
normalized_item_tags = {t.strip().lower() for t in (item_tags or []) if isinstance(t, str)}
return all(tag in normalized_item_tags for tag in non_special_tags)
return True
# Normal case: all tags must match
return all(tag in (item_tags or []) for tag in non_special_tags)
# Normal case: all tags must match (case-insensitive)
normalized_item_tags = {t.strip().lower() for t in (item_tags or []) if isinstance(t, str)}
return all(tag in normalized_item_tags for tag in non_special_tags)
else:
# OR logic (default): item must have ANY include tag
return any(tag in include_tags for tag in (item_tags or []))
# OR logic (default): item must have ANY include tag (case-insensitive)
normalized_item_tags = {t.strip().lower() for t in (item_tags or []) if isinstance(t, str)}
return bool(normalized_item_tags & include_tags)
items = [item for item in items if matches_include(item.get("tags"))]
@@ -332,7 +339,9 @@ class ModelFilterSet:
def matches_exclude(item_tags):
if not item_tags and "__no_tags__" in exclude_tags:
return True
return any(tag in exclude_tags for tag in (item_tags or []))
# Case-insensitive: normalize item tags
normalized_item_tags = {t.strip().lower() for t in (item_tags or []) if isinstance(t, str)}
return bool(normalized_item_tags & exclude_tags)
items = [
item for item in items if not matches_exclude(item.get("tags"))
@@ -359,10 +368,37 @@ class ModelFilterSet:
]
model_types_duration = time.perf_counter() - t0
auto_tags_duration = 0
auto_tag_filters = criteria.auto_tags or {}
if auto_tag_filters:
t0 = time.perf_counter()
include_at = set()
exclude_at = set()
for tag, state in auto_tag_filters.items():
if not tag:
continue
if state == "exclude":
exclude_at.add(tag)
else:
include_at.add(tag)
if include_at:
items = [
item for item in items
if any(tag in include_at for tag in (item.get("auto_tags") or []))
]
if exclude_at:
items = [
item for item in items
if not any(tag in exclude_at for tag in (item.get("auto_tags") or []))
]
auto_tags_duration = time.perf_counter() - t0
duration = time.perf_counter() - overall_start
if duration > 0.1: # Only log if it's potentially slow
logger.debug(
"ModelFilterSet.apply took %.3fs (sfw: %.3fs, fav: %.3fs, folder: %.3fs, base: %.3fs, tags: %.3fs, types: %.3fs). "
"ModelFilterSet.apply took %.3fs (sfw: %.3fs, fav: %.3fs, folder: %.3fs, base: %.3fs, tags: %.3fs, types: %.3fs, auto_tags: %.3fs). "
"Count: %d -> %d",
duration,
sfw_duration,
@@ -371,6 +407,7 @@ class ModelFilterSet:
base_models_duration,
tags_duration,
model_types_duration,
auto_tags_duration,
initial_count,
len(items),
)
+139 -23
View File
@@ -9,7 +9,7 @@ from typing import Any, Awaitable, Callable, Dict, List, Mapping, Optional, Set,
from ..utils.models import BaseModelMetadata
from ..config import config
from ..utils.file_utils import find_preview_file, get_preview_extension
from ..utils.file_utils import find_preview_file, get_preview_extension, calculate_sha256
from ..utils.metadata_manager import MetadataManager
from ..utils.civitai_utils import resolve_license_info
from .model_cache import ModelCache
@@ -248,6 +248,7 @@ class ModelScanner:
'civitai': civitai_slim,
'civitai_deleted': bool(get_value('civitai_deleted', False)),
'skip_metadata_refresh': bool(get_value('skip_metadata_refresh', False)),
'hf_url': get_value('hf_url', '') or '',
}
license_source: Dict[str, Any] = {}
@@ -476,11 +477,20 @@ class ModelScanner:
for tag in adjusted_item.get('tags') or []:
tags_count[tag] = tags_count.get(tag, 0) + 1
# Validate cache entries and check health
# Validate cache entries and check health.
# Always use the validated/repaired entries — even when there are no
# invalid entries, auto_repair may have filled in missing optional
# fields (model_name, file_name, folder) with safe defaults on a copied
# working_entry. Without this unconditional replacement the repaired
# copies are discarded and None values propagate to format_response.
# See issue #730.
valid_entries, invalid_entries = CacheEntryValidator.validate_batch(
adjusted_raw_data, auto_repair=True
)
# Always use the validated entries (repaired copies)
adjusted_raw_data = valid_entries
if invalid_entries:
monitor = CacheHealthMonitor()
report = monitor.check_health(adjusted_raw_data, auto_repair=True)
@@ -532,6 +542,13 @@ class ModelScanner:
if not scan_result or not getattr(self, '_persistent_cache', None):
return
if self.is_cancelled():
logger.info(
f"{self.model_type.capitalize()} Scanner: Skipping _save_persistent_cache "
"after cancellation"
)
return
hash_snapshot = self._build_hash_index_snapshot(scan_result.hash_index)
loop = asyncio.get_event_loop()
try:
@@ -705,14 +722,20 @@ class ModelScanner:
# Determine the page type based on model type
# Scan for new data
scan_result = await self._gather_model_data()
await self._apply_scan_result(scan_result)
await self._save_persistent_cache(scan_result)
await self._sync_download_history(scan_result.raw_data, source='scan')
if not self.is_cancelled():
await self._apply_scan_result(scan_result)
await self._save_persistent_cache(scan_result)
await self._sync_download_history(scan_result.raw_data, source='scan')
logger.info(
f"{self.model_type.capitalize()} Scanner: Cache initialization completed in {time.time() - start_time:.2f} seconds, "
f"found {len(scan_result.raw_data)} models"
)
logger.info(
f"{self.model_type.capitalize()} Scanner: Cache initialization completed in {time.time() - start_time:.2f} seconds, "
f"found {len(scan_result.raw_data)} models"
)
else:
logger.info(
f"{self.model_type.capitalize()} Scanner: Cache initialization cancelled "
f"after {time.time() - start_time:.2f} seconds"
)
except Exception as e:
logger.error(f"{self.model_type.capitalize()} Scanner: Error initializing cache: {e}")
# Ensure cache is at least an empty structure on error
@@ -1067,19 +1090,27 @@ class ModelScanner:
model_data = self._build_cache_entry(metadata, folder=normalized_folder)
# Compute SHA256 hash when metadata provided none (e.g., CivitAI API response has empty hashes).
# Respect hash_status='pending' (set by CheckpointScanner for large models) to defer
# hash calculation until on-demand — avoids reading entire checkpoint files at startup.
hash_status = model_data.get('hash_status', '')
if not model_data.get('sha256') and hash_status != 'pending' and file_path:
try:
logger.info(f"Computing SHA256 hash for {file_path} (was empty from metadata)")
sha256 = await calculate_sha256(file_path)
if sha256:
model_data['sha256'] = sha256.lower()
if isinstance(metadata, BaseModelMetadata):
metadata.sha256 = sha256.lower()
await MetadataManager.save_metadata(file_path, metadata)
except Exception as e:
logger.error(f"Failed to compute SHA256 for {file_path}: {e}")
# Skip excluded models
if model_data.get('exclude', False):
excluded_models.append(model_data['file_path'])
return None
# Check for duplicate filename before adding to hash index
# filename = os.path.splitext(os.path.basename(file_path))[0]
# existing_hash = hash_index.get_hash_by_filename(filename)
# if existing_hash and existing_hash != model_data.get('sha256', '').lower():
# existing_path = hash_index.get_path(existing_hash)
# if existing_path and existing_path != file_path:
# logger.warning(f"Duplicate filename detected: '{filename}' - files: '{existing_path}' and '{file_path}'")
return model_data
async def _apply_scan_result(self, scan_result: CacheBuildResult) -> None:
@@ -1088,6 +1119,13 @@ class ModelScanner:
if scan_result is None:
return
if self.is_cancelled():
logger.info(
f"{self.model_type.capitalize()} Scanner: Skipping _apply_scan_result "
"after cancellation"
)
return
self._hash_index = scan_result.hash_index
self._tags_count = dict(scan_result.tags_count)
self._excluded_models = list(scan_result.excluded_models)
@@ -1105,6 +1143,39 @@ class ModelScanner:
await self._cache.resort()
self._log_duplicate_filename_summary()
def _log_duplicate_filename_summary(self) -> None:
"""Log a batched summary of duplicate filename conflicts once per scan."""
# Duplicate filename detection is only relevant for LoRAs, which use
# basename-only syntax (<lora:name:strength>). Checkpoints and embeddings
# use full relative paths for resolution, so conflicts are not ambiguous.
if self._hash_index is None or self.model_type != "lora":
return
# When full path syntax is active, duplicate filenames across subfolders
# are fully qualified, so there is no ambiguity — skip the warning.
if get_settings_manager().get("lora_syntax_format", "legacy") == "full":
return
duplicates = self._hash_index.get_duplicate_filenames()
if not duplicates:
return
total_files = sum(len(paths) for paths in duplicates.values())
conflict_count = len(duplicates)
model_type_label = self.model_type or "model"
logger.warning(
"Duplicate filename conflict detected: %d %s filename(s) "
"are shared by %d files total, causing ambiguity in %s resolution. "
"Open the Doctor panel to resolve one-click.",
conflict_count,
model_type_label,
total_files,
model_type_label.capitalize(),
)
async def _sync_download_history(
self,
raw_data: List[Mapping[str, Any]],
@@ -1456,6 +1527,15 @@ class ModelScanner:
file_path_override=normalized_new_path,
)
# Ensure sha256 is populated even when metadata doesn't have it
if not cache_entry.get('sha256') and normalized_new_path and os.path.exists(normalized_new_path):
try:
sha256 = await calculate_sha256(normalized_new_path)
if sha256:
cache_entry['sha256'] = sha256.lower()
except Exception as e:
logger.error(f"Failed to compute SHA256 for {normalized_new_path}: {e}")
if recalculate_type:
cache_entry = self.adjust_cached_entry(cache_entry)
@@ -1535,7 +1615,7 @@ class ModelScanner:
return sorted_tags[:limit]
async def get_base_models(self, limit: int = 20) -> List[Dict[str, any]]:
"""Get base models sorted by frequency"""
"""Get base models sorted by count. If limit is 0, return all."""
cache = await self.get_cached_data()
base_model_counts = {}
@@ -1546,19 +1626,48 @@ class ModelScanner:
sorted_models = [{'name': model, 'count': count} for model, count in base_model_counts.items()]
sorted_models.sort(key=lambda x: x['count'], reverse=True)
if limit == 0:
return sorted_models
return sorted_models[:limit]
async def get_model_info_by_name(self, name):
"""Get model information by name"""
try:
cache = await self.get_cached_data()
name_normalized = name.replace("\\", "/")
name_no_ext = name_normalized
for ext in (".safetensors", ".ckpt", ".pt", ".bin"):
if name_no_ext.lower().endswith(ext):
name_no_ext = name_no_ext[: -len(ext)]
break
has_path = "/" in name_no_ext
basename = os.path.basename(name_no_ext) if has_path else name_no_ext
best_fallback = None
for model in cache.raw_data:
if model.get("file_name") == name:
file_name = model.get("file_name", "")
folder = model.get("folder", "")
file_name_no_ext = file_name
for ext in (".safetensors", ".ckpt", ".pt", ".bin"):
if file_name_no_ext.lower().endswith(ext):
file_name_no_ext = file_name_no_ext[: -len(ext)]
break
path_name = f"{folder}/{file_name_no_ext}".replace("\\", "/") if folder else file_name_no_ext
if name_no_ext == file_name_no_ext or name_no_ext == path_name:
return model
return None
if has_path and file_name_no_ext == basename:
if folder and name_no_ext.startswith(folder.replace("\\", "/") + "/"):
best_fallback = model
elif best_fallback is None:
best_fallback = model
return best_fallback
except Exception as e:
logger.error(f"Error getting model info by name: {e}", exc_info=True)
return None
@@ -1685,6 +1794,13 @@ class ModelScanner:
"""
if not file_paths or self._cache is None:
return False
if self.is_cancelled():
logger.info(
f"{self.model_type.capitalize()} Scanner: Skipping cache update "
"after cancelled bulk delete"
)
return False
try:
# Get all models that need to be removed from cache
+263 -29
View File
@@ -69,6 +69,7 @@ class ModelVersionRecord:
early_access_ends_at: Optional[str] = None
sort_index: int = 0
is_early_access: bool = False
usage_control: Optional[str] = None # "Download", "Generation", "InternalGeneration"
@dataclass
@@ -101,11 +102,14 @@ class ModelUpdateRecord:
return [version.version_id for version in self.versions if version.is_in_library]
def has_update(self, hide_early_access: bool = False) -> bool:
def has_update(
self, hide_early_access: bool = False, hide_non_downloadable: bool = True
) -> bool:
"""Return True when a non-ignored remote version newer than the newest local copy is available.
Args:
hide_early_access: If True, exclude early access versions from update check.
hide_non_downloadable: If True, exclude versions that don't allow downloads.
"""
if self.should_ignore_model:
@@ -121,6 +125,7 @@ class ModelUpdateRecord:
not version.is_in_library
and not version.should_ignore
and not (hide_early_access and ModelUpdateRecord._is_early_access_active(version))
and not (hide_non_downloadable and not ModelUpdateRecord._is_downloadable(version))
for version in self.versions
)
@@ -129,6 +134,8 @@ class ModelUpdateRecord:
continue
if hide_early_access and ModelUpdateRecord._is_early_access_active(version):
continue
if hide_non_downloadable and not ModelUpdateRecord._is_downloadable(version):
continue
if version.version_id > max_in_library:
return True
return False
@@ -155,11 +162,18 @@ class ModelUpdateRecord:
# Phase 1: Basic EA flag from bulk API
return version.is_early_access
@staticmethod
def _is_downloadable(version: ModelVersionRecord) -> bool:
if version.usage_control is None:
return True
return version.usage_control == "Download"
def has_update_for_base(
self,
local_version_id: Optional[int],
local_base_model: Optional[str],
hide_early_access: bool = False,
hide_non_downloadable: bool = True,
) -> bool:
"""Return True when a newer remote version with the same base model exists.
@@ -167,6 +181,7 @@ class ModelUpdateRecord:
local_version_id: The current local version id.
local_base_model: The base model to filter by.
hide_early_access: If True, exclude early access versions from update check.
hide_non_downloadable: If True, exclude versions that don't allow downloads.
"""
if self.should_ignore_model:
@@ -197,6 +212,8 @@ class ModelUpdateRecord:
continue
if hide_early_access and ModelUpdateRecord._is_early_access_active(version):
continue
if hide_non_downloadable and not ModelUpdateRecord._is_downloadable(version):
continue
version_base = _normalize_base_model(version.base_model)
if version_base != normalized_base:
continue
@@ -209,6 +226,8 @@ class ModelUpdateRecord:
class ModelUpdateService:
"""Persist and query remote model version metadata."""
_SQLITE_MAX_VARIABLES = 500
_SCHEMA = """
PRAGMA foreign_keys = ON;
CREATE TABLE IF NOT EXISTS model_update_status (
@@ -228,6 +247,7 @@ class ModelUpdateService:
preview_url TEXT,
is_in_library INTEGER NOT NULL DEFAULT 0,
should_ignore INTEGER NOT NULL DEFAULT 0,
usage_control TEXT,
PRIMARY KEY (model_id, version_id),
FOREIGN KEY(model_id) REFERENCES model_update_status(model_id) ON DELETE CASCADE
);
@@ -463,6 +483,10 @@ class ModelUpdateService:
"ALTER TABLE model_update_versions "
"ADD COLUMN is_early_access INTEGER NOT NULL DEFAULT 0"
),
"usage_control": (
"ALTER TABLE model_update_versions "
"ADD COLUMN usage_control TEXT"
),
}
for column, statement in migrations.items():
@@ -665,6 +689,7 @@ class ModelUpdateService:
*,
force_refresh: bool = False,
target_model_ids: Optional[Sequence[int]] = None,
folder_path: Optional[str] = None,
) -> Dict[int, ModelUpdateRecord]:
"""Refresh update information for every model present in the cache."""
scanner.reset_cancellation()
@@ -679,6 +704,7 @@ class ModelUpdateService:
local_versions = await self._collect_local_versions(
scanner,
target_model_ids=target_filter,
folder_path=folder_path,
)
total_models = len(local_versions)
if total_models == 0:
@@ -698,6 +724,16 @@ class ModelUpdateService:
"Refreshing update metadata for %d %s models", total_models, model_type
)
# When filtering by folder, also collect the cross-folder version set
# so that versions already present in other folders are not reported
# as available updates. See issue #997.
all_local_versions: Optional[Dict[int, List[int]]] = None
if folder_path is not None:
all_local_versions = await self._collect_local_versions(
scanner,
target_model_ids=target_filter,
)
results: Dict[int, ModelUpdateRecord] = {}
prefetched: Dict[int, Mapping] = {}
@@ -736,6 +772,12 @@ class ModelUpdateService:
for index, (model_id, version_ids) in enumerate(
local_versions.items(), start=1
):
# Use cross-folder version IDs for is_in_library if available
all_vids: Sequence[int] = (
all_local_versions.get(model_id, [])
if all_local_versions is not None
else version_ids
)
record = await self._refresh_single_model(
model_type,
model_id,
@@ -743,6 +785,7 @@ class ModelUpdateService:
metadata_provider,
force_refresh=force_refresh,
prefetched_response=prefetched.get(model_id),
all_local_version_ids=all_vids,
)
if scanner.is_cancelled():
logger.info(f"{model_type.capitalize()} Update Service: Refresh cancelled by user")
@@ -938,8 +981,16 @@ class ModelUpdateService:
*,
force_refresh: bool = False,
prefetched_response: Optional[Mapping] = None,
all_local_version_ids: Optional[Sequence[int]] = None,
) -> Optional[ModelUpdateRecord]:
normalized_local = self._normalize_sequence(local_versions)
# When folder-filtering, this carries the cross-folder version set
# for is_in_library; otherwise it falls back to normalized_local.
normalized_all = (
self._normalize_sequence(all_local_version_ids)
if all_local_version_ids is not None
else normalized_local
)
now = time.time()
async with self._lock:
existing = self._get_record(model_type, model_id)
@@ -947,6 +998,7 @@ class ModelUpdateService:
record = self._merge_with_local_versions(
existing,
normalized_local,
all_local_version_ids=normalized_all,
)
self._upsert_record(record)
return record
@@ -965,18 +1017,22 @@ class ModelUpdateService:
fallback_attempted = True
try:
response = await metadata_provider.get_model_versions(model_id)
if response is not None:
await self._enrich_version_entries(
metadata_provider,
{model_id: response},
)
except RateLimitError:
raise
except ResourceNotFoundError as exc:
fallback_error_message = str(exc) or "resource not found"
mark_model_as_ignored = True
except Exception as exc: # pragma: no cover - defensive log
logger.error(
logger.warning(
"Failed to fetch versions for model %s (%s): %s",
model_id,
model_type,
exc,
exc_info=True,
)
fallback_error_message = str(exc)
if response is not None:
@@ -1018,6 +1074,7 @@ class ModelUpdateService:
record = self._merge_with_local_versions(
existing,
normalized_local,
all_local_version_ids=normalized_all,
)
self._upsert_record(record)
return record
@@ -1029,6 +1086,7 @@ class ModelUpdateService:
model_type=model_type,
model_id=model_id,
last_checked_at=now,
all_local_version_ids=normalized_all,
)
record = replace(record, should_ignore_model=True)
self._upsert_record(record)
@@ -1047,6 +1105,7 @@ class ModelUpdateService:
fetched_versions,
existing,
now,
all_local_version_ids=normalized_all,
)
else:
record = self._merge_with_local_versions(
@@ -1055,10 +1114,141 @@ class ModelUpdateService:
model_type=model_type,
model_id=model_id,
last_checked_at=existing.last_checked_at if existing else None,
all_local_version_ids=normalized_all,
)
self._upsert_record(record)
return record
async def _enrich_version_entries(
self,
metadata_provider,
responses_by_model_id: Dict[int, Mapping],
) -> None:
"""Enrich version entries with ``usageControl`` via batch hash endpoint.
The model-level API does not include ``usageControl`` on version
entries. This method collects SHA256 hashes from every version's
primary model file, calls ``POST /api/v1/model-versions/by-hash``
(up to 100 hashes per request), and injects ``usageControl`` +
``earlyAccessEndsAt`` into each version entry dict in-place.
"""
if not metadata_provider or not responses_by_model_id:
return
hashes_by_version: Dict[int, str] = {}
for response in responses_by_model_id.values():
hashes_by_version.update(
self._collect_hashes_from_response(response)
)
if not hashes_by_version:
return
version_ids_by_hash: Dict[str, List[int]] = {}
for version_id, sha256 in hashes_by_version.items():
version_ids_by_hash.setdefault(sha256, []).append(version_id)
all_hashes = list(version_ids_by_hash.keys())
BATCH_SIZE = 100
enrichment: Dict[int, Dict] = {}
try:
for start in range(0, len(all_hashes), BATCH_SIZE):
batch = all_hashes[start : start + BATCH_SIZE]
try:
enriched = await metadata_provider.get_model_versions_by_hashes(
batch
)
except NotImplementedError:
return
except RateLimitError:
raise
except Exception:
continue
if not enriched:
continue
for entry in enriched:
if not isinstance(entry, dict):
continue
version_id = entry.get("id")
if version_id is None:
continue
enrichment[version_id] = {
"usageControl": _normalize_string(
entry.get("usageControl")
),
"earlyAccessEndsAt": _normalize_string(
entry.get("earlyAccessEndsAt")
),
}
except RateLimitError:
raise
if not enrichment:
return
for response in responses_by_model_id.values():
versions = response.get("modelVersions")
if not isinstance(versions, list):
continue
for version in versions:
if not isinstance(version, dict):
continue
version_id = version.get("id")
if version_id not in enrichment:
continue
extra = enrichment[version_id]
if extra.get("usageControl") and not version.get("usageControl"):
version["usageControl"] = extra["usageControl"]
if extra.get("earlyAccessEndsAt") and not version.get(
"earlyAccessEndsAt"
):
version["earlyAccessEndsAt"] = extra["earlyAccessEndsAt"]
@staticmethod
def _collect_hashes_from_response(response: Mapping) -> Dict[int, str]:
"""Extract ``{version_id: sha256}`` from a model-level API response.
Returns an empty dict if the response structure is unexpected.
"""
result: Dict[int, str] = {}
versions = response.get("modelVersions")
if not isinstance(versions, list):
return result
for entry in versions:
if not isinstance(entry, dict):
continue
version_id = _normalize_int(entry.get("id"))
if version_id is None:
continue
sha256 = ModelUpdateService._extract_sha256_from_version_entry(entry)
if sha256:
result[version_id] = sha256
return result
@staticmethod
def _extract_sha256_from_version_entry(entry: Mapping) -> Optional[str]:
"""Return the SHA256 hash from the primary model file of a version entry."""
files = entry.get("files")
if not isinstance(files, list):
return None
for file_info in files:
if not isinstance(file_info, dict):
continue
if file_info.get("type") != "Model":
continue
primary = file_info.get("primary")
if primary is not True and str(primary).strip().lower() != "true":
continue
hashes = file_info.get("hashes")
if isinstance(hashes, dict):
sha256 = hashes.get("SHA256")
if sha256:
return sha256
return None
async def _fetch_model_versions_bulk(
self,
metadata_provider,
@@ -1110,6 +1300,7 @@ class ModelUpdateService:
len(aggregated),
provider_name,
)
await self._enrich_version_entries(metadata_provider, aggregated)
return aggregated
async def _collect_local_versions(
@@ -1117,6 +1308,7 @@ class ModelUpdateService:
scanner,
*,
target_model_ids: Optional[Sequence[int]] = None,
folder_path: Optional[str] = None,
) -> Dict[int, List[int]]:
cache = await scanner.get_cached_data()
mapping: Dict[int, set[int]] = {}
@@ -1129,7 +1321,19 @@ class ModelUpdateService:
if not target_set:
return {}
normalized_folder = None
if folder_path is not None:
normalized_folder = folder_path.replace("\\", "/").strip("/")
for item in cache.raw_data:
# Apply folder filter first (cheapest check)
if normalized_folder is not None:
if not isinstance(item, dict):
continue
item_folder = (item.get("folder") or "").replace("\\", "/").strip("/")
if item_folder != normalized_folder and not item_folder.startswith(normalized_folder + "/"):
continue
civitai = item.get("civitai") if isinstance(item, dict) else None
if not isinstance(civitai, dict):
continue
@@ -1148,12 +1352,20 @@ class ModelUpdateService:
existing: Optional[ModelUpdateRecord],
normalized_local: Sequence[int],
*,
all_local_version_ids: Optional[Sequence[int]] = None,
model_type: Optional[str] = None,
model_id: Optional[int] = None,
last_checked_at: Optional[float] = None,
version_info: Optional[Mapping] = None,
) -> ModelUpdateRecord:
local_set = set(normalized_local)
# When folder-filtering, also consider versions in other folders
# as in-library so they are not reported as available updates.
effective_local_set: set[int] = (
local_set | set(all_local_version_ids)
if all_local_version_ids is not None
else local_set
)
versions: List[ModelVersionRecord] = []
ignore_map: Dict[int, bool] = {}
if existing:
@@ -1165,7 +1377,7 @@ class ModelUpdateService:
versions.append(
replace(
version,
is_in_library=version.version_id in local_set,
is_in_library=version.version_id in effective_local_set,
)
)
elif model_type is None or model_id is None:
@@ -1212,8 +1424,17 @@ class ModelUpdateService:
remote_versions: Sequence[ModelVersionRecord],
existing: Optional[ModelUpdateRecord],
timestamp: float,
*,
all_local_version_ids: Optional[Sequence[int]] = None,
) -> ModelUpdateRecord:
local_set = set(local_versions)
# When folder-filtering, also consider versions in other folders
# as in-library so they are not reported as available updates.
effective_local_set: set[int] = (
local_set | set(all_local_version_ids)
if all_local_version_ids is not None
else local_set
)
ignore_map = {version.version_id: version.should_ignore for version in existing.versions} if existing else {}
preview_map = {version.version_id: version.preview_url for version in existing.versions} if existing else {}
sort_map = {version.version_id: version.sort_index for version in existing.versions} if existing else {}
@@ -1232,11 +1453,12 @@ class ModelUpdateService:
released_at=remote_version.released_at,
size_bytes=remote_version.size_bytes,
preview_url=remote_version.preview_url or preview_map.get(version_id),
is_in_library=version_id in local_set,
is_in_library=version_id in effective_local_set,
should_ignore=ignore_map.get(version_id, remote_version.should_ignore),
sort_index=sort_map.get(version_id, index),
early_access_ends_at=remote_version.early_access_ends_at,
is_early_access=remote_version.is_early_access,
usage_control=remote_version.usage_control,
)
)
@@ -1335,6 +1557,7 @@ class ModelUpdateService:
# Check availability field from bulk API for basic EA detection
availability = _normalize_string(entry.get("availability"))
is_early_access = availability == "EarlyAccess"
usage_control = _normalize_string(entry.get("usageControl"))
return ModelVersionRecord(
version_id=version_id,
@@ -1348,6 +1571,7 @@ class ModelUpdateService:
early_access_ends_at=early_access_ends_at,
sort_index=index,
is_early_access=is_early_access,
usage_control=usage_control,
)
def _extract_size_bytes(self, files) -> Optional[int]:
@@ -1439,33 +1663,41 @@ class ModelUpdateService:
if not model_ids:
return {}
params = tuple(model_ids)
placeholders = ",".join("?" for _ in params)
ids = list(model_ids)
status_rows: list = []
version_rows: list = []
with self._connect() as conn:
status_rows = conn.execute(
f"""
SELECT model_id, model_type, last_checked_at, should_ignore_model
FROM model_update_status
WHERE model_id IN ({placeholders})
""",
params,
).fetchall()
for start in range(0, len(ids), self._SQLITE_MAX_VARIABLES):
chunk = tuple(ids[start : start + self._SQLITE_MAX_VARIABLES])
placeholders = ",".join("?" for _ in chunk)
chunk_status = conn.execute(
f"""
SELECT model_id, model_type, last_checked_at, should_ignore_model
FROM model_update_status
WHERE model_id IN ({placeholders})
""",
chunk,
).fetchall()
status_rows.extend(chunk_status)
chunk_versions = conn.execute(
f"""
SELECT model_id, version_id, sort_index, name, base_model, released_at,
size_bytes, preview_url, is_in_library, should_ignore, early_access_ends_at,
is_early_access, usage_control
FROM model_update_versions
WHERE model_id IN ({placeholders})
ORDER BY model_id ASC, sort_index ASC, version_id ASC
""",
chunk,
).fetchall()
version_rows.extend(chunk_versions)
if not status_rows:
return {}
version_rows = conn.execute(
f"""
SELECT model_id, version_id, sort_index, name, base_model, released_at,
size_bytes, preview_url, is_in_library, should_ignore, early_access_ends_at,
is_early_access
FROM model_update_versions
WHERE model_id IN ({placeholders})
ORDER BY model_id ASC, sort_index ASC, version_id ASC
""",
params,
).fetchall()
versions_by_model: Dict[int, List[ModelVersionRecord]] = {}
for row in version_rows:
model_id = int(row["model_id"])
@@ -1482,6 +1714,7 @@ class ModelUpdateService:
early_access_ends_at=row["early_access_ends_at"],
sort_index=_normalize_int(row["sort_index"]) or 0,
is_early_access=bool(row["is_early_access"]),
usage_control=row["usage_control"],
)
)
@@ -1538,8 +1771,8 @@ class ModelUpdateService:
INSERT INTO model_update_versions (
version_id, model_id, sort_index, name, base_model, released_at,
size_bytes, preview_url, is_in_library, should_ignore, early_access_ends_at,
is_early_access
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
is_early_access, usage_control
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
version.version_id,
@@ -1554,6 +1787,7 @@ class ModelUpdateService:
1 if version.should_ignore else 0,
version.early_access_ends_at,
1 if version.is_early_access else 0,
version.usage_control,
),
)
conn.commit()
+14 -9
View File
@@ -57,6 +57,7 @@ class PersistentModelCache:
"db_checked",
"last_checked_at",
"hash_status",
"hf_url",
)
_MODEL_UPDATE_COLUMNS: Tuple[str, ...] = _MODEL_COLUMNS[2:]
_instances: Dict[str, "PersistentModelCache"] = {}
@@ -165,8 +166,8 @@ class PersistentModelCache:
item = {
"file_path": file_path,
"file_name": row["file_name"],
"model_name": row["model_name"],
"file_name": row["file_name"] or "",
"model_name": row["model_name"] or "",
"folder": row["folder"] or "",
"size": row["size"] or 0,
"modified": row["modified"] or 0.0,
@@ -188,6 +189,7 @@ class PersistentModelCache:
"skip_metadata_refresh": bool(row["skip_metadata_refresh"]),
"license_flags": int(license_value),
"hash_status": row["hash_status"] or "completed",
"hf_url": row["hf_url"] or "",
}
raw_data.append(item)
@@ -452,6 +454,7 @@ class PersistentModelCache:
db_checked INTEGER,
last_checked_at REAL,
hash_status TEXT,
hf_url TEXT DEFAULT '',
PRIMARY KEY (model_type, file_path)
);
@@ -500,6 +503,7 @@ class PersistentModelCache:
# Persisting without explicit flags should assume CivitAI's documented defaults (0b111001 == 57).
"license_flags": f"INTEGER DEFAULT {DEFAULT_LICENSE_FLAGS}",
"hash_status": "TEXT DEFAULT 'completed'",
"hf_url": "TEXT DEFAULT ''",
}
for column, definition in required_columns.items():
@@ -548,19 +552,19 @@ class PersistentModelCache:
return (
model_type,
item.get("file_path"),
item.get("file_name"),
item.get("model_name"),
item.get("folder"),
item.get("file_name") or "",
item.get("model_name") or "",
item.get("folder") or "",
int(item.get("size") or 0),
float(item.get("modified") or 0.0),
(item.get("sha256") or "").lower() or None,
item.get("base_model"),
item.get("preview_url"),
item.get("base_model") or "",
item.get("preview_url") or "",
int(item.get("preview_nsfw_level") or 0),
1 if item.get("from_civitai", True) else 0,
1 if item.get("favorite") else 0,
item.get("notes"),
item.get("usage_tips"),
item.get("notes") or "",
item.get("usage_tips") or "",
metadata_source,
civitai.get("id"),
civitai.get("modelId"),
@@ -575,6 +579,7 @@ class PersistentModelCache:
1 if item.get("db_checked") else 0,
float(item.get("last_checked_at") or 0.0),
item.get("hash_status", "completed"),
item.get("hf_url") or "",
)
def _insert_model_sql(self) -> str:
+69 -3
View File
@@ -12,7 +12,7 @@ import logging
import os
import sqlite3
import threading
from dataclasses import dataclass
from dataclasses import dataclass, field
from typing import Dict, List, Optional, Set, Tuple
from ..utils.cache_paths import CacheType, resolve_cache_path_with_migration
@@ -26,6 +26,8 @@ class PersistedRecipeData:
raw_data: List[Dict]
file_stats: Dict[str, Tuple[float, int]] # json_path -> (mtime, size)
image_id_map: Dict[str, str] = field(default_factory=dict)
"""Precomputed mapping of civitai image_id → recipe_id."""
class PersistentRecipeCache:
@@ -38,6 +40,7 @@ class PersistentRecipeCache:
"json_path",
"title",
"folder",
"source_path",
"base_model",
"fingerprint",
"created_date",
@@ -115,6 +118,20 @@ class PersistentRecipeCache:
if not rows:
return None
# Restore precomputed image_id_map if available
image_id_map: Dict[str, str] = {}
try:
meta_row = conn.execute(
"SELECT value FROM cache_metadata WHERE key = ?",
("image_id_map",),
).fetchone()
if meta_row:
parsed = json.loads(meta_row["value"])
if isinstance(parsed, dict):
image_id_map = parsed
except Exception:
pass # missing or corrupt — rebuilt on next cache refresh
finally:
conn.close()
except FileNotFoundError:
@@ -137,14 +154,24 @@ class PersistentRecipeCache:
row["file_size"] or 0,
)
return PersistedRecipeData(raw_data=raw_data, file_stats=file_stats)
return PersistedRecipeData(
raw_data=raw_data,
file_stats=file_stats,
image_id_map=image_id_map,
)
def save_cache(self, recipes: List[Dict], json_paths: Optional[Dict[str, str]] = None) -> None:
def save_cache(
self,
recipes: List[Dict],
json_paths: Optional[Dict[str, str]] = None,
image_id_map: Optional[Dict[str, str]] = None,
) -> None:
"""Save all recipes to SQLite cache.
Args:
recipes: List of recipe dictionaries to persist.
json_paths: Optional mapping of recipe_id -> json_path for file stats.
image_id_map: Optional precomputed civitai image_id → recipe_id mapping.
"""
if not self.is_enabled():
return
@@ -185,6 +212,12 @@ class PersistentRecipeCache:
recipe_rows,
)
# Persist image_id_map for O(1) lookups on cache load
conn.execute(
"INSERT OR REPLACE INTO cache_metadata (key, value) VALUES (?, ?)",
("image_id_map", json.dumps(image_id_map or {})),
)
conn.commit()
logger.debug("Persisted %d recipes to cache", len(recipe_rows))
finally:
@@ -272,6 +305,29 @@ class PersistentRecipeCache:
except Exception as exc:
logger.debug("Failed to remove recipe %s from cache: %s", recipe_id, exc)
def save_image_id_map(self, image_id_map: Dict[str, str]) -> None:
"""Persist the image_id_map to cache_metadata without rewriting the full cache.
This is called after ``add_recipe`` / ``remove_recipe`` mutations so
the persistent copy does not go stale between full ``save_cache`` calls.
"""
if not self.is_enabled() or not self._schema_initialized:
return
try:
with self._db_lock:
conn = self._connect()
try:
conn.execute(
"INSERT OR REPLACE INTO cache_metadata (key, value) VALUES (?, ?)",
("image_id_map", json.dumps(image_id_map)),
)
conn.commit()
finally:
conn.close()
except Exception as exc:
logger.debug("Failed to persist image_id_map: %s", exc)
def get_indexed_recipe_ids(self) -> Set[str]:
"""Return all recipe IDs in the cache.
@@ -334,6 +390,7 @@ class PersistentRecipeCache:
json_path TEXT,
title TEXT,
folder TEXT,
source_path TEXT,
base_model TEXT,
fingerprint TEXT,
created_date REAL,
@@ -358,6 +415,13 @@ class PersistentRecipeCache:
);
"""
)
# Migration: add source_path column to existing databases
try:
conn.execute(
"ALTER TABLE recipes ADD COLUMN source_path TEXT"
)
except Exception:
pass # column already exists
conn.commit()
self._schema_initialized = True
except Exception as exc:
@@ -406,6 +470,7 @@ class PersistentRecipeCache:
json_path,
recipe.get("title"),
recipe.get("folder"),
recipe.get("source_path"),
recipe.get("base_model"),
recipe.get("fingerprint"),
float(recipe.get("created_date") or 0.0),
@@ -456,6 +521,7 @@ class PersistentRecipeCache:
"file_path": row["file_path"] or "",
"title": row["title"] or "",
"folder": row["folder"] or "",
"source_path": row["source_path"] or "",
"base_model": row["base_model"] or "",
"fingerprint": row["fingerprint"] or "",
"created_date": row["created_date"] or 0.0,
+10 -1
View File
@@ -1,6 +1,6 @@
import asyncio
from typing import Iterable, List, Dict, Optional
from dataclasses import dataclass
from dataclasses import dataclass, field
from operator import itemgetter
from natsort import natsorted
@@ -14,6 +14,15 @@ class RecipeCache:
sorted_by_date: List[Dict]
folders: List[str] | None = None
folder_tree: Dict | None = None
image_id_map: Dict[str, str] = field(default_factory=dict)
"""Mapping of civitai image_id → recipe_id, precomputed at cache build time.
Built once during cache initialization (O(n)) so that
``check_image_exists`` and ``import_from_url`` duplicate checks
can look up image_id in O(1) instead of scanning all recipes.
Recipes imported from local files have no valid civitai image_id
and are naturally excluded from this map.
"""
def __post_init__(self):
self._lock = asyncio.Lock()
+147 -5
View File
@@ -20,6 +20,7 @@ from .metadata_service import get_default_metadata_provider
from .checkpoint_scanner import CheckpointScanner
from .settings_manager import get_settings_manager
from .recipes.errors import RecipeNotFoundError
from ..utils.civitai_utils import extract_civitai_image_id
from ..utils.utils import calculate_recipe_fingerprint, fuzzy_match
from natsort import natsorted
import sys
@@ -65,7 +66,7 @@ class RecipeScanner:
cls._instance._civitai_client = None # Will be lazily initialized
return cls._instance
REPAIR_VERSION = 3
REPAIR_VERSION = 4
def __init__(
self,
@@ -292,6 +293,32 @@ class RecipeScanner:
if recipe.get("repair_version", 0) >= self.REPAIR_VERSION:
return False
# 1.5 Detect and clear corrupted checkpoint (LoRA data saved as checkpoint).
# A checkpoint whose modelVersionId also appears in a LoRA entry is
# definitely wrong — the CivitAI import code used to pick
# modelVersionIds[0] as the checkpoint, which was often a LoRA.
# Clearing it lets the enrichment flow re-resolve the correct
# checkpoint from CivitAI image metadata.
cp = recipe.get("checkpoint")
lora_mvids = {
l.get("modelVersionId")
for l in recipe.get("loras", [])
if l.get("modelVersionId")
}
if cp and cp.get("modelVersionId") and cp["modelVersionId"] in lora_mvids:
cp_mvid = cp["modelVersionId"]
logger.info(
"Recipe %s: checkpoint modelVersionId %s matches a LoRA — "
"clearing corrupted checkpoint and removing matching LoRA entry",
recipe.get("id"),
cp_mvid,
)
recipe["checkpoint"] = None
recipe["loras"] = [
l for l in recipe.get("loras", [])
if l.get("modelVersionId") != cp_mvid
]
# 2. Identification: Is repair needed?
has_checkpoint = (
"checkpoint" in recipe
@@ -504,6 +531,23 @@ class RecipeScanner:
self._cache.raw_data = recipes
self._update_folder_metadata(self._cache)
self._sort_cache_sync()
# Backfill source_path from JSON files if missing (schema migration)
if self._backfill_source_path_if_needed(recipes, json_paths):
self._cache.image_id_map = self._build_image_id_map()
self._persistent_cache.save_cache(
recipes, json_paths, self._cache.image_id_map
)
else:
# Use persisted map, or rebuild if empty (e.g. first startup
# after deploying the image_id_map feature).
if persisted.image_id_map:
self._cache.image_id_map = dict(persisted.image_id_map)
else:
self._cache.image_id_map = self._build_image_id_map()
if self._cache.image_id_map:
self._persistent_cache.save_image_id_map(
self._cache.image_id_map
)
return self._cache
else:
# Partial update: some files changed
@@ -514,8 +558,13 @@ class RecipeScanner:
self._cache.raw_data = recipes
self._update_folder_metadata(self._cache)
self._sort_cache_sync()
# Backfill source_path from JSON files if missing (schema migration)
self._backfill_source_path_if_needed(recipes, json_paths)
self._cache.image_id_map = self._build_image_id_map()
# Persist updated cache
self._persistent_cache.save_cache(recipes, json_paths)
self._persistent_cache.save_cache(
recipes, json_paths, self._cache.image_id_map
)
return self._cache
# Fall back to full directory scan
@@ -527,9 +576,12 @@ class RecipeScanner:
self._cache.raw_data = recipes
self._update_folder_metadata(self._cache)
self._sort_cache_sync()
self._cache.image_id_map = self._build_image_id_map()
# Persist for next startup
self._persistent_cache.save_cache(recipes, json_paths)
self._persistent_cache.save_cache(
recipes, json_paths, self._cache.image_id_map
)
return self._cache
except Exception as e:
@@ -642,6 +694,34 @@ class RecipeScanner:
return recipes, changed, json_paths
def _backfill_source_path_if_needed(
self,
recipes: List[Dict],
json_paths: Dict[str, str],
) -> bool:
"""Backfill source_path from recipe JSON files if missing from cache.
Returns True if any recipes were updated (caller should persist cache).
"""
updated = False
for recipe in recipes:
if recipe.get("source_path"):
continue
recipe_id = str(recipe.get("id", ""))
json_path = json_paths.get(recipe_id)
if not json_path or not os.path.exists(json_path):
continue
try:
with open(json_path, "r", encoding="utf-8") as f:
json_data = json.load(f)
file_source_path = json_data.get("source_path")
if file_source_path:
recipe["source_path"] = file_source_path
updated = True
except Exception:
pass
return updated
def _full_directory_scan_sync(
self, recipes_dir: str
) -> Tuple[List[Dict], Dict[str, str]]:
@@ -773,6 +853,28 @@ class RecipeScanner:
except Exception as e:
logger.error(f"Error sorting recipe cache: {e}")
def _build_image_id_map(self) -> Dict[str, str]:
"""Build civitai image_id → recipe_id mapping from cached recipes.
Only recipes with a valid CivitAI image URL source_path produce an
entry. Recipes imported from local files are naturally excluded.
"""
mapping: Dict[str, str] = {}
if not self._cache:
return mapping
for recipe in getattr(self._cache, "raw_data", []):
if not isinstance(recipe, dict):
continue
source = recipe.get("source_path")
if not source:
continue
image_id = extract_civitai_image_id(source)
if image_id and image_id not in mapping:
recipe_id = recipe.get("id")
if recipe_id is not None:
mapping[image_id] = str(recipe_id)
return mapping
async def _wait_for_lora_scanner(self) -> None:
"""Ensure the LoRA scanner has initialized before recipe enrichment."""
@@ -1237,11 +1339,20 @@ class RecipeScanner:
# Update FTS index
self._update_fts_index_for_recipe(recipe_data, "add")
source = recipe_data.get("source_path")
if source:
image_id = extract_civitai_image_id(source)
if image_id:
recipe_id_value = recipe_data.get("id")
if recipe_id_value is not None:
cache.image_id_map[image_id] = str(recipe_id_value)
# Persist to SQLite cache
if self._persistent_cache:
recipe_id = str(recipe_data.get("id", ""))
json_path = self._json_path_map.get(recipe_id, "")
self._persistent_cache.update_recipe(recipe_data, json_path)
self._persistent_cache.save_image_id_map(cache.image_id_map)
async def remove_recipe(self, recipe_id: str) -> bool:
"""Remove a recipe from the cache by ID."""
@@ -1260,9 +1371,15 @@ class RecipeScanner:
# Update FTS index
self._update_fts_index_for_recipe(recipe_id, "remove")
# Remove any image_id entry pointing to this recipe
stale = [k for k, v in cache.image_id_map.items() if v == recipe_id]
for k in stale:
del cache.image_id_map[k]
# Remove from SQLite cache
if self._persistent_cache:
self._persistent_cache.remove_recipe(recipe_id)
self._persistent_cache.save_image_id_map(cache.image_id_map)
self._json_path_map.pop(recipe_id, None)
return True
@@ -1273,14 +1390,21 @@ class RecipeScanner:
cache = await self.get_cached_data()
removed = await cache.bulk_remove(recipe_ids, resort=False)
if removed:
removed_ids = {str(r.get("id", "")) for r in removed}
stale = [k for k, v in cache.image_id_map.items() if v in removed_ids]
for k in stale:
del cache.image_id_map[k]
self._schedule_resort()
# Update FTS index and persistent cache for each removed recipe
for recipe in removed:
recipe_id = str(recipe.get("id", ""))
self._update_fts_index_for_recipe(recipe_id, "remove")
if self._persistent_cache:
self._persistent_cache.remove_recipe(recipe_id)
self._json_path_map.pop(recipe_id, None)
if self._persistent_cache:
self._persistent_cache.save_image_id_map(cache.image_id_map)
return len(removed)
async def scan_all_recipes(self) -> List[Dict]:
@@ -1815,6 +1939,15 @@ class RecipeScanner:
return await self._lora_scanner.get_model_info_by_name(name)
async def get_local_checkpoint(self, name: str) -> Optional[Dict[str, Any]]:
"""Lookup a local checkpoint model by name."""
checkpoint_scanner = getattr(self, "_checkpoint_scanner", None)
if not checkpoint_scanner or not name:
return None
return await checkpoint_scanner.get_model_info_by_name(name)
async def get_paginated_data(
self,
page: int,
@@ -2475,6 +2608,7 @@ class RecipeScanner:
continue
file_name = None
folder = ""
hash_value = (lora.get("hash") or "").lower()
if (
hash_value
@@ -2484,6 +2618,11 @@ class RecipeScanner:
file_path = self._lora_scanner._hash_index.get_path(hash_value)
if file_path:
file_name = os.path.splitext(os.path.basename(file_path))[0]
if lora_cache is not None:
for cached_lora in getattr(lora_cache, "raw_data", []):
if cached_lora.get("file_path") == file_path:
folder = cached_lora.get("folder", "")
break
if not file_name and lora.get("modelVersionId") and lora_cache is not None:
for cached_lora in getattr(lora_cache, "raw_data", []):
@@ -2498,13 +2637,16 @@ class RecipeScanner:
file_name = os.path.splitext(os.path.basename(cached_path))[
0
]
folder = cached_lora.get("folder", "")
break
if not file_name:
file_name = lora.get("file_name", "unknown-lora")
folder = lora.get("folder", "")
lora_name = f"{folder}/{file_name}" if folder else file_name
strength = lora.get("strength", 1.0)
syntax_parts.append(f"<lora:{file_name}:{strength}>")
syntax_parts.append(f"<lora:{lora_name}:{strength}>")
return syntax_parts
+143 -9
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import asyncio
import base64
import io
import os
@@ -14,6 +15,7 @@ from PIL import Image
from ...utils.utils import calculate_recipe_fingerprint
from ...utils.civitai_utils import extract_civitai_image_id, rewrite_preview_url
from ...recipes.enrichment import RecipeEnricher
from .errors import (
RecipeDownloadError,
RecipeNotFoundError,
@@ -144,11 +146,38 @@ class RecipeAnalysisService:
):
metadata = metadata["meta"]
# Include modelVersionIds from root level if available
# Civitai API returns modelVersionIds at root level, not in meta
# Include modelVersionIds from root level if available.
# CivitAI API returns modelVersionIds at root level, not in meta.
# When meta is null (None), create a minimal dict so downstream
# parsers can still discover LoRAs and checkpoints.
model_version_ids = image_info.get("modelVersionIds")
if model_version_ids and isinstance(metadata, dict):
metadata["modelVersionIds"] = model_version_ids
if model_version_ids:
if isinstance(metadata, dict):
metadata["modelVersionIds"] = model_version_ids
else:
metadata = {"modelVersionIds": model_version_ids}
# Inject browsingLevel (canonical integer) so the recipe's
# preview_nsfw_level can be set, enabling proper NSFW blur
# of the preview image. Fall back to nsfwLevel (string)
# when browsingLevel is absent.
if isinstance(metadata, dict):
browsing_level = image_info.get("browsingLevel")
nsfw_level_str = image_info.get("nsfwLevel")
if isinstance(browsing_level, int) and browsing_level > 0:
metadata["browsingLevel"] = browsing_level
elif (
isinstance(nsfw_level_str, str)
and nsfw_level_str
in (
"PG", "PG13", "R", "X", "XXX", "Blocked",
)
):
from ...utils.constants import NSFW_LEVELS
metadata["browsingLevel"] = NSFW_LEVELS.get(
nsfw_level_str, 0
)
# Validate that metadata contains meaningful recipe fields
# If not, treat as None to trigger EXIF extraction from downloaded image
@@ -169,17 +198,116 @@ class RecipeAnalysisService:
temp_path = self._create_temp_path(suffix=extension)
await self._download_image(url, temp_path)
if metadata is None and not is_video:
metadata = self._exif_utils.extract_image_metadata(temp_path)
# Always extract EXIF from the downloaded image for generation
# params (prompt, negative prompt, sampler, steps, etc.).
# Previously this was gated on ``metadata is None``, but that
# skipped EXIF entirely when API metadata (modelVersionIds,
# browsingLevel) is present, losing all generation parameters.
exif_metadata = None
if not is_video:
exif_metadata = await asyncio.to_thread(
self._exif_utils.extract_image_metadata, temp_path
)
return await self._parse_metadata(
metadata or {},
# Fallback: try the original (non-optimized) image for EXIF data
if not exif_metadata and civitai_image_id and image_info:
original_url = image_info.get("url")
if original_url:
self._logger.debug(
"Optimized image lacks embedded metadata, "
"falling back to original image: %s",
original_url,
)
orig_temp_path = self._create_temp_path(suffix=".png")
try:
await self._download_image(original_url, orig_temp_path)
exif_metadata = await asyncio.to_thread(
self._exif_utils.extract_image_metadata,
orig_temp_path,
)
finally:
self._safe_cleanup(orig_temp_path)
# Parse EXIF data (typically a string like parameters/prompt/workflow)
# and API metadata (dict with modelVersionIds, browsingLevel) separately,
# then merge: API loras/checkpoint override, EXIF gen_params fill in gaps.
# This mirrors the two-pass approach in _do_import_from_url.
exif_parsed_result = None
if isinstance(exif_metadata, str):
exif_parser = self._recipe_parser_factory.create_parser(exif_metadata)
if exif_parser:
exif_data = await exif_parser.parse_metadata(
exif_metadata, recipe_scanner=recipe_scanner,
)
if exif_data and not exif_data.get("error"):
exif_parsed_result = exif_data
# Merge API metadata (dict) with EXIF data (if dict) for the
# CivitaiApiMetadataParser. If EXIF data is a string it was
# parsed above — don't try to merge a string into a dict.
merged = {}
if isinstance(exif_metadata, dict):
merged.update(exif_metadata)
if isinstance(metadata, dict):
merged.update(metadata)
result = await self._parse_metadata(
merged,
recipe_scanner=recipe_scanner,
image_path=temp_path,
include_image_base64=True,
is_video=is_video,
extension=extension,
)
# Merge EXIF string-parsed gen_params into the API result.
# API gen_params take priority (they come later via update).
if exif_parsed_result and not result.payload.get("error"):
exif_gp = exif_parsed_result.get("gen_params") or {}
result_gp = result.payload.get("gen_params") or {}
merged_gp = {**exif_gp, **result_gp}
if merged_gp:
result.payload["gen_params"] = merged_gp
if civitai_image_id and image_info and not result.payload.get("error"):
# Use the metadata dict we built (may contain modelVersionIds
# and browsingLevel from the API root level). Do NOT pass
# image_info.get("meta") — it is null for images whose meta
# lives at the root level only. Also do NOT derive
# model_version_id from modelVersionIds[0] — that array mixes
# checkpoints, LoRAs, and other types without ordering
# guarantees; the parser already resolved them correctly.
recipe_for_enrich = {
"gen_params": result.payload.get("gen_params", {}),
"loras": result.payload.get("loras", []),
"base_model": result.payload.get("base_model", "") or "",
"checkpoint": result.payload.get("checkpoint") or result.payload.get("model"),
"source_path": url,
}
await RecipeEnricher.enrich_recipe(
recipe=recipe_for_enrich,
civitai_client=civitai_client,
request_params=None,
prefetched_civitai_meta_raw=(
metadata if isinstance(metadata, dict) else None
),
prefetched_model_version_id=None,
)
result.payload["gen_params"] = recipe_for_enrich["gen_params"]
if recipe_for_enrich.get("checkpoint"):
result.payload["checkpoint"] = recipe_for_enrich["checkpoint"]
if recipe_for_enrich.get("base_model"):
result.payload["base_model"] = recipe_for_enrich["base_model"]
# Extract browsingLevel from our constructed metadata for NSFW blur
if isinstance(metadata, dict):
bl = metadata.get("browsingLevel")
if isinstance(bl, int) and bl > 0:
result.payload["preview_nsfw_level"] = bl
return result
finally:
if temp_path:
self._safe_cleanup(temp_path)
@@ -199,7 +327,9 @@ class RecipeAnalysisService:
if not os.path.isfile(normalized_path):
raise RecipeNotFoundError("File not found")
metadata = self._exif_utils.extract_image_metadata(normalized_path)
metadata = await asyncio.to_thread(
self._exif_utils.extract_image_metadata, normalized_path
)
if not metadata:
return self._metadata_not_found_response(normalized_path)
@@ -259,6 +389,10 @@ class RecipeAnalysisService:
"prompt_type",
"positive",
"negative",
# modelVersionIds is injected at the root level by CivitAI's image
# API when meta is null. It carries the version IDs of ALL models
# (checkpoint + LoRAs) used to generate the image.
"modelVersionIds",
}
return any(field in metadata for field in recipe_fields)
+121 -5
View File
@@ -49,8 +49,18 @@ class RecipePersistenceService:
tags: Iterable[str],
metadata: Optional[dict[str, Any]],
extension: str | None = None,
recipe_id: str | None = None,
target_dir: str | None = None,
) -> PersistenceResult:
"""Persist a user uploaded recipe."""
"""Persist a user uploaded recipe.
Args:
recipe_id: If provided, reuse this ID instead of generating a new
UUID. Used by re-import to preserve the original recipe identity.
target_dir: If provided, save recipe files to this directory instead
of the default recipes_dir. Used by re-import to preserve the
original folder location.
"""
missing_fields = []
if not name:
@@ -63,10 +73,10 @@ class RecipePersistenceService:
)
resolved_image_bytes = self._resolve_image_bytes(image_bytes, image_base64)
recipes_dir = recipe_scanner.recipes_dir
recipes_dir = target_dir or recipe_scanner.recipes_dir
os.makedirs(recipes_dir, exist_ok=True)
recipe_id = str(uuid.uuid4())
recipe_id = recipe_id or str(uuid.uuid4())
# Handle video formats by bypassing optimization and metadata embedding
is_video = extension in [".mp4", ".webm"]
@@ -115,6 +125,22 @@ class RecipePersistenceService:
if metadata.get("source_path"):
recipe_data["source_path"] = metadata.get("source_path")
nsfw_level = metadata.get("preview_nsfw_level")
if nsfw_level is not None and isinstance(nsfw_level, int):
recipe_data["preview_nsfw_level"] = nsfw_level
# Compute recipe folder relative to recipes root, mirroring
# RecipeScanner._calculate_folder() which is only called during scan/load.
if recipe_scanner.recipes_dir:
recipe_file_dir = os.path.dirname(normalized_image_path)
try:
relative_folder = os.path.relpath(recipe_file_dir, recipe_scanner.recipes_dir)
if relative_folder in (".", ""):
relative_folder = ""
recipe_data["folder"] = relative_folder.replace(os.path.sep, "/")
except Exception:
recipe_data["folder"] = ""
json_filename = f"{recipe_id}.recipe.json"
json_path = os.path.join(recipes_dir, json_filename)
json_path = os.path.normpath(json_path)
@@ -508,6 +534,10 @@ class RecipePersistenceService:
most_common_base_model = (
max(base_model_counts.items(), key=lambda item: item[1])[0] if base_model_counts else ""
)
checkpoint_entry = await self._build_widget_checkpoint_entry(
recipe_scanner,
metadata.get("checkpoint"),
)
recipe_data = {
"id": recipe_id,
@@ -515,9 +545,8 @@ class RecipePersistenceService:
"title": recipe_name,
"modified": time.time(),
"created_date": time.time(),
"base_model": most_common_base_model,
"base_model": most_common_base_model or (checkpoint_entry or {}).get("baseModel", ""),
"loras": loras_data,
"checkpoint": self._sanitize_checkpoint_entry(metadata.get("checkpoint", "")),
"gen_params": {
key: value
for key, value in metadata.items()
@@ -525,6 +554,8 @@ class RecipePersistenceService:
},
"loras_stack": lora_stack,
}
if checkpoint_entry:
recipe_data["checkpoint"] = checkpoint_entry
json_filename = f"{recipe_id}.recipe.json"
json_path = os.path.join(recipes_dir, json_filename)
@@ -546,6 +577,91 @@ class RecipePersistenceService:
# Helper methods ---------------------------------------------------
async def _build_widget_checkpoint_entry(
self,
recipe_scanner,
checkpoint_raw: Any,
) -> Optional[dict[str, Any]]:
"""Build recipe checkpoint metadata from widget generation metadata."""
if isinstance(checkpoint_raw, dict):
return self._sanitize_checkpoint_entry(checkpoint_raw)
if not isinstance(checkpoint_raw, str):
return None
checkpoint_name = checkpoint_raw.strip()
if not checkpoint_name:
return None
file_name = os.path.splitext(os.path.basename(checkpoint_name))[0]
checkpoint_info = await self._lookup_widget_checkpoint(
recipe_scanner,
checkpoint_name,
)
if not checkpoint_info:
return {
"type": "checkpoint",
"name": checkpoint_name,
"file_name": file_name,
"hash": "",
}
civitai = checkpoint_info.get("civitai") or {}
civitai_model = civitai.get("model") or {}
file_path = checkpoint_info.get("file_path") or checkpoint_info.get("path") or ""
cached_file_name = (
checkpoint_info.get("file_name")
or (os.path.splitext(os.path.basename(file_path))[0] if file_path else "")
or file_name
)
return {
"type": "checkpoint",
"modelId": civitai_model.get("id", 0),
"modelVersionId": civitai.get("id", 0),
"name": civitai_model.get("name") or checkpoint_info.get("model_name") or checkpoint_name,
"version": civitai.get("name", ""),
"hash": (checkpoint_info.get("sha256") or checkpoint_info.get("hash") or "").lower(),
"file_name": cached_file_name,
"modelName": civitai_model.get("name", ""),
"modelVersionName": civitai.get("name", ""),
"baseModel": checkpoint_info.get("base_model") or civitai.get("baseModel", ""),
}
async def _lookup_widget_checkpoint(
self,
recipe_scanner,
checkpoint_name: str,
) -> Optional[dict[str, Any]]:
lookup = getattr(recipe_scanner, "get_local_checkpoint", None)
if not callable(lookup):
return None
candidates = []
for candidate in (
checkpoint_name,
os.path.basename(checkpoint_name),
os.path.splitext(os.path.basename(checkpoint_name))[0],
):
if candidate and candidate not in candidates:
candidates.append(candidate)
for candidate in candidates:
try:
checkpoint_info = await lookup(candidate)
except Exception as exc:
self._logger.debug(
"Failed to lookup checkpoint %s while saving widget recipe: %s",
candidate,
exc,
)
continue
if checkpoint_info:
return checkpoint_info
return None
def _extract_checkpoint_entry(self, metadata: dict[str, Any]) -> Optional[dict[str, Any]]:
"""Pull a checkpoint entry from various metadata locations."""
+19
View File
@@ -188,6 +188,25 @@ class ServiceRegistry:
logger.debug(f"Created and registered {service_name}")
return service
@classmethod
async def get_download_queue_service(cls):
"""Get or create the download queue service."""
service_name = "download_queue_service"
if service_name in cls._services:
return cls._services[service_name]
async with cls._get_lock(service_name):
if service_name in cls._services:
return cls._services[service_name]
from .download_queue_service import DownloadQueueService
service = await DownloadQueueService.get_instance()
cls._services[service_name] = service
logger.debug(f"Created and registered {service_name}")
return service
@classmethod
async def get_backup_service(cls):
"""Get or create the backup service."""
+110 -51
View File
@@ -2,6 +2,7 @@ import asyncio
import copy
import json
import os
import posixpath
import shutil
import tempfile
import logging
@@ -55,6 +56,8 @@ DEFAULT_KEYS_CLEANUP_THRESHOLD = 10
DEFAULT_SETTINGS: Dict[str, Any] = {
"civitai_api_key": "",
"civitai_host": "civitai.com",
"download_backend": "python",
"aria2c_path": "",
"use_portable_settings": False,
"hash_chunk_size_mb": DEFAULT_HASH_CHUNK_SIZE_MB,
"language": "en",
@@ -78,6 +81,9 @@ DEFAULT_SETTINGS: Dict[str, Any] = {
"folder_paths": {},
"extra_folder_paths": {},
"example_images_path": "",
"example_images_open_mode": "system",
"example_images_local_root": "",
"example_images_open_uri_template": "",
"optimize_example_images": True,
"auto_download_example_images": False,
"blur_mature_content": True,
@@ -85,22 +91,34 @@ DEFAULT_SETTINGS: Dict[str, Any] = {
"autoplay_on_hover": False,
"display_density": "default",
"card_info_display": "always",
"show_folder_sidebar": True,
"include_trigger_words": False,
"compact_mode": False,
"priority_tags": DEFAULT_PRIORITY_TAG_CONFIG.copy(),
"model_name_display": "model_name",
"lora_syntax_format": "legacy",
"model_card_footer_action": "replace_preview",
"update_flag_strategy": "same_base",
"show_version_on_card": True,
"version_grouping": "same_base",
"auto_organize_exclusions": [],
"metadata_refresh_skip_paths": [],
"skip_previously_downloaded_model_versions": False,
"download_skip_base_models": [],
"backup_auto_enabled": True,
"backup_retention_count": 5,
"use_new_license_icons": True,
"group_by_model": False,
}
def _normalize_root_identity(path: str) -> str:
"""Normalize a root path for equality checks across slash styles."""
normalized = posixpath.normpath(path.strip().replace("\\", "/"))
if len(normalized) >= 2 and normalized[1] == ":":
return normalized.lower()
return normalized
class SettingsManager:
def __init__(self):
self.settings_file = ensure_settings_file(logger)
@@ -117,6 +135,9 @@ class SettingsManager:
self._template_path = (
Path(__file__).resolve().parents[2] / "settings.json.example"
)
# Known placeholder value in settings.json.example; any file containing
# this value should be treated as "not configured".
self._TEMPLATE_PLACEHOLDER_API_KEY = "your_civitai_api_key_here"
self.settings = self._load_settings()
self._migrate_setting_keys()
self._ensure_default_settings()
@@ -148,6 +169,12 @@ class SettingsManager:
self._original_disk_payload = copy.deepcopy(data)
if self._matches_template_payload(data):
self._preserve_disk_template = True
# Clean up the template placeholder so it is not treated
# as a real key (affects both the frontend boolean and
# the downloader's Authorization header).
placeholder = self._TEMPLATE_PLACEHOLDER_API_KEY
if data.get("civitai_api_key") == placeholder:
data["civitai_api_key"] = ""
return data
except json.JSONDecodeError as exc:
logger.error("Failed to parse settings.json: %s", exc)
@@ -718,6 +745,7 @@ class SettingsManager:
"includeTriggerWords": "include_trigger_words",
"compactMode": "compact_mode",
"modelCardFooterAction": "model_card_footer_action",
"update_flag_strategy": "version_grouping",
}
updated = False
@@ -761,34 +789,29 @@ class SettingsManager:
if self._preserve_disk_template:
return
folder_paths = self.settings.get("folder_paths", {})
updated = False
def _check_and_auto_set(key: str, setting_key: str) -> bool:
"""Repair default roots when empty or no longer present."""
current = self.settings.get(setting_key, "")
candidates = folder_paths.get(key, [])
if not isinstance(candidates, list) or not candidates:
primary_candidates = self._get_valid_root_candidates(key)
if not primary_candidates:
return False
# Filter valid path strings
valid_paths = [p for p in candidates if isinstance(p, str) and p.strip()]
if not valid_paths:
allowed_roots = self._get_allowed_roots(key)
if current and _normalize_root_identity(current) in allowed_roots:
return False
if current in valid_paths:
return False
self.settings[setting_key] = valid_paths[0]
self.settings[setting_key] = primary_candidates[0]
if current:
logger.info(
"Repaired stale %s from '%s' to '%s'",
"Repaired stale %s from '%s' to '%s' because it is not present in primary or extra roots",
setting_key,
current,
valid_paths[0],
primary_candidates[0],
)
else:
logger.info("Auto-set %s to '%s'", setting_key, valid_paths[0])
logger.info("Auto-set %s to '%s'", setting_key, primary_candidates[0])
return True
# Process all model types
@@ -811,6 +834,36 @@ class SettingsManager:
else:
self._save_settings()
def _get_valid_root_candidates(self, key: str) -> List[str]:
"""Return stable root candidates, preferring primary roots over extra roots."""
candidates: List[str] = []
seen: set[str] = set()
for mapping_key in ("folder_paths", "extra_folder_paths"):
raw_paths = self.settings.get(mapping_key, {})
if not isinstance(raw_paths, Mapping):
continue
values = raw_paths.get(key, [])
if not isinstance(values, list):
continue
for value in values:
if not isinstance(value, str):
continue
normalized = value.strip()
if not normalized:
continue
identity = _normalize_root_identity(normalized)
if identity in seen:
continue
seen.add(identity)
candidates.append(normalized)
return candidates
def _get_allowed_roots(self, key: str) -> set[str]:
"""Return all valid roots for a model type, including extra roots."""
return {_normalize_root_identity(path) for path in self._get_valid_root_candidates(key)}
def _check_environment_variables(self) -> None:
"""Check for environment variables and update settings if needed"""
env_api_key = os.environ.get("CIVITAI_API_KEY")
@@ -1515,7 +1568,7 @@ class SettingsManager:
previous_dir = os.path.dirname(previous_path) or target_dir
if os.path.abspath(previous_path) != os.path.abspath(target_path):
self._copy_model_cache_directory(previous_dir, target_dir)
self._migrate_settings_directory_content(previous_dir, target_dir)
logger.info("Switching settings file to: %s", target_path)
self._pending_portable_switch = {"other_path": other_path}
@@ -1550,46 +1603,52 @@ class SettingsManager:
finally:
self._pending_portable_switch = None
def _copy_model_cache_directory(self, source_dir: str, target_dir: str) -> None:
"""Copy model_cache artifacts when switching storage locations."""
def _migrate_settings_directory_content(
self, source_dir: str, target_dir: str
) -> None:
"""Migrate settings directory subdirectories when switching storage locations.
Copies the canonical subdirectories (cache, backups, logs, stats, wildcards)
from the old settings directory to the new one. Legacy cache artifacts
(model_cache, recipe_cache, etc.) are migrated lazily by
``resolve_cache_path_with_migration`` on first access.
Args:
source_dir: The previous settings directory path.
target_dir: The new settings directory path.
"""
if not source_dir or not target_dir:
return
source_cache_dir = os.path.join(source_dir, "model_cache")
target_cache_dir = os.path.join(target_dir, "model_cache")
if os.path.isdir(source_cache_dir) and os.path.abspath(
source_cache_dir
) != os.path.abspath(target_cache_dir):
try:
shutil.copytree(
source_cache_dir,
target_cache_dir,
dirs_exist_ok=True,
ignore=shutil.ignore_patterns("*.sqlite-shm", "*.sqlite-wal"),
)
except Exception as exc:
logger.warning(
"Failed to copy model_cache directory from %s to %s: %s",
source_cache_dir,
target_cache_dir,
exc,
)
def _copy_dir(name: str) -> None:
source = os.path.join(source_dir, name)
target = os.path.join(target_dir, name)
if os.path.isdir(source) and os.path.abspath(source) != os.path.abspath(
target
):
try:
shutil.copytree(
source,
target,
dirs_exist_ok=True,
ignore=shutil.ignore_patterns("*.sqlite-shm", "*.sqlite-wal"),
)
except Exception as exc:
logger.warning(
"Failed to copy directory %s from %s to %s: %s",
name,
source,
target,
exc,
)
source_cache_file = os.path.join(source_dir, "model_cache.sqlite")
target_cache_file = os.path.join(target_dir, "model_cache.sqlite")
if os.path.isfile(source_cache_file) and os.path.abspath(
source_cache_file
) != os.path.abspath(target_cache_file):
try:
shutil.copy2(source_cache_file, target_cache_file)
except Exception as exc:
logger.warning(
"Failed to copy model_cache.sqlite from %s to %s: %s",
source_cache_file,
target_cache_file,
exc,
)
# Managed subdirectories under settings_dir
_copy_dir("cache")
_copy_dir("backups")
_copy_dir("logs")
_copy_dir("stats")
_copy_dir("wildcards")
def _get_user_config_directory(self) -> str:
"""Return the user configuration directory, falling back to ~/.config."""
+9 -7
View File
@@ -4,7 +4,9 @@ from __future__ import annotations
import os
from typing import Awaitable, Callable, Dict, List, Sequence
from typing import Awaitable, Callable, Dict, List, Sequence, Tuple
from .auto_tag_service import extract_auto_tags
class TagUpdateService:
@@ -20,9 +22,8 @@ class TagUpdateService:
new_tags: Sequence[str],
metadata_loader: Callable[[str], Awaitable[Dict[str, object]]],
update_cache: Callable[[str, str, Dict[str, object]], Awaitable[bool]],
) -> List[str]:
"""Add tags to a metadata entry while keeping case-insensitive uniqueness."""
) -> Tuple[List[str], List[str]]:
"""Add tags to a metadata entry and return updated tags and auto_tags."""
base, _ = os.path.splitext(file_path)
metadata_path = f"{base}.metadata.json"
metadata = await metadata_loader(metadata_path)
@@ -35,14 +36,15 @@ class TagUpdateService:
if isinstance(tag, str) and tag.strip():
# Convert all tags to lowercase to avoid case sensitivity issues on Windows
normalized = tag.strip().lower()
if normalized.lower() not in existing_lower:
if normalized not in existing_lower:
existing_tags.append(normalized)
existing_lower.append(normalized.lower())
existing_lower.append(normalized)
tags_added.append(normalized)
metadata["tags"] = existing_tags
await self._metadata_manager.save_metadata(file_path, metadata)
await update_cache(file_path, file_path, metadata)
return existing_tags
auto_tags = extract_auto_tags(metadata)
return existing_tags, auto_tags
@@ -3,6 +3,7 @@
from __future__ import annotations
import logging
import time
from typing import Any, Dict, List, Optional, Protocol, Sequence
from ..metadata_sync_service import MetadataSyncService
@@ -62,26 +63,48 @@ class BulkMetadataRefreshUseCase:
]
total_to_process = len(to_process)
initial_skipped = total_models - total_to_process # models excluded from fetch queue
processed = 0
success = 0
skipped_count = initial_skipped
handled_count = initial_skipped
needs_resort = False
start_time = time.monotonic()
failures: List[Dict[str, str]] = []
self._service.scanner.reset_cancellation()
async def emit(status: str, **extra: Any) -> None:
if progress_callback is None:
return
payload = {"status": status, "total": total_to_process, "processed": processed, "success": success}
payload = {
"status": status,
"total": total_models,
"processed": processed,
"success": success,
"failure_count": len(failures),
"skipped_count": skipped_count,
"handled": handled_count,
"elapsed_seconds": int(time.monotonic() - start_time),
}
# Only include full failure details in terminal emits (completed,
# cancelled, rate_limited) to avoid serializing the list on every
# per-model progress update.
if failures and status in ("completed", "cancelled", "rate_limited"):
payload["failures"] = failures
payload.update(extra)
await progress_callback.on_progress(payload)
await emit("started")
RATE_LIMIT_ABORT_THRESHOLD = 3
consecutive_rate_limits = 0
for model in to_process:
if self._service.scanner.is_cancelled():
self._logger.info("Bulk metadata refresh cancelled by user")
await emit("cancelled", processed=processed, success=success)
return {"success": False, "message": "Operation cancelled", "processed": processed, "updated": success, "total": total_models}
return {"success": False, "message": "Operation cancelled", "processed": processed, "updated": success, "total": total_models, "failures": failures, "failure_count": len(failures), "skipped_count": skipped_count, "elapsed_seconds": int(time.monotonic() - start_time)}
try:
original_name = model.get("model_name")
@@ -101,31 +124,76 @@ class BulkMetadataRefreshUseCase:
model["hash_status"] = "completed"
else:
self._logger.error(f"Failed to calculate hash for {file_path}")
failures.append({"name": model.get("model_name", file_path or "Unknown"), "error": "Failed to calculate hash"})
processed += 1
handled_count += 1
continue
else:
self._logger.warning(f"Scanner does not support lazy hash calculation for {file_path}")
skipped_count += 1
processed += 1
handled_count += 1
continue
# Skip models without valid hash
if not model.get("sha256"):
self._logger.warning(f"Skipping model without hash: {file_path}")
skipped_count += 1
processed += 1
handled_count += 1
continue
await MetadataManager.hydrate_model_data(model)
result, _ = await self._metadata_sync.fetch_and_update_model(
result, error_msg = await self._metadata_sync.fetch_and_update_model(
sha256=model["sha256"],
file_path=model["file_path"],
model_data=model,
update_cache_func=self._service.scanner.update_single_model_cache,
)
if not result and error_msg and "Rate limited" in error_msg:
consecutive_rate_limits += 1
else:
consecutive_rate_limits = 0
if not result:
current_name = model.get("model_name", file_path or "Unknown")
failures.append({"name": current_name, "error": error_msg or "Unknown error"})
self._logger.warning("Failed to fetch metadata for %s: %s", current_name, error_msg)
if consecutive_rate_limits >= RATE_LIMIT_ABORT_THRESHOLD:
# The current model was attempted and failed due to rate limiting;
# count it before aborting so the summary is consistent.
processed += 1
handled_count += 1
self._logger.warning(
"Bulk metadata refresh aborted: %d consecutive rate limits detected. "
"Processed %d/%d models.",
consecutive_rate_limits,
processed,
total_to_process,
)
await emit(
"rate_limited",
)
return {
"success": False,
"message": f"Rate limit detected; {total_to_process - processed} models skipped",
"processed": processed,
"updated": success,
"total": total_models,
"failures": failures,
"failure_count": len(failures),
"skipped_count": skipped_count,
"elapsed_seconds": int(time.monotonic() - start_time),
}
if result:
success += 1
if original_name != model.get("model_name"):
needs_resort = True
processed += 1
handled_count += 1
await emit(
"processing",
processed=processed,
@@ -134,6 +202,9 @@ class BulkMetadataRefreshUseCase:
)
except Exception as exc: # pragma: no cover - logging path
processed += 1
handled_count += 1
current_name = model.get("model_name", model.get("file_path", "Unknown"))
failures.append({"name": current_name, "error": str(exc)})
self._logger.error(
"Error fetching CivitAI data for %s: %s",
model.get("file_path"),
@@ -150,7 +221,7 @@ class BulkMetadataRefreshUseCase:
f"{success} of {processed} processed {self._service.model_type}s (total: {total_models})"
)
return {"success": True, "message": message, "processed": processed, "updated": success, "total": total_models}
return {"success": True, "message": message, "processed": processed, "updated": success, "total": total_models, "failures": failures, "failure_count": len(failures), "skipped_count": skipped_count, "elapsed_seconds": int(time.monotonic() - start_time)}
@staticmethod
def _is_in_skip_path(folder: str, skip_paths: List[str]) -> bool:
+46 -4
View File
@@ -7,7 +7,7 @@ from typing import Any, Dict, Iterable, Mapping, Sequence
from urllib.parse import parse_qs, urlparse, urlunparse
_SUPPORTED_CIVITAI_PAGE_HOSTS = frozenset({"civitai.com", "civitai.red"})
_SUPPORTED_CIVITAI_PAGE_HOSTS = frozenset({"civitai.com", "civitai.red", "civitai.green"})
DEFAULT_CIVITAI_PAGE_HOST = "civitai.com"
_DEFAULT_ALLOW_COMMERCIAL_USE: Sequence[str] = ("Sell",)
_LICENSE_DEFAULTS: Dict[str, Any] = {
@@ -66,6 +66,46 @@ def build_civitai_model_page_url(
return None
_RE_CDN_IMAGE_ID = re.compile(r"/(\d+)\.(?:jpeg|jpg|png|webp|gif)(?:\?|#|$)")
def extract_civitai_image_id_from_cdn_url(url: str | None) -> str | None:
"""Extract the numeric image ID from a Cloudflare CDN image URL.
CivitAI image CDN URLs follow the pattern::
https://image.civitai.com/{cf_uuid}/{params}/{image_id}.{ext}
The image database ID is always the last path segment (minus extension)
because ``getEdgeUrl(…, name=id.toString())`` embeds it explicitly
in the model-versions REST API response.
"""
if not url:
return None
match = _RE_CDN_IMAGE_ID.search(url)
return match.group(1) if match else None
def build_civitai_image_page_url(
image_id: str | int | None,
*,
host: str | None = None,
) -> str | None:
"""Build a Civitai image page URL.
Returns something like ``https://civitai.com/images/12345``.
The host is resolved through :func:`normalize_civitai_page_host` and
therefore respects the user's ``civitai_host`` setting.
"""
if not image_id:
return None
normalized_host = normalize_civitai_page_host(host)
normalized_id = str(image_id).strip()
if not normalized_id:
return None
return urlunparse(("https", normalized_host, f"/images/{normalized_id}", "", "", ""))
def _parse_supported_civitai_page_url(url: str | None):
if not url:
return None
@@ -239,9 +279,9 @@ def _resolve_commercial_bits(values: Sequence[str]) -> int:
normalized_values.add(normalized)
has_sell = "sell" in normalized_values
has_rent = has_sell or "rent" in normalized_values
has_rentcivit = has_rent or "rentcivit" in normalized_values
has_image = has_sell or "image" in normalized_values
has_rent = "rent" in normalized_values
has_rentcivit = "rentcivit" in normalized_values
has_image = "image" in normalized_values
commercial_bits = (
(1 if has_sell else 0) << 3
@@ -328,8 +368,10 @@ def rewrite_preview_url(
__all__ = [
"build_civitai_image_page_url",
"build_license_flags",
"extract_civitai_image_id",
"extract_civitai_image_id_from_cdn_url",
"extract_civitai_page_host",
"extract_civitai_model_url_parts",
"is_supported_civitai_page_host",
+58 -5
View File
@@ -31,6 +31,8 @@ PREVIEW_EXTENSIONS = [
".mp4",
".gif",
".webm",
".avif",
".jxl",
]
# Card preview image width
@@ -41,10 +43,24 @@ EXAMPLE_IMAGE_WIDTH = 832
# Supported media extensions for example downloads
SUPPORTED_MEDIA_EXTENSIONS = {
"images": [".jpg", ".jpeg", ".png", ".webp", ".gif"],
"images": [".jpg", ".jpeg", ".png", ".webp", ".gif", ".avif", ".jxl"],
"videos": [".mp4", ".webm"],
}
# Model weight file extensions recognised by scanners.
# This is the union of all scanner extensions (lora, checkpoint, embedding).
MODEL_FILE_EXTENSIONS = {
".safetensors",
".ckpt",
".pt",
".pt2",
".bin",
".pth",
".pkl",
".sft",
".gguf",
}
# Valid sub-types for each scanner type
VALID_LORA_SUB_TYPES = ["lora", "locon", "dora"]
VALID_CHECKPOINT_SUB_TYPES = ["checkpoint", "diffusion_model"]
@@ -100,8 +116,35 @@ DEFAULT_PRIORITY_TAG_CONFIG = {
# These model types are incorrectly labeled as "checkpoint" by CivitAI but are actually diffusion models
DIFFUSION_MODEL_BASE_MODELS = frozenset(
[
"ZImageTurbo",
"ZImageBase",
"Anima",
# Flux series — DiT architecture, loaded via UNETLoader in ComfyUI
"Flux.1 D",
"Flux.1 S",
"Flux.1 Krea",
"Flux.1 Kontext",
"Flux.2 D",
"Flux.2 Klein 9B",
"Flux.2 Klein 9B-base",
"Flux.2 Klein 4B",
"Flux.2 Klein 4B-base",
# Non-UNet / DiT image diffusion models
"AuraFlow",
"Chroma",
"HiDream",
"Hunyuan 1",
"Kolors",
"Lumina",
"PixArt a",
"PixArt E",
# Video diffusion models
"CogVideoX",
"Hunyuan Video",
"LTXV",
"LTXV2",
"LTXV 2.3",
"Mochi",
"SVD",
"Wan Video",
"Wan Video 1.3B t2v",
"Wan Video 14B t2v",
"Wan Video 14B i2v 480p",
@@ -111,9 +154,15 @@ DIFFUSION_MODEL_BASE_MODELS = frozenset(
"Wan Video 2.2 T2V-A14B",
"Wan Video 2.5 T2V",
"Wan Video 2.5 I2V",
"CogVideoX",
"Mochi",
# Other diffusion models
"Ernie",
"Ernie Turbo",
"Nucleus",
"Qwen",
"ZImageBase",
"ZImageTurbo",
# Krea 2 — loaded via UNETLoader in ComfyUI
"Krea 2",
]
)
@@ -177,5 +226,9 @@ SUPPORTED_DOWNLOAD_SKIP_BASE_MODELS = frozenset(
"Wan Video 2.5 I2V",
"Hunyuan Video",
"Anima",
"Ernie",
"Ernie Turbo",
"Nucleus",
"Krea 2",
]
)
+4 -5
View File
@@ -397,13 +397,12 @@ class DownloadManager:
models_with_hash = len(all_models_with_hash)
# Calculate pending count: check which models actually need processing
# A model is pending if it has a hash, is not in processed_models,
# and its folder doesn't exist or is empty
# Calculate pending count: check which models actually need processing.
# A model is pending if it has a hash, is not already processed or known-failed,
# and its folder doesn't exist or is empty.
pending_hashes = set()
for model_hash, model_name in all_models_with_hash:
if model_hash not in processed_models:
# Check if model folder exists with files
if model_hash not in processed_models and model_hash not in failed_models:
model_dir = ExampleImagePathResolver.get_model_folder(
model_hash, active_library
)
+103 -19
View File
@@ -1,17 +1,81 @@
import logging
import os
import sys
import re
import subprocess
import sys
from urllib.parse import quote
from aiohttp import web
from ..services.settings_manager import get_settings_manager
from ..utils.example_images_paths import (
get_model_folder,
get_model_relative_path,
)
from ..utils.constants import SUPPORTED_MEDIA_EXTENSIONS
logger = logging.getLogger(__name__)
_WINDOWS_DRIVE_PATTERN = re.compile(r"^[A-Za-z]:/")
def _is_within_root(path: str, root: str) -> bool:
try:
return os.path.commonpath([os.path.abspath(path), os.path.abspath(root)]) == os.path.abspath(root)
except ValueError:
return False
def _join_local_example_path(local_root: str, relative_path: str) -> str:
separator = "\\" if "\\" in local_root and "/" not in local_root else "/"
normalized_root = local_root.rstrip("\\/")
normalized_relative = relative_path.replace("/", separator)
if not normalized_root:
return normalized_relative
return f"{normalized_root}{separator}{normalized_relative}"
def _build_file_uri(path: str) -> str:
normalized = path.replace("\\", "/")
if _WINDOWS_DRIVE_PATTERN.match(normalized):
return f"file:///{quote(normalized, safe='/:')}"
if normalized.startswith("/"):
return f"file://{quote(normalized, safe='/:')}"
return f"file:///{quote(normalized.lstrip('/'), safe='/:')}"
def _render_open_uri_template(template: str, local_path: str, relative_path: str) -> str:
file_uri = _build_file_uri(local_path)
replacements = {
"{{local_path}}": local_path,
"{{encoded_local_path}}": quote(local_path, safe=""),
"{{relative_path}}": relative_path,
"{{encoded_relative_path}}": quote(relative_path, safe=""),
"{{file_uri}}": file_uri,
"{{encoded_file_uri}}": quote(file_uri, safe=""),
}
rendered = template
for placeholder, value in replacements.items():
rendered = rendered.replace(placeholder, value)
return rendered
def _open_system_folder(model_folder: str) -> dict[str, object]:
if os.name == "nt": # Windows
os.startfile(model_folder)
elif os.name == "posix": # macOS and Linux
if sys.platform == "darwin": # macOS
subprocess.Popen(["open", model_folder])
else: # Linux
subprocess.Popen(["xdg-open", model_folder])
return {
"success": True,
"message": f"Opened example images folder for {model_folder}",
"path": model_folder,
}
class ExampleImagesFileManager:
"""Manages access and operations for example image files"""
@@ -54,7 +118,7 @@ class ExampleImagesFileManager:
}, status=500)
# Path validation: ensure model_folder is under example_images_path
if not model_folder.startswith(os.path.abspath(example_images_path)):
if not _is_within_root(model_folder, example_images_path):
return web.json_response({
'success': False,
'error': 'Invalid model folder path'
@@ -66,20 +130,40 @@ class ExampleImagesFileManager:
'success': False,
'error': 'No example images found for this model. Download example images first.'
}, status=404)
# Open folder in file explorer
if os.name == 'nt': # Windows
os.startfile(model_folder)
elif os.name == 'posix': # macOS and Linux
if sys.platform == 'darwin': # macOS
subprocess.Popen(['open', model_folder])
else: # Linux
subprocess.Popen(['xdg-open', model_folder])
return web.json_response({
'success': True,
'message': f'Opened example images folder for model {model_hash}'
})
root_path = os.path.abspath(example_images_path)
relative_path = os.path.relpath(model_folder, root_path).replace("\\", "/")
open_mode = settings_manager.get("example_images_open_mode") or "system"
if open_mode == "clipboard":
local_root = settings_manager.get("example_images_local_root") or root_path
local_path = _join_local_example_path(local_root, relative_path)
return web.json_response({
'success': True,
'mode': 'clipboard',
'path': local_path,
'relative_path': relative_path,
})
if open_mode == "uri_template":
local_root = settings_manager.get("example_images_local_root") or root_path
uri_template = settings_manager.get("example_images_open_uri_template") or ""
if not uri_template.strip():
return web.json_response({
'success': False,
'error': 'No example image open URI template configured.'
}, status=400)
local_path = _join_local_example_path(local_root, relative_path)
return web.json_response({
'success': True,
'mode': 'uri',
'path': local_path,
'relative_path': relative_path,
'uri': _render_open_uri_template(uri_template, local_path, relative_path),
})
return web.json_response(_open_system_folder(model_folder))
except Exception as e:
logger.error(f"Failed to open example images folder: {e}", exc_info=True)
@@ -143,7 +227,7 @@ class ExampleImagesFileManager:
file_ext = os.path.splitext(file)[1].lower()
if (file_ext in SUPPORTED_MEDIA_EXTENSIONS['images'] or
file_ext in SUPPORTED_MEDIA_EXTENSIONS['videos']):
relative_path = get_model_relative_path(model_hash)
relative_path = os.path.relpath(model_folder, os.path.abspath(example_images_path)).replace("\\", "/")
files.append({
'name': file,
'path': f'/example_images_static/{relative_path}/{file}',
@@ -227,4 +311,4 @@ class ExampleImagesFileManager:
return web.json_response({
'has_images': False,
'error': str(e)
})
})
+108
View File
@@ -452,3 +452,111 @@ class MetadataUpdater:
except Exception as e:
logger.error(f"Error parsing image metadata: {e}", exc_info=True)
return None
@staticmethod
async def prune_stale_example_images(metadata) -> bool:
"""Remove example-image metadata entries whose files no longer exist on disk.
Checks ``civitai.customImages`` (by ``id``) and ``civitai.images`` entries
that have an empty ``url`` (no remote fallback) against actual files in
the model's example-image folder. Stale entries are removed in-place so
the caller can persist the cleaned metadata afterwards.
Args:
metadata: A ``BaseModelMetadata`` instance (modified in place).
Returns:
True if at least one entry was removed.
"""
from ..utils.example_images_paths import get_model_folder
model_hash = getattr(metadata, "sha256", None)
if not model_hash:
return False
model_folder = get_model_folder(model_hash)
if not model_folder:
return False
civitai = getattr(metadata, "civitai", None)
if not isinstance(civitai, dict):
return False
has_changes = False
custom_images = civitai.get("customImages")
if isinstance(custom_images, list) and custom_images:
stale: list[int] = []
for idx, img in enumerate(custom_images):
img_id = img.get("id", "")
if not img_id:
continue
if not os.path.isdir(model_folder):
stale.append(idx)
else:
found = False
try:
prefix = f"custom_{img_id}"
for fname in os.listdir(model_folder):
if fname.startswith(prefix) and os.path.isfile(
os.path.join(model_folder, fname)
):
found = True
break
except OSError:
stale.append(idx)
continue
if not found:
stale.append(idx)
if stale:
for idx in reversed(stale):
custom_images.pop(idx)
has_changes = True
logger.info(
"Pruned %d stale custom image(s) for %s",
len(stale),
getattr(metadata, "model_name", model_hash),
)
images = civitai.get("images")
if isinstance(images, list) and images:
stale: list[int] = []
for idx, img in enumerate(images):
if img.get("url", ""):
# Has a remote fallback – keep it even if the local copy
# is gone.
continue
if not os.path.isdir(model_folder):
stale.append(idx)
else:
found = False
try:
prefix = f"image_{idx}."
for fname in os.listdir(model_folder):
if fname.startswith(prefix):
found = True
break
except OSError:
stale.append(idx)
continue
if not found:
stale.append(idx)
if stale:
for idx in reversed(stale):
images.pop(idx)
has_changes = True
logger.info(
"Pruned %d stale image entry(ies) for %s",
len(stale),
getattr(metadata, "model_name", model_hash),
)
return has_changes
+70
View File
@@ -12,6 +12,18 @@ from ..services.settings_manager import get_settings_manager
_HEX_PATTERN = re.compile(r"[a-fA-F0-9]{64}")
# Filesystem/metadata files that are never created by the example images system
# and are safe to ignore during validation. The cleanup service only operates on
# directories, so these files pose no data-loss risk.
_SAFE_FILENAMES: frozenset[str] = frozenset({
".DS_Store", # macOS folder metadata
"Thumbs.db", # Windows thumbnail cache
"desktop.ini", # Windows folder customization
".localized", # macOS folder name localization
".gitkeep", # Placeholder to keep empty dirs in git
".gitignore", # Git ignore rules
})
logger = logging.getLogger(__name__)
@@ -180,6 +192,22 @@ def is_hash_folder(name: str) -> bool:
return bool(_HEX_PATTERN.fullmatch(name or ""))
def _is_safe_ignorable_entry(item: str, item_path: str) -> bool:
"""Return True if *item* is a harmless system/hidden file we can skip.
These files are never created by the example images system and are safe to
ignore because the cleanup/delete operations only act on **directories**,
never on individual files (other than ``.download_progress.json``).
"""
if item in _SAFE_FILENAMES:
return True
# Hide Unix hidden files (dotfiles) that are regular files,
# since the cleanup system never deletes or moves files.
if item.startswith(".") and os.path.isfile(item_path):
return True
return False
def is_valid_example_images_root(folder_path: str) -> bool:
"""Check whether a folder looks like a dedicated example images root."""
@@ -190,9 +218,16 @@ def is_valid_example_images_root(folder_path: str) -> bool:
for item in items:
item_path = os.path.join(folder_path, item)
# .download_progress.json is an expected metadata file — check before
# the generic dotfile rule so it stays explicitly documented.
if item == ".download_progress.json" and os.path.isfile(item_path):
continue
# Skip harmless system/hidden files — cleanup only touches directories
if _is_safe_ignorable_entry(item, item_path):
continue
if os.path.isdir(item_path):
if is_hash_folder(item):
continue
@@ -211,6 +246,41 @@ def is_valid_example_images_root(folder_path: str) -> bool:
return True
def find_non_compliant_items_in_example_images_root(folder_path: str) -> list[str]:
"""Return the names of items that prevent *folder_path* from being a valid
example images root, or an empty list if the folder is valid.
This mirrors ``is_valid_example_images_root`` but **returns** the offending
names instead of a boolean, so callers can produce actionable error messages.
"""
try:
items = os.listdir(folder_path)
except OSError as exc:
return [f"<cannot list directory: {exc}>"]
offending: list[str] = []
for item in items:
item_path = os.path.join(folder_path, item)
# Same skip rules as is_valid_example_images_root
if item == ".download_progress.json" and os.path.isfile(item_path):
continue
if _is_safe_ignorable_entry(item, item_path):
continue
if os.path.isdir(item_path):
if is_hash_folder(item):
continue
if item == "_deleted":
continue
if _library_folder_has_only_hash_dirs(item_path):
continue
offending.append(item)
return offending
def _library_folder_has_only_hash_dirs(path: str) -> bool:
"""Return True when a library subfolder only contains hash folders or metadata files."""
+6
View File
@@ -62,6 +62,10 @@ class ExampleImagesProcessor:
return '.gif'
elif content.startswith(b'RIFF') and b'WEBP' in content[:12]:
return '.webp'
elif len(content) >= 12 and content[4:8] == b'ftyp' and b'avif' in content[8:24]:
return '.avif'
elif content.startswith(b'\x00\x00\x00\x0cJXL \x0d\x0a\x87\x0a'):
return '.jxl'
elif content.startswith(b'\x00\x00\x00\x18ftypmp4') or content.startswith(b'\x00\x00\x00\x20ftypmp4'):
return '.mp4'
elif content.startswith(b'\x1A\x45\xDF\xA3'):
@@ -75,6 +79,8 @@ class ExampleImagesProcessor:
'image/png': '.png',
'image/gif': '.gif',
'image/webp': '.webp',
'image/avif': '.avif',
'image/jxl': '.jxl',
'video/mp4': '.mp4',
'video/webm': '.webm',
'video/quicktime': '.mov'
+278 -101
View File
@@ -1,15 +1,256 @@
import piexif
import json
import logging
from typing import Optional
from io import BytesIO
import os
import struct
from io import BytesIO
from typing import Any, Optional
import piexif
from PIL import Image, PngImagePlugin
try:
import brotli
_BROTLI_AVAILABLE = True
except ImportError:
brotli = None
_BROTLI_AVAILABLE = False
logger = logging.getLogger(__name__)
class ExifUtils:
"""Utility functions for working with EXIF data in images"""
@staticmethod
def _parse_isobmff_boxes(data: bytes, offset: int = 0) -> list[dict]:
boxes = []
while offset + 8 <= len(data):
size = struct.unpack('>I', data[offset:offset + 4])[0]
box_type = data[offset + 4:offset + 8]
if size == 0:
break
if size < 8 or offset + size > len(data):
break
box_data = data[offset + 8:offset + size]
boxes.append({'type': box_type, 'data': box_data, 'size': size})
offset += size
return boxes
@staticmethod
def _is_jxl_container(data: bytes) -> bool:
if len(data) < 32:
return False
return (
struct.unpack('>I', data[:4])[0] == 12
and data[4:8] == b'JXL '
and data[8:12] == bytes([0x0d, 0x0a, 0x87, 0x0a])
and struct.unpack('>I', data[12:16])[0] >= 16
and data[16:20] == b'ftyp'
and data[20:24] == b'jxl '
)
@staticmethod
def _is_avif_container(data: bytes) -> bool:
if len(data) < 16:
return False
for box in ExifUtils._parse_isobmff_boxes(data):
if box['type'] == b'ftyp' and b'avif' in box['data']:
return True
return False
# Max decompressed size for brotli metadata (2 MB)
_BROTLI_MAX_DECOMPRESSED = 2 * 1024 * 1024
@staticmethod
def _extract_isobmff_brotli(image_path: str) -> Optional[dict]:
try:
with open(image_path, 'rb') as f:
data = f.read()
except Exception:
return None
if ExifUtils._is_jxl_container(data):
boxes = ExifUtils._parse_isobmff_boxes(data, offset=12)
elif ExifUtils._is_avif_container(data):
boxes = ExifUtils._parse_isobmff_boxes(data)
else:
return None
brob = None
for box in boxes:
if box['type'] == b'brob':
brob = box
break
if brob is None:
return None
payload = brob['data']
if payload[:4] != b'comf':
return None
compressed = payload[4:]
if _BROTLI_AVAILABLE:
try:
decompressed = brotli.decompress(compressed)
if len(decompressed) > ExifUtils._BROTLI_MAX_DECOMPRESSED:
logger.warning(
"Brotli metadata too large (%d bytes, max %d), ignoring",
len(decompressed),
ExifUtils._BROTLI_MAX_DECOMPRESSED,
)
decompressed = None
except Exception:
decompressed = None
else:
decompressed = None
raw = decompressed if decompressed is not None else compressed
try:
meta = json.loads(raw.decode('utf-8'))
except Exception:
return None
result = {"parameters": None, "prompt": None, "workflow": None, "comment": None}
if isinstance(meta.get("prompt"), (dict, list)):
result["prompt"] = json.dumps(meta["prompt"])
elif isinstance(meta.get("prompt"), str):
result["prompt"] = meta["prompt"]
if isinstance(meta.get("workflow"), (dict, list)):
result["workflow"] = json.dumps(meta["workflow"])
elif isinstance(meta.get("workflow"), str):
result["workflow"] = meta["workflow"]
return result
@staticmethod
def _decode_user_comment(user_comment: Any) -> Optional[str]:
if user_comment is None:
return None
if isinstance(user_comment, bytes):
if user_comment.startswith(b"UNICODE\0"):
return user_comment[8:].decode("utf-16be", errors="ignore")
return user_comment.decode("utf-8", errors="ignore")
if isinstance(user_comment, str):
return user_comment
return str(user_comment)
@staticmethod
def _decode_exif_text(value: Any) -> Optional[str]:
if value is None:
return None
if isinstance(value, bytes):
return value.decode("utf-8", errors="ignore")
if isinstance(value, str):
return value
return str(value)
@staticmethod
def _load_structured_metadata(image_path: str) -> dict[str, Optional[str]]:
metadata = {
"parameters": None,
"prompt": None,
"workflow": None,
"comment": None,
}
ext = os.path.splitext(image_path)[1].lower()
if ext in ('.avif', '.jxl'):
brotli_meta = ExifUtils._extract_isobmff_brotli(image_path)
if brotli_meta:
return brotli_meta
with Image.open(image_path) as img:
info = getattr(img, "info", {}) or {}
if "parameters" in info:
metadata["parameters"] = info["parameters"]
if "prompt" in info:
metadata["prompt"] = info["prompt"]
if "workflow" in info:
metadata["workflow"] = info["workflow"]
if img.format not in ["JPEG", "TIFF", "WEBP"]:
exif = img.getexif()
if exif and piexif.ExifIFD.UserComment in exif:
metadata["comment"] = ExifUtils._decode_user_comment(
exif[piexif.ExifIFD.UserComment]
)
try:
exif_dict = piexif.load(image_path)
except Exception as e:
logger.debug(f"Error loading EXIF data: {e}")
exif_dict = {}
if piexif.ExifIFD.UserComment in exif_dict.get("Exif", {}):
metadata["comment"] = ExifUtils._decode_user_comment(
exif_dict["Exif"][piexif.ExifIFD.UserComment]
)
image_description = ExifUtils._decode_exif_text(
exif_dict.get("0th", {}).get(piexif.ImageIFD.ImageDescription)
)
if image_description:
if image_description.startswith("Workflow:"):
metadata["workflow"] = image_description[len("Workflow:") :]
elif not metadata["prompt"]:
metadata["prompt"] = image_description
if not metadata["parameters"] and metadata["comment"]:
metadata["parameters"] = metadata["comment"]
return metadata
@staticmethod
def _build_pnginfo(img: Image.Image, metadata_fields: dict[str, Optional[str]]) -> PngImagePlugin.PngInfo:
png_info = PngImagePlugin.PngInfo()
existing_info = getattr(img, "info", {}) or {}
managed_keys = {"parameters", "prompt", "workflow"}
for key, value in existing_info.items():
if key in {"exif", "dpi", "transparency", "gamma", "aspect"}:
continue
if key in managed_keys:
continue
if isinstance(value, str):
png_info.add_text(key, value)
for key in managed_keys:
value = metadata_fields.get(key)
if value:
png_info.add_text(key, value)
return png_info
@staticmethod
def _build_exif_bytes(
metadata_fields: dict[str, Optional[str]], existing_exif: bytes | None = None
) -> bytes:
try:
exif_dict = piexif.load(existing_exif or b"")
except Exception:
exif_dict = {"0th": {}, "Exif": {}, "GPS": {}, "Interop": {}, "1st": {}}
exif_dict.setdefault("0th", {})
exif_dict.setdefault("Exif", {})
parameters = metadata_fields.get("parameters")
workflow = metadata_fields.get("workflow")
prompt = metadata_fields.get("prompt")
if parameters:
exif_dict["Exif"][piexif.ExifIFD.UserComment] = (
b"UNICODE\0" + parameters.encode("utf-16be")
)
else:
exif_dict["Exif"].pop(piexif.ExifIFD.UserComment, None)
if workflow:
exif_dict["0th"][piexif.ImageIFD.ImageDescription] = f"Workflow:{workflow}"
elif prompt:
exif_dict["0th"][piexif.ImageIFD.ImageDescription] = prompt
else:
exif_dict["0th"].pop(piexif.ImageIFD.ImageDescription, None)
return piexif.dump(exif_dict)
@staticmethod
def extract_image_metadata(image_path: str) -> Optional[str]:
@@ -22,54 +263,17 @@ class ExifUtils:
Optional[str]: Extracted metadata or None if not found
"""
try:
# Skip for video files
if image_path:
ext = os.path.splitext(image_path)[1].lower()
if ext in ['.mp4', '.webm']:
return None
# First try to open the image
with Image.open(image_path) as img:
# Method 1: Check for parameters in image info
if hasattr(img, 'info') and 'parameters' in img.info:
return img.info['parameters']
# Method 2: Check EXIF UserComment field
if img.format not in ['JPEG', 'TIFF', 'WEBP']:
# For non-JPEG/TIFF/WEBP images, try to get EXIF through PIL
exif = img.getexif()
if exif and piexif.ExifIFD.UserComment in exif:
user_comment = exif[piexif.ExifIFD.UserComment]
if isinstance(user_comment, bytes):
if user_comment.startswith(b'UNICODE\0'):
return user_comment[8:].decode('utf-16be')
return user_comment.decode('utf-8', errors='ignore')
return user_comment
# For JPEG/TIFF/WEBP, use piexif
try:
exif_dict = piexif.load(image_path)
if piexif.ExifIFD.UserComment in exif_dict.get('Exif', {}):
user_comment = exif_dict['Exif'][piexif.ExifIFD.UserComment]
if isinstance(user_comment, bytes):
if user_comment.startswith(b'UNICODE\0'):
user_comment = user_comment[8:].decode('utf-16be')
else:
user_comment = user_comment.decode('utf-8', errors='ignore')
return user_comment
except Exception as e:
logger.debug(f"Error loading EXIF data: {e}")
# Method 3: Check PNG metadata for workflow info (for ComfyUI images)
if img.format == 'PNG':
# Look for workflow or prompt metadata in PNG chunks
for key in img.info:
if key in ['workflow', 'prompt', 'parameters']:
return img.info[key]
return None
metadata = ExifUtils._load_structured_metadata(image_path)
return (
metadata.get("parameters")
or metadata.get("prompt")
or metadata.get("workflow")
)
except Exception as e:
logger.error(f"Error extracting image metadata: {e}", exc_info=True)
return None
@@ -86,56 +290,31 @@ class ExifUtils:
str: Path to the updated image
"""
try:
# Skip for video files
if image_path:
ext = os.path.splitext(image_path)[1].lower()
if ext in ['.mp4', '.webm']:
if ext in ['.mp4', '.webm', '.avif', '.jxl']:
return image_path
# Load the image and check its format
metadata_fields = ExifUtils._load_structured_metadata(image_path)
metadata_fields["parameters"] = metadata
with Image.open(image_path) as img:
img_format = img.format
# For PNG, try to update parameters directly
if img_format == 'PNG':
# Use PngInfo instead of plain dictionary
png_info = PngImagePlugin.PngInfo()
png_info.add_text("parameters", metadata)
img.save(image_path, format='PNG', pnginfo=png_info)
if img_format == "PNG":
png_info = ExifUtils._build_pnginfo(img, metadata_fields)
img.save(image_path, format="PNG", pnginfo=png_info)
return image_path
# For WebP format, use PIL's exif parameter directly
elif img_format == 'WEBP':
exif_dict = {'Exif': {piexif.ExifIFD.UserComment: b'UNICODE\0' + metadata.encode('utf-16be')}}
exif_bytes = piexif.dump(exif_dict)
# Save with the exif data
img.save(image_path, format='WEBP', exif=exif_bytes, quality=85)
return image_path
# For other formats, use standard EXIF approach
else:
try:
exif_dict = piexif.load(img.info.get('exif', b''))
except:
exif_dict = {'0th':{}, 'Exif':{}, 'GPS':{}, 'Interop':{}, '1st':{}}
# If no Exif dictionary exists, create one
if 'Exif' not in exif_dict:
exif_dict['Exif'] = {}
# Update the UserComment field - use UNICODE format
unicode_bytes = metadata.encode('utf-16be')
metadata_bytes = b'UNICODE\0' + unicode_bytes
exif_dict['Exif'][piexif.ExifIFD.UserComment] = metadata_bytes
# Convert EXIF dict back to bytes
exif_bytes = piexif.dump(exif_dict)
# Save the image with updated EXIF data
img.save(image_path, exif=exif_bytes)
exif_bytes = ExifUtils._build_exif_bytes(
metadata_fields, img.info.get("exif")
)
save_kwargs = {"exif": exif_bytes}
if img_format == "WEBP":
save_kwargs["quality"] = 85
img.save(image_path, format=img_format, **save_kwargs)
return image_path
except Exception as e:
logger.error(f"Error updating metadata in {image_path}: {e}")
@@ -145,10 +324,9 @@ class ExifUtils:
def append_recipe_metadata(image_path, recipe_data) -> str:
"""Append recipe metadata to an image's EXIF data"""
try:
# Skip for video files
if image_path:
ext = os.path.splitext(image_path)[1].lower()
if ext in ['.mp4', '.webm']:
if ext in ['.mp4', '.webm', '.avif', '.jxl']:
return image_path
# First, extract existing metadata
@@ -260,10 +438,9 @@ class ExifUtils:
Tuple of (optimized_image_data, extension)
"""
try:
# Skip for video files early if it's a file path
if isinstance(image_data, str) and os.path.exists(image_data):
ext = os.path.splitext(image_data)[1].lower()
if ext in ['.mp4', '.webm']:
if ext in ['.mp4', '.webm', '.avif', '.jxl']:
try:
with open(image_data, 'rb') as f:
return f.read(), ext
@@ -297,12 +474,12 @@ class ExifUtils:
raise ValueError(f"Cannot process corrupt image data: {e}")
# Extract metadata if needed and valid
metadata = None
metadata_fields = None
if preserve_metadata:
try:
if isinstance(image_data, str) and os.path.exists(image_data):
# For file path, extract directly
metadata = ExifUtils.extract_image_metadata(image_data)
metadata_fields = ExifUtils._load_structured_metadata(image_data)
else:
# For binary data, save to temp file first
import tempfile
@@ -310,7 +487,7 @@ class ExifUtils:
temp_path = temp_file.name
temp_file.write(image_data)
try:
metadata = ExifUtils.extract_image_metadata(temp_path)
metadata_fields = ExifUtils._load_structured_metadata(temp_path)
except Exception as e:
logger.warning(f"Failed to extract metadata from temp file: {e}")
finally:
@@ -363,14 +540,13 @@ class ExifUtils:
optimized_data = output.getvalue()
# Handle metadata preservation if requested and available
if preserve_metadata and metadata:
if preserve_metadata and metadata_fields:
try:
if save_format == 'WEBP':
# For WebP format, directly save with metadata
try:
output_with_metadata = BytesIO()
exif_dict = {'Exif': {piexif.ExifIFD.UserComment: b'UNICODE\0' + metadata.encode('utf-16be')}}
exif_bytes = piexif.dump(exif_dict)
exif_bytes = ExifUtils._build_exif_bytes(metadata_fields)
resized_img.save(output_with_metadata, format='WEBP', exif=exif_bytes, quality=quality)
optimized_data = output_with_metadata.getvalue()
except Exception as e:
@@ -383,8 +559,9 @@ class ExifUtils:
temp_file.write(optimized_data)
try:
# Add metadata
ExifUtils.update_image_metadata(temp_path, metadata)
ExifUtils.update_image_metadata(
temp_path, metadata_fields.get("parameters") or ""
)
# Read back the file
with open(temp_path, 'rb') as f:
optimized_data = f.read()
+91 -1
View File
@@ -1,7 +1,10 @@
import hashlib
import json
import logging
import os
import struct
from typing import Any
from .constants import (
CARD_PREVIEW_WIDTH,
@@ -31,14 +34,101 @@ def _get_hash_chunk_size_bytes() -> int:
async def calculate_sha256(file_path: str) -> str:
"""Calculate SHA256 hash of a file"""
"""Calculate SHA256 hash of a file (full file content).
Uses ``posix_fadvise`` with ``POSIX_FADV_DONTNEED`` to avoid polluting the OS page
cache — critical on WSL where cached file pages live inside the VM and are not
accounted for in guest ``used`` memory, causing VmmemWSL to balloon.
On Windows/macOS where ``posix_fadvise`` is not available the hint is silently
skipped.
"""
sha256_hash = hashlib.sha256()
chunk_size = _get_hash_chunk_size_bytes()
with open(file_path, "rb") as f:
fd = f.fileno()
for byte_block in iter(lambda: f.read(chunk_size), b""):
sha256_hash.update(byte_block)
# Evict pages after reading so the data doesn't linger in the kernel page
# cache — on WSL this otherwise appears as unreclaimable VmmemWSL growth.
# Guard against platforms (Windows, macOS) that lack posix_fadvise.
if hasattr(os, "posix_fadvise") and hasattr(os, "POSIX_FADV_DONTNEED"):
os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED)
return sha256_hash.hexdigest()
def calculate_autov2(file_path: str) -> str:
"""Calculate CivitAI AutoV2 hash.
AutoV2 is the first 10 characters of the full file SHA256.
Used by CivitAI as a shortened file identifier.
Reference: https://developer.civitai.com/site/reference/model-versions
"""
full_hash = hashlib.sha256()
chunk_size = _get_hash_chunk_size_bytes()
with open(file_path, "rb") as f:
for byte_block in iter(lambda: f.read(chunk_size), b""):
full_hash.update(byte_block)
return full_hash.hexdigest()[:10]
def read_safetensors_metadata(file_path: str) -> dict[str, Any]:
"""Read the ``__metadata__`` dict from a safetensors file header.
Safetensors file format:
- 8 bytes: header length (little-endian 64-bit)
- N bytes: UTF-8 JSON header
- The header JSON contains a ``__metadata__`` key holding arbitrary metadata.
Returns an empty dict if the file is not a valid safetensors file or has no
metadata.
"""
try:
with open(file_path, "rb") as f:
header_len_bytes = f.read(8)
if len(header_len_bytes) < 8:
return {}
header_len = struct.unpack("<Q", header_len_bytes)[0]
header_bytes = f.read(header_len)
if len(header_bytes) < header_len:
return {}
header = json.loads(header_bytes.decode("utf-8"))
return header.get("__metadata__", {})
except (OSError, json.JSONDecodeError, UnicodeDecodeError, struct.error, MemoryError, Exception):
return {}
def calculate_autov3(file_path: str) -> str | None:
"""Calculate CivitAI AutoV3 hash from a safetensors file.
AutoV3 is extracted from the safetensors file's embedded metadata, not
computed from the file bytes directly. The orchestrator reads the
``sshs_model_hash`` (kohya-ss format) or ``modelspec.hash_sha256`` field
from the safetensors header and stores the first 12 characters.
The embedded hash itself is the SHA256 of the file after skipping the
8-byte header length + JSON header (a.k.a. the addnet hash / tensor-only
hash).
Reference:
- CivitAI DB trigger: ``SUBSTRING(NEW.hash FROM 1 FOR 12)``
- https://developer.civitai.com/site/reference/model-versions
Returns ``None`` when no AutoV3 hash can be determined (e.g. the file is
not safetensors, or the metadata doesn't contain a recognised hash field).
"""
metadata = read_safetensors_metadata(file_path)
if not metadata:
return None
embedded_hash = metadata.get("sshs_model_hash") or metadata.get("modelspec.hash_sha256")
if embedded_hash and isinstance(embedded_hash, str) and len(embedded_hash) >= 12:
return embedded_hash[:12]
return None
def find_preview_file(base_name: str, dir_path: str) -> str:
"""Find preview file for given base name in directory.
+22
View File
@@ -64,6 +64,27 @@ def _build_log_file_path(settings_file: str | None, started_at: datetime) -> str
return os.path.join(log_dir, f"standalone-session-{timestamp}.log")
_KEEP_LOG_COUNT = 3
def _prune_old_logs(log_dir: str) -> None:
"""Remove older session log files, keeping only the ``_KEEP_LOG_COUNT`` newest."""
try:
files = [
os.path.join(log_dir, name)
for name in os.listdir(log_dir)
if name.startswith("standalone-session-") and name.endswith(".log")
]
except OSError:
return
files.sort(key=os.path.getmtime, reverse=True)
for path in files[_KEEP_LOG_COUNT:]:
try:
os.remove(path)
except OSError:
pass
def setup_standalone_session_logging(settings_file: str | None) -> StandaloneSessionLogState:
global _session_state
@@ -90,6 +111,7 @@ def setup_standalone_session_logging(settings_file: str | None) -> StandaloneSes
file_handler.set_name(_FILE_HANDLER_NAME)
file_handler.setFormatter(formatter)
root_logger.addHandler(file_handler)
_prune_old_logs(os.path.dirname(log_file_path))
_session_state = StandaloneSessionLogState(
started_at=started_at,
+115 -17
View File
@@ -1,4 +1,5 @@
import os
import re
import json
import time
import asyncio
@@ -9,6 +10,7 @@ from typing import Dict, Set
from ..config import config
from ..services.service_registry import ServiceRegistry
from ..utils.settings_paths import get_settings_dir
# Check if running in standalone mode
standalone_mode = os.environ.get("LORA_MANAGER_STANDALONE", "0") == "1" or os.environ.get("HF_HUB_DISABLE_TELEMETRY", "0") == "0"
@@ -16,14 +18,18 @@ standalone_mode = os.environ.get("LORA_MANAGER_STANDALONE", "0") == "1" or os.en
# Define constants locally to avoid dependency on conditional imports
MODELS = "models"
LORAS = "loras"
EMBEDDINGS = "embeddings"
PROMPTS = "prompts"
if not standalone_mode:
from ..metadata_collector.metadata_registry import MetadataRegistry
# Import constants from metadata_collector to ensure consistency, but we have fallbacks defined above
try:
from ..metadata_collector.constants import MODELS as _MODELS, LORAS as _LORAS
from ..metadata_collector.constants import MODELS as _MODELS, LORAS as _LORAS, EMBEDDINGS as _EMBEDDINGS, PROMPTS as _PROMPTS
MODELS = _MODELS
LORAS = _LORAS
EMBEDDINGS = _EMBEDDINGS
PROMPTS = _PROMPTS
except ImportError:
pass # Use the local definitions
@@ -65,6 +71,7 @@ class UsageStats:
self.stats = {
"checkpoints": {}, # sha256 -> { total: count, history: { date: count } }
"loras": {}, # sha256 -> { total: count, history: { date: count } }
"embeddings": {}, # sha256 -> { total: count, history: { date: count } }
"total_executions": 0,
"last_save_time": 0
}
@@ -77,6 +84,7 @@ class UsageStats:
# Load existing stats if available
self._stats_file_path = self._get_stats_file_path()
self._migrate_from_old_location()
self._load_stats()
# Save interval in seconds
@@ -89,14 +97,38 @@ class UsageStats:
logger.debug("Usage statistics tracker initialized")
def _get_stats_file_path(self) -> str:
"""Get the path to the stats JSON file"""
"""Get the path to the stats JSON file in the settings directory."""
settings_dir = get_settings_dir(create=True)
return os.path.join(settings_dir, "stats", self.STATS_FILENAME)
@staticmethod
def _get_old_stats_file_path() -> str:
"""Get the legacy stats file path in the first lora root directory."""
if not config.loras_roots or len(config.loras_roots) == 0:
# If no lora roots are available, we can't save stats
# This will be handled by the caller
raise RuntimeError("No LoRA root directories configured. Cannot initialize usage statistics.")
# Use the first lora root
return os.path.join(config.loras_roots[0], self.STATS_FILENAME)
return ""
return os.path.join(config.loras_roots[0], UsageStats.STATS_FILENAME)
def _migrate_from_old_location(self) -> None:
"""Migrate stats file from old location (first lora root) to new location (settings_dir/stats/)."""
new_path = self._stats_file_path
if os.path.exists(new_path):
return
old_path = self._get_old_stats_file_path()
if not old_path or not os.path.exists(old_path):
return
try:
os.makedirs(os.path.dirname(new_path), exist_ok=True)
shutil.copy2(old_path, new_path)
logger.info("Migrated usage stats from %s to %s", old_path, new_path)
try:
os.remove(old_path)
logger.info("Cleaned up old stats file: %s", old_path)
except Exception as e:
logger.warning("Failed to remove old stats file %s: %s", old_path, e)
except Exception as e:
logger.error("Failed to migrate usage stats from %s to %s: %s", old_path, new_path, e)
def _backup_old_stats(self):
"""Backup the old stats file before conversion"""
@@ -115,6 +147,7 @@ class UsageStats:
new_stats = {
"checkpoints": {},
"loras": {},
"embeddings": {},
"total_executions": old_stats.get("total_executions", 0),
"last_save_time": old_stats.get("last_save_time", time.time())
}
@@ -142,21 +175,27 @@ class UsageStats:
}
}
# Convert embedding stats (if present in old format)
if "embeddings" in old_stats and isinstance(old_stats["embeddings"], dict):
for hash_id, count in old_stats["embeddings"].items():
new_stats["embeddings"][hash_id] = {
"total": count,
"history": {
today: count
}
}
logger.info("Successfully converted stats from old format to new format with history")
return new_stats
def _is_old_format(self, stats):
"""Check if the stats are in the old format (direct count values)"""
# Check if any lora or checkpoint entry is a direct number instead of an object
if "loras" in stats and isinstance(stats["loras"], dict):
for hash_id, data in stats["loras"].items():
if isinstance(data, (int, float)):
return True
if "checkpoints" in stats and isinstance(stats["checkpoints"], dict):
for hash_id, data in stats["checkpoints"].items():
if isinstance(data, (int, float)):
return True
for category in ("loras", "checkpoints", "embeddings"):
if category in stats and isinstance(stats[category], dict):
for hash_id, data in stats[category].items():
if isinstance(data, (int, float)):
return True
return False
@@ -182,6 +221,9 @@ class UsageStats:
if "loras" in loaded_stats and isinstance(loaded_stats["loras"], dict):
self.stats["loras"] = loaded_stats["loras"]
if "embeddings" in loaded_stats and isinstance(loaded_stats["embeddings"], dict):
self.stats["embeddings"] = loaded_stats["embeddings"]
if "total_executions" in loaded_stats:
self.stats["total_executions"] = loaded_stats["total_executions"]
@@ -304,6 +346,10 @@ class UsageStats:
if LORAS in metadata and isinstance(metadata[LORAS], dict):
await self._process_loras(metadata[LORAS], today)
# Process embeddings — parse prompt text for embedding:name references
if PROMPTS in metadata and isinstance(metadata[PROMPTS], dict):
await self._process_embeddings(metadata[PROMPTS], today)
def _increment_usage_counter(self, category: str, stat_key: str, today_date: str) -> None:
"""Increment usage counters for a resolved stats key."""
if stat_key not in self.stats[category]:
@@ -510,6 +556,55 @@ class UsageStats:
except Exception as e:
logger.error(f"Error processing LoRA usage: {e}", exc_info=True)
@staticmethod
def _extract_embedding_names(prompt_text: str) -> set:
"""Parse embedding:name references from prompt text.
ComfyUI's SDTokenizer resolves ``embedding:<name>`` during tokenization
(see ``sd1_clip.py _try_get_embedding``). This mirrors the same pattern
to extract embedding file names from the captured prompt strings.
"""
if not prompt_text:
return set()
# Matches ``embedding:name`` where name is alphanumeric plus _ . - /
names = re.findall(r"embedding:([a-zA-Z0-9_.\-/]+)", prompt_text)
return set(names)
async def _process_embeddings(self, prompts_data, today_date):
"""Extract embedding usage from prompt texts and record it.
Iterates every prompt node's text field captured by the metadata
collector, extracts ``embedding:<name>`` references, resolves each
name to its SHA256 hash via the embedding scanner, and increments
usage counters.
"""
try:
embedding_scanner = await ServiceRegistry.get_embedding_scanner()
if not embedding_scanner:
logger.warning("Embedding scanner not available for usage tracking")
return
seen_names = set()
for _node_id, prompt_data in prompts_data.items():
if not isinstance(prompt_data, dict):
continue
for text_field in ("text", "positive_text", "negative_text"):
text = prompt_data.get(text_field)
if isinstance(text, str):
seen_names.update(self._extract_embedding_names(text))
for emb_name in seen_names:
emb_hash = embedding_scanner.get_hash_by_filename(emb_name)
if emb_hash:
self._increment_usage_counter("embeddings", emb_hash, today_date)
else:
logger.debug(
"No hash found for embedding '%s', skipping usage tracking",
emb_name,
)
except Exception as e:
logger.error("Error processing embedding usage: %s", e, exc_info=True)
async def get_stats(self):
"""Get current usage statistics"""
return self.stats
@@ -522,6 +617,9 @@ class UsageStats:
elif model_type == "lora":
if sha256 in self.stats["loras"]:
return self.stats["loras"][sha256]["total"]
elif model_type == "embedding":
if sha256 in self.stats["embeddings"]:
return self.stats["embeddings"][sha256]["total"]
return 0
async def process_execution(self, prompt_id):
+97 -24
View File
@@ -15,30 +15,64 @@ def get_lora_info(lora_name):
scanner = await ServiceRegistry.get_lora_scanner()
cache = await scanner.get_cached_data()
lora_name_normalized = lora_name.replace("\\", "/")
lora_name_no_ext = lora_name_normalized
for ext in (".safetensors", ".ckpt", ".pt", ".bin"):
if lora_name_no_ext.lower().endswith(ext):
lora_name_no_ext = lora_name_no_ext[: -len(ext)]
break
has_path = "/" in lora_name_no_ext
basename = os.path.basename(lora_name_no_ext) if has_path else lora_name_no_ext
best_fallback = None
for item in cache.raw_data:
if item.get("file_name") == lora_name:
file_path = item.get("file_path")
if file_path:
# Check all lora roots including extra paths
all_roots = list(config.loras_roots or []) + list(
config.extra_loras_roots or []
file_name = item.get("file_name", "")
folder = item.get("folder", "")
file_name_no_ext = file_name
for ext in (".safetensors", ".ckpt", ".pt", ".bin"):
if file_name_no_ext.lower().endswith(ext):
file_name_no_ext = file_name_no_ext[: -len(ext)]
break
path_name = f"{folder}/{file_name_no_ext}".replace("\\", "/") if folder else file_name_no_ext
if lora_name_no_ext not in (file_name_no_ext, path_name):
if has_path and file_name_no_ext == basename:
if folder and lora_name_no_ext.startswith(folder.replace("\\", "/") + "/"):
best_fallback = item
elif best_fallback is None:
best_fallback = item
continue
file_path = item.get("file_path")
if not file_path:
continue
all_roots = list(config.loras_roots or []) + list(
config.extra_loras_roots or []
)
for root in all_roots:
root = root.replace(os.sep, "/")
if file_path.startswith(root):
relative_path = os.path.relpath(file_path, root).replace(
os.sep, "/"
)
for root in all_roots:
root = root.replace(os.sep, "/")
if file_path.startswith(root):
relative_path = os.path.relpath(file_path, root).replace(
os.sep, "/"
)
# Get trigger words from civitai metadata
civitai = item.get("civitai", {})
trigger_words = (
civitai.get("trainedWords", []) if civitai else []
)
return relative_path, trigger_words
# If not found in any root, return path with trigger words from cache
civitai = item.get("civitai", {})
trigger_words = civitai.get("trainedWords", []) if civitai else []
return file_path, trigger_words
trigger_words = (
civitai.get("trainedWords", []) if civitai else []
)
return relative_path, trigger_words
civitai = item.get("civitai", {})
trigger_words = civitai.get("trainedWords", []) if civitai else []
return file_path, trigger_words
if best_fallback:
file_path = best_fallback.get("file_path")
if file_path:
civitai = best_fallback.get("civitai", {})
trigger_words = civitai.get("trainedWords", []) if civitai else []
return file_path, trigger_words
return lora_name, []
try:
@@ -77,15 +111,54 @@ def get_lora_info_absolute(lora_name):
scanner = await ServiceRegistry.get_lora_scanner()
cache = await scanner.get_cached_data()
lora_name_normalized = lora_name.replace("\\", "/")
lora_name_no_ext = lora_name_normalized
for ext in (".safetensors", ".ckpt", ".pt", ".bin"):
if lora_name_no_ext.lower().endswith(ext):
lora_name_no_ext = lora_name_no_ext[: -len(ext)]
break
has_path = "/" in lora_name_no_ext
basename = os.path.basename(lora_name_no_ext) if has_path else lora_name_no_ext
best_fallback = None
for item in cache.raw_data:
if item.get("file_name") == lora_name:
file_name = item.get("file_name", "")
folder = item.get("folder", "")
file_name_no_ext = file_name
for ext in (".safetensors", ".ckpt", ".pt", ".bin"):
if file_name_no_ext.lower().endswith(ext):
file_name_no_ext = file_name_no_ext[: -len(ext)]
break
path_name = f"{folder}/{file_name_no_ext}".replace("\\", "/") if folder else file_name_no_ext
if lora_name_no_ext == file_name_no_ext:
file_path = item.get("file_path")
if file_path:
# Return absolute path directly
# Get trigger words from civitai metadata
civitai = item.get("civitai", {})
trigger_words = civitai.get("trainedWords", []) if civitai else []
return file_path, trigger_words
if lora_name_no_ext == path_name:
file_path = item.get("file_path")
if file_path:
civitai = item.get("civitai", {})
trigger_words = civitai.get("trainedWords", []) if civitai else []
return file_path, trigger_words
if has_path and file_name_no_ext == basename:
if folder and lora_name_no_ext.startswith(folder.replace("\\", "/") + "/"):
best_fallback = item
elif best_fallback is None:
best_fallback = item
if best_fallback:
file_path = best_fallback.get("file_path")
if file_path:
civitai = best_fallback.get("civitai", {})
trigger_words = civitai.get("trainedWords", []) if civitai else []
return file_path, trigger_words
return lora_name, []
try:

Some files were not shown because too many files have changed in this diff Show More