`Config._dedupe_existing_paths()` drops roots whose directory does not exist
when the root list is built, so a drive that is switched off while LM starts was
not "an offline root" — it was not a root at all. The Refresh ▾ menu had no row
for it in either state, a full refresh reported nothing (its cached entries
survived only because they fall outside every configured prefix), and plugging
the drive back in changed nothing until the next restart.
* `Config` now records the configured (existence-unfiltered) paths per model
type — from ComfyUI's list in plugin mode, from the active library snapshot in
standalone mode, where the host mock filters non-existent paths itself — and
exposes them through `configured_roots_for()`.
* `ModelScanner.describe_model_roots()` appends configured-but-unavailable roots
with a live `reachable`, an `available` flag and their cached entry count, so
`/roots` (and therefore the menu) offers a row in both states: greyed while the
directory is missing, normal and clickable once it is back.
* `_reconcile_cache()` reports them as `skipped_roots` / `unavailable_paths` with
reason `root_unavailable` and counts their cached entries in `kept_unreachable`,
which is what makes the "N models kept" toast appear in the startup-offline
case. Report-only: they stay outside the scan scope, so nothing about pruning
changes. A root-scoped scan stays quiet about roots it was not asked about, and
a folder-scoped scan only mentions a missing root that holds cached entries of
that folder.
* `Config.admit_configured_roots()` re-runs the per-type prepare helpers against
the configured paths and APPENDS what is readable now (plus the
checkpoint/unet/other side maps), then refreshes the preview allowlist, so a
drive plugged in mid-session can be scanned without restarting. Append-only and
order-preserving by design: removing a root mid-session would let a later
settings save persist the loss, and re-sorting would move `*_roots[0]`, which
derives the recipes directory and the usage-stats file location. `_roots[0]`
never moves.
* `/roots` and `/scan` admit first, so `describe_model_roots()` stays a pure read
and a caller that never opens the menu (the browser extension) still gets a
drive that came back.
Verified on the sandbox with drive-Z switched off before startup: `/roots`
reports `drive-Z reachable=false available=false models=60`, a full refresh
returns `skipped_roots=[drive-Z root_unavailable]`,
`unavailable_paths=[{... kept: 60}]`, `kept_unreachable=60`, and all 432 models
stay cached. Renaming the drive back while the server runs admits it
(`available=true`) and `GET /scan?roots=<drive-Z>` walks it with no restart.
3714 passed, 7 skipped; frontend 1495 passed (149 files); vue widgets 96 passed.
Refreshing had no way to say "scan only this drive": a user with three external
drives had to spin all of them up for every refresh, and switching a drive off
made the next refresh treat its whole library as deleted (rows pruned from the
memory cache and the SQLite cache, preview_url stripped on the next scroll).
Backend (py/services/model_scanner.py, py/config.py):
* ReconcileScope(roots, folder) + _reconcile_cache(scope=...): files inside the
scope reconcile normally, everything outside is neither re-read nor removed.
The folder half is plumbing for the sidebar entry in the next change.
* Path-level pruning guard: cached entries under a path this walk could not
read are kept and reported instead of removed. Sources: a configured root that
is not reachable (drive switched off while LM runs), a directory os.walk
failed to enter (permissions / I/O error / Windows junction to an offline
drive), and a known first-level symlink whose target is gone
(Config.iter_path_mappings()).
* The recorded folder list is unioned instead of replaced whenever the scan did
not verify every root, so a scoped scan cannot empty the sidebar.
* _reconcile_cache returns a summary (added / removed / repaired /
scanned_roots / skipped_roots / unavailable_paths / kept_unreachable),
exposed as ModelScanner.last_reconcile_summary, returned by
BaseModelService.scan_models() and broadcast in the completed WS payload.
* _root_display_labels(): set-aware labels ("G: loras", "usb/loras") grown
leftwards with real parent segments until unique, shared by the walk-progress
line and the roots API.
* GET /scan accepts repeated `roots` (400 for unknown roots, 400 combined with
full_rebuild=true); GET /roots gains root_details (label / reachable / cached
count) while `roots` stays a plain path list for existing callers.
* serve_preview: a 404 no longer clears the cached preview_url when the file's
own directory is unreachable - browsing the grid with a drive off used to
strip preview references from the persistent cache.
Frontend:
* Refresh ▾ gains a "Scan one folder" section listing the page's roots with
their cached counts; offline roots stay clickable and explain themselves; rows
are wired by delegation (new static/js/components/controls/ScanScopeMenu.js).
* A scoped scan reports "Scanned <root>: N new, M removed"; a scan that kept
entries reports "<N> models kept: <paths> not reachable".
* registerAPI() now injects the two cross-page passthroughs (fetchModelRoots and
an argument-forwarding refreshModels) so a page facade cannot drop them: the
first version rendered an empty menu and would have run a full refresh.
* createToastElement whitelists toast types, so a wrong `type` argument degrades
to the info style instead of rendering an unstyled box.
Verified in a sandbox instance with three roots: a scoped scan walks only the
requested root (progress roots=0/1, 240 files); a full refresh with one root
offline reports kept_unreachable=60 and leaves all 420 models cached; /roots
reports the offline root with its cached count. 3699 passed, 7 skipped;
frontend 1488 passed (148 files); vue widgets 96 passed.
A repository is not a model identity: collection repos on Hugging Face
and ModelScope host many unrelated models, which were wrongly shown as
versions of each other.
- Hugging Face models no longer auto-group (the Hub exposes no
site-native model id)
- ModelScope models group by the site's native published-model id
(MuseInfo modelVersion.modelId), extracted during enrichment and
persisted on the sidecar as source_model_id/source_version_id;
unenriched models stay standalone instead of collapsing a whole repo
into one group
- TensorArt grouping unchanged (its URL id is already model-level)
- Frontend group-key derivation mirrors the new backend semantics
A model file could only ever be linked to huggingface.co: `set_hf_url`
validated the URL with a huggingface-only regex, the agent fetched the card
from a hardcoded HF URL, and the readme processor built every relative image
path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the
same model-card convention (README.md + YAML frontmatter, often carrying
`base_model:` and `trigger_words:`) behind a public, key-less API, so the
enrichment pipeline could already serve it - it was the plumbing that was
HF-shaped, not the idea.
Make the external source a first-class, provider-driven concept:
- New `py/services/model_sources/` registry. A `ModelSource` owns URL
recognition (lenient for stored values, strict for user input), the
canonical page URL, model-card fetching, the asset base URL and the
capability flags. `HuggingFaceSource` is the previous logic relocated;
`ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md`
and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is
link-only on purpose: tensor.art answers plain HTTP clients with a
Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud /
cn.tensorart.net) rejects every /v1/model/* route with "invalid
authorization header", so it declares supports_enrichment=False rather than
failing silently later.
- Metadata gains `source_platform` + `source_url`; `hf_url` stays as a
read/write alias, written only for Hugging Face, so existing sidecars,
cached rows and third-party consumers keep working. Normalisation runs at
the scanner, the persistent cache (both directions, plus two new columns
behind an ALTER migration) and the linking handler - which is what stops a
user who switches sources from leaving a stale `hf_url` on a ModelScope
model.
- The agent pipeline keys off the provider instead of `hf_url`: the fast-fail
gate now explains *why* a model is skipped (no source / unknown source /
source without a reachable card), the prompt context exposes
source_url/source_id/source_label/asset_base_url while still filling the
legacy hf_url/repo aliases, and the four README image extractors take a
base_url (defaulting to HF) so relative paths resolve against the right
site. Version grouping generalises to hf: / ms: / ta: keys.
- `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but
accepts `source_url`, validates against every provider and returns the
platform. `GET /api/lm/model-sources` lets the UI render the supported-site
list from the server.
- Frontend: a `modelSourceHelpers` mirror of the registry drives the link
dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the
version-group key and the enrichment gate; the versions tab no longer sends
ms:/ta: keys to the CivitAI API.
TensorArt stays in the list because provenance is worth keeping even when the
card is unreadable - the dialog says so plainly ("Sites that don't expose one
(currently TensorArt) can only be linked") and the context menu disables
enrichment with a matching tooltip, instead of the user getting
"Unsupported URL".
Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a
1882-byte card whose frontmatter carries base_model/tags/trigger_words, and
relative images resolve to .../resolve/master/....
Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest
tests/i18n and a Jinja compile pass over templates/. The nine locales carry
[TODO: Translate] for the new strings, completed in the next commit.
Autocomplete suggestions were ranked purely by relevance across the whole
library, so same-named loras from different subfolders interleaved and were
hard to tell apart. Results are now bucketed by folder (root first, then
alphabetically, with nested paths sorting naturally) while keeping the
existing relevance ordering within each folder group.
CivitAI's PaidAccess cutover deprecated the availability=EarlyAccess and
earlyAccessEndsAt signals; gated versions now report availability=Public
with a paidAccess DTO that LoRA Manager previously ignored, so "Hide
Early Access Updates" missed paid/early-access models and downloads
failed with 401.
Parse and persist paidAccess from model-level, bulk, and by-hash
responses; treat timed paid gates as early access and permanent paid
versions as a distinct is_paid state; add a hide_paid_updates setting
with a "Paid" badge in the versions tab; warn before downloading gated
versions. Includes SQLite migration, i18n for all locales, and
backend/frontend tests.
- Three-state autov3 field (not-checked / checked-unavailable / 12-hex value)
in .metadata.json sidecars, in-memory ModelHashIndex, and SQLite
(models.autov3 column + autov3_index table) with column-presence migration
- Background self-terminating backfill for legacy rows: per-model-type
concurrency guard, executor-offloaded I/O, Civitai-first resolution
(SHA256-matched version file) falling back to the embedded safetensors
header hash
- Civitai-first propagation on metadata refresh, scan, and download paths;
reject the empty-string SHA256 placeholder and strip OneTrainer 0x prefix
- List API hash filters and hash index lookups accept 12-char AutoV3
- Cap safetensors header reads at 64 MiB to prevent crafted-file allocation
- Prevent stale AutoV3 mappings on file replacement while preserving them on
same-file re-registration (lazy-hash completion)
Add /af and /noaf toggle commands (plus /activefilters aliases) to the
loras autocomplete widget. When enabled (default off), suggestions are
matched within the active filters (folder, base model, tags, auto-tags,
license, tag logic) persisted by the LoRA Manager page in localStorage,
keeping the match pool consistent with the list endpoint, including the
global show_only_sfw setting.
Backend: /lm/{prefix}/relative-paths accepts the filter query params and
pre-filters the scanner cache with ModelFilterSet. The presence of the
recursive param signals the filter pipeline to run even without concrete
filters so global settings stay in parity with the list endpoint.
- Add PersistentModelCache.update_single_model() for lightweight targeted
SQL update (single row + incremental tag/hash deltas, no full table scan)
- Add ModelScanner.sync_cache_from_metadata() with compare-first logic:
skips entirely when cache is already in sync; when stale, updates the
entry in-place (O(1) instead of O(n) remove+append), incrementally
adjusts tag counts/hash index/version index, and resorts only when
sort-relevant fields changed
- Wire sync_cache_from_metadata() into BaseModelService.get_model_metadata()
via fire-and-forget asyncio.create_task — disk I/O is already paid for
- Include identity re-validation guard against concurrent cache replacement
- Add 16 tests covering _cache_entries_differ, sync_cache_from_metadata
(no-change, in-place, fallback, conditional resort), and
update_single_model (insert, tag delta, hash delta)
Add a pure frontend node that shows filename and editable notes for
a selected LoRA. Connect any output from a LoRA Loader/Stacker/Randomizer/
WanVideoSelect to the lora_source input — selecting a LoRA in the source
widget updates the info display automatically.
- Python node (LoraInfoLM): display-only, no workflow execution
- Vue widget: filename label, auto-sizing notes textarea, save button
with ComfyUI toast feedback on save
- Frontend extension: wire-based selection propagation with stale-response
race guard; clears display on wire disconnect
- Backend: get-notes endpoint now returns file_path alongside notes;
matching supports full-path lora syntax; fix NoneType crash in
trigger words endpoint; document cache file_name invariant
- Wired into all four lora widget nodes (Loader, Stacker, Randomizer,
WanVideoSelect)
Cache corruption (NULL model_name/file_name from legacy DB rows or partial
writes) caused format_response to raise KeyError/AttributeError, failing the
entire /loras/list request and showing no models in the UI.
Fix across three layers:
- format_response (lora/checkpoint/embedding): replace direct dict[] access
with .get() fallbacks; return None for entries missing file_path
- handlers: filter None entries from list/excluded/fetch/duplicate/conflict
endpoints instead of letting them crash or appear as null in responses
- model_scanner: always use validate_batch repaired copies (previously
discarded when no invalid entries, leaving None values in raw_data)
- persistent_model_cache: add or-empty-string guards on read and write for
nullable TEXT columns (model_name, file_name, folder, base_model, etc.)
Sort by Most/Fewest versions first now works when Group by model is off.
- Backend: group items by modelId (respecting version_grouping setting),
count versions per group, sort groups by count, expand groups with
versions sorted by version id descending
- CSS: remove rule that hid the sort option in non-grouped mode
- Tests: add 3 tests covering desc, asc, and same_base variants
When viewing all versions of a model (VLM mode via 'x versions' button):
- Backend always sorts by version ID descending, ignoring current sort_by
- A temporary 'Newest version first' option is injected into the sort
dropdown (removed on exit, not a permanent option)
- The sort dropdown is disabled (greyed out) while VLM is active
- On clearing VLM, the previous sort preference is restored and the
dropdown re-enabled
- Handles stale VLM state (e.g. after page reload with leftover session)
- Covers all three model page types: loras, checkpoints, embeddings
Also fixes review nits:
- Correct i18n call pattern (defaultValue in options object)
- Shared _restoreSortAfterVlm() helper to avoid triple duplication
- group_by_model dedup now counts versions per group and attaches
version_count; respects update_flag_strategy (same_base) by
sub-grouping on base_model
- Card footer shows clickable 'x versions' link instead of version
name when grouped (hides HIGH/LOW badges); clicking triggers
View Local Versions without page reload
- Added 'Local Versions' sort option (versions_count), auto-hidden
when group_by_model is off
- Sort preference is saved/restored separately for normal and
grouped modes
- VLM flow (triggerVlmView, clearCustomFilter) uses resetAndReload()
via API instead of window.location.reload()
- Fixed cache mutation bug: version_count is now set on a shallow
copy, not the cached dict, preventing stale version_count leaking
into VLM responses
- i18n: all 9 locale files translated
Clicking the button closes the modal, writes filter params to sessionStorage,
and reloads the page to show all local versions of the model as individual
cards (bypassing group-by-model dedup). The filter respects the update flag
strategy and the versions-filter-toggle state (same-base vs all versions).
Supporting changes:
- sessionStorage keys vlm_model_id / vlm_model_name / vlm_base_model
- BaseModelApiClient._addModelSpecificParams adds civitai_model_id param
- LoraApiClient calls super._addModelSpecificParams for VLM detection
- LorasControls / CheckpointsControls clearCustomFilter checks VLM first
- PageControls.checkVlmFilter shows customFilterIndicator with label
- Backend parses civitai_model_id, filters before group_by_model dedup
Adds a 'Group by Model' toggle in Layout Settings. When enabled, only the
latest version (highest civitai.id) of each Civitai model is shown as a
single card — older versions sharing the same modelId are hidden.
Backend dedup runs in BaseModelService.get_paginated_data() before
filtering/pagination, ensuring correct paginated results. The setting
is persisted via the existing settings pipeline and passed as a query
parameter to the listing endpoint.
Includes:
- Backend: dedup logic, route param parsing, settings default
- Frontend: API param, SettingsManager wiring, toggle UI
- i18n: translations for all 10 locales
- Tests: unit test covering dedup on/off and standalone items
Autocomplete, copy/send-to-workflow, and recipe syntax now emit
<lora:folder/name:strength> instead of <lora:name:strength>, using
relative paths to disambiguate identically-named loras in different
subfolders without requiring file renames.
Backend: 3-tier hybrid resolution (path → bare → basename fallback)
across get_lora_info, get_lora_info_absolute, get_model_preview_url,
get_model_civitai_url, get_model_info_by_name, get_lora_metadata_by_filename,
and get_hash_by_filename. Also fix get_random_loras and get_cycler_list
to return path-prefixed names for randomizer/cycler consistency.
Frontend: autocomplete, copyLoraSyntax, handleSendToWorkflow emit
folder-prefixed syntax. extract_lora_name preserves relative paths.
Saved image metadata (<lora:...> in EXIF) intentionally keeps basename-only
for compatibility with A1111/Forge ecosystem.
- Backend auto-tag extraction service: detect HIGH/LOW (Wan-only), I2V/T2V/TI2V,
Lightning/Turbo from filename, base_model, and CivitAI version name
- HIGH/LOW badge in card footer (inline before version name), color-coded:
blue for HIGH, teal for LOW; abbreviated to H/L in medium/compact density
- Auto-tag filter panel (I2V, T2V, TI2V, Lightning, Turbo) with tri-state
include/exclude filtering
- Full filter pipeline: FilterCriteria → ModelFilterSet → baseModelApi params
- AUTO_TAG_GROUPS exported for frontend use
- 19 unit tests for auto-tag extraction edge cases
Add file_path as a tie-breaker for all sort modes in ModelCache, BaseModelService, LoraService, and RecipeCache to ensure deterministic ordering when primary keys are identical. Resolves issue #859.
Remove .safetensors/.ckpt/.pt/.bin extensions from model names in autocomplete
suggestions to improve UX and search relevance:
Frontend (web/comfyui/autocomplete.js):
- Add _getDisplayText() helper to strip extensions from model paths
- Update _matchItem() to match against filename without extension
- Update render() and createItemElement() to display clean names
Backend (py/services/base_model_service.py):
- Add _remove_model_extension() helper method
- Update _relative_path_matches_tokens() to ignore extensions in matching
- Update _relative_path_sort_key() to sort based on names without extensions
Tests (tests/services/test_relative_path_search.py):
- Add tests to verify 's' and 'safe' queries don't match all .safetensors files
Fixes issue where typing 's' would match all .safetensors files and cluttered
suggestions with redundant extension names.
Add Early Access version support with filtering and improved UI:
Backend:
- Add is_early_access and early_access_ends_at fields to ModelVersionRecord
- Implement two-phase EA detection (bulk API + single API enrichment)
- Add hide_early_access_updates setting to filter EA updates
- Update has_update() and has_updates_bulk() to respect EA filter setting
- Add _enrich_early_access_details() for precise EA time fetching
- Fix setting propagation through base_model_service and model_update_service
Frontend:
- Add smart relative time display for EA (in Xh, in Xd, or date)
- Replace EA label with clock icon in metadata (fa-clock)
- Show Download button with bolt icon for EA versions (fa-bolt)
- Change EA badge color to #F59F00 (CivitAI Buzz theme)
- Fix toggle UI for hide_early_access_updates setting
- Add translation keys for EA time formatting
Tests:
- Update all tests to pass with new EA functionality
- Add test coverage for EA filtering logic
Closes#815
Add a segmented toggle in the Filter Panel to switch between 'Any' (OR)
and 'All' (AND) logic when filtering by multiple include tags.
Changes:
- Backend: Add tag_logic field to FilterCriteria and ModelFilterSet
- Backend: Parse tag_logic parameter in model handlers
- Frontend: Add segmented toggle UI in filter panel header
- Frontend: Add interaction logic and state management for tag logic
- Add translations for all supported languages
- Add comprehensive tests for the new feature
Closes#802
This commit resolves the semantic confusion around the model_type field by
clearly distinguishing between:
- scanner_type: architecture-level (lora/checkpoint/embedding)
- sub_type: business-level subtype (lora/locon/dora/checkpoint/diffusion_model/embedding)
Backend Changes:
- Rename model_type to sub_type in CheckpointMetadata and EmbeddingMetadata
- Add resolve_sub_type() and normalize_sub_type() in model_query.py
- Update checkpoint_scanner to use _resolve_sub_type()
- Update service format_response to include both sub_type and model_type
- Add VALID_*_SUB_TYPES constants with backward compatible aliases
Frontend Changes:
- Add MODEL_SUBTYPE_DISPLAY_NAMES constants
- Keep MODEL_TYPE_DISPLAY_NAMES as backward compatible alias
Testing:
- Add 43 new tests covering sub_type resolution and API response
Documentation:
- Add refactoring todo document to docs/technical/
BREAKING CHANGE: None - full backward compatibility maintained
- Add folder_include parameter support in backend API handlers
- Add folder_include to FilterCriteria and implement multi-folder filtering logic
- Update frontend to send all include folders instead of only the first
- Add tests for single/multiple include folders, include with exclude, and non-recursive filtering
- Add blank line after module docstring for better PEP 8 compliance
- Reformat long lines to adhere to 88-character limit using Black-style formatting
- Improve string consistency by using double quotes consistently
- Enhance readability of complex list comprehensions and method calls
- Maintain all existing functionality while improving code structure
Changed logging level from INFO to DEBUG for performance-related messages in model management service. This reduces noise in production logs while maintaining debugging capability for performance analysis.
Removed the forced normalization of path separators to forward slashes in BaseModelService to maintain platform-specific separators. Updated test cases to use os.sep for constructing expected paths, ensuring tests work correctly across different operating systems while preserving native path representations.
Add `_seed_root_symlink_mappings` method to ensure symlinked root folders are recorded before deep scanning, preventing them from being missed during directory traversal. This ensures that root symlinks are properly captured in the path mappings.
Additionally, normalize separators in relative paths for cross-platform consistency in `BaseModelService`, and update tests to verify root symlinks are preserved in the cache.
- Add token parsing to support include/exclude search terms using "-" prefix
- Implement token-based matching logic for relative path searches
- Improve search result sorting by prioritizing prefix matches and match position
- Add frontend test for multi-token highlighting with exclusion support
- Add model_types parameter to ModelListingHandler to support filtering by model type
- Implement get_model_types endpoint in ModelQueryHandler to retrieve available model types
- Register new /api/lm/{prefix}/model-types route for model type queries
- Extend BaseModelService to handle model type filtering in queries
- Support both model_type and civitai_model_type query parameters for backward compatibility
This enables users to filter models by specific types, improving model discovery and organization capabilities.
Add new "updateFlags" section to settings navigation and implement update flag strategy configuration. The strategy allows users to choose when update badges appear:
- Match updates by base model (only show when new release shares same base model)
- Flag any available update (show whenever newer version exists)
Includes translations for English, German, Spanish, and French locales.
Add support for configurable update flag strategy with new "same_base" mode that considers base model versions when determining update availability. The strategy is controlled by the "update_flag_strategy" setting.
When strategy is set to "same_base":
- Uses get_records_bulk instead of has_updates_bulk
- Compares model versions against highest local versions per base model
- Provides more granular update detection based on base model relationships
Fallback to existing bulk or individual update checks when:
- Strategy is not "same_base"
- Bulk operations fail
- Records are unavailable
This enables more precise update flagging for models sharing common bases.
- Update frontend tag filter to cycle through include/exclude/clear states
- Add backend support for tag_include and tag_exclude query parameters
- Maintain backward compatibility with legacy tag parameter
- Store tag states as dictionary with 'include'/'exclude' values
- Update test matrix documentation to reflect new tag behavior
The changes enable more granular tag filtering where users can now explicitly include or exclude specific tags, rather than just adding tags to a simple inclusion list. This provides better control over search results and improves the filtering user experience.
Add support for filtering models by license requirements:
- credit_required: filter models that require credits or allow free use
- allow_selling_generated_content: filter models based on commercial usage rights
These filters use license_flags bitmask to determine model permissions and enable users to find models that match their specific usage requirements and budget constraints.
Will Miao