Follows the folder create/delete work: a typo'd directory could be
removed but not corrected, and for a folder holding models the only fix
was to move every model out by hand.
Adds POST /api/lm/{prefix}/rename-folder. Unlike the delete path this one
deliberately works on folders that hold models — a rename keeps every
file, so nothing is cascaded over: the directory is renamed on disk and
the scanner re-keys the records that pointed at the old prefix (recorded
folder list, cache file_path/folder/preview_url, hash and autov3 index
paths, excluded-model paths, and the metadata sidecars that travelled
with the directory). Ancestors are never touched, and only the leaf name
is accepted so a rename can never escape its parent.
Library roots, top-level symlinks and folders holding a staged delete are
refused; the last because a staging manifest records absolute
original/staged paths, so moving it would break undo and purge. A name
collision is a 409 target_exists conflict.
The sidebar reuses the inline-row idiom from folder creation: prefilled
with the current name, inserted in place of the node with that node
hidden while editing, Enter confirms and Escape/blur cancels. The
persisted selection and the expanded set are re-keyed across the rename
so the user keeps their place in the refreshed tree.
Folders created from the sidebar had no in-app way back out: the only
removal path was to leave ComfyUI, delete the directory by hand and
rescan. A typo'd folder also polluted the move/download destination
picker permanently, since it reads the same all_folders source.
Adds POST /api/lm/{prefix}/delete-folder, restricted to directories
whose subtree holds no model weight files — a folder-level cascade would
bypass the per-model lifecycle bookkeeping (metadata sidecars, previews,
cache entries, pending-delete staging, recipe references). The service
walks the directory itself instead of trusting the possibly stale cache,
reports what it would remove (models / files / subfolders / symlinks),
and refuses library roots, top-level symlinks (shutil.rmtree rejects
those) and folders holding a staged delete, whose manifest would be
invalidated by the move. Symbolic links inside the subtree are counted
but never followed.
ModelScanner.remove_known_folder mirrors add_known_folder: the removed
subtree leaves all_folders while ancestors are kept (every recorded
ancestor exists on disk in its own right), stale cache entries under the
prefix are purged and the folder list recomputed. The handler broadcasts
models_changed so destination pickers drop the folder too.
The sidebar entry is a destructive context-menu item. The modal opens in
a confirm state for model-free folders and an explanatory one when the
subtree still holds models, decided from the models-only set that already
dims empty nodes; a stale tree is caught by the 409 not_empty/busy
conflict. Truly empty folders get the existing 20s undo affordance,
implemented by re-creating the directory.
Empty folders (tracked in the scan-recorded all_folders list, same source
the move/download destination picker uses) can now be surfaced in the
folder sidebar via a view-options toggle, dimmed when their subtree holds
no models. Folders can be created directly from the sidebar through a new
POST /api/lm/{prefix}/create-folder endpoint with library-root
containment checks; the scanner records the new directory incrementally
so the tree reflects it without a rescan.
The sidebar header moves its view toggles (tree/list, recursive, empty
folders) into a "..." menu to fit the new create-folder button.
A model file could only ever be linked to huggingface.co: `set_hf_url`
validated the URL with a huggingface-only regex, the agent fetched the card
from a hardcoded HF URL, and the readme processor built every relative image
path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the
same model-card convention (README.md + YAML frontmatter, often carrying
`base_model:` and `trigger_words:`) behind a public, key-less API, so the
enrichment pipeline could already serve it - it was the plumbing that was
HF-shaped, not the idea.
Make the external source a first-class, provider-driven concept:
- New `py/services/model_sources/` registry. A `ModelSource` owns URL
recognition (lenient for stored values, strict for user input), the
canonical page URL, model-card fetching, the asset base URL and the
capability flags. `HuggingFaceSource` is the previous logic relocated;
`ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md`
and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is
link-only on purpose: tensor.art answers plain HTTP clients with a
Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud /
cn.tensorart.net) rejects every /v1/model/* route with "invalid
authorization header", so it declares supports_enrichment=False rather than
failing silently later.
- Metadata gains `source_platform` + `source_url`; `hf_url` stays as a
read/write alias, written only for Hugging Face, so existing sidecars,
cached rows and third-party consumers keep working. Normalisation runs at
the scanner, the persistent cache (both directions, plus two new columns
behind an ALTER migration) and the linking handler - which is what stops a
user who switches sources from leaving a stale `hf_url` on a ModelScope
model.
- The agent pipeline keys off the provider instead of `hf_url`: the fast-fail
gate now explains *why* a model is skipped (no source / unknown source /
source without a reachable card), the prompt context exposes
source_url/source_id/source_label/asset_base_url while still filling the
legacy hf_url/repo aliases, and the four README image extractors take a
base_url (defaulting to HF) so relative paths resolve against the right
site. Version grouping generalises to hf: / ms: / ta: keys.
- `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but
accepts `source_url`, validates against every provider and returns the
platform. `GET /api/lm/model-sources` lets the UI render the supported-site
list from the server.
- Frontend: a `modelSourceHelpers` mirror of the registry drives the link
dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the
version-group key and the enrichment gate; the versions tab no longer sends
ms:/ta: keys to the CivitAI API.
TensorArt stays in the list because provenance is worth keeping even when the
card is unreadable - the dialog says so plainly ("Sites that don't expose one
(currently TensorArt) can only be linked") and the context menu disables
enrichment with a matching tooltip, instead of the user getting
"Unsupported URL".
Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a
1882-byte card whose frontmatter carries base_model/tags/trigger_words, and
relative images resolve to .../resolve/master/....
Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest
tests/i18n and a Jinja compile pass over templates/. The nine locales carry
[TODO: Translate] for the new strings, completed in the next commit.
The include_empty folder tree (download/move modals) walked every model
root synchronously on the event loop via get_all_folders(). On network
(NAS) roots this froze the whole server for the duration of the walk —
blocking WebSocket progress, aria2 RPC and the download queue — and the
5s TTL re-triggered the walk on nearly every modal interaction.
The scanners already visit every directory during cache scans, so record
the full directory list (including empty folders) there instead:
- _gather_model_data/_reconcile_cache collect directories during the
existing walks; reconcile refreshes and persists the list even when no
model files changed.
- ModelCache gains an all_folders field (None = never recorded).
- PersistentModelCache stores the list in a new folders table, with a
cache_meta flag distinguishing 'recorded empty' from legacy snapshots.
- get_all_folders() is now a pure in-memory read. A legacy snapshot
triggers a one-shot backfill walk in a worker thread (never on the
event loop) that records and persists the list.
- Moves add the destination folder (and parents) incrementally instead
of invalidating a TTL cache.
A no-change Refresh still computed os.path.realpath for every model file
in the library and for every cached entry. Both values are only ever
consulted when a discovered file is missing from the cache, so on a
50k-file library they cost ~1.3s and ~0.6s while being used zero times.
- Compute the per-file realpath only after the exact cache match fails
- Build the physical-path alias map lazily on the first miss; the
cross-run alias guard (overlapping roots / symlink layout changes)
still keeps the cached entry instead of a delete + re-add, which would
re-read metadata and re-hash the whole library
- Snapshot get_model_roots() once for the new-file pass instead of
re-reading it for every added file
- Run the duplicate-path integrity pass only when the snapshot already
contained duplicates or files were appended; a clean, unchanged cache
has nothing to clean. Duplicates can only be introduced by external
code rewriting raw_data or by this pass's own appends.
Zero-change reconcile drops from ~1400ms to ~120ms on 50k files, and an
alias flip still re-processes 0 files (#1108 investigation).
Broadcast typed scan_progress messages over /ws/fetch-progress from the
manual refresh/rebuild paths of ModelScanner and RecipeScanner, and
render percent, processed/total, current file name and an EMA-smoothed
ETA in the loading overlay. Hardcoded refresh strings move to i18n
(common.scanProgress); WS connection failure falls back to the previous
static loading behavior.
Bulk delete merged staged batches by physically moving each loser's
files into the winner's batch dir with os.rename. Cross-volume bulks
(winner and loser on different filesystems) always hit EXDEV, forcing a
rollback and degrading to the batch_ids array with per-batch undo.
Merge is now manifest-only: loser entries are appended to the winner's
manifest with their staged paths unchanged, so staged files keep living
in each model's own .lm-pending-delete/<batch_id> dir (no data IO, no
EXDEV). Loser dirs are recorded in the winner manifest's merged_sources
and each loser manifest is stamped merged_into so its own purge timer, a
post-restart sweep or a direct undo call no-op. A cross-volume bulk is
one undoable batch again, and undo/purge clean up the loser dirs once
the merged batch settles.
- Remove delete_undo_enabled setting (backend default, frontend state,
settings modal UI, 10 locales); staged deletes with 30s undo are now
the only delete path and stale settings keys are silently ignored
- Remove the 1500ms delete-button arm delay (armDeleteButton) from all
delete modals; misclicks are recoverable via the undo toast
- Delete modal always shows the recoverable warning
- Log the first staged file path in staging log lines for easier support
Fix ~790 basedpyright errors across the test suite:
- Type stub subclasses of real production classes with super().__init__()
- Add missing generic type arguments and Dict[str, Any] annotations
- Add None guards before subscript/member access
- Adapt tests to production API changes (removed dead handlers,
PersistentModelCache.get_default, _i18n_filter_added location)
- Add PersistentModelCache.update_single_model() for lightweight targeted
SQL update (single row + incremental tag/hash deltas, no full table scan)
- Add ModelScanner.sync_cache_from_metadata() with compare-first logic:
skips entirely when cache is already in sync; when stale, updates the
entry in-place (O(1) instead of O(n) remove+append), incrementally
adjusts tag counts/hash index/version index, and resorts only when
sort-relevant fields changed
- Wire sync_cache_from_metadata() into BaseModelService.get_model_metadata()
via fire-and-forget asyncio.create_task — disk I/O is already paid for
- Include identity re-validation guard against concurrent cache replacement
- Add 16 tests covering _cache_entries_differ, sync_cache_from_metadata
(no-change, in-place, fallback, conditional resort), and
update_single_model (insert, tag delta, hash delta)
- _resolve_commercial_bits() no longer has Sell-implies-Image
cascading; each CommercialUse value sets only its own bit,
matching CivitAI's modern array-format API.
- Keep filter tag label as 'Allow Selling' for brevity; add
title/tooltip 'Allow selling generated images' on hover.
- Same tooltip treatment for 'No Credit Required'.
- Add i18n keys for both tooltips across all 10 locales.
Duplicate filename detection is only relevant for LoRAs, which use
basename-only syntax (<lora:name:strength>). Checkpoints and diffusion
models reference files via relative paths with extensions, so filename
conflicts there are false positives — there is no resolution ambiguity.
Both _log_duplicate_filename_summary() and DoctorHandler's
_check_filename_conflicts() now skip scanners with model_type != 'lora'.
Detects when multiple model files share the same basename (causing
ambiguity in LoRA resolution), logs warnings during scanning, and
provides a "Resolve Conflicts" button in the Doctor panel. Resolution
renames duplicates with hash-prefixed unique filenames, migrates all
sidecar and preview files, and updates the cache and frontend scroller
in-place so the model modal immediately reflects the new filename.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Remove backward compatibility code for `model_type` in `ModelScanner._build_cache_entry()`
- Update `CheckpointScanner` to only handle `sub_type` in `adjust_metadata()` and `adjust_cached_entry()`
- Delete deprecated aliases `resolve_civitai_model_type` and `normalize_civitai_model_type` from `model_query.py`
- Update frontend components (`RecipeModal.js`, `ModelCard.js`, etc.) to use `sub_type` instead of `model_type`
- Update API response format to return only `sub_type`, removing `model_type` from service responses
- Revise technical documentation to mark Phase 5 as completed and remove outdated TODO items
All cleanup tasks for the model type refactoring are now complete, ensuring consistent use of `sub_type` across the codebase.
Add license resolution utilities and integrate license information into model metadata processing. The changes include:
- Add `resolve_license_payload` function to extract license data from Civitai model responses
- Integrate license information into model metadata in CivitaiClient and MetadataSyncService
- Add license flags support in model scanning and caching
- Implement CommercialUseLevel enum for standardized license classification
- Update model scanner to handle unknown fields when extracting metadata values
This ensures proper license attribution and compliance when working with Civitai models.
Will Miao