ModelScope became a linkable source, but downloading from it was impossible:
the URL picker only recognised huggingface.co, the file listing hit a
huggingface-only endpoint, the resolve URL was hardcoded, and the default
path template always wrote into a `huggingface/` directory.
Move the download knowledge into the providers so the handlers stay generic:
- `ModelSource` gains `list_files()`, `file_download_url()`,
`default_revision` and `default_subdir`. `HuggingFaceSource` keeps the Hub
tree API (`/api/models/{id}/tree/{rev}`, LFS-aware sizes, `main`).
`ModelScopeSource` uses `/api/v1/models/{id}/repo/files?Revision=master`
— which reports real byte sizes for LFS files, so no HEAD probe is needed,
and which only accepts `master` (an HF-imported repo still 404s on `main`)
— and downloads through `/models/{id}/resolve/{rev}/{path}`. That URL
redirects to a CDN target carrying a time-limited `auth_key`, so it is
rebuilt on every request and never cached, which is also what keeps
resumable Range requests working.
- `hf_handlers.py`/`HfHandler` become `model_source_handlers.py`/
`ModelSourceHandler` with `list_model_source_files` and
`download_model_source`. New routes `/api/lm/model-source-files` and
`/api/lm/download-model-source`; the old `/api/lm/hf-repo-files` and
`/api/lm/download-hf-model` paths stay as aliases, and a payload without
`platform` still means Hugging Face, so existing callers are unaffected.
- A downloaded sidecar now records `source_platform` + `source_url` (with the
`hf_url` alias only for Hugging Face) instead of always writing `hf_url`,
and `use_default_paths` files ModelScope downloads under
`modelscope/<owner>/<repo>`. The now-unused shared HF aiohttp session and
its shutdown hook are gone; providers open short-lived sessions.
- Frontend: `detectUrlType` returns the platform-neutral
`model-source-repo` / `model-source-file` plus an explicit `platform`, the
DownloadManager's `hf*` state and methods are renamed to `source*`, every
`source === 'huggingface'` check becomes `isExternalModelSource()`, and
batch groups are keyed by `platform:repo` so the same `owner/name` on two
sites renders as two groups. A bare `owner/name` still means Hugging Face.
- `is_valid_source_id()` centralises repo-id validation (exactly
`owner/name`, no traversal, no leading dot). This also fixes the old HF
download check that rejected any dot in the name, i.e. legitimate repos
such as `black-forest-labs/FLUX.1-dev`.
Verified against the live APIs: the example repo lists 8 weight files with
correct sizes, and a ranged GET of the built resolve URL returns 206 after
following the redirect to the CDN. Backend 2853 passed; frontend 1143 JS +
91 Vue passed. The nine locales carry the refreshed download copy in the
next commit.
A model file could only ever be linked to huggingface.co: `set_hf_url`
validated the URL with a huggingface-only regex, the agent fetched the card
from a hardcoded HF URL, and the readme processor built every relative image
path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the
same model-card convention (README.md + YAML frontmatter, often carrying
`base_model:` and `trigger_words:`) behind a public, key-less API, so the
enrichment pipeline could already serve it - it was the plumbing that was
HF-shaped, not the idea.
Make the external source a first-class, provider-driven concept:
- New `py/services/model_sources/` registry. A `ModelSource` owns URL
recognition (lenient for stored values, strict for user input), the
canonical page URL, model-card fetching, the asset base URL and the
capability flags. `HuggingFaceSource` is the previous logic relocated;
`ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md`
and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is
link-only on purpose: tensor.art answers plain HTTP clients with a
Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud /
cn.tensorart.net) rejects every /v1/model/* route with "invalid
authorization header", so it declares supports_enrichment=False rather than
failing silently later.
- Metadata gains `source_platform` + `source_url`; `hf_url` stays as a
read/write alias, written only for Hugging Face, so existing sidecars,
cached rows and third-party consumers keep working. Normalisation runs at
the scanner, the persistent cache (both directions, plus two new columns
behind an ALTER migration) and the linking handler - which is what stops a
user who switches sources from leaving a stale `hf_url` on a ModelScope
model.
- The agent pipeline keys off the provider instead of `hf_url`: the fast-fail
gate now explains *why* a model is skipped (no source / unknown source /
source without a reachable card), the prompt context exposes
source_url/source_id/source_label/asset_base_url while still filling the
legacy hf_url/repo aliases, and the four README image extractors take a
base_url (defaulting to HF) so relative paths resolve against the right
site. Version grouping generalises to hf: / ms: / ta: keys.
- `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but
accepts `source_url`, validates against every provider and returns the
platform. `GET /api/lm/model-sources` lets the UI render the supported-site
list from the server.
- Frontend: a `modelSourceHelpers` mirror of the registry drives the link
dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the
version-group key and the enrichment gate; the versions tab no longer sends
ms:/ta: keys to the CivitAI API.
TensorArt stays in the list because provenance is worth keeping even when the
card is unreadable - the dialog says so plainly ("Sites that don't expose one
(currently TensorArt) can only be linked") and the context menu disables
enrichment with a matching tooltip, instead of the user getting
"Unsupported URL".
Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a
1882-byte card whose frontmatter carries base_model/tags/trigger_words, and
relative images resolve to .../resolve/master/....
Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest
tests/i18n and a Jinja compile pass over templates/. The nine locales carry
[TODO: Translate] for the new strings, completed in the next commit.
Other Models management is opt-in and its folders come from
folder_paths.get_folder_paths(). In plugin mode ComfyUI registers vae,
upscale_models, text_encoders, clip_vision and controlnet out of the box, so
enabling the feature works immediately. Standalone only knows the keys present
in settings.json.folder_paths, and that file is edited by hand - there is no UI
for those keys - so a standalone user who followed the announcement banner
reached "Enable Other Models" and then an empty page.
Gate the announcement on the capability instead of on how the process was
started:
- Config.get_other_models_availability() probes every canonical other key
(legacy clip collapses into text_encoders where the host exposes
map_legacy) and reports which sub_types resolve to a folder that exists on
disk. It deliberately ignores enable_other_models: the question is "could
this work here at all?". An empty folder counts, because CivitAI downloads
can target it.
- /api/lm/settings exposes it as the derived, non-persisted
other_models_paths_available flag; a probe failure yields null and the
banner fails open.
- BannerService only registers the announcement when the flag is not false.
`=== false` (not falsy) keeps a cached/older payload working, and nothing is
written to dismissed_banners, so the banner can return once folders exist.
- The Other page grows an "enabled but nothing to scan" empty state driven by
config.other_roots, showing the settings.json snippet for standalone and a
pointer to ComfyUI model paths otherwise, plus an Open Settings action. It
also covers the corner where only a non-default sub_type has a folder.
Translate the six other.noPaths.* keys into all nine locales and record the
new "folder key" / "on disk" terminology in the i18n guidelines.
Backend tests and pytest tests/i18n could not run in this environment (no
pytest/platformdirs); the probe was exercised against a stubbed folder_paths.
Frontend: 120 files / 1101 JS tests passed.
A model could have CivitAI metadata and a HuggingFace link at the same time,
but only one of the two "View on ..." entries ever rendered, because both the
model modal and the card globe asked the `from_civitai` provenance flag which
source to show. `set_hf_url` wrote `false` and a CivitAI refresh wrote `true`,
so whichever ran last erased the other: linking HF hid "View on CivitAI" even
though the civitai payload was still in the sidecar, and (on the card) a later
refresh pointed the single globe icon back at CivitAI, hiding the HF entry.
Decide the links from the data itself instead:
- `set_hf_url` no longer touches `from_civitai`; it records where the metadata
came from, and HF provenance is already tracked by `hf_url`.
- Add `hasCivitaiSource(civitai)` in the shared card/modal utils and gate the
modal's CivitAI link, the card globe (title, enabled state, click target,
new `data-has_civitai`) and the context-menu `civitai` action on actual
CivitAI data (`modelId` / `model_id` / `id`). A dual-source model now shows
both links, and a CivitAI-only model with no `hf_url` stays as before.
- Agent HF enrichment (`PostProcessor.is_hf_model`) keyed off
`not from_civitai`, which stopped being a synonym for "has an HF source" once
both sources can coexist (and already broke after a CivitAI refresh flipped
the flag back to true). Key it off `hf_url` directly; the post-processor
tests move to that discriminator and gain a dual-source case.
Regression tests: the set-hf-url handler preserves civitai + `from_civitai`
and no longer forces the flag false, the modal renders both links (including
with `from_civitai: false`), and the card globe targets/opens the right source
and is disabled when neither is available.
Backend: 2749 passed. Frontend: 1098 JS + 91 Vue tests passed.
DEFAULT_ENABLED_OTHER_SUB_TYPES managed vae, upscaler, text_encoder and
clip_vision while controlnet was the sole opt-in type. That split was not
defensible on demand breadth: ControlNet is the broader category by install
base, and clip_vision is the narrower one (IPAdapter/SVD image conditioning,
usually one to three files) whose CivitAI type is retired upstream.
Keep the default set to the dependency-style assets every pipeline needs and
where "which one am I actually using" is the real problem - VAE, upscalers
and text encoders - and treat clip_vision and controlnet symmetrically as
opt-in. The feature is still unreleased, so the change needs no migration.
- Sync all five surfaces holding a default: DEFAULT_ENABLED_OTHER_SUB_TYPES,
DEFAULT_SETTINGS, both DEFAULT_SETTINGS_BASE/createDefaultSettings lists,
updateOtherModelsControls()'s fallback and the Jinja fallback.
- The selection is persisted per user, so only the untouched default moves;
existing default_other_roots entries for a disabled sub_type are preserved.
- Fix the Jinja fallback using `or`, which treated an all-unchecked empty
allow-list as "unset" and re-checked every box on render; `is none` keeps
the empty list empty.
- Document the revised defaults and rationale in the plan.
Tests assert the new default trio, the normalize fallback, that both opt-in
types stay out of the default scan, and the auto-set iteration test now
enables clip_vision explicitly since it exercises the loop, not the default.
Enabling Other Models logged two warnings on a stock ComfyUI install:
Detected the same folder '.../clip' under multiple other-model categories
('.../clip' is already mapped). Keeping the first category; please fix
your path configuration.
Nothing was wrong with the configuration. ComfyUI's folder_paths rewrites
legacy names before every access (map_legacy: clip -> text_encoders,
unet -> diffusion_models) and registers both legacy directories under the
canonical key, so get_folder_paths("clip") returns exactly the same list as
get_folder_paths("text_encoders"). Both keys are in the enabled allow-list,
so the second pass hit the overlap guard for every text-encoder folder and
printed advice the user cannot act on. The path list itself was correct
(deduped), only the message was wrong.
- Config._collapse_legacy_folder_keys() drops a key when the host exposes
map_legacy and resolves it to another queried key. That is provably
lossless: an empty canonical list implies an empty alias list. The
standalone MockFolderPaths has no map_legacy and its keys are independent
settings.json entries, so every key is still queried there.
- _prepare_other_paths() now tracks the claiming sub_type alongside the
business path and downgrades a same-sub_type duplicate to debug, keeping
the warning for a genuine cross-category collision (and naming the other
category in the message).
Regression tests cover the aliased-key layout (no warning, no redundant
query, both folders still managed) and the same-sub_type duplicate, and the
opt-in test is parametrized over controlnet and clip_vision.
get_download_path_template() fell back to "{base_model}/{first_tag}" for any
unconfigured model type, so other-model downloads were silently nested under an
arbitrary CivitAI tag even though the settings UI exposes no template row for
"other" and priority_tags has no "other" entry (making {first_tag} resolve to
tags[0]).
Add DEFAULT_DOWNLOAD_PATH_TEMPLATES with other -> "" so unconfigured and
unknown types resolve to a flat layout under the already sub_type-scoped
default_other_roots; explicit settings.json values still win. Mirror the flat
default in the frontend DEFAULT_PATH_TEMPLATES and stop the download/move
default-path previews from rendering "/undefined" or a dangling slash.
Implements the backend slice (B1-B7) of
lm-civitai-extension/docs/other-models-support.md, which lets the companion
browser extension detect, badge and download the opt-in Other Models types
(VAE / upscaler / text encoder / CLIP vision / ControlNet).
ModelLibraryHandler:
- _normalize_model_type() learns the CivitAI other aliases (vae, upscaler,
textencoder, clip, clipvision, controlnet, other) and maps them to "other".
- _get_scanner_for_type() resolves "other" through the other scanner, but only
while enable_other_models is on, so model-versions-status and
model-version-download-status keep their legacy 400 when the feature is off.
- check_model_exists() / check_models_exist() consult the other scanner last
(lora -> checkpoint -> embedding -> other) and report modelType "other".
With the feature disabled both endpoints stay byte-identical to before and
the other scanner is never touched.
DownloadManager:
- The four other-type default-path failures now carry a machine-readable
"reason" (contract C4): other_models_disabled, other_sub_type_disabled,
other_no_default_root, other_sub_type_undecidable. The user-facing "error"
strings are unchanged; the key is additive and reaches the client because
both download endpoints pass the result dict through verbatim.
Tests cover the opt-in on/off branches for both existence endpoints, mixed
lora + other ids in the batch endpoint, the CivitAI alias acceptance and the
400 regression for unknown types, and the exact reason/error pairs for all
four download failure modes.
Other Models management is now opt-in: enable_other_models (default false)
plus the enabled_other_sub_types allow-list replace the unreleased additive
enabled_other_folders key.
- config._get_enabled_other_folder_keys() is the single scan gate; a new
refresh_other_roots() rebuilds roots and preview roots on toggle.
- ModelScanner gains a _should_keep_cached_entry() hydration hook and
on_library_changed(reconcile=...) so switching a sub_type off drops its
entries (and hash/autov3 rows) at load time and switching it on rescans.
- OtherScanner filters location-derived entries accordingly.
- Other routes reject every other type while off (or a disabled sub_type) and
expose an "other_disabled" page flag; download routing returns a disabled
marker instead of guessing; the download manager refuses other-type
downloads and default-path routing for switched-off sub_types.
- Doctor / init-status / refresh-all skip the other scanner while off; the
scanner stays registered so staged pending-deletes still merge.
- Tests updated with explicit opt-in fixtures plus new gating coverage.
The include_empty folder tree (download/move modals) walked every model
root synchronously on the event loop via get_all_folders(). On network
(NAS) roots this froze the whole server for the duration of the walk —
blocking WebSocket progress, aria2 RPC and the download queue — and the
5s TTL re-triggered the walk on nearly every modal interaction.
The scanners already visit every directory during cache scans, so record
the full directory list (including empty folders) there instead:
- _gather_model_data/_reconcile_cache collect directories during the
existing walks; reconcile refreshes and persists the list even when no
model files changed.
- ModelCache gains an all_folders field (None = never recorded).
- PersistentModelCache stores the list in a new folders table, with a
cache_meta flag distinguishing 'recorded empty' from legacy snapshots.
- get_all_folders() is now a pure in-memory read. A legacy snapshot
triggers a one-shot backfill walk in a worker thread (never on the
event loop) that records and persists the list.
- Moves add the destination folder (and parents) incrementally instead
of invalidating a TTL cache.
The browse endpoint and its frontend were written with POSIX-only
assumptions, so on Windows pressing Browse immediately failed with
"Access denied to this directory":
- The frontend opened the browser at "/", which resolves to the
current drive root on Windows.
- The allowlist check used Path("/"), which has no drive letter on
Windows, so relative_to() rejected every drive-qualified path —
anything outside the user profile was denied.
Fixes:
- Empty browse path now defaults to the user home directory instead of
erroring; the frontend sends "" rather than the POSIX-only "/".
- The access check is platform-aware (drive-qualified on Windows,
absolute on POSIX).
- Parent navigation uses the server-provided parent_path; the root
check is now path.parent == path (the old str/anchor comparison
self-looped at Windows drive roots).
- Browsing up from a Windows drive root shows a virtual list of
available drives so users can switch drives without typing a path.
A no-change Refresh still computed os.path.realpath for every model file
in the library and for every cached entry. Both values are only ever
consulted when a discovered file is missing from the cache, so on a
50k-file library they cost ~1.3s and ~0.6s while being used zero times.
- Compute the per-file realpath only after the exact cache match fails
- Build the physical-path alias map lazily on the first miss; the
cross-run alias guard (overlapping roots / symlink layout changes)
still keeps the cached entry instead of a delete + re-add, which would
re-read metadata and re-hash the whole library
- Snapshot get_model_roots() once for the new-file pass instead of
re-reading it for every added file
- Run the duplicate-path integrity pass only when the snapshot already
contained duplicates or files were appended; a clean, unchanged cache
has nothing to clean. Duplicates can only be introduced by external
code rewriting raw_data or by this pass's own appends.
Zero-change reconcile drops from ~1400ms to ~120ms on 50k files, and an
alias flip still re-processes 0 files (#1108 investigation).
The download modal's location step decided between checkpoint and unet
roots using only the CivitAI file-type signal, while the backend also
falls back to DIFFUSION_MODEL_BASE_MODELS. Models like Anima (file type
"Model") were offered checkpoint roots in the UI even though
use_default_paths would route them to the unet root.
- Extract the two-tier decision into py/services/download_routing.py and
reuse it in DownloadManager._execute_download
- Add POST /api/lm/download/routing so the UI asks the backend for the
routing decision; fall back to the local file-type check on failure
- ModelVersionsTab: search both checkpoint and unet roots when resolving
an existing version's download path
A cancel landing between aria2.addUri acceptance and the _transfers
registration found no tracked transfer, so DownloadManager tolerated the
"not found" and only cancelled the asyncio task — the daemon kept
downloading the file untracked while history showed the download as
cancelled.
- Register the gid in _transfers immediately after addUri returns,
before any further await (state-store persist moved after it)
- Shield the addUri RPC so a mid-flight cancellation still learns the
accepted gid and forceRemoves it before re-raising CancelledError
- On cancellation during the state persist, remove the daemon transfer
unless it is paused (skip_download relies on paused gids surviving)
Moving a checkpoint into a unet root (or vice versa) moved the file and
updated the in-memory cache, but three stale spots survived until a
manual cache rebuild:
- The moved .metadata.json kept the old sub_type, and the opportunistic
sync_cache_from_metadata path (fired by get_model_metadata and example
image metadata updates) trusted it, reverting the cache entry and the
SQLite snapshot to the pre-move sub_type. Loader nodes filter strictly
on sub_type, so the model stayed listed under the old type.
- The manager page discarded the move response's cache_entry, so the
card badge (CKPT/DM) and context menu label kept showing the old type.
Fixes:
- move_model now re-resolves sub_type from the target location (new
resolve_sub_type_for_path hook) and persists it into the moved
.metadata.json.
- _sync_cache_from_metadata_impl runs desired entries through
adjust_cached_entry so location-derived fields cannot be re-poisoned
by stale metadata snapshots.
- MoveManager carries cache_entry.sub_type into the in-place card
update so badge and context menu reflect the new type immediately.
- Snapshot pre-rematch entry state (reconnectSnapshot) so rematched
entries can be undone via the existing restore flow
- Bulk missing-LoRA downloads mark unresolvable failures hash-invalid,
flipping those entries from download to reconnect candidacy
- Recipe modal always offers a reconnect action next to download for
missing LoRA entries
- Rematch runs collect an opt-in relaxed-matching choice (also reconnect
missing models by file name) via a pre-run options dialog on the
global, bulk and single-recipe entries
- L4 (filename-level) matches are listed in a results dialog with
per-entry undo
The field dates back to a development-stage bug in the enrich-metadata
(agent) pipeline, which briefly wrote trigger words at the top level of
model metadata instead of the established civitai.trainedWords location.
The write path was fixed before the feature merged to main (PR #1013)
and never shipped in any release, so no writer has existed since.
Remove the leftover pieces:
- BaseModelMetadata.trainedWords field (py/utils/models.py); sidecars
from that dev window now pass the key through _unknown_fields instead
- HF download handler's strip-empty-trainedWords special case, reverting
to saving the metadata object directly (py/routes/handlers/hf_handlers.py)
- trainedWords in the LLM enrichment context (agent_service.py)
- matching fallbacks/fixtures in the enrich_hf_validation harness and
post-processor test
Trigger words continue to live in civitai.trainedWords for all model
sources, which is what the UI, agent post-processor, and metadata sync
all read and write.
models.dev is served by Cloudflare with brotli compression when the
client advertises it, and brotli is a required dependency here, so
aiohttp always negotiates br. A corrupted br stream can crash the
native decoder with a Windows access violation (a Python-level
exception handler cannot catch it), or produce garbage bytes.
Send an explicit "Accept-Encoding: gzip, deflate" header on the model
catalog and Ollama model-list requests so the server never returns
brotli. zlib handles corrupt gzip data by raising ContentEncodingError
(an aiohttp.ClientError subclass), which the existing handlers already
catch and degrade to a warning with an empty-catalog fallback.
The public REST API rewrites files[].name to "{model}_{version}" for
non-LoRA model types, so every precision variant of a multi-file version
shared one name and landed on disk with a random short-hash suffix.
Fetch the raw stored filename from the model-versions/mini endpoint
(always pinned with modelFileId) and use it for the on-disk name and
metadata when available; fall back silently to the REST name otherwise.
CivArchive already serves raw names and is skipped.
Recipes imported from CivitAI image URLs can contain 0 LoRAs: the backend
only sees the REST image API + EXIF, while the complete generation data
lives in the image page's internal trpc payload (see
docs/recipe-civitai-image-no-metadata.md). When the companion
lm-civitai-extension is installed with a valid license, re-import (single
and bulk) of CivitAI-image-sourced recipes is now delegated to the
extension via DOM CustomEvents; the extension scrapes the image page with
the user's session and calls back into the reimport endpoint with the
full metadata payload. Without the extension (or with an invalid license)
the native path runs unchanged.
- POST /api/lm/recipe/{id}/reimport accepts optional payload params
(image_url/name/resources/gen_params/base_model/tags); the payload path
reuses the import-remote engine with reimport semantics (user-edit
carryover, delete-after-save), and malformed/failed payloads fall back
to the legacy URL import. Response gains loras_count.
- The endpoint also accepts GET: the extension is GET-only by convention
(documented in AGENTS.md).
- New static/js/utils/extensionReimportBridge.js (probeExtension /
delegateReimport / getCivitaiImageInfo) wired into RecipeContextMenu
and BulkManager with silent native fallback.
- i18n: toast.recipes.reimportingViaExtension added and translated in
all 9 locales.
resp.json() raises UnicodeDecodeError (not JSONDecodeError) when the
remote body contains invalid UTF-8 bytes, which the exception handler
did not catch and could crash the app. Apply the same fix to both
_load_model_catalog and fetch_ollama_models so they fall back to an
empty catalog. Add regression tests for both paths.
The recipe "Repair Metadata" action has been marked Deprecated in the UI
for a while and cannot reliably recover recipes imported from CivitAI URLs
whose REST meta has no resources/hashes and whose image has no embedded
metadata (e.g. CivitAI-only generation data). Drop the feature end to end.
Backend:
- remove repair routes (repair, cancel-repair, recipe/{id}/repair,
repair-bulk, repair-progress) and their handler mappings/methods
- remove RecipeScanner repair_all_recipes / repair_recipe_by_id /
_repair_single_recipe and REPAIR_VERSION
- remove WebSocketManager recipe-repair progress channel
- drop repair_version column from the persistent recipe cache
- rematch mutual-exclusion now only checks rematch
Frontend:
- remove repair entries from per-recipe, bulk and global context menus
- remove repairRecipe / repairSelectedRecipes / repairRecipes + cancelRepair
and the repairBulk API client method/endpoint
- drop recipe-repair i18n keys (synced across locales; doctor keys kept)
Tests/docs: delete test_recipe_repair.py, update scaffolding/routes/ws/
persistent-cache/integration tests and i18n guideline examples.
Adds a 'Keep Action Bar Visible' toggle (default off) under
Settings > Interface > Layout Settings. When enabled, the controls bar
(Refresh, Download, etc.) and the breadcrumb nav are wrapped in a shared
sticky container (.sticky-topbar) so both stay pinned as one unit; when
disabled, the wrapper is display: contents and the original behavior
(only the breadcrumb stays visible) is preserved.
Re-importing a file-imported recipe fell back to its own saved preview
image, then recorded that internal path as the new recipe's source_path.
Since the old preview is deleted with the old recipe, this left a
dangling source_path that showed up as a bogus source URL and blocked
any further re-import with 'no re-importable source'.
Only persist source_path when the re-import source is an accessible
external file; otherwise keep it empty. Also let a dangling non-URL
source_path fall back to the recipe's own image so existing affected
recipes can re-import again.
Broadcast typed scan_progress messages over /ws/fetch-progress from the
manual refresh/rebuild paths of ModelScanner and RecipeScanner, and
render percent, processed/total, current file name and an EMA-smoothed
ETA in the loading overlay. Hardcoded refresh strings move to i18n
(common.scanProgress); WS connection failure falls back to the previous
static loading behavior.
Autocomplete suggestions were ranked purely by relevance across the whole
library, so same-named loras from different subfolders interleaved and were
hard to tell apart. Results are now bucketed by folder (root first, then
alphabetically, with nested paths sorting naturally) while keeping the
existing relevance ordering within each folder group.
The LoRA Manager page kept its active filters in localStorage, which the
ComfyUI-side autocomplete read directly. When the two run in different
browsers, origins, or the ComfyUI Desktop Electron shell, localStorage is
not shared and the active-filters search silently did nothing.
The manager page now mirrors its filter state to a server-side in-memory
store (PUT /api/lm/{prefix}/active-filters), pushed on every change via a
storage-listener hook and once on page load. The autocomplete widget sends
only use_active_filters=true, and the relative-paths endpoint injects the
stored filters into the search, with explicit query params taking
precedence.
Add a de-emphasized meta footer to the recipe modal, mirroring the model
modal's hash footnote: a clickable file location on the left (opens the
recipe JSON via the generic open-file-location route, with the Docker
clipboard fallback) and a middle-truncated recipe ID with copy button on
the right.
The recipe detail API now exposes recipe_json_path so the frontend does
not have to guess the on-disk storage layout. Translations for the new
recipes.modal.* keys are filled in for all 9 locales, reusing the model
modal's openFileLocation wording per locale.
The SHA256 of an empty byte string (written by repackaging tools into
safetensors metadata, or produced by hashing an empty/unreadable file)
was previously resolved against CivitAI's by-hash API, which can contain
polluted entries for it (e.g. a broken SD 1.5 LoRA whose AutoV3 equals
the placeholder) and falsely attributed the wrong model to a recipe.
Guard all lookup paths for the 10/12/64-char AutoV2/AutoV3/full-SHA256
spellings: CivitaiClient.get_model_by_hash/_fetch_version_by_hash return
not-found without a request, and ModelHashIndex ignores the placeholder
in has_hash/get_path/add_autov3.
The Automatic1111 metadata parser keeps the LoRA item itself when its
hash is the placeholder: it matches by filename locally, or retains the
entry with an empty hash flagged hashInvalid (unresolvable-hash state in
the UI, with reconnect as the remedy) instead of dropping it or resolving
it to a polluted CivitAI entry.
Recipes imported by drag & drop / file-picker record no source_path and
were rejected by re-import. Fall back to the recipe's own saved image,
which still carries the original embedded generation metadata.
Re-import now re-parses that original metadata instead of the appended
recipe JSON block, so parser upgrades produce fresh results. The
already-optimized preview image is kept verbatim: only its WebP EXIF
chunk is rewritten in place to replace the recipe metadata block, and
the recipe JSON is rewritten with the new analysis plus carried-over
user edits.
Normalize undetermined recipe base_model to None in RecipeFormatParser
(previously ''). get_base_models now reports an "Unknown" bucket backed
by a dedicated __unknown__ marker, and the listing filter matches it
against recipes whose base model is falsy. Frontend renders the bucket
label as "Unknown" while filtering via the marker.
Tests: handler, scanner, parser, and frontend filtering.
Checkpoint entries that cannot be restored by download (deleted,
unresolvable hash, or name-only remnants with no CivitAI identifiers)
now get the same remediation chain LoRAs already had:
- scanner: parameterized reconnect-suggestion ranking, update/restore/
set-hash-invalid for the checkpoint entry, and clear hashInvalid on
rematch write-back (was only done for LoRAs)
- persistence/handlers/routes: reconnect/restore/reconnect-suggestions/
mark-hash-invalid endpoints under /api/lm/recipe/checkpoint/*
- modal: checkpoint reconnect UI (deleted/hash-invalid badges, inline
form with suggestions, undo for reconnected entries); download
failures mark the hash invalid only on explicit unresolvable signals
(not found/deleted/404/410), matching the LoRA rule
- css: checkpoint undo button shares the LoRA undo styles
- i18n: the 14 new keys translated in all 9 locales
Record import provenance on every recipe: a new import_info block
(channel, machine-readable no-LoRA reason, diagnostic details) built at
import time across all channels (batch import, single URL, local file,
upload, widget save, re-imports) and persisted in the recipe JSON plus
the SQLite persistent cache (new import_info_json column with ALTER
TABLE migration).
The recipe modal renders the empty LoRA list with a collapsed details
panel showing the import method, the reason (CivitAI API returned no
LoRA resource data, API meta missing, no embedded metadata, ComfyUI
workflow metadata, video, unparsable format), and recorded diagnostics.
Legacy recipes without import_info fall back to heuristics labeled as
inferred. Genuine no-LoRA generations show no panel.
CivitAI images are always classified by API meta shape: the onsite
generator writes A1111-style EXIF without LoRA references, so parsed
EXIF cannot prove "no LoRAs used".
Adds recipes.resources.noLoras* i18n keys (all 10 locales) plus
frontend vitest and backend pytest coverage.
- Remove py/nodes/random_checkpoint_loader.py and random_unet_loader.py
- Remove their dedicated test file
- Clean up imports and NODE_CLASS_MAPPINGS in __init__.py
- Update loader-pool comments/docstrings to reference the remaining Checkpoint/Unet Loader nodes' control_after_generate feature
Enhance the deleted-LoRA reconnect flow in the recipe modal:
- Suggest local reconnect candidates when the panel opens, ranked by
identity (same hash / same CivitAI version) then filename/name
similarity, with a hard filter on confident base-model mismatches;
the input gets a Combobox backed by the same endpoint as you type.
- Snapshot the pre-reconnect entry and offer a permanent restore:
reconnected entries show an undo icon at the right end of the info
row, with the original filename in the tooltip.
- Relax the manual reconnect base-model guard to a three-tier check:
exact/unknown labels pass silently, same-architecture families
(e.g. Pony <-> Illustrious) pass with a warning toast, and only
cross-architecture mismatches stay hard-rejected.
The recipe card pill counted LoRAs deleted from the source (isDeleted) as
available, showing a green 'ready 2/2' for recipes that cannot be fully
reproduced. LoRAs with an unresolvable hash (hashInvalid) were counted as
missing/downloadable even though downloads always fail, and recipe syntax
generation emitted broken tokens for them.
- Four-state status on RecipeCard pill and RecipeTab badge: ready (all in
library), missing (downloadable, red, keeps the action cue), partial
(unobtainable entries skipped when used, amber, fa-circle-minus),
unavailable (nothing usable, gray, fa-ban)
- Pill numerator is now the real in-library count; tooltips spell out
missing vs unavailable (deleted from source or unresolvable hash)
- get_recipe_syntax_tokens skips hashInvalid entries like deleted ones
instead of emitting tokens pointing at nonexistent files
- Bulk missing-download manager and recipe context menu exclude
hashInvalid LoRAs, matching the modal's per-item download block
- New locale keys loraStatus.missingAndUnavailable/partial/noneUsable,
translated for all 9 non-en locales
A failed aria2 transfer deleted the partial payload while keeping its
.aria2 control file, and "No URI available" (expired CivitAI signed URL)
was treated as a permanent failure, wasting nearly-complete downloads.
- Re-schedule the transfer with a freshly resolved signed URL and
continue=true when aria2 reports "No URI available", bounded by
MAX_TRANSFER_RECOVERY_ATTEMPTS
- Keep payload and .aria2 control file together as a resumable pair
after a failed transfer instead of deleting the payload
- Report and remove orphaned .aria2 control files that have no payload,
both after failures and when restoring persisted downloads
Fixes#1088
- import: prefer A1111 Lora hashes (12-char AutoV3) over conflicting Hashes
JSON values; recover the quote-wrapped AutoV3 from CivitAI image API meta;
merge EXIF-parsed LoRAs when the API-only parse yields none (meta=null)
- rematch: treat entries whose hash failed CivitAI resolution (hashInvalid)
as unresolved candidates; clear the flag on rematch/reconnect write-back
- download: persist hashInvalid and show a distinct toast when hash lookup
returns "Model not found", so unresolvable entries become recoverable
- ui: add Unresolvable Hash badge styling and reconnect affordance
- i18n: translate the new keys across all 10 locales
Bulk delete merged staged batches by physically moving each loser's
files into the winner's batch dir with os.rename. Cross-volume bulks
(winner and loser on different filesystems) always hit EXDEV, forcing a
rollback and degrading to the batch_ids array with per-batch undo.
Merge is now manifest-only: loser entries are appended to the winner's
manifest with their staged paths unchanged, so staged files keep living
in each model's own .lm-pending-delete/<batch_id> dir (no data IO, no
EXDEV). Loser dirs are recorded in the winner manifest's merged_sources
and each loser manifest is stamped merged_into so its own purge timer, a
post-restart sweep or a direct undo call no-op. A cross-volume bulk is
one undoable batch again, and undo/purge clean up the loser dirs once
the merged batch settles.
Phase 2 of docs/plans/issue-1085-rate-limit-design.md:
- Batch import: items that fail due to vendor rate limiting are now
SKIPPED with a "re-run the import later" hint instead of FAILED, so a
transient 429 no longer pollutes failure accounting; the progress
broadcast carries a rate_limited flag.
- Batch import UI: show a one-time "rate limited — slowing down" toast
and swap the running status text while rate_limited; i18n keys synced
to all locales.
- Downloader: download_file / download_to_memory / get_response_headers
register 429 cooldowns with the RateLimitCoordinator, so subsequent
API calls queue behind a download-triggered rate-limit window.
Implement Phase 1 of docs/plans/issue-1085-rate-limit-design.md:
- New RateLimitCoordinator: per-host shared Retry-After gate with
exponential backoff (30s base, 1800s cap), minimum inter-request pacing
(default 0.75s), herd-free waiter serialization via per-destination
locks, and a bounded wait (default 300s) that raises instead of parking.
- Downloader.make_request: connectivity-guard fail-fast first, then gate
pacing; on 429 register the cooldown and wait-and-resend (bounded);
errors that passed through the gate are marked gate_handled.
- FallbackMetadataProvider / MetadataSyncService: a network provider 429
no longer fails over to other network providers (stops the CivArchive
flood); sqlite stays as local last resort. Rate-limited lookups now
report "Rate limited" instead of "Model not found", so transient 429s
no longer mark models civitai_deleted.
- _RateLimitRetryHelper skips its own sleep for gate_handled errors,
removing the double wait.
- New settings: rate_limit_gate_enabled, rate_limit_max_wait_seconds,
rate_limit_min_interval_seconds.
Address the rate-limit flood and secondary errors seen during large
recipe ingestion (example-images directory import):
- batch import: share one adaptive-concurrency semaphore across the whole
batch (previously each item got a fresh semaphore, so the min/max
concurrency bounds never applied and every item ran concurrently);
synchronize the shared semaphore capacity after each completed item.
- comfy parser: guard ckpt_name against list/None values so re.search no
longer raises TypeError and fails the whole image import.
- civarchive client: normalize empty-string failure payloads to
"Request failed" and treat a missing payload as an error, fixing the
"'NoneType' object has no attribute 'get'" crash.
- civarchive client: log connectivity-guard offline-cooldown
short-circuits at DEBUG instead of one ERROR per request.
Add LORA_MANAGER_SETTINGS_DIR env var and standalone --settings-path to pin
the settings location (settings.json, cache/, wildcards/, backups/, logs/,
stats/) to an arbitrary directory. The override takes precedence over
portable mode and the platform user config dir, and skips legacy migration,
so sandboxed dev/E2E runs no longer need to write settings.json in the repo
root or collide with the real instance.
standalone.py pre-scans argv for --settings-path at import time because the
settings location is resolved before main() parses arguments. SettingsManager
portable-switch migration is a no-op while the directory is pinned.
Update the lora-manager-e2e skill (prefer --settings-path sandboxing;
start_server.py passes it through) and the lora-manager-runtime-context
skill (document precedence; inspect script honors the override).
Will Miao