Commit Graph

80 Commits

Author SHA1 Message Date
Will Miao 4064ea7d3a refactor(agent): apply model-source data without an LLM
`_build_prompt_context()` was only reached when the LLM was configured,
so a user with no provider got nothing at all from a linked model source
— no preview, no example images, no author summary, no tags — even
though all of that is deterministic data from a public API.

Split the model-card fetch into `_load_source_card()`, which runs for
every source-backed enrichment, and have the post-processor apply its
result whether or not the LLM runs. The prompt is then built from the
already-fetched card rather than re-fetching it.

Invoking "Enrich Metadata with AI" still always calls the provider; a
model source supplying a description, images and tags is not treated as
a reason to skip it, since the LLM's summary and notes are richer and an
action that silently does not call out to the provider would be
unpredictable. The site data acts as a fallback for the gaps the LLM
leaves.

Add `base_model_resolver.resolve_base_model()` to map the site's own
names (`krea/Krea-2-Turbo`, `KREA_2_TURBO`) onto the canonical
vocabulary, used only when the LLM returns no base model. It is strictly
conservative — exact normalised matching plus a bounded set of variant
suffixes, and it only ever returns a name that is already in the
vocabulary — so an uncertain hint defers to the LLM instead of writing a
plausible-looking wrong value.
2026-09-14 20:39:10 +08:00
Will Miao 38d4c59b4c feat(download): support ModelScope repositories in the URL downloader
ModelScope became a linkable source, but downloading from it was impossible:
the URL picker only recognised huggingface.co, the file listing hit a
huggingface-only endpoint, the resolve URL was hardcoded, and the default
path template always wrote into a `huggingface/` directory.

Move the download knowledge into the providers so the handlers stay generic:

- `ModelSource` gains `list_files()`, `file_download_url()`,
  `default_revision` and `default_subdir`. `HuggingFaceSource` keeps the Hub
  tree API (`/api/models/{id}/tree/{rev}`, LFS-aware sizes, `main`).
  `ModelScopeSource` uses `/api/v1/models/{id}/repo/files?Revision=master`
  — which reports real byte sizes for LFS files, so no HEAD probe is needed,
  and which only accepts `master` (an HF-imported repo still 404s on `main`)
  — and downloads through `/models/{id}/resolve/{rev}/{path}`. That URL
  redirects to a CDN target carrying a time-limited `auth_key`, so it is
  rebuilt on every request and never cached, which is also what keeps
  resumable Range requests working.
- `hf_handlers.py`/`HfHandler` become `model_source_handlers.py`/
  `ModelSourceHandler` with `list_model_source_files` and
  `download_model_source`. New routes `/api/lm/model-source-files` and
  `/api/lm/download-model-source`; the old `/api/lm/hf-repo-files` and
  `/api/lm/download-hf-model` paths stay as aliases, and a payload without
  `platform` still means Hugging Face, so existing callers are unaffected.
- A downloaded sidecar now records `source_platform` + `source_url` (with the
  `hf_url` alias only for Hugging Face) instead of always writing `hf_url`,
  and `use_default_paths` files ModelScope downloads under
  `modelscope/<owner>/<repo>`. The now-unused shared HF aiohttp session and
  its shutdown hook are gone; providers open short-lived sessions.
- Frontend: `detectUrlType` returns the platform-neutral
  `model-source-repo` / `model-source-file` plus an explicit `platform`, the
  DownloadManager's `hf*` state and methods are renamed to `source*`, every
  `source === 'huggingface'` check becomes `isExternalModelSource()`, and
  batch groups are keyed by `platform:repo` so the same `owner/name` on two
  sites renders as two groups. A bare `owner/name` still means Hugging Face.
- `is_valid_source_id()` centralises repo-id validation (exactly
  `owner/name`, no traversal, no leading dot). This also fixes the old HF
  download check that rejected any dot in the name, i.e. legitimate repos
  such as `black-forest-labs/FLUX.1-dev`.

Verified against the live APIs: the example repo lists 8 weight files with
correct sizes, and a ranged GET of the built resolve URL returns 206 after
following the redirect to the CDN. Backend 2853 passed; frontend 1143 JS +
91 Vue passed. The nine locales carry the refreshed download copy in the
next commit.
2026-09-14 07:42:51 +08:00
Will Miao b9bf006998 i18n: translate model-source strings into 9 locales
Complete the 15 [TODO: Translate] keys the model-source feature left behind
(modelCard.actions.viewOnSource, loras.contextMenu.linkModelSource,
modals.linkModelSource.*, modals.model.versions.sourceGroupInfo,
toast.contextMenu.enrichNeedsSource, toast.contextMenu.enrichUnsupportedSource),
and refresh the two enrichment labels that feature made stale.

- Brands stay Latin per R3: Hugging Face / ModelScope / TensorArt appear
  verbatim, and {source} is substituted by the caller at runtime, so no locale
  embeds a transliterated platform name. The placeholder-URL value
  (modals.linkModelSource.urlPlaceholder) stays byte-identical to en.json per
  the §6 URL exception.
- "model source" / "model page" / "model card" are new nouns and each locale
  gets exactly one rendering; "AI enrichment" reuses the noun already in each
  file from the previous enrichHfAgent copy. All of it is recorded in §2.
- modelCard.actions.viewOnSource follows each locale's existing
  viewOnHuggingFace pattern rather than the neighbouring viewOnCivitai one, so
  de/ru/he/ja/ko do not gain a third "View on ..." shape.
- loras.contextMenu.enrichHfAgent and loras.bulkOperations.enrichHfAgent read
  "AI HF metadata" in all nine locales. The feature invalidated that by also
  covering ModelScope, so both values drop the HF qualifier (the key names keep
  the historical Hf, and the guidelines now say so).
- Script conventions: fr keeps ASCII apostrophes and a space before ':' (the
  file is 351 ASCII vs 26 U+2019 and the modal being replaced was ASCII); ko
  keeps ASCII ':' and '()' (188 vs 6); CJK locales keep full-width punctuation;
  every ellipsis is ASCII '...'. Placeholders are verbatim per R2.
- modals.linkModelSource.enrichNote is phrased as a rule with the current
  exception in parentheses, so the guidelines call that out for whoever adds
  the next link-only source.

pytest tests/i18n: 20 passed, and scripts/sync_translation_keys.py --dry-run is
a no-op (no missing and no stale keys). Frontend: 1130 JS + 91 Vue passed.
Backend: 2815 passed. en.json is untouched by this commit.
2026-09-14 07:28:27 +08:00
Will Miao 5ab0e88abc feat(links): support ModelScope and TensorArt as model sources
A model file could only ever be linked to huggingface.co: `set_hf_url`
validated the URL with a huggingface-only regex, the agent fetched the card
from a hardcoded HF URL, and the readme processor built every relative image
path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the
same model-card convention (README.md + YAML frontmatter, often carrying
`base_model:` and `trigger_words:`) behind a public, key-less API, so the
enrichment pipeline could already serve it - it was the plumbing that was
HF-shaped, not the idea.

Make the external source a first-class, provider-driven concept:

- New `py/services/model_sources/` registry. A `ModelSource` owns URL
  recognition (lenient for stored values, strict for user input), the
  canonical page URL, model-card fetching, the asset base URL and the
  capability flags. `HuggingFaceSource` is the previous logic relocated;
  `ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md`
  and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is
  link-only on purpose: tensor.art answers plain HTTP clients with a
  Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud /
  cn.tensorart.net) rejects every /v1/model/* route with "invalid
  authorization header", so it declares supports_enrichment=False rather than
  failing silently later.
- Metadata gains `source_platform` + `source_url`; `hf_url` stays as a
  read/write alias, written only for Hugging Face, so existing sidecars,
  cached rows and third-party consumers keep working. Normalisation runs at
  the scanner, the persistent cache (both directions, plus two new columns
  behind an ALTER migration) and the linking handler - which is what stops a
  user who switches sources from leaving a stale `hf_url` on a ModelScope
  model.
- The agent pipeline keys off the provider instead of `hf_url`: the fast-fail
  gate now explains *why* a model is skipped (no source / unknown source /
  source without a reachable card), the prompt context exposes
  source_url/source_id/source_label/asset_base_url while still filling the
  legacy hf_url/repo aliases, and the four README image extractors take a
  base_url (defaulting to HF) so relative paths resolve against the right
  site. Version grouping generalises to hf: / ms: / ta: keys.
- `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but
  accepts `source_url`, validates against every provider and returns the
  platform. `GET /api/lm/model-sources` lets the UI render the supported-site
  list from the server.
- Frontend: a `modelSourceHelpers` mirror of the registry drives the link
  dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the
  version-group key and the enrichment gate; the versions tab no longer sends
  ms:/ta: keys to the CivitAI API.

TensorArt stays in the list because provenance is worth keeping even when the
card is unreadable - the dialog says so plainly ("Sites that don't expose one
(currently TensorArt) can only be linked") and the context menu disables
enrichment with a matching tooltip, instead of the user getting
"Unsupported URL".

Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a
1882-byte card whose frontmatter carries base_model/tags/trigger_words, and
relative images resolve to .../resolve/master/....

Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest
tests/i18n and a Jinja compile pass over templates/. The nine locales carry
[TODO: Translate] for the new strings, completed in the next commit.
2026-09-14 07:24:08 +08:00
Will Miao 84146b62fd feat(other-models): announce the feature only when folders are available
Other Models management is opt-in and its folders come from
folder_paths.get_folder_paths(). In plugin mode ComfyUI registers vae,
upscale_models, text_encoders, clip_vision and controlnet out of the box, so
enabling the feature works immediately. Standalone only knows the keys present
in settings.json.folder_paths, and that file is edited by hand - there is no UI
for those keys - so a standalone user who followed the announcement banner
reached "Enable Other Models" and then an empty page.

Gate the announcement on the capability instead of on how the process was
started:

- Config.get_other_models_availability() probes every canonical other key
  (legacy clip collapses into text_encoders where the host exposes
  map_legacy) and reports which sub_types resolve to a folder that exists on
  disk. It deliberately ignores enable_other_models: the question is "could
  this work here at all?". An empty folder counts, because CivitAI downloads
  can target it.
- /api/lm/settings exposes it as the derived, non-persisted
  other_models_paths_available flag; a probe failure yields null and the
  banner fails open.
- BannerService only registers the announcement when the flag is not false.
  `=== false` (not falsy) keeps a cached/older payload working, and nothing is
  written to dismissed_banners, so the banner can return once folders exist.
- The Other page grows an "enabled but nothing to scan" empty state driven by
  config.other_roots, showing the settings.json snippet for standalone and a
  pointer to ComfyUI model paths otherwise, plus an Open Settings action. It
  also covers the corner where only a non-default sub_type has a folder.

Translate the six other.noPaths.* keys into all nine locales and record the
new "folder key" / "on disk" terminology in the i18n guidelines.

Backend tests and pytest tests/i18n could not run in this environment (no
pytest/platformdirs); the probe was exercised against a stubbed folder_paths.
Frontend: 120 files / 1101 JS tests passed.
2026-09-13 21:47:16 +08:00
Will Miao 8a21837ca2 i18n: name all five sub_types in the Other Models opt-in copy
Three pre-enable strings listed exactly the old default set (VAE, upscaler,
text encoder, CLIP vision), so they read as "these are what enabling
manages" - now wrong twice over, since clip_vision became opt-in and
ControlNet was never named.

Point them at the capability instead: other.disabled.description and
banners.otherModels.content enumerate all five sub_types, and
settings.folderSettings.enableOtherModelsHelp names all five folder
categories the master switch gates. Model-type names stay in Latin per the
model-type rule; de compounds as CLIP-Vision- und ControlNet-Ordner and the
slash-list locales keep their existing VAE / Upscaler / Text Encoder / ...
casing. No placeholders or HTML are involved.

Editing en.json leaves the nine locales stale, and the sync script only adds
missing keys, so each locale is updated in the same pass by exact-literal
replacement of the one line - no JSON round-trip, no formatting churn (three
changed lines per file). Record the refreshed strings and the
"capability, not defaults" rule in the i18n guidelines.
2026-09-13 20:12:52 +08:00
Will Miao 3302147a43 fix(other-models): make clip_vision opt-in like controlnet
DEFAULT_ENABLED_OTHER_SUB_TYPES managed vae, upscaler, text_encoder and
clip_vision while controlnet was the sole opt-in type. That split was not
defensible on demand breadth: ControlNet is the broader category by install
base, and clip_vision is the narrower one (IPAdapter/SVD image conditioning,
usually one to three files) whose CivitAI type is retired upstream.

Keep the default set to the dependency-style assets every pipeline needs and
where "which one am I actually using" is the real problem - VAE, upscalers
and text encoders - and treat clip_vision and controlnet symmetrically as
opt-in. The feature is still unreleased, so the change needs no migration.

- Sync all five surfaces holding a default: DEFAULT_ENABLED_OTHER_SUB_TYPES,
  DEFAULT_SETTINGS, both DEFAULT_SETTINGS_BASE/createDefaultSettings lists,
  updateOtherModelsControls()'s fallback and the Jinja fallback.
- The selection is persisted per user, so only the untouched default moves;
  existing default_other_roots entries for a disabled sub_type are preserved.
- Fix the Jinja fallback using `or`, which treated an all-unchecked empty
  allow-list as "unset" and re-checked every box on render; `is none` keeps
  the empty list empty.
- Document the revised defaults and rationale in the plan.

Tests assert the new default trio, the normalize fallback, that both opt-in
types stay out of the default scan, and the auto-set iteration test now
enables clip_vision explicitly since it exercises the loop, not the default.
2026-09-13 20:12:49 +08:00
Will Miao 6fe0543d2e fix(other-models): default downloads to a flat path, not {base_model}/{first_tag}
get_download_path_template() fell back to "{base_model}/{first_tag}" for any
unconfigured model type, so other-model downloads were silently nested under an
arbitrary CivitAI tag even though the settings UI exposes no template row for
"other" and priority_tags has no "other" entry (making {first_tag} resolve to
tags[0]).

Add DEFAULT_DOWNLOAD_PATH_TEMPLATES with other -> "" so unconfigured and
unknown types resolve to a flat layout under the already sub_type-scoped
default_other_roots; explicit settings.json values still win. Mirror the flat
default in the frontend DEFAULT_PATH_TEMPLATES and stop the download/move
default-path previews from rendering "/undefined" or a dangling slash.
2026-09-13 11:30:57 +08:00
Will Miao f3b3393a16 i18n: translate Other Models feature strings into 9 locales
Complete the 36 keys left as [TODO: Translate] by the Other Models
feature (VAE / Upscaler / Text Encoder / CLIP Vision / ControlNet
management page and its opt-in toggles): settings.folderSettings.*,
other.*, initialization.other.*, toast.settings.otherRootsFailed and
banners.otherModels.*.

Model-type names (VAE, Upscaler, Text Encoder, CLIP Vision, ControlNet)
stay in Latin per the model-type rule, so the five subType* values are
intentionally identical to en.json; "Other Models" is a page/feature
name and is translated. Document the new terminology in the i18n
translation guidelines and note the completed i18n phase in the plan.
2026-09-13 08:08:57 +08:00
Will Miao 480a3f4ea5 docs: record Other Models opt-in toggles in the plan (Phase 3)
Documents the settings keys and defaults, the enabled/disabled behaviour
matrix, the backend and frontend touch points, cache consistency, the
discoverability surfaces (hidden nav + announcement banner + download CTA)
and the minimal settings.json.example policy.
2026-09-13 07:59:28 +08:00
Will Miao 1dbdf5b00c docs: mark Phase 2 implemented in other-models plan 2026-09-12 15:56:47 +08:00
Will Miao 57729375b6 docs: detail Phase 2 download design for Other Models page 2026-09-12 14:24:35 +08:00
Will Miao 3070838a42 docs: plan for Other Models page (VAE/upscaler/text encoder management) 2026-09-12 09:29:42 +08:00
Will Miao 3112869a21 docs(technical): record Windows case-fold fallback follow-up in reconcile
The Windows-only case-insensitive match in ModelScanner._reconcile_cache
is the only pass left unverified by the recent realpath cleanup: realpath
may already cover case differences on Windows, and if the branch is ever
reachable it is O(files x cache entries). Records the reachability
question, the verification steps for a Windows run, and the two possible
fixes.
2026-09-11 22:23:55 +08:00
Will Miao a17399d667 feat(recipes): delegate CivitAI-image re-import to companion browser extension
Recipes imported from CivitAI image URLs can contain 0 LoRAs: the backend
only sees the REST image API + EXIF, while the complete generation data
lives in the image page's internal trpc payload (see
docs/recipe-civitai-image-no-metadata.md). When the companion
lm-civitai-extension is installed with a valid license, re-import (single
and bulk) of CivitAI-image-sourced recipes is now delegated to the
extension via DOM CustomEvents; the extension scrapes the image page with
the user's session and calls back into the reimport endpoint with the
full metadata payload. Without the extension (or with an invalid license)
the native path runs unchanged.

- POST /api/lm/recipe/{id}/reimport accepts optional payload params
  (image_url/name/resources/gen_params/base_model/tags); the payload path
  reuses the import-remote engine with reimport semantics (user-edit
  carryover, delete-after-save), and malformed/failed payloads fall back
  to the legacy URL import. Response gains loras_count.
- The endpoint also accepts GET: the extension is GET-only by convention
  (documented in AGENTS.md).
- New static/js/utils/extensionReimportBridge.js (probeExtension /
  delegateReimport / getCivitaiImageInfo) wired into RecipeContextMenu
  and BulkManager with silent native fallback.
- i18n: toast.recipes.reimportingViaExtension added and translated in
  all 9 locales.
2026-09-06 20:26:14 +08:00
Will Miao f86b7b55d6 feat(recipes): remove deprecated Repair Metadata feature
The recipe "Repair Metadata" action has been marked Deprecated in the UI
for a while and cannot reliably recover recipes imported from CivitAI URLs
whose REST meta has no resources/hashes and whose image has no embedded
metadata (e.g. CivitAI-only generation data). Drop the feature end to end.

Backend:
- remove repair routes (repair, cancel-repair, recipe/{id}/repair,
  repair-bulk, repair-progress) and their handler mappings/methods
- remove RecipeScanner repair_all_recipes / repair_recipe_by_id /
  _repair_single_recipe and REPAIR_VERSION
- remove WebSocketManager recipe-repair progress channel
- drop repair_version column from the persistent recipe cache
- rematch mutual-exclusion now only checks rematch

Frontend:
- remove repair entries from per-recipe, bulk and global context menus
- remove repairRecipe / repairSelectedRecipes / repairRecipes + cancelRepair
  and the repairBulk API client method/endpoint
- drop recipe-repair i18n keys (synced across locales; doctor keys kept)

Tests/docs: delete test_recipe_repair.py, update scaffolding/routes/ws/
persistent-cache/integration tests and i18n guideline examples.
2026-09-05 16:56:20 +08:00
Will Miao bce7d1d30c docs(i18n): resolve R1 vs R8/§7 contradiction on proactive translation
R1 instructed agents to "translate the newly added keys in every locale"
right after syncing, while R8 and §7 make [TODO: Translate] placeholders
the sanctioned end state during feature development until the feature
owner explicitly asks for translations. Reword R1 and the AGENTS.md
Localization section to say stop after syncing and never translate
proactively.
2026-08-30 16:28:53 +08:00
Will Miao 838a374a56 feat(recipes): reconnect suggestions, undo, and base-model family tolerance
Enhance the deleted-LoRA reconnect flow in the recipe modal:

- Suggest local reconnect candidates when the panel opens, ranked by
  identity (same hash / same CivitAI version) then filename/name
  similarity, with a hard filter on confident base-model mismatches;
  the input gets a Combobox backed by the same endpoint as you type.
- Snapshot the pre-reconnect entry and offer a permanent restore:
  reconnected entries show an undo icon at the right end of the info
  row, with the original filename in the tooltip.
- Relax the manual reconnect base-model guard to a three-tier check:
  exact/unknown labels pass silently, same-architecture families
  (e.g. Pony <-> Illustrious) pass with a warning toast, and only
  cross-architecture mismatches stay hard-rejected.
2026-08-30 08:17:35 +08:00
Will Miao 2ff98ae089 docs(i18n): defer non-en translations until UI wording is final
[TODO: Translate] placeholders are now the sanctioned intermediate state
during feature development; translate all pending keys in one pass only
when the feature owner asks. R8 notes the exemption so placeholders are
not 'fixed' prematurely.
2026-08-29 16:41:53 +08:00
Will Miao ebe3df7d22 docs(i18n): mark guidelines as the post-sweep target state; translate de playlist title
§3/§5/§6 now describe the resolved state (regression watch-list instead of a
to-do list), §4 documents the single intentional placeholder deviation
(mappingsUpdated drops {plural} where '<noun>s' cannot be appended). de
help.updateVlogs.playlistTitle translated.
2026-08-29 12:49:38 +08:00
Will Miao 1a1be95a64 docs(i18n): add translation guidelines with per-locale term conventions
Audit of all 10 locale files found recipe/checkpoint mistranslations,
inverted ko tag logic, stale help texts, placeholder contract deviations,
and untranslated feature blocks. Document the conventions (R1-R9), per-
language term maps, confusion hot-spots, and the translation workflow so
future agents and translators follow the established decisions (e.g. keep
'Recipe' untranslated in French, use 配方 in Chinese).
2026-08-29 12:16:52 +08:00
Will Miao df34efafbc feat(recipes): skip rate-limited batch-import items and register download 429s (#1085)
Phase 2 of docs/plans/issue-1085-rate-limit-design.md:

- Batch import: items that fail due to vendor rate limiting are now
  SKIPPED with a "re-run the import later" hint instead of FAILED, so a
  transient 429 no longer pollutes failure accounting; the progress
  broadcast carries a rate_limited flag.
- Batch import UI: show a one-time "rate limited — slowing down" toast
  and swap the running status text while rate_limited; i18n keys synced
  to all locales.
- Downloader: download_file / download_to_memory / get_response_headers
  register 429 cooldowns with the RateLimitCoordinator, so subsequent
  API calls queue behind a download-triggered rate-limit window.
2026-08-27 10:08:32 +08:00
Will Miao c2a2048c8b feat(services): add per-destination rate-limit gate for API traffic (#1085)
Implement Phase 1 of docs/plans/issue-1085-rate-limit-design.md:

- New RateLimitCoordinator: per-host shared Retry-After gate with
  exponential backoff (30s base, 1800s cap), minimum inter-request pacing
  (default 0.75s), herd-free waiter serialization via per-destination
  locks, and a bounded wait (default 300s) that raises instead of parking.
- Downloader.make_request: connectivity-guard fail-fast first, then gate
  pacing; on 429 register the cooldown and wait-and-resend (bounded);
  errors that passed through the gate are marked gate_handled.
- FallbackMetadataProvider / MetadataSyncService: a network provider 429
  no longer fails over to other network providers (stops the CivArchive
  flood); sqlite stays as local last resort. Rate-limited lookups now
  report "Rate limited" instead of "Model not found", so transient 429s
  no longer mark models civitai_deleted.
- _RateLimitRetryHelper skips its own sleep for gate_handled errors,
  removing the double wait.
- New settings: rate_limit_gate_enabled, rate_limit_max_wait_seconds,
  rate_limit_min_interval_seconds.
2026-08-27 09:53:07 +08:00
Will Miao 1e1921cabb docs(plans): rate-limit abidance design for recipe ingest (#1085) 2026-08-27 09:02:42 +08:00
Will Miao cef4129fc9 fix(download): allow downloading additional files of an in-library model version (#1058) 2026-08-19 16:29:59 +08:00
Will Miao 97b9b1f62b feat(metadata): add CivitAI AutoV3 hash support across all storage layers
- Three-state autov3 field (not-checked / checked-unavailable / 12-hex value)
  in .metadata.json sidecars, in-memory ModelHashIndex, and SQLite
  (models.autov3 column + autov3_index table) with column-presence migration
- Background self-terminating backfill for legacy rows: per-model-type
  concurrency guard, executor-offloaded I/O, Civitai-first resolution
  (SHA256-matched version file) falling back to the embedded safetensors
  header hash
- Civitai-first propagation on metadata refresh, scan, and download paths;
  reject the empty-string SHA256 placeholder and strip OneTrainer 0x prefix
- List API hash filters and hash index lookups accept 12-char AutoV3
- Cap safetensors header reads at 64 MiB to prevent crafted-file allocation
- Prevent stale AutoV3 mappings on file replacement while preserving them on
  same-file re-registration (lazy-hash completion)
2026-08-08 14:30:34 +08:00
Will Miao 6f71335be4 feat(widgets): add Description tab to LoraInfoWidget with dual-mode rendering support
- Add Notes/Description tab switching with tab state persistence in widget value
- Lazy-load model description and version description from /lm/loras/metadata
- Render CivitAI HTML descriptions inline via v-html
- Auto-fetch description when LoRA selection changes while on Description tab
- Fix Vue mode height containment via contain:layout size (lm-vue-node class)
- Fix scroll wheel isolation: widget scroll vs canvas zoom in both render modes
- Add docs/comfyui-dual-mode-widgets.md with widget rendering patterns
2026-07-17 15:04:47 +08:00
Will Miao cf898da193 feat(agent): add LLM-powered metadata enrichment system with AgentCLI and PostProcessor
Introduce an agent skill framework for LLM-driven metadata enrichment:

- AgentCLI (py/agent_cli/): in-process wrappers around internal services
  using standard relative imports, eliminating the need for sys.path hacks
- LLMService: centralized BYOK (bring-your-own-key) LLM client supporting
  OpenAI, Ollama, and custom OpenAI-compatible endpoints
- PostProcessor: deterministic engine that applies LLM output via AgentCLI
  (replaces old handler.py + _BASE_MODEL_ALIASES approach)
- SkillRegistry: filesystem-based skill discovery (skill.yaml + prompt.md)
- AgentService: orchestrates skill execution with WebSocket progress
- Frontend AgentManager: WebSocket listeners, skill execution, config UI
- Context menu entries (single + bulk) for "Enrich Metadata (Agent)"
- Settings UI for AI Provider configuration (BYOK)
- Full i18n support across 9 locales

Bug fixes found during review:
- aiohttp.web.json_response: status_code= -> status=
- settings_modal cancelEditApiKey: wrong argument position
- AgentManager.isLlmConfigured: allow Ollama without API key
- PostProcessor._merge_tags: lowercase all tags to match TagUpdateService
2026-07-02 21:27:01 +08:00
Will Miao 88da754504 docs: migrate wiki-images to wiki repo, remove stale docs
Moved wiki-images to the wiki repo (willmiao/ComfyUI-Lora-Manager.wiki). Updated README.md image reference to use wiki raw URL. Removed docs/LM-Extension-Wiki.md (superseded by wiki pages).
2026-05-05 22:20:19 +08:00
Will Miao ef38bda04f docs: remove redundant example metadata files (#856)
- Delete examples/metadata/ directory and all example files
  - Real metadata.json files in model roots are better examples
  - Examples were artificial and could become outdated
  - Maintenance burden outweighs benefit

- Remove 'Complete Examples' section from docs/metadata-json-schema.md
- Remove reference to example files in 'See Also' section

Rationale:
Users have access to real-world metadata.json files in their actual
model directories, which contain complete Civitai API responses with
authentic data structures (images arrays with prompts, files with hashes,
creator information, etc.). These are more valuable than simplified
artificial examples.
2026-03-16 09:41:58 +08:00
Will Miao 8b91920058 docs: add comprehensive metadata.json schema documentation (#856)
- Create docs/metadata-json-schema.md with complete field reference
  - All base fields for LoRA, Checkpoint, and Embedding models
  - Complete civitai object structure with Used vs Stored field classification
  - Model-level fields (allowCommercialUse, allowDerivatives, etc.)
  - Creator fields (username, image)
  - customImages structure with actual field names and types
  - Field behavior categories (Auto-Updated, Set Once, User-Editable)

- Add .specs/metadata.schema.json for programmatic validation
  - JSON Schema draft-07 format
  - oneOf schemas for each model type
  - Definitions for civitaiObject and usageTips

- Add example metadata files for each model type
  - lora-civitai.json: LoRA with full Civitai data
  - lora-custom.json: User-defined LoRA with trigger words
  - lora-no-triggerwords.json: LoRA without trigger words
  - checkpoint-civitai.json: Checkpoint from Civitai
  - embedding-custom.json: Custom embedding

Key clarifications:
  - modified: Import timestamp (Set Once, never changes after import)
  - size: File size at import time (Set Once)
  - base_model: Optional with actual values (SDXL 1.0, Flux.1 D, etc.)
  - model_type: Used in metadata.json (not sub_type which is internal)
  - allowCommercialUse: ["Image", "Video", "RentCivit", "Rent"]
  - civitai.files/images: Marked as Used by Lora Manager
  - User-editable fields clearly documented (model_name, tags, etc.)
2026-03-16 09:41:58 +08:00
Will Miao a221682a0d refactor(settings): implement macOS Settings style for settings modal
- Reorganize settings into 4 sections: General, Interface, Download, Advanced
- Implement section switching instead of scrolling (macOS Settings style)
- Remove collapsible/expandable sections and redundant 'SETTINGS' label
- Add accent-colored underline for section headers
- Update navigation with larger, more prominent active state
- Add fade-in animation for section transitions
- Update search to auto-switch to matching section
- Refactor CSS: 800x600 fixed modal size, remove collapse styles
- Refactor JS: simplify navigation logic, remove scroll spy and collapse code

Refs: Phase 0 settings modal optimization
2026-02-24 07:19:32 +08:00
Will Miao 3f0227ba9d feat(settings): add search functionality to settings modal (P2)
Implement Phase 2 search bar feature for settings modal:

- Add search input to settings modal header with icon and clear button
- Implement real-time filtering with 150ms debounce for performance
- Add visual highlighting for matched search terms using accent color
- Implement empty search results state with user-friendly message
- Add keyboard shortcuts (Escape to clear search)
- Auto-expand sections containing matching content during search
- Fix header layout to prevent overlap with close button
- Update progress tracker documenting P2 completion
- Add translation keys for search feature (placeholder, clear, no results)
- Sync translations across all language files

Files changed:
- templates/components/modals/settings_modal.html
- static/css/components/modal/settings-modal.css
- static/js/managers/SettingsManager.js
- locales/*.json (10 language files)
- docs/ui-ux-optimization/progress-tracker.md
2026-02-24 06:36:49 +08:00
Will Miao 528225ffbd feat(settings): add left navigation sidebar to settings modal
Implement two-column layout for improved settings navigation:
- Add 200px fixed navigation sidebar with 4 groups (General, Interface, Download, Advanced)
- Implement scroll spy to highlight current section during scroll
- Add smooth scrolling when clicking navigation items
- Extend modal width from 700px to 950px for better content display
- Add responsive mobile layout (switches to stacked view below 768px)
- Add i18n keys for navigation group titles
- Create documentation for optimization phases and progress tracking

Files changed:
- settings-modal.css: Add sidebar, navigation, and responsive styles
- settings_modal.html: Restructure with two-column layout and section IDs
- SettingsManager.js: Add initializeNavigation() with scroll spy
- locales/*.json: Add settings.nav translations (en, zh-CN, zh-TW, ja, ru, de, fr, es, ko, he)
- docs/ui-ux-optimization/: Add proposal and progress tracker documentation
2026-02-23 21:12:15 +08:00
Will Miao 1f5baec7fd docs: add recipe batch import feature requirements document 2026-02-23 17:07:03 +08:00
Will Miao 85e511d81c feat(testing): implement Phase 4 advanced testing
- Add Hypothesis property-based tests (19 tests)
- Add Syrupy snapshot tests (7 tests)
- Add pytest-benchmark performance tests (11 tests)
- Fix Hypothesis plugin compatibility by creating MockModule class
- Update pytest.ini to exclude .hypothesis directory
- Add .hypothesis/ to .gitignore
- Update requirements-dev.txt with testing dependencies
- Mark Phase 4 complete in backend-testing-improvement-plan.md

All 947 tests passing.
2026-02-11 11:58:28 +08:00
Will Miao 8e30008b29 test: complete Phase 3 of backend testing improvement plan
Centralize test fixtures:
- Add mock_downloader fixture for configurable downloader mocking
- Add mock_websocket_manager fixture for WebSocket broadcast recording
- Add reset_singletons autouse fixture for test isolation
- Consolidate singleton cleanup in conftest.py

Split large test files:
- test_download_manager.py (1422 lines) → 3 focused files:
  - test_download_manager_basic.py: 12 core functionality tests
  - test_download_manager_error.py: 15 error handling tests
  - test_download_manager_concurrent.py: 6 advanced scenario tests

- test_cache_paths.py (530 lines) → 3 focused files:
  - test_cache_paths_resolution.py: 11 path resolution tests
  - test_cache_paths_validation.py: 9 legacy validation tests
  - test_cache_paths_migration.py: 9 migration scenario tests

Update documentation:
- Mark all Phase 3 checklist items as complete
- Add Phase 3 completion summary with test results

All 894 tests passing.
2026-02-11 11:10:31 +08:00
Will Miao e335a527d4 test: Complete Phase 2 - Integration & Coverage improvements
- Create tests/integration/ directory with conftest.py fixtures
- Add 7 download flow integration tests (test_download_flow.py)
- Add 9 recipe flow integration tests (test_recipe_flow.py)
- Add 12 ModelLifecycleService tests (exclude_model, bulk_delete, error paths)
- Add 5 PersistentRecipeCache concurrent access tests
- Update backend-testing-improvement-plan.md with Phase 2 completion

Total: 28 new tests, all passing (51/51)
2026-02-11 10:55:19 +08:00
Will Miao 25e6d72c4f test(backend): Phase 1 - Improve testing infrastructure and add error path tests
## Changes

### pytest-asyncio Integration
- Add pytest-asyncio>=0.21.0 to requirements-dev.txt
- Update pytest.ini with asyncio_mode=auto and fixture loop scope
- Remove custom pytest_pyfunc_call handler from conftest.py
- Add @pytest.mark.asyncio to 21 async test functions

### Error Path Tests
- Create test_downloader_error_paths.py with 19 new tests covering:
  - DownloadStreamControl state management (6 tests)
  - Downloader configuration and initialization (4 tests)
  - DownloadProgress dataclass validation (1 test)
  - Custom exception handling (2 tests)
  - Authentication header generation (3 tests)
  - Session management (3 tests)

### Documentation
- Update backend-testing-improvement-plan.md with Phase 1 completion status

## Test Results
- All 458 service tests pass
- No regressions introduced

Relates to backend testing improvement plan Phase 1
2026-02-11 10:29:21 +08:00
Will Miao e2f1520e7f docs: update LM-Extension-Wiki with CivArchive support and v0.4.8 features
- Reframe supporter access section to emphasize sustainability and gratitude
- Add CivArchive support announcement and image
- Document new dedicated download button and hide models feature in v0.4.8
- Improve readability and flow of the overview and supporter sections
2026-02-07 23:27:01 +08:00
Will Miao 84c62f2954 refactor(model-type): complete phase 5 cleanup by removing deprecated model_type field
- Remove backward compatibility code for `model_type` in `ModelScanner._build_cache_entry()`
- Update `CheckpointScanner` to only handle `sub_type` in `adjust_metadata()` and `adjust_cached_entry()`
- Delete deprecated aliases `resolve_civitai_model_type` and `normalize_civitai_model_type` from `model_query.py`
- Update frontend components (`RecipeModal.js`, `ModelCard.js`, etc.) to use `sub_type` instead of `model_type`
- Update API response format to return only `sub_type`, removing `model_type` from service responses
- Revise technical documentation to mark Phase 5 as completed and remove outdated TODO items

All cleanup tasks for the model type refactoring are now complete, ensuring consistent use of `sub_type` across the codebase.
2026-01-30 07:48:31 +08:00
Will Miao 5e91073476 refactor: unify model_type semantics by introducing sub_type field
This commit resolves the semantic confusion around the model_type field by
clearly distinguishing between:
- scanner_type: architecture-level (lora/checkpoint/embedding)
- sub_type: business-level subtype (lora/locon/dora/checkpoint/diffusion_model/embedding)

Backend Changes:
- Rename model_type to sub_type in CheckpointMetadata and EmbeddingMetadata
- Add resolve_sub_type() and normalize_sub_type() in model_query.py
- Update checkpoint_scanner to use _resolve_sub_type()
- Update service format_response to include both sub_type and model_type
- Add VALID_*_SUB_TYPES constants with backward compatible aliases

Frontend Changes:
- Add MODEL_SUBTYPE_DISPLAY_NAMES constants
- Keep MODEL_TYPE_DISPLAY_NAMES as backward compatible alias

Testing:
- Add 43 new tests covering sub_type resolution and API response

Documentation:
- Add refactoring todo document to docs/technical/

BREAKING CHANGE: None - full backward compatibility maintained
2026-01-30 06:56:10 +08:00
Will Miao 822ac046e0 docs: update DOM widget value persistence best practices guide
- Restructure document to clearly separate simple vs complex widget patterns
- Add detailed explanation of ComfyUI's built-in callback mechanism
- Provide complete implementation examples for both patterns
- Remove outdated sync chain diagrams and replace with practical guidance
- Emphasize using DOM element as source of truth for simple widgets
- Document proper use of internal state with widget.callback for complex widgets
2026-01-27 22:51:09 +08:00
Will Miao 9032226724 fix(autocomplete): fix value persistence in DOM text widgets
Remove multiple sources of truth and async sync chains that caused
values to be lost during load/switch workflow or reload page.

Changes:
- Remove internalValue state variable from main.ts
- Update getValue/setValue to read/write DOM directly via widget.inputEl
- Remove textValue reactive ref and v-model from Vue component
- Remove serializeValue, onSetValue, and watch callbacks
- Register textarea reference on mount, clean up on unmount
- Simplify AutocompleteTextWidgetInterface

Follows ComfyUI built-in addMultilineWidget pattern:
- Single source of truth (DOM element value only)
- Direct sync (no intermediate variables or async chains)

Also adds documentation:
- docs/dom-widgets/value-persistence-best-practices.md
- docs/dom-widgets/README.md
- Update docs/dom_widget_dev_guide.md with reference
2026-01-26 23:24:16 +08:00
Will Miao 42f35be9d3 feat(autocomplete): add Danbooru/e621 tag search with category filtering
- Add TagFTSIndex service for fast SQLite FTS5-based tag search (221k+ tags)
- Implement command-mode autocomplete: /char, /artist, /general, /meta, etc.
- Support category filtering via category IDs or names
- Return enriched results with post counts and category badges
- Add UI styling for category badges and command list dropdown
2026-01-26 13:51:45 +08:00
Will Miao 88e7f671d2 fix(autocomplete): resolve instability in Vue DOM mode and fix WanVideo node binding
- Fix infinite reinitialization loop by only validating stale widget.inputEl when it's actually in DOM
- Improve findWidgetInputElement to specifically search for textarea for text widgets, avoiding mismatches with checkbox inputs on nodes like WanVideo Lora Select that have toggle switches
- Add data-node-id based element search as primary strategy for better reliability across rendering modes
- Fix autocomplete initialization to properly handle element DOM state transitions

Fixes autocomplete failing after Canvas ↔ Vue DOM mode switches and WanVideo node always failing to trigger autocomplete.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2026-01-17 14:19:20 +08:00
Will Miao cde6151c71 fix: make sliders compatible with Vue DOM render mode
Add data-capture-wheel attribute to SingleSlider and DualRangeSlider
components to prevent wheel events from being intercepted by the canvas
in ComfyUI's new Vue DOM render mode. This allows mouse wheel to work
for adjusting slider values while still enabling workflow zoom on
non-interactive widget areas.

Also update event handling to use pointer events with proper stop
propagation and pointer capture for reliable drag operations in both
rendering modes.

Update development guide with Section 8 documenting Vue DOM render mode
event handling patterns and best practices.
2026-01-15 07:03:05 +08:00
Will Miao 688baef2f0 feat(dom-widgets): forward middle mouse events to canvas for panning
Add `forwardMiddleMouseToCanvas` utility to forward middle mouse button events from DOM widgets to the ComfyUI canvas, enabling workflow panning when the cursor is over a widget. The function is implemented in `vue-widgets/src/main.ts` and documented in the developer guide. Additionally, fix `getPoolConfigFromConnectedNode` to return null for inactive pool nodes.
2026-01-13 11:45:12 +08:00
Will Miao 6a17e75782 docs: add frontend UI architecture and ComfyUI widget guidelines
- Document dual UI systems: standalone web UI and ComfyUI custom node widgets
- Add ComfyUI widget development guidelines including styling and constraints
- Update terminology in LoraRandomizerNode from 'frontend/backend' to 'fixed/always' for clarity
- Include UI constraints for ComfyUI widgets: minimize vertical space, avoid dynamic height changes, keep UI simple
2026-01-13 11:20:50 +08:00
Will Miao 9719dd4d07 docs(dom_widget_dev_guide): clarify dynamic resizing and add performance note
- Add performance note explaining that providing `getMinHeight` and `getHeight` via `options` avoids expensive DOM measurements
- Expand dynamic resizing section with detailed update sequence and common scenarios table
- Update LoraPoolSummaryView.vue with `min-height: 0` to allow flex shrinking
- Update main.ts to provide `getMinHeight` via options and adjust `computeLayoutSize` for performance
2026-01-12 09:22:18 +08:00