A model file could only ever be linked to huggingface.co: `set_hf_url`
validated the URL with a huggingface-only regex, the agent fetched the card
from a hardcoded HF URL, and the readme processor built every relative image
path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the
same model-card convention (README.md + YAML frontmatter, often carrying
`base_model:` and `trigger_words:`) behind a public, key-less API, so the
enrichment pipeline could already serve it - it was the plumbing that was
HF-shaped, not the idea.
Make the external source a first-class, provider-driven concept:
- New `py/services/model_sources/` registry. A `ModelSource` owns URL
recognition (lenient for stored values, strict for user input), the
canonical page URL, model-card fetching, the asset base URL and the
capability flags. `HuggingFaceSource` is the previous logic relocated;
`ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md`
and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is
link-only on purpose: tensor.art answers plain HTTP clients with a
Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud /
cn.tensorart.net) rejects every /v1/model/* route with "invalid
authorization header", so it declares supports_enrichment=False rather than
failing silently later.
- Metadata gains `source_platform` + `source_url`; `hf_url` stays as a
read/write alias, written only for Hugging Face, so existing sidecars,
cached rows and third-party consumers keep working. Normalisation runs at
the scanner, the persistent cache (both directions, plus two new columns
behind an ALTER migration) and the linking handler - which is what stops a
user who switches sources from leaving a stale `hf_url` on a ModelScope
model.
- The agent pipeline keys off the provider instead of `hf_url`: the fast-fail
gate now explains *why* a model is skipped (no source / unknown source /
source without a reachable card), the prompt context exposes
source_url/source_id/source_label/asset_base_url while still filling the
legacy hf_url/repo aliases, and the four README image extractors take a
base_url (defaulting to HF) so relative paths resolve against the right
site. Version grouping generalises to hf: / ms: / ta: keys.
- `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but
accepts `source_url`, validates against every provider and returns the
platform. `GET /api/lm/model-sources` lets the UI render the supported-site
list from the server.
- Frontend: a `modelSourceHelpers` mirror of the registry drives the link
dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the
version-group key and the enrichment gate; the versions tab no longer sends
ms:/ta: keys to the CivitAI API.
TensorArt stays in the list because provenance is worth keeping even when the
card is unreadable - the dialog says so plainly ("Sites that don't expose one
(currently TensorArt) can only be linked") and the context menu disables
enrichment with a matching tooltip, instead of the user getting
"Unsupported URL".
Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a
1882-byte card whose frontmatter carries base_model/tags/trigger_words, and
relative images resolve to .../resolve/master/....
Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest
tests/i18n and a Jinja compile pass over templates/. The nine locales carry
[TODO: Translate] for the new strings, completed in the next commit.
A model could have CivitAI metadata and a HuggingFace link at the same time,
but only one of the two "View on ..." entries ever rendered, because both the
model modal and the card globe asked the `from_civitai` provenance flag which
source to show. `set_hf_url` wrote `false` and a CivitAI refresh wrote `true`,
so whichever ran last erased the other: linking HF hid "View on CivitAI" even
though the civitai payload was still in the sidecar, and (on the card) a later
refresh pointed the single globe icon back at CivitAI, hiding the HF entry.
Decide the links from the data itself instead:
- `set_hf_url` no longer touches `from_civitai`; it records where the metadata
came from, and HF provenance is already tracked by `hf_url`.
- Add `hasCivitaiSource(civitai)` in the shared card/modal utils and gate the
modal's CivitAI link, the card globe (title, enabled state, click target,
new `data-has_civitai`) and the context-menu `civitai` action on actual
CivitAI data (`modelId` / `model_id` / `id`). A dual-source model now shows
both links, and a CivitAI-only model with no `hf_url` stays as before.
- Agent HF enrichment (`PostProcessor.is_hf_model`) keyed off
`not from_civitai`, which stopped being a synonym for "has an HF source" once
both sources can coexist (and already broke after a CivitAI refresh flipped
the flag back to true). Key it off `hf_url` directly; the post-processor
tests move to that discriminator and gain a dual-source case.
Regression tests: the set-hf-url handler preserves civitai + `from_civitai`
and no longer forces the flag false, the modal renders both links (including
with `from_civitai: false`), and the card globe targets/opens the right source
and is disabled when neither is available.
Backend: 2749 passed. Frontend: 1098 JS + 91 Vue tests passed.
The option only existed in the LoRA page menu. Move updateEnrichMenuItem
and enrichWithAgent into ModelContextMenuMixin so both pages share the
implementation, and add the menu item to the checkpoints template.
Split the single-model and bulk context menu actions into 'Download
Missing Example Images' (regular endpoint, skips already-processed
models) and 'Re-process Example Images' (force endpoint, retries
failed models).
- start_download accepts model_hashes so a selected subset can be
processed with the progress-aware skip logic; explicitly targeted
models bypass the failed/processed model-level guards so per-image
gaps are filled
- pre-download existence check in the processor skips network requests
for image files already on disk across all download paths
- force download retries previously failed models and clears their
failed status on success
- add i18n keys for the new menu items across all locales
- Merge Relink to Civitai and new Link to HuggingFace into a single
'Link Model' submenu with sub-options for each source
- Add POST /api/lm/set-hf-url endpoint to associate a model with a
HuggingFace repo URL, saving hf_url to .metadata.json
- Add link_hf_modal.html for URL input, following relink-civitai pattern
- Use update_single_model_cache instead of add_model_to_cache to
prevent duplicate cache entries after linking
- Remove os.path.realpath usage for consistency with relink-civitai
- Raise errors instead of silently falling back to LoRA scanner when
model root cannot be determined
- Scope .input-group CSS rules to modal IDs to fix style conflicts
with download-modal.css
- Add i18n keys across all 10 locales with translations for
zh-CN, zh-TW, ja, ko, de, es, fr, he, ru
- Add `cardPath` parameter to `show` method in NsfwLevelSelector component
- Include `filePath` from card dataset when calling selector in ModelContextMenuMixin
- Clear `cardPath` from dataset when hiding selector to prevent stale data
This enables the NSFW level selector to access the file path context, which may be needed for backend operations when changing NSFW levels.
Add new POST endpoint `/api/lm/example-images/set-nsfw-level` to allow updating NSFW classification for individual example images. The endpoint supports both regular and custom images, validates required parameters, and updates the corresponding model metadata. This enables users to manually adjust NSFW ratings for better content filtering.
Add initialization tracking to prevent multiple event listener attachments
in context menu components. Use dataset.initialized flag to ensure NSFW
selector events are only set up once per component instance.
In ModelContextMenuMixin, replace DOM elements and reattach event listeners
to avoid duplicates when components are reinitialized. This fixes issues
where multiple click handlers could be attached to the same elements.
- Updated all relevant routes in `stats_routes.py` and `update_routes.py` to include the new '/api/lm/' prefix for consistency.
- Modified API endpoint configurations in `apiConfig.js` to reflect the new structure, ensuring all CRUD and bulk operations are correctly routed.
- Adjusted fetch calls in various components and managers to utilize the updated API paths, including recipe, model, and example image operations.
- Ensured all instances of the old API paths were replaced with the new '/api/lm/' prefix across the codebase for uniformity and to prevent broken links.
- Introduced ModelContextMenuMixin to encapsulate shared methods for Lora and Checkpoint context menus.
- Updated CheckpointContextMenu to utilize the mixin for common actions and NSFW level handling.
- Simplified LoraContextMenu by integrating the mixin, removing redundant methods.
- Removed duplicated NSFW handling logic and centralized it in the mixin.
- Adjusted import/export statements to reflect the new structure and ensure proper functionality.
Will Miao