feat(links): support ModelScope and TensorArt as model sources

A model file could only ever be linked to huggingface.co: `set_hf_url`
validated the URL with a huggingface-only regex, the agent fetched the card
from a hardcoded HF URL, and the readme processor built every relative image
path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the
same model-card convention (README.md + YAML frontmatter, often carrying
`base_model:` and `trigger_words:`) behind a public, key-less API, so the
enrichment pipeline could already serve it - it was the plumbing that was
HF-shaped, not the idea.

Make the external source a first-class, provider-driven concept:

- New `py/services/model_sources/` registry. A `ModelSource` owns URL
  recognition (lenient for stored values, strict for user input), the
  canonical page URL, model-card fetching, the asset base URL and the
  capability flags. `HuggingFaceSource` is the previous logic relocated;
  `ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md`
  and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is
  link-only on purpose: tensor.art answers plain HTTP clients with a
  Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud /
  cn.tensorart.net) rejects every /v1/model/* route with "invalid
  authorization header", so it declares supports_enrichment=False rather than
  failing silently later.
- Metadata gains `source_platform` + `source_url`; `hf_url` stays as a
  read/write alias, written only for Hugging Face, so existing sidecars,
  cached rows and third-party consumers keep working. Normalisation runs at
  the scanner, the persistent cache (both directions, plus two new columns
  behind an ALTER migration) and the linking handler - which is what stops a
  user who switches sources from leaving a stale `hf_url` on a ModelScope
  model.
- The agent pipeline keys off the provider instead of `hf_url`: the fast-fail
  gate now explains *why* a model is skipped (no source / unknown source /
  source without a reachable card), the prompt context exposes
  source_url/source_id/source_label/asset_base_url while still filling the
  legacy hf_url/repo aliases, and the four README image extractors take a
  base_url (defaulting to HF) so relative paths resolve against the right
  site. Version grouping generalises to hf: / ms: / ta: keys.
- `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but
  accepts `source_url`, validates against every provider and returns the
  platform. `GET /api/lm/model-sources` lets the UI render the supported-site
  list from the server.
- Frontend: a `modelSourceHelpers` mirror of the registry drives the link
  dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the
  version-group key and the enrichment gate; the versions tab no longer sends
  ms:/ta: keys to the CivitAI API.

TensorArt stays in the list because provenance is worth keeping even when the
card is unreadable - the dialog says so plainly ("Sites that don't expose one
(currently TensorArt) can only be linked") and the context menu disables
enrichment with a matching tooltip, instead of the user getting
"Unsupported URL".

Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a
1882-byte card whose frontmatter carries base_model/tags/trigger_words, and
relative images resolve to .../resolve/master/....

Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest
tests/i18n and a Jinja compile pass over templates/. The nine locales carry
[TODO: Translate] for the new strings, completed in the next commit.
This commit is contained in:
Will Miao
2026-09-14 07:24:08 +08:00
parent 84146b62fd
commit 5ab0e88abc
51 changed files with 2518 additions and 279 deletions
+18 -11
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "View on CivitAI",
"notAvailableFromCivitai": "Not available from CivitAI",
"viewOnHuggingFace": "View on Hugging Face",
"viewOnSource": "View on {source}",
"sendToWorkflow": "Send to ComfyUI (Click: Append, Shift+Click: Replace)",
"copyLoRASyntax": "Copy LoRA Syntax",
"checkpointNameCopied": "Checkpoint name copied",
@@ -867,14 +868,14 @@
"complete": "Auto-organize complete",
"error": "Error: {error}"
},
"enrichHfAgent": "Enrich HF Metadata (AI)"
"enrichHfAgent": "Enrich Metadata with AI"
},
"contextMenu": {
"refreshMetadata": "Refresh CivitAI Data",
"checkUpdates": "Check Updates",
"linkModel": "Link Model",
"linkCivitai": "Link to CivitAI",
"linkHuggingFace": "Link to HuggingFace",
"linkModelSource": "Link to Model Source",
"copySyntax": "Copy LoRA Syntax",
"copyFilename": "Copy Model Filename",
"copyRecipeSyntax": "Copy Recipe Syntax",
@@ -896,7 +897,7 @@
"viewAllLoras": "View All LoRAs",
"downloadMissingLoras": "Download Missing LoRAs",
"deleteRecipe": "Delete Recipe",
"enrichHfAgent": "Enrich HF Metadata (AI)"
"enrichHfAgent": "Enrich Metadata with AI"
}
},
"recipes": {
@@ -1596,12 +1597,16 @@
"pathPlaceholder": "Type folder path or select from tree below...",
"root": "Root"
},
"linkHuggingFace": {
"title": "Link to HuggingFace",
"infoText": "Paste the HuggingFace repository URL to associate this model with its source. This enables AI-powered metadata enrichment.",
"urlLabel": "HuggingFace Repository URL:",
"linkModelSource": {
"title": "Link to Model Source",
"infoText": "Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "Model Page URL:",
"urlPlaceholder": "https://huggingface.co/user/repo",
"helpText": "Enter the full URL of the HuggingFace repository.",
"helpText": "Enter the full URL of the model page. Supported sites:",
"enrichNote": "AI enrichment needs a readable model card. Sites that don't expose one (currently TensorArt) can only be linked.",
"urlRequired": "Please enter a model page URL.",
"invalidUrl": "Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "Linking model source...",
"confirmAction": "Save & Link"
},
"relinkCivitai": {
@@ -1847,7 +1852,7 @@
"empty": "No version history available for this model yet.",
"error": "Failed to load versions.",
"missingModelId": "This model is missing a CivitAI model id.",
"hfGroupInfo": "This is a HuggingFace model group. Open the library to see all versions in the grid.",
"sourceGroupInfo": "This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": {
"delete": "Delete this version from your library?"
},
@@ -2478,12 +2483,14 @@
"contentRatingFailed": "Failed to set content rating: {message}",
"relinkSuccess": "Model successfully re-linked to CivitAI",
"relinkFailed": "Error: {message}",
"linkHfSuccess": "Model successfully linked to HuggingFace",
"linkHfSuccess": "Model successfully linked to its model source",
"linkHfFailed": "Error: {message}",
"linkCivArchSuccess": "Model successfully re-linked via CivitArchive",
"fetchMetadataFirst": "Please fetch metadata from CivitAI first",
"noCivitaiInfo": "No CivitAI information available",
"missingHash": "Model hash not available"
"missingHash": "Model hash not available",
"enrichNeedsSource": "Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "AI enrichment is not available for {source} models"
},
"exampleImages": {
"pathUpdated": "Example images path updated successfully",