feat(links): support ModelScope and TensorArt as model sources

A model file could only ever be linked to huggingface.co: `set_hf_url`
validated the URL with a huggingface-only regex, the agent fetched the card
from a hardcoded HF URL, and the readme processor built every relative image
path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the
same model-card convention (README.md + YAML frontmatter, often carrying
`base_model:` and `trigger_words:`) behind a public, key-less API, so the
enrichment pipeline could already serve it - it was the plumbing that was
HF-shaped, not the idea.

Make the external source a first-class, provider-driven concept:

- New `py/services/model_sources/` registry. A `ModelSource` owns URL
  recognition (lenient for stored values, strict for user input), the
  canonical page URL, model-card fetching, the asset base URL and the
  capability flags. `HuggingFaceSource` is the previous logic relocated;
  `ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md`
  and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is
  link-only on purpose: tensor.art answers plain HTTP clients with a
  Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud /
  cn.tensorart.net) rejects every /v1/model/* route with "invalid
  authorization header", so it declares supports_enrichment=False rather than
  failing silently later.
- Metadata gains `source_platform` + `source_url`; `hf_url` stays as a
  read/write alias, written only for Hugging Face, so existing sidecars,
  cached rows and third-party consumers keep working. Normalisation runs at
  the scanner, the persistent cache (both directions, plus two new columns
  behind an ALTER migration) and the linking handler - which is what stops a
  user who switches sources from leaving a stale `hf_url` on a ModelScope
  model.
- The agent pipeline keys off the provider instead of `hf_url`: the fast-fail
  gate now explains *why* a model is skipped (no source / unknown source /
  source without a reachable card), the prompt context exposes
  source_url/source_id/source_label/asset_base_url while still filling the
  legacy hf_url/repo aliases, and the four README image extractors take a
  base_url (defaulting to HF) so relative paths resolve against the right
  site. Version grouping generalises to hf: / ms: / ta: keys.
- `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but
  accepts `source_url`, validates against every provider and returns the
  platform. `GET /api/lm/model-sources` lets the UI render the supported-site
  list from the server.
- Frontend: a `modelSourceHelpers` mirror of the registry drives the link
  dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the
  version-group key and the enrichment gate; the versions tab no longer sends
  ms:/ta: keys to the CivitAI API.

TensorArt stays in the list because provenance is worth keeping even when the
card is unreadable - the dialog says so plainly ("Sites that don't expose one
(currently TensorArt) can only be linked") and the context menu disables
enrichment with a matching tooltip, instead of the user getting
"Unsupported URL".

Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a
1882-byte card whose frontmatter carries base_model/tags/trigger_words, and
relative images resolve to .../resolve/master/....

Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest
tests/i18n and a Jinja compile pass over templates/. The nine locales carry
[TODO: Translate] for the new strings, completed in the next commit.

py/services/agent/agent_service.py

@@ -19,16 +19,18 @@ from __future__ import annotations
 import asyncio
 import json
 import logging
+import os
 import re
 from dataclasses import dataclass, field
 from typing import Any, Dict, List, Optional
 
-import aiohttp
-
-import os
-
 from ...config import config
 from ..llm_service import LLMService
+from ..model_sources import (
+    get_source,
+    resolve_source_ref,
+    source_label,
+)
 from ..websocket_manager import ws_manager
 from .post_processor import PostProcessor
 from .skill_registry import SkillRegistry
@@ -267,14 +269,17 @@ class AgentService:
                 from ...metadata_ops import read_metadata
                 metadata = await read_metadata(model_path)
 
-                # Fast-fail: enrich_hf_metadata requires hf_url to have HF README context
-                if skill_name == "enrich_hf_metadata" and not metadata.get("hf_url", ""):
-                    logger.info(
-                        "[%s] SKIP %s — no hf_url in metadata",
-                        skill_name, model_filename,
-                    )
-                    skipped_count += 1
-                    skip_model = True
+                # Fast-fail: enrich_hf_metadata needs an external model source
+                # that exposes an accessible model card.
+                if skill_name == "enrich_hf_metadata":
+                    skip_reason = self._enrichment_skip_reason(metadata)
+                    if skip_reason:
+                        logger.info(
+                            "[%s] SKIP %s — %s",
+                            skill_name, model_filename, skip_reason,
+                        )
+                        skipped_count += 1
+                        skip_model = True
 
                 if not skip_model:
                     prompt_vars: Dict[str, Any] = {"model_path": model_path}
@@ -358,6 +363,28 @@ class AgentService:
     # Base model grouping (keeps the prompt compact)
     # ------------------------------------------------------------------
 
+    @staticmethod
+    def _enrichment_skip_reason(metadata: Dict[str, Any]) -> str:
+        """Return why ``enrich_hf_metadata`` cannot run, or ``""`` if it can.
+
+        Distinguishes the three cases the user can act on: no source linked,
+        a source we don't know, and a known source whose model card is not
+        reachable from the backend (TensorArt).
+        """
+
+        ref = resolve_source_ref(metadata)
+        if ref is None:
+            return "no model source linked (source_url missing)"
+        source = get_source(ref.platform)
+        if source is None:
+            return f"unsupported model source platform '{ref.platform}'"
+        if not source.supports_enrichment:
+            return (
+                f"{source.label} does not expose a model card to the backend; "
+                "AI metadata enrichment is not available for this source"
+            )
+        return ""
+
     @staticmethod
     def _format_base_models(models: List[str]) -> str:
         """Format the base model list as a flat, one-per-line list.
@@ -388,6 +415,14 @@ class AgentService:
         context: Dict[str, Any] = {
             "model_path": model_path,
             "model_basename": "",
+            # Canonical external-source variables
+            "source_url": "",
+            "source_id": "",
+            "source_platform": "",
+            "source_label": "",
+            "asset_base_url": "",
+            # Legacy Hugging Face aliases (kept so older prompt templates and
+            # third-party skills keep rendering)
             "hf_url": "",
             "repo": "",
             "readme_content": "",
@@ -411,12 +446,20 @@ class AgentService:
             "size": metadata.get("size", 0),
         }
 
-        hf_url = metadata.get("hf_url", "")
-        context["hf_url"] = hf_url
-        repo = self._extract_repo_from_url(hf_url) if hf_url else ""
-        context["repo"] = repo or ""
-        if repo:
-            readme = await self._fetch_readme(repo)
+        ref = resolve_source_ref(metadata)
+        if ref is not None:
+            context["source_url"] = ref.url
+            context["source_id"] = ref.source_id
+            context["source_platform"] = ref.platform
+            context["source_label"] = source_label(ref.platform, ref.platform)
+            if ref.platform == "huggingface":
+                context["hf_url"] = ref.url
+            context["repo"] = ref.source_id
+
+        source = get_source(ref.platform) if ref is not None else None
+        if ref is not None and source is not None and source.supports_enrichment:
+            context["asset_base_url"] = source.asset_base_url(ref.source_id)
+            readme = await source.fetch_model_card(ref.source_id)
             # Trim README to the section relevant to this model file
             # (collection repos often have multiple models in one README).
             if readme and raw_basename:
@@ -458,20 +501,14 @@ class AgentService:
 
     @staticmethod
     async def _fetch_readme(repo: str) -> str:
-        """Fetch README.md from HuggingFace (tries ``main``, then ``master``)."""
-        async with aiohttp.ClientSession(
-            headers={"User-Agent": "ComfyUI-LoRA-Manager/1.0"},
-            timeout=aiohttp.ClientTimeout(total=30),
-        ) as session:
-            for branch in ("main", "master"):
-                url = f"https://huggingface.co/{repo}/raw/{branch}/README.md"
-                try:
-                    async with session.get(url) as resp:
-                        if resp.status == 200:
-                            return await resp.text()
-                except Exception as exc:
-                    logger.debug("Failed to fetch README from %s: %s", url, exc)
-        return ""
+        """Fetch a Hugging Face README (tries ``main``, then ``master``).
+
+        Kept for backward compatibility; new code should go through the
+        model-source registry so every supported site works.
+        """
+        from ..model_sources import HuggingFaceSource
+
+        return await HuggingFaceSource().fetch_model_card(repo)
 
     async def _emit_progress(
         self,

py/services/agent/post_processor.py

@@ -78,6 +78,7 @@ class PostProcessor:
             download_preview,
             refresh_cache,
         )
+        from ..model_sources import get_source, has_external_source, resolve_source_ref
         from .skills.enrich_hf_metadata.readme_processor import (
             convert_readme_to_html,
             extract_gallery_images,
@@ -85,17 +86,25 @@ class PostProcessor:
             extract_relevant_section,
             extract_simple_markdown_images,
             extract_html_img_tags,
-            extract_repo_from_hf_url,
         )
 
         updated_fields: List[str] = []
         preview_downloaded = False
 
-        # -- Determine whether this is an HF-sourced model -----------------
-        # Key off `hf_url` directly: `from_civitai` records provenance and can
-        # be true for a model that is also linked to HuggingFace (both sources
-        # coexist, see #1094), so it must not gate HF enrichment.
-        is_hf_model = bool(metadata.get("hf_url", ""))
+        # -- Determine whether this is an externally-sourced model ---------
+        # Key off the source fields directly: `from_civitai` records provenance
+        # and can be true for a model that is also linked to an external site
+        # (both sources coexist, see #1094), so it must not gate enrichment.
+        is_source_model = has_external_source(metadata)
+
+        source_ref = resolve_source_ref(metadata)
+        source = get_source(source_ref.platform) if source_ref else None
+        source_id = source_ref.source_id if source_ref else ""
+        asset_base_url = (
+            source.asset_base_url(source_id)
+            if source is not None and source_id
+            else None
+        )
 
         # -- Collect updates -----------------------------------------------
         updates: Dict[str, Any] = {}
@@ -103,7 +112,7 @@ class PostProcessor:
         # base_model
         new_base = (llm_output.get("base_model") or "").strip()
         current_base = metadata.get("base_model", "") or ""
-        if new_base and self._should_overwrite(current_base, is_hf_model):
+        if new_base and self._should_overwrite(current_base, is_source_model):
             updates["base_model"] = new_base
 
         # trigger words → civitai.trainedWords
@@ -115,7 +124,7 @@ class PostProcessor:
             trigger_words_empty = not cleaned
             current_civitai = metadata.get("civitai") or {}
             current_triggers = current_civitai.get("trainedWords") or []
-            if self._should_overwrite_list(current_triggers, is_hf_model):
+            if self._should_overwrite_list(current_triggers, is_source_model):
                 trig_civitai = dict(current_civitai)
                 if "civitai" in updates and isinstance(updates["civitai"], dict):
                     trig_civitai.update(updates["civitai"])
@@ -123,14 +132,14 @@ class PostProcessor:
                 updates["civitai"] = trig_civitai
 
         # modelDescription — from raw README content (converted to HTML)
-        if readme_content and is_hf_model:
+        if readme_content and is_source_model:
             converted = convert_readme_to_html(readme_content)
             if converted:
                 updates["modelDescription"] = converted
 
         # short_description → civitai.description (for "About this version")
         short_desc = (llm_output.get("short_description") or "").strip()
-        if short_desc and is_hf_model:
+        if short_desc and is_source_model:
             current_civitai = metadata.get("civitai") or {}
             desc_civitai = dict(current_civitai)
             if "civitai" in updates and isinstance(updates["civitai"], dict):
@@ -141,9 +150,8 @@ class PostProcessor:
         # gallery images → civitai.images (from YAML frontmatter widget entries
         # and Sample Gallery markdown tables in the README body)
         gallery_images: List[Dict[str, Any]] = []
-        if readme_content and is_hf_model:
-            hf_url = metadata.get("hf_url", "") or ""
-            repo = extract_repo_from_hf_url(hf_url)
+        if readme_content and is_source_model:
+            repo = source_id
             if repo:
                 rec_w = llm_output.get("recommended_width") or 0
                 rec_h = llm_output.get("recommended_height") or 0
@@ -152,6 +160,7 @@ class PostProcessor:
                 gallery = extract_gallery_images(
                     readme_content, repo,
                     default_width=rec_w, default_height=rec_h,
+                    base_url=asset_base_url,
                 )
 
                 # 2. Sample Gallery table images (markdown body), deduplicated
@@ -160,6 +169,7 @@ class PostProcessor:
                     readme_content, repo,
                     existing_urls=existing_urls,
                     default_width=rec_w, default_height=rec_h,
+                    base_url=asset_base_url,
                 )
                 existing_urls.update(img["url"] for img in table_images if img.get("url"))
 
@@ -168,6 +178,7 @@ class PostProcessor:
                     readme_content, repo,
                     existing_urls=existing_urls,
                     default_width=rec_w, default_height=rec_h,
+                    base_url=asset_base_url,
                 )
                 existing_urls.update(img["url"] for img in simple_images if img.get("url"))
 
@@ -176,6 +187,7 @@ class PostProcessor:
                     readme_content, repo,
                     existing_urls=existing_urls,
                     default_width=rec_w, default_height=rec_h,
+                    base_url=asset_base_url,
                 )
 
                 all_images = gallery + table_images + simple_images + html_images
@@ -193,7 +205,7 @@ class PostProcessor:
         if isinstance(new_tags, list) and new_tags:
             existing_tags = metadata.get("tags") or []
             merged = self._merge_tags(existing_tags, new_tags)
-            if len(merged) > len(existing_tags) or is_hf_model:
+            if len(merged) > len(existing_tags) or is_source_model:
                 updates["tags"] = merged
 
         # metadata_source & llm_enriched_at (always set)
@@ -222,7 +234,7 @@ class PostProcessor:
         # README, find the first gallery image from the *model-specific
         # section* of the README (not the repo-wide first image, which
         # belongs to a different model in collection repos).
-        if not preview_remote_url and readme_content and is_hf_model:
+        if not preview_remote_url and readme_content and is_source_model:
             model_basename = os.path.splitext(os.path.basename(model_path))[0]
             relevant_section = extract_relevant_section(
                 readme_content, model_basename,
@@ -279,16 +291,16 @@ class PostProcessor:
     # ------------------------------------------------------------------
 
     @staticmethod
-    def _should_overwrite(current_value: str, is_hf_model: bool) -> bool:
+    def _should_overwrite(current_value: str, is_source_model: bool) -> bool:
         """Return ``True`` when a scalar field should be overwritten."""
-        return is_hf_model or not current_value or current_value.lower() in (
+        return is_source_model or not current_value or current_value.lower() in (
             "", "unknown",
         )
 
     @staticmethod
-    def _should_overwrite_list(current_list: List[str], is_hf_model: bool) -> bool:
+    def _should_overwrite_list(current_list: List[str], is_source_model: bool) -> bool:
         """Return ``True`` when a list field should be overwritten."""
-        return is_hf_model or not current_list
+        return is_source_model or not current_list
 
     @staticmethod
     def _merge_tags(existing: List[str], new: List[str]) -> List[str]:

py/services/agent/skills/enrich_hf_metadata/prompt.md

@@ -1,20 +1,23 @@
 ---
 name: enrich_hf_metadata
-title: "Enrich Metadata from HuggingFace"
+title: "Enrich Metadata from Model Card"
 description: >
-  Parse the HuggingFace model card via LLM to extract description, trigger
-  words, base model, tags, and preview image URL.
+  Parse the model card (README) from HuggingFace, ModelScope, or any other
+  supported model site via LLM to extract description, trigger words, base
+  model, tags, and preview image URL.
 llm_required: true
 ---
 
-You are an expert assistant for AI image generation models. Your task is to extract structured metadata from a HuggingFace model card (README.md).
+You are an expert assistant for AI image generation models. Your task is to extract structured metadata from a model card (README).
 
 ## Model Information
 
-- **Repository**: {{hf_url}}
+- **Source site**: {{source_label}} ({{source_platform}})
+- **Model page**: {{source_url}}
 - **Model file path**: {{model_path}}
 - **Model filename**: {{model_basename}}
-- **Repository ID**: {{repo}}
+- **Repository ID**: {{source_id}}
+- **Repository raw-file base URL**: {{asset_base_url}}
 
 ## Current Metadata (may be incomplete)
 
@@ -39,7 +42,7 @@ name listed — do not invent aliases or modify variant suffixes.
 
 {{base_models}}
 
-## HuggingFace README Content
+## Model Card Content
 
 ```
 {{readme_content}}
@@ -92,7 +95,7 @@ The URL of the most suitable preview image from the README. Look for:
 - The YAML frontmatter `widget:` section (which often has `output.url` fields)
 - In collection repos: the sample images listed **under the section** for this specific model version
 - Generic `![alt](url)` in the body
-Choose the first image that appears to be a generation example (not a logo or diagram). Construct the absolute URL as `https://huggingface.co/{{repo}}/resolve/main/{filename}`. If no suitable image is found, return an empty string.
+Choose the first image that appears to be a generation example (not a logo or diagram). Construct the absolute URL from the repository raw-file base URL (`{{asset_base_url}}`) plus the relative path. If no suitable image is found, return an empty string.
 
 ### notes
 A plain-text summary of the model card's key practical usage information. Combine trigger words, style modifiers, recommended parameters (steps, CFG, resolution, sampler), and any setup tips into a readable paragraph.  For collection repos, focus on the **specific model version** matching `{{model_basename}}`.  Return empty string if the README has no useful usage info.
@@ -121,7 +124,7 @@ Your confidence level in the extracted data:
 
 ## Important: Handling Collection Repos (multiple model files)
 
-Many HuggingFace repos contain **multiple model files** in a single repository
+Many model repositories contain **multiple model files** in a single repository
 (e.g. a "LoRA collection" with different styles/characters in separate files).
 
 The model file currently being enriched is: **`{{model_basename}}`**

py/services/agent/skills/enrich_hf_metadata/readme_processor.py

@@ -1,8 +1,15 @@
-"""HF README processing for the ``enrich_hf_metadata`` skill.
+"""Model card (README) processing for the ``enrich_hf_metadata`` skill.
 
 Provides README cleaning for LLM injection, gallery/image extraction from
 multiple formats (YAML widget, markdown, HTML ``<img>``, gallery tables),
 and section-based README trimming for collection repos.
+
+The extractors default to Hugging Face asset URLs, but every one of them
+accepts an explicit ``base_url`` so the same parsing works for any model
+source (ModelScope, ...).  See :mod:`py.services.model_sources`.
+
+This module deliberately has no package-relative imports: it is also loaded
+standalone by the README-processing test harness.
 """
 
 from __future__ import annotations
@@ -15,12 +22,25 @@ from typing import Any, List, Tuple
 _REPO_URL_PATTERN = re.compile(r"https?://huggingface\.co/([^/]+/[^/]+)")
 
 
+def resolve_asset_base_url(repo: str, base_url: str | None = None) -> str:
+    """Return the base URL used to resolve repository-relative assets.
+
+    Falls back to the historical Hugging Face layout when *base_url* is not
+    supplied, so existing callers keep their behaviour.
+    """
+
+    if base_url:
+        return base_url.rstrip("/")
+    return f"https://huggingface.co/{repo}/resolve/main"
+
+
 def extract_simple_markdown_images(
     markdown_text: str,
     repo: str,
     existing_urls: set[str] | None = None,
     default_width: int = 512,
     default_height: int = 512,
+    base_url: str | None = None,
 ) -> list[dict[str, Any]]:
     """Extract standalone markdown images from the README body.
 
@@ -32,10 +52,10 @@ def extract_simple_markdown_images(
     Returns a list of dicts in the same ``civitai.images`` format as
     :func:`extract_gallery_images`.
     """
-    if not markdown_text or not repo:
+    if not markdown_text or not (repo or base_url):
         return []
 
-    base_url = f"https://huggingface.co/{repo}/resolve/main"
+    base_url = resolve_asset_base_url(repo, base_url)
     images: list[dict[str, Any]] = []
     seen_urls: set[str] = set(existing_urls) if existing_urls else set()
 
@@ -89,20 +109,21 @@ def extract_html_img_tags(
     existing_urls: set[str] | None = None,
     default_width: int = 512,
     default_height: int = 512,
+    base_url: str | None = None,
 ) -> list[dict[str, Any]]:
     """Extract image URLs from HTML ``<img src=\"...\">`` tags in the README.
 
     Many HF collection repos (e.g. ``deadman44/Z-Image_LoRA``) use raw HTML
     ``<img>`` tags exclusively for their sample images, with no markdown
     ``![]()`` equivalents.  This function finds those tags and constructs
-    resolvable HF URLs.
+    resolvable URLs.
 
     Returns a list of dicts in the ``civitai.images`` format.
     """
-    if not markdown_text or not repo:
+    if not markdown_text or not (repo or base_url):
         return []
 
-    base_url = f"https://huggingface.co/{repo}/resolve/main"
+    base_url = resolve_asset_base_url(repo, base_url)
     images: list[dict[str, Any]] = []
     seen_urls: set[str] = set(existing_urls) if existing_urls else set()
 
@@ -166,7 +187,7 @@ def extract_html_img_tags(
 
 def extract_repo_from_hf_url(hf_url: str) -> str:
     """Extract ``user/repo`` from a HuggingFace URL."""
-    m = _REPO_URL_PATTERN.match(hf_url)
+    m = _REPO_URL_PATTERN.match(hf_url or "")
     return m.group(1) if m else ""
 
 
@@ -175,21 +196,23 @@ def extract_gallery_images(
     repo: str,
     default_width: int = 512,
     default_height: int = 512,
+    base_url: str | None = None,
 ) -> List[dict[str, Any]]:
-    """Extract widget/gallery images from the YAML frontmatter of a HF README.
+    """Extract widget/gallery images from the YAML frontmatter of a README.
 
     Args:
         markdown_text: Raw README content.
-        repo: HF repo identifier (``user/repo``).
+        repo: Repository identifier (``user/repo``).
         default_width: Fallback width when the README provides no dimension.
         default_height: Fallback height when the README provides no dimension.
+        base_url: Overrides the asset base URL (defaults to Hugging Face).
 
     Returns a list of dicts compatible with the ``civitai.images`` metadata
-    format, each containing ``url`` (absolute HF URL), ``meta.prompt``,
+    format, each containing ``url`` (absolute), ``meta.prompt``,
     ``width``, ``height``, and ``type``.  Returns an empty list when no
     widget entries are found or when *repo* is empty.
     """
-    if not markdown_text or not repo:
+    if not markdown_text or not (repo or base_url):
         return []
 
     frontmatter = _extract_frontmatter(markdown_text)
@@ -197,7 +220,7 @@ def extract_gallery_images(
         return []
 
     images: List[dict[str, Any]] = []
-    base_url = f"https://huggingface.co/{repo}/resolve/main"
+    base_url = resolve_asset_base_url(repo, base_url)
     w = default_width or 512
     h = default_height or 512
 
@@ -279,10 +302,11 @@ def extract_gallery_table_images(
     existing_urls: set[str] | None = None,
     default_width: int = 512,
     default_height: int = 512,
+    base_url: str | None = None,
 ) -> list[dict[str, Any]]:
     """Extract images from ``| Preview | Prompt |`` markdown gallery tables.
 
-    Many HF READMEs include a sample-gallery table in the body (outside
+    Many READMEs include a sample-gallery table in the body (outside
     the YAML frontmatter) that shows generation examples with their
     prompts.  This function parses those tables and merges results with
     the widget-sourced images from :func:`extract_gallery_images`.
@@ -291,10 +315,10 @@ def extract_gallery_table_images(
     :func:`extract_gallery_images`.  Already-seen URLs (from *existing_urls*)
     are skipped.
     """
-    if not markdown_text or not repo:
+    if not markdown_text or not (repo or base_url):
         return []
 
-    base_url = f"https://huggingface.co/{repo}/resolve/main"
+    base_url = resolve_asset_base_url(repo, base_url)
     images: list[dict[str, Any]] = []
     seen_urls: set[str] = set(existing_urls) if existing_urls else set()
     lines = markdown_text.split("\n")