feat(links): support ModelScope and TensorArt as model sources

A model file could only ever be linked to huggingface.co: `set_hf_url` validated the URL with a huggingface-only regex, the agent fetched the card from a hardcoded HF URL, and the readme processor built every relative image path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the same model-card convention (README.md + YAML frontmatter, often carrying `base_model:` and `trigger_words:`) behind a public, key-less API, so the enrichment pipeline could already serve it - it was the plumbing that was HF-shaped, not the idea. Make the external source a first-class, provider-driven concept: - New `py/services/model_sources/` registry. A `ModelSource` owns URL recognition (lenient for stored values, strict for user input), the canonical page URL, model-card fetching, the asset base URL and the capability flags. `HuggingFaceSource` is the previous logic relocated; `ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md` and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is link-only on purpose: tensor.art answers plain HTTP clients with a Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud / cn.tensorart.net) rejects every /v1/model/* route with "invalid authorization header", so it declares supports_enrichment=False rather than failing silently later. - Metadata gains `source_platform` + `source_url`; `hf_url` stays as a read/write alias, written only for Hugging Face, so existing sidecars, cached rows and third-party consumers keep working. Normalisation runs at the scanner, the persistent cache (both directions, plus two new columns behind an ALTER migration) and the linking handler - which is what stops a user who switches sources from leaving a stale `hf_url` on a ModelScope model. - The agent pipeline keys off the provider instead of `hf_url`: the fast-fail gate now explains *why* a model is skipped (no source / unknown source / source without a reachable card), the prompt context exposes source_url/source_id/source_label/asset_base_url while still filling the legacy hf_url/repo aliases, and the four README image extractors take a base_url (defaulting to HF) so relative paths resolve against the right site. Version grouping generalises to hf: / ms: / ta: keys. - `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but accepts `source_url`, validates against every provider and returns the platform. `GET /api/lm/model-sources` lets the UI render the supported-site list from the server. - Frontend: a `modelSourceHelpers` mirror of the registry drives the link dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the version-group key and the enrichment gate; the versions tab no longer sends ms:/ta: keys to the CivitAI API. TensorArt stays in the list because provenance is worth keeping even when the card is unreadable - the dialog says so plainly ("Sites that don't expose one (currently TensorArt) can only be linked") and the context menu disables enrichment with a matching tooltip, instead of the user getting "Unsupported URL". Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a 1882-byte card whose frontmatter carries base_model/tags/trigger_words, and relative images resolve to .../resolve/master/.... Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest tests/i18n and a Jinja compile pass over templates/. The nine locales carry [TODO: Translate] for the new strings, completed in the next commit.
2026-09-21 03:01:27 -03:00 · 2026-09-14 07:24:08 +08:00
parent 84146b62fd
commit 5ab0e88abc
51 changed files with 2518 additions and 279 deletions
@@ -0,0 +1,182 @@
+"""Tests for source-aware AI enrichment orchestration.
+
+Covers the fast-fail gate (:meth:`AgentService._enrichment_skip_reason`) and
+the prompt-context builder for non-Hugging Face model sources.
+"""
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+
+from py.services.agent.agent_service import AgentService
+
+
+class TestEnrichmentSkipReason:
+    def test_skips_when_no_source_linked(self):
+        reason = AgentService._enrichment_skip_reason({})
+        assert "source_url" in reason
+
+    def test_allows_huggingface(self):
+        assert (
+            AgentService._enrichment_skip_reason(
+                {"hf_url": "https://huggingface.co/user/repo"}
+            )
+            == ""
+        )
+
+    def test_allows_modelscope(self):
+        assert (
+            AgentService._enrichment_skip_reason(
+                {
+                    "source_platform": "modelscope",
+                    "source_url": "https://modelscope.cn/models/user/repo",
+                }
+            )
+            == ""
+        )
+
+    def test_skips_tensorart_with_reason(self):
+        reason = AgentService._enrichment_skip_reason(
+            {
+                "source_platform": "tensorart",
+                "source_url": "https://tensor.art/models/827823520299086029",
+            }
+        )
+        assert "TensorArt" in reason
+        assert "not available" in reason
+
+    def test_skips_unknown_platform(self):
+        reason = AgentService._enrichment_skip_reason(
+            {"source_platform": "somewhere", "source_url": "https://somewhere.example/m/1"}
+        )
+        assert "somewhere" in reason
+
+
+class TestBuildPromptContext:
+    @pytest.mark.asyncio
+    async def test_modelscope_card_populates_source_variables(self):
+        service = AgentService()
+        readme = "---\nbase_model: krea/Krea-2-Turbo\n---\n# krea\n"
+
+        with (
+            mock.patch(
+                "py.services.model_sources.modelscope.ModelScopeSource.fetch_model_card",
+                new=mock.AsyncMock(return_value=readme),
+            ) as mock_fetch,
+            mock.patch(
+                "py.metadata_ops.list_base_models",
+                new=mock.AsyncMock(return_value=["Krea 2 Turbo"]),
+            ),
+            mock.patch(
+                "py.metadata_ops.identify_model_type",
+                new=mock.AsyncMock(return_value="lora"),
+            ),
+            mock.patch(
+                "py.services.settings_manager.SettingsManager.get_priority_tag_config",
+                return_value={"lora": "style, subject"},
+            ),
+        ):
+            context = await service._build_prompt_context(
+                skill_name="enrich_hf_metadata",
+                model_path="/models/loras/krea.safetensors",
+                metadata={
+                    "source_platform": "modelscope",
+                    "source_url": "https://modelscope.cn/models/jj3550945163/Krea-2-LORA",
+                    "file_name": "krea",
+                },
+                registry=mock.Mock(),
+                llm=mock.Mock(),
+            )
+
+        mock_fetch.assert_awaited_once_with("jj3550945163/Krea-2-LORA")
+        assert context["source_platform"] == "modelscope"
+        assert context["source_id"] == "jj3550945163/Krea-2-LORA"
+        assert context["source_label"] == "ModelScope"
+        assert (
+            context["asset_base_url"]
+            == "https://modelscope.cn/models/jj3550945163/Krea-2-LORA/resolve/master"
+        )
+        assert readme in context["readme_content_full"]
+        # Hugging Face aliases stay empty for a non-HF source.
+        assert context["hf_url"] == ""
+        assert context["repo"] == "jj3550945163/Krea-2-LORA"
+
+    @pytest.mark.asyncio
+    async def test_huggingface_keeps_legacy_aliases(self):
+        service = AgentService()
+        readme = "# card\n"
+
+        with (
+            mock.patch(
+                "py.services.model_sources.huggingface.HuggingFaceSource.fetch_model_card",
+                new=mock.AsyncMock(return_value=readme),
+            ) as mock_fetch,
+            mock.patch(
+                "py.metadata_ops.list_base_models",
+                new=mock.AsyncMock(return_value=[]),
+            ),
+            mock.patch(
+                "py.metadata_ops.identify_model_type",
+                new=mock.AsyncMock(return_value="lora"),
+            ),
+            mock.patch(
+                "py.services.settings_manager.SettingsManager.get_priority_tag_config",
+                return_value={},
+            ),
+        ):
+            context = await service._build_prompt_context(
+                skill_name="enrich_hf_metadata",
+                model_path="/models/loras/thing.safetensors",
+                metadata={"hf_url": "https://huggingface.co/user/repo"},
+                registry=mock.Mock(),
+                llm=mock.Mock(),
+            )
+
+        mock_fetch.assert_awaited_once_with("user/repo")
+        assert context["source_platform"] == "huggingface"
+        assert context["hf_url"] == "https://huggingface.co/user/repo"
+        assert context["repo"] == "user/repo"
+
+    @pytest.mark.asyncio
+    async def test_tensorart_never_fetches_a_card(self):
+        service = AgentService()
+
+        with (
+            mock.patch(
+                "py.services.model_sources.huggingface.HuggingFaceSource.fetch_model_card",
+                new=mock.AsyncMock(),
+            ) as hf_fetch,
+            mock.patch(
+                "py.services.model_sources.modelscope.ModelScopeSource.fetch_model_card",
+                new=mock.AsyncMock(),
+            ) as ms_fetch,
+            mock.patch(
+                "py.metadata_ops.list_base_models",
+                new=mock.AsyncMock(return_value=[]),
+            ),
+            mock.patch(
+                "py.metadata_ops.identify_model_type",
+                new=mock.AsyncMock(return_value="lora"),
+            ),
+            mock.patch(
+                "py.services.settings_manager.SettingsManager.get_priority_tag_config",
+                return_value={},
+            ),
+        ):
+            context = await service._build_prompt_context(
+                skill_name="enrich_hf_metadata",
+                model_path="/models/loras/thing.safetensors",
+                metadata={
+                    "source_platform": "tensorart",
+                    "source_url": "https://tensor.art/models/827823520299086029",
+                },
+                registry=mock.Mock(),
+                llm=mock.Mock(),
+            )
+
+        hf_fetch.assert_not_awaited()
+        ms_fetch.assert_not_awaited()
+        assert context["readme_content"] == ""
+        assert context["source_platform"] == "tensorart"
@@ -964,6 +964,8 @@ def _make_cache_entry(**overrides) -> Dict[str, Any]:
        "civitai": {"id": 111, "modelId": 222, "name": "v1"},
        "civitai_deleted": False,
        "skip_metadata_refresh": False,
+        "source_platform": "",
+        "source_url": "",
        "hf_url": "",
        "license_flags": 113,
        "hash_status": "completed",
@@ -0,0 +1,324 @@
+"""Tests for the external model-source provider registry.
+
+Covers URL recognition for Hugging Face / ModelScope / TensorArt, the
+legacy ``hf_url`` → ``source_url`` normalisation, version-group keys, and
+each provider's model-card fetching and capability flags.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from py.services.model_sources import (
+    HuggingFaceSource,
+    ModelScopeSource,
+    TensorArtSource,
+    detect_source,
+    get_source,
+    get_source_platform,
+    has_external_source,
+    list_sources,
+    normalize_metadata_source,
+    resolve_source_ref,
+    source_group_key,
+    source_label,
+)
+
+
+# ---------------------------------------------------------------------------
+# URL recognition
+# ---------------------------------------------------------------------------
+
+
+class TestDetectSource:
+    @pytest.mark.parametrize(
+        ("url", "platform", "source_id"),
+        [
+            ("https://huggingface.co/user/repo", "huggingface", "user/repo"),
+            ("https://www.huggingface.co/user/repo", "huggingface", "user/repo"),
+            (
+                "https://huggingface.co/user/repo/resolve/main/model.safetensors",
+                "huggingface",
+                "user/repo",
+            ),
+            (
+                "https://modelscope.cn/models/jj3550945163/Krea-2-LORA",
+                "modelscope",
+                "jj3550945163/Krea-2-LORA",
+            ),
+            (
+                "https://www.modelscope.cn/models/jj3550945163/Krea-2-LORA/summary",
+                "modelscope",
+                "jj3550945163/Krea-2-LORA",
+            ),
+            (
+                "https://tensor.art/models/827823520299086029/Vivid-Impressions-Storybook-Sstyle-V1.0",
+                "tensorart",
+                "827823520299086029",
+            ),
+            ("https://tusi.cn/models/827823520299086029", "tensorart", "827823520299086029"),
+        ],
+    )
+    def test_recognises_supported_urls(self, url, platform, source_id):
+        ref = detect_source(url)
+        assert ref is not None
+        assert ref.platform == platform
+        assert ref.source_id == source_id
+
+    @pytest.mark.parametrize(
+        "url",
+        [
+            "",
+            None,
+            "not-a-url",
+            "https://example.com/x",
+            "https://civitai.com/models/123",
+        ],
+    )
+    def test_ignores_unsupported_urls(self, url):
+        assert detect_source(url) is None
+
+    def test_canonical_url_is_stable(self):
+        assert detect_source("https://huggingface.co/u/r").url == "https://huggingface.co/u/r"
+        assert (
+            detect_source("https://modelscope.cn/models/u/r/summary").url
+            == "https://modelscope.cn/models/u/r"
+        )
+        assert (
+            detect_source("https://tensor.art/models/123/some-slug").url
+            == "https://tensor.art/models/123"
+        )
+
+
+class TestStrictParsing:
+    @pytest.mark.parametrize(
+        "url",
+        [
+            "https://huggingface.co/user/repo",
+            "https://huggingface.co/user/repo/",
+            "https://modelscope.cn/models/user/repo",
+            "https://modelscope.cn/models/user/repo/summary",
+            "https://tensor.art/models/827823520299086029",
+            "https://tensor.art/models/827823520299086029/Vivid-Impressions",
+        ],
+    )
+    def test_accepts_user_facing_urls(self, url):
+        assert detect_source(url, strict=True) is not None
+
+    @pytest.mark.parametrize(
+        "url",
+        [
+            "https://huggingface.co/user/repo/resolve/main/model.safetensors",
+            "https://example.com/x",
+            "https://tensor.art/models/not-a-number",
+        ],
+    )
+    def test_rejects_non_page_urls(self, url):
+        assert detect_source(url, strict=True) is None
+
+
+# ---------------------------------------------------------------------------
+# Capabilities
+# ---------------------------------------------------------------------------
+
+
+class TestCapabilities:
+    def test_huggingface_supports_everything(self):
+        source = get_source("huggingface")
+        assert source.supports_enrichment is True
+        assert source.supports_download is True
+
+    def test_modelscope_supports_enrichment_but_not_download(self):
+        source = get_source("modelscope")
+        assert source.supports_enrichment is True
+        assert source.supports_download is False
+
+    def test_tensorart_is_link_only(self):
+        source = get_source("tensorart")
+        assert source.supports_enrichment is False
+        assert source.supports_download is False
+
+    def test_registry_lists_every_source(self):
+        platforms = {s.platform for s in list_sources()}
+        assert platforms == {"huggingface", "modelscope", "tensorart"}
+
+    def test_labels_are_brand_names(self):
+        assert source_label("huggingface") == "Hugging Face"
+        assert source_label("modelscope") == "ModelScope"
+        assert source_label("tensorart") == "TensorArt"
+        assert source_label("unknown", "fallback") == "fallback"
+
+
+# ---------------------------------------------------------------------------
+# Metadata normalisation
+# ---------------------------------------------------------------------------
+
+
+class TestNormalizeMetadataSource:
+    def test_derives_source_fields_from_legacy_hf_url(self):
+        metadata = {"hf_url": "https://huggingface.co/user/repo"}
+        normalize_metadata_source(metadata)
+        assert metadata["source_platform"] == "huggingface"
+        assert metadata["source_url"] == "https://huggingface.co/user/repo"
+        assert metadata["hf_url"] == "https://huggingface.co/user/repo"
+
+    def test_canonicalises_modelscope_url_and_clears_hf_alias(self):
+        metadata = {
+            "source_platform": "modelscope",
+            "source_url": "https://modelscope.cn/models/user/repo/summary",
+            "hf_url": "https://huggingface.co/old/repo",
+        }
+        normalize_metadata_source(metadata)
+        assert metadata["source_platform"] == "modelscope"
+        assert metadata["source_url"] == "https://modelscope.cn/models/user/repo"
+        # A stale HF alias must not make a ModelScope model look like HF.
+        assert metadata["hf_url"] == ""
+
+    def test_preserves_unknown_url_for_unknown_platform(self):
+        metadata = {"source_url": "https://example.com/model/1", "source_platform": "other"}
+        normalize_metadata_source(metadata)
+        assert metadata["source_url"] == "https://example.com/model/1"
+        assert metadata["source_platform"] == "other"
+
+    def test_empty_metadata_gets_default_fields(self):
+        metadata: dict = {}
+        normalize_metadata_source(metadata)
+        assert metadata["source_platform"] == ""
+        assert metadata["source_url"] == ""
+
+    def test_infers_platform_from_url_when_missing(self):
+        metadata = {"source_url": "https://modelscope.cn/models/user/repo"}
+        normalize_metadata_source(metadata)
+        assert metadata["source_platform"] == "modelscope"
+
+
+class TestResolveSourceRef:
+    def test_resolves_from_canonical_fields(self):
+        ref = resolve_source_ref(
+            {"source_platform": "modelscope", "source_url": "https://modelscope.cn/models/u/r"}
+        )
+        assert ref is not None
+        assert ref.platform == "modelscope"
+        assert ref.source_id == "u/r"
+
+    def test_resolves_from_legacy_hf_url(self):
+        ref = resolve_source_ref({"hf_url": "https://huggingface.co/u/r"})
+        assert ref is not None
+        assert ref.platform == "huggingface"
+
+    def test_returns_none_without_any_source(self):
+        assert resolve_source_ref({}) is None
+        assert resolve_source_ref({"hf_url": ""}) is None
+
+
+class TestHelpers:
+    def test_has_external_source_accepts_both_field_shapes(self):
+        assert has_external_source({"hf_url": "https://huggingface.co/u/r"}) is True
+        assert has_external_source({"source_url": "https://modelscope.cn/models/u/r"}) is True
+        assert has_external_source({"source_url": ""}) is False
+        assert has_external_source({}) is False
+
+    def test_get_source_platform_infers_from_url(self):
+        assert get_source_platform({"hf_url": "https://huggingface.co/u/r"}) == "huggingface"
+        assert get_source_platform({"source_platform": "tensorart"}) == "tensorart"
+        assert get_source_platform({}) == ""
+
+    def test_group_keys_match_legacy_hf_shape(self):
+        assert source_group_key({"hf_url": "https://huggingface.co/u/r"}) == "hf:u/r"
+        assert (
+            source_group_key({"source_url": "https://modelscope.cn/models/u/r"}) == "ms:u/r"
+        )
+        assert (
+            source_group_key({"source_url": "https://tensor.art/models/123"}) == "ta:123"
+        )
+
+    def test_group_key_is_none_without_source(self):
+        assert source_group_key({}) is None
+        assert source_group_key({"hf_url": "https://example.com/x"}) is None
+
+
+# ---------------------------------------------------------------------------
+# Model card fetching
+# ---------------------------------------------------------------------------
+
+
+class TestFetchModelCard:
+    @pytest.mark.asyncio
+    async def test_huggingface_tries_main_then_master(self, monkeypatch):
+        calls: list[str] = []
+
+        async def fake_fetch_text(url: str, **_kwargs) -> str:
+            calls.append(url)
+            if url.endswith("/master/README.md"):
+                return "# card"
+            return ""
+
+        monkeypatch.setattr("py.services.model_sources.huggingface.fetch_text", fake_fetch_text)
+
+        card = await HuggingFaceSource().fetch_model_card("user/repo")
+
+        assert card == "# card"
+        assert calls == [
+            "https://huggingface.co/user/repo/raw/main/README.md",
+            "https://huggingface.co/user/repo/raw/master/README.md",
+        ]
+
+    @pytest.mark.asyncio
+    async def test_modelscope_prefers_resolve_url(self, monkeypatch):
+        calls: list[str] = []
+
+        async def fake_fetch_text(url: str, **_kwargs) -> str:
+            calls.append(url)
+            return "---\nbase_model: krea/Krea-2-Turbo\n---\n# krea"
+
+        monkeypatch.setattr("py.services.model_sources.modelscope.fetch_text", fake_fetch_text)
+
+        card = await ModelScopeSource().fetch_model_card("u/r")
+
+        assert card.startswith("---")
+        assert calls == ["https://modelscope.cn/models/u/r/resolve/master/README.md"]
+
+    @pytest.mark.asyncio
+    async def test_modelscope_falls_back_to_repo_api(self, monkeypatch):
+        calls: list[str] = []
+
+        async def fake_fetch_text(url: str, **_kwargs) -> str:
+            calls.append(url)
+            if "/api/v1/models/" in url:
+                return "# from api"
+            return ""
+
+        monkeypatch.setattr("py.services.model_sources.modelscope.fetch_text", fake_fetch_text)
+
+        card = await ModelScopeSource().fetch_model_card("u/r")
+
+        assert card == "# from api"
+        assert "resolve/master/README.md" in calls[0]
+        assert (
+            "https://modelscope.cn/api/v1/models/u/r/repo?Revision=master&FilePath=README.md"
+            in calls
+        )
+
+    @pytest.mark.asyncio
+    async def test_tensorart_never_fetches(self):
+        # TensorArt enrichment is disabled: the provider must not issue any
+        # HTTP request, so it deliberately does not import `fetch_text`.
+        import importlib
+
+        module = importlib.import_module("py.services.model_sources.tensorart")
+        assert not hasattr(module, "fetch_text")
+        assert await TensorArtSource().fetch_model_card("123") == ""
+
+
+class TestAssetBaseUrl:
+    def test_huggingface_uses_main_revision(self):
+        assert (
+            HuggingFaceSource().asset_base_url("u/r")
+            == "https://huggingface.co/u/r/resolve/main"
+        )
+
+    def test_modelscope_uses_master_revision(self):
+        assert (
+            ModelScopeSource().asset_base_url("u/r")
+            == "https://modelscope.cn/models/u/r/resolve/master"
+        )
@@ -292,6 +292,61 @@ Content
        )
        assert images[0]["meta"]["prompt"] == "a cat"

+    @pytest.mark.asyncio
+    async def test_gallery_images_use_modelscope_asset_base_url(self, processor):
+        """A ModelScope-linked model resolves relative images against ModelScope."""
+        readme = """---
+widget:
+- text: "a cat"
+  output:
+    url: images/cat.png
+---
+Content
+"""
+        with (
+            mock.patch("py.metadata_ops.apply_metadata_updates") as mock_apply,
+            mock.patch("py.metadata_ops.download_preview", return_value=None),
+            mock.patch("py.metadata_ops.refresh_cache"),
+        ):
+            await processor.process(
+                skill_name="enrich_hf_metadata",
+                model_path="/p.safetensors",
+                llm_output=self.MIN_LLM_OUTPUT,
+                metadata={
+                    "from_civitai": False,
+                    "source_platform": "modelscope",
+                    "source_url": "https://modelscope.cn/models/user/repo",
+                },
+                readme_content=readme,
+            )
+        applied = mock_apply.call_args[0][1]
+        images = applied.get("civitai", {}).get("images", [])
+        assert len(images) == 1
+        assert images[0]["url"] == (
+            "https://modelscope.cn/models/user/repo/resolve/master/images/cat.png"
+        )
+
+    @pytest.mark.asyncio
+    async def test_base_model_overwrites_existing_modelscope_model(self, processor):
+        """ModelScope is an external source, so the LLM may overwrite base_model."""
+        llm = {**self.MIN_LLM_OUTPUT, "base_model": "Flux.1 D"}
+        with (
+            mock.patch("py.metadata_ops.apply_metadata_updates") as mock_apply,
+            mock.patch("py.metadata_ops.download_preview", return_value=False),
+            mock.patch("py.metadata_ops.refresh_cache"),
+        ):
+            await processor.process(
+                skill_name="enrich_hf_metadata",
+                model_path="/p.safetensors",
+                llm_output=llm,
+                metadata={
+                    "base_model": "SD 1.5",
+                    "source_platform": "modelscope",
+                    "source_url": "https://modelscope.cn/models/user/repo",
+                },
+            )
+        assert mock_apply.call_args[0][1]["base_model"] == "Flux.1 D"
+
    @pytest.mark.asyncio
    async def test_gallery_images_skipped_without_hf_url(self, processor):
        """Gallery images NOT extracted when the model has no HF source."""