feat(links): support ModelScope and TensorArt as model sources

A model file could only ever be linked to huggingface.co: `set_hf_url` validated the URL with a huggingface-only regex, the agent fetched the card from a hardcoded HF URL, and the readme processor built every relative image path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the same model-card convention (README.md + YAML frontmatter, often carrying `base_model:` and `trigger_words:`) behind a public, key-less API, so the enrichment pipeline could already serve it - it was the plumbing that was HF-shaped, not the idea. Make the external source a first-class, provider-driven concept: - New `py/services/model_sources/` registry. A `ModelSource` owns URL recognition (lenient for stored values, strict for user input), the canonical page URL, model-card fetching, the asset base URL and the capability flags. `HuggingFaceSource` is the previous logic relocated; `ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md` and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is link-only on purpose: tensor.art answers plain HTTP clients with a Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud / cn.tensorart.net) rejects every /v1/model/* route with "invalid authorization header", so it declares supports_enrichment=False rather than failing silently later. - Metadata gains `source_platform` + `source_url`; `hf_url` stays as a read/write alias, written only for Hugging Face, so existing sidecars, cached rows and third-party consumers keep working. Normalisation runs at the scanner, the persistent cache (both directions, plus two new columns behind an ALTER migration) and the linking handler - which is what stops a user who switches sources from leaving a stale `hf_url` on a ModelScope model. - The agent pipeline keys off the provider instead of `hf_url`: the fast-fail gate now explains *why* a model is skipped (no source / unknown source / source without a reachable card), the prompt context exposes source_url/source_id/source_label/asset_base_url while still filling the legacy hf_url/repo aliases, and the four README image extractors take a base_url (defaulting to HF) so relative paths resolve against the right site. Version grouping generalises to hf: / ms: / ta: keys. - `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but accepts `source_url`, validates against every provider and returns the platform. `GET /api/lm/model-sources` lets the UI render the supported-site list from the server. - Frontend: a `modelSourceHelpers` mirror of the registry drives the link dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the version-group key and the enrichment gate; the versions tab no longer sends ms:/ta: keys to the CivitAI API. TensorArt stays in the list because provenance is worth keeping even when the card is unreadable - the dialog says so plainly ("Sites that don't expose one (currently TensorArt) can only be linked") and the context menu disables enrichment with a matching tooltip, instead of the user getting "Unsupported URL". Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a 1882-byte card whose frontmatter carries base_model/tags/trigger_words, and relative images resolve to .../resolve/master/.... Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest tests/i18n and a Jinja compile pass over templates/. The nine locales carry [TODO: Translate] for the new strings, completed in the next commit.
2026-09-20 18:51:26 -03:00 · 2026-09-14 07:24:08 +08:00
parent 84146b62fd
commit 5ab0e88abc
51 changed files with 2518 additions and 279 deletions
@@ -191,4 +191,60 @@ describe('ModelCard source globe (#1094)', () => {
    expect(openHuggingFace).toHaveBeenCalledWith('https://huggingface.co/user/repo');
    expect(openCivitai).not.toHaveBeenCalled();
  });
+
+  it('points the globe at ModelScope for a ModelScope-linked model', () => {
+    const card = mountCard(
+      createModelCard,
+      makeModel({
+        from_civitai: false,
+        civitai: {},
+        source_platform: 'modelscope',
+        source_url: 'https://modelscope.cn/models/user/repo',
+      })
+    );
+
+    expect(card.dataset.has_civitai).toBe('false');
+    expect(card.dataset.source_platform).toBe('modelscope');
+    expect(card.dataset.hf_url).toBe('');
+    expect(card.querySelector('.fa-globe').getAttribute('title')).toBe('View on ModelScope');
+  });
+
+  it('opens the ModelScope page when the globe is clicked', () => {
+    const openSpy = vi.spyOn(window, 'open').mockImplementation(() => {});
+    const card = mountCard(
+      createModelCard,
+      makeModel({
+        from_civitai: false,
+        civitai: {},
+        source_platform: 'modelscope',
+        source_url: 'https://modelscope.cn/models/user/repo',
+      })
+    );
+    setupModelCardEventDelegation('loras');
+
+    card.querySelector('.fa-globe').dispatchEvent(new MouseEvent('click', { bubbles: true }));
+
+    expect(openSpy).toHaveBeenCalledWith(
+      'https://modelscope.cn/models/user/repo',
+      '_blank',
+      'noopener,noreferrer'
+    );
+    expect(openCivitai).not.toHaveBeenCalled();
+    expect(openHuggingFace).not.toHaveBeenCalled();
+    openSpy.mockRestore();
+  });
+
+  it('points the globe at TensorArt for a TensorArt-linked model', () => {
+    const card = mountCard(
+      createModelCard,
+      makeModel({
+        from_civitai: false,
+        civitai: {},
+        source_platform: 'tensorart',
+        source_url: 'https://tensor.art/models/827823520299086029',
+      })
+    );
+
+    expect(card.querySelector('.fa-globe').getAttribute('title')).toBe('View on TensorArt');
+  });
 });
@@ -1,4 +1,4 @@
-import { describe, expect, it } from 'vitest';
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';

 import { ModelContextMenuMixin } from '../../../static/js/components/ContextMenu/ModelContextMenuMixin.js';

@@ -14,3 +14,115 @@ describe('ModelContextMenuMixin.getModelTypePrefix', () => {
    expect(ModelContextMenuMixin.getModelTypePrefix.call({})).toBe('loras');
  });
 });
+
+describe('ModelContextMenuMixin.updateEnrichMenuItem', () => {
+  function setupMenu() {
+    document.body.innerHTML = '<div id="menu"><div data-action="enrich-hf-llm"></div></div>';
+    return { menu: document.getElementById('menu') };
+  }
+
+  function cardWith(dataset) {
+    return { dataset };
+  }
+
+  it('enables enrichment for Hugging Face links', () => {
+    const context = setupMenu();
+    ModelContextMenuMixin.updateEnrichMenuItem.call(
+      context,
+      cardWith({ hf_url: 'https://huggingface.co/user/repo' })
+    );
+
+    const item = context.menu.querySelector('[data-action="enrich-hf-llm"]');
+    expect(item.classList.contains('disabled')).toBe(false);
+    expect(item.title).toBe('');
+  });
+
+  it('enables enrichment for ModelScope links', () => {
+    const context = setupMenu();
+    ModelContextMenuMixin.updateEnrichMenuItem.call(
+      context,
+      cardWith({
+        source_platform: 'modelscope',
+        source_url: 'https://modelscope.cn/models/user/repo',
+      })
+    );
+
+    const item = context.menu.querySelector('[data-action="enrich-hf-llm"]');
+    expect(item.classList.contains('disabled')).toBe(false);
+  });
+
+  it('disables enrichment for TensorArt and explains why', () => {
+    const context = setupMenu();
+    ModelContextMenuMixin.updateEnrichMenuItem.call(
+      context,
+      cardWith({
+        source_platform: 'tensorart',
+        source_url: 'https://tensor.art/models/827823520299086029',
+      })
+    );
+
+    const item = context.menu.querySelector('[data-action="enrich-hf-llm"]');
+    expect(item.classList.contains('disabled')).toBe(true);
+    expect(item.title).toContain('TensorArt');
+  });
+
+  it('disables enrichment when no source is linked', () => {
+    const context = setupMenu();
+    ModelContextMenuMixin.updateEnrichMenuItem.call(context, cardWith({}));
+
+    const item = context.menu.querySelector('[data-action="enrich-hf-llm"]');
+    expect(item.classList.contains('disabled')).toBe(true);
+    expect(item.title).toContain('Link this model to a model source');
+  });
+});
+
+describe('ModelContextMenuMixin._renderSupportedSources', () => {
+  const originalFetch = global.fetch;
+
+  beforeEach(() => {
+    document.body.innerHTML = '<div id="hfSupportedSources">static fallback</div>';
+  });
+
+  afterEach(() => {
+    global.fetch = originalFetch;
+  });
+
+  it('renders the server-provided example URLs', async () => {
+    global.fetch = vi.fn().mockResolvedValue({
+      ok: true,
+      json: async () => [
+        { platform: 'huggingface', example_url: 'https://huggingface.co/user/repo' },
+        { platform: 'modelscope', example_url: 'https://modelscope.cn/models/user/repo' },
+        { platform: 'tensorart', example_url: 'https://tensor.art/models/123' },
+      ],
+    });
+
+    await ModelContextMenuMixin._renderSupportedSources.call({});
+
+    const html = document.getElementById('hfSupportedSources').innerHTML;
+    expect(html).toContain('https://huggingface.co/user/repo');
+    expect(html).toContain('https://modelscope.cn/models/user/repo');
+    expect(html).toContain('https://tensor.art/models/123');
+  });
+
+  it('keeps the static fallback when the request fails', async () => {
+    global.fetch = vi.fn().mockRejectedValue(new Error('offline'));
+
+    await ModelContextMenuMixin._renderSupportedSources.call({});
+
+    expect(document.getElementById('hfSupportedSources').innerHTML).toBe('static fallback');
+  });
+
+  it('escapes markup from the server payload', async () => {
+    global.fetch = vi.fn().mockResolvedValue({
+      ok: true,
+      json: async () => [{ example_url: '<img src=x onerror=alert(1)>' }],
+    });
+
+    await ModelContextMenuMixin._renderSupportedSources.call({});
+
+    const html = document.getElementById('hfSupportedSources').innerHTML;
+    expect(html).not.toContain('<img');
+    expect(html).toContain('&lt;img');
+  });
+});