mirror of
https://github.com/willmiao/ComfyUI-Lora-Manager.git
synced 2026-09-20 18:51:26 -03:00
feat(links): support ModelScope and TensorArt as model sources
A model file could only ever be linked to huggingface.co: `set_hf_url` validated the URL with a huggingface-only regex, the agent fetched the card from a hardcoded HF URL, and the readme processor built every relative image path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the same model-card convention (README.md + YAML frontmatter, often carrying `base_model:` and `trigger_words:`) behind a public, key-less API, so the enrichment pipeline could already serve it - it was the plumbing that was HF-shaped, not the idea. Make the external source a first-class, provider-driven concept: - New `py/services/model_sources/` registry. A `ModelSource` owns URL recognition (lenient for stored values, strict for user input), the canonical page URL, model-card fetching, the asset base URL and the capability flags. `HuggingFaceSource` is the previous logic relocated; `ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md` and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is link-only on purpose: tensor.art answers plain HTTP clients with a Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud / cn.tensorart.net) rejects every /v1/model/* route with "invalid authorization header", so it declares supports_enrichment=False rather than failing silently later. - Metadata gains `source_platform` + `source_url`; `hf_url` stays as a read/write alias, written only for Hugging Face, so existing sidecars, cached rows and third-party consumers keep working. Normalisation runs at the scanner, the persistent cache (both directions, plus two new columns behind an ALTER migration) and the linking handler - which is what stops a user who switches sources from leaving a stale `hf_url` on a ModelScope model. - The agent pipeline keys off the provider instead of `hf_url`: the fast-fail gate now explains *why* a model is skipped (no source / unknown source / source without a reachable card), the prompt context exposes source_url/source_id/source_label/asset_base_url while still filling the legacy hf_url/repo aliases, and the four README image extractors take a base_url (defaulting to HF) so relative paths resolve against the right site. Version grouping generalises to hf: / ms: / ta: keys. - `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but accepts `source_url`, validates against every provider and returns the platform. `GET /api/lm/model-sources` lets the UI render the supported-site list from the server. - Frontend: a `modelSourceHelpers` mirror of the registry drives the link dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the version-group key and the enrichment gate; the versions tab no longer sends ms:/ta: keys to the CivitAI API. TensorArt stays in the list because provenance is worth keeping even when the card is unreadable - the dialog says so plainly ("Sites that don't expose one (currently TensorArt) can only be linked") and the context menu disables enrichment with a matching tooltip, instead of the user getting "Unsupported URL". Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a 1882-byte card whose frontmatter carries base_model/tags/trigger_words, and relative images resolve to .../resolve/master/.... Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest tests/i18n and a Jinja compile pass over templates/. The nine locales carry [TODO: Translate] for the new strings, completed in the next commit.
This commit is contained in:
@@ -0,0 +1,61 @@
|
||||
"""External model-source providers (Hugging Face, ModelScope, TensorArt).
|
||||
|
||||
This package is the single abstraction over "a site that hosts models and
|
||||
a model card". See :mod:`py.services.model_sources.base` for the provider
|
||||
protocol and :mod:`py.services.model_sources.registry` for the lookup and
|
||||
metadata-normalisation helpers used across the codebase.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from .base import (
|
||||
GROUP_PREFIXES,
|
||||
HTTP_TIMEOUT,
|
||||
ModelSource,
|
||||
SourceRef,
|
||||
USER_AGENT,
|
||||
clean_source_url,
|
||||
fetch_text,
|
||||
)
|
||||
from .huggingface import HuggingFaceSource
|
||||
from .modelscope import ModelScopeSource
|
||||
from .registry import (
|
||||
LEGACY_HF_URL_FIELD,
|
||||
SOURCE_PLATFORM_FIELD,
|
||||
SOURCE_URL_FIELD,
|
||||
detect_source,
|
||||
get_source,
|
||||
get_source_platform,
|
||||
has_external_source,
|
||||
list_sources,
|
||||
normalize_metadata_source,
|
||||
resolve_source_ref,
|
||||
source_group_key,
|
||||
source_label,
|
||||
)
|
||||
from .tensorart import TensorArtSource
|
||||
|
||||
__all__ = [
|
||||
"GROUP_PREFIXES",
|
||||
"HTTP_TIMEOUT",
|
||||
"LEGACY_HF_URL_FIELD",
|
||||
"ModelSource",
|
||||
"HuggingFaceSource",
|
||||
"ModelScopeSource",
|
||||
"SOURCE_PLATFORM_FIELD",
|
||||
"SOURCE_URL_FIELD",
|
||||
"SourceRef",
|
||||
"TensorArtSource",
|
||||
"USER_AGENT",
|
||||
"clean_source_url",
|
||||
"detect_source",
|
||||
"fetch_text",
|
||||
"get_source",
|
||||
"get_source_platform",
|
||||
"has_external_source",
|
||||
"list_sources",
|
||||
"normalize_metadata_source",
|
||||
"resolve_source_ref",
|
||||
"source_group_key",
|
||||
"source_label",
|
||||
]
|
||||
@@ -0,0 +1,186 @@
|
||||
"""Base types for the external model-source provider abstraction.
|
||||
|
||||
A *model source* is a third-party site that hosts model files and a model
|
||||
card (README) describing them — Hugging Face, ModelScope, TensorArt, and
|
||||
whatever gets added later. Everything the rest of the codebase needs to
|
||||
know about such a site is expressed by :class:`ModelSource`:
|
||||
|
||||
* how to recognise one of its URLs (:meth:`ModelSource.parse`)
|
||||
* the canonical page URL for a source id (:meth:`ModelSource.canonical_url`)
|
||||
* how to fetch the model card (:meth:`ModelSource.fetch_model_card`)
|
||||
* how to turn repository-relative asset paths into absolute URLs
|
||||
(:meth:`ModelSource.asset_base_url`)
|
||||
* which capabilities the site actually supports
|
||||
(``supports_enrichment`` / ``supports_download``)
|
||||
|
||||
Keeping this in one place means the agent pipeline, the scanners, and the
|
||||
HTTP handlers never need site-specific branching.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Optional
|
||||
|
||||
import aiohttp
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: Shared HTTP timeout for model-card fetches.
|
||||
HTTP_TIMEOUT = 30
|
||||
|
||||
#: User agent used for all model-source HTTP requests.
|
||||
USER_AGENT = "ComfyUI-LoRA-Manager/1.0"
|
||||
|
||||
#: Platform → short prefix used when building version-group keys.
|
||||
#: ``huggingface`` keeps the historical ``hf:`` prefix for backward
|
||||
#: compatibility with already-cached group keys.
|
||||
GROUP_PREFIXES: dict[str, str] = {
|
||||
"huggingface": "hf",
|
||||
"modelscope": "ms",
|
||||
"tensorart": "ta",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SourceRef:
|
||||
"""A parsed reference to a model hosted on an external site."""
|
||||
|
||||
platform: str
|
||||
"""Canonical platform id, e.g. ``"huggingface"``."""
|
||||
|
||||
source_id: str
|
||||
"""Site-specific identity, e.g. ``"user/repo"`` or ``"827823520299086029"``."""
|
||||
|
||||
url: str
|
||||
"""Canonical URL of the model page."""
|
||||
|
||||
|
||||
async def fetch_text(url: str, *, timeout: int = HTTP_TIMEOUT) -> str:
|
||||
"""Fetch *url* and return its body as text, or ``""`` on any failure.
|
||||
|
||||
Network problems are expected (offline installs, rate limits, dead
|
||||
repos) and must never bubble up into the pipeline, so every error is
|
||||
logged at debug level and normalised to an empty string.
|
||||
"""
|
||||
|
||||
try:
|
||||
async with aiohttp.ClientSession(
|
||||
headers={"User-Agent": USER_AGENT},
|
||||
timeout=aiohttp.ClientTimeout(total=timeout),
|
||||
) as session:
|
||||
async with session.get(url) as resp:
|
||||
if resp.status == 200:
|
||||
return await resp.text()
|
||||
logger.debug("Fetch %s returned HTTP %s", url, resp.status)
|
||||
except Exception as exc: # pragma: no cover - network dependent
|
||||
logger.debug("Failed to fetch %s: %s", url, exc)
|
||||
return ""
|
||||
|
||||
|
||||
class ModelSource:
|
||||
"""Description and I/O for one external model hosting site."""
|
||||
|
||||
#: Canonical platform id stored in metadata.
|
||||
platform: str = ""
|
||||
|
||||
#: Human-readable name used in UI copy and prompts.
|
||||
label: str = ""
|
||||
|
||||
#: Whether the agent skill can fetch a model card and run AI extraction.
|
||||
supports_enrichment: bool = False
|
||||
|
||||
#: Whether models can be downloaded directly from this site.
|
||||
supports_download: bool = False
|
||||
|
||||
#: Lenient pattern used to recognise URLs already stored in metadata.
|
||||
#: Captures the site-specific source id in group ``id``.
|
||||
url_pattern: re.Pattern[str] | None = None
|
||||
|
||||
#: Strict pattern used to validate user input. Must match the whole URL.
|
||||
strict_url_pattern: re.Pattern[str] | None = None
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Parsing
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def parse(self, url: str, *, strict: bool = False) -> Optional[str]:
|
||||
"""Return the source id contained in *url*, or ``None``.
|
||||
|
||||
With ``strict=True`` the URL must match this site's canonical shape
|
||||
exactly (used when validating what a user pasted); with
|
||||
``strict=False`` sub-paths such as ``/resolve/main/file.bin`` are
|
||||
tolerated (used when normalising already-stored values).
|
||||
"""
|
||||
|
||||
if not url or not isinstance(url, str):
|
||||
return None
|
||||
candidate = url.strip()
|
||||
if not candidate:
|
||||
return None
|
||||
pattern = self.strict_url_pattern if strict else self.url_pattern
|
||||
if pattern is None:
|
||||
return None
|
||||
match = pattern.match(candidate)
|
||||
return match.group("id") if match else None
|
||||
|
||||
def ref(self, url: str, *, strict: bool = False) -> Optional[SourceRef]:
|
||||
"""Return a :class:`SourceRef` for *url*, or ``None`` if not ours."""
|
||||
|
||||
source_id = self.parse(url, strict=strict)
|
||||
if not source_id:
|
||||
return None
|
||||
return SourceRef(
|
||||
platform=self.platform,
|
||||
source_id=source_id,
|
||||
url=self.canonical_url(source_id),
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# URLs and content
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def canonical_url(self, source_id: str) -> str:
|
||||
"""Return the canonical model-page URL for *source_id*."""
|
||||
|
||||
raise NotImplementedError
|
||||
|
||||
def asset_base_url(self, source_id: str, revision: str = "") -> str:
|
||||
"""Base URL used to resolve repository-relative asset paths."""
|
||||
|
||||
return ""
|
||||
|
||||
def group_key(self, source_id: str) -> str:
|
||||
"""Return the version-group key for *source_id*."""
|
||||
|
||||
prefix = GROUP_PREFIXES.get(self.platform, self.platform)
|
||||
return f"{prefix}:{source_id}"
|
||||
|
||||
async def fetch_model_card(self, source_id: str) -> str:
|
||||
"""Fetch the raw model card (README) markdown for *source_id*."""
|
||||
|
||||
return ""
|
||||
|
||||
def __repr__(self) -> str: # pragma: no cover - debugging aid
|
||||
return f"<ModelSource {self.platform}>"
|
||||
|
||||
|
||||
def clean_source_url(url: Any) -> str:
|
||||
"""Normalise a stored source URL value into a stripped string."""
|
||||
|
||||
if not isinstance(url, str):
|
||||
return ""
|
||||
return url.strip()
|
||||
|
||||
|
||||
__all__ = [
|
||||
"GROUP_PREFIXES",
|
||||
"HTTP_TIMEOUT",
|
||||
"ModelSource",
|
||||
"SourceRef",
|
||||
"USER_AGENT",
|
||||
"clean_source_url",
|
||||
"fetch_text",
|
||||
]
|
||||
@@ -0,0 +1,49 @@
|
||||
"""Hugging Face model source."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
from .base import ModelSource, fetch_text
|
||||
|
||||
#: Lenient — used to normalise URLs already stored in metadata; tolerates
|
||||
#: sub-paths such as ``/resolve/main/model.safetensors``.
|
||||
_URL_PATTERN = re.compile(
|
||||
r"https?://(?:www\.)?huggingface\.co/(?P<id>[^/?#\s]+/[^/?#\s]+)"
|
||||
)
|
||||
|
||||
#: Strict — validates what the user pasted into the "link model" dialog.
|
||||
_STRICT_URL_PATTERN = re.compile(
|
||||
r"https?://(?:www\.)?huggingface\.co/(?P<id>[^/?#\s]+/[^/?#\s]+)/?$"
|
||||
)
|
||||
|
||||
|
||||
class HuggingFaceSource(ModelSource):
|
||||
"""Hugging Face Hub (``huggingface.co``)."""
|
||||
|
||||
platform = "huggingface"
|
||||
label = "Hugging Face"
|
||||
supports_enrichment = True
|
||||
supports_download = True
|
||||
url_pattern = _URL_PATTERN
|
||||
strict_url_pattern = _STRICT_URL_PATTERN
|
||||
|
||||
def canonical_url(self, source_id: str) -> str:
|
||||
return f"https://huggingface.co/{source_id}"
|
||||
|
||||
def asset_base_url(self, source_id: str, revision: str = "") -> str:
|
||||
return f"https://huggingface.co/{source_id}/resolve/{revision or 'main'}"
|
||||
|
||||
async def fetch_model_card(self, source_id: str) -> str:
|
||||
"""Fetch ``README.md`` from Hugging Face (tries ``main``, then ``master``)."""
|
||||
|
||||
for branch in ("main", "master"):
|
||||
text = await fetch_text(
|
||||
f"https://huggingface.co/{source_id}/raw/{branch}/README.md"
|
||||
)
|
||||
if text:
|
||||
return text
|
||||
return ""
|
||||
|
||||
|
||||
__all__ = ["HuggingFaceSource"]
|
||||
@@ -0,0 +1,76 @@
|
||||
"""ModelScope (魔搭社区) model source.
|
||||
|
||||
ModelScope exposes the same "model card as README.md" convention as
|
||||
Hugging Face, including a YAML frontmatter block that often carries
|
||||
``base_model:`` and ``trigger_words:``. Two public endpoints are used,
|
||||
neither of which requires an API key for public models:
|
||||
|
||||
* ``/models/{owner}/{name}/resolve/{revision}/README.md`` — raw model card
|
||||
* ``/api/v1/models/{owner}/{name}/repo?Revision=..&FilePath=README.md`` —
|
||||
the same content through the API, used as a fallback when the resolve
|
||||
URL is unavailable.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
from .base import ModelSource, fetch_text
|
||||
|
||||
_URL_PATTERN = re.compile(
|
||||
r"https?://(?:www\.)?modelscope\.(?:cn|com)/models/(?P<id>[^/?#\s]+/[^/?#\s]+)"
|
||||
)
|
||||
|
||||
#: Trailing view segments the site appends to a model URL; accepted verbatim
|
||||
#: when the user pastes a browser tab URL.
|
||||
_VIEW_SEGMENTS = r"(?:summary|files|model-file|readme|community|evaluation)?"
|
||||
|
||||
_STRICT_URL_PATTERN = re.compile(
|
||||
r"https?://(?:www\.)?modelscope\.(?:cn|com)/models/(?P<id>[^/?#\s]+/[^/?#\s]+)"
|
||||
rf"/?{_VIEW_SEGMENTS}/?$"
|
||||
)
|
||||
|
||||
#: ``master`` is ModelScope's default branch; ``main`` is tried as a fallback
|
||||
#: for repos imported from Hugging Face.
|
||||
_REVISIONS = ("master", "main")
|
||||
|
||||
|
||||
class ModelScopeSource(ModelSource):
|
||||
"""ModelScope (``modelscope.cn``)."""
|
||||
|
||||
platform = "modelscope"
|
||||
label = "ModelScope"
|
||||
supports_enrichment = True
|
||||
supports_download = False
|
||||
url_pattern = _URL_PATTERN
|
||||
strict_url_pattern = _STRICT_URL_PATTERN
|
||||
|
||||
def canonical_url(self, source_id: str) -> str:
|
||||
return f"https://modelscope.cn/models/{source_id}"
|
||||
|
||||
def asset_base_url(self, source_id: str, revision: str = "") -> str:
|
||||
return f"https://modelscope.cn/models/{source_id}/resolve/{revision or 'master'}"
|
||||
|
||||
async def fetch_model_card(self, source_id: str) -> str:
|
||||
"""Fetch the model card, preferring the raw resolve URL."""
|
||||
|
||||
for revision in _REVISIONS:
|
||||
text = await fetch_text(
|
||||
f"https://modelscope.cn/models/{source_id}/resolve/{revision}/README.md"
|
||||
)
|
||||
if text:
|
||||
return text
|
||||
|
||||
# Fallback: the repo API proxies the same file and is reachable in
|
||||
# environments where the CDN resolve host is blocked.
|
||||
for revision in _REVISIONS:
|
||||
text = await fetch_text(
|
||||
"https://modelscope.cn/api/v1/models/"
|
||||
f"{source_id}/repo?Revision={revision}&FilePath=README.md"
|
||||
)
|
||||
if text:
|
||||
return text
|
||||
return ""
|
||||
|
||||
|
||||
__all__ = ["ModelScopeSource"]
|
||||
@@ -0,0 +1,208 @@
|
||||
"""Registry and metadata helpers for external model sources.
|
||||
|
||||
The registry is the single place the rest of the codebase asks "which site
|
||||
is this URL from?", "what is this model's source?", and "can we enrich it?".
|
||||
Import from :mod:`py.services.model_sources` rather than this module
|
||||
directly.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any, Dict, Mapping, Optional
|
||||
|
||||
from .base import GROUP_PREFIXES, ModelSource, SourceRef, clean_source_url
|
||||
from .huggingface import HuggingFaceSource
|
||||
from .modelscope import ModelScopeSource
|
||||
from .tensorart import TensorArtSource
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: Order matters only for disambiguation; the URL patterns are disjoint.
|
||||
_SOURCES: tuple[ModelSource, ...] = (
|
||||
HuggingFaceSource(),
|
||||
ModelScopeSource(),
|
||||
TensorArtSource(),
|
||||
)
|
||||
|
||||
_BY_PLATFORM: Dict[str, ModelSource] = {s.platform: s for s in _SOURCES}
|
||||
|
||||
#: Metadata keys that carry the canonical external-source identity.
|
||||
SOURCE_PLATFORM_FIELD = "source_platform"
|
||||
SOURCE_URL_FIELD = "source_url"
|
||||
#: Legacy field kept as a read/write alias for Hugging Face models so that
|
||||
#: older sidecars, cached rows, and third-party consumers keep working.
|
||||
LEGACY_HF_URL_FIELD = "hf_url"
|
||||
|
||||
|
||||
def list_sources() -> list[ModelSource]:
|
||||
"""Return every known model source."""
|
||||
|
||||
return list(_SOURCES)
|
||||
|
||||
|
||||
def get_source(platform: Optional[str]) -> Optional[ModelSource]:
|
||||
"""Return the source registered for *platform*, or ``None``."""
|
||||
|
||||
if not platform or not isinstance(platform, str):
|
||||
return None
|
||||
return _BY_PLATFORM.get(platform.strip().lower())
|
||||
|
||||
|
||||
def source_label(platform: Optional[str], default: str = "") -> str:
|
||||
"""Return the human-readable label for *platform*."""
|
||||
|
||||
source = get_source(platform)
|
||||
return source.label if source else default
|
||||
|
||||
|
||||
def detect_source(url: Optional[str], *, strict: bool = False) -> Optional[SourceRef]:
|
||||
"""Return the :class:`SourceRef` for *url*, or ``None`` if unsupported."""
|
||||
|
||||
if not url or not isinstance(url, str):
|
||||
return None
|
||||
for source in _SOURCES:
|
||||
ref = source.ref(url, strict=strict)
|
||||
if ref is not None:
|
||||
return ref
|
||||
return None
|
||||
|
||||
|
||||
def resolve_source_ref(metadata: Mapping[str, Any]) -> Optional[SourceRef]:
|
||||
"""Return the source reference described by a model's metadata.
|
||||
|
||||
Handles all three storage states found in the wild:
|
||||
|
||||
1. ``source_url`` + ``source_platform`` (current format)
|
||||
2. ``hf_url`` only (legacy Hugging Face storage)
|
||||
3. ``hf_url`` plus a newer ``source_url`` (both written by older builds)
|
||||
"""
|
||||
|
||||
if not isinstance(metadata, Mapping):
|
||||
return None
|
||||
|
||||
platform = clean_source_url(metadata.get(SOURCE_PLATFORM_FIELD)).lower()
|
||||
url = clean_source_url(metadata.get(SOURCE_URL_FIELD))
|
||||
legacy = clean_source_url(metadata.get(LEGACY_HF_URL_FIELD))
|
||||
|
||||
source = get_source(platform)
|
||||
if url:
|
||||
if source is not None:
|
||||
ref = source.ref(url)
|
||||
if ref is not None:
|
||||
return ref
|
||||
ref = detect_source(url)
|
||||
if ref is not None:
|
||||
return ref
|
||||
# Unknown platform but a URL is present: keep it addressable.
|
||||
return SourceRef(platform=platform or "unknown", source_id="", url=url)
|
||||
|
||||
if legacy:
|
||||
return detect_source(legacy)
|
||||
return None
|
||||
|
||||
|
||||
def normalize_metadata_source(metadata: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Normalise the external-source fields on *metadata* in place.
|
||||
|
||||
Guarantees that ``source_url``/``source_platform`` are present and
|
||||
consistent, and that ``hf_url`` mirrors ``source_url`` for Hugging Face
|
||||
models (never for other platforms, so a stale alias can't make a
|
||||
ModelScope model look like a Hugging Face one).
|
||||
|
||||
Returns the same dict for convenient chaining.
|
||||
"""
|
||||
|
||||
if not isinstance(metadata, dict):
|
||||
return metadata
|
||||
|
||||
platform = clean_source_url(metadata.get(SOURCE_PLATFORM_FIELD)).lower()
|
||||
url = clean_source_url(metadata.get(SOURCE_URL_FIELD))
|
||||
legacy = clean_source_url(metadata.get(LEGACY_HF_URL_FIELD))
|
||||
|
||||
source = get_source(platform)
|
||||
ref: Optional[SourceRef] = None
|
||||
|
||||
if url:
|
||||
ref = source.ref(url) if source is not None else None
|
||||
if ref is None:
|
||||
ref = detect_source(url)
|
||||
elif legacy:
|
||||
ref = detect_source(legacy)
|
||||
|
||||
if ref is not None and ref.source_id:
|
||||
platform = ref.platform
|
||||
url = ref.url or url
|
||||
|
||||
if platform:
|
||||
metadata[SOURCE_PLATFORM_FIELD] = platform
|
||||
else:
|
||||
metadata.setdefault(SOURCE_PLATFORM_FIELD, "")
|
||||
|
||||
metadata[SOURCE_URL_FIELD] = url
|
||||
|
||||
# Keep the legacy alias in sync, but only for Hugging Face.
|
||||
if url and platform == "huggingface":
|
||||
metadata[LEGACY_HF_URL_FIELD] = url
|
||||
elif LEGACY_HF_URL_FIELD in metadata and platform and platform != "huggingface":
|
||||
metadata[LEGACY_HF_URL_FIELD] = ""
|
||||
elif legacy and not url:
|
||||
metadata[LEGACY_HF_URL_FIELD] = legacy
|
||||
|
||||
return metadata
|
||||
|
||||
|
||||
def has_external_source(item: Mapping[str, Any]) -> bool:
|
||||
"""Return ``True`` when *item* is linked to any external model site."""
|
||||
|
||||
if not isinstance(item, Mapping):
|
||||
return False
|
||||
return bool(
|
||||
clean_source_url(item.get(SOURCE_URL_FIELD))
|
||||
or clean_source_url(item.get(LEGACY_HF_URL_FIELD))
|
||||
)
|
||||
|
||||
|
||||
def get_source_platform(item: Mapping[str, Any]) -> str:
|
||||
"""Return the platform id stored on *item* (may be empty)."""
|
||||
|
||||
if not isinstance(item, Mapping):
|
||||
return ""
|
||||
platform = clean_source_url(item.get(SOURCE_PLATFORM_FIELD)).lower()
|
||||
if platform:
|
||||
return platform
|
||||
ref = resolve_source_ref(item)
|
||||
return ref.platform if ref else ""
|
||||
|
||||
|
||||
def source_group_key(item: Mapping[str, Any]) -> Optional[str]:
|
||||
"""Return the version-group key for *item*, or ``None``.
|
||||
|
||||
Hugging Face keeps the historical ``hf:{owner}/{repo}`` shape; other
|
||||
platforms use their own short prefix (see :data:`GROUP_PREFIXES`).
|
||||
"""
|
||||
|
||||
ref = resolve_source_ref(item)
|
||||
if ref is None or not ref.source_id:
|
||||
return None
|
||||
source = get_source(ref.platform)
|
||||
if source is None:
|
||||
return None
|
||||
return source.group_key(ref.source_id)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"GROUP_PREFIXES",
|
||||
"LEGACY_HF_URL_FIELD",
|
||||
"SOURCE_PLATFORM_FIELD",
|
||||
"SOURCE_URL_FIELD",
|
||||
"detect_source",
|
||||
"get_source",
|
||||
"get_source_platform",
|
||||
"has_external_source",
|
||||
"list_sources",
|
||||
"normalize_metadata_source",
|
||||
"resolve_source_ref",
|
||||
"source_group_key",
|
||||
"source_label",
|
||||
]
|
||||
@@ -0,0 +1,56 @@
|
||||
"""TensorArt model source (link / provenance only).
|
||||
|
||||
TensorArt support is intentionally limited to *linking* a model to its
|
||||
TensorArt page. Automatic metadata extraction is not possible without a
|
||||
user session:
|
||||
|
||||
* ``tensor.art`` sits behind a Cloudflare managed challenge, so plain
|
||||
HTTP clients (aiohttp, requests, curl) receive ``403 "Just a moment..."``.
|
||||
* Its internal API (``ap-east-1.tensorart.cloud`` / ``cn.tensorart.net``)
|
||||
answers every ``/v1/model/*`` route with
|
||||
``{"code":100002,"message":"invalid authorization header"}``.
|
||||
* The official TAMS API requires an AccessKey/SecretKey pair and request
|
||||
signatures, which is a poor fit for a "paste a URL" workflow.
|
||||
|
||||
``supports_enrichment`` is therefore ``False``: the agent pipeline skips
|
||||
these models with an explicit reason instead of failing silently, and the
|
||||
UI keeps showing the "View on TensorArt" link. ``tusi.cn`` is TensorArt's
|
||||
Chinese mirror and is accepted as the same platform.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
from .base import ModelSource
|
||||
|
||||
_DOMAINS = r"(?:tensor\.art|tusi\.cn)"
|
||||
|
||||
_URL_PATTERN = re.compile(
|
||||
rf"https?://(?:www\.)?{_DOMAINS}/models/(?P<id>\d+)"
|
||||
)
|
||||
|
||||
_STRICT_URL_PATTERN = re.compile(
|
||||
rf"https?://(?:www\.)?{_DOMAINS}/models/(?P<id>\d+)(?:/[^/?#\s]+)?/?$"
|
||||
)
|
||||
|
||||
|
||||
class TensorArtSource(ModelSource):
|
||||
"""TensorArt (``tensor.art``)."""
|
||||
|
||||
platform = "tensorart"
|
||||
label = "TensorArt"
|
||||
supports_enrichment = False
|
||||
supports_download = False
|
||||
url_pattern = _URL_PATTERN
|
||||
strict_url_pattern = _STRICT_URL_PATTERN
|
||||
|
||||
def canonical_url(self, source_id: str) -> str:
|
||||
return f"https://tensor.art/models/{source_id}"
|
||||
|
||||
def asset_base_url(self, source_id: str, revision: str = "") -> str:
|
||||
# Unreachable today: enrichment is disabled for this platform.
|
||||
return f"https://tensor.art/models/{source_id}"
|
||||
|
||||
|
||||
__all__ = ["TensorArtSource"]
|
||||
Reference in New Issue
Block a user