feat(download): fill model metadata from the source API on download

A ModelScope or Hugging Face download landed as a bare filename, hash and
source link; the model card stayed empty until the user ran "Enrich
Metadata with AI" by hand. But everything that makes a CivitAI download
useful — the display name, the description, the tags, the trigger words,
the example images, the preview — is already published by those sites'
public APIs, so asking for it at download time is deterministic work, not
model work.

Add `py/services/model_sources/hydration.py`, called by
`_save_source_metadata()` once the sidecar exists and the file is in the
scanner cache. It fetches the model card plus the site's card extras and
hands them to the same `PostProcessor` the AI skill uses, with an empty
`llm_output`, so the two paths cannot drift apart. What lands:

* `model_name` from the site's own display name (ModelScope's `Name`), so
  the card stops showing the local filename — written only while the value
  still equals the file stem, since once a user renames a model that
  choice is theirs to keep
* `civitai.name` from the matched version's label (`showName`), which the
  card renders as the version chip
* `civitai.description` / `modelDescription` from the author summary plus
  the README as HTML
* `civitai.images` / `preview_url` from the per-file example images
* `civitai.trainedWords` from the per-file trigger words
* `base_model`, `tags` and `usage_tips` as before

Provenance stays honest: the pass records
`metadata_source = "source:<platform>"` rather than the skill's
`agent:enrich_hf_metadata`, and — because no provider ran — it no longer
stamps `llm_enriched_at`; that stamp is now conditional on the LLM
actually answering, which is what the field means. The five hand-rolled
`civitai` dict merges in the post-processor collapse into one
`_merge_civitai()` helper.

Two guards keep it safe. Only a model whose stored
`source_platform`/`source_url` match the repository being downloaded is
updated, so a local file that merely shares a name never receives another
model's card; and a file already on disk is topped up too, which
back-fills models downloaded before this existed. READMEs and detail
payloads describe the repository rather than the file, so a short-lived
process-wide `ModelSourceCache` (300 s, 32 entries) keeps a batch over one
repository to two HTTP requests. Every failure is logged and swallowed:
hydration can never fail a download.

Fix the hash policy while here. `_save_source_metadata()` went straight to
`MetadataManager.create_default_metadata()`, bypassing the per-type
factory on the owning scanner, so a checkpoint paid a full SHA256 inside
the download request — `CheckpointScanner`/`OtherScanner` deliberately
record `hash_status="pending"` with an empty `sha256` for their multi-GB
files. Metadata is now created through `scanner._create_default_metadata()`.
Hydration copes with the empty hash: `_matching_versions()` falls back to
the repository basename, which is exactly what the download just wrote.

Report both post-transfer stages, which advance no byte counter and so
read as a stall: the bar sat at 100% showing `0 B/s` for the seconds spent
hashing and fetching. `_report_phase()` broadcasts
`{"status": "metadata", "stage": "indexing" | "source", "platform": ...}`,
and `LoadingManager` names the stage in the status line (keeping the batch
position), retitles the item line, replaces the dead speed figure and runs
a sheen over the bar. `stage`/`platform` are machine-readable; the wording
is localised in the frontend.

Finally, `modelscope.ai` is its own catalogue rather than an alias of
`modelscope.cn` — `referall13/EM1` exists only on `.ai` and
`jj3550945163/Krea-2-LORA` only on `.cn` — so its URLs were rejected with
"Invalid model URL format". Register it as `ModelScopeIntlSource`
(`platform="modelscope-ai"`, `msai:` group prefix, its own default
download directory) and derive every URL either deployment builds from a
per-class `base_url`. `modelscope.com` stays an alias of `.cn`, which is
what it redirects to. The frontend source table, the link dialog hints and
the docs mirror the split.

Verified against the live APIs: both reported `.ai` repositories list
their files, read their READMEs and yield name / version / base model /
trigger words / example images. Backend 3092 passed; frontend 1259 JS +
91 Vue passed. The nine locales carry the new progress copy in the next
commit.
This commit is contained in:
Will Miao
2026-09-17 07:47:07 +08:00
parent 1b1a8d63db
commit d572292142
34 changed files with 2385 additions and 135 deletions
+169 -38
View File
@@ -1,4 +1,4 @@
"""ModelScope (魔搭社区) model source.
"""ModelScope (魔搭社区) model sources.
ModelScope exposes the same "model card as README.md" convention as
Hugging Face, including a YAML frontmatter block that often carries
@@ -10,11 +10,13 @@ none of which requires an API key for public models:
the same content through the API, used as a fallback when the resolve
URL is unavailable.
* ``/api/v1/models/{owner}/{name}`` — the model-detail payload behind the
model page. It carries the author's summary (``Description``), the
site-curated tags (``OfficialTags``), and, per published version, the
model filenames (``MuseInfo.versions[].stats.fileList``) together with
that file's example images (``coverImages``) and trigger words. See
:meth:`ModelScopeSource.fetch_model_card_context`.
model page. It carries the repository's display name (``Name`` /
``ChineseName``), the author's summary (``Description``), the license, the
AIGC type, the site tags (``OfficialTags``, falling back to ``Tags``), and,
per published version, the model filenames
(``MuseInfo.versions[].stats.fileList``) together with that version's label
(``modelVersion.showName``), example images (``coverImages``) and trigger
words. See :meth:`ModelScopeSource.fetch_model_card_context`.
* ``/api/v1/models/{owner}/{name}/repo/files?Revision=..`` — the file
listing backing the download picker. It reports real sizes for LFS
files (not the pointer size), so no extra HEAD request is needed.
@@ -28,6 +30,12 @@ valid; the CDN URL must never be cached.
The README and the detail payload both describe the whole repository rather
than one file, so a per-run ``ModelSourceCache`` keeps them from being read
again for every checkpoint of a collection repository.
Two deployments are served by this module. ``modelscope.cn`` (with
``modelscope.com`` as a redirect alias) and ``modelscope.ai`` are *separate
catalogues*, not mirrors, so they are registered as distinct sources:
:class:`ModelScopeSource` and :class:`ModelScopeIntlSource`. Every URL either
class builds is derived from its ``base_url``.
"""
from __future__ import annotations
@@ -36,7 +44,7 @@ import json
import logging
import os
import re
from typing import TYPE_CHECKING, Any, Optional
from typing import TYPE_CHECKING, Any, Iterable, Optional
from .base import (
ModelCardContext,
@@ -52,18 +60,28 @@ if TYPE_CHECKING: # pragma: no cover - typing only
logger = logging.getLogger(__name__)
_URL_PATTERN = re.compile(
r"https?://(?:www\.)?modelscope\.(?:cn|com)/models/(?P<id>[^/?#\s]+/[^/?#\s]+)"
)
#: ModelScope runs two independent catalogues. ``modelscope.com`` is a
#: redirect alias of the mainland site, but ``modelscope.ai`` is the
#: *international* deployment with its own repository catalogue — a repository
#: published on one is routinely absent from the other (``referall13/EM1``
#: exists only on ``.ai``, ``jj3550945163/Krea-2-LORA`` only on ``.cn``). The
#: host therefore decides which site, API and CDN a model belongs to, and the
#: two deployments are registered as separate sources rather than folded into
#: one id.
_MAINLAND_HOSTS = r"modelscope\.(?:cn|com)"
_INTERNATIONAL_HOSTS = r"modelscope\.ai"
#: Trailing view segments the site appends to a model URL; accepted verbatim
#: when the user pastes a browser tab URL.
_VIEW_SEGMENTS = r"(?:summary|files|model-file|readme|community|evaluation)?"
_STRICT_URL_PATTERN = re.compile(
r"https?://(?:www\.)?modelscope\.(?:cn|com)/models/(?P<id>[^/?#\s]+/[^/?#\s]+)"
rf"/?{_VIEW_SEGMENTS}/?$"
)
def _url_patterns(hosts: str) -> tuple[re.Pattern[str], re.Pattern[str]]:
"""Build the lenient and strict model-URL patterns for *hosts*."""
body = rf"https?://(?:www\.)?(?:{hosts})/models/(?P<id>[^/?#\s]+/[^/?#\s]+)"
return re.compile(body), re.compile(rf"{body}/?{_VIEW_SEGMENTS}/?$")
#: ``master`` is ModelScope's default branch; ``main`` is tried as a fallback
#: for repos imported from Hugging Face.
@@ -71,7 +89,12 @@ _REVISIONS = ("master", "main")
class ModelScopeSource(ModelSource):
"""ModelScope (``modelscope.cn``)."""
"""ModelScope's mainland site (``modelscope.cn``).
``modelscope.com`` is accepted as an alias of it. The international
deployment is :class:`ModelScopeIntlSource`; everything below is written in
terms of ``base_url`` so both share one implementation.
"""
platform = "modelscope"
label = "ModelScope"
@@ -79,15 +102,18 @@ class ModelScopeSource(ModelSource):
supports_download = True
default_revision = "master"
default_subdir = "modelscope"
url_pattern = _URL_PATTERN
strict_url_pattern = _STRICT_URL_PATTERN
#: Origin every outgoing URL is built from.
base_url = "https://modelscope.cn"
url_pattern, strict_url_pattern = _url_patterns(_MAINLAND_HOSTS)
def canonical_url(self, source_id: str) -> str:
return f"https://modelscope.cn/models/{source_id}"
return f"{self.base_url}/models/{source_id}"
def asset_base_url(self, source_id: str, revision: str = "") -> str:
return (
f"https://modelscope.cn/models/{source_id}/resolve/"
f"{self.base_url}/models/{source_id}/resolve/"
f"{self.resolve_revision(revision)}"
)
@@ -96,7 +122,7 @@ class ModelScopeSource(ModelSource):
for revision in _REVISIONS:
text = await fetch_text(
f"https://modelscope.cn/models/{source_id}/resolve/{revision}/README.md"
f"{self.base_url}/models/{source_id}/resolve/{revision}/README.md"
)
if text:
return text
@@ -105,7 +131,7 @@ class ModelScopeSource(ModelSource):
# environments where the CDN resolve host is blocked.
for revision in _REVISIONS:
text = await fetch_text(
"https://modelscope.cn/api/v1/models/"
f"{self.base_url}/api/v1/models/"
f"{source_id}/repo?Revision={revision}&FilePath=README.md"
)
if text:
@@ -158,7 +184,7 @@ class ModelScopeSource(ModelSource):
return cache.provider[cache_key]
status, payload = await fetch_json(
f"https://modelscope.cn/api/v1/models/{source_id}"
f"{self.base_url}/api/v1/models/{source_id}"
)
if status != 200 or not isinstance(payload, dict):
logger.debug(
@@ -185,7 +211,7 @@ class ModelScopeSource(ModelSource):
revision = self.resolve_revision(revision)
status, payload = await fetch_json(
"https://modelscope.cn/api/v1/models/"
f"{self.base_url}/api/v1/models/"
f"{source_id}/repo/files?Revision={revision}"
)
@@ -208,18 +234,37 @@ class ModelScopeSource(ModelSource):
self, source_id: str, filename: str, revision: str = ""
) -> str:
return (
f"https://modelscope.cn/models/{source_id}/resolve/"
f"{self.base_url}/models/{source_id}/resolve/"
f"{self.resolve_revision(revision)}/{filename}"
)
def page_url_for_file(self, source_id: str, filename: str) -> str:
return (
f"https://modelscope.cn/models/{source_id}/file/view/"
f"{self.base_url}/models/{source_id}/file/view/"
f"{self.default_revision}/{filename}"
)
__all__ = ["ModelScopeSource"]
class ModelScopeIntlSource(ModelScopeSource):
"""ModelScope's international site (``modelscope.ai``).
A separate catalogue rather than a mirror, so it is registered under its
own platform id: the two deployments must not share a version group, a
"use default paths" directory, or a stored ``source_url``. The detail API,
the file listing, the resolve URLs and the CDN redirect all behave exactly
like the mainland site, which is why every URL here is derived from
:attr:`base_url` instead of being duplicated.
"""
platform = "modelscope-ai"
label = "ModelScope (International)"
default_subdir = "modelscope-ai"
base_url = "https://www.modelscope.ai"
url_pattern, strict_url_pattern = _url_patterns(_INTERNATIONAL_HOSTS)
__all__ = ["ModelScopeIntlSource", "ModelScopeSource"]
# ---------------------------------------------------------------------------
@@ -229,6 +274,33 @@ __all__ = ["ModelScopeSource"]
#: Trigger-word values that mean "the author left this blank".
_EMPTY_TRIGGER_VALUES = frozenset({"none", "null", "n/a"})
#: Repository tags that only restate what the model *is* (its library, task or
#: framework) rather than what it depicts. ModelScope mixes both into the
#: plain ``Tags`` list, and a card tagged "lora" or "text-to-image" is noise.
_GENERIC_TAGS = frozenset(
{
"any-to-any",
"checkpoint",
"controlnet",
"diffusers",
"embedding",
"image-text-to-text",
"image-to-image",
"image-to-video",
"lora",
"lycoris",
"onnx",
"pytorch",
"safetensors",
"tensorflow",
"text-to-image",
"text-to-speech",
"text-to-video",
"textual-inversion",
"vae",
}
)
def _clean_text(value: Any) -> str:
"""Return a stripped string for *value*, or ``""`` for anything else."""
@@ -259,9 +331,13 @@ def _build_card_context(
context = ModelCardContext(
description=_clean_text(data.get("Description")),
model_name=_clean_text(data.get("Name")),
model_name_localized=_clean_text(data.get("ChineseName")),
license=_clean_text(data.get("License")),
model_type=_clean_text(data.get("AigcType")),
base_model=_first_string(data.get("BaseModel")),
base_model_aliases=_base_model_aliases(data),
official_tags=_official_tags(data.get("OfficialTags")),
official_tags=_official_tags(data),
)
versions = _matching_versions(
@@ -271,6 +347,7 @@ def _build_card_context(
sha256=sha256,
)
if versions:
context.version_name = _version_label(versions)
context.example_images = _cover_image_urls(versions)
context.trigger_words = _version_trigger_words(versions)
return context
@@ -303,26 +380,63 @@ def _base_model_aliases(data: dict[str, Any]) -> list[str]:
return aliases
def _official_tags(value: Any) -> list[str]:
"""Extract the site-curated tag values from ``OfficialTags``.
def _official_tags(data: dict[str, Any]) -> list[str]:
"""Return the content tags the site publishes for the repository.
ModelScope's entries are dicts carrying an English ``Tag`` plus a
``ChineseName``; the English value is the curated content vocabulary, so
that is the one surfaced here.
``OfficialTags`` is ModelScope's curated content vocabulary and is
preferred whenever it is populated. Plenty of AIGC repositories leave it
empty and carry only the plain ``Tags`` list, which mixes content tags with
framework and task categories; those categories are dropped so a card is
not handed "lora" and "text-to-image" as if they described the model.
"""
curated = _dedupe(_tag_values(data.get("OfficialTags")))
if curated:
return curated
generic = set(_GENERIC_TAGS)
for value in (
data.get("AigcType"),
data.get("Libraries"),
data.get("Frameworks"),
):
for item in value if isinstance(value, list) else [value]:
text = _clean_text(item).lower()
if text:
generic.add(text)
return _dedupe(
tag for tag in _tag_values(data.get("Tags")) if tag.lower() not in generic
)
def _tag_values(value: Any) -> list[str]:
"""Return the tag strings from either shape ModelScope publishes.
``OfficialTags`` is a list of ``{"Tag": ..., "ChineseName": ...}`` dicts
carrying an English value; the plain ``Tags`` list is already strings.
"""
tags: list[str] = []
if not isinstance(value, list):
return tags
return []
tags: list[str] = []
for entry in value:
if not isinstance(entry, dict):
continue
tag = _clean_text(entry.get("Tag"))
if tag and tag not in tags:
tag = _clean_text(entry.get("Tag") if isinstance(entry, dict) else entry)
if tag:
tags.append(tag)
return tags
def _dedupe(values: Iterable[str]) -> list[str]:
"""Drop empties and repeats, keeping the first spelling seen."""
unique: list[str] = []
for value in values:
if value and value not in unique:
unique.append(value)
return unique
def _version_files(version: dict[str, Any]) -> list[str]:
"""Return the model filenames covered by one ``MuseInfo.versions`` entry.
@@ -359,6 +473,23 @@ def _version_show_name(version: dict[str, Any]) -> str:
return _clean_text(model_version.get("showName")).lower()
def _version_label(versions: list[dict[str, Any]]) -> str:
"""Return the first published version label, preserving its spelling.
Unlike :func:`_version_show_name` this is for display, so the label is
not lowercased.
"""
for version in versions:
model_version = version.get("modelVersion")
if not isinstance(model_version, dict):
continue
label = _clean_text(model_version.get("showName"))
if label:
return label
return ""
def _file_digests(data: dict[str, Any]) -> dict[str, str]:
"""Return ``basename -> sha256`` for every published weight file.