mirror of
https://github.com/willmiao/ComfyUI-Lora-Manager.git
synced 2026-09-20 18:51:26 -03:00
feat(download): fill model metadata from the source API on download
A ModelScope or Hugging Face download landed as a bare filename, hash and
source link; the model card stayed empty until the user ran "Enrich
Metadata with AI" by hand. But everything that makes a CivitAI download
useful — the display name, the description, the tags, the trigger words,
the example images, the preview — is already published by those sites'
public APIs, so asking for it at download time is deterministic work, not
model work.
Add `py/services/model_sources/hydration.py`, called by
`_save_source_metadata()` once the sidecar exists and the file is in the
scanner cache. It fetches the model card plus the site's card extras and
hands them to the same `PostProcessor` the AI skill uses, with an empty
`llm_output`, so the two paths cannot drift apart. What lands:
* `model_name` from the site's own display name (ModelScope's `Name`), so
the card stops showing the local filename — written only while the value
still equals the file stem, since once a user renames a model that
choice is theirs to keep
* `civitai.name` from the matched version's label (`showName`), which the
card renders as the version chip
* `civitai.description` / `modelDescription` from the author summary plus
the README as HTML
* `civitai.images` / `preview_url` from the per-file example images
* `civitai.trainedWords` from the per-file trigger words
* `base_model`, `tags` and `usage_tips` as before
Provenance stays honest: the pass records
`metadata_source = "source:<platform>"` rather than the skill's
`agent:enrich_hf_metadata`, and — because no provider ran — it no longer
stamps `llm_enriched_at`; that stamp is now conditional on the LLM
actually answering, which is what the field means. The five hand-rolled
`civitai` dict merges in the post-processor collapse into one
`_merge_civitai()` helper.
Two guards keep it safe. Only a model whose stored
`source_platform`/`source_url` match the repository being downloaded is
updated, so a local file that merely shares a name never receives another
model's card; and a file already on disk is topped up too, which
back-fills models downloaded before this existed. READMEs and detail
payloads describe the repository rather than the file, so a short-lived
process-wide `ModelSourceCache` (300 s, 32 entries) keeps a batch over one
repository to two HTTP requests. Every failure is logged and swallowed:
hydration can never fail a download.
Fix the hash policy while here. `_save_source_metadata()` went straight to
`MetadataManager.create_default_metadata()`, bypassing the per-type
factory on the owning scanner, so a checkpoint paid a full SHA256 inside
the download request — `CheckpointScanner`/`OtherScanner` deliberately
record `hash_status="pending"` with an empty `sha256` for their multi-GB
files. Metadata is now created through `scanner._create_default_metadata()`.
Hydration copes with the empty hash: `_matching_versions()` falls back to
the repository basename, which is exactly what the download just wrote.
Report both post-transfer stages, which advance no byte counter and so
read as a stall: the bar sat at 100% showing `0 B/s` for the seconds spent
hashing and fetching. `_report_phase()` broadcasts
`{"status": "metadata", "stage": "indexing" | "source", "platform": ...}`,
and `LoadingManager` names the stage in the status line (keeping the batch
position), retitles the item line, replaces the dead speed figure and runs
a sheen over the bar. `stage`/`platform` are machine-readable; the wording
is localised in the frontend.
Finally, `modelscope.ai` is its own catalogue rather than an alias of
`modelscope.cn` — `referall13/EM1` exists only on `.ai` and
`jj3550945163/Krea-2-LORA` only on `.cn` — so its URLs were rejected with
"Invalid model URL format". Register it as `ModelScopeIntlSource`
(`platform="modelscope-ai"`, `msai:` group prefix, its own default
download directory) and derive every URL either deployment builds from a
per-class `base_url`. `modelscope.com` stays an alias of `.cn`, which is
what it redirects to. The frontend source table, the link dialog hints and
the docs mirror the split.
Verified against the live APIs: both reported `.ai` repositories list
their files, read their READMEs and yield name / version / base model /
trigger words / example images. Backend 3092 passed; frontend 1259 JS +
91 Vue passed. The nine locales carry the new progress copy in the next
commit.
This commit is contained in:
+53
-1
@@ -71,9 +71,18 @@ Enriches models linked to an external model site with metadata extracted by an L
|
|||||||
| Platform | Link | AI enrichment | Direct download |
|
| Platform | Link | AI enrichment | Direct download |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| Hugging Face | yes | yes | yes |
|
| Hugging Face | yes | yes | yes |
|
||||||
| ModelScope | yes | yes | yes |
|
| ModelScope (`modelscope.cn`) | yes | yes | yes |
|
||||||
|
| ModelScope International (`modelscope.ai`) | yes | yes | yes |
|
||||||
| TensorArt | yes | no (see below) | no |
|
| TensorArt | yes | no (see below) | no |
|
||||||
|
|
||||||
|
`modelscope.cn` and `modelscope.ai` are **separate catalogues, not mirrors** — a
|
||||||
|
repository published on one is routinely absent from the other — so each is
|
||||||
|
registered as its own source (`ModelScopeSource` / `ModelScopeIntlSource` in
|
||||||
|
`py/services/model_sources/modelscope.py`). The host therefore decides which
|
||||||
|
API and CDN a model resolves against, and the two deployments get separate
|
||||||
|
version groups (`ms:` / `msai:`) and default download directories. Keep the two
|
||||||
|
tables in `modelSourceHelpers.js` and `registry.py` in step when adding a site.
|
||||||
|
|
||||||
TensorArt is link-only: `tensor.art` sits behind a Cloudflare managed challenge and its internal API requires session authorization, so the backend cannot read its model pages. Linking still stores the canonical page URL and the "View on TensorArt" link works.
|
TensorArt is link-only: `tensor.art` sits behind a Cloudflare managed challenge and its internal API requires session authorization, so the backend cannot read its model pages. Linking still stores the canonical page URL and the "View on TensorArt" link works.
|
||||||
|
|
||||||
**What it does**:
|
**What it does**:
|
||||||
@@ -133,7 +142,9 @@ gaps the LLM leaves behind:
|
|||||||
|
|
||||||
| Field | Deterministic source | LLM role |
|
| Field | Deterministic source | LLM role |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
|
| `model_name` | site display name (`Name`), written only while the value is still the file stem | — |
|
||||||
| `modelDescription` | author summary + README as HTML | — |
|
| `modelDescription` | author summary + README as HTML | — |
|
||||||
|
| `civitai.name` | the matched version's label (`modelVersion.showName`) | — |
|
||||||
| `civitai.images` | site example images, then README images | — |
|
| `civitai.images` | site example images, then README images | — |
|
||||||
| `preview_url` | first available example image | may propose one from the README |
|
| `preview_url` | first available example image | may propose one from the README |
|
||||||
| `tags` | site-curated tags, always merged in | proposes additional content tags |
|
| `tags` | site-curated tags, always merged in | proposes additional content tags |
|
||||||
@@ -147,6 +158,47 @@ Models with no source, an unknown source, or a source without model-card access
|
|||||||
|
|
||||||
**Model types**: LoRA, Checkpoint, Embedding
|
**Model types**: LoRA, Checkpoint, Embedding
|
||||||
|
|
||||||
|
### Download-time hydration
|
||||||
|
|
||||||
|
The same deterministic mapping runs automatically when a model is downloaded
|
||||||
|
from a model source, so a ModelScope or Hugging Face download lands with the
|
||||||
|
populated card a CivitAI download produces instead of a bare filename and
|
||||||
|
hash. Nothing needs to be triggered by hand and no provider is called.
|
||||||
|
|
||||||
|
`py/services/model_sources/hydration.py` owns this path:
|
||||||
|
|
||||||
|
* `_save_source_metadata()` in `py/routes/handlers/model_source_handlers.py`
|
||||||
|
creates the sidecar (hash, source link, scanner-cache entry) and then calls
|
||||||
|
`hydrate_from_source()`. It also runs for a file that was already on disk, so
|
||||||
|
models downloaded before this existed get topped up on the next attempt.
|
||||||
|
* Metadata is created through the **owning scanner**
|
||||||
|
(`scanner._create_default_metadata()`) rather than
|
||||||
|
`MetadataManager.create_default_metadata()`, so the per-type lazy-hash rule
|
||||||
|
applies: `CheckpointScanner` and `OtherScanner` store
|
||||||
|
`hash_status="pending"` with an empty `sha256` for their multi-GB files, and
|
||||||
|
the generic helper would read a 10 GB checkpoint end to end inside the
|
||||||
|
download request. Hydration copes with the empty hash — `_matching_versions()`
|
||||||
|
falls back to the repository basename, which the download just wrote.
|
||||||
|
* Hydration reuses `PostProcessor` with an empty `llm_output`, so the two paths
|
||||||
|
cannot drift apart. It reports `metadata_source = "source:<platform>"` rather
|
||||||
|
than the skill's `agent:enrich_hf_metadata`, and — because no provider ran —
|
||||||
|
it does not stamp `llm_enriched_at`.
|
||||||
|
* `model_name` is only written while it still equals the file stem: once a user
|
||||||
|
renames a model, that choice is kept.
|
||||||
|
* Only a model whose stored `source_platform`/`source_url` match the repository
|
||||||
|
being downloaded is updated; a local file that merely shares a name must not
|
||||||
|
receive another model's card.
|
||||||
|
* The README and repository payload describe the *repository*, so a short-lived
|
||||||
|
process-wide `ModelSourceCache` (`shared_source_cache`, 300 s, 32 entries)
|
||||||
|
keeps a batch over one repository to two HTTP requests.
|
||||||
|
* Every failure — unreachable site, changed payload shape, broken post-processor
|
||||||
|
— is logged and swallowed. Metadata hydration can never fail a download.
|
||||||
|
* Neither stage advances the byte counter, so both are announced to the
|
||||||
|
progress UI (`_report_phase()` → `{"status": "metadata", "stage": ...}`) as
|
||||||
|
they start. Without that the bar sits at 100% reporting `0 B/s` for several
|
||||||
|
seconds and the download looks stuck. `stage` and `platform` are
|
||||||
|
machine-readable; the wording is localised in `LoadingManager`.
|
||||||
|
|
||||||
## Adding a New Skill
|
## Adding a New Skill
|
||||||
|
|
||||||
### 1. Create the skill directory
|
### 1. Create the skill directory
|
||||||
|
|||||||
@@ -1488,6 +1488,10 @@
|
|||||||
"progress": {
|
"progress": {
|
||||||
"currentFile": "Aktuelle Datei:",
|
"currentFile": "Aktuelle Datei:",
|
||||||
"downloading": "Wird heruntergeladen: {name}",
|
"downloading": "Wird heruntergeladen: {name}",
|
||||||
|
"metadata": "[TODO: Translate] Metadata: {name}",
|
||||||
|
"indexingFile": "[TODO: Translate] Reading model file...",
|
||||||
|
"fetchingSourceMetadata": "[TODO: Translate] Fetching metadata from {source}...",
|
||||||
|
"fetchingMetadata": "[TODO: Translate] Fetching metadata...",
|
||||||
"transferred": "Heruntergeladen: {downloaded} / {total}",
|
"transferred": "Heruntergeladen: {downloaded} / {total}",
|
||||||
"transferredSimple": "Heruntergeladen: {downloaded}",
|
"transferredSimple": "Heruntergeladen: {downloaded}",
|
||||||
"transferredUnknown": "Heruntergeladen: --",
|
"transferredUnknown": "Heruntergeladen: --",
|
||||||
|
|||||||
@@ -1488,6 +1488,10 @@
|
|||||||
"progress": {
|
"progress": {
|
||||||
"currentFile": "Current file:",
|
"currentFile": "Current file:",
|
||||||
"downloading": "Downloading: {name}",
|
"downloading": "Downloading: {name}",
|
||||||
|
"metadata": "Metadata: {name}",
|
||||||
|
"indexingFile": "Reading model file...",
|
||||||
|
"fetchingSourceMetadata": "Fetching metadata from {source}...",
|
||||||
|
"fetchingMetadata": "Fetching metadata...",
|
||||||
"transferred": "Transferred: {downloaded} / {total}",
|
"transferred": "Transferred: {downloaded} / {total}",
|
||||||
"transferredSimple": "Transferred: {downloaded}",
|
"transferredSimple": "Transferred: {downloaded}",
|
||||||
"transferredUnknown": "Transferred: --",
|
"transferredUnknown": "Transferred: --",
|
||||||
|
|||||||
@@ -1488,6 +1488,10 @@
|
|||||||
"progress": {
|
"progress": {
|
||||||
"currentFile": "Archivo actual:",
|
"currentFile": "Archivo actual:",
|
||||||
"downloading": "Descargando: {name}",
|
"downloading": "Descargando: {name}",
|
||||||
|
"metadata": "[TODO: Translate] Metadata: {name}",
|
||||||
|
"indexingFile": "[TODO: Translate] Reading model file...",
|
||||||
|
"fetchingSourceMetadata": "[TODO: Translate] Fetching metadata from {source}...",
|
||||||
|
"fetchingMetadata": "[TODO: Translate] Fetching metadata...",
|
||||||
"transferred": "Descargado: {downloaded} / {total}",
|
"transferred": "Descargado: {downloaded} / {total}",
|
||||||
"transferredSimple": "Descargado: {downloaded}",
|
"transferredSimple": "Descargado: {downloaded}",
|
||||||
"transferredUnknown": "Descargado: --",
|
"transferredUnknown": "Descargado: --",
|
||||||
|
|||||||
@@ -1488,6 +1488,10 @@
|
|||||||
"progress": {
|
"progress": {
|
||||||
"currentFile": "Fichier actuel :",
|
"currentFile": "Fichier actuel :",
|
||||||
"downloading": "Téléchargement : {name}",
|
"downloading": "Téléchargement : {name}",
|
||||||
|
"metadata": "[TODO: Translate] Metadata: {name}",
|
||||||
|
"indexingFile": "[TODO: Translate] Reading model file...",
|
||||||
|
"fetchingSourceMetadata": "[TODO: Translate] Fetching metadata from {source}...",
|
||||||
|
"fetchingMetadata": "[TODO: Translate] Fetching metadata...",
|
||||||
"transferred": "Téléchargé : {downloaded} / {total}",
|
"transferred": "Téléchargé : {downloaded} / {total}",
|
||||||
"transferredSimple": "Téléchargé : {downloaded}",
|
"transferredSimple": "Téléchargé : {downloaded}",
|
||||||
"transferredUnknown": "Téléchargé : --",
|
"transferredUnknown": "Téléchargé : --",
|
||||||
|
|||||||
@@ -1488,6 +1488,10 @@
|
|||||||
"progress": {
|
"progress": {
|
||||||
"currentFile": "הקובץ הנוכחי:",
|
"currentFile": "הקובץ הנוכחי:",
|
||||||
"downloading": "מוריד: {name}",
|
"downloading": "מוריד: {name}",
|
||||||
|
"metadata": "[TODO: Translate] Metadata: {name}",
|
||||||
|
"indexingFile": "[TODO: Translate] Reading model file...",
|
||||||
|
"fetchingSourceMetadata": "[TODO: Translate] Fetching metadata from {source}...",
|
||||||
|
"fetchingMetadata": "[TODO: Translate] Fetching metadata...",
|
||||||
"transferred": "הורד: {downloaded} / {total}",
|
"transferred": "הורד: {downloaded} / {total}",
|
||||||
"transferredSimple": "הורד: {downloaded}",
|
"transferredSimple": "הורד: {downloaded}",
|
||||||
"transferredUnknown": "הורד: --",
|
"transferredUnknown": "הורד: --",
|
||||||
|
|||||||
@@ -1488,6 +1488,10 @@
|
|||||||
"progress": {
|
"progress": {
|
||||||
"currentFile": "現在のファイル:",
|
"currentFile": "現在のファイル:",
|
||||||
"downloading": "ダウンロード中: {name}",
|
"downloading": "ダウンロード中: {name}",
|
||||||
|
"metadata": "[TODO: Translate] Metadata: {name}",
|
||||||
|
"indexingFile": "[TODO: Translate] Reading model file...",
|
||||||
|
"fetchingSourceMetadata": "[TODO: Translate] Fetching metadata from {source}...",
|
||||||
|
"fetchingMetadata": "[TODO: Translate] Fetching metadata...",
|
||||||
"transferred": "ダウンロード済み: {downloaded} / {total}",
|
"transferred": "ダウンロード済み: {downloaded} / {total}",
|
||||||
"transferredSimple": "ダウンロード済み: {downloaded}",
|
"transferredSimple": "ダウンロード済み: {downloaded}",
|
||||||
"transferredUnknown": "ダウンロード済み: --",
|
"transferredUnknown": "ダウンロード済み: --",
|
||||||
|
|||||||
@@ -1488,6 +1488,10 @@
|
|||||||
"progress": {
|
"progress": {
|
||||||
"currentFile": "현재 파일:",
|
"currentFile": "현재 파일:",
|
||||||
"downloading": "다운로드 중: {name}",
|
"downloading": "다운로드 중: {name}",
|
||||||
|
"metadata": "[TODO: Translate] Metadata: {name}",
|
||||||
|
"indexingFile": "[TODO: Translate] Reading model file...",
|
||||||
|
"fetchingSourceMetadata": "[TODO: Translate] Fetching metadata from {source}...",
|
||||||
|
"fetchingMetadata": "[TODO: Translate] Fetching metadata...",
|
||||||
"transferred": "다운로드됨: {downloaded} / {total}",
|
"transferred": "다운로드됨: {downloaded} / {total}",
|
||||||
"transferredSimple": "다운로드됨: {downloaded}",
|
"transferredSimple": "다운로드됨: {downloaded}",
|
||||||
"transferredUnknown": "다운로드됨: --",
|
"transferredUnknown": "다운로드됨: --",
|
||||||
|
|||||||
@@ -1488,6 +1488,10 @@
|
|||||||
"progress": {
|
"progress": {
|
||||||
"currentFile": "Текущий файл:",
|
"currentFile": "Текущий файл:",
|
||||||
"downloading": "Скачивается: {name}",
|
"downloading": "Скачивается: {name}",
|
||||||
|
"metadata": "[TODO: Translate] Metadata: {name}",
|
||||||
|
"indexingFile": "[TODO: Translate] Reading model file...",
|
||||||
|
"fetchingSourceMetadata": "[TODO: Translate] Fetching metadata from {source}...",
|
||||||
|
"fetchingMetadata": "[TODO: Translate] Fetching metadata...",
|
||||||
"transferred": "Скачано: {downloaded} / {total}",
|
"transferred": "Скачано: {downloaded} / {total}",
|
||||||
"transferredSimple": "Скачано: {downloaded}",
|
"transferredSimple": "Скачано: {downloaded}",
|
||||||
"transferredUnknown": "Скачано: --",
|
"transferredUnknown": "Скачано: --",
|
||||||
|
|||||||
@@ -1488,6 +1488,10 @@
|
|||||||
"progress": {
|
"progress": {
|
||||||
"currentFile": "当前文件:",
|
"currentFile": "当前文件:",
|
||||||
"downloading": "下载中:{name}",
|
"downloading": "下载中:{name}",
|
||||||
|
"metadata": "[TODO: Translate] Metadata: {name}",
|
||||||
|
"indexingFile": "[TODO: Translate] Reading model file...",
|
||||||
|
"fetchingSourceMetadata": "[TODO: Translate] Fetching metadata from {source}...",
|
||||||
|
"fetchingMetadata": "[TODO: Translate] Fetching metadata...",
|
||||||
"transferred": "已下载:{downloaded} / {total}",
|
"transferred": "已下载:{downloaded} / {total}",
|
||||||
"transferredSimple": "已下载:{downloaded}",
|
"transferredSimple": "已下载:{downloaded}",
|
||||||
"transferredUnknown": "已下载:--",
|
"transferredUnknown": "已下载:--",
|
||||||
|
|||||||
@@ -1488,6 +1488,10 @@
|
|||||||
"progress": {
|
"progress": {
|
||||||
"currentFile": "目前檔案:",
|
"currentFile": "目前檔案:",
|
||||||
"downloading": "下載中:{name}",
|
"downloading": "下載中:{name}",
|
||||||
|
"metadata": "[TODO: Translate] Metadata: {name}",
|
||||||
|
"indexingFile": "[TODO: Translate] Reading model file...",
|
||||||
|
"fetchingSourceMetadata": "[TODO: Translate] Fetching metadata from {source}...",
|
||||||
|
"fetchingMetadata": "[TODO: Translate] Fetching metadata...",
|
||||||
"transferred": "已下載:{downloaded} / {total}",
|
"transferred": "已下載:{downloaded} / {total}",
|
||||||
"transferredSimple": "已下載:{downloaded}",
|
"transferredSimple": "已下載:{downloaded}",
|
||||||
"transferredUnknown": "已下載:--",
|
"transferredUnknown": "已下載:--",
|
||||||
|
|||||||
@@ -1910,6 +1910,11 @@ class ModelDownloadHandler:
|
|||||||
response_payload["status"] = status
|
response_payload["status"] = status
|
||||||
if "message" in progress_data:
|
if "message" in progress_data:
|
||||||
response_payload["message"] = progress_data["message"]
|
response_payload["message"] = progress_data["message"]
|
||||||
|
# Post-transfer stage (indexing / source metadata); polling
|
||||||
|
# consumers need it to tell "working" from "stuck".
|
||||||
|
for field in ("stage", "platform"):
|
||||||
|
if field in progress_data:
|
||||||
|
response_payload[field] = progress_data[field]
|
||||||
elif status is None and "message" in progress_data:
|
elif status is None and "message" in progress_data:
|
||||||
response_payload["message"] = progress_data["message"]
|
response_payload["message"] = progress_data["message"]
|
||||||
|
|
||||||
|
|||||||
@@ -30,6 +30,7 @@ from ...services.model_sources import (
|
|||||||
SourceRef,
|
SourceRef,
|
||||||
detect_source,
|
detect_source,
|
||||||
get_download_source,
|
get_download_source,
|
||||||
|
hydrate_from_source,
|
||||||
is_valid_source_id,
|
is_valid_source_id,
|
||||||
list_sources,
|
list_sources,
|
||||||
normalize_metadata_source,
|
normalize_metadata_source,
|
||||||
@@ -85,25 +86,77 @@ def _infer_model_type(model_root: str) -> tuple[Any, str]:
|
|||||||
return _DEFAULT_MODEL_CLASS, _DEFAULT_SCANNER_GETTER
|
return _DEFAULT_MODEL_CLASS, _DEFAULT_SCANNER_GETTER
|
||||||
|
|
||||||
|
|
||||||
|
async def _report_phase(
|
||||||
|
download_id: str | None, stage: str, platform: str = ""
|
||||||
|
) -> None:
|
||||||
|
"""Tell the progress UI which post-transfer stage is running.
|
||||||
|
|
||||||
|
A download's byte counter stops the moment the last byte lands, but the
|
||||||
|
backend still has to index the file and read the model site's API. Without
|
||||||
|
this the bar sits at 100% reporting "0 B/s" and the download looks stuck for
|
||||||
|
several seconds. *stage* is machine-readable — the UI localises it — and
|
||||||
|
*platform* lets it name the site the metadata comes from.
|
||||||
|
"""
|
||||||
|
|
||||||
|
if not download_id:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
await ws_manager.broadcast_download_progress(
|
||||||
|
download_id,
|
||||||
|
{
|
||||||
|
"status": "metadata",
|
||||||
|
"stage": stage,
|
||||||
|
"platform": platform,
|
||||||
|
"progress": 100,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
except Exception as exc: # pragma: no cover - progress must never be fatal
|
||||||
|
logger.debug("Failed to report the '%s' phase: %s", stage, exc)
|
||||||
|
|
||||||
|
|
||||||
async def _save_source_metadata(
|
async def _save_source_metadata(
|
||||||
dest_path: str, ref: SourceRef, model_root: str
|
dest_path: str, ref: SourceRef, model_root: str, *, download_id: str | None = None
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Create a proper .metadata.json and add the model to the scanner cache.
|
"""Create a proper .metadata.json and add the model to the scanner cache.
|
||||||
|
|
||||||
Uses ``MetadataManager.create_default_metadata()`` which computes the
|
The metadata is created through the owning scanner rather than
|
||||||
SHA256 hash, extracts safetensors header metadata (base_model), and
|
``MetadataManager.create_default_metadata()``, because that is the only
|
||||||
produces a fully-populated ``LoraMetadata`` (or ``CheckpointMetadata`` /
|
factory that knows when hashing must be deferred: ``CheckpointScanner`` and
|
||||||
``EmbeddingMetadata``) object. We then overlay the external-source fields
|
``OtherScanner`` deliberately record ``hash_status="pending"`` with an empty
|
||||||
and register the model in the in-memory scanner cache so it appears
|
``sha256`` for their multi-GB files, and the generic helper would read a
|
||||||
immediately without a full filesystem walk.
|
10 GB checkpoint end to end *inside the download request*. Scanners for the
|
||||||
|
small types delegate straight back to it, so nothing changes for them.
|
||||||
|
|
||||||
|
The external-source fields are then overlaid and the model is registered in
|
||||||
|
the in-memory scanner cache so it appears immediately without a full
|
||||||
|
filesystem walk.
|
||||||
|
|
||||||
|
Finally the site's own published metadata is applied (see
|
||||||
|
:func:`~py.services.model_sources.hydration.hydrate_from_source`), so a
|
||||||
|
ModelScope or Hugging Face download lands with the same populated model
|
||||||
|
card a CivitAI download produces instead of a bare filename and hash.
|
||||||
|
|
||||||
|
Both post-transfer stages are reported through *download_id* when the UI is
|
||||||
|
watching one, because neither advances the byte counter.
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
model_class, scanner_getter_name = _infer_model_type(model_root)
|
model_class, scanner_getter_name = _infer_model_type(model_root)
|
||||||
|
|
||||||
# 1. Create proper metadata (computes SHA256, reads safetensors headers)
|
scanner = None
|
||||||
metadata = await MetadataManager.create_default_metadata(
|
scanner_getter = getattr(ServiceRegistry, scanner_getter_name, None)
|
||||||
dest_path, model_class=model_class
|
if scanner_getter is not None:
|
||||||
)
|
scanner = await scanner_getter()
|
||||||
|
|
||||||
|
# 1. Create proper metadata (reads safetensors headers; hashes only for
|
||||||
|
# the model types whose scanner does not defer it)
|
||||||
|
await _report_phase(download_id, "indexing", ref.platform)
|
||||||
|
create_metadata = getattr(scanner, "_create_default_metadata", None)
|
||||||
|
if create_metadata is not None:
|
||||||
|
metadata = await create_metadata(dest_path)
|
||||||
|
else:
|
||||||
|
metadata = await MetadataManager.create_default_metadata(
|
||||||
|
dest_path, model_class=model_class
|
||||||
|
)
|
||||||
if metadata is None:
|
if metadata is None:
|
||||||
logger.warning("create_default_metadata returned None for %s", dest_path)
|
logger.warning("create_default_metadata returned None for %s", dest_path)
|
||||||
return
|
return
|
||||||
@@ -120,8 +173,8 @@ async def _save_source_metadata(
|
|||||||
# 3. Save metadata atomically
|
# 3. Save metadata atomically
|
||||||
await MetadataManager.save_metadata(dest_path, metadata)
|
await MetadataManager.save_metadata(dest_path, metadata)
|
||||||
logger.info(
|
logger.info(
|
||||||
"Saved %s metadata (source=%s) for %s",
|
"Saved %s metadata (source=%s, hash_status=%s) for %s",
|
||||||
ref.platform, ref.url, dest_path,
|
ref.platform, ref.url, getattr(metadata, "hash_status", "?"), dest_path,
|
||||||
)
|
)
|
||||||
|
|
||||||
# 4. Determine relative folder path for cache
|
# 4. Determine relative folder path for cache
|
||||||
@@ -132,13 +185,16 @@ async def _save_source_metadata(
|
|||||||
folder = rel.replace(os.sep, "/") if rel != "." else ""
|
folder = rel.replace(os.sep, "/") if rel != "." else ""
|
||||||
|
|
||||||
# 5. Add to scanner cache (same as CivitAI's _execute_download does)
|
# 5. Add to scanner cache (same as CivitAI's _execute_download does)
|
||||||
scanner_getter = getattr(ServiceRegistry, scanner_getter_name, None)
|
if scanner is not None:
|
||||||
if scanner_getter is not None:
|
metadata_dict = normalize_metadata_source(metadata.to_dict())
|
||||||
scanner = await scanner_getter()
|
await scanner.add_model_to_cache(metadata_dict, folder)
|
||||||
if scanner is not None:
|
logger.info("Added %s to scanner cache (folder=%s)", dest_path, folder)
|
||||||
metadata_dict = normalize_metadata_source(metadata.to_dict())
|
|
||||||
await scanner.add_model_to_cache(metadata_dict, folder)
|
# 6. Top up from the site's public API. Runs last so the scanner-cache
|
||||||
logger.info("Added %s to scanner cache (folder=%s)", dest_path, folder)
|
# refresh it performs lands on the entry created above. It never
|
||||||
|
# raises and never fails the download.
|
||||||
|
await _report_phase(download_id, "source", ref.platform)
|
||||||
|
await hydrate_from_source(dest_path, ref=ref)
|
||||||
|
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
logger.warning("Failed to save source metadata for %s: %s", dest_path, exc)
|
logger.warning("Failed to save source metadata for %s: %s", dest_path, exc)
|
||||||
@@ -466,15 +522,6 @@ class ModelSourceHandler:
|
|||||||
os.makedirs(target_dir, exist_ok=True)
|
os.makedirs(target_dir, exist_ok=True)
|
||||||
dest_path = os.path.join(target_dir, file_base)
|
dest_path = os.path.join(target_dir, file_base)
|
||||||
|
|
||||||
# Check if already exists (simple skip)
|
|
||||||
if os.path.exists(dest_path) and os.path.getsize(dest_path) > 0:
|
|
||||||
logger.info("download_model_source: file already exists, skipping — %s", dest_path)
|
|
||||||
return web.json_response({
|
|
||||||
"success": True,
|
|
||||||
"message": f"File already exists: {dest_path}",
|
|
||||||
"path": dest_path,
|
|
||||||
})
|
|
||||||
|
|
||||||
# Built per request: sites that redirect to a CDN hand out a
|
# Built per request: sites that redirect to a CDN hand out a
|
||||||
# time-limited token in the redirect, so the URL must never be cached.
|
# time-limited token in the redirect, so the URL must never be cached.
|
||||||
resolve_url = source.file_download_url(repo, filename, revision)
|
resolve_url = source.file_download_url(repo, filename, revision)
|
||||||
@@ -482,6 +529,20 @@ class ModelSourceHandler:
|
|||||||
platform=source.platform, source_id=repo, url=source.canonical_url(repo)
|
platform=source.platform, source_id=repo, url=source.canonical_url(repo)
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Check if already exists (simple skip)
|
||||||
|
if os.path.exists(dest_path) and os.path.getsize(dest_path) > 0:
|
||||||
|
logger.info("download_model_source: file already exists, skipping — %s", dest_path)
|
||||||
|
# The sidecar may predate the source metadata being fetched, or may
|
||||||
|
# have been deleted, so top it up instead of skipping past it.
|
||||||
|
# Hydration no-ops when there is no sidecar to update.
|
||||||
|
await _report_phase(download_id, "source", source.platform)
|
||||||
|
await hydrate_from_source(dest_path, ref=ref)
|
||||||
|
return web.json_response({
|
||||||
|
"success": True,
|
||||||
|
"message": f"File already exists: {dest_path}",
|
||||||
|
"path": dest_path,
|
||||||
|
})
|
||||||
|
|
||||||
# Set up progress callback if download_id is provided
|
# Set up progress callback if download_id is provided
|
||||||
progress_callback = None
|
progress_callback = None
|
||||||
if download_id:
|
if download_id:
|
||||||
@@ -530,7 +591,9 @@ class ModelSourceHandler:
|
|||||||
progress_callback=progress_callback,
|
progress_callback=progress_callback,
|
||||||
)
|
)
|
||||||
if ok:
|
if ok:
|
||||||
await _save_source_metadata(dest_path, ref, model_root)
|
await _save_source_metadata(
|
||||||
|
dest_path, ref, model_root, download_id=download_id
|
||||||
|
)
|
||||||
return web.json_response({
|
return web.json_response({
|
||||||
"success": True,
|
"success": True,
|
||||||
"message": f"Downloaded to {dest_path}",
|
"message": f"Downloaded to {dest_path}",
|
||||||
@@ -557,7 +620,9 @@ class ModelSourceHandler:
|
|||||||
progress_callback=progress_callback,
|
progress_callback=progress_callback,
|
||||||
)
|
)
|
||||||
if success:
|
if success:
|
||||||
await _save_source_metadata(dest_path, ref, model_root)
|
await _save_source_metadata(
|
||||||
|
dest_path, ref, model_root, download_id=download_id
|
||||||
|
)
|
||||||
return web.json_response({
|
return web.json_response({
|
||||||
"success": True,
|
"success": True,
|
||||||
"message": f"Downloaded to {result}",
|
"message": f"Downloaded to {result}",
|
||||||
|
|||||||
@@ -33,8 +33,8 @@ from ..model_sources import (
|
|||||||
resolve_source_ref,
|
resolve_source_ref,
|
||||||
source_label,
|
source_label,
|
||||||
)
|
)
|
||||||
|
from ..model_sources.hydration import load_model_card, resolve_site_base_model
|
||||||
from ..websocket_manager import ws_manager
|
from ..websocket_manager import ws_manager
|
||||||
from .base_model_resolver import resolve_base_model
|
|
||||||
from .post_processor import PostProcessor
|
from .post_processor import PostProcessor
|
||||||
from .skill_registry import SkillRegistry
|
from .skill_registry import SkillRegistry
|
||||||
from .skills.enrich_hf_metadata.readme_processor import (
|
from .skills.enrich_hf_metadata.readme_processor import (
|
||||||
@@ -466,12 +466,7 @@ class AgentService:
|
|||||||
raw_basename = os.path.splitext(os.path.basename(model_path))[0]
|
raw_basename = os.path.splitext(os.path.basename(model_path))[0]
|
||||||
variables["asset_base_url"] = source.asset_base_url(ref.source_id)
|
variables["asset_base_url"] = source.asset_base_url(ref.source_id)
|
||||||
|
|
||||||
cache_key = f"{ref.platform}:{ref.source_id}"
|
readme = await load_model_card(source, ref.source_id, cache)
|
||||||
readme = cache.readmes.get(cache_key) if cache is not None else None
|
|
||||||
if readme is None:
|
|
||||||
readme = await source.fetch_model_card(ref.source_id)
|
|
||||||
if cache is not None and readme:
|
|
||||||
cache.readmes[cache_key] = readme
|
|
||||||
|
|
||||||
# Sites such as ModelScope keep part of the model card outside the
|
# Sites such as ModelScope keep part of the model card outside the
|
||||||
# README (author summary, curated tags, per-file example images). The
|
# README (author summary, curated tags, per-file example images). The
|
||||||
@@ -507,17 +502,7 @@ class AgentService:
|
|||||||
async def _resolve_site_base_model(self, source_context: ModelCardContext) -> str:
|
async def _resolve_site_base_model(self, source_context: ModelCardContext) -> str:
|
||||||
"""Resolve the site's base-model hints to a canonical name, or ``""``."""
|
"""Resolve the site's base-model hints to a canonical name, or ``""``."""
|
||||||
|
|
||||||
from ...metadata_ops import list_base_models
|
return await resolve_site_base_model(source_context)
|
||||||
|
|
||||||
hints = [*source_context.base_model_aliases, source_context.base_model]
|
|
||||||
if not any(hints):
|
|
||||||
return ""
|
|
||||||
try:
|
|
||||||
known_names = await list_base_models()
|
|
||||||
except Exception as exc:
|
|
||||||
logger.debug("Failed to list base models for site resolution: %s", exc)
|
|
||||||
return ""
|
|
||||||
return resolve_base_model(hints, known_names)
|
|
||||||
|
|
||||||
async def _build_prompt_context(
|
async def _build_prompt_context(
|
||||||
self,
|
self,
|
||||||
|
|||||||
@@ -48,6 +48,7 @@ class PostProcessor:
|
|||||||
readme_content: str = "",
|
readme_content: str = "",
|
||||||
source_context: Optional["ModelCardContext"] = None,
|
source_context: Optional["ModelCardContext"] = None,
|
||||||
resolved_base_model: str = "",
|
resolved_base_model: str = "",
|
||||||
|
metadata_source: str = "agent:enrich_hf_metadata",
|
||||||
) -> Dict[str, Any]:
|
) -> Dict[str, Any]:
|
||||||
"""Route *llm_output* to the correct skill post-processor.
|
"""Route *llm_output* to the correct skill post-processor.
|
||||||
|
|
||||||
@@ -63,13 +64,18 @@ class PostProcessor:
|
|||||||
hints resolve to, used when the LLM did not supply one (which is the
|
hints resolve to, used when the LLM did not supply one (which is the
|
||||||
normal case when the LLM was skipped).
|
normal case when the LLM was skipped).
|
||||||
|
|
||||||
|
*metadata_source* records who produced the metadata. The AI skill
|
||||||
|
keeps its historical value; the deterministic download-time hydration
|
||||||
|
passes its own so the two remain distinguishable. ``llm_enriched_at``
|
||||||
|
is only stamped when *llm_output* actually carries a provider answer.
|
||||||
|
|
||||||
Returns a dict with keys ``success`` (bool), ``updated_fields`` (list),
|
Returns a dict with keys ``success`` (bool), ``updated_fields`` (list),
|
||||||
``preview_downloaded`` (bool), and ``errors`` (list).
|
``preview_downloaded`` (bool), and ``errors`` (list).
|
||||||
"""
|
"""
|
||||||
if skill_name == "enrich_hf_metadata":
|
if skill_name == "enrich_hf_metadata":
|
||||||
return await self._process_enrich_hf_metadata(
|
return await self._process_enrich_hf_metadata(
|
||||||
model_path, llm_output, metadata, readme_content, source_context,
|
model_path, llm_output, metadata, readme_content, source_context,
|
||||||
resolved_base_model,
|
resolved_base_model, metadata_source,
|
||||||
)
|
)
|
||||||
return {
|
return {
|
||||||
"success": False,
|
"success": False,
|
||||||
@@ -89,6 +95,7 @@ class PostProcessor:
|
|||||||
readme_content: str = "",
|
readme_content: str = "",
|
||||||
source_context: Optional["ModelCardContext"] = None,
|
source_context: Optional["ModelCardContext"] = None,
|
||||||
resolved_base_model: str = "",
|
resolved_base_model: str = "",
|
||||||
|
metadata_source: str = "agent:enrich_hf_metadata",
|
||||||
) -> Dict[str, Any]:
|
) -> Dict[str, Any]:
|
||||||
from ...metadata_ops import (
|
from ...metadata_ops import (
|
||||||
apply_metadata_updates,
|
apply_metadata_updates,
|
||||||
@@ -135,6 +142,17 @@ class PostProcessor:
|
|||||||
if new_base and self._should_overwrite(current_base, is_source_model):
|
if new_base and self._should_overwrite(current_base, is_source_model):
|
||||||
updates["base_model"] = new_base
|
updates["base_model"] = new_base
|
||||||
|
|
||||||
|
# model_name — the site's own display name, so a source download never
|
||||||
|
# shows up under its local filename. Written only while the name is
|
||||||
|
# still the untouched file stem: once a user renames a model that
|
||||||
|
# choice is theirs to keep.
|
||||||
|
site_name = ((source_context.model_name if source_context else "") or "").strip()
|
||||||
|
if is_source_model and site_name:
|
||||||
|
current_name = (metadata.get("model_name") or "").strip()
|
||||||
|
file_stem = (metadata.get("file_name") or "").strip()
|
||||||
|
if not current_name or current_name == file_stem:
|
||||||
|
updates["model_name"] = site_name
|
||||||
|
|
||||||
# trigger words → civitai.trainedWords
|
# trigger words → civitai.trainedWords
|
||||||
new_triggers = llm_output.get("trigger_words", [])
|
new_triggers = llm_output.get("trigger_words", [])
|
||||||
trigger_words_empty = True
|
trigger_words_empty = True
|
||||||
@@ -142,14 +160,9 @@ class PostProcessor:
|
|||||||
cleaned = [t.strip() for t in new_triggers if t.strip()]
|
cleaned = [t.strip() for t in new_triggers if t.strip()]
|
||||||
cleaned = [t for t in cleaned if t.lower() not in ("none", "null", "n/a")]
|
cleaned = [t for t in cleaned if t.lower() not in ("none", "null", "n/a")]
|
||||||
trigger_words_empty = not cleaned
|
trigger_words_empty = not cleaned
|
||||||
current_civitai = metadata.get("civitai") or {}
|
current_triggers = (metadata.get("civitai") or {}).get("trainedWords") or []
|
||||||
current_triggers = current_civitai.get("trainedWords") or []
|
|
||||||
if self._should_overwrite_list(current_triggers, is_source_model):
|
if self._should_overwrite_list(current_triggers, is_source_model):
|
||||||
trig_civitai = dict(current_civitai)
|
self._merge_civitai(updates, metadata, trainedWords=cleaned)
|
||||||
if "civitai" in updates and isinstance(updates["civitai"], dict):
|
|
||||||
trig_civitai.update(updates["civitai"])
|
|
||||||
trig_civitai["trainedWords"] = cleaned
|
|
||||||
updates["civitai"] = trig_civitai
|
|
||||||
|
|
||||||
# modelDescription — the author's own summary (when the site keeps one
|
# modelDescription — the author's own summary (when the site keeps one
|
||||||
# outside the README, e.g. ModelScope's ``Description``) followed by the
|
# outside the README, e.g. ModelScope's ``Description``) followed by the
|
||||||
@@ -175,12 +188,16 @@ class PostProcessor:
|
|||||||
if not short_desc:
|
if not short_desc:
|
||||||
short_desc = site_description
|
short_desc = site_description
|
||||||
if short_desc and is_source_model:
|
if short_desc and is_source_model:
|
||||||
current_civitai = metadata.get("civitai") or {}
|
self._merge_civitai(updates, metadata, description=short_desc)
|
||||||
desc_civitai = dict(current_civitai)
|
|
||||||
if "civitai" in updates and isinstance(updates["civitai"], dict):
|
# The version label completes the card the way a CivitAI download does:
|
||||||
desc_civitai.update(updates["civitai"])
|
# the UI renders `civitai.name` as the version chip. It is per file,
|
||||||
desc_civitai["description"] = short_desc
|
# so a collection repository shows that checkpoint's own label.
|
||||||
updates["civitai"] = desc_civitai
|
site_version = (
|
||||||
|
(source_context.version_name if source_context else "") or ""
|
||||||
|
).strip()
|
||||||
|
if is_source_model and site_version:
|
||||||
|
self._merge_civitai(updates, metadata, name=site_version)
|
||||||
|
|
||||||
# gallery images → civitai.images (site example images, YAML frontmatter
|
# gallery images → civitai.images (site example images, YAML frontmatter
|
||||||
# widget entries, and Sample Gallery markdown tables in the README body)
|
# widget entries, and Sample Gallery markdown tables in the README body)
|
||||||
@@ -244,12 +261,7 @@ class PostProcessor:
|
|||||||
all_images = _dedupe_images(site_images + readme_images)
|
all_images = _dedupe_images(site_images + readme_images)
|
||||||
if all_images:
|
if all_images:
|
||||||
gallery_images = all_images
|
gallery_images = all_images
|
||||||
current_civitai = metadata.get("civitai") or {}
|
self._merge_civitai(updates, metadata, images=all_images)
|
||||||
gallery_civitai = dict(current_civitai)
|
|
||||||
if "civitai" in updates and isinstance(updates["civitai"], dict):
|
|
||||||
gallery_civitai.update(updates["civitai"])
|
|
||||||
gallery_civitai["images"] = all_images
|
|
||||||
updates["civitai"] = gallery_civitai
|
|
||||||
|
|
||||||
# tags — the site's curated tags are authoritative content vocabulary, so
|
# tags — the site's curated tags are authoritative content vocabulary, so
|
||||||
# they are kept alongside whatever the LLM proposed (the LLM is skipped
|
# they are kept alongside whatever the LLM proposed (the LLM is skipped
|
||||||
@@ -269,9 +281,12 @@ class PostProcessor:
|
|||||||
if len(merged) > len(existing_tags) or is_source_model:
|
if len(merged) > len(existing_tags) or is_source_model:
|
||||||
updates["tags"] = merged
|
updates["tags"] = merged
|
||||||
|
|
||||||
# metadata_source & llm_enriched_at (always set)
|
# metadata_source is recorded for provenance; llm_enriched_at only means
|
||||||
updates["metadata_source"] = "agent:enrich_hf_metadata"
|
# something when a provider actually answered, so the deterministic
|
||||||
updates["llm_enriched_at"] = datetime.now(timezone.utc).isoformat()
|
# download-time hydration does not claim an enrichment that never ran.
|
||||||
|
updates["metadata_source"] = metadata_source
|
||||||
|
if llm_output:
|
||||||
|
updates["llm_enriched_at"] = datetime.now(timezone.utc).isoformat()
|
||||||
|
|
||||||
# LLM confidence, stored for the enrichment evaluation harness. The key
|
# LLM confidence, stored for the enrichment evaluation harness. The key
|
||||||
# must NOT start with an underscore: `BaseModelMetadata.from_dict()`
|
# must NOT start with an underscore: `BaseModelMetadata.from_dict()`
|
||||||
@@ -292,12 +307,7 @@ class PostProcessor:
|
|||||||
if instance_prompt:
|
if instance_prompt:
|
||||||
site_triggers = [instance_prompt]
|
site_triggers = [instance_prompt]
|
||||||
if site_triggers:
|
if site_triggers:
|
||||||
current_civitai = metadata.get("civitai") or {}
|
self._merge_civitai(updates, metadata, trainedWords=site_triggers)
|
||||||
trig_civitai = dict(current_civitai)
|
|
||||||
if "civitai" in updates and isinstance(updates["civitai"], dict):
|
|
||||||
trig_civitai.update(updates["civitai"])
|
|
||||||
trig_civitai["trainedWords"] = site_triggers
|
|
||||||
updates["civitai"] = trig_civitai
|
|
||||||
|
|
||||||
preview_remote_url = (llm_output.get("preview_url") or "").strip()
|
preview_remote_url = (llm_output.get("preview_url") or "").strip()
|
||||||
# Fallback: if the LLM couldn't find a preview image in the cleaned
|
# Fallback: if the LLM couldn't find a preview image in the cleaned
|
||||||
@@ -371,6 +381,25 @@ class PostProcessor:
|
|||||||
"", "unknown",
|
"", "unknown",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _merge_civitai(
|
||||||
|
updates: Dict[str, Any], metadata: Dict[str, Any], **fields: Any
|
||||||
|
) -> None:
|
||||||
|
"""Layer *fields* onto the ``civitai`` block being assembled.
|
||||||
|
|
||||||
|
Description, version label, trigger words and gallery images all live
|
||||||
|
in the same dict and are contributed by separate branches, so each one
|
||||||
|
starts from what is already on disk and then applies whatever an
|
||||||
|
earlier branch queued in *updates*.
|
||||||
|
"""
|
||||||
|
|
||||||
|
merged = dict(metadata.get("civitai") or {})
|
||||||
|
queued = updates.get("civitai")
|
||||||
|
if isinstance(queued, dict):
|
||||||
|
merged.update(queued)
|
||||||
|
merged.update(fields)
|
||||||
|
updates["civitai"] = merged
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _should_overwrite_list(current_list: List[str], is_source_model: bool) -> bool:
|
def _should_overwrite_list(current_list: List[str], is_source_model: bool) -> bool:
|
||||||
"""Return ``True`` when a list field should be overwritten."""
|
"""Return ``True`` when a list field should be overwritten."""
|
||||||
|
|||||||
@@ -24,7 +24,12 @@ from .base import (
|
|||||||
is_valid_source_id,
|
is_valid_source_id,
|
||||||
)
|
)
|
||||||
from .huggingface import HuggingFaceSource
|
from .huggingface import HuggingFaceSource
|
||||||
from .modelscope import ModelScopeSource
|
from .hydration import (
|
||||||
|
hydrate_from_source,
|
||||||
|
load_model_card,
|
||||||
|
resolve_site_base_model,
|
||||||
|
)
|
||||||
|
from .modelscope import ModelScopeIntlSource, ModelScopeSource
|
||||||
from .registry import (
|
from .registry import (
|
||||||
LEGACY_HF_URL_FIELD,
|
LEGACY_HF_URL_FIELD,
|
||||||
SOURCE_PLATFORM_FIELD,
|
SOURCE_PLATFORM_FIELD,
|
||||||
@@ -52,6 +57,7 @@ __all__ = [
|
|||||||
"ModelSourceCache",
|
"ModelSourceCache",
|
||||||
"ModelSourceError",
|
"ModelSourceError",
|
||||||
"HuggingFaceSource",
|
"HuggingFaceSource",
|
||||||
|
"ModelScopeIntlSource",
|
||||||
"ModelScopeSource",
|
"ModelScopeSource",
|
||||||
"SOURCE_PLATFORM_FIELD",
|
"SOURCE_PLATFORM_FIELD",
|
||||||
"SOURCE_URL_FIELD",
|
"SOURCE_URL_FIELD",
|
||||||
@@ -68,9 +74,12 @@ __all__ = [
|
|||||||
"get_source",
|
"get_source",
|
||||||
"get_source_platform",
|
"get_source_platform",
|
||||||
"has_external_source",
|
"has_external_source",
|
||||||
|
"hydrate_from_source",
|
||||||
"is_valid_source_id",
|
"is_valid_source_id",
|
||||||
"list_sources",
|
"list_sources",
|
||||||
|
"load_model_card",
|
||||||
"normalize_metadata_source",
|
"normalize_metadata_source",
|
||||||
|
"resolve_site_base_model",
|
||||||
"resolve_source_ref",
|
"resolve_source_ref",
|
||||||
"source_group_key",
|
"source_group_key",
|
||||||
"source_label",
|
"source_label",
|
||||||
|
|||||||
@@ -45,6 +45,7 @@ USER_AGENT = "ComfyUI-LoRA-Manager/1.0"
|
|||||||
GROUP_PREFIXES: dict[str, str] = {
|
GROUP_PREFIXES: dict[str, str] = {
|
||||||
"huggingface": "hf",
|
"huggingface": "hf",
|
||||||
"modelscope": "ms",
|
"modelscope": "ms",
|
||||||
|
"modelscope-ai": "msai",
|
||||||
"tensorart": "ta",
|
"tensorart": "ta",
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -77,6 +78,30 @@ class ModelCardContext:
|
|||||||
description: str = ""
|
description: str = ""
|
||||||
"""Author-written summary shown on the model page, outside the README."""
|
"""Author-written summary shown on the model page, outside the README."""
|
||||||
|
|
||||||
|
model_name: str = ""
|
||||||
|
"""Site-published display name for the repository.
|
||||||
|
|
||||||
|
Sites publish this next to the repository id (ModelScope's ``Name``).
|
||||||
|
It is what a CivitAI download would store as the model's name, so the
|
||||||
|
card never has to fall back to the local filename.
|
||||||
|
"""
|
||||||
|
|
||||||
|
model_name_localized: str = ""
|
||||||
|
"""Site-published localized name (ModelScope's ``ChineseName``)."""
|
||||||
|
|
||||||
|
version_name: str = ""
|
||||||
|
"""Site-published label for the requested file's version.
|
||||||
|
|
||||||
|
Resolved per file, like :attr:`example_images`: a repository publishes
|
||||||
|
one label per checkpoint (ModelScope's ``modelVersion.showName``).
|
||||||
|
"""
|
||||||
|
|
||||||
|
license: str = ""
|
||||||
|
"""License the site records for the repository."""
|
||||||
|
|
||||||
|
model_type: str = ""
|
||||||
|
"""Site-reported model type, e.g. ModelScope's ``AigcType`` (``LoRA``)."""
|
||||||
|
|
||||||
base_model: str = ""
|
base_model: str = ""
|
||||||
"""Base model as reported by the site (possibly a site-local id)."""
|
"""Base model as reported by the site (possibly a site-local id)."""
|
||||||
|
|
||||||
@@ -104,6 +129,11 @@ class ModelCardContext:
|
|||||||
return not any(
|
return not any(
|
||||||
(
|
(
|
||||||
self.description,
|
self.description,
|
||||||
|
self.model_name,
|
||||||
|
self.model_name_localized,
|
||||||
|
self.version_name,
|
||||||
|
self.license,
|
||||||
|
self.model_type,
|
||||||
self.base_model,
|
self.base_model,
|
||||||
self.base_model_aliases,
|
self.base_model_aliases,
|
||||||
self.official_tags,
|
self.official_tags,
|
||||||
|
|||||||
@@ -0,0 +1,235 @@
|
|||||||
|
"""Deterministic metadata hydration for freshly downloaded source models.
|
||||||
|
|
||||||
|
A CivitAI download writes a fully-populated metadata sidecar as part of the
|
||||||
|
download itself: the name, the description, the tags, the trigger words and
|
||||||
|
the example images all arrive with the file. A download from an external
|
||||||
|
model source (ModelScope, Hugging Face) has the same information behind a
|
||||||
|
public API, but historically landed as a bare filename plus a source URL that
|
||||||
|
the user had to enrich by hand ("Enrich Metadata with AI").
|
||||||
|
|
||||||
|
This module closes that gap without involving an LLM. It fetches the linked
|
||||||
|
site's model card, hands it to the same :class:`~py.services.agent.post_processor.PostProcessor`
|
||||||
|
the AI skill uses, and writes the result. Everything it applies is data the
|
||||||
|
site published, so it is safe to run automatically on every download and to
|
||||||
|
treat as a fallback for the gaps the LLM would otherwise fill.
|
||||||
|
|
||||||
|
Nothing here may break a download: every failure is logged and normalised to
|
||||||
|
"the site had nothing to contribute".
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
from typing import TYPE_CHECKING, Optional
|
||||||
|
|
||||||
|
from .base import ModelCardContext, ModelSourceCache
|
||||||
|
from .registry import get_source, resolve_source_ref
|
||||||
|
|
||||||
|
if TYPE_CHECKING: # pragma: no cover - typing only
|
||||||
|
from .base import ModelSource, SourceRef
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
#: How long a fetched repository payload stays usable. A download batch walks
|
||||||
|
#: a repository's files one HTTP request at a time, and the README plus the
|
||||||
|
#: detail payload describe the *repository*, not the file, so re-fetching them
|
||||||
|
#: per file would be pure waste. They expire so an edited model card is still
|
||||||
|
#: picked up by the next batch.
|
||||||
|
SHARED_CACHE_TTL = 300.0
|
||||||
|
|
||||||
|
#: Upper bound on memoised repositories; a long-running server must not grow
|
||||||
|
#: without limit.
|
||||||
|
SHARED_CACHE_MAX_ENTRIES = 32
|
||||||
|
|
||||||
|
#: ``"<platform>:<source_id>"`` → ``(expiry, memo)``.
|
||||||
|
_shared_caches: dict[str, tuple[float, ModelSourceCache]] = {}
|
||||||
|
|
||||||
|
|
||||||
|
def shared_source_cache(platform: str, source_id: str) -> ModelSourceCache:
|
||||||
|
"""Return a short-lived per-repository memo for download-time hydration."""
|
||||||
|
|
||||||
|
now = time.monotonic()
|
||||||
|
key = f"{platform}:{source_id}"
|
||||||
|
entry = _shared_caches.get(key)
|
||||||
|
if entry is not None and entry[0] > now:
|
||||||
|
return entry[1]
|
||||||
|
|
||||||
|
for expired in [k for k, (expiry, _) in _shared_caches.items() if expiry <= now]:
|
||||||
|
_shared_caches.pop(expired, None)
|
||||||
|
if len(_shared_caches) >= SHARED_CACHE_MAX_ENTRIES:
|
||||||
|
oldest = min(_shared_caches, key=lambda k: _shared_caches[k][0])
|
||||||
|
_shared_caches.pop(oldest, None)
|
||||||
|
|
||||||
|
cache = ModelSourceCache()
|
||||||
|
_shared_caches[key] = (now + SHARED_CACHE_TTL, cache)
|
||||||
|
return cache
|
||||||
|
|
||||||
|
|
||||||
|
def reset_shared_caches() -> None:
|
||||||
|
"""Drop every memoised repository — used by tests."""
|
||||||
|
|
||||||
|
_shared_caches.clear()
|
||||||
|
|
||||||
|
|
||||||
|
async def load_model_card(
|
||||||
|
source: "ModelSource",
|
||||||
|
source_id: str,
|
||||||
|
cache: Optional[ModelSourceCache] = None,
|
||||||
|
) -> str:
|
||||||
|
"""Return *source_id*'s README, reusing *cache* when one is supplied.
|
||||||
|
|
||||||
|
Only successful reads are memoised, leaving a transient failure to be
|
||||||
|
retried for the next file of the same repository.
|
||||||
|
"""
|
||||||
|
|
||||||
|
key = f"{source.platform}:{source_id}"
|
||||||
|
if cache is not None:
|
||||||
|
cached = cache.readmes.get(key)
|
||||||
|
if cached is not None:
|
||||||
|
return cached
|
||||||
|
|
||||||
|
readme = await source.fetch_model_card(source_id)
|
||||||
|
if cache is not None and readme:
|
||||||
|
cache.readmes[key] = readme
|
||||||
|
return readme or ""
|
||||||
|
|
||||||
|
|
||||||
|
async def resolve_site_base_model(context: ModelCardContext) -> str:
|
||||||
|
"""Resolve the site's base-model hints to a canonical name, or ``""``.
|
||||||
|
|
||||||
|
Sites name base models in their own vocabulary (ModelScope publishes both
|
||||||
|
``krea/Krea-2-Turbo`` and the ``KREA_2_TURBO`` enum). The resolver is
|
||||||
|
strict and only ever returns a name the canonical vocabulary already
|
||||||
|
contains, so an uncertain hint yields ``""`` rather than a plausible-looking
|
||||||
|
wrong value.
|
||||||
|
"""
|
||||||
|
|
||||||
|
hints = [*context.base_model_aliases, context.base_model]
|
||||||
|
if not any(hints):
|
||||||
|
return ""
|
||||||
|
|
||||||
|
# Imported lazily: pulling in the agent package at module scope would make
|
||||||
|
# the model-source package import itself while it is still initialising.
|
||||||
|
try:
|
||||||
|
from ...metadata_ops import list_base_models
|
||||||
|
from ..agent.base_model_resolver import resolve_base_model
|
||||||
|
|
||||||
|
known_names = await list_base_models()
|
||||||
|
except Exception as exc:
|
||||||
|
logger.warning("Could not resolve a site base model: %s", exc)
|
||||||
|
return ""
|
||||||
|
return resolve_base_model(hints, known_names)
|
||||||
|
|
||||||
|
|
||||||
|
async def hydrate_from_source(
|
||||||
|
file_path: str,
|
||||||
|
*,
|
||||||
|
ref: "SourceRef",
|
||||||
|
cache: Optional[ModelSourceCache] = None,
|
||||||
|
) -> list[str]:
|
||||||
|
"""Apply the linked site's published metadata to a downloaded model.
|
||||||
|
|
||||||
|
This is the deterministic counterpart of the ``enrich_hf_metadata`` skill:
|
||||||
|
it produces the same populated model card a CivitAI download produces,
|
||||||
|
without an LLM and without user action.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
file_path: The just-downloaded model file, whose sidecar already
|
||||||
|
carries the SHA256 used to match the right file in a collection
|
||||||
|
repository.
|
||||||
|
ref: The source the file came from.
|
||||||
|
cache: Optional per-call memo; defaults to a short-lived shared one so
|
||||||
|
a batch over one repository fetches its card only once.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
The names of the metadata fields that changed. Never raises — a site
|
||||||
|
that is down, or an API that changed shape, must not fail a download.
|
||||||
|
"""
|
||||||
|
|
||||||
|
try:
|
||||||
|
source = get_source(ref.platform)
|
||||||
|
if source is None or not source.supports_enrichment:
|
||||||
|
return []
|
||||||
|
|
||||||
|
from ...metadata_ops import read_metadata
|
||||||
|
|
||||||
|
metadata = await read_metadata(file_path)
|
||||||
|
if not metadata:
|
||||||
|
logger.debug("No metadata to hydrate for %s", file_path)
|
||||||
|
return []
|
||||||
|
|
||||||
|
# Only a model that is actually linked to this repository may be
|
||||||
|
# updated. The download path writes those fields just before calling
|
||||||
|
# us; a file that merely shares a name with the requested one must not
|
||||||
|
# be given another model's card.
|
||||||
|
linked = resolve_source_ref(metadata)
|
||||||
|
if linked is None or (linked.platform, linked.source_id) != (
|
||||||
|
ref.platform,
|
||||||
|
ref.source_id,
|
||||||
|
):
|
||||||
|
logger.debug(
|
||||||
|
"Not hydrating %s: linked to %s, not %s",
|
||||||
|
file_path, linked.url if linked else "no model source", ref.url,
|
||||||
|
)
|
||||||
|
return []
|
||||||
|
|
||||||
|
memo = cache if cache is not None else shared_source_cache(
|
||||||
|
ref.platform, ref.source_id
|
||||||
|
)
|
||||||
|
readme = await load_model_card(source, ref.source_id, memo)
|
||||||
|
context = await source.fetch_model_card_context(
|
||||||
|
ref.source_id,
|
||||||
|
os.path.basename(file_path),
|
||||||
|
sha256=(metadata.get("sha256") or "").strip(),
|
||||||
|
cache=memo,
|
||||||
|
)
|
||||||
|
if context.is_empty() and not readme:
|
||||||
|
logger.debug(
|
||||||
|
"No published metadata for %s on %s", ref.source_id, ref.platform
|
||||||
|
)
|
||||||
|
return []
|
||||||
|
|
||||||
|
resolved_base_model = await resolve_site_base_model(context)
|
||||||
|
|
||||||
|
from ..agent.post_processor import PostProcessor
|
||||||
|
|
||||||
|
result = await PostProcessor().process(
|
||||||
|
skill_name="enrich_hf_metadata",
|
||||||
|
model_path=file_path,
|
||||||
|
llm_output={},
|
||||||
|
metadata=metadata,
|
||||||
|
readme_content=readme,
|
||||||
|
source_context=context,
|
||||||
|
resolved_base_model=resolved_base_model,
|
||||||
|
metadata_source=f"source:{ref.platform}",
|
||||||
|
)
|
||||||
|
if not result.get("success", True):
|
||||||
|
logger.debug(
|
||||||
|
"Hydration reported failure for %s: %s",
|
||||||
|
file_path, result.get("errors"),
|
||||||
|
)
|
||||||
|
return []
|
||||||
|
|
||||||
|
updated = list(result.get("updated_fields") or [])
|
||||||
|
logger.info(
|
||||||
|
"Hydrated %s from %s (%s): %s",
|
||||||
|
file_path, source.label or ref.platform, ref.source_id,
|
||||||
|
", ".join(updated) or "nothing to change",
|
||||||
|
)
|
||||||
|
return updated
|
||||||
|
except Exception as exc: # pragma: no cover - defensive by design
|
||||||
|
logger.warning("Source hydration failed for %s: %s", file_path, exc)
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"SHARED_CACHE_MAX_ENTRIES",
|
||||||
|
"SHARED_CACHE_TTL",
|
||||||
|
"hydrate_from_source",
|
||||||
|
"load_model_card",
|
||||||
|
"reset_shared_caches",
|
||||||
|
"resolve_site_base_model",
|
||||||
|
"shared_source_cache",
|
||||||
|
]
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
"""ModelScope (魔搭社区) model source.
|
"""ModelScope (魔搭社区) model sources.
|
||||||
|
|
||||||
ModelScope exposes the same "model card as README.md" convention as
|
ModelScope exposes the same "model card as README.md" convention as
|
||||||
Hugging Face, including a YAML frontmatter block that often carries
|
Hugging Face, including a YAML frontmatter block that often carries
|
||||||
@@ -10,11 +10,13 @@ none of which requires an API key for public models:
|
|||||||
the same content through the API, used as a fallback when the resolve
|
the same content through the API, used as a fallback when the resolve
|
||||||
URL is unavailable.
|
URL is unavailable.
|
||||||
* ``/api/v1/models/{owner}/{name}`` — the model-detail payload behind the
|
* ``/api/v1/models/{owner}/{name}`` — the model-detail payload behind the
|
||||||
model page. It carries the author's summary (``Description``), the
|
model page. It carries the repository's display name (``Name`` /
|
||||||
site-curated tags (``OfficialTags``), and, per published version, the
|
``ChineseName``), the author's summary (``Description``), the license, the
|
||||||
model filenames (``MuseInfo.versions[].stats.fileList``) together with
|
AIGC type, the site tags (``OfficialTags``, falling back to ``Tags``), and,
|
||||||
that file's example images (``coverImages``) and trigger words. See
|
per published version, the model filenames
|
||||||
:meth:`ModelScopeSource.fetch_model_card_context`.
|
(``MuseInfo.versions[].stats.fileList``) together with that version's label
|
||||||
|
(``modelVersion.showName``), example images (``coverImages``) and trigger
|
||||||
|
words. See :meth:`ModelScopeSource.fetch_model_card_context`.
|
||||||
* ``/api/v1/models/{owner}/{name}/repo/files?Revision=..`` — the file
|
* ``/api/v1/models/{owner}/{name}/repo/files?Revision=..`` — the file
|
||||||
listing backing the download picker. It reports real sizes for LFS
|
listing backing the download picker. It reports real sizes for LFS
|
||||||
files (not the pointer size), so no extra HEAD request is needed.
|
files (not the pointer size), so no extra HEAD request is needed.
|
||||||
@@ -28,6 +30,12 @@ valid; the CDN URL must never be cached.
|
|||||||
The README and the detail payload both describe the whole repository rather
|
The README and the detail payload both describe the whole repository rather
|
||||||
than one file, so a per-run ``ModelSourceCache`` keeps them from being read
|
than one file, so a per-run ``ModelSourceCache`` keeps them from being read
|
||||||
again for every checkpoint of a collection repository.
|
again for every checkpoint of a collection repository.
|
||||||
|
|
||||||
|
Two deployments are served by this module. ``modelscope.cn`` (with
|
||||||
|
``modelscope.com`` as a redirect alias) and ``modelscope.ai`` are *separate
|
||||||
|
catalogues*, not mirrors, so they are registered as distinct sources:
|
||||||
|
:class:`ModelScopeSource` and :class:`ModelScopeIntlSource`. Every URL either
|
||||||
|
class builds is derived from its ``base_url``.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -36,7 +44,7 @@ import json
|
|||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
from typing import TYPE_CHECKING, Any, Optional
|
from typing import TYPE_CHECKING, Any, Iterable, Optional
|
||||||
|
|
||||||
from .base import (
|
from .base import (
|
||||||
ModelCardContext,
|
ModelCardContext,
|
||||||
@@ -52,18 +60,28 @@ if TYPE_CHECKING: # pragma: no cover - typing only
|
|||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
_URL_PATTERN = re.compile(
|
#: ModelScope runs two independent catalogues. ``modelscope.com`` is a
|
||||||
r"https?://(?:www\.)?modelscope\.(?:cn|com)/models/(?P<id>[^/?#\s]+/[^/?#\s]+)"
|
#: redirect alias of the mainland site, but ``modelscope.ai`` is the
|
||||||
)
|
#: *international* deployment with its own repository catalogue — a repository
|
||||||
|
#: published on one is routinely absent from the other (``referall13/EM1``
|
||||||
|
#: exists only on ``.ai``, ``jj3550945163/Krea-2-LORA`` only on ``.cn``). The
|
||||||
|
#: host therefore decides which site, API and CDN a model belongs to, and the
|
||||||
|
#: two deployments are registered as separate sources rather than folded into
|
||||||
|
#: one id.
|
||||||
|
_MAINLAND_HOSTS = r"modelscope\.(?:cn|com)"
|
||||||
|
_INTERNATIONAL_HOSTS = r"modelscope\.ai"
|
||||||
|
|
||||||
#: Trailing view segments the site appends to a model URL; accepted verbatim
|
#: Trailing view segments the site appends to a model URL; accepted verbatim
|
||||||
#: when the user pastes a browser tab URL.
|
#: when the user pastes a browser tab URL.
|
||||||
_VIEW_SEGMENTS = r"(?:summary|files|model-file|readme|community|evaluation)?"
|
_VIEW_SEGMENTS = r"(?:summary|files|model-file|readme|community|evaluation)?"
|
||||||
|
|
||||||
_STRICT_URL_PATTERN = re.compile(
|
|
||||||
r"https?://(?:www\.)?modelscope\.(?:cn|com)/models/(?P<id>[^/?#\s]+/[^/?#\s]+)"
|
def _url_patterns(hosts: str) -> tuple[re.Pattern[str], re.Pattern[str]]:
|
||||||
rf"/?{_VIEW_SEGMENTS}/?$"
|
"""Build the lenient and strict model-URL patterns for *hosts*."""
|
||||||
)
|
|
||||||
|
body = rf"https?://(?:www\.)?(?:{hosts})/models/(?P<id>[^/?#\s]+/[^/?#\s]+)"
|
||||||
|
return re.compile(body), re.compile(rf"{body}/?{_VIEW_SEGMENTS}/?$")
|
||||||
|
|
||||||
|
|
||||||
#: ``master`` is ModelScope's default branch; ``main`` is tried as a fallback
|
#: ``master`` is ModelScope's default branch; ``main`` is tried as a fallback
|
||||||
#: for repos imported from Hugging Face.
|
#: for repos imported from Hugging Face.
|
||||||
@@ -71,7 +89,12 @@ _REVISIONS = ("master", "main")
|
|||||||
|
|
||||||
|
|
||||||
class ModelScopeSource(ModelSource):
|
class ModelScopeSource(ModelSource):
|
||||||
"""ModelScope (``modelscope.cn``)."""
|
"""ModelScope's mainland site (``modelscope.cn``).
|
||||||
|
|
||||||
|
``modelscope.com`` is accepted as an alias of it. The international
|
||||||
|
deployment is :class:`ModelScopeIntlSource`; everything below is written in
|
||||||
|
terms of ``base_url`` so both share one implementation.
|
||||||
|
"""
|
||||||
|
|
||||||
platform = "modelscope"
|
platform = "modelscope"
|
||||||
label = "ModelScope"
|
label = "ModelScope"
|
||||||
@@ -79,15 +102,18 @@ class ModelScopeSource(ModelSource):
|
|||||||
supports_download = True
|
supports_download = True
|
||||||
default_revision = "master"
|
default_revision = "master"
|
||||||
default_subdir = "modelscope"
|
default_subdir = "modelscope"
|
||||||
url_pattern = _URL_PATTERN
|
|
||||||
strict_url_pattern = _STRICT_URL_PATTERN
|
#: Origin every outgoing URL is built from.
|
||||||
|
base_url = "https://modelscope.cn"
|
||||||
|
|
||||||
|
url_pattern, strict_url_pattern = _url_patterns(_MAINLAND_HOSTS)
|
||||||
|
|
||||||
def canonical_url(self, source_id: str) -> str:
|
def canonical_url(self, source_id: str) -> str:
|
||||||
return f"https://modelscope.cn/models/{source_id}"
|
return f"{self.base_url}/models/{source_id}"
|
||||||
|
|
||||||
def asset_base_url(self, source_id: str, revision: str = "") -> str:
|
def asset_base_url(self, source_id: str, revision: str = "") -> str:
|
||||||
return (
|
return (
|
||||||
f"https://modelscope.cn/models/{source_id}/resolve/"
|
f"{self.base_url}/models/{source_id}/resolve/"
|
||||||
f"{self.resolve_revision(revision)}"
|
f"{self.resolve_revision(revision)}"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -96,7 +122,7 @@ class ModelScopeSource(ModelSource):
|
|||||||
|
|
||||||
for revision in _REVISIONS:
|
for revision in _REVISIONS:
|
||||||
text = await fetch_text(
|
text = await fetch_text(
|
||||||
f"https://modelscope.cn/models/{source_id}/resolve/{revision}/README.md"
|
f"{self.base_url}/models/{source_id}/resolve/{revision}/README.md"
|
||||||
)
|
)
|
||||||
if text:
|
if text:
|
||||||
return text
|
return text
|
||||||
@@ -105,7 +131,7 @@ class ModelScopeSource(ModelSource):
|
|||||||
# environments where the CDN resolve host is blocked.
|
# environments where the CDN resolve host is blocked.
|
||||||
for revision in _REVISIONS:
|
for revision in _REVISIONS:
|
||||||
text = await fetch_text(
|
text = await fetch_text(
|
||||||
"https://modelscope.cn/api/v1/models/"
|
f"{self.base_url}/api/v1/models/"
|
||||||
f"{source_id}/repo?Revision={revision}&FilePath=README.md"
|
f"{source_id}/repo?Revision={revision}&FilePath=README.md"
|
||||||
)
|
)
|
||||||
if text:
|
if text:
|
||||||
@@ -158,7 +184,7 @@ class ModelScopeSource(ModelSource):
|
|||||||
return cache.provider[cache_key]
|
return cache.provider[cache_key]
|
||||||
|
|
||||||
status, payload = await fetch_json(
|
status, payload = await fetch_json(
|
||||||
f"https://modelscope.cn/api/v1/models/{source_id}"
|
f"{self.base_url}/api/v1/models/{source_id}"
|
||||||
)
|
)
|
||||||
if status != 200 or not isinstance(payload, dict):
|
if status != 200 or not isinstance(payload, dict):
|
||||||
logger.debug(
|
logger.debug(
|
||||||
@@ -185,7 +211,7 @@ class ModelScopeSource(ModelSource):
|
|||||||
|
|
||||||
revision = self.resolve_revision(revision)
|
revision = self.resolve_revision(revision)
|
||||||
status, payload = await fetch_json(
|
status, payload = await fetch_json(
|
||||||
"https://modelscope.cn/api/v1/models/"
|
f"{self.base_url}/api/v1/models/"
|
||||||
f"{source_id}/repo/files?Revision={revision}"
|
f"{source_id}/repo/files?Revision={revision}"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -208,18 +234,37 @@ class ModelScopeSource(ModelSource):
|
|||||||
self, source_id: str, filename: str, revision: str = ""
|
self, source_id: str, filename: str, revision: str = ""
|
||||||
) -> str:
|
) -> str:
|
||||||
return (
|
return (
|
||||||
f"https://modelscope.cn/models/{source_id}/resolve/"
|
f"{self.base_url}/models/{source_id}/resolve/"
|
||||||
f"{self.resolve_revision(revision)}/{filename}"
|
f"{self.resolve_revision(revision)}/{filename}"
|
||||||
)
|
)
|
||||||
|
|
||||||
def page_url_for_file(self, source_id: str, filename: str) -> str:
|
def page_url_for_file(self, source_id: str, filename: str) -> str:
|
||||||
return (
|
return (
|
||||||
f"https://modelscope.cn/models/{source_id}/file/view/"
|
f"{self.base_url}/models/{source_id}/file/view/"
|
||||||
f"{self.default_revision}/{filename}"
|
f"{self.default_revision}/{filename}"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["ModelScopeSource"]
|
class ModelScopeIntlSource(ModelScopeSource):
|
||||||
|
"""ModelScope's international site (``modelscope.ai``).
|
||||||
|
|
||||||
|
A separate catalogue rather than a mirror, so it is registered under its
|
||||||
|
own platform id: the two deployments must not share a version group, a
|
||||||
|
"use default paths" directory, or a stored ``source_url``. The detail API,
|
||||||
|
the file listing, the resolve URLs and the CDN redirect all behave exactly
|
||||||
|
like the mainland site, which is why every URL here is derived from
|
||||||
|
:attr:`base_url` instead of being duplicated.
|
||||||
|
"""
|
||||||
|
|
||||||
|
platform = "modelscope-ai"
|
||||||
|
label = "ModelScope (International)"
|
||||||
|
default_subdir = "modelscope-ai"
|
||||||
|
base_url = "https://www.modelscope.ai"
|
||||||
|
|
||||||
|
url_pattern, strict_url_pattern = _url_patterns(_INTERNATIONAL_HOSTS)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["ModelScopeIntlSource", "ModelScopeSource"]
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -229,6 +274,33 @@ __all__ = ["ModelScopeSource"]
|
|||||||
#: Trigger-word values that mean "the author left this blank".
|
#: Trigger-word values that mean "the author left this blank".
|
||||||
_EMPTY_TRIGGER_VALUES = frozenset({"none", "null", "n/a"})
|
_EMPTY_TRIGGER_VALUES = frozenset({"none", "null", "n/a"})
|
||||||
|
|
||||||
|
#: Repository tags that only restate what the model *is* (its library, task or
|
||||||
|
#: framework) rather than what it depicts. ModelScope mixes both into the
|
||||||
|
#: plain ``Tags`` list, and a card tagged "lora" or "text-to-image" is noise.
|
||||||
|
_GENERIC_TAGS = frozenset(
|
||||||
|
{
|
||||||
|
"any-to-any",
|
||||||
|
"checkpoint",
|
||||||
|
"controlnet",
|
||||||
|
"diffusers",
|
||||||
|
"embedding",
|
||||||
|
"image-text-to-text",
|
||||||
|
"image-to-image",
|
||||||
|
"image-to-video",
|
||||||
|
"lora",
|
||||||
|
"lycoris",
|
||||||
|
"onnx",
|
||||||
|
"pytorch",
|
||||||
|
"safetensors",
|
||||||
|
"tensorflow",
|
||||||
|
"text-to-image",
|
||||||
|
"text-to-speech",
|
||||||
|
"text-to-video",
|
||||||
|
"textual-inversion",
|
||||||
|
"vae",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _clean_text(value: Any) -> str:
|
def _clean_text(value: Any) -> str:
|
||||||
"""Return a stripped string for *value*, or ``""`` for anything else."""
|
"""Return a stripped string for *value*, or ``""`` for anything else."""
|
||||||
@@ -259,9 +331,13 @@ def _build_card_context(
|
|||||||
|
|
||||||
context = ModelCardContext(
|
context = ModelCardContext(
|
||||||
description=_clean_text(data.get("Description")),
|
description=_clean_text(data.get("Description")),
|
||||||
|
model_name=_clean_text(data.get("Name")),
|
||||||
|
model_name_localized=_clean_text(data.get("ChineseName")),
|
||||||
|
license=_clean_text(data.get("License")),
|
||||||
|
model_type=_clean_text(data.get("AigcType")),
|
||||||
base_model=_first_string(data.get("BaseModel")),
|
base_model=_first_string(data.get("BaseModel")),
|
||||||
base_model_aliases=_base_model_aliases(data),
|
base_model_aliases=_base_model_aliases(data),
|
||||||
official_tags=_official_tags(data.get("OfficialTags")),
|
official_tags=_official_tags(data),
|
||||||
)
|
)
|
||||||
|
|
||||||
versions = _matching_versions(
|
versions = _matching_versions(
|
||||||
@@ -271,6 +347,7 @@ def _build_card_context(
|
|||||||
sha256=sha256,
|
sha256=sha256,
|
||||||
)
|
)
|
||||||
if versions:
|
if versions:
|
||||||
|
context.version_name = _version_label(versions)
|
||||||
context.example_images = _cover_image_urls(versions)
|
context.example_images = _cover_image_urls(versions)
|
||||||
context.trigger_words = _version_trigger_words(versions)
|
context.trigger_words = _version_trigger_words(versions)
|
||||||
return context
|
return context
|
||||||
@@ -303,26 +380,63 @@ def _base_model_aliases(data: dict[str, Any]) -> list[str]:
|
|||||||
return aliases
|
return aliases
|
||||||
|
|
||||||
|
|
||||||
def _official_tags(value: Any) -> list[str]:
|
def _official_tags(data: dict[str, Any]) -> list[str]:
|
||||||
"""Extract the site-curated tag values from ``OfficialTags``.
|
"""Return the content tags the site publishes for the repository.
|
||||||
|
|
||||||
ModelScope's entries are dicts carrying an English ``Tag`` plus a
|
``OfficialTags`` is ModelScope's curated content vocabulary and is
|
||||||
``ChineseName``; the English value is the curated content vocabulary, so
|
preferred whenever it is populated. Plenty of AIGC repositories leave it
|
||||||
that is the one surfaced here.
|
empty and carry only the plain ``Tags`` list, which mixes content tags with
|
||||||
|
framework and task categories; those categories are dropped so a card is
|
||||||
|
not handed "lora" and "text-to-image" as if they described the model.
|
||||||
|
"""
|
||||||
|
|
||||||
|
curated = _dedupe(_tag_values(data.get("OfficialTags")))
|
||||||
|
if curated:
|
||||||
|
return curated
|
||||||
|
|
||||||
|
generic = set(_GENERIC_TAGS)
|
||||||
|
for value in (
|
||||||
|
data.get("AigcType"),
|
||||||
|
data.get("Libraries"),
|
||||||
|
data.get("Frameworks"),
|
||||||
|
):
|
||||||
|
for item in value if isinstance(value, list) else [value]:
|
||||||
|
text = _clean_text(item).lower()
|
||||||
|
if text:
|
||||||
|
generic.add(text)
|
||||||
|
|
||||||
|
return _dedupe(
|
||||||
|
tag for tag in _tag_values(data.get("Tags")) if tag.lower() not in generic
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _tag_values(value: Any) -> list[str]:
|
||||||
|
"""Return the tag strings from either shape ModelScope publishes.
|
||||||
|
|
||||||
|
``OfficialTags`` is a list of ``{"Tag": ..., "ChineseName": ...}`` dicts
|
||||||
|
carrying an English value; the plain ``Tags`` list is already strings.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
tags: list[str] = []
|
|
||||||
if not isinstance(value, list):
|
if not isinstance(value, list):
|
||||||
return tags
|
return []
|
||||||
|
tags: list[str] = []
|
||||||
for entry in value:
|
for entry in value:
|
||||||
if not isinstance(entry, dict):
|
tag = _clean_text(entry.get("Tag") if isinstance(entry, dict) else entry)
|
||||||
continue
|
if tag:
|
||||||
tag = _clean_text(entry.get("Tag"))
|
|
||||||
if tag and tag not in tags:
|
|
||||||
tags.append(tag)
|
tags.append(tag)
|
||||||
return tags
|
return tags
|
||||||
|
|
||||||
|
|
||||||
|
def _dedupe(values: Iterable[str]) -> list[str]:
|
||||||
|
"""Drop empties and repeats, keeping the first spelling seen."""
|
||||||
|
|
||||||
|
unique: list[str] = []
|
||||||
|
for value in values:
|
||||||
|
if value and value not in unique:
|
||||||
|
unique.append(value)
|
||||||
|
return unique
|
||||||
|
|
||||||
|
|
||||||
def _version_files(version: dict[str, Any]) -> list[str]:
|
def _version_files(version: dict[str, Any]) -> list[str]:
|
||||||
"""Return the model filenames covered by one ``MuseInfo.versions`` entry.
|
"""Return the model filenames covered by one ``MuseInfo.versions`` entry.
|
||||||
|
|
||||||
@@ -359,6 +473,23 @@ def _version_show_name(version: dict[str, Any]) -> str:
|
|||||||
return _clean_text(model_version.get("showName")).lower()
|
return _clean_text(model_version.get("showName")).lower()
|
||||||
|
|
||||||
|
|
||||||
|
def _version_label(versions: list[dict[str, Any]]) -> str:
|
||||||
|
"""Return the first published version label, preserving its spelling.
|
||||||
|
|
||||||
|
Unlike :func:`_version_show_name` this is for display, so the label is
|
||||||
|
not lowercased.
|
||||||
|
"""
|
||||||
|
|
||||||
|
for version in versions:
|
||||||
|
model_version = version.get("modelVersion")
|
||||||
|
if not isinstance(model_version, dict):
|
||||||
|
continue
|
||||||
|
label = _clean_text(model_version.get("showName"))
|
||||||
|
if label:
|
||||||
|
return label
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
def _file_digests(data: dict[str, Any]) -> dict[str, str]:
|
def _file_digests(data: dict[str, Any]) -> dict[str, str]:
|
||||||
"""Return ``basename -> sha256`` for every published weight file.
|
"""Return ``basename -> sha256`` for every published weight file.
|
||||||
|
|
||||||
|
|||||||
@@ -13,15 +13,18 @@ from typing import Any, Dict, Mapping, Optional
|
|||||||
|
|
||||||
from .base import GROUP_PREFIXES, ModelSource, SourceRef, clean_source_url
|
from .base import GROUP_PREFIXES, ModelSource, SourceRef, clean_source_url
|
||||||
from .huggingface import HuggingFaceSource
|
from .huggingface import HuggingFaceSource
|
||||||
from .modelscope import ModelScopeSource
|
from .modelscope import ModelScopeIntlSource, ModelScopeSource
|
||||||
from .tensorart import TensorArtSource
|
from .tensorart import TensorArtSource
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
#: Order matters only for disambiguation; the URL patterns are disjoint.
|
#: Order matters only for disambiguation; the URL patterns are disjoint.
|
||||||
|
#: ``modelscope.ai`` is a separate catalogue from ``modelscope.cn`` rather than
|
||||||
|
#: an alias, which is why it gets its own entry (see ``modelscope.py``).
|
||||||
_SOURCES: tuple[ModelSource, ...] = (
|
_SOURCES: tuple[ModelSource, ...] = (
|
||||||
HuggingFaceSource(),
|
HuggingFaceSource(),
|
||||||
ModelScopeSource(),
|
ModelScopeSource(),
|
||||||
|
ModelScopeIntlSource(),
|
||||||
TensorArtSource(),
|
TensorArtSource(),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -170,6 +170,13 @@ class WebSocketManager:
|
|||||||
progress_entry['status'] = data['status']
|
progress_entry['status'] = data['status']
|
||||||
if 'message' in data:
|
if 'message' in data:
|
||||||
progress_entry['message'] = data['message']
|
progress_entry['message'] = data['message']
|
||||||
|
# Post-transfer stage reporting (see `model_source_handlers._report_phase`):
|
||||||
|
# the byte counter has stopped by then, so the stage is the only thing
|
||||||
|
# that still says the download is working.
|
||||||
|
if 'stage' in data:
|
||||||
|
progress_entry['stage'] = data['stage']
|
||||||
|
if 'platform' in data:
|
||||||
|
progress_entry['platform'] = data['platform']
|
||||||
|
|
||||||
self._download_progress[download_id] = progress_entry
|
self._download_progress[download_id] = progress_entry
|
||||||
|
|
||||||
|
|||||||
@@ -97,6 +97,32 @@
|
|||||||
width: 0%;
|
width: 0%;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* The transfer is done but the backend is still indexing the file and reading
|
||||||
|
the model site's API. A sheen over the full bar reads as "busy" where a
|
||||||
|
motionless 100% bar reads as "stuck". */
|
||||||
|
.current-item-bar.is-indeterminate {
|
||||||
|
position: relative;
|
||||||
|
overflow: hidden;
|
||||||
|
}
|
||||||
|
|
||||||
|
.current-item-bar.is-indeterminate::after {
|
||||||
|
content: '';
|
||||||
|
position: absolute;
|
||||||
|
inset: 0;
|
||||||
|
background: linear-gradient(
|
||||||
|
90deg,
|
||||||
|
transparent 0%,
|
||||||
|
rgba(255, 255, 255, 0.5) 50%,
|
||||||
|
transparent 100%
|
||||||
|
);
|
||||||
|
animation: progress-sheen 1.2s ease-in-out infinite;
|
||||||
|
}
|
||||||
|
|
||||||
|
@keyframes progress-sheen {
|
||||||
|
from { transform: translateX(-100%); }
|
||||||
|
to { transform: translateX(100%); }
|
||||||
|
}
|
||||||
|
|
||||||
.current-item-percent {
|
.current-item-percent {
|
||||||
font-size: 0.8rem;
|
font-size: 0.8rem;
|
||||||
color: var(--text-color-secondary, var(--text-color));
|
color: var(--text-color-secondary, var(--text-color));
|
||||||
@@ -131,4 +157,8 @@
|
|||||||
.current-item-bar {
|
.current-item-bar {
|
||||||
transition: none;
|
transition: none;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.current-item-bar.is-indeterminate::after {
|
||||||
|
animation: none;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -340,6 +340,27 @@ export class DownloadManager {
|
|||||||
|
|
||||||
// ---- External repository download flow (Hugging Face / ModelScope) ----
|
// ---- External repository download flow (Hugging Face / ModelScope) ----
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Report a post-transfer stage frame to the progress UI.
|
||||||
|
*
|
||||||
|
* The backend keeps working after the last byte lands — it indexes the
|
||||||
|
* file and reads the model site's API — and announces those stages with
|
||||||
|
* `status: 'metadata'`. Without them the bar sits at 100% showing "0 B/s"
|
||||||
|
* and the download looks stuck. The stage and platform are machine
|
||||||
|
* readable so LoadingManager can localise the wording.
|
||||||
|
*
|
||||||
|
* @returns {boolean} `true` when the frame was a stage frame.
|
||||||
|
*/
|
||||||
|
_applyMetadataStage(data, updateProgress, completed, name) {
|
||||||
|
if (data?.status !== 'metadata') return false;
|
||||||
|
updateProgress(100, completed, name, {}, {
|
||||||
|
phase: 'metadata',
|
||||||
|
stage: data.stage || '',
|
||||||
|
platform: data.platform || '',
|
||||||
|
});
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
/** Rendering group key: the same repo on two sites is two groups. */
|
/** Rendering group key: the same repo on two sites is two groups. */
|
||||||
_externalGroupKey(item) {
|
_externalGroupKey(item) {
|
||||||
return `${item.source}:${item.repo || 'unknown'}`;
|
return `${item.source}:${item.repo || 'unknown'}`;
|
||||||
@@ -1708,6 +1729,12 @@ export class DownloadManager {
|
|||||||
cancelled = true;
|
cancelled = true;
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
// Indexing / site metadata: the transfer is over but the
|
||||||
|
// backend is still working, so say so instead of
|
||||||
|
// leaving the bar frozen at 100%.
|
||||||
|
if (this._applyMetadataStage(data, updateProgress, snapshotCompleted, filename)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
if (data.status === 'progress') {
|
if (data.status === 'progress') {
|
||||||
const metrics = {
|
const metrics = {
|
||||||
bytesDownloaded: data.bytes_downloaded,
|
bytesDownloaded: data.bytes_downloaded,
|
||||||
@@ -2331,6 +2358,9 @@ export class DownloadManager {
|
|||||||
const snapshotCompleted = completedDownloads;
|
const snapshotCompleted = completedDownloads;
|
||||||
wsHf.onmessage = (event) => {
|
wsHf.onmessage = (event) => {
|
||||||
const data = JSON.parse(event.data);
|
const data = JSON.parse(event.data);
|
||||||
|
if (this._applyMetadataStage(data, updateProgress, snapshotCompleted, name)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
if (data.status === 'progress') {
|
if (data.status === 'progress') {
|
||||||
const metrics = {
|
const metrics = {
|
||||||
bytesDownloaded: data.bytes_downloaded,
|
bytesDownloaded: data.bytes_downloaded,
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
import { translate } from '../utils/i18nHelpers.js';
|
import { translate } from '../utils/i18nHelpers.js';
|
||||||
import { formatFileSize } from '../utils/formatters.js';
|
import { formatFileSize } from '../utils/formatters.js';
|
||||||
|
import { getModelSource } from '../utils/modelSourceHelpers.js';
|
||||||
|
|
||||||
// Loading management
|
// Loading management
|
||||||
export class LoadingManager {
|
export class LoadingManager {
|
||||||
@@ -278,6 +279,35 @@ export class LoadingManager {
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Describe a post-transfer stage in the status line.
|
||||||
|
*
|
||||||
|
* The byte counter stops as soon as the last byte lands, but the
|
||||||
|
* backend still hashes the file and reads the model site's API. Naming
|
||||||
|
* that work is what stops the bar looking frozen at 100%.
|
||||||
|
*/
|
||||||
|
const describeMetadataStage = (stage, platform) => {
|
||||||
|
if (stage === 'indexing') {
|
||||||
|
return translate(
|
||||||
|
'modals.download.progress.indexingFile',
|
||||||
|
{},
|
||||||
|
'Reading model file...'
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const label = getModelSource(platform)?.label || platform || '';
|
||||||
|
return label
|
||||||
|
? translate(
|
||||||
|
'modals.download.progress.fetchingSourceMetadata',
|
||||||
|
{ source: label },
|
||||||
|
`Fetching metadata from ${label}...`
|
||||||
|
)
|
||||||
|
: translate(
|
||||||
|
'modals.download.progress.fetchingMetadata',
|
||||||
|
{},
|
||||||
|
'Fetching metadata...'
|
||||||
|
);
|
||||||
|
};
|
||||||
|
|
||||||
// Initialize transfer stats with empty data
|
// Initialize transfer stats with empty data
|
||||||
updateTransferStats();
|
updateTransferStats();
|
||||||
|
|
||||||
@@ -285,19 +315,62 @@ export class LoadingManager {
|
|||||||
this.loadingContent.appendChild(this.cancelButton);
|
this.loadingContent.appendChild(this.cancelButton);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Return update function
|
/**
|
||||||
return (currentProgress, currentIndex = 0, currentName = '', metrics = {}) => {
|
* Update the progress UI.
|
||||||
|
*
|
||||||
|
* @param {number} currentProgress Percentage of the current item.
|
||||||
|
* @param {number} [currentIndex] Items finished so far.
|
||||||
|
* @param {string} [currentName] File being processed.
|
||||||
|
* @param {object} [metrics] Byte counters; only meaningful while
|
||||||
|
* transferring.
|
||||||
|
* @param {object} [phase] `{ phase: 'metadata', stage, platform }` once
|
||||||
|
* the transfer has finished, so the UI can show what is still running
|
||||||
|
* instead of a 0 B/s speed.
|
||||||
|
*/
|
||||||
|
return (
|
||||||
|
currentProgress,
|
||||||
|
currentIndex = 0,
|
||||||
|
currentName = '',
|
||||||
|
metrics = {},
|
||||||
|
phase = null
|
||||||
|
) => {
|
||||||
|
const isMetadata = phase?.phase === 'metadata';
|
||||||
|
|
||||||
// Update current item progress
|
// Update current item progress
|
||||||
currentItemProgress.style.width = `${currentProgress}%`;
|
currentItemProgress.style.width = `${currentProgress}%`;
|
||||||
currentItemPercent.textContent = `${Math.floor(currentProgress)}%`;
|
currentItemPercent.textContent = `${Math.floor(currentProgress)}%`;
|
||||||
|
currentItemProgress.classList.toggle('is-indeterminate', isMetadata);
|
||||||
|
|
||||||
// Update current item label if name provided
|
// Update current item label if name provided
|
||||||
if (currentName) {
|
if (currentName) {
|
||||||
currentItemLabel.textContent = translate(
|
currentItemLabel.textContent = isMetadata
|
||||||
'modals.download.progress.downloading',
|
? translate(
|
||||||
{ name: currentName },
|
'modals.download.progress.metadata',
|
||||||
`Downloading: ${currentName}`
|
{ name: currentName },
|
||||||
|
`Metadata: ${currentName}`
|
||||||
|
)
|
||||||
|
: translate(
|
||||||
|
'modals.download.progress.downloading',
|
||||||
|
{ name: currentName },
|
||||||
|
`Downloading: ${currentName}`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// No bytes are moving any more, so report the stage instead of a
|
||||||
|
// rate that has dropped to zero.
|
||||||
|
if (isMetadata) {
|
||||||
|
updateTransferStats({ bytesDownloaded: metrics.bytesDownloaded, totalBytes: metrics.totalBytes });
|
||||||
|
const stageText = describeMetadataStage(phase.stage, phase.platform);
|
||||||
|
speedDetail.textContent = stageText;
|
||||||
|
// Keep the batch position visible; the status line is the one
|
||||||
|
// place a caller also writes to.
|
||||||
|
this.setStatus(
|
||||||
|
totalItems > 1
|
||||||
|
? `${Math.min(currentIndex + 1, totalItems)}/${totalItems}: ${stageText}`
|
||||||
|
: stageText
|
||||||
);
|
);
|
||||||
|
} else {
|
||||||
|
updateTransferStats(metrics);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Update overall label if multiple items
|
// Update overall label if multiple items
|
||||||
@@ -311,8 +384,6 @@ export class LoadingManager {
|
|||||||
// Single item, just update main progress
|
// Single item, just update main progress
|
||||||
this.setProgress(currentProgress);
|
this.setProgress(currentProgress);
|
||||||
}
|
}
|
||||||
|
|
||||||
updateTransferStats(metrics);
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -49,6 +49,26 @@ export const MODEL_SOURCES = [
|
|||||||
filePage: (id, filename) =>
|
filePage: (id, filename) =>
|
||||||
`https://modelscope.cn/models/${id}/file/view/master/${filename}`,
|
`https://modelscope.cn/models/${id}/file/view/master/${filename}`,
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
// A separate catalogue from `modelscope.cn`, not an alias: a repository
|
||||||
|
// published on one is routinely absent from the other, so the host is part
|
||||||
|
// of the model's identity. Mirrors ModelScopeIntlSource in the backend.
|
||||||
|
platform: 'modelscope-ai',
|
||||||
|
label: 'ModelScope (International)',
|
||||||
|
groupPrefix: 'msai',
|
||||||
|
supportsEnrichment: true,
|
||||||
|
supportsDownload: true,
|
||||||
|
defaultRevision: 'master',
|
||||||
|
defaultSubdir: 'modelscope-ai',
|
||||||
|
exampleUrl: 'https://www.modelscope.ai/models/user/repo',
|
||||||
|
placeholder: 'https://www.modelscope.ai/models/user/repo',
|
||||||
|
pattern: /^https?:\/\/(?:www\.)?modelscope\.ai\/models\/([^/?#\s]+\/[^/?#\s]+)/i,
|
||||||
|
filePattern:
|
||||||
|
/^https?:\/\/(?:www\.)?modelscope\.ai\/models\/([^/?#\s]+\/[^/?#\s]+)\/resolve\/([^/?#\s]+)\/(.+)$/i,
|
||||||
|
canonical: (id) => `https://www.modelscope.ai/models/${id}`,
|
||||||
|
filePage: (id, filename) =>
|
||||||
|
`https://www.modelscope.ai/models/${id}/file/view/master/${filename}`,
|
||||||
|
},
|
||||||
{
|
{
|
||||||
platform: 'tensorart',
|
platform: 'tensorart',
|
||||||
label: 'TensorArt',
|
label: 'TensorArt',
|
||||||
|
|||||||
@@ -16,6 +16,7 @@
|
|||||||
<div id="hfSupportedSources">
|
<div id="hfSupportedSources">
|
||||||
<strong>https://huggingface.co/user/repo</strong><br>
|
<strong>https://huggingface.co/user/repo</strong><br>
|
||||||
<strong>https://modelscope.cn/models/user/repo</strong><br>
|
<strong>https://modelscope.cn/models/user/repo</strong><br>
|
||||||
|
<strong>https://www.modelscope.ai/models/user/repo</strong><br>
|
||||||
<strong>https://tensor.art/models/827823520299086029</strong>
|
<strong>https://tensor.art/models/827823520299086029</strong>
|
||||||
</div>
|
</div>
|
||||||
{{ t('modals.linkModelSource.enrichNote') }}
|
{{ t('modals.linkModelSource.enrichNote') }}
|
||||||
|
|||||||
@@ -266,4 +266,101 @@ describe('DownloadManager external model source downloads', () => {
|
|||||||
expect(manager._externalGroupKey(ms)).toBe('modelscope:u/r');
|
expect(manager._externalGroupKey(ms)).toBe('modelscope:u/r');
|
||||||
expect(manager._externalGroupKey(hf)).not.toBe(manager._externalGroupKey(ms));
|
expect(manager._externalGroupKey(hf)).not.toBe(manager._externalGroupKey(ms));
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe('post-transfer stage reporting', () => {
|
||||||
|
it('ignores ordinary frames', () => {
|
||||||
|
const updateProgress = vi.fn();
|
||||||
|
|
||||||
|
expect(
|
||||||
|
manager._applyMetadataStage(
|
||||||
|
{ status: 'progress', progress: 40, bytes_per_second: 10 },
|
||||||
|
updateProgress,
|
||||||
|
0,
|
||||||
|
'f.safetensors'
|
||||||
|
)
|
||||||
|
).toBe(false);
|
||||||
|
expect(updateProgress).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('routes a metadata stage to the progress bar at 100%', () => {
|
||||||
|
const updateProgress = vi.fn();
|
||||||
|
|
||||||
|
expect(
|
||||||
|
manager._applyMetadataStage(
|
||||||
|
{ status: 'metadata', stage: 'source', platform: 'modelscope' },
|
||||||
|
updateProgress,
|
||||||
|
3,
|
||||||
|
'f.safetensors'
|
||||||
|
)
|
||||||
|
).toBe(true);
|
||||||
|
expect(updateProgress).toHaveBeenCalledWith(100, 3, 'f.safetensors', {}, {
|
||||||
|
phase: 'metadata',
|
||||||
|
stage: 'source',
|
||||||
|
platform: 'modelscope',
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it('tolerates a stage frame with no stage or platform', () => {
|
||||||
|
const updateProgress = vi.fn();
|
||||||
|
|
||||||
|
expect(
|
||||||
|
manager._applyMetadataStage({ status: 'metadata' }, updateProgress, 0, 'f')
|
||||||
|
).toBe(true);
|
||||||
|
expect(updateProgress).toHaveBeenCalledWith(100, 0, 'f', {}, {
|
||||||
|
phase: 'metadata',
|
||||||
|
stage: '',
|
||||||
|
platform: '',
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it('surfaces a metadata frame received while the request is in flight', async () => {
|
||||||
|
// End-to-end through the websocket handler: the backend keeps the socket
|
||||||
|
// open while it hydrates, and the frame has to reach the progress bar.
|
||||||
|
const sockets = [];
|
||||||
|
class RecordingWebSocket {
|
||||||
|
constructor(url) {
|
||||||
|
this.url = url;
|
||||||
|
this.onopen = null;
|
||||||
|
this.onmessage = null;
|
||||||
|
this.onerror = null;
|
||||||
|
this.close = vi.fn();
|
||||||
|
sockets.push(this);
|
||||||
|
queueMicrotask(() => this.onopen && this.onopen());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
vi.stubGlobal('WebSocket', RecordingWebSocket);
|
||||||
|
|
||||||
|
const updateProgress = vi.fn();
|
||||||
|
mockLoadingManager.showDownloadProgress.mockReturnValue(updateProgress);
|
||||||
|
|
||||||
|
mockApiClient.downloadModelSource.mockImplementation(async () => {
|
||||||
|
sockets.at(-1).onmessage({
|
||||||
|
data: JSON.stringify({
|
||||||
|
status: 'metadata',
|
||||||
|
stage: 'source',
|
||||||
|
platform: 'modelscope',
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
return { success: true };
|
||||||
|
});
|
||||||
|
|
||||||
|
manager.sourcePlatform = 'modelscope';
|
||||||
|
manager.sourceRepoId = 'u/r';
|
||||||
|
manager.sourceSelectedFiles = ['a.safetensors'];
|
||||||
|
|
||||||
|
await manager._downloadExternalRepoFiles({
|
||||||
|
modelRoot: '/models',
|
||||||
|
targetFolder: '',
|
||||||
|
useDefaultPaths: false,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(updateProgress).toHaveBeenCalledWith(100, 0, 'a.safetensors', {}, {
|
||||||
|
phase: 'metadata',
|
||||||
|
stage: 'source',
|
||||||
|
platform: 'modelscope',
|
||||||
|
});
|
||||||
|
|
||||||
|
mockLoadingManager.showDownloadProgress.mockReturnValue(vi.fn());
|
||||||
|
});
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -0,0 +1,141 @@
|
|||||||
|
import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||||
|
|
||||||
|
const { I18N_MODULE } = vi.hoisted(() => ({
|
||||||
|
I18N_MODULE: new URL(
|
||||||
|
'../../../static/js/utils/i18nHelpers.js',
|
||||||
|
import.meta.url
|
||||||
|
).pathname,
|
||||||
|
}));
|
||||||
|
|
||||||
|
// Interpolate the English fallback the way the real helper does when a locale
|
||||||
|
// has not been loaded, so assertions can name the visible text.
|
||||||
|
vi.mock(I18N_MODULE, () => ({
|
||||||
|
translate: vi.fn((key, params = {}, fallback) => {
|
||||||
|
if (typeof fallback !== 'string') return key;
|
||||||
|
return Object.entries(params).reduce(
|
||||||
|
(text, [name, value]) => text.replace(`{${name}}`, String(value)),
|
||||||
|
fallback
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
}));
|
||||||
|
|
||||||
|
const { LoadingManager } = await import(
|
||||||
|
'../../../static/js/managers/LoadingManager.js'
|
||||||
|
);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A download's byte counter stops when the last byte lands, but the backend
|
||||||
|
* still hashes the file and reads the model site's API. These tests pin the
|
||||||
|
* rendering that says so, instead of leaving the bar at 100% showing 0 B/s.
|
||||||
|
*/
|
||||||
|
describe('LoadingManager download progress phases', () => {
|
||||||
|
let manager;
|
||||||
|
let updateProgress;
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
document.body.innerHTML = '';
|
||||||
|
LoadingManager.instance = null;
|
||||||
|
manager = new LoadingManager();
|
||||||
|
updateProgress = manager.showDownloadProgress(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
const speedText = () =>
|
||||||
|
document.querySelector('.download-transfer-speed')?.textContent;
|
||||||
|
const itemLabel = () =>
|
||||||
|
document.querySelector('.current-item-label')?.textContent;
|
||||||
|
const itemPercent = () =>
|
||||||
|
document.querySelector('.current-item-percent')?.textContent;
|
||||||
|
const itemBar = () => document.querySelector('.current-item-bar');
|
||||||
|
|
||||||
|
it('shows the byte rate while transferring', () => {
|
||||||
|
updateProgress(42, 0, 'model.safetensors', {
|
||||||
|
bytesDownloaded: 1024,
|
||||||
|
totalBytes: 2048,
|
||||||
|
bytesPerSecond: 512,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(itemLabel()).toBe('Downloading: model.safetensors');
|
||||||
|
expect(itemPercent()).toBe('42%');
|
||||||
|
expect(speedText()).toMatch(/^Speed: /);
|
||||||
|
expect(itemBar().classList.contains('is-indeterminate')).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('names the indexing stage instead of a stopped speed', () => {
|
||||||
|
updateProgress(100, 0, 'model.safetensors', {}, {
|
||||||
|
phase: 'metadata',
|
||||||
|
stage: 'indexing',
|
||||||
|
platform: 'modelscope',
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(itemLabel()).toBe('Metadata: model.safetensors');
|
||||||
|
expect(itemPercent()).toBe('100%');
|
||||||
|
expect(speedText()).toBe('Reading model file...');
|
||||||
|
expect(manager.statusText.textContent).toBe('Reading model file...');
|
||||||
|
expect(itemBar().classList.contains('is-indeterminate')).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('names the site the metadata is fetched from', () => {
|
||||||
|
updateProgress(100, 0, 'model.safetensors', {}, {
|
||||||
|
phase: 'metadata',
|
||||||
|
stage: 'source',
|
||||||
|
platform: 'modelscope-ai',
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(speedText()).toBe('Fetching metadata from ModelScope (International)...');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('falls back to a generic message for an unknown site', () => {
|
||||||
|
updateProgress(100, 0, 'model.safetensors', {}, {
|
||||||
|
phase: 'metadata',
|
||||||
|
stage: 'source',
|
||||||
|
platform: '',
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(speedText()).toBe('Fetching metadata...');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('returns to the transfer rendering for the next file', () => {
|
||||||
|
updateProgress(100, 0, 'a.safetensors', {}, {
|
||||||
|
phase: 'metadata',
|
||||||
|
stage: 'source',
|
||||||
|
platform: 'modelscope',
|
||||||
|
});
|
||||||
|
|
||||||
|
updateProgress(0, 1, 'b.safetensors');
|
||||||
|
|
||||||
|
expect(itemLabel()).toBe('Downloading: b.safetensors');
|
||||||
|
expect(speedText()).toMatch(/^Speed: /);
|
||||||
|
expect(itemBar().classList.contains('is-indeterminate')).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('keeps the byte counters visible during the metadata stage', () => {
|
||||||
|
updateProgress(100, 0, 'model.safetensors', {
|
||||||
|
bytesDownloaded: 2048,
|
||||||
|
totalBytes: 2048,
|
||||||
|
bytesPerSecond: 0,
|
||||||
|
}, {
|
||||||
|
phase: 'metadata',
|
||||||
|
stage: 'source',
|
||||||
|
platform: 'modelscope',
|
||||||
|
});
|
||||||
|
|
||||||
|
const transferred = document.querySelector('.download-transfer-bytes');
|
||||||
|
expect(transferred.textContent).toContain('/');
|
||||||
|
// The 0 B/s figure is what made the pause look like a stall.
|
||||||
|
expect(speedText()).not.toContain('0 B');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('keeps the batch position visible in the status line', () => {
|
||||||
|
updateProgress = manager.showDownloadProgress(4);
|
||||||
|
|
||||||
|
updateProgress(100, 2, 'c.safetensors', {}, {
|
||||||
|
phase: 'metadata',
|
||||||
|
stage: 'source',
|
||||||
|
platform: 'huggingface',
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(manager.statusText.textContent).toBe(
|
||||||
|
'3/4: Fetching metadata from Hugging Face...'
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -26,6 +26,7 @@ describe('modelSourceHelpers', () => {
|
|||||||
expect(MODEL_SOURCES.map((s) => s.platform)).toEqual([
|
expect(MODEL_SOURCES.map((s) => s.platform)).toEqual([
|
||||||
'huggingface',
|
'huggingface',
|
||||||
'modelscope',
|
'modelscope',
|
||||||
|
'modelscope-ai',
|
||||||
'tensorart',
|
'tensorart',
|
||||||
]);
|
]);
|
||||||
});
|
});
|
||||||
@@ -44,6 +45,16 @@ describe('modelSourceHelpers', () => {
|
|||||||
expect(info.url).toBe('https://modelscope.cn/models/user/repo');
|
expect(info.url).toBe('https://modelscope.cn/models/user/repo');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('recognises ModelScope International as its own platform', () => {
|
||||||
|
const info = parseModelSourceUrl(
|
||||||
|
'https://www.modelscope.ai/models/referall13/EM1/files'
|
||||||
|
);
|
||||||
|
expect(info.platform).toBe('modelscope-ai');
|
||||||
|
expect(info.groupPrefix).toBe('msai');
|
||||||
|
expect(info.sourceId).toBe('referall13/EM1');
|
||||||
|
expect(info.url).toBe('https://www.modelscope.ai/models/referall13/EM1');
|
||||||
|
});
|
||||||
|
|
||||||
it('recognises TensorArt URLs and keeps only the numeric id', () => {
|
it('recognises TensorArt URLs and keeps only the numeric id', () => {
|
||||||
const info = parseModelSourceUrl(
|
const info = parseModelSourceUrl(
|
||||||
'https://tensor.art/models/827823520299086029/Vivid-Impressions-Storybook-Sstyle-V1.0'
|
'https://tensor.art/models/827823520299086029/Vivid-Impressions-Storybook-Sstyle-V1.0'
|
||||||
|
|||||||
@@ -174,6 +174,54 @@ describe('DownloadManager.detectUrlType — external model source URLs', () => {
|
|||||||
expect(result.platform).toBe('huggingface');
|
expect(result.platform).toBe('huggingface');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// modelscope.ai is a separate catalogue from modelscope.cn, not an alias,
|
||||||
|
// so it carries its own platform id all the way to the backend.
|
||||||
|
it('detects a ModelScope International repo URL', () => {
|
||||||
|
const result = DownloadManager.detectUrlType(
|
||||||
|
'https://www.modelscope.ai/models/referall13/EM1'
|
||||||
|
);
|
||||||
|
expect(result).toEqual({
|
||||||
|
type: 'model-source-repo',
|
||||||
|
platform: 'modelscope-ai',
|
||||||
|
repo: 'referall13/EM1',
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it('detects a ModelScope International repo URL without the www prefix', () => {
|
||||||
|
const result = DownloadManager.detectUrlType(
|
||||||
|
'https://modelscope.ai/models/ErLubu/krea2_style_260911_02'
|
||||||
|
);
|
||||||
|
expect(result).toEqual({
|
||||||
|
type: 'model-source-repo',
|
||||||
|
platform: 'modelscope-ai',
|
||||||
|
repo: 'ErLubu/krea2_style_260911_02',
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it('detects a ModelScope International file URL', () => {
|
||||||
|
const result = DownloadManager.detectUrlType(
|
||||||
|
'https://www.modelscope.ai/models/referall13/EM1/resolve/master/EM1_c1-st1000.safetensors'
|
||||||
|
);
|
||||||
|
expect(result).toEqual({
|
||||||
|
type: 'model-source-file',
|
||||||
|
platform: 'modelscope-ai',
|
||||||
|
repo: 'referall13/EM1',
|
||||||
|
revision: 'master',
|
||||||
|
filename: 'EM1_c1-st1000.safetensors',
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it('keeps the two ModelScope deployments distinct', () => {
|
||||||
|
const mainland = DownloadManager.detectUrlType(
|
||||||
|
'https://modelscope.cn/models/referall13/EM1'
|
||||||
|
);
|
||||||
|
const intl = DownloadManager.detectUrlType(
|
||||||
|
'https://www.modelscope.ai/models/referall13/EM1'
|
||||||
|
);
|
||||||
|
expect(mainland.platform).toBe('modelscope');
|
||||||
|
expect(intl.platform).toBe('modelscope-ai');
|
||||||
|
});
|
||||||
|
|
||||||
it('rejects path traversal in either platform', () => {
|
it('rejects path traversal in either platform', () => {
|
||||||
expect(
|
expect(
|
||||||
DownloadManager.detectUrlType('https://modelscope.cn/models/../etc/passwd')
|
DownloadManager.detectUrlType('https://modelscope.cn/models/../etc/passwd')
|
||||||
|
|||||||
@@ -307,7 +307,12 @@ async def test_get_model_sources_lists_capabilities():
|
|||||||
sources = _json_payload(response)
|
sources = _json_payload(response)
|
||||||
|
|
||||||
by_platform = {s["platform"]: s for s in sources}
|
by_platform = {s["platform"]: s for s in sources}
|
||||||
assert set(by_platform) == {"huggingface", "modelscope", "tensorart"}
|
assert set(by_platform) == {
|
||||||
|
"huggingface",
|
||||||
|
"modelscope",
|
||||||
|
"modelscope-ai",
|
||||||
|
"tensorart",
|
||||||
|
}
|
||||||
assert by_platform["huggingface"]["supports_enrichment"] is True
|
assert by_platform["huggingface"]["supports_enrichment"] is True
|
||||||
assert by_platform["modelscope"]["supports_enrichment"] is True
|
assert by_platform["modelscope"]["supports_enrichment"] is True
|
||||||
# TensorArt is link-only: no accessible model card for the backend.
|
# TensorArt is link-only: no accessible model card for the backend.
|
||||||
@@ -315,6 +320,12 @@ async def test_get_model_sources_lists_capabilities():
|
|||||||
assert by_platform["modelscope"]["supports_download"] is True
|
assert by_platform["modelscope"]["supports_download"] is True
|
||||||
assert by_platform["modelscope"]["default_revision"] == "master"
|
assert by_platform["modelscope"]["default_revision"] == "master"
|
||||||
assert by_platform["tensorart"]["supports_download"] is False
|
assert by_platform["tensorart"]["supports_download"] is False
|
||||||
|
# The international deployment is advertised with its own example URL, so
|
||||||
|
# the Link dialog names the host a user actually has open.
|
||||||
|
assert by_platform["modelscope-ai"]["supports_download"] is True
|
||||||
|
assert by_platform["modelscope-ai"]["example_url"].startswith(
|
||||||
|
"https://www.modelscope.ai/"
|
||||||
|
)
|
||||||
assert all(s["example_url"] for s in sources)
|
assert all(s["example_url"] for s in sources)
|
||||||
|
|
||||||
|
|
||||||
@@ -492,6 +503,44 @@ async def test_download_model_source_modelscope_default_paths(tmp_path, monkeypa
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_download_model_source_modelscope_intl_uses_its_own_host(
|
||||||
|
tmp_path, monkeypatch
|
||||||
|
):
|
||||||
|
"""`.ai` is a separate catalogue, so the download must not go to `.cn`."""
|
||||||
|
captured = _stub_download_backend(monkeypatch)
|
||||||
|
saved = AsyncMock()
|
||||||
|
monkeypatch.setattr(model_source_handlers, "_save_source_metadata", saved)
|
||||||
|
|
||||||
|
response = await ModelSourceHandler().download_model_source(
|
||||||
|
FakeRequest(
|
||||||
|
json_data={
|
||||||
|
"platform": "modelscope-ai",
|
||||||
|
"repo": "referall13/EM1",
|
||||||
|
"filename": "EM1_c1-st1000.safetensors",
|
||||||
|
"model_root": str(tmp_path),
|
||||||
|
"use_default_paths": True,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert response.status == 200
|
||||||
|
assert captured["url"] == (
|
||||||
|
"https://www.modelscope.ai/models/referall13/EM1/resolve/master/"
|
||||||
|
"EM1_c1-st1000.safetensors"
|
||||||
|
)
|
||||||
|
# Its own default directory, so the same owner/name on both deployments
|
||||||
|
# cannot overwrite each other.
|
||||||
|
assert captured["save_path"] == str(
|
||||||
|
tmp_path / "modelscope-ai" / "referall13" / "EM1" / "EM1_c1-st1000.safetensors"
|
||||||
|
)
|
||||||
|
|
||||||
|
ref = saved.await_args.args[1]
|
||||||
|
assert ref.platform == "modelscope-ai"
|
||||||
|
assert ref.source_id == "referall13/EM1"
|
||||||
|
assert ref.url == "https://www.modelscope.ai/models/referall13/EM1"
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_download_model_source_defaults_to_huggingface(tmp_path, monkeypatch):
|
async def test_download_model_source_defaults_to_huggingface(tmp_path, monkeypatch):
|
||||||
"""The legacy /api/lm/download-hf-model payload has no `platform` key."""
|
"""The legacy /api/lm/download-hf-model payload has no `platform` key."""
|
||||||
@@ -609,18 +658,19 @@ async def test_save_source_metadata_writes_platform_fields(
|
|||||||
base_model="SDXL 1.0",
|
base_model="SDXL 1.0",
|
||||||
preview_url="",
|
preview_url="",
|
||||||
)
|
)
|
||||||
monkeypatch.setattr(
|
scanner = SimpleNamespace(
|
||||||
model_source_handlers.MetadataManager,
|
# A real scanner owns metadata creation (see the lazy-hash test below).
|
||||||
"create_default_metadata",
|
_create_default_metadata=AsyncMock(return_value=metadata),
|
||||||
AsyncMock(return_value=metadata),
|
add_model_to_cache=AsyncMock(),
|
||||||
)
|
)
|
||||||
scanner = SimpleNamespace(add_model_to_cache=AsyncMock())
|
|
||||||
monkeypatch.setattr(
|
monkeypatch.setattr(
|
||||||
ServiceRegistry, "get_lora_scanner", AsyncMock(return_value=scanner)
|
ServiceRegistry, "get_lora_scanner", AsyncMock(return_value=scanner)
|
||||||
)
|
)
|
||||||
monkeypatch.setattr(
|
monkeypatch.setattr(
|
||||||
model_source_handlers, "_infer_model_type", lambda _root: (LoraMetadata, "get_lora_scanner")
|
model_source_handlers, "_infer_model_type", lambda _root: (LoraMetadata, "get_lora_scanner")
|
||||||
)
|
)
|
||||||
|
hydrate = AsyncMock()
|
||||||
|
monkeypatch.setattr(model_source_handlers, "hydrate_from_source", hydrate)
|
||||||
|
|
||||||
ref = SourceRef(platform=platform, source_id="u/r", url=url)
|
ref = SourceRef(platform=platform, source_id="u/r", url=url)
|
||||||
await model_source_handlers._save_source_metadata(str(model_path), ref, str(tmp_path))
|
await model_source_handlers._save_source_metadata(str(model_path), ref, str(tmp_path))
|
||||||
@@ -630,6 +680,471 @@ async def test_save_source_metadata_writes_platform_fields(
|
|||||||
assert saved["source_url"] == url
|
assert saved["source_url"] == url
|
||||||
assert bool(saved.get("hf_url", "")) is expect_hf_alias
|
assert bool(saved.get("hf_url", "")) is expect_hf_alias
|
||||||
|
|
||||||
|
assert scanner._create_default_metadata.await_args.args == (str(model_path),)
|
||||||
|
|
||||||
cached = scanner.add_model_to_cache.await_args.args[0]
|
cached = scanner.add_model_to_cache.await_args.args[0]
|
||||||
assert cached["source_platform"] == platform
|
assert cached["source_platform"] == platform
|
||||||
assert cached["source_url"] == url
|
assert cached["source_url"] == url
|
||||||
|
|
||||||
|
# The site's own API is consulted last, so the scanner-cache refresh it
|
||||||
|
# performs lands on the entry created above.
|
||||||
|
assert hydrate.await_args.args == (str(model_path),)
|
||||||
|
assert hydrate.await_args.kwargs["ref"] == ref
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_checkpoint_download_defers_the_hash(tmp_path, monkeypatch):
|
||||||
|
"""A multi-GB checkpoint must not be hashed inside the download request.
|
||||||
|
|
||||||
|
``CheckpointScanner`` records ``hash_status="pending"`` and lets the hash be
|
||||||
|
computed on demand; going through the generic
|
||||||
|
``MetadataManager.create_default_metadata`` would read the whole file before
|
||||||
|
the download response could return, which is exactly the pause this code
|
||||||
|
path is supposed to avoid.
|
||||||
|
"""
|
||||||
|
from py.services.checkpoint_scanner import CheckpointScanner
|
||||||
|
from py.utils.models import CheckpointMetadata
|
||||||
|
|
||||||
|
model_path = tmp_path / "big_checkpoint.safetensors"
|
||||||
|
model_path.write_bytes(b"stub")
|
||||||
|
|
||||||
|
real_scanner = CheckpointScanner()
|
||||||
|
scanner = SimpleNamespace(
|
||||||
|
_create_default_metadata=real_scanner._create_default_metadata,
|
||||||
|
add_model_to_cache=AsyncMock(),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
ServiceRegistry, "get_checkpoint_scanner", AsyncMock(return_value=scanner)
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
model_source_handlers,
|
||||||
|
"_infer_model_type",
|
||||||
|
lambda _root: (CheckpointMetadata, "get_checkpoint_scanner"),
|
||||||
|
)
|
||||||
|
generic = AsyncMock(
|
||||||
|
# Stands in for the eager helper: if the handler reaches for it, the
|
||||||
|
# sidecar ends up hashed and the assertions below say so plainly.
|
||||||
|
return_value=LoraMetadata(
|
||||||
|
file_name="big_checkpoint",
|
||||||
|
model_name="big_checkpoint",
|
||||||
|
file_path=str(model_path),
|
||||||
|
size=4,
|
||||||
|
modified=1.0,
|
||||||
|
sha256="d" * 64,
|
||||||
|
base_model="Unknown",
|
||||||
|
preview_url="",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
model_source_handlers.MetadataManager, "create_default_metadata", generic
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(model_source_handlers, "hydrate_from_source", AsyncMock())
|
||||||
|
|
||||||
|
ref = SourceRef(
|
||||||
|
platform="huggingface",
|
||||||
|
source_id="u/r",
|
||||||
|
url="https://huggingface.co/u/r",
|
||||||
|
)
|
||||||
|
await model_source_handlers._save_source_metadata(str(model_path), ref, str(tmp_path))
|
||||||
|
|
||||||
|
saved = json.loads(open(_sidecar_path(model_path), encoding="utf-8").read())
|
||||||
|
assert saved["sha256"] == ""
|
||||||
|
assert saved["hash_status"] == "pending"
|
||||||
|
assert saved["from_civitai"] is False
|
||||||
|
# The download link is still recorded on top of the deferred hash.
|
||||||
|
assert saved["source_platform"] == "huggingface"
|
||||||
|
assert saved["source_url"] == "https://huggingface.co/u/r"
|
||||||
|
|
||||||
|
# The scanner cache must carry the pending state too, or the cache fill
|
||||||
|
# would compute the hash after all.
|
||||||
|
cached = scanner.add_model_to_cache.await_args.args[0]
|
||||||
|
assert cached["hash_status"] == "pending"
|
||||||
|
assert cached["sha256"] == ""
|
||||||
|
|
||||||
|
generic.assert_not_awaited()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Post-transfer phase reporting
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _stub_hydration_pipeline(tmp_path, monkeypatch):
|
||||||
|
"""Wire `_save_source_metadata`'s collaborators and record call order."""
|
||||||
|
model_path = tmp_path / "downloaded.safetensors"
|
||||||
|
model_path.write_bytes(b"x" * 32)
|
||||||
|
|
||||||
|
metadata = LoraMetadata(
|
||||||
|
file_name="downloaded",
|
||||||
|
model_name="Downloaded",
|
||||||
|
file_path=str(model_path),
|
||||||
|
size=32,
|
||||||
|
modified=1.0,
|
||||||
|
sha256="a" * 64,
|
||||||
|
base_model="SDXL 1.0",
|
||||||
|
preview_url="",
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
model_source_handlers.MetadataManager,
|
||||||
|
"create_default_metadata",
|
||||||
|
AsyncMock(return_value=metadata),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
ServiceRegistry,
|
||||||
|
"get_lora_scanner",
|
||||||
|
AsyncMock(return_value=SimpleNamespace(add_model_to_cache=AsyncMock())),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
model_source_handlers,
|
||||||
|
"_infer_model_type",
|
||||||
|
lambda _root: (LoraMetadata, "get_lora_scanner"),
|
||||||
|
)
|
||||||
|
|
||||||
|
events: list = []
|
||||||
|
|
||||||
|
async def fake_broadcast(download_id, data):
|
||||||
|
events.append(("broadcast", data["stage"], data, download_id))
|
||||||
|
|
||||||
|
async def fake_hydrate(*_args, **_kwargs):
|
||||||
|
events.append(("hydrate", None, None, None))
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
model_source_handlers.ws_manager, "broadcast_download_progress", fake_broadcast
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(model_source_handlers, "hydrate_from_source", fake_hydrate)
|
||||||
|
return model_path, events
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_save_source_metadata_reports_post_transfer_stages(tmp_path, monkeypatch):
|
||||||
|
"""The byte counter stops before indexing and the site fetch, so the UI has
|
||||||
|
to be told what is still running — otherwise the bar looks stuck."""
|
||||||
|
model_path, events = _stub_hydration_pipeline(tmp_path, monkeypatch)
|
||||||
|
|
||||||
|
ref = SourceRef(
|
||||||
|
platform="modelscope", source_id="u/r", url="https://modelscope.cn/models/u/r"
|
||||||
|
)
|
||||||
|
await model_source_handlers._save_source_metadata(
|
||||||
|
str(model_path), ref, str(tmp_path), download_id="dl-1"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Each stage is announced *before* its work starts, so the label is never
|
||||||
|
# describing something that already finished.
|
||||||
|
assert [event[:2] for event in events] == [
|
||||||
|
("broadcast", "indexing"),
|
||||||
|
("broadcast", "source"),
|
||||||
|
("hydrate", None),
|
||||||
|
]
|
||||||
|
for kind, stage, data, download_id in events:
|
||||||
|
if kind != "broadcast":
|
||||||
|
continue
|
||||||
|
assert download_id == "dl-1"
|
||||||
|
assert data["status"] == "metadata"
|
||||||
|
assert data["progress"] == 100
|
||||||
|
assert data["platform"] == "modelscope"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_save_source_metadata_is_silent_without_a_watcher(tmp_path, monkeypatch):
|
||||||
|
"""No `download_id` means no UI is watching; nothing should be broadcast."""
|
||||||
|
model_path, events = _stub_hydration_pipeline(tmp_path, monkeypatch)
|
||||||
|
|
||||||
|
ref = SourceRef(
|
||||||
|
platform="modelscope", source_id="u/r", url="https://modelscope.cn/models/u/r"
|
||||||
|
)
|
||||||
|
await model_source_handlers._save_source_metadata(str(model_path), ref, str(tmp_path))
|
||||||
|
|
||||||
|
assert events == [("hydrate", None, None, None)]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_report_phase_never_breaks_a_download(monkeypatch):
|
||||||
|
"""Progress reporting is cosmetic; a dead socket must not fail the file."""
|
||||||
|
monkeypatch.setattr(
|
||||||
|
model_source_handlers.ws_manager,
|
||||||
|
"broadcast_download_progress",
|
||||||
|
AsyncMock(side_effect=RuntimeError("socket gone")),
|
||||||
|
)
|
||||||
|
|
||||||
|
await model_source_handlers._report_phase("dl-1", "source", "modelscope")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_download_passes_its_watch_id_into_metadata_work(tmp_path, monkeypatch):
|
||||||
|
"""The stages are only visible if the handler hands its id down."""
|
||||||
|
_stub_download_backend(monkeypatch)
|
||||||
|
saved = AsyncMock()
|
||||||
|
monkeypatch.setattr(model_source_handlers, "_save_source_metadata", saved)
|
||||||
|
|
||||||
|
await ModelSourceHandler().download_model_source(
|
||||||
|
FakeRequest(
|
||||||
|
json_data={
|
||||||
|
"platform": "modelscope",
|
||||||
|
"repo": "owner/name",
|
||||||
|
"filename": "model.safetensors",
|
||||||
|
"model_root": str(tmp_path),
|
||||||
|
"download_id": "dl-42",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert saved.await_args.kwargs["download_id"] == "dl-42"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_skipped_download_still_reports_the_site_stage(tmp_path, monkeypatch):
|
||||||
|
"""An already-present file is hydrated too, so it needs the same signal."""
|
||||||
|
_stub_download_backend(monkeypatch)
|
||||||
|
hydrate = AsyncMock()
|
||||||
|
monkeypatch.setattr(model_source_handlers, "hydrate_from_source", hydrate)
|
||||||
|
broadcast = AsyncMock()
|
||||||
|
monkeypatch.setattr(
|
||||||
|
model_source_handlers.ws_manager, "broadcast_download_progress", broadcast
|
||||||
|
)
|
||||||
|
|
||||||
|
existing = tmp_path / "model.safetensors"
|
||||||
|
existing.write_bytes(b"x" * 32)
|
||||||
|
|
||||||
|
await ModelSourceHandler().download_model_source(
|
||||||
|
FakeRequest(
|
||||||
|
json_data={
|
||||||
|
"platform": "modelscope",
|
||||||
|
"repo": "owner/name",
|
||||||
|
"filename": "model.safetensors",
|
||||||
|
"model_root": str(tmp_path),
|
||||||
|
"download_id": "dl-7",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert broadcast.await_args.args[1]["stage"] == "source"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_save_source_metadata_survives_a_hydration_failure(tmp_path, monkeypatch):
|
||||||
|
"""Metadata hydration must never be able to fail a completed download."""
|
||||||
|
model_path = tmp_path / "downloaded.safetensors"
|
||||||
|
model_path.write_bytes(b"x" * 32)
|
||||||
|
|
||||||
|
metadata = LoraMetadata(
|
||||||
|
file_name="downloaded",
|
||||||
|
model_name="Downloaded",
|
||||||
|
file_path=str(model_path),
|
||||||
|
size=32,
|
||||||
|
modified=1.0,
|
||||||
|
sha256="a" * 64,
|
||||||
|
base_model="SDXL 1.0",
|
||||||
|
preview_url="",
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
model_source_handlers.MetadataManager,
|
||||||
|
"create_default_metadata",
|
||||||
|
AsyncMock(return_value=metadata),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
ServiceRegistry,
|
||||||
|
"get_lora_scanner",
|
||||||
|
AsyncMock(return_value=SimpleNamespace(add_model_to_cache=AsyncMock())),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
model_source_handlers, "_infer_model_type", lambda _root: (LoraMetadata, "get_lora_scanner")
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
model_source_handlers,
|
||||||
|
"hydrate_from_source",
|
||||||
|
AsyncMock(side_effect=RuntimeError("site down")),
|
||||||
|
)
|
||||||
|
|
||||||
|
ref = SourceRef(
|
||||||
|
platform="modelscope", source_id="u/r", url="https://modelscope.cn/models/u/r"
|
||||||
|
)
|
||||||
|
await model_source_handlers._save_source_metadata(str(model_path), ref, str(tmp_path))
|
||||||
|
|
||||||
|
saved = json.loads(open(_sidecar_path(model_path), encoding="utf-8").read())
|
||||||
|
assert saved["source_platform"] == "modelscope"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_downloading_an_existing_file_still_hydrates(tmp_path, monkeypatch):
|
||||||
|
"""A pre-existing file may still be missing the site's metadata."""
|
||||||
|
_stub_download_backend(monkeypatch)
|
||||||
|
saved = AsyncMock()
|
||||||
|
monkeypatch.setattr(model_source_handlers, "_save_source_metadata", saved)
|
||||||
|
hydrate = AsyncMock()
|
||||||
|
monkeypatch.setattr(model_source_handlers, "hydrate_from_source", hydrate)
|
||||||
|
|
||||||
|
existing = tmp_path / "model.safetensors"
|
||||||
|
existing.write_bytes(b"x" * 32)
|
||||||
|
|
||||||
|
response = await ModelSourceHandler().download_model_source(
|
||||||
|
FakeRequest(
|
||||||
|
json_data={
|
||||||
|
"platform": "modelscope",
|
||||||
|
"repo": "owner/name",
|
||||||
|
"filename": "model.safetensors",
|
||||||
|
"model_root": str(tmp_path),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert response.status == 200
|
||||||
|
saved.assert_not_awaited()
|
||||||
|
assert hydrate.await_args.args == (str(existing),)
|
||||||
|
assert hydrate.await_args.kwargs["ref"].source_id == "owner/name"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Download-time metadata hydration (end to end)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _modelscope_card_payload() -> dict:
|
||||||
|
"""A trimmed ModelScope model-detail response for the hydration test."""
|
||||||
|
|
||||||
|
return {
|
||||||
|
"Code": 200,
|
||||||
|
"Data": {
|
||||||
|
"Name": "Krea-2-LORA",
|
||||||
|
"ChineseName": "krea脸模",
|
||||||
|
"AigcType": "LoRA",
|
||||||
|
"Description": "权重0.5-1.2。配合《风格滤镜》lora一起使用。",
|
||||||
|
"BaseModel": ["krea/Krea-2-Turbo"],
|
||||||
|
"OfficialTags": [{"Tag": "photography"}, {"Tag": "woman"}],
|
||||||
|
"ModelInfos": {
|
||||||
|
"safetensor": {
|
||||||
|
"files": [
|
||||||
|
{
|
||||||
|
"name": "Krea-2-LORA_c1-st1000.safetensors",
|
||||||
|
"sha256": "a" * 64,
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"MuseInfo": {
|
||||||
|
"versions": [
|
||||||
|
{
|
||||||
|
"stats": {"fileList": ["Krea-2-LORA_c1-st1000.safetensors"]},
|
||||||
|
"modelVersion": {
|
||||||
|
"showName": "c1-st1000",
|
||||||
|
"triggerWords": '["kreaface","kreamodel"]',
|
||||||
|
},
|
||||||
|
"coverImages": [
|
||||||
|
{"url": "https://resources.modelscope.cn/cover-images/b.png"},
|
||||||
|
{"url": "https://resources.modelscope.cn/cover-images/c.png"},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_download_hydrates_the_card_from_the_site(tmp_path, monkeypatch):
|
||||||
|
"""A ModelScope download must land with a populated model card.
|
||||||
|
|
||||||
|
Only the network, the scanner and the file transfer are faked, so this
|
||||||
|
exercises the real handler, the real `ModelScopeSource` and the real
|
||||||
|
post-processor together. Breaking the wiring between them fails here even
|
||||||
|
when each half still passes its own unit tests.
|
||||||
|
"""
|
||||||
|
model_path = tmp_path / "Krea-2-LORA_c1-st1000.safetensors"
|
||||||
|
|
||||||
|
async def fake_download_file(**kwargs):
|
||||||
|
with open(kwargs["save_path"], "wb") as handle:
|
||||||
|
handle.write(b"stub")
|
||||||
|
return True, kwargs["save_path"]
|
||||||
|
|
||||||
|
class _Downloader:
|
||||||
|
download_file = staticmethod(fake_download_file)
|
||||||
|
|
||||||
|
class _Settings:
|
||||||
|
def get(self, key, default=None):
|
||||||
|
return default
|
||||||
|
|
||||||
|
async def fake_get_downloader():
|
||||||
|
return _Downloader()
|
||||||
|
|
||||||
|
monkeypatch.setattr(model_source_handlers, "get_downloader", fake_get_downloader)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
model_source_handlers, "get_settings_manager", lambda: _Settings()
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
model_source_handlers,
|
||||||
|
"_infer_model_type",
|
||||||
|
lambda _root: (LoraMetadata, "get_lora_scanner"),
|
||||||
|
)
|
||||||
|
|
||||||
|
scanner = SimpleNamespace(
|
||||||
|
get_cached_data=AsyncMock(
|
||||||
|
return_value=SimpleNamespace(raw_data=[{"file_path": str(model_path)}])
|
||||||
|
),
|
||||||
|
add_model_to_cache=AsyncMock(),
|
||||||
|
update_single_model_cache=AsyncMock(),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
ServiceRegistry, "get_lora_scanner", AsyncMock(return_value=scanner)
|
||||||
|
)
|
||||||
|
|
||||||
|
async def fake_fetch_text(url, **_kwargs):
|
||||||
|
return "# Krea-2-LORA\n\n权重0.5-1.2。"
|
||||||
|
|
||||||
|
async def fake_fetch_json(url, **_kwargs):
|
||||||
|
return 200, _modelscope_card_payload()
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"py.services.model_sources.modelscope.fetch_text", fake_fetch_text
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"py.services.model_sources.modelscope.fetch_json", fake_fetch_json
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"py.metadata_ops.list_base_models", AsyncMock(return_value=["Krea 2"])
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"py.metadata_ops.download_preview",
|
||||||
|
AsyncMock(return_value=str(tmp_path / "preview.webp")),
|
||||||
|
)
|
||||||
|
|
||||||
|
response = await ModelSourceHandler().download_model_source(
|
||||||
|
FakeRequest(
|
||||||
|
json_data={
|
||||||
|
"platform": "modelscope",
|
||||||
|
"repo": "jj3550945163/Krea-2-LORA",
|
||||||
|
"filename": "Krea-2-LORA_c1-st1000.safetensors",
|
||||||
|
"model_root": str(tmp_path),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert response.status == 200
|
||||||
|
saved = json.loads(open(_sidecar_path(model_path), encoding="utf-8").read())
|
||||||
|
|
||||||
|
# The download's own provenance is unchanged.
|
||||||
|
assert saved["source_platform"] == "modelscope"
|
||||||
|
assert saved["source_url"] == "https://modelscope.cn/models/jj3550945163/Krea-2-LORA"
|
||||||
|
assert saved["from_civitai"] is False
|
||||||
|
|
||||||
|
# The site's published metadata, with no LLM involved.
|
||||||
|
assert saved["model_name"] == "Krea-2-LORA"
|
||||||
|
assert saved["base_model"] == "Krea 2"
|
||||||
|
assert saved["tags"] == ["photography", "woman"]
|
||||||
|
assert saved["civitai"]["name"] == "c1-st1000"
|
||||||
|
assert saved["civitai"]["trainedWords"] == ["kreaface", "kreamodel"]
|
||||||
|
assert saved["civitai"]["description"] == "权重0.5-1.2。配合《风格滤镜》lora一起使用。"
|
||||||
|
assert [img["url"] for img in saved["civitai"]["images"]] == [
|
||||||
|
"https://resources.modelscope.cn/cover-images/b.png",
|
||||||
|
"https://resources.modelscope.cn/cover-images/c.png",
|
||||||
|
]
|
||||||
|
assert saved["preview_url"] == str(tmp_path / "preview.webp")
|
||||||
|
assert saved["usage_tips"] == (
|
||||||
|
'{"strength_min": 0.5, "strength_max": 1.2, "strength_range": "0.5-1.2"}'
|
||||||
|
)
|
||||||
|
assert saved["metadata_source"] == "source:modelscope"
|
||||||
|
# No provider answered, so claiming an AI enrichment would be a lie.
|
||||||
|
assert "llm_enriched_at" not in saved
|
||||||
|
|
||||||
|
# The enriched card reaches the scanner cache, not just the file.
|
||||||
|
assert scanner.update_single_model_cache.await_count == 1
|
||||||
|
cached = scanner.update_single_model_cache.await_args.args[2]
|
||||||
|
assert cached["model_name"] == "Krea-2-LORA"
|
||||||
|
|||||||
@@ -59,6 +59,17 @@ class TestDetectSource:
|
|||||||
"modelscope",
|
"modelscope",
|
||||||
"jj3550945163/Krea-2-LORA",
|
"jj3550945163/Krea-2-LORA",
|
||||||
),
|
),
|
||||||
|
# modelscope.ai is a separate catalogue with its own platform id.
|
||||||
|
(
|
||||||
|
"https://www.modelscope.ai/models/referall13/EM1",
|
||||||
|
"modelscope-ai",
|
||||||
|
"referall13/EM1",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"https://modelscope.ai/models/ErLubu/krea2_style_260911_02/summary",
|
||||||
|
"modelscope-ai",
|
||||||
|
"ErLubu/krea2_style_260911_02",
|
||||||
|
),
|
||||||
(
|
(
|
||||||
"https://tensor.art/models/827823520299086029/Vivid-Impressions-Storybook-Sstyle-V1.0",
|
"https://tensor.art/models/827823520299086029/Vivid-Impressions-Storybook-Sstyle-V1.0",
|
||||||
"tensorart",
|
"tensorart",
|
||||||
@@ -97,6 +108,21 @@ class TestDetectSource:
|
|||||||
== "https://tensor.art/models/123"
|
== "https://tensor.art/models/123"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def test_modelscope_com_is_an_alias_of_the_mainland_site(self):
|
||||||
|
"""``.com`` 301-redirects to ``.cn``, so it is not a third catalogue."""
|
||||||
|
ref = detect_source("https://www.modelscope.com/models/u/r")
|
||||||
|
assert ref.platform == "modelscope"
|
||||||
|
assert ref.url == "https://modelscope.cn/models/u/r"
|
||||||
|
|
||||||
|
def test_the_two_modelscope_catalogues_do_not_cross_match(self):
|
||||||
|
"""A host must never be accepted by the other deployment's patterns."""
|
||||||
|
mainland = get_source("modelscope")
|
||||||
|
international = get_source("modelscope-ai")
|
||||||
|
|
||||||
|
assert mainland.parse("https://www.modelscope.ai/models/u/r") is None
|
||||||
|
assert international.parse("https://modelscope.cn/models/u/r") is None
|
||||||
|
assert international.parse("https://www.modelscope.com/models/u/r") is None
|
||||||
|
|
||||||
|
|
||||||
class TestStrictParsing:
|
class TestStrictParsing:
|
||||||
@pytest.mark.parametrize(
|
@pytest.mark.parametrize(
|
||||||
@@ -106,6 +132,8 @@ class TestStrictParsing:
|
|||||||
"https://huggingface.co/user/repo/",
|
"https://huggingface.co/user/repo/",
|
||||||
"https://modelscope.cn/models/user/repo",
|
"https://modelscope.cn/models/user/repo",
|
||||||
"https://modelscope.cn/models/user/repo/summary",
|
"https://modelscope.cn/models/user/repo/summary",
|
||||||
|
"https://www.modelscope.ai/models/user/repo",
|
||||||
|
"https://www.modelscope.ai/models/user/repo/files",
|
||||||
"https://tensor.art/models/827823520299086029",
|
"https://tensor.art/models/827823520299086029",
|
||||||
"https://tensor.art/models/827823520299086029/Vivid-Impressions",
|
"https://tensor.art/models/827823520299086029/Vivid-Impressions",
|
||||||
],
|
],
|
||||||
@@ -145,6 +173,16 @@ class TestCapabilities:
|
|||||||
assert source.default_revision == "master"
|
assert source.default_revision == "master"
|
||||||
assert source.default_subdir == "modelscope"
|
assert source.default_subdir == "modelscope"
|
||||||
|
|
||||||
|
def test_modelscope_intl_is_the_same_site_on_another_catalogue(self):
|
||||||
|
source = get_source("modelscope-ai")
|
||||||
|
assert source.supports_enrichment is True
|
||||||
|
assert source.supports_download is True
|
||||||
|
assert source.default_revision == "master"
|
||||||
|
# A distinct directory: the same owner/name can exist on both
|
||||||
|
# deployments with different content.
|
||||||
|
assert source.default_subdir == "modelscope-ai"
|
||||||
|
assert source.base_url == "https://www.modelscope.ai"
|
||||||
|
|
||||||
def test_tensorart_is_link_only(self):
|
def test_tensorart_is_link_only(self):
|
||||||
source = get_source("tensorart")
|
source = get_source("tensorart")
|
||||||
assert source.supports_enrichment is False
|
assert source.supports_enrichment is False
|
||||||
@@ -152,11 +190,17 @@ class TestCapabilities:
|
|||||||
|
|
||||||
def test_registry_lists_every_source(self):
|
def test_registry_lists_every_source(self):
|
||||||
platforms = {s.platform for s in list_sources()}
|
platforms = {s.platform for s in list_sources()}
|
||||||
assert platforms == {"huggingface", "modelscope", "tensorart"}
|
assert platforms == {
|
||||||
|
"huggingface",
|
||||||
|
"modelscope",
|
||||||
|
"modelscope-ai",
|
||||||
|
"tensorart",
|
||||||
|
}
|
||||||
|
|
||||||
def test_labels_are_brand_names(self):
|
def test_labels_are_brand_names(self):
|
||||||
assert source_label("huggingface") == "Hugging Face"
|
assert source_label("huggingface") == "Hugging Face"
|
||||||
assert source_label("modelscope") == "ModelScope"
|
assert source_label("modelscope") == "ModelScope"
|
||||||
|
assert source_label("modelscope-ai") == "ModelScope (International)"
|
||||||
assert source_label("tensorart") == "TensorArt"
|
assert source_label("tensorart") == "TensorArt"
|
||||||
assert source_label("unknown", "fallback") == "fallback"
|
assert source_label("unknown", "fallback") == "fallback"
|
||||||
|
|
||||||
@@ -311,6 +355,44 @@ class TestFetchModelCard:
|
|||||||
in calls
|
in calls
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_modelscope_intl_fetches_from_its_own_catalogue(self, monkeypatch):
|
||||||
|
"""The mainland site 404s for a `.ai`-only repository, so every fetch
|
||||||
|
has to stay on the host the URL came from."""
|
||||||
|
calls: list[str] = []
|
||||||
|
|
||||||
|
async def fake_fetch_text(url: str, **_kwargs) -> str:
|
||||||
|
calls.append(url)
|
||||||
|
return "# card"
|
||||||
|
|
||||||
|
json_calls: list[str] = []
|
||||||
|
|
||||||
|
async def fake_fetch_json(url: str, **_kwargs):
|
||||||
|
json_calls.append(url)
|
||||||
|
return 200, {"Data": {"Name": "EM1", "MuseInfo": {"versions": []}}}
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"py.services.model_sources.modelscope.fetch_text", fake_fetch_text
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"py.services.model_sources.modelscope.fetch_json", fake_fetch_json
|
||||||
|
)
|
||||||
|
|
||||||
|
source = get_source("modelscope-ai")
|
||||||
|
await source.fetch_model_card("referall13/EM1")
|
||||||
|
await source.fetch_model_card_context("referall13/EM1")
|
||||||
|
await source.list_files("referall13/EM1")
|
||||||
|
|
||||||
|
assert calls == [
|
||||||
|
"https://www.modelscope.ai/models/referall13/EM1/resolve/master/README.md"
|
||||||
|
]
|
||||||
|
assert json_calls == [
|
||||||
|
"https://www.modelscope.ai/api/v1/models/referall13/EM1",
|
||||||
|
"https://www.modelscope.ai/api/v1/models/referall13/EM1/repo/files"
|
||||||
|
"?Revision=master",
|
||||||
|
]
|
||||||
|
assert not any("modelscope.cn" in url for url in calls + json_calls)
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_tensorart_never_fetches(self):
|
async def test_tensorart_never_fetches(self):
|
||||||
# TensorArt enrichment is disabled: the provider must not issue any
|
# TensorArt enrichment is disabled: the provider must not issue any
|
||||||
@@ -335,6 +417,12 @@ class TestAssetBaseUrl:
|
|||||||
== "https://modelscope.cn/models/u/r/resolve/master"
|
== "https://modelscope.cn/models/u/r/resolve/master"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def test_modelscope_intl_uses_master_revision(self):
|
||||||
|
assert (
|
||||||
|
get_source("modelscope-ai").asset_base_url("u/r")
|
||||||
|
== "https://www.modelscope.ai/models/u/r/resolve/master"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Model card context (site extras kept outside the README)
|
# Model card context (site extras kept outside the README)
|
||||||
@@ -354,6 +442,9 @@ def _modelscope_detail_payload() -> dict:
|
|||||||
"Data": {
|
"Data": {
|
||||||
"Name": "Krea-2-LORA",
|
"Name": "Krea-2-LORA",
|
||||||
"ChineseName": "krea脸模",
|
"ChineseName": "krea脸模",
|
||||||
|
"AigcType": "LoRA",
|
||||||
|
"License": "Apache License 2.0",
|
||||||
|
"Tags": ["LoRA", "text-to-image", "portrait"],
|
||||||
"Description": "权重0.5-1.2。配合《风格滤镜》lora一起使用。",
|
"Description": "权重0.5-1.2。配合《风格滤镜》lora一起使用。",
|
||||||
"BaseModel": ["krea/Krea-2-Turbo"],
|
"BaseModel": ["krea/Krea-2-Turbo"],
|
||||||
"License": "Apache License 2.0",
|
"License": "Apache License 2.0",
|
||||||
@@ -422,6 +513,81 @@ class TestFetchModelCardContext:
|
|||||||
# OfficialTag values only, de-duplicated, order preserved.
|
# OfficialTag values only, de-duplicated, order preserved.
|
||||||
assert context.official_tags == ["photography", "woman"]
|
assert context.official_tags == ["photography", "woman"]
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_modelscope_reads_site_identity_fields(self, monkeypatch):
|
||||||
|
async def fake_fetch_json(url, **_kwargs):
|
||||||
|
return 200, _modelscope_detail_payload()
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"py.services.model_sources.modelscope.fetch_json", fake_fetch_json
|
||||||
|
)
|
||||||
|
|
||||||
|
context = await ModelScopeSource().fetch_model_card_context(
|
||||||
|
"u/r", "Krea-2-LORA_c1-st1000.safetensors"
|
||||||
|
)
|
||||||
|
|
||||||
|
assert context.model_name == "Krea-2-LORA"
|
||||||
|
assert context.model_name_localized == "krea脸模"
|
||||||
|
assert context.license == "Apache License 2.0"
|
||||||
|
assert context.model_type == "LoRA"
|
||||||
|
# The version label is taken from the file that was matched, not from
|
||||||
|
# whichever version happens to come first in the payload.
|
||||||
|
assert context.version_name == "c1-st1000"
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_modelscope_version_label_is_empty_for_an_unknown_file(
|
||||||
|
self, monkeypatch
|
||||||
|
):
|
||||||
|
async def fake_fetch_json(url, **_kwargs):
|
||||||
|
return 200, _modelscope_detail_payload()
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"py.services.model_sources.modelscope.fetch_json", fake_fetch_json
|
||||||
|
)
|
||||||
|
|
||||||
|
context = await ModelScopeSource().fetch_model_card_context(
|
||||||
|
"u/r", "other.safetensors"
|
||||||
|
)
|
||||||
|
|
||||||
|
assert context.version_name == ""
|
||||||
|
# The repository-wide fields are still published.
|
||||||
|
assert context.model_name == "Krea-2-LORA"
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_modelscope_falls_back_to_plain_tags(self, monkeypatch):
|
||||||
|
"""An empty ``OfficialTags`` must not mean "no tags at all".
|
||||||
|
|
||||||
|
The plain ``Tags`` list mixes genuine content tags with library and
|
||||||
|
task categories; the latter are dropped so the card is not tagged
|
||||||
|
"lora" / "text-to-image".
|
||||||
|
"""
|
||||||
|
payload = _modelscope_detail_payload()
|
||||||
|
payload["Data"]["OfficialTags"] = None
|
||||||
|
|
||||||
|
async def fake_fetch_json(url, **_kwargs):
|
||||||
|
return 200, payload
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"py.services.model_sources.modelscope.fetch_json", fake_fetch_json
|
||||||
|
)
|
||||||
|
|
||||||
|
context = await ModelScopeSource().fetch_model_card_context("u/r")
|
||||||
|
|
||||||
|
assert context.official_tags == ["portrait"]
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_modelscope_curated_tags_win_over_plain_tags(self, monkeypatch):
|
||||||
|
async def fake_fetch_json(url, **_kwargs):
|
||||||
|
return 200, _modelscope_detail_payload()
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"py.services.model_sources.modelscope.fetch_json", fake_fetch_json
|
||||||
|
)
|
||||||
|
|
||||||
|
context = await ModelScopeSource().fetch_model_card_context("u/r")
|
||||||
|
|
||||||
|
assert "portrait" not in context.official_tags
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_modelscope_matches_example_images_by_filename(self, monkeypatch):
|
async def test_modelscope_matches_example_images_by_filename(self, monkeypatch):
|
||||||
async def fake_fetch_json(url, **_kwargs):
|
async def fake_fetch_json(url, **_kwargs):
|
||||||
@@ -655,6 +821,22 @@ class TestDownloadUrls:
|
|||||||
"https://modelscope.cn/models/u/r/resolve/master/sub/f.safetensors"
|
"https://modelscope.cn/models/u/r/resolve/master/sub/f.safetensors"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def test_modelscope_intl_builds_every_url_on_its_own_host(self):
|
||||||
|
"""The two deployments serve different catalogues, so a URL built for
|
||||||
|
one must never point at the other."""
|
||||||
|
source = get_source("modelscope-ai")
|
||||||
|
|
||||||
|
assert source.canonical_url("u/r") == "https://www.modelscope.ai/models/u/r"
|
||||||
|
assert source.file_download_url("u/r", "sub/f.safetensors") == (
|
||||||
|
"https://www.modelscope.ai/models/u/r/resolve/master/sub/f.safetensors"
|
||||||
|
)
|
||||||
|
assert source.asset_base_url("u/r") == (
|
||||||
|
"https://www.modelscope.ai/models/u/r/resolve/master"
|
||||||
|
)
|
||||||
|
assert source.page_url_for_file("u/r", "sub/f.safetensors") == (
|
||||||
|
"https://www.modelscope.ai/models/u/r/file/view/master/sub/f.safetensors"
|
||||||
|
)
|
||||||
|
|
||||||
def test_explicit_revision_wins(self):
|
def test_explicit_revision_wins(self):
|
||||||
assert ModelScopeSource().file_download_url("u/r", "f.bin", "v1") == (
|
assert ModelScopeSource().file_download_url("u/r", "f.bin", "v1") == (
|
||||||
"https://modelscope.cn/models/u/r/resolve/v1/f.bin"
|
"https://modelscope.cn/models/u/r/resolve/v1/f.bin"
|
||||||
@@ -701,12 +883,13 @@ class TestSourceIdValidation:
|
|||||||
class TestDownloadSourceRegistry:
|
class TestDownloadSourceRegistry:
|
||||||
def test_downloadable_sources_excludes_link_only_sites(self):
|
def test_downloadable_sources_excludes_link_only_sites(self):
|
||||||
platforms = {source.platform for source in downloadable_sources()}
|
platforms = {source.platform for source in downloadable_sources()}
|
||||||
assert platforms == {"huggingface", "modelscope"}
|
assert platforms == {"huggingface", "modelscope", "modelscope-ai"}
|
||||||
|
|
||||||
def test_get_download_source_rejects_link_only_platform(self):
|
def test_get_download_source_rejects_link_only_platform(self):
|
||||||
assert get_download_source("tensorart") is None
|
assert get_download_source("tensorart") is None
|
||||||
assert get_download_source("nope") is None
|
assert get_download_source("nope") is None
|
||||||
assert get_download_source("modelscope").platform == "modelscope"
|
assert get_download_source("modelscope").platform == "modelscope"
|
||||||
|
assert get_download_source("modelscope-ai").platform == "modelscope-ai"
|
||||||
assert get_download_source("huggingface").platform == "huggingface"
|
assert get_download_source("huggingface").platform == "huggingface"
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,397 @@
|
|||||||
|
"""Tests for download-time metadata hydration.
|
||||||
|
|
||||||
|
`py/services/model_sources/hydration.py` is the deterministic counterpart of
|
||||||
|
the `enrich_hf_metadata` skill: it turns a freshly downloaded ModelScope /
|
||||||
|
Hugging Face file into the populated model card a CivitAI download produces,
|
||||||
|
without an LLM and without the user running anything.
|
||||||
|
|
||||||
|
These tests cover the orchestration — which source data is fetched, what is
|
||||||
|
handed to the post-processor, and that nothing here can fail a download. The
|
||||||
|
field-by-field mapping lives in `tests/services/test_post_processor.py`.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from py.services.model_sources import ModelCardContext, ModelSourceCache, SourceRef
|
||||||
|
from py.services.model_sources import hydration
|
||||||
|
from py.services.model_sources.base import ModelSource
|
||||||
|
from py.services.model_sources.hydration import (
|
||||||
|
SHARED_CACHE_MAX_ENTRIES,
|
||||||
|
hydrate_from_source,
|
||||||
|
load_model_card,
|
||||||
|
reset_shared_caches,
|
||||||
|
resolve_site_base_model,
|
||||||
|
shared_source_cache,
|
||||||
|
)
|
||||||
|
|
||||||
|
REF = SourceRef(
|
||||||
|
platform="modelscope",
|
||||||
|
source_id="user/repo",
|
||||||
|
url="https://modelscope.cn/models/user/repo",
|
||||||
|
)
|
||||||
|
|
||||||
|
SIDECAR = {
|
||||||
|
"sha256": "a" * 64,
|
||||||
|
"base_model": "Unknown",
|
||||||
|
# Written by the download handler just before hydration runs.
|
||||||
|
"source_platform": "modelscope",
|
||||||
|
"source_url": "https://modelscope.cn/models/user/repo",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeSource(ModelSource):
|
||||||
|
"""Minimal provider that records what hydration asked of it."""
|
||||||
|
|
||||||
|
platform = "modelscope"
|
||||||
|
label = "ModelScope"
|
||||||
|
supports_enrichment = True
|
||||||
|
|
||||||
|
def __init__(self, *, context=None, readme="", fail=False):
|
||||||
|
self.context = context if context is not None else ModelCardContext()
|
||||||
|
self.readme = readme
|
||||||
|
self.fail = fail
|
||||||
|
self.readme_calls = 0
|
||||||
|
self.context_calls = 0
|
||||||
|
self.context_kwargs: dict = {}
|
||||||
|
|
||||||
|
async def fetch_model_card(self, source_id):
|
||||||
|
self.readme_calls += 1
|
||||||
|
if self.fail:
|
||||||
|
raise RuntimeError("network down")
|
||||||
|
return self.readme
|
||||||
|
|
||||||
|
async def fetch_model_card_context(
|
||||||
|
self, source_id, filename="", *, sha256="", cache=None
|
||||||
|
):
|
||||||
|
self.context_calls += 1
|
||||||
|
self.context_kwargs = {"filename": filename, "sha256": sha256}
|
||||||
|
if self.fail:
|
||||||
|
raise RuntimeError("network down")
|
||||||
|
return self.context
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def _isolated_shared_caches():
|
||||||
|
reset_shared_caches()
|
||||||
|
yield
|
||||||
|
reset_shared_caches()
|
||||||
|
|
||||||
|
|
||||||
|
def _async(value):
|
||||||
|
async def _call(*_args, **_kwargs):
|
||||||
|
return value
|
||||||
|
|
||||||
|
return _call
|
||||||
|
|
||||||
|
|
||||||
|
def _wire(monkeypatch, source, *, metadata=SIDECAR, result=None):
|
||||||
|
"""Patch hydration's collaborators; return the recorded process() calls."""
|
||||||
|
|
||||||
|
monkeypatch.setattr(hydration, "get_source", lambda _platform: source)
|
||||||
|
monkeypatch.setattr("py.metadata_ops.read_metadata", _async(metadata))
|
||||||
|
|
||||||
|
calls: list = []
|
||||||
|
|
||||||
|
class _Processor:
|
||||||
|
async def process(self, **kwargs):
|
||||||
|
calls.append(kwargs)
|
||||||
|
if result is not None:
|
||||||
|
return result
|
||||||
|
return {"success": True, "updated_fields": ["model_name"]}
|
||||||
|
|
||||||
|
monkeypatch.setattr("py.services.agent.post_processor.PostProcessor", _Processor)
|
||||||
|
return calls
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Orchestration
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestHydrateFromSource:
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_applies_the_site_card_without_an_llm(self, monkeypatch):
|
||||||
|
source = _FakeSource(
|
||||||
|
context=ModelCardContext(
|
||||||
|
model_name="Krea-2-LORA",
|
||||||
|
version_name="c1-st1000",
|
||||||
|
description="权重0.5-1.2。",
|
||||||
|
official_tags=["photography"],
|
||||||
|
),
|
||||||
|
readme="# Krea-2-LORA",
|
||||||
|
)
|
||||||
|
calls = _wire(monkeypatch, source)
|
||||||
|
|
||||||
|
updated = await hydrate_from_source("/models/lora.safetensors", ref=REF)
|
||||||
|
|
||||||
|
assert updated == ["model_name"]
|
||||||
|
assert len(calls) == 1
|
||||||
|
call = calls[0]
|
||||||
|
# No provider is consulted: everything applied is what the site published.
|
||||||
|
assert call["llm_output"] == {}
|
||||||
|
assert call["skill_name"] == "enrich_hf_metadata"
|
||||||
|
assert call["readme_content"] == "# Krea-2-LORA"
|
||||||
|
assert call["source_context"].model_name == "Krea-2-LORA"
|
||||||
|
assert call["metadata_source"] == "source:modelscope"
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_matches_the_file_by_hash_and_basename(self, monkeypatch):
|
||||||
|
source = _FakeSource(context=ModelCardContext(model_name="X"))
|
||||||
|
_wire(monkeypatch, source)
|
||||||
|
|
||||||
|
await hydrate_from_source("/models/sub/Krea-2-LORA_c1-st1000.safetensors", ref=REF)
|
||||||
|
|
||||||
|
assert source.context_kwargs == {
|
||||||
|
"filename": "Krea-2-LORA_c1-st1000.safetensors",
|
||||||
|
"sha256": "a" * 64,
|
||||||
|
}
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_returns_early_for_an_unknown_platform(self, monkeypatch):
|
||||||
|
source = _FakeSource(context=ModelCardContext(model_name="X"))
|
||||||
|
calls = _wire(monkeypatch, source)
|
||||||
|
monkeypatch.setattr(hydration, "get_source", lambda _platform: None)
|
||||||
|
|
||||||
|
assert await hydrate_from_source("/models/lora.safetensors", ref=REF) == []
|
||||||
|
assert calls == []
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_returns_early_for_a_link_only_source(self, monkeypatch):
|
||||||
|
source = _FakeSource(context=ModelCardContext(model_name="X"))
|
||||||
|
source.supports_enrichment = False
|
||||||
|
calls = _wire(monkeypatch, source)
|
||||||
|
|
||||||
|
assert await hydrate_from_source("/models/lora.safetensors", ref=REF) == []
|
||||||
|
assert calls == []
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_returns_early_without_a_sidecar(self, monkeypatch):
|
||||||
|
source = _FakeSource(context=ModelCardContext(model_name="X"))
|
||||||
|
calls = _wire(monkeypatch, source, metadata={})
|
||||||
|
|
||||||
|
assert await hydrate_from_source("/models/lora.safetensors", ref=REF) == []
|
||||||
|
assert calls == []
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_returns_early_when_the_site_published_nothing(self, monkeypatch):
|
||||||
|
source = _FakeSource(context=ModelCardContext(), readme="")
|
||||||
|
calls = _wire(monkeypatch, source)
|
||||||
|
|
||||||
|
assert await hydrate_from_source("/models/lora.safetensors", ref=REF) == []
|
||||||
|
assert calls == []
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_a_deferred_hash_still_matches_by_filename(self, monkeypatch):
|
||||||
|
"""Checkpoints and other large files are stored with
|
||||||
|
``hash_status="pending"`` and an empty ``sha256`` (see
|
||||||
|
``CheckpointScanner._create_default_metadata``), so hydration has to
|
||||||
|
work from the filename alone."""
|
||||||
|
source = _FakeSource(context=ModelCardContext(model_name="X"))
|
||||||
|
calls = _wire(
|
||||||
|
monkeypatch,
|
||||||
|
source,
|
||||||
|
metadata={**SIDECAR, "sha256": "", "hash_status": "pending"},
|
||||||
|
)
|
||||||
|
|
||||||
|
await hydrate_from_source("/models/big_checkpoint.safetensors", ref=REF)
|
||||||
|
|
||||||
|
assert calls[0]["source_context"].model_name == "X"
|
||||||
|
assert source.context_kwargs == {
|
||||||
|
"filename": "big_checkpoint.safetensors",
|
||||||
|
"sha256": "",
|
||||||
|
}
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_returns_early_when_the_model_is_not_linked(self, monkeypatch):
|
||||||
|
"""A file that merely shares a name must not get another model's card."""
|
||||||
|
source = _FakeSource(context=ModelCardContext(model_name="X"))
|
||||||
|
calls = _wire(
|
||||||
|
monkeypatch, source, metadata={"sha256": "a" * 64, "base_model": "Unknown"}
|
||||||
|
)
|
||||||
|
|
||||||
|
assert await hydrate_from_source("/models/lora.safetensors", ref=REF) == []
|
||||||
|
assert calls == []
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_returns_early_when_linked_to_another_repository(self, monkeypatch):
|
||||||
|
source = _FakeSource(context=ModelCardContext(model_name="X"))
|
||||||
|
calls = _wire(
|
||||||
|
monkeypatch,
|
||||||
|
source,
|
||||||
|
metadata={
|
||||||
|
**SIDECAR,
|
||||||
|
"source_url": "https://modelscope.cn/models/user/other",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert await hydrate_from_source("/models/lora.safetensors", ref=REF) == []
|
||||||
|
assert calls == []
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_returns_early_when_linked_to_another_platform(self, monkeypatch):
|
||||||
|
source = _FakeSource(context=ModelCardContext(model_name="X"))
|
||||||
|
calls = _wire(
|
||||||
|
monkeypatch,
|
||||||
|
source,
|
||||||
|
metadata={
|
||||||
|
"sha256": "a" * 64,
|
||||||
|
"source_platform": "huggingface",
|
||||||
|
"source_url": "https://huggingface.co/user/repo",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert await hydrate_from_source("/models/lora.safetensors", ref=REF) == []
|
||||||
|
assert calls == []
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_readme_alone_is_enough_to_run(self, monkeypatch):
|
||||||
|
source = _FakeSource(context=ModelCardContext(), readme="# hi")
|
||||||
|
calls = _wire(monkeypatch, source)
|
||||||
|
|
||||||
|
await hydrate_from_source("/models/lora.safetensors", ref=REF)
|
||||||
|
|
||||||
|
assert len(calls) == 1
|
||||||
|
assert calls[0]["readme_content"] == "# hi"
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_a_failing_site_never_breaks_the_download(self, monkeypatch):
|
||||||
|
source = _FakeSource(fail=True)
|
||||||
|
calls = _wire(monkeypatch, source)
|
||||||
|
|
||||||
|
assert await hydrate_from_source("/models/lora.safetensors", ref=REF) == []
|
||||||
|
assert calls == []
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_a_failing_post_processor_never_breaks_the_download(
|
||||||
|
self, monkeypatch
|
||||||
|
):
|
||||||
|
source = _FakeSource(context=ModelCardContext(model_name="X"))
|
||||||
|
_wire(monkeypatch, source, result={"success": False, "errors": ["boom"]})
|
||||||
|
|
||||||
|
assert await hydrate_from_source("/models/lora.safetensors", ref=REF) == []
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_updates_are_reported_for_logging(self, monkeypatch):
|
||||||
|
source = _FakeSource(context=ModelCardContext(model_name="X"))
|
||||||
|
_wire(
|
||||||
|
monkeypatch,
|
||||||
|
source,
|
||||||
|
result={"success": True, "updated_fields": ["tags", "civitai"]},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert await hydrate_from_source("/models/lora.safetensors", ref=REF) == [
|
||||||
|
"tags",
|
||||||
|
"civitai",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Per-repository memo
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestSharedSourceCache:
|
||||||
|
def test_same_repository_reuses_one_memo(self):
|
||||||
|
assert shared_source_cache("modelscope", "u/r") is shared_source_cache(
|
||||||
|
"modelscope", "u/r"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_different_repositories_get_different_memos(self):
|
||||||
|
assert shared_source_cache("modelscope", "u/r") is not shared_source_cache(
|
||||||
|
"modelscope", "u/other"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_entry_expires(self, monkeypatch):
|
||||||
|
clock = {"now": 1000.0}
|
||||||
|
monkeypatch.setattr(hydration.time, "monotonic", lambda: clock["now"])
|
||||||
|
|
||||||
|
first = shared_source_cache("modelscope", "u/r")
|
||||||
|
clock["now"] += hydration.SHARED_CACHE_TTL + 1
|
||||||
|
|
||||||
|
assert shared_source_cache("modelscope", "u/r") is not first
|
||||||
|
|
||||||
|
def test_cache_is_bounded(self):
|
||||||
|
for index in range(SHARED_CACHE_MAX_ENTRIES + 5):
|
||||||
|
shared_source_cache("modelscope", f"u/r{index}")
|
||||||
|
|
||||||
|
assert len(hydration._shared_caches) == SHARED_CACHE_MAX_ENTRIES
|
||||||
|
|
||||||
|
|
||||||
|
class TestLoadModelCard:
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_successful_read_is_memoised(self):
|
||||||
|
source = _FakeSource(readme="# hi")
|
||||||
|
cache = ModelSourceCache()
|
||||||
|
|
||||||
|
assert await load_model_card(source, "u/r", cache) == "# hi"
|
||||||
|
assert await load_model_card(source, "u/r", cache) == "# hi"
|
||||||
|
assert source.readme_calls == 1
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_empty_read_is_retried(self):
|
||||||
|
"""A transient failure must not be cached as "this repo has no card"."""
|
||||||
|
source = _FakeSource(readme="")
|
||||||
|
cache = ModelSourceCache()
|
||||||
|
|
||||||
|
await load_model_card(source, "u/r", cache)
|
||||||
|
await load_model_card(source, "u/r", cache)
|
||||||
|
|
||||||
|
assert source.readme_calls == 2
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_works_without_a_cache(self):
|
||||||
|
source = _FakeSource(readme="# hi")
|
||||||
|
|
||||||
|
assert await load_model_card(source, "u/r") == "# hi"
|
||||||
|
assert source.readme_calls == 1
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Base-model resolution
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestResolveSiteBaseModel:
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_maps_the_sites_own_vocabulary(self, monkeypatch):
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"py.metadata_ops.list_base_models",
|
||||||
|
_async(["Krea 2", "Flux.1 D"]),
|
||||||
|
)
|
||||||
|
|
||||||
|
context = ModelCardContext(
|
||||||
|
base_model="krea/Krea-2-Turbo",
|
||||||
|
base_model_aliases=["KREA_2_TURBO", "krea/Krea-2-Turbo"],
|
||||||
|
)
|
||||||
|
|
||||||
|
assert await resolve_site_base_model(context) == "Krea 2"
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_unknown_hint_defers_instead_of_guessing(self, monkeypatch):
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"py.metadata_ops.list_base_models", _async(["Flux.1 D"])
|
||||||
|
)
|
||||||
|
|
||||||
|
context = ModelCardContext(base_model="something/else")
|
||||||
|
|
||||||
|
assert await resolve_site_base_model(context) == ""
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_no_hints_needs_no_vocabulary_lookup(self, monkeypatch):
|
||||||
|
async def _boom(*_args, **_kwargs): # pragma: no cover - must not run
|
||||||
|
raise AssertionError("list_base_models should not be called")
|
||||||
|
|
||||||
|
monkeypatch.setattr("py.metadata_ops.list_base_models", _boom)
|
||||||
|
|
||||||
|
assert await resolve_site_base_model(ModelCardContext()) == ""
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_a_vocabulary_failure_is_not_fatal(self, monkeypatch):
|
||||||
|
async def _boom(*_args, **_kwargs):
|
||||||
|
raise RuntimeError("civitai down")
|
||||||
|
|
||||||
|
monkeypatch.setattr("py.metadata_ops.list_base_models", _boom)
|
||||||
|
|
||||||
|
assert await resolve_site_base_model(ModelCardContext(base_model="x")) == ""
|
||||||
@@ -1121,3 +1121,118 @@ pip install modelscope
|
|||||||
)
|
)
|
||||||
|
|
||||||
assert "modelDescription" not in mock_apply.call_args[0][1]
|
assert "modelDescription" not in mock_apply.call_args[0][1]
|
||||||
|
|
||||||
|
|
||||||
|
# ======================================================================
|
||||||
|
# Site identity and provenance fields
|
||||||
|
# ======================================================================
|
||||||
|
|
||||||
|
|
||||||
|
class TestSiteIdentityFields:
|
||||||
|
"""The fields that make a source download look like a CivitAI one."""
|
||||||
|
|
||||||
|
METADATA = {
|
||||||
|
"from_civitai": False,
|
||||||
|
"source_platform": "modelscope",
|
||||||
|
"source_url": "https://modelscope.cn/models/user/repo",
|
||||||
|
"file_name": "Krea-2-LORA_c1-st1000",
|
||||||
|
"model_name": "Krea-2-LORA_c1-st1000",
|
||||||
|
"base_model": "Unknown",
|
||||||
|
}
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _run(processor, *, metadata, context, llm_output=None, **kwargs):
|
||||||
|
async def _call():
|
||||||
|
with (
|
||||||
|
mock.patch("py.metadata_ops.apply_metadata_updates") as mock_apply,
|
||||||
|
mock.patch("py.metadata_ops.download_preview", return_value=None),
|
||||||
|
mock.patch("py.metadata_ops.refresh_cache"),
|
||||||
|
):
|
||||||
|
await processor.process(
|
||||||
|
skill_name="enrich_hf_metadata",
|
||||||
|
model_path="/p.safetensors",
|
||||||
|
llm_output=llm_output if llm_output is not None else {},
|
||||||
|
metadata=metadata,
|
||||||
|
source_context=context,
|
||||||
|
**kwargs,
|
||||||
|
)
|
||||||
|
return mock_apply.call_args[0][1]
|
||||||
|
|
||||||
|
return _call()
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_model_name_is_taken_from_the_site(self, processor):
|
||||||
|
applied = await self._run(
|
||||||
|
processor,
|
||||||
|
metadata=dict(self.METADATA),
|
||||||
|
context=ModelCardContext(model_name="Krea-2-LORA"),
|
||||||
|
)
|
||||||
|
assert applied["model_name"] == "Krea-2-LORA"
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_model_name_is_also_written_when_absent(self, processor):
|
||||||
|
metadata = {**self.METADATA, "model_name": ""}
|
||||||
|
applied = await self._run(
|
||||||
|
processor,
|
||||||
|
metadata=metadata,
|
||||||
|
context=ModelCardContext(model_name="Krea-2-LORA"),
|
||||||
|
)
|
||||||
|
assert applied["model_name"] == "Krea-2-LORA"
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_renamed_model_keeps_the_users_name(self, processor):
|
||||||
|
metadata = {**self.METADATA, "model_name": "my own name"}
|
||||||
|
applied = await self._run(
|
||||||
|
processor,
|
||||||
|
metadata=metadata,
|
||||||
|
context=ModelCardContext(model_name="Krea-2-LORA"),
|
||||||
|
)
|
||||||
|
assert "model_name" not in applied
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_version_label_becomes_the_civitai_name(self, processor):
|
||||||
|
applied = await self._run(
|
||||||
|
processor,
|
||||||
|
metadata=dict(self.METADATA),
|
||||||
|
context=ModelCardContext(
|
||||||
|
model_name="Krea-2-LORA",
|
||||||
|
version_name="c1-st1000",
|
||||||
|
description="权重0.5-1.2。",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
assert applied["civitai"]["name"] == "c1-st1000"
|
||||||
|
# Every civitai branch contributes to one dict, so an earlier branch
|
||||||
|
# must survive a later one.
|
||||||
|
assert applied["civitai"]["description"] == "权重0.5-1.2。"
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_llm_enriched_at_is_stamped_only_when_the_llm_answered(
|
||||||
|
self, processor
|
||||||
|
):
|
||||||
|
applied = await self._run(
|
||||||
|
processor,
|
||||||
|
metadata=dict(self.METADATA),
|
||||||
|
context=ModelCardContext(model_name="Krea-2-LORA"),
|
||||||
|
)
|
||||||
|
assert applied["metadata_source"] == "agent:enrich_hf_metadata"
|
||||||
|
assert "llm_enriched_at" not in applied
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_llm_answer_stamps_llm_enriched_at(self, processor):
|
||||||
|
applied = await self._run(
|
||||||
|
processor,
|
||||||
|
metadata=dict(self.METADATA),
|
||||||
|
context=ModelCardContext(model_name="Krea-2-LORA"),
|
||||||
|
llm_output={"base_model": "", "confidence": "high"},
|
||||||
|
)
|
||||||
|
assert "llm_enriched_at" in applied
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_metadata_source_can_be_overridden(self, processor):
|
||||||
|
applied = await self._run(
|
||||||
|
processor,
|
||||||
|
metadata=dict(self.METADATA),
|
||||||
|
context=ModelCardContext(model_name="Krea-2-LORA"),
|
||||||
|
metadata_source="source:modelscope",
|
||||||
|
)
|
||||||
|
assert applied["metadata_source"] == "source:modelscope"
|
||||||
|
|||||||
Reference in New Issue
Block a user