alphard/ComfyUI-Lora-Manager
1
0
Fork 0
mirror of https://github.com/willmiao/ComfyUI-Lora-Manager.git synced 2026-09-20 18:51:26 -03:00
Code Issues Packages Projects Releases Wiki Activity

feat(links): support ModelScope and TensorArt as model sources

Browse Source 
A model file could only ever be linked to huggingface.co: `set_hf_url`
validated the URL with a huggingface-only regex, the agent fetched the card
from a hardcoded HF URL, and the readme processor built every relative image
path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the
same model-card convention (README.md + YAML frontmatter, often carrying
`base_model:` and `trigger_words:`) behind a public, key-less API, so the
enrichment pipeline could already serve it - it was the plumbing that was
HF-shaped, not the idea.

Make the external source a first-class, provider-driven concept:

- New `py/services/model_sources/` registry. A `ModelSource` owns URL
  recognition (lenient for stored values, strict for user input), the
  canonical page URL, model-card fetching, the asset base URL and the
  capability flags. `HuggingFaceSource` is the previous logic relocated;
  `ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md`
  and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is
  link-only on purpose: tensor.art answers plain HTTP clients with a
  Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud /
  cn.tensorart.net) rejects every /v1/model/* route with "invalid
  authorization header", so it declares supports_enrichment=False rather than
  failing silently later.
- Metadata gains `source_platform` + `source_url`; `hf_url` stays as a
  read/write alias, written only for Hugging Face, so existing sidecars,
  cached rows and third-party consumers keep working. Normalisation runs at
  the scanner, the persistent cache (both directions, plus two new columns
  behind an ALTER migration) and the linking handler - which is what stops a
  user who switches sources from leaving a stale `hf_url` on a ModelScope
  model.
- The agent pipeline keys off the provider instead of `hf_url`: the fast-fail
  gate now explains *why* a model is skipped (no source / unknown source /
  source without a reachable card), the prompt context exposes
  source_url/source_id/source_label/asset_base_url while still filling the
  legacy hf_url/repo aliases, and the four README image extractors take a
  base_url (defaulting to HF) so relative paths resolve against the right
  site. Version grouping generalises to hf: / ms: / ta: keys.
- `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but
  accepts `source_url`, validates against every provider and returns the
  platform. `GET /api/lm/model-sources` lets the UI render the supported-site
  list from the server.
- Frontend: a `modelSourceHelpers` mirror of the registry drives the link
  dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the
  version-group key and the enrichment gate; the versions tab no longer sends
  ms:/ta: keys to the CivitAI API.

TensorArt stays in the list because provenance is worth keeping even when the
card is unreadable - the dialog says so plainly ("Sites that don't expose one
(currently TensorArt) can only be linked") and the context menu disables
enrichment with a matching tooltip, instead of the user getting
"Unsupported URL".

Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a
1882-byte card whose frontmatter carries base_model/tags/trigger_words, and
relative images resolve to .../resolve/master/....

Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest
tests/i18n and a Jinja compile pass over templates/. The nine locales carry
[TODO: Translate] for the new strings, completed in the next commit.
This commit is contained in:
Will Miao
2026-09-14 07:24:08 +08:00
parent 84146b62fd
commit 5ab0e88abc
51 changed files with 2518 additions and 279 deletions
Download Patch File Download Diff File Expand all files Collapse all files
py/routes/handlers/hf_handlers.py
+97 -21
View File
@@ -22,6 +22,11 @@ from ...services.downloader import (
get_downloader,
)
from ...services.aria2_downloader import Aria2Downloader
from ...services.model_sources import (
detect_source,
list_sources,
normalize_metadata_source,
)
from ...services.settings_manager import get_settings_manager
from ...services.service_registry import ServiceRegistry
from ...services.websocket_manager import ws_manager
@@ -120,6 +125,8 @@ async def _save_hf_metadata(dest_path: str, repo: str, model_root: str) -> None:
# 2. Overlay HF-specific fields
metadata._unknown_fields["hf_url"] = hf_url
metadata._unknown_fields["source_url"] = hf_url
metadata._unknown_fields["source_platform"] = "huggingface"
metadata.from_civitai = False # HF models are not from CivitAI
# 3. Save metadata atomically
@@ -189,27 +196,72 @@ async def _add_to_scanner_cache(dest_path: str, metadata: dict[str, Any]) -> Non
class HfHandler:
"""Handle Hugging Face model browsing and download."""
async def get_model_sources(self, request: web.Request) -> web.Response:
"""List the external model sites the UI can link a model to.
Used by the "Link Model" dialog to validate URLs client-side and to
explain which sites support AI metadata enrichment.
"""
return web.json_response([
{
"platform": source.platform,
"label": source.label,
"supports_enrichment": source.supports_enrichment,
"supports_download": source.supports_download,
"example_url": source.canonical_url(
"user/repo" if source.platform != "tensorart" else "827823520299086029"
),
}
for source in list_sources()
])
async def set_hf_url(self, request: web.Request) -> web.Response:
"""Link a model file to its page on an external model site.
Accepts ``source_url`` (preferred) or the legacy ``hf_url`` /
``url`` payload key. Hugging Face, ModelScope, and TensorArt URLs
are recognised; the platform is stored alongside the canonical URL.
TensorArt models can be linked and browsed, but not AI-enriched.
"""
try:
payload: dict[str, Any] = await request.json()
except json.JSONDecodeError:
return web.json_response({"success": False, "error": "Invalid JSON"}, status=400)
file_path = (payload.get("file_path") or "").strip()
hf_url = (payload.get("hf_url") or "").strip()
raw_url = (
payload.get("source_url")
or payload.get("hf_url")
or payload.get("url")
or ""
)
source_url = raw_url.strip() if isinstance(raw_url, str) else ""
if not file_path or not hf_url:
return web.json_response(
{"success": False, "error": "Missing required fields: 'file_path' and 'hf_url'"},
status=400,
)
m = re.match(r"^https?://huggingface\.co/([^/]+/[^/]+)/?$", hf_url)
if not m:
if not file_path or not source_url:
return web.json_response(
{
"success": False,
"error": "Invalid HuggingFace URL. Expected format: https://huggingface.co/user/repo",
"error": "Missing required fields: 'file_path' and 'source_url'",
},
status=400,
)
ref = detect_source(source_url, strict=True)
if ref is None:
return web.json_response(
{
"success": False,
"error": (
"Unsupported model URL. Supported formats: "
+ ", ".join(
f"{s.label} ({s.canonical_url('user/repo')})"
if s.platform != "tensorart"
else f"{s.label} (https://tensor.art/models/<id>)"
for s in list_sources()
)
),
},
status=400,
)
@@ -225,37 +277,61 @@ class HfHandler:
return web.json_response(
{
"success": False,
"error": "File is not within any configured model directory. Cannot link to HuggingFace.",
"error": "File is not within any configured model directory. Cannot link to a model source.",
},
status=400,
)
try:
existing = await MetadataManager.load_metadata_payload(file_path)
if existing.get("hf_url") == hf_url:
already_linked = (
(existing.get("source_url") or "").strip() == ref.url
and (existing.get("source_platform") or "").strip().lower()
== ref.platform
) or (
not existing.get("source_url")
and ref.platform == "huggingface"
and (existing.get("hf_url") or "").strip() == ref.url
)
if already_linked:
return web.json_response({
"success": True,
"message": "hf_url already set",
"hf_url": hf_url,
"message": "source_url already set",
"source_url": ref.url,
"source_platform": ref.platform,
"hf_url": ref.url if ref.platform == "huggingface" else "",
})
existing["hf_url"] = hf_url
existing["source_url"] = ref.url
existing["source_platform"] = ref.platform
if ref.platform == "huggingface":
existing["hf_url"] = ref.url
else:
existing.pop("hf_url", None)
normalize_metadata_source(existing)
# NOTE: deliberately do NOT touch `from_civitai` here. It records
# where the metadata came from, and the UI must show the CivitAI
# link whenever CivitAI data is present — linking HuggingFace must
# not hide it (#1094). HF provenance is tracked via `hf_url`.
# link whenever CivitAI data is present — linking an external
# source must not hide it (#1094). Source provenance is tracked
# via `source_platform` / `source_url`.
await MetadataManager.save_metadata(file_path, existing)
await _add_to_scanner_cache(file_path, existing)
logger.info("Set hf_url=%s for %s", hf_url, file_path)
logger.info(
"Linked %s to %s source (%s)", file_path, ref.platform, ref.url
)
return web.json_response({
"success": True,
"message": f"hf_url set to {hf_url}",
"hf_url": hf_url,
"message": f"Linked to {ref.url}",
"source_url": ref.url,
"source_platform": ref.platform,
"hf_url": existing.get("hf_url", ""),
})
except Exception as exc:
logger.error("Failed to set hf_url for %s: %s", file_path, exc)
logger.error("Failed to link %s to a model source: %s", file_path, exc)
return web.json_response(
{"success": False, "error": str(exc)},
status=500,
py/routes/handlers/misc_handlers.py
+1
View File
@@ -4079,6 +4079,7 @@ class MiscHandlerSet:
"get_hf_repo_files": self.hf_handler.get_hf_repo_files,
"download_hf_model": self.hf_handler.download_hf_model,
"set_hf_url": self.hf_handler.set_hf_url,
"get_model_sources": self.hf_handler.get_model_sources,
# Agent skill handlers
"get_agent_skills": self.agent_handler.get_agent_skills,
"execute_agent_skill": self.agent_handler.execute_agent_skill,
py/routes/misc_route_registrar.py
+4
View File
@@ -113,6 +113,10 @@ MISC_ROUTE_DEFINITIONS: tuple[RouteDefinition, ...] = (
RouteDefinition(
"POST", "/api/lm/set-hf-url", "set_hf_url"
),
# Supported external model sites (Hugging Face / ModelScope / TensorArt)
RouteDefinition(
"GET", "/api/lm/model-sources", "get_model_sources"
),
# Agent skill endpoints
RouteDefinition(
"GET", "/api/lm/agent/skills", "get_agent_skills"
py/services/agent/agent_service.py
+69 -32
View File
@@ -19,16 +19,18 @@ from __future__ import annotations
import asyncio
import json
import logging
import os
import re
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional
import aiohttp
import os
from ...config import config
from ..llm_service import LLMService
from ..model_sources import (
get_source,
resolve_source_ref,
source_label,
)
from ..websocket_manager import ws_manager
from .post_processor import PostProcessor
from .skill_registry import SkillRegistry
@@ -267,14 +269,17 @@ class AgentService:
from ...metadata_ops import read_metadata
metadata = await read_metadata(model_path)
# Fast-fail: enrich_hf_metadata requires hf_url to have HF README context
if skill_name == "enrich_hf_metadata" and not metadata.get("hf_url", ""):
logger.info(
"[%s] SKIP %s — no hf_url in metadata",
skill_name, model_filename,
)
skipped_count += 1
skip_model = True
# Fast-fail: enrich_hf_metadata needs an external model source
# that exposes an accessible model card.
if skill_name == "enrich_hf_metadata":
skip_reason = self._enrichment_skip_reason(metadata)
if skip_reason:
logger.info(
"[%s] SKIP %s — %s",
skill_name, model_filename, skip_reason,
)
skipped_count += 1
skip_model = True
if not skip_model:
prompt_vars: Dict[str, Any] = {"model_path": model_path}
@@ -358,6 +363,28 @@ class AgentService:
# Base model grouping (keeps the prompt compact)
# ------------------------------------------------------------------
@staticmethod
def _enrichment_skip_reason(metadata: Dict[str, Any]) -> str:
"""Return why ``enrich_hf_metadata`` cannot run, or ``""`` if it can.
Distinguishes the three cases the user can act on: no source linked,
a source we don't know, and a known source whose model card is not
reachable from the backend (TensorArt).
"""
ref = resolve_source_ref(metadata)
if ref is None:
return "no model source linked (source_url missing)"
source = get_source(ref.platform)
if source is None:
return f"unsupported model source platform '{ref.platform}'"
if not source.supports_enrichment:
return (
f"{source.label} does not expose a model card to the backend; "
"AI metadata enrichment is not available for this source"
)
return ""
@staticmethod
def _format_base_models(models: List[str]) -> str:
"""Format the base model list as a flat, one-per-line list.
@@ -388,6 +415,14 @@ class AgentService:
context: Dict[str, Any] = {
"model_path": model_path,
"model_basename": "",
# Canonical external-source variables
"source_url": "",
"source_id": "",
"source_platform": "",
"source_label": "",
"asset_base_url": "",
# Legacy Hugging Face aliases (kept so older prompt templates and
# third-party skills keep rendering)
"hf_url": "",
"repo": "",
"readme_content": "",
@@ -411,12 +446,20 @@ class AgentService:
"size": metadata.get("size", 0),
}
hf_url = metadata.get("hf_url", "")
context["hf_url"] = hf_url
repo = self._extract_repo_from_url(hf_url) if hf_url else ""
context["repo"] = repo or ""
if repo:
readme = await self._fetch_readme(repo)
ref = resolve_source_ref(metadata)
if ref is not None:
context["source_url"] = ref.url
context["source_id"] = ref.source_id
context["source_platform"] = ref.platform
context["source_label"] = source_label(ref.platform, ref.platform)
if ref.platform == "huggingface":
context["hf_url"] = ref.url
context["repo"] = ref.source_id
source = get_source(ref.platform) if ref is not None else None
if ref is not None and source is not None and source.supports_enrichment:
context["asset_base_url"] = source.asset_base_url(ref.source_id)
readme = await source.fetch_model_card(ref.source_id)
# Trim README to the section relevant to this model file
# (collection repos often have multiple models in one README).
if readme and raw_basename:
@@ -458,20 +501,14 @@ class AgentService:
@staticmethod
async def _fetch_readme(repo: str) -> str:
"""Fetch README.md from HuggingFace (tries ``main``, then ``master``)."""
async with aiohttp.ClientSession(
headers={"User-Agent": "ComfyUI-LoRA-Manager/1.0"},
timeout=aiohttp.ClientTimeout(total=30),
) as session:
for branch in ("main", "master"):
url = f"https://huggingface.co/{repo}/raw/{branch}/README.md"
try:
async with session.get(url) as resp:
if resp.status == 200:
return await resp.text()
except Exception as exc:
logger.debug("Failed to fetch README from %s: %s", url, exc)
return ""
"""Fetch a Hugging Face README (tries ``main``, then ``master``).
Kept for backward compatibility; new code should go through the
model-source registry so every supported site works.
"""
from ..model_sources import HuggingFaceSource
return await HuggingFaceSource().fetch_model_card(repo)
async def _emit_progress(
self,
py/services/agent/post_processor.py
+31 -19
View File
@@ -78,6 +78,7 @@ class PostProcessor:
download_preview,
refresh_cache,
)
from ..model_sources import get_source, has_external_source, resolve_source_ref
from .skills.enrich_hf_metadata.readme_processor import (
convert_readme_to_html,
extract_gallery_images,
@@ -85,17 +86,25 @@ class PostProcessor:
extract_relevant_section,
extract_simple_markdown_images,
extract_html_img_tags,
extract_repo_from_hf_url,
)
updated_fields: List[str] = []
preview_downloaded = False
# -- Determine whether this is an HF-sourced model -----------------
# Key off `hf_url` directly: `from_civitai` records provenance and can
# be true for a model that is also linked to HuggingFace (both sources
# coexist, see #1094), so it must not gate HF enrichment.
is_hf_model = bool(metadata.get("hf_url", ""))
# -- Determine whether this is an externally-sourced model ---------
# Key off the source fields directly: `from_civitai` records provenance
# and can be true for a model that is also linked to an external site
# (both sources coexist, see #1094), so it must not gate enrichment.
is_source_model = has_external_source(metadata)
source_ref = resolve_source_ref(metadata)
source = get_source(source_ref.platform) if source_ref else None
source_id = source_ref.source_id if source_ref else ""
asset_base_url = (
source.asset_base_url(source_id)
if source is not None and source_id
else None
)
# -- Collect updates -----------------------------------------------
updates: Dict[str, Any] = {}
@@ -103,7 +112,7 @@ class PostProcessor:
# base_model
new_base = (llm_output.get("base_model") or "").strip()
current_base = metadata.get("base_model", "") or ""
if new_base and self._should_overwrite(current_base, is_hf_model):
if new_base and self._should_overwrite(current_base, is_source_model):
updates["base_model"] = new_base
# trigger words → civitai.trainedWords
@@ -115,7 +124,7 @@ class PostProcessor:
trigger_words_empty = not cleaned
current_civitai = metadata.get("civitai") or {}
current_triggers = current_civitai.get("trainedWords") or []
if self._should_overwrite_list(current_triggers, is_hf_model):
if self._should_overwrite_list(current_triggers, is_source_model):
trig_civitai = dict(current_civitai)
if "civitai" in updates and isinstance(updates["civitai"], dict):
trig_civitai.update(updates["civitai"])
@@ -123,14 +132,14 @@ class PostProcessor:
updates["civitai"] = trig_civitai
# modelDescription — from raw README content (converted to HTML)
if readme_content and is_hf_model:
if readme_content and is_source_model:
converted = convert_readme_to_html(readme_content)
if converted:
updates["modelDescription"] = converted
# short_description → civitai.description (for "About this version")
short_desc = (llm_output.get("short_description") or "").strip()
if short_desc and is_hf_model:
if short_desc and is_source_model:
current_civitai = metadata.get("civitai") or {}
desc_civitai = dict(current_civitai)
if "civitai" in updates and isinstance(updates["civitai"], dict):
@@ -141,9 +150,8 @@ class PostProcessor:
# gallery images → civitai.images (from YAML frontmatter widget entries
# and Sample Gallery markdown tables in the README body)
gallery_images: List[Dict[str, Any]] = []
if readme_content and is_hf_model:
hf_url = metadata.get("hf_url", "") or ""
repo = extract_repo_from_hf_url(hf_url)
if readme_content and is_source_model:
repo = source_id
if repo:
rec_w = llm_output.get("recommended_width") or 0
rec_h = llm_output.get("recommended_height") or 0
@@ -152,6 +160,7 @@ class PostProcessor:
gallery = extract_gallery_images(
readme_content, repo,
default_width=rec_w, default_height=rec_h,
base_url=asset_base_url,
)
# 2. Sample Gallery table images (markdown body), deduplicated
@@ -160,6 +169,7 @@ class PostProcessor:
readme_content, repo,
existing_urls=existing_urls,
default_width=rec_w, default_height=rec_h,
base_url=asset_base_url,
)
existing_urls.update(img["url"] for img in table_images if img.get("url"))
@@ -168,6 +178,7 @@ class PostProcessor:
readme_content, repo,
existing_urls=existing_urls,
default_width=rec_w, default_height=rec_h,
base_url=asset_base_url,
)
existing_urls.update(img["url"] for img in simple_images if img.get("url"))
@@ -176,6 +187,7 @@ class PostProcessor:
readme_content, repo,
existing_urls=existing_urls,
default_width=rec_w, default_height=rec_h,
base_url=asset_base_url,
)
all_images = gallery + table_images + simple_images + html_images
@@ -193,7 +205,7 @@ class PostProcessor:
if isinstance(new_tags, list) and new_tags:
existing_tags = metadata.get("tags") or []
merged = self._merge_tags(existing_tags, new_tags)
if len(merged) > len(existing_tags) or is_hf_model:
if len(merged) > len(existing_tags) or is_source_model:
updates["tags"] = merged
# metadata_source & llm_enriched_at (always set)
@@ -222,7 +234,7 @@ class PostProcessor:
# README, find the first gallery image from the *model-specific
# section* of the README (not the repo-wide first image, which
# belongs to a different model in collection repos).
if not preview_remote_url and readme_content and is_hf_model:
if not preview_remote_url and readme_content and is_source_model:
model_basename = os.path.splitext(os.path.basename(model_path))[0]
relevant_section = extract_relevant_section(
readme_content, model_basename,
@@ -279,16 +291,16 @@ class PostProcessor:
# ------------------------------------------------------------------
@staticmethod
def _should_overwrite(current_value: str, is_hf_model: bool) -> bool:
def _should_overwrite(current_value: str, is_source_model: bool) -> bool:
"""Return ``True`` when a scalar field should be overwritten."""
return is_hf_model or not current_value or current_value.lower() in (
return is_source_model or not current_value or current_value.lower() in (
"", "unknown",
)
@staticmethod
def _should_overwrite_list(current_list: List[str], is_hf_model: bool) -> bool:
def _should_overwrite_list(current_list: List[str], is_source_model: bool) -> bool:
"""Return ``True`` when a list field should be overwritten."""
return is_hf_model or not current_list
return is_source_model or not current_list
@staticmethod
def _merge_tags(existing: List[str], new: List[str]) -> List[str]:
py/services/agent/skills/enrich_hf_metadata/prompt.md
+12 -9
View File
@@ -1,20 +1,23 @@
---
name: enrich_hf_metadata
title: "Enrich Metadata from HuggingFace"
title: "Enrich Metadata from Model Card"
description: >
Parse the HuggingFace model card via LLM to extract description, trigger
words, base model, tags, and preview image URL.
Parse the model card (README) from HuggingFace, ModelScope, or any other
supported model site via LLM to extract description, trigger words, base
model, tags, and preview image URL.
llm_required: true
---
You are an expert assistant for AI image generation models. Your task is to extract structured metadata from a HuggingFace model card (README.md).
You are an expert assistant for AI image generation models. Your task is to extract structured metadata from a model card (README).
## Model Information
- **Repository**: {{hf_url}}
- **Source site**: {{source_label}} ({{source_platform}})
- **Model page**: {{source_url}}
- **Model file path**: {{model_path}}
- **Model filename**: {{model_basename}}
- **Repository ID**: {{repo}}
- **Repository ID**: {{source_id}}
- **Repository raw-file base URL**: {{asset_base_url}}
## Current Metadata (may be incomplete)
@@ -39,7 +42,7 @@ name listed — do not invent aliases or modify variant suffixes.
{{base_models}}
## HuggingFace README Content
## Model Card Content
```
{{readme_content}}
@@ -92,7 +95,7 @@ The URL of the most suitable preview image from the README. Look for:
- The YAML frontmatter `widget:` section (which often has `output.url` fields)
- In collection repos: the sample images listed **under the section** for this specific model version
- Generic `![alt](url)` in the body
Choose the first image that appears to be a generation example (not a logo or diagram). Construct the absolute URL as `https://huggingface.co/{{repo}}/resolve/main/{filename}`. If no suitable image is found, return an empty string.
Choose the first image that appears to be a generation example (not a logo or diagram). Construct the absolute URL from the repository raw-file base URL (`{{asset_base_url}}`) plus the relative path. If no suitable image is found, return an empty string.
### notes
A plain-text summary of the model card's key practical usage information. Combine trigger words, style modifiers, recommended parameters (steps, CFG, resolution, sampler), and any setup tips into a readable paragraph. For collection repos, focus on the **specific model version** matching `{{model_basename}}`. Return empty string if the README has no useful usage info.
@@ -121,7 +124,7 @@ Your confidence level in the extracted data:
## Important: Handling Collection Repos (multiple model files)
Many HuggingFace repos contain **multiple model files** in a single repository
Many model repositories contain **multiple model files** in a single repository
(e.g. a "LoRA collection" with different styles/characters in separate files).
The model file currently being enriched is: **`{{model_basename}}`**
py/services/agent/skills/enrich_hf_metadata/readme_processor.py
+39 -15
View File
@@ -1,8 +1,15 @@
"""HF README processing for the ``enrich_hf_metadata`` skill.
"""Model card (README) processing for the ``enrich_hf_metadata`` skill.
Provides README cleaning for LLM injection, gallery/image extraction from
multiple formats (YAML widget, markdown, HTML ``<img>``, gallery tables),
and section-based README trimming for collection repos.
The extractors default to Hugging Face asset URLs, but every one of them
accepts an explicit ``base_url`` so the same parsing works for any model
source (ModelScope, ...). See :mod:`py.services.model_sources`.
This module deliberately has no package-relative imports: it is also loaded
standalone by the README-processing test harness.
"""
from __future__ import annotations
@@ -15,12 +22,25 @@ from typing import Any, List, Tuple
_REPO_URL_PATTERN = re.compile(r"https?://huggingface\.co/([^/]+/[^/]+)")
def resolve_asset_base_url(repo: str, base_url: str | None = None) -> str:
"""Return the base URL used to resolve repository-relative assets.
Falls back to the historical Hugging Face layout when *base_url* is not
supplied, so existing callers keep their behaviour.
"""
if base_url:
return base_url.rstrip("/")
return f"https://huggingface.co/{repo}/resolve/main"
def extract_simple_markdown_images(
markdown_text: str,
repo: str,
existing_urls: set[str] | None = None,
default_width: int = 512,
default_height: int = 512,
base_url: str | None = None,
) -> list[dict[str, Any]]:
"""Extract standalone markdown images from the README body.
@@ -32,10 +52,10 @@ def extract_simple_markdown_images(
Returns a list of dicts in the same ``civitai.images`` format as
:func:`extract_gallery_images`.
"""
if not markdown_text or not repo:
if not markdown_text or not (repo or base_url):
return []
base_url = f"https://huggingface.co/{repo}/resolve/main"
base_url = resolve_asset_base_url(repo, base_url)
images: list[dict[str, Any]] = []
seen_urls: set[str] = set(existing_urls) if existing_urls else set()
@@ -89,20 +109,21 @@ def extract_html_img_tags(
existing_urls: set[str] | None = None,
default_width: int = 512,
default_height: int = 512,
base_url: str | None = None,
) -> list[dict[str, Any]]:
"""Extract image URLs from HTML ``<img src=\"...\">`` tags in the README.
Many HF collection repos (e.g. ``deadman44/Z-Image_LoRA``) use raw HTML
``<img>`` tags exclusively for their sample images, with no markdown
``![]()`` equivalents. This function finds those tags and constructs
resolvable HF URLs.
resolvable URLs.
Returns a list of dicts in the ``civitai.images`` format.
"""
if not markdown_text or not repo:
if not markdown_text or not (repo or base_url):
return []
base_url = f"https://huggingface.co/{repo}/resolve/main"
base_url = resolve_asset_base_url(repo, base_url)
images: list[dict[str, Any]] = []
seen_urls: set[str] = set(existing_urls) if existing_urls else set()
@@ -166,7 +187,7 @@ def extract_html_img_tags(
def extract_repo_from_hf_url(hf_url: str) -> str:
"""Extract ``user/repo`` from a HuggingFace URL."""
m = _REPO_URL_PATTERN.match(hf_url)
m = _REPO_URL_PATTERN.match(hf_url or "")
return m.group(1) if m else ""
@@ -175,21 +196,23 @@ def extract_gallery_images(
repo: str,
default_width: int = 512,
default_height: int = 512,
base_url: str | None = None,
) -> List[dict[str, Any]]:
"""Extract widget/gallery images from the YAML frontmatter of a HF README.
"""Extract widget/gallery images from the YAML frontmatter of a README.
Args:
markdown_text: Raw README content.
repo: HF repo identifier (``user/repo``).
repo: Repository identifier (``user/repo``).
default_width: Fallback width when the README provides no dimension.
default_height: Fallback height when the README provides no dimension.
base_url: Overrides the asset base URL (defaults to Hugging Face).
Returns a list of dicts compatible with the ``civitai.images`` metadata
format, each containing ``url`` (absolute HF URL), ``meta.prompt``,
format, each containing ``url`` (absolute), ``meta.prompt``,
``width``, ``height``, and ``type``. Returns an empty list when no
widget entries are found or when *repo* is empty.
"""
if not markdown_text or not repo:
if not markdown_text or not (repo or base_url):
return []
frontmatter = _extract_frontmatter(markdown_text)
@@ -197,7 +220,7 @@ def extract_gallery_images(
return []
images: List[dict[str, Any]] = []
base_url = f"https://huggingface.co/{repo}/resolve/main"
base_url = resolve_asset_base_url(repo, base_url)
w = default_width or 512
h = default_height or 512
@@ -279,10 +302,11 @@ def extract_gallery_table_images(
existing_urls: set[str] | None = None,
default_width: int = 512,
default_height: int = 512,
base_url: str | None = None,
) -> list[dict[str, Any]]:
"""Extract images from ``| Preview | Prompt |`` markdown gallery tables.
Many HF READMEs include a sample-gallery table in the body (outside
Many READMEs include a sample-gallery table in the body (outside
the YAML frontmatter) that shows generation examples with their
prompts. This function parses those tables and merges results with
the widget-sourced images from :func:`extract_gallery_images`.
@@ -291,10 +315,10 @@ def extract_gallery_table_images(
:func:`extract_gallery_images`. Already-seen URLs (from *existing_urls*)
are skipped.
"""
if not markdown_text or not repo:
if not markdown_text or not (repo or base_url):
return []
base_url = f"https://huggingface.co/{repo}/resolve/main"
base_url = resolve_asset_base_url(repo, base_url)
images: list[dict[str, Any]] = []
seen_urls: set[str] = set(existing_urls) if existing_urls else set()
lines = markdown_text.split("\n")
py/services/base_model_service.py
+16 -12
View File
@@ -21,6 +21,7 @@ from .model_query import (
resolve_sub_type,
)
from .settings_manager import get_settings_manager
from .model_sources import source_group_key
from ..utils.civitai_utils import build_civitai_model_page_url
logger = logging.getLogger(__name__)
@@ -742,29 +743,32 @@ class BaseModelService(ABC):
@staticmethod
def _extract_hf_group_key(item: Dict[str, Any]) -> Optional[str]:
"""Extract `hf:{owner}/{repo}` from item's ``hf_url``, or None."""
hf_url = item.get("hf_url") if isinstance(item, dict) else None
if not hf_url or not isinstance(hf_url, str):
return None
m = re.match(
r"https?://huggingface\.co/([^/]+/[^/]+)", hf_url.strip()
)
if not m:
return None
return f"hf:{m.group(1)}"
key = BaseModelService._extract_source_group_key(item)
return key if key and key.startswith("hf:") else None
@staticmethod
def _extract_source_group_key(item: Dict[str, Any]) -> Optional[str]:
"""Return the external-source group key for *item*, or None.
Hugging Face keeps the historical ``hf:{owner}/{repo}`` shape; other
platforms use their own short prefix (``ms:`` / ``ta:``).
"""
return source_group_key(item)
@staticmethod
def _extract_group_key(item: Dict[str, Any]) -> Union[int, str, None]:
"""Return the group identity key: CivitAI modelId (int) or HF repo (str).
"""Return the group identity key.
Preference order:
1. CivitAI ``modelId`` (int)
2. HF repo identity ``hf:{owner}/{repo}`` (str)
2. External model source identity, e.g. ``hf:{owner}/{repo}``,
``ms:{owner}/{repo}``, ``ta:{model_id}`` (str)
3. ``None`` (no known grouping source)
"""
mid = BaseModelService._extract_model_id(item)
if mid is not None:
return mid
return BaseModelService._extract_hf_group_key(item)
return BaseModelService._extract_source_group_key(item)
@staticmethod
def _extract_model_id(item: Dict[str, Any]) -> Optional[int]:
py/services/checkpoint_service.py
+2
View File
@@ -67,6 +67,8 @@ class CheckpointService(BaseModelService):
"civitai": self.filter_civitai_data(model_data.get("civitai", {}), minimal=True),
"auto_tags": model_data.get("auto_tags") or extract_auto_tags(model_data),
"version_count": model_data.get("version_count"),
"source_platform": model_data.get("source_platform", ""),
"source_url": model_data.get("source_url", ""),
"hf_url": model_data.get("hf_url", ""),
}
py/services/embedding_service.py
+2
View File
@@ -67,6 +67,8 @@ class EmbeddingService(BaseModelService):
"civitai": self.filter_civitai_data(model_data.get("civitai", {}), minimal=True),
"auto_tags": model_data.get("auto_tags") or extract_auto_tags(model_data),
"version_count": model_data.get("version_count"),
"source_platform": model_data.get("source_platform", ""),
"source_url": model_data.get("source_url", ""),
"hf_url": model_data.get("hf_url", ""),
}
py/services/lora_service.py
+2
View File
@@ -79,6 +79,8 @@ class LoraService(BaseModelService):
),
"auto_tags": model_data.get("auto_tags") or extract_auto_tags(model_data),
"version_count": model_data.get("version_count"),
"source_platform": model_data.get("source_platform", ""),
"source_url": model_data.get("source_url", ""),
"hf_url": model_data.get("hf_url", ""),
}
py/services/metadata_sync_service.py
+4 -2
View File
@@ -14,6 +14,7 @@ from ..utils.model_utils import determine_base_model
from ..utils.models import autov3_from_civitai_files
from .connectivity_guard import OFFLINE_FRIENDLY_MESSAGE, is_expected_offline_error
from .errors import RateLimitError
from .model_sources import has_external_source
logger = logging.getLogger(__name__)
@@ -222,9 +223,10 @@ class MetadataSyncService:
error_msg = "CivitAI model is deleted and no archive provider is available"
return False, error_msg
else:
is_hf_source = bool(model_data.get("hf_url"))
is_hf_source = has_external_source(model_data)
if is_hf_source:
# HF-sourced model: only check CivitAI API directly.
# External-source model (Hugging Face / ModelScope /
# TensorArt): only check CivitAI API directly.
# CivArchive is almost guaranteed to have no record, and
# hitting it wastes rate-limit budget.
# Use a distinct provider name ("civitai_api" not None) so
py/services/model_scanner.py
+7
View File
@@ -15,6 +15,7 @@ from ..utils.civitai_utils import resolve_license_info
from .model_cache import ModelCache
from .model_hash_index import ModelHashIndex
from .model_lifecycle_service import delete_model_artifacts, _require_path_in_library_roots
from .model_sources import normalize_metadata_source
from .service_registry import ServiceRegistry
from .websocket_manager import ws_manager
from .persistent_model_cache import get_persistent_cache
@@ -387,8 +388,14 @@ class ModelScanner:
'civitai': civitai_slim,
'civitai_deleted': bool(get_value('civitai_deleted', False)),
'skip_metadata_refresh': bool(get_value('skip_metadata_refresh', False)),
# External model source (Hugging Face / ModelScope / TensorArt).
# `source_url` + `source_platform` are canonical; `hf_url` stays in
# sync as a legacy alias (normalised below).
'source_platform': get_value('source_platform', '') or '',
'source_url': get_value('source_url', '') or '',
'hf_url': get_value('hf_url', '') or '',
}
normalize_metadata_source(entry)
license_source: Dict[str, Any] = {}
if isinstance(civitai_full, Mapping):
py/services/model_sources/__init__.py
+61
View File
@@ -0,0 +1,61 @@
"""External model-source providers (Hugging Face, ModelScope, TensorArt).
This package is the single abstraction over "a site that hosts models and
a model card". See :mod:`py.services.model_sources.base` for the provider
protocol and :mod:`py.services.model_sources.registry` for the lookup and
metadata-normalisation helpers used across the codebase.
"""
from __future__ import annotations
from .base import (
GROUP_PREFIXES,
HTTP_TIMEOUT,
ModelSource,
SourceRef,
USER_AGENT,
clean_source_url,
fetch_text,
)
from .huggingface import HuggingFaceSource
from .modelscope import ModelScopeSource
from .registry import (
LEGACY_HF_URL_FIELD,
SOURCE_PLATFORM_FIELD,
SOURCE_URL_FIELD,
detect_source,
get_source,
get_source_platform,
has_external_source,
list_sources,
normalize_metadata_source,
resolve_source_ref,
source_group_key,
source_label,
)
from .tensorart import TensorArtSource
__all__ = [
"GROUP_PREFIXES",
"HTTP_TIMEOUT",
"LEGACY_HF_URL_FIELD",
"ModelSource",
"HuggingFaceSource",
"ModelScopeSource",
"SOURCE_PLATFORM_FIELD",
"SOURCE_URL_FIELD",
"SourceRef",
"TensorArtSource",
"USER_AGENT",
"clean_source_url",
"detect_source",
"fetch_text",
"get_source",
"get_source_platform",
"has_external_source",
"list_sources",
"normalize_metadata_source",
"resolve_source_ref",
"source_group_key",
"source_label",
]
py/services/model_sources/base.py
+186
View File
@@ -0,0 +1,186 @@
"""Base types for the external model-source provider abstraction.
A *model source* is a third-party site that hosts model files and a model
card (README) describing them — Hugging Face, ModelScope, TensorArt, and
whatever gets added later. Everything the rest of the codebase needs to
know about such a site is expressed by :class:`ModelSource`:
* how to recognise one of its URLs (:meth:`ModelSource.parse`)
* the canonical page URL for a source id (:meth:`ModelSource.canonical_url`)
* how to fetch the model card (:meth:`ModelSource.fetch_model_card`)
* how to turn repository-relative asset paths into absolute URLs
(:meth:`ModelSource.asset_base_url`)
* which capabilities the site actually supports
(``supports_enrichment`` / ``supports_download``)
Keeping this in one place means the agent pipeline, the scanners, and the
HTTP handlers never need site-specific branching.
"""
from __future__ import annotations
import logging
import re
from dataclasses import dataclass
from typing import Any, Optional
import aiohttp
logger = logging.getLogger(__name__)
#: Shared HTTP timeout for model-card fetches.
HTTP_TIMEOUT = 30
#: User agent used for all model-source HTTP requests.
USER_AGENT = "ComfyUI-LoRA-Manager/1.0"
#: Platform → short prefix used when building version-group keys.
#: ``huggingface`` keeps the historical ``hf:`` prefix for backward
#: compatibility with already-cached group keys.
GROUP_PREFIXES: dict[str, str] = {
"huggingface": "hf",
"modelscope": "ms",
"tensorart": "ta",
}
@dataclass(frozen=True)
class SourceRef:
"""A parsed reference to a model hosted on an external site."""
platform: str
"""Canonical platform id, e.g. ``"huggingface"``."""
source_id: str
"""Site-specific identity, e.g. ``"user/repo"`` or ``"827823520299086029"``."""
url: str
"""Canonical URL of the model page."""
async def fetch_text(url: str, *, timeout: int = HTTP_TIMEOUT) -> str:
"""Fetch *url* and return its body as text, or ``""`` on any failure.
Network problems are expected (offline installs, rate limits, dead
repos) and must never bubble up into the pipeline, so every error is
logged at debug level and normalised to an empty string.
"""
try:
async with aiohttp.ClientSession(
headers={"User-Agent": USER_AGENT},
timeout=aiohttp.ClientTimeout(total=timeout),
) as session:
async with session.get(url) as resp:
if resp.status == 200:
return await resp.text()
logger.debug("Fetch %s returned HTTP %s", url, resp.status)
except Exception as exc: # pragma: no cover - network dependent
logger.debug("Failed to fetch %s: %s", url, exc)
return ""
class ModelSource:
"""Description and I/O for one external model hosting site."""
#: Canonical platform id stored in metadata.
platform: str = ""
#: Human-readable name used in UI copy and prompts.
label: str = ""
#: Whether the agent skill can fetch a model card and run AI extraction.
supports_enrichment: bool = False
#: Whether models can be downloaded directly from this site.
supports_download: bool = False
#: Lenient pattern used to recognise URLs already stored in metadata.
#: Captures the site-specific source id in group ``id``.
url_pattern: re.Pattern[str] | None = None
#: Strict pattern used to validate user input. Must match the whole URL.
strict_url_pattern: re.Pattern[str] | None = None
# ------------------------------------------------------------------
# Parsing
# ------------------------------------------------------------------
def parse(self, url: str, *, strict: bool = False) -> Optional[str]:
"""Return the source id contained in *url*, or ``None``.
With ``strict=True`` the URL must match this site's canonical shape
exactly (used when validating what a user pasted); with
``strict=False`` sub-paths such as ``/resolve/main/file.bin`` are
tolerated (used when normalising already-stored values).
"""
if not url or not isinstance(url, str):
return None
candidate = url.strip()
if not candidate:
return None
pattern = self.strict_url_pattern if strict else self.url_pattern
if pattern is None:
return None
match = pattern.match(candidate)
return match.group("id") if match else None
def ref(self, url: str, *, strict: bool = False) -> Optional[SourceRef]:
"""Return a :class:`SourceRef` for *url*, or ``None`` if not ours."""
source_id = self.parse(url, strict=strict)
if not source_id:
return None
return SourceRef(
platform=self.platform,
source_id=source_id,
url=self.canonical_url(source_id),
)
# ------------------------------------------------------------------
# URLs and content
# ------------------------------------------------------------------
def canonical_url(self, source_id: str) -> str:
"""Return the canonical model-page URL for *source_id*."""
raise NotImplementedError
def asset_base_url(self, source_id: str, revision: str = "") -> str:
"""Base URL used to resolve repository-relative asset paths."""
return ""
def group_key(self, source_id: str) -> str:
"""Return the version-group key for *source_id*."""
prefix = GROUP_PREFIXES.get(self.platform, self.platform)
return f"{prefix}:{source_id}"
async def fetch_model_card(self, source_id: str) -> str:
"""Fetch the raw model card (README) markdown for *source_id*."""
return ""
def __repr__(self) -> str: # pragma: no cover - debugging aid
return f"<ModelSource {self.platform}>"
def clean_source_url(url: Any) -> str:
"""Normalise a stored source URL value into a stripped string."""
if not isinstance(url, str):
return ""
return url.strip()
__all__ = [
"GROUP_PREFIXES",
"HTTP_TIMEOUT",
"ModelSource",
"SourceRef",
"USER_AGENT",
"clean_source_url",
"fetch_text",
]
py/services/model_sources/huggingface.py
+49
View File
@@ -0,0 +1,49 @@
"""Hugging Face model source."""
from __future__ import annotations
import re
from .base import ModelSource, fetch_text
#: Lenient — used to normalise URLs already stored in metadata; tolerates
#: sub-paths such as ``/resolve/main/model.safetensors``.
_URL_PATTERN = re.compile(
r"https?://(?:www\.)?huggingface\.co/(?P<id>[^/?#\s]+/[^/?#\s]+)"
)
#: Strict — validates what the user pasted into the "link model" dialog.
_STRICT_URL_PATTERN = re.compile(
r"https?://(?:www\.)?huggingface\.co/(?P<id>[^/?#\s]+/[^/?#\s]+)/?$"
)
class HuggingFaceSource(ModelSource):
"""Hugging Face Hub (``huggingface.co``)."""
platform = "huggingface"
label = "Hugging Face"
supports_enrichment = True
supports_download = True
url_pattern = _URL_PATTERN
strict_url_pattern = _STRICT_URL_PATTERN
def canonical_url(self, source_id: str) -> str:
return f"https://huggingface.co/{source_id}"
def asset_base_url(self, source_id: str, revision: str = "") -> str:
return f"https://huggingface.co/{source_id}/resolve/{revision or 'main'}"
async def fetch_model_card(self, source_id: str) -> str:
"""Fetch ``README.md`` from Hugging Face (tries ``main``, then ``master``)."""
for branch in ("main", "master"):
text = await fetch_text(
f"https://huggingface.co/{source_id}/raw/{branch}/README.md"
)
if text:
return text
return ""
__all__ = ["HuggingFaceSource"]
py/services/model_sources/modelscope.py
+76
View File
@@ -0,0 +1,76 @@
"""ModelScope (魔搭社区) model source.
ModelScope exposes the same "model card as README.md" convention as
Hugging Face, including a YAML frontmatter block that often carries
``base_model:`` and ``trigger_words:``. Two public endpoints are used,
neither of which requires an API key for public models:
* ``/models/{owner}/{name}/resolve/{revision}/README.md`` — raw model card
* ``/api/v1/models/{owner}/{name}/repo?Revision=..&FilePath=README.md`` —
the same content through the API, used as a fallback when the resolve
URL is unavailable.
"""
from __future__ import annotations
import re
from .base import ModelSource, fetch_text
_URL_PATTERN = re.compile(
r"https?://(?:www\.)?modelscope\.(?:cn|com)/models/(?P<id>[^/?#\s]+/[^/?#\s]+)"
)
#: Trailing view segments the site appends to a model URL; accepted verbatim
#: when the user pastes a browser tab URL.
_VIEW_SEGMENTS = r"(?:summary|files|model-file|readme|community|evaluation)?"
_STRICT_URL_PATTERN = re.compile(
r"https?://(?:www\.)?modelscope\.(?:cn|com)/models/(?P<id>[^/?#\s]+/[^/?#\s]+)"
rf"/?{_VIEW_SEGMENTS}/?$"
)
#: ``master`` is ModelScope's default branch; ``main`` is tried as a fallback
#: for repos imported from Hugging Face.
_REVISIONS = ("master", "main")
class ModelScopeSource(ModelSource):
"""ModelScope (``modelscope.cn``)."""
platform = "modelscope"
label = "ModelScope"
supports_enrichment = True
supports_download = False
url_pattern = _URL_PATTERN
strict_url_pattern = _STRICT_URL_PATTERN
def canonical_url(self, source_id: str) -> str:
return f"https://modelscope.cn/models/{source_id}"
def asset_base_url(self, source_id: str, revision: str = "") -> str:
return f"https://modelscope.cn/models/{source_id}/resolve/{revision or 'master'}"
async def fetch_model_card(self, source_id: str) -> str:
"""Fetch the model card, preferring the raw resolve URL."""
for revision in _REVISIONS:
text = await fetch_text(
f"https://modelscope.cn/models/{source_id}/resolve/{revision}/README.md"
)
if text:
return text
# Fallback: the repo API proxies the same file and is reachable in
# environments where the CDN resolve host is blocked.
for revision in _REVISIONS:
text = await fetch_text(
"https://modelscope.cn/api/v1/models/"
f"{source_id}/repo?Revision={revision}&FilePath=README.md"
)
if text:
return text
return ""
__all__ = ["ModelScopeSource"]
py/services/model_sources/registry.py
+208
View File
@@ -0,0 +1,208 @@
"""Registry and metadata helpers for external model sources.
The registry is the single place the rest of the codebase asks "which site
is this URL from?", "what is this model's source?", and "can we enrich it?".
Import from :mod:`py.services.model_sources` rather than this module
directly.
"""
from __future__ import annotations
import logging
from typing import Any, Dict, Mapping, Optional
from .base import GROUP_PREFIXES, ModelSource, SourceRef, clean_source_url
from .huggingface import HuggingFaceSource
from .modelscope import ModelScopeSource
from .tensorart import TensorArtSource
logger = logging.getLogger(__name__)
#: Order matters only for disambiguation; the URL patterns are disjoint.
_SOURCES: tuple[ModelSource, ...] = (
HuggingFaceSource(),
ModelScopeSource(),
TensorArtSource(),
)
_BY_PLATFORM: Dict[str, ModelSource] = {s.platform: s for s in _SOURCES}
#: Metadata keys that carry the canonical external-source identity.
SOURCE_PLATFORM_FIELD = "source_platform"
SOURCE_URL_FIELD = "source_url"
#: Legacy field kept as a read/write alias for Hugging Face models so that
#: older sidecars, cached rows, and third-party consumers keep working.
LEGACY_HF_URL_FIELD = "hf_url"
def list_sources() -> list[ModelSource]:
"""Return every known model source."""
return list(_SOURCES)
def get_source(platform: Optional[str]) -> Optional[ModelSource]:
"""Return the source registered for *platform*, or ``None``."""
if not platform or not isinstance(platform, str):
return None
return _BY_PLATFORM.get(platform.strip().lower())
def source_label(platform: Optional[str], default: str = "") -> str:
"""Return the human-readable label for *platform*."""
source = get_source(platform)
return source.label if source else default
def detect_source(url: Optional[str], *, strict: bool = False) -> Optional[SourceRef]:
"""Return the :class:`SourceRef` for *url*, or ``None`` if unsupported."""
if not url or not isinstance(url, str):
return None
for source in _SOURCES:
ref = source.ref(url, strict=strict)
if ref is not None:
return ref
return None
def resolve_source_ref(metadata: Mapping[str, Any]) -> Optional[SourceRef]:
"""Return the source reference described by a model's metadata.
Handles all three storage states found in the wild:
1. ``source_url`` + ``source_platform`` (current format)
2. ``hf_url`` only (legacy Hugging Face storage)
3. ``hf_url`` plus a newer ``source_url`` (both written by older builds)
"""
if not isinstance(metadata, Mapping):
return None
platform = clean_source_url(metadata.get(SOURCE_PLATFORM_FIELD)).lower()
url = clean_source_url(metadata.get(SOURCE_URL_FIELD))
legacy = clean_source_url(metadata.get(LEGACY_HF_URL_FIELD))
source = get_source(platform)
if url:
if source is not None:
ref = source.ref(url)
if ref is not None:
return ref
ref = detect_source(url)
if ref is not None:
return ref
# Unknown platform but a URL is present: keep it addressable.
return SourceRef(platform=platform or "unknown", source_id="", url=url)
if legacy:
return detect_source(legacy)
return None
def normalize_metadata_source(metadata: Dict[str, Any]) -> Dict[str, Any]:
"""Normalise the external-source fields on *metadata* in place.
Guarantees that ``source_url``/``source_platform`` are present and
consistent, and that ``hf_url`` mirrors ``source_url`` for Hugging Face
models (never for other platforms, so a stale alias can't make a
ModelScope model look like a Hugging Face one).
Returns the same dict for convenient chaining.
"""
if not isinstance(metadata, dict):
return metadata
platform = clean_source_url(metadata.get(SOURCE_PLATFORM_FIELD)).lower()
url = clean_source_url(metadata.get(SOURCE_URL_FIELD))
legacy = clean_source_url(metadata.get(LEGACY_HF_URL_FIELD))
source = get_source(platform)
ref: Optional[SourceRef] = None
if url:
ref = source.ref(url) if source is not None else None
if ref is None:
ref = detect_source(url)
elif legacy:
ref = detect_source(legacy)
if ref is not None and ref.source_id:
platform = ref.platform
url = ref.url or url
if platform:
metadata[SOURCE_PLATFORM_FIELD] = platform
else:
metadata.setdefault(SOURCE_PLATFORM_FIELD, "")
metadata[SOURCE_URL_FIELD] = url
# Keep the legacy alias in sync, but only for Hugging Face.
if url and platform == "huggingface":
metadata[LEGACY_HF_URL_FIELD] = url
elif LEGACY_HF_URL_FIELD in metadata and platform and platform != "huggingface":
metadata[LEGACY_HF_URL_FIELD] = ""
elif legacy and not url:
metadata[LEGACY_HF_URL_FIELD] = legacy
return metadata
def has_external_source(item: Mapping[str, Any]) -> bool:
"""Return ``True`` when *item* is linked to any external model site."""
if not isinstance(item, Mapping):
return False
return bool(
clean_source_url(item.get(SOURCE_URL_FIELD))
or clean_source_url(item.get(LEGACY_HF_URL_FIELD))
)
def get_source_platform(item: Mapping[str, Any]) -> str:
"""Return the platform id stored on *item* (may be empty)."""
if not isinstance(item, Mapping):
return ""
platform = clean_source_url(item.get(SOURCE_PLATFORM_FIELD)).lower()
if platform:
return platform
ref = resolve_source_ref(item)
return ref.platform if ref else ""
def source_group_key(item: Mapping[str, Any]) -> Optional[str]:
"""Return the version-group key for *item*, or ``None``.
Hugging Face keeps the historical ``hf:{owner}/{repo}`` shape; other
platforms use their own short prefix (see :data:`GROUP_PREFIXES`).
"""
ref = resolve_source_ref(item)
if ref is None or not ref.source_id:
return None
source = get_source(ref.platform)
if source is None:
return None
return source.group_key(ref.source_id)
__all__ = [
"GROUP_PREFIXES",
"LEGACY_HF_URL_FIELD",
"SOURCE_PLATFORM_FIELD",
"SOURCE_URL_FIELD",
"detect_source",
"get_source",
"get_source_platform",
"has_external_source",
"list_sources",
"normalize_metadata_source",
"resolve_source_ref",
"source_group_key",
"source_label",
]
py/services/model_sources/tensorart.py
+56
View File
@@ -0,0 +1,56 @@
"""TensorArt model source (link / provenance only).
TensorArt support is intentionally limited to *linking* a model to its
TensorArt page. Automatic metadata extraction is not possible without a
user session:
* ``tensor.art`` sits behind a Cloudflare managed challenge, so plain
HTTP clients (aiohttp, requests, curl) receive ``403 "Just a moment..."``.
* Its internal API (``ap-east-1.tensorart.cloud`` / ``cn.tensorart.net``)
answers every ``/v1/model/*`` route with
``{"code":100002,"message":"invalid authorization header"}``.
* The official TAMS API requires an AccessKey/SecretKey pair and request
signatures, which is a poor fit for a "paste a URL" workflow.
``supports_enrichment`` is therefore ``False``: the agent pipeline skips
these models with an explicit reason instead of failing silently, and the
UI keeps showing the "View on TensorArt" link. ``tusi.cn`` is TensorArt's
Chinese mirror and is accepted as the same platform.
"""
from __future__ import annotations
import re
from .base import ModelSource
_DOMAINS = r"(?:tensor\.art|tusi\.cn)"
_URL_PATTERN = re.compile(
rf"https?://(?:www\.)?{_DOMAINS}/models/(?P<id>\d+)"
)
_STRICT_URL_PATTERN = re.compile(
rf"https?://(?:www\.)?{_DOMAINS}/models/(?P<id>\d+)(?:/[^/?#\s]+)?/?$"
)
class TensorArtSource(ModelSource):
"""TensorArt (``tensor.art``)."""
platform = "tensorart"
label = "TensorArt"
supports_enrichment = False
supports_download = False
url_pattern = _URL_PATTERN
strict_url_pattern = _STRICT_URL_PATTERN
def canonical_url(self, source_id: str) -> str:
return f"https://tensor.art/models/{source_id}"
def asset_base_url(self, source_id: str, revision: str = "") -> str:
# Unreachable today: enrichment is disabled for this platform.
return f"https://tensor.art/models/{source_id}"
__all__ = ["TensorArtSource"]
py/services/other_model_service.py
+2
View File
@@ -67,6 +67,8 @@ class OtherModelService(BaseModelService):
"civitai": self.filter_civitai_data(model_data.get("civitai", {}), minimal=True),
"auto_tags": model_data.get("auto_tags") or extract_auto_tags(model_data),
"version_count": model_data.get("version_count"),
"source_platform": model_data.get("source_platform", ""),
"source_url": model_data.get("source_url", ""),
"hf_url": model_data.get("hf_url", ""),
}
py/services/persistent_model_cache.py
+17
View File
@@ -7,6 +7,7 @@ from dataclasses import dataclass, field
from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple
from ..utils.cache_paths import CacheType, resolve_cache_path_with_migration
from .model_sources import normalize_metadata_source
logger = logging.getLogger(__name__)
@@ -62,6 +63,8 @@ class PersistentModelCache:
"db_checked",
"last_checked_at",
"hash_status",
"source_platform",
"source_url",
"hf_url",
)
_MODEL_UPDATE_COLUMNS: Tuple[str, ...] = _MODEL_COLUMNS[2:]
@@ -206,8 +209,13 @@ class PersistentModelCache:
"skip_metadata_refresh": bool(row["skip_metadata_refresh"]),
"license_flags": int(license_value),
"hash_status": row["hash_status"] or "completed",
"source_platform": row["source_platform"] or "",
"source_url": row["source_url"] or "",
"hf_url": row["hf_url"] or "",
}
# Legacy rows only carry `hf_url`; derive the canonical pair so
# every consumer sees the same shape.
normalize_metadata_source(item)
if row["autov3"] is not None:
item["autov3"] = (row["autov3"] or "").lower()
raw_data.append(item)
@@ -562,6 +570,8 @@ class PersistentModelCache:
db_checked INTEGER,
last_checked_at REAL,
hash_status TEXT,
source_platform TEXT DEFAULT '',
source_url TEXT DEFAULT '',
hf_url TEXT DEFAULT '',
PRIMARY KEY (model_type, file_path)
);
@@ -629,6 +639,8 @@ class PersistentModelCache:
# Persisting without explicit flags should assume CivitAI's documented defaults (0b111001 == 57).
"license_flags": f"INTEGER DEFAULT {DEFAULT_LICENSE_FLAGS}",
"hash_status": "TEXT DEFAULT 'completed'",
"source_platform": "TEXT DEFAULT ''",
"source_url": "TEXT DEFAULT ''",
"hf_url": "TEXT DEFAULT ''",
"autov3": "TEXT",
}
@@ -650,6 +662,9 @@ class PersistentModelCache:
return conn
def _prepare_model_row(self, model_type: str, item: Dict[str, Any]) -> Tuple[Any, ...]:
# Keep `source_*` and the legacy `hf_url` alias consistent no matter
# which caller populated the item.
normalize_metadata_source(item)
civitai = item.get("civitai") or {}
trained_words = civitai.get("trainedWords")
if isinstance(trained_words, str):
@@ -713,6 +728,8 @@ class PersistentModelCache:
1 if item.get("db_checked") else 0,
float(item.get("last_checked_at") or 0.0),
item.get("hash_status", "completed"),
item.get("source_platform") or "",
item.get("source_url") or "",
item.get("hf_url") or "",
)
py/services/use_cases/bulk_metadata_refresh_use_case.py
+6 -4
View File
@@ -7,6 +7,7 @@ import time
from typing import Any, Dict, List, Optional, Protocol, Sequence
from ..metadata_sync_service import MetadataSyncService
from ..model_sources import has_external_source
from ...utils.metadata_manager import MetadataManager
@@ -51,10 +52,11 @@ class BulkMetadataRefreshUseCase:
if not model.get("skip_metadata_refresh", False)
and not self._is_in_skip_path(model.get("folder", ""), skip_paths)
and (not model.get("civitai") or not model["civitai"].get("id"))
# Skip models downloaded from Hugging Face — they are not on
# CivitAI / CivArchive. Users can still refresh them individually
# via the right-click context menu.
and not model.get("hf_url", "")
# Skip models linked to an external model site (Hugging Face /
# ModelScope / TensorArt) — they are not on CivitAI / CivArchive.
# Users can still refresh them individually via the right-click
# context menu.
and not has_external_source(model)
and not (
# Skip models confirmed not on CivitAI when no need to retry
model.get("from_civitai") is False