alphard/ComfyUI-Lora-Manager
1
0
Fork 0
mirror of https://github.com/willmiao/ComfyUI-Lora-Manager.git synced 2026-09-21 03:01:27 -03:00
Code Issues Packages Projects Releases Wiki Activity

feat(links): support ModelScope and TensorArt as model sources

Browse Source 
A model file could only ever be linked to huggingface.co: `set_hf_url`
validated the URL with a huggingface-only regex, the agent fetched the card
from a hardcoded HF URL, and the readme processor built every relative image
path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the
same model-card convention (README.md + YAML frontmatter, often carrying
`base_model:` and `trigger_words:`) behind a public, key-less API, so the
enrichment pipeline could already serve it - it was the plumbing that was
HF-shaped, not the idea.

Make the external source a first-class, provider-driven concept:

- New `py/services/model_sources/` registry. A `ModelSource` owns URL
  recognition (lenient for stored values, strict for user input), the
  canonical page URL, model-card fetching, the asset base URL and the
  capability flags. `HuggingFaceSource` is the previous logic relocated;
  `ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md`
  and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is
  link-only on purpose: tensor.art answers plain HTTP clients with a
  Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud /
  cn.tensorart.net) rejects every /v1/model/* route with "invalid
  authorization header", so it declares supports_enrichment=False rather than
  failing silently later.
- Metadata gains `source_platform` + `source_url`; `hf_url` stays as a
  read/write alias, written only for Hugging Face, so existing sidecars,
  cached rows and third-party consumers keep working. Normalisation runs at
  the scanner, the persistent cache (both directions, plus two new columns
  behind an ALTER migration) and the linking handler - which is what stops a
  user who switches sources from leaving a stale `hf_url` on a ModelScope
  model.
- The agent pipeline keys off the provider instead of `hf_url`: the fast-fail
  gate now explains *why* a model is skipped (no source / unknown source /
  source without a reachable card), the prompt context exposes
  source_url/source_id/source_label/asset_base_url while still filling the
  legacy hf_url/repo aliases, and the four README image extractors take a
  base_url (defaulting to HF) so relative paths resolve against the right
  site. Version grouping generalises to hf: / ms: / ta: keys.
- `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but
  accepts `source_url`, validates against every provider and returns the
  platform. `GET /api/lm/model-sources` lets the UI render the supported-site
  list from the server.
- Frontend: a `modelSourceHelpers` mirror of the registry drives the link
  dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the
  version-group key and the enrichment gate; the versions tab no longer sends
  ms:/ta: keys to the CivitAI API.

TensorArt stays in the list because provenance is worth keeping even when the
card is unreadable - the dialog says so plainly ("Sites that don't expose one
(currently TensorArt) can only be linked") and the context menu disables
enrichment with a matching tooltip, instead of the user getting
"Unsupported URL".

Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a
1882-byte card whose frontmatter carries base_model/tags/trigger_words, and
relative images resolve to .../resolve/master/....

Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest
tests/i18n and a Jinja compile pass over templates/. The nine locales carry
[TODO: Translate] for the new strings, completed in the next commit.
This commit is contained in:
Will Miao
2026-09-14 07:24:08 +08:00
parent 84146b62fd
commit 5ab0e88abc
51 changed files with 2518 additions and 279 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/agent_skills.md
+17 -5
View File
@@ -62,13 +62,23 @@ Environment variable overrides: `LLM_API_KEY`, `LLM_MODEL`, `LLM_API_BASE`, `LLM
### enrich_hf_metadata
Enriches HuggingFace-downloaded models with metadata extracted by an LLM from the HF model card.
Enriches models linked to an external model site with metadata extracted by an LLM from the site's model card (README).
**Entry point**: Right-click context menu → "Enrich Metadata (Agent)"
**Entry point**: Right-click context menu → "Enrich Metadata with AI"
**Supported model sources**:
| Platform | Link | AI enrichment | Direct download |
| --- | --- | --- | --- |
| Hugging Face | yes | yes | yes |
| ModelScope | yes | yes | no |
| TensorArt | yes | no (see below) | no |
TensorArt is link-only: `tensor.art` sits behind a Cloudflare managed challenge and its internal API requires session authorization, so the backend cannot read its model pages. Linking still stores the canonical page URL and the "View on TensorArt" link works.
**What it does**:
1. Reads the model's `.metadata.json` to get the `hf_url`
2. Fetches the README.md from the HuggingFace repository
1. Reads the model's `.metadata.json` to get the source (`source_platform` + `source_url`, or the legacy `hf_url`)
2. Fetches the model card through the provider in `py/services/model_sources/`
3. Sends the README + local metadata to the LLM for structured extraction
4. Writes extracted fields to `.metadata.json`:
- `base_model` — only if current value is empty
@@ -81,6 +91,8 @@ Enriches HuggingFace-downloaded models with metadata extracted by an LLM from th
6. Updates the scanner cache
7. Broadcasts WebSocket progress events
Models with no source, an unknown source, or a source without model-card access (TensorArt) are skipped with an explicit reason and counted in the run summary.
**Model types**: LoRA, Checkpoint, Embedding
## Adding a New Skill
@@ -129,7 +141,7 @@ Use `{{variable}}` placeholders that will be replaced with data from the `prepar
```markdown
You are an expert assistant...
Model URL: {{hf_url}}
Model URL: {{source_url}}
README content:
{{readme_content}}
locales/de.json
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "Auf CivitAI anzeigen",
"notAvailableFromCivitai": "Nicht auf CivitAI verfügbar",
"viewOnHuggingFace": "Auf Hugging Face ansehen",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "An ComfyUI senden (Klick: Anhängen, Shift+Klick: Ersetzen)",
"copyLoRASyntax": "LoRA-Syntax kopieren",
"checkpointNameCopied": "Checkpoint-Name kopiert",
@@ -874,7 +875,7 @@
"checkUpdates": "Updates prüfen",
"linkModel": "Modell verknüpfen",
"linkCivitai": "Mit CivitAI neu verknüpfen",
"linkHuggingFace": "Mit HuggingFace verknüpfen",
"linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "LoRA-Syntax kopieren",
"copyFilename": "Modell-Dateiname kopieren",
"copyRecipeSyntax": "Rezept-Syntax kopieren",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "Ordnerpfad eingeben oder aus Baum unten auswählen...",
"root": "Stammverzeichnis"
},
"linkHuggingFace": {
"title": "Mit HuggingFace verknüpfen",
"infoText": "Fügen Sie die HuggingFace-Repository-URL ein, um dieses Modell zuzuordnen. Dies ermöglicht die KI-gestützte Metadatenanreicherung.",
"urlLabel": "HuggingFace-Repository-URL:",
"urlPlaceholder": "https://huggingface.co/user/repo",
"helpText": "Geben Sie die vollständige URL des HuggingFace-Repositorys ein.",
"confirmAction": "Speichern & Verknüpfen"
"linkModelSource": {
"title": "[TODO: Translate] Link to Model Source",
"infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
},
"relinkCivitai": {
"title": "Mit CivitAI neu verknüpfen",
@@ -1847,7 +1852,7 @@
"empty": "Noch keine Versionshistorie für dieses Modell vorhanden.",
"error": "Versionen konnten nicht geladen werden.",
"missingModelId": "Für dieses Modell ist keine CivitAI-Model-ID vorhanden.",
"hfGroupInfo": "Dies ist eine HuggingFace-Modellgruppe. Öffnen Sie die Bibliothek, um alle Versionen im Raster zu sehen.",
"sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": {
"delete": "Diese Version aus Ihrer Bibliothek löschen?"
},
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "Modell erfolgreich über CivitArchive neu verknüpft",
"fetchMetadataFirst": "Bitte rufen Sie zuerst Metadaten von CivitAI ab",
"noCivitaiInfo": "Keine CivitAI-Informationen verfügbar",
"missingHash": "Modell-Hash nicht verfügbar"
"missingHash": "Modell-Hash nicht verfügbar",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
},
"exampleImages": {
"pathUpdated": "Beispielbilder-Pfad erfolgreich aktualisiert",
locales/en.json
+18 -11
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "View on CivitAI",
"notAvailableFromCivitai": "Not available from CivitAI",
"viewOnHuggingFace": "View on Hugging Face",
"viewOnSource": "View on {source}",
"sendToWorkflow": "Send to ComfyUI (Click: Append, Shift+Click: Replace)",
"copyLoRASyntax": "Copy LoRA Syntax",
"checkpointNameCopied": "Checkpoint name copied",
@@ -867,14 +868,14 @@
"complete": "Auto-organize complete",
"error": "Error: {error}"
},
"enrichHfAgent": "Enrich HF Metadata (AI)"
"enrichHfAgent": "Enrich Metadata with AI"
},
"contextMenu": {
"refreshMetadata": "Refresh CivitAI Data",
"checkUpdates": "Check Updates",
"linkModel": "Link Model",
"linkCivitai": "Link to CivitAI",
"linkHuggingFace": "Link to HuggingFace",
"linkModelSource": "Link to Model Source",
"copySyntax": "Copy LoRA Syntax",
"copyFilename": "Copy Model Filename",
"copyRecipeSyntax": "Copy Recipe Syntax",
@@ -896,7 +897,7 @@
"viewAllLoras": "View All LoRAs",
"downloadMissingLoras": "Download Missing LoRAs",
"deleteRecipe": "Delete Recipe",
"enrichHfAgent": "Enrich HF Metadata (AI)"
"enrichHfAgent": "Enrich Metadata with AI"
}
},
"recipes": {
@@ -1596,12 +1597,16 @@
"pathPlaceholder": "Type folder path or select from tree below...",
"root": "Root"
},
"linkHuggingFace": {
"title": "Link to HuggingFace",
"infoText": "Paste the HuggingFace repository URL to associate this model with its source. This enables AI-powered metadata enrichment.",
"urlLabel": "HuggingFace Repository URL:",
"linkModelSource": {
"title": "Link to Model Source",
"infoText": "Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "Model Page URL:",
"urlPlaceholder": "https://huggingface.co/user/repo",
"helpText": "Enter the full URL of the HuggingFace repository.",
"helpText": "Enter the full URL of the model page. Supported sites:",
"enrichNote": "AI enrichment needs a readable model card. Sites that don't expose one (currently TensorArt) can only be linked.",
"urlRequired": "Please enter a model page URL.",
"invalidUrl": "Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "Linking model source...",
"confirmAction": "Save & Link"
},
"relinkCivitai": {
@@ -1847,7 +1852,7 @@
"empty": "No version history available for this model yet.",
"error": "Failed to load versions.",
"missingModelId": "This model is missing a CivitAI model id.",
"hfGroupInfo": "This is a HuggingFace model group. Open the library to see all versions in the grid.",
"sourceGroupInfo": "This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": {
"delete": "Delete this version from your library?"
},
@@ -2478,12 +2483,14 @@
"contentRatingFailed": "Failed to set content rating: {message}",
"relinkSuccess": "Model successfully re-linked to CivitAI",
"relinkFailed": "Error: {message}",
"linkHfSuccess": "Model successfully linked to HuggingFace",
"linkHfSuccess": "Model successfully linked to its model source",
"linkHfFailed": "Error: {message}",
"linkCivArchSuccess": "Model successfully re-linked via CivitArchive",
"fetchMetadataFirst": "Please fetch metadata from CivitAI first",
"noCivitaiInfo": "No CivitAI information available",
"missingHash": "Model hash not available"
"missingHash": "Model hash not available",
"enrichNeedsSource": "Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "AI enrichment is not available for {source} models"
},
"exampleImages": {
"pathUpdated": "Example images path updated successfully",
locales/es.json
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "Ver en CivitAI",
"notAvailableFromCivitai": "No disponible en CivitAI",
"viewOnHuggingFace": "Ver en Hugging Face",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "Enviar a ComfyUI (Clic: Añadir, Shift+Clic: Reemplazar)",
"copyLoRASyntax": "Copiar sintaxis de LoRA",
"checkpointNameCopied": "Nombre del checkpoint copiado",
@@ -874,7 +875,7 @@
"checkUpdates": "Comprobar actualizaciones",
"linkModel": "Vincular modelo",
"linkCivitai": "Re-vincular a CivitAI",
"linkHuggingFace": "Vincular a HuggingFace",
"linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "Copiar sintaxis de LoRA",
"copyFilename": "Copiar nombre de archivo del modelo",
"copyRecipeSyntax": "Copiar sintaxis de receta",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "Escribe la ruta de la carpeta o selecciona del árbol de abajo...",
"root": "Raíz"
},
"linkHuggingFace": {
"title": "Vincular a HuggingFace",
"infoText": "Pegue la URL del repositorio de HuggingFace para asociar este modelo. Esto permite el enriquecimiento de metadatos con IA.",
"urlLabel": "URL del repositorio de HuggingFace:",
"urlPlaceholder": "https://huggingface.co/user/repo",
"helpText": "Ingrese la URL completa del repositorio de HuggingFace.",
"confirmAction": "Guardar y vincular"
"linkModelSource": {
"title": "[TODO: Translate] Link to Model Source",
"infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
},
"relinkCivitai": {
"title": "Re-vincular a CivitAI",
@@ -1847,7 +1852,7 @@
"empty": "Aún no hay historial de versiones para este modelo.",
"error": "No se pudieron cargar las versiones.",
"missingModelId": "Este modelo no tiene un ID de modelo de CivitAI.",
"hfGroupInfo": "Este es un grupo de modelos de HuggingFace. Abra la biblioteca para ver todas las versiones en la cuadrícula.",
"sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": {
"delete": "¿Eliminar esta versión de tu biblioteca?"
},
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "Modelo re-vinculado exitosamente mediante CivitArchive",
"fetchMetadataFirst": "Por favor obtén metadatos de CivitAI primero",
"noCivitaiInfo": "No hay información de CivitAI disponible",
"missingHash": "Hash del modelo no disponible"
"missingHash": "Hash del modelo no disponible",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
},
"exampleImages": {
"pathUpdated": "Ruta de imágenes de ejemplo actualizada exitosamente",
locales/fr.json
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "Voir sur CivitAI",
"notAvailableFromCivitai": "Non disponible sur CivitAI",
"viewOnHuggingFace": "Voir sur Hugging Face",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "Envoyer vers ComfyUI (Clic: Ajouter, Maj+Clic: Remplacer)",
"copyLoRASyntax": "Copier la syntaxe LoRA",
"checkpointNameCopied": "Nom du checkpoint copié",
@@ -874,7 +875,7 @@
"checkUpdates": "Vérifier les mises à jour",
"linkModel": "Lier le modèle",
"linkCivitai": "Relier à nouveau à CivitAI",
"linkHuggingFace": "Lier à HuggingFace",
"linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "Copier la syntaxe LoRA",
"copyFilename": "Copier le nom de fichier du modèle",
"copyRecipeSyntax": "Copier la syntaxe de la recipe",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "Tapez le chemin du dossier ou sélectionnez dans l'arbre ci-dessous...",
"root": "Racine"
},
"linkHuggingFace": {
"title": "Lier à HuggingFace",
"infoText": "Collez l'URL du dépôt HuggingFace pour associer ce modèle à sa source. Cela permet l'enrichissement des métadonnées par IA.",
"urlLabel": "URL du dépôt HuggingFace :",
"urlPlaceholder": "https://huggingface.co/user/repo",
"helpText": "Entrez l'URL complète du dépôt HuggingFace.",
"confirmAction": "Enregistrer & lier"
"linkModelSource": {
"title": "[TODO: Translate] Link to Model Source",
"infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
},
"relinkCivitai": {
"title": "Relier à nouveau à CivitAI",
@@ -1847,7 +1852,7 @@
"empty": "Aucun historique de versions n'est disponible pour ce modèle pour le moment.",
"error": "Échec du chargement des versions.",
"missingModelId": "Ce modèle ne possède pas d'identifiant de modèle CivitAI.",
"hfGroupInfo": "Ceci est un groupe de modèles HuggingFace. Ouvrez la bibliothèque pour voir toutes les versions dans la grille.",
"sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": {
"delete": "Supprimer cette version de votre bibliothèque ?"
},
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "Modèle relié via CivitArchive avec succès",
"fetchMetadataFirst": "Veuillez d'abord récupérer les métadonnées depuis CivitAI",
"noCivitaiInfo": "Aucune information CivitAI disponible",
"missingHash": "Hash du modèle non disponible"
"missingHash": "Hash du modèle non disponible",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
},
"exampleImages": {
"pathUpdated": "Chemin des images d'exemple mis à jour avec succès",
locales/he.json
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "הצג ב-CivitAI",
"notAvailableFromCivitai": "לא זמין מ-CivitAI",
"viewOnHuggingFace": "צפייה ב-Hugging Face",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "שלח ל-ComfyUI (לחיצה: הוסף, Shift+לחיצה: החלף)",
"copyLoRASyntax": "העתק תחביר LoRA",
"checkpointNameCopied": "שם Checkpoint הועתק",
@@ -874,7 +875,7 @@
"checkUpdates": "בדוק עדכונים",
"linkModel": "קישור מודל",
"linkCivitai": "קשר מחדש ל-CivitAI",
"linkHuggingFace": "קישור ל-HuggingFace",
"linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "העתק תחביר LoRA",
"copyFilename": "העתק שם קובץ מודל",
"copyRecipeSyntax": "העתק תחביר מתכון",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "הקלד נתיב תיקייה או בחר מהעץ למטה...",
"root": "שורש"
},
"linkHuggingFace": {
"title": "קישור ל-HuggingFace",
"infoText": "הדבק את כתובת ה-URL של מאגר HuggingFace כדי לשייך מודל זה למקורו. פעולה זו מאפשרת העשרת מטא-נתונים באמצעות AI.",
"urlLabel": "כתובת URL של מאגר HuggingFace:",
"urlPlaceholder": "https://huggingface.co/user/repo",
"helpText": "הזן את כתובת ה-URL המלאה של מאגר HuggingFace.",
"confirmAction": "שמור וקשר"
"linkModelSource": {
"title": "[TODO: Translate] Link to Model Source",
"infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
},
"relinkCivitai": {
"title": "קשר מחדש ל-CivitAI",
@@ -1847,7 +1852,7 @@
"empty": "אין עדיין היסטוריית גרסאות למודל זה.",
"error": "טעינת הגרסאות נכשלה.",
"missingModelId": "למודל זה אין מזהה מודל של CivitAI.",
"hfGroupInfo": "זוהי קבוצת מודלים של HuggingFace. פתח את הספרייה כדי לראות את כל הגרסאות ברשת.",
"sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": {
"delete": "למחוק גרסה זו מהספרייה שלך?"
},
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "המודל קושר מחדש דרך CivitArchive בהצלחה",
"fetchMetadataFirst": "אנא אחזר מטא-נתונים מ-CivitAI תחילה",
"noCivitaiInfo": "אין מידע מ-CivitAI זמין",
"missingHash": "ה-hash של המודל אינו זמין"
"missingHash": "ה-hash של המודל אינו זמין",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
},
"exampleImages": {
"pathUpdated": "נתיב תמונות הדוגמה עודכן בהצלחה",
locales/ja.json
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "CivitAIで表示",
"notAvailableFromCivitai": "CivitAIでは利用できません",
"viewOnHuggingFace": "Hugging Face で見る",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "ComfyUIに送信（クリック：追加、Shift+クリック：置換）",
"copyLoRASyntax": "LoRA構文をコピー",
"checkpointNameCopied": "Checkpointの名前をコピーしました",
@@ -874,7 +875,7 @@
"checkUpdates": "更新確認",
"linkModel": "モデルをリンク",
"linkCivitai": "CivitAI にリンク",
"linkHuggingFace": "HuggingFace にリンク",
"linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "LoRA構文をコピー",
"copyFilename": "モデルファイル名をコピー",
"copyRecipeSyntax": "レシピ構文をコピー",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "フォルダパスを入力するか、下のツリーから選択...",
"root": "ルート"
},
"linkHuggingFace": {
"title": "HuggingFace にリンク",
"infoText": "HuggingFace リポジトリの URL を貼り付けてモデルを関連付けます。AI によるメタデータ補完が有効になります。",
"urlLabel": "HuggingFace リポジトリ URL：",
"urlPlaceholder": "https://huggingface.co/user/repo",
"helpText": "完全な HuggingFace リポジトリ URL を入力してください。",
"confirmAction": "保存＆リンク"
"linkModelSource": {
"title": "[TODO: Translate] Link to Model Source",
"infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
},
"relinkCivitai": {
"title": "CivitAIに再リンク",
@@ -1847,7 +1852,7 @@
"empty": "このモデルにはまだバージョン履歴がありません。",
"error": "バージョンの読み込みに失敗しました。",
"missingModelId": "このモデルにはCivitAIのモデルIDがありません。",
"hfGroupInfo": "これは HuggingFace モデルグループです。ライブラリを開いてグリッドですべてのバージョンを表示してください。",
"sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": {
"delete": "このバージョンをライブラリから削除しますか？"
},
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "モデルがCivitArchive経由で正常に再リンクされました",
"fetchMetadataFirst": "最初にCivitAIからメタデータを取得してください",
"noCivitaiInfo": "CivitAI情報が利用できません",
"missingHash": "モデルハッシュが利用できません"
"missingHash": "モデルハッシュが利用できません",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
},
"exampleImages": {
"pathUpdated": "例画像パスが正常に更新されました",
locales/ko.json
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "CivitAI에서 보기",
"notAvailableFromCivitai": "CivitAI에서 사용할 수 없음",
"viewOnHuggingFace": "Hugging Face에서 보기",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "ComfyUI로 전송 (클릭: 추가, Shift+클릭: 교체)",
"copyLoRASyntax": "LoRA 문법 복사",
"checkpointNameCopied": "Checkpoint 이름 복사됨",
@@ -874,7 +875,7 @@
"checkUpdates": "업데이트 확인",
"linkModel": "모델 연결",
"linkCivitai": "CivitAI에 연결",
"linkHuggingFace": "HuggingFace에 연결",
"linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "LoRA 문법 복사",
"copyFilename": "모델 파일명 복사",
"copyRecipeSyntax": "레시피 문법 복사",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "폴더 경로를 입력하거나 아래 트리에서 선택하세요...",
"root": "루트"
},
"linkHuggingFace": {
"title": "HuggingFace에 연결",
"infoText": "HuggingFace 저장소 URL을 붙여넣어 모델을 연결합니다. AI 메타데이터 보강 기능을 사용할 수 있습니다.",
"urlLabel": "HuggingFace 저장소 URL：",
"urlPlaceholder": "https://huggingface.co/user/repo",
"helpText": "전체 HuggingFace 저장소 URL을 입력하세요.",
"confirmAction": "저장 및 연결"
"linkModelSource": {
"title": "[TODO: Translate] Link to Model Source",
"infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
},
"relinkCivitai": {
"title": "CivitAI에 다시 연결",
@@ -1847,7 +1852,7 @@
"empty": "이 모델에는 아직 버전 기록이 없습니다.",
"error": "버전을 불러오지 못했습니다.",
"missingModelId": "이 모델에는 CivitAI 모델 ID가 없습니다.",
"hfGroupInfo": "HuggingFace 모델 그룹입니다. 라이브러리를 열어 그리드에서 모든 버전을 확인하세요.",
"sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": {
"delete": "이 버전을 라이브러리에서 삭제하시겠습니까?"
},
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "모델이 CivitArchive을 통해 성공적으로 다시 연결되었습니다",
"fetchMetadataFirst": "먼저 CivitAI에서 메타데이터를 가져와주세요",
"noCivitaiInfo": "사용 가능한 CivitAI 정보가 없습니다",
"missingHash": "모델 해시를 사용할 수 없습니다"
"missingHash": "모델 해시를 사용할 수 없습니다",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
},
"exampleImages": {
"pathUpdated": "예시 이미지 경로가 성공적으로 업데이트되었습니다",
locales/ru.json
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "Посмотреть на CivitAI",
"notAvailableFromCivitai": "Недоступно на CivitAI",
"viewOnHuggingFace": "Открыть Hugging Face",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "Отправить в ComfyUI (Клик: Добавить, Shift+Клик: Заменить)",
"copyLoRASyntax": "Копировать синтаксис LoRA",
"checkpointNameCopied": "Имя checkpoint скопировано",
@@ -874,7 +875,7 @@
"checkUpdates": "Проверить обновления",
"linkModel": "Связать модель",
"linkCivitai": "Пересвязать с CivitAI",
"linkHuggingFace": "Связать с HuggingFace",
"linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "Копировать синтаксис LoRA",
"copyFilename": "Копировать имя файла модели",
"copyRecipeSyntax": "Копировать синтаксис рецепта",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "Введите путь к папке или выберите из дерева ниже...",
"root": "Корень"
},
"linkHuggingFace": {
"title": "Связать с HuggingFace",
"infoText": "Вставьте URL репозитория HuggingFace, чтобы связать эту модель с её источником. Это позволит обогащать метаданные с помощью ИИ.",
"urlLabel": "URL репозитория HuggingFace:",
"urlPlaceholder": "https://huggingface.co/user/repo",
"helpText": "Введите полный URL репозитория HuggingFace.",
"confirmAction": "Сохранить и связать"
"linkModelSource": {
"title": "[TODO: Translate] Link to Model Source",
"infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
},
"relinkCivitai": {
"title": "Пересвязать с CivitAI",
@@ -1847,7 +1852,7 @@
"empty": "Для этой модели пока нет истории версий.",
"error": "Не удалось загрузить версии.",
"missingModelId": "У этой модели отсутствует идентификатор модели CivitAI.",
"hfGroupInfo": "Это группа моделей HuggingFace. Откройте библиотеку, чтобы увидеть все версии в сетке.",
"sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": {
"delete": "Удалить эту версию из библиотеки?"
},
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "Модель успешно пересвязана через CivitArchive",
"fetchMetadataFirst": "Пожалуйста, сначала получите метаданные с CivitAI",
"noCivitaiInfo": "Информация CivitAI недоступна",
"missingHash": "Хеш модели недоступен"
"missingHash": "Хеш модели недоступен",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
},
"exampleImages": {
"pathUpdated": "Путь к примерам изображений успешно обновлен",
locales/zh-CN.json
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "在 CivitAI 查看",
"notAvailableFromCivitai": "CivitAI 上不可用",
"viewOnHuggingFace": "在 Hugging Face 查看",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "发送到 ComfyUI（点击：追加，Shift+点击：替换）",
"copyLoRASyntax": "复制 LoRA 语法",
"checkpointNameCopied": "Checkpoint 名称已复制",
@@ -874,7 +875,7 @@
"checkUpdates": "检查更新",
"linkModel": "链接模型",
"linkCivitai": "链接到 CivitAI",
"linkHuggingFace": "链接到 HuggingFace",
"linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "复制 LoRA 语法",
"copyFilename": "复制模型文件名",
"copyRecipeSyntax": "复制配方语法",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "输入文件夹路径或从下方树中选择...",
"root": "根目录"
},
"linkHuggingFace": {
"title": "链接到 HuggingFace",
"infoText": "粘贴 HuggingFace 仓库 URL 以关联此模型。关联后可启用 AI 元数据增强功能。",
"urlLabel": "HuggingFace 仓库 URL：",
"urlPlaceholder": "https://huggingface.co/user/repo",
"helpText": "请输入完整的 HuggingFace 仓库 URL。",
"confirmAction": "保存并链接"
"linkModelSource": {
"title": "[TODO: Translate] Link to Model Source",
"infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
},
"relinkCivitai": {
"title": "重新关联到 CivitAI",
@@ -1847,7 +1852,7 @@
"empty": "该模型还没有版本历史。",
"error": "加载版本失败。",
"missingModelId": "该模型缺少 CivitAI 模型 ID。",
"hfGroupInfo": "这是一个 HuggingFace 模型组。打开库页面即可在网格中查看所有版本。",
"sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": {
"delete": "从库中删除此版本？"
},
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "模型已成功通过 CivitArchive 重新关联",
"fetchMetadataFirst": "请先从 CivitAI 获取元数据",
"noCivitaiInfo": "无 CivitAI 信息",
"missingHash": "模型哈希不可用"
"missingHash": "模型哈希不可用",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
},
"exampleImages": {
"pathUpdated": "示例图片路径更新成功",
locales/zh-TW.json
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "在 CivitAI 查看",
"notAvailableFromCivitai": "CivitAI 不提供",
"viewOnHuggingFace": "在 Hugging Face 查看",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "傳送到 ComfyUI（點擊：附加，Shift+點擊：取代）",
"copyLoRASyntax": "複製 LoRA 語法",
"checkpointNameCopied": "Checkpoint 名稱已複製",
@@ -874,7 +875,7 @@
"checkUpdates": "檢查更新",
"linkModel": "連結模型",
"linkCivitai": "連結到 CivitAI",
"linkHuggingFace": "連結到 HuggingFace",
"linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "複製 LoRA 語法",
"copyFilename": "複製模型檔名",
"copyRecipeSyntax": "複製配方語法",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "輸入資料夾路徑或從下方樹狀結構選擇...",
"root": "根目錄"
},
"linkHuggingFace": {
"title": "連結到 HuggingFace",
"infoText": "貼上 HuggingFace 倉庫 URL 以關聯此模型。關聯後可啟用 AI 中繼資料增強功能。",
"urlLabel": "HuggingFace 倉庫 URL：",
"urlPlaceholder": "https://huggingface.co/user/repo",
"helpText": "請輸入完整的 HuggingFace 倉庫 URL。",
"confirmAction": "儲存並連結"
"linkModelSource": {
"title": "[TODO: Translate] Link to Model Source",
"infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
},
"relinkCivitai": {
"title": "重新連結至 CivitAI",
@@ -1847,7 +1852,7 @@
"empty": "此模型尚無版本歷史。",
"error": "載入版本失敗。",
"missingModelId": "此模型缺少 CivitAI 模型 ID。",
"hfGroupInfo": "這是一個 HuggingFace 模型組。打開庫頁面即可在網格中查看所有版本。",
"sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": {
"delete": "要從庫中刪除此版本嗎？"
},
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "模型已成功透過 CivitArchive 重新連結",
"fetchMetadataFirst": "請先從 CivitAI 取得 metadata",
"noCivitaiInfo": "無 CivitAI 資訊",
"missingHash": "模型雜湊不可用"
"missingHash": "模型雜湊不可用",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
},
"exampleImages": {
"pathUpdated": "範例圖片路徑已更新",
py/routes/handlers/hf_handlers.py
+97 -21
View File
@@ -22,6 +22,11 @@ from ...services.downloader import (
get_downloader,
)
from ...services.aria2_downloader import Aria2Downloader
from ...services.model_sources import (
detect_source,
list_sources,
normalize_metadata_source,
)
from ...services.settings_manager import get_settings_manager
from ...services.service_registry import ServiceRegistry
from ...services.websocket_manager import ws_manager
@@ -120,6 +125,8 @@ async def _save_hf_metadata(dest_path: str, repo: str, model_root: str) -> None:
# 2. Overlay HF-specific fields
metadata._unknown_fields["hf_url"] = hf_url
metadata._unknown_fields["source_url"] = hf_url
metadata._unknown_fields["source_platform"] = "huggingface"
metadata.from_civitai = False # HF models are not from CivitAI
# 3. Save metadata atomically
@@ -189,27 +196,72 @@ async def _add_to_scanner_cache(dest_path: str, metadata: dict[str, Any]) -> Non
class HfHandler:
"""Handle Hugging Face model browsing and download."""
async def get_model_sources(self, request: web.Request) -> web.Response:
"""List the external model sites the UI can link a model to.
Used by the "Link Model" dialog to validate URLs client-side and to
explain which sites support AI metadata enrichment.
"""
return web.json_response([
{
"platform": source.platform,
"label": source.label,
"supports_enrichment": source.supports_enrichment,
"supports_download": source.supports_download,
"example_url": source.canonical_url(
"user/repo" if source.platform != "tensorart" else "827823520299086029"
),
}
for source in list_sources()
])
async def set_hf_url(self, request: web.Request) -> web.Response:
"""Link a model file to its page on an external model site.
Accepts ``source_url`` (preferred) or the legacy ``hf_url`` /
``url`` payload key. Hugging Face, ModelScope, and TensorArt URLs
are recognised; the platform is stored alongside the canonical URL.
TensorArt models can be linked and browsed, but not AI-enriched.
"""
try:
payload: dict[str, Any] = await request.json()
except json.JSONDecodeError:
return web.json_response({"success": False, "error": "Invalid JSON"}, status=400)
file_path = (payload.get("file_path") or "").strip()
hf_url = (payload.get("hf_url") or "").strip()
raw_url = (
payload.get("source_url")
or payload.get("hf_url")
or payload.get("url")
or ""
)
source_url = raw_url.strip() if isinstance(raw_url, str) else ""
if not file_path or not hf_url:
return web.json_response(
{"success": False, "error": "Missing required fields: 'file_path' and 'hf_url'"},
status=400,
)
m = re.match(r"^https?://huggingface\.co/([^/]+/[^/]+)/?$", hf_url)
if not m:
if not file_path or not source_url:
return web.json_response(
{
"success": False,
"error": "Invalid HuggingFace URL. Expected format: https://huggingface.co/user/repo",
"error": "Missing required fields: 'file_path' and 'source_url'",
},
status=400,
)
ref = detect_source(source_url, strict=True)
if ref is None:
return web.json_response(
{
"success": False,
"error": (
"Unsupported model URL. Supported formats: "
+ ", ".join(
f"{s.label} ({s.canonical_url('user/repo')})"
if s.platform != "tensorart"
else f"{s.label} (https://tensor.art/models/<id>)"
for s in list_sources()
)
),
},
status=400,
)
@@ -225,37 +277,61 @@ class HfHandler:
return web.json_response(
{
"success": False,
"error": "File is not within any configured model directory. Cannot link to HuggingFace.",
"error": "File is not within any configured model directory. Cannot link to a model source.",
},
status=400,
)
try:
existing = await MetadataManager.load_metadata_payload(file_path)
if existing.get("hf_url") == hf_url:
already_linked = (
(existing.get("source_url") or "").strip() == ref.url
and (existing.get("source_platform") or "").strip().lower()
== ref.platform
) or (
not existing.get("source_url")
and ref.platform == "huggingface"
and (existing.get("hf_url") or "").strip() == ref.url
)
if already_linked:
return web.json_response({
"success": True,
"message": "hf_url already set",
"hf_url": hf_url,
"message": "source_url already set",
"source_url": ref.url,
"source_platform": ref.platform,
"hf_url": ref.url if ref.platform == "huggingface" else "",
})
existing["hf_url"] = hf_url
existing["source_url"] = ref.url
existing["source_platform"] = ref.platform
if ref.platform == "huggingface":
existing["hf_url"] = ref.url
else:
existing.pop("hf_url", None)
normalize_metadata_source(existing)
# NOTE: deliberately do NOT touch `from_civitai` here. It records
# where the metadata came from, and the UI must show the CivitAI
# link whenever CivitAI data is present — linking HuggingFace must
# not hide it (#1094). HF provenance is tracked via `hf_url`.
# link whenever CivitAI data is present — linking an external
# source must not hide it (#1094). Source provenance is tracked
# via `source_platform` / `source_url`.
await MetadataManager.save_metadata(file_path, existing)
await _add_to_scanner_cache(file_path, existing)
logger.info("Set hf_url=%s for %s", hf_url, file_path)
logger.info(
"Linked %s to %s source (%s)", file_path, ref.platform, ref.url
)
return web.json_response({
"success": True,
"message": f"hf_url set to {hf_url}",
"hf_url": hf_url,
"message": f"Linked to {ref.url}",
"source_url": ref.url,
"source_platform": ref.platform,
"hf_url": existing.get("hf_url", ""),
})
except Exception as exc:
logger.error("Failed to set hf_url for %s: %s", file_path, exc)
logger.error("Failed to link %s to a model source: %s", file_path, exc)
return web.json_response(
{"success": False, "error": str(exc)},
status=500,
py/routes/handlers/misc_handlers.py
+1
View File
@@ -4079,6 +4079,7 @@ class MiscHandlerSet:
"get_hf_repo_files": self.hf_handler.get_hf_repo_files,
"download_hf_model": self.hf_handler.download_hf_model,
"set_hf_url": self.hf_handler.set_hf_url,
"get_model_sources": self.hf_handler.get_model_sources,
# Agent skill handlers
"get_agent_skills": self.agent_handler.get_agent_skills,
"execute_agent_skill": self.agent_handler.execute_agent_skill,
py/routes/misc_route_registrar.py
+4
View File
@@ -113,6 +113,10 @@ MISC_ROUTE_DEFINITIONS: tuple[RouteDefinition, ...] = (
RouteDefinition(
"POST", "/api/lm/set-hf-url", "set_hf_url"
),
# Supported external model sites (Hugging Face / ModelScope / TensorArt)
RouteDefinition(
"GET", "/api/lm/model-sources", "get_model_sources"
),
# Agent skill endpoints
RouteDefinition(
"GET", "/api/lm/agent/skills", "get_agent_skills"
py/services/agent/agent_service.py
+69 -32
View File
@@ -19,16 +19,18 @@ from __future__ import annotations
import asyncio
import json
import logging
import os
import re
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional
import aiohttp
import os
from ...config import config
from ..llm_service import LLMService
from ..model_sources import (
get_source,
resolve_source_ref,
source_label,
)
from ..websocket_manager import ws_manager
from .post_processor import PostProcessor
from .skill_registry import SkillRegistry
@@ -267,14 +269,17 @@ class AgentService:
from ...metadata_ops import read_metadata
metadata = await read_metadata(model_path)
# Fast-fail: enrich_hf_metadata requires hf_url to have HF README context
if skill_name == "enrich_hf_metadata" and not metadata.get("hf_url", ""):
logger.info(
"[%s] SKIP %s — no hf_url in metadata",
skill_name, model_filename,
)
skipped_count += 1
skip_model = True
# Fast-fail: enrich_hf_metadata needs an external model source
# that exposes an accessible model card.
if skill_name == "enrich_hf_metadata":
skip_reason = self._enrichment_skip_reason(metadata)
if skip_reason:
logger.info(
"[%s] SKIP %s — %s",
skill_name, model_filename, skip_reason,
)
skipped_count += 1
skip_model = True
if not skip_model:
prompt_vars: Dict[str, Any] = {"model_path": model_path}
@@ -358,6 +363,28 @@ class AgentService:
# Base model grouping (keeps the prompt compact)
# ------------------------------------------------------------------
@staticmethod
def _enrichment_skip_reason(metadata: Dict[str, Any]) -> str:
"""Return why ``enrich_hf_metadata`` cannot run, or ``""`` if it can.
Distinguishes the three cases the user can act on: no source linked,
a source we don't know, and a known source whose model card is not
reachable from the backend (TensorArt).
"""
ref = resolve_source_ref(metadata)
if ref is None:
return "no model source linked (source_url missing)"
source = get_source(ref.platform)
if source is None:
return f"unsupported model source platform '{ref.platform}'"
if not source.supports_enrichment:
return (
f"{source.label} does not expose a model card to the backend; "
"AI metadata enrichment is not available for this source"
)
return ""
@staticmethod
def _format_base_models(models: List[str]) -> str:
"""Format the base model list as a flat, one-per-line list.
@@ -388,6 +415,14 @@ class AgentService:
context: Dict[str, Any] = {
"model_path": model_path,
"model_basename": "",
# Canonical external-source variables
"source_url": "",
"source_id": "",
"source_platform": "",
"source_label": "",
"asset_base_url": "",
# Legacy Hugging Face aliases (kept so older prompt templates and
# third-party skills keep rendering)
"hf_url": "",
"repo": "",
"readme_content": "",
@@ -411,12 +446,20 @@ class AgentService:
"size": metadata.get("size", 0),
}
hf_url = metadata.get("hf_url", "")
context["hf_url"] = hf_url
repo = self._extract_repo_from_url(hf_url) if hf_url else ""
context["repo"] = repo or ""
if repo:
readme = await self._fetch_readme(repo)
ref = resolve_source_ref(metadata)
if ref is not None:
context["source_url"] = ref.url
context["source_id"] = ref.source_id
context["source_platform"] = ref.platform
context["source_label"] = source_label(ref.platform, ref.platform)
if ref.platform == "huggingface":
context["hf_url"] = ref.url
context["repo"] = ref.source_id
source = get_source(ref.platform) if ref is not None else None
if ref is not None and source is not None and source.supports_enrichment:
context["asset_base_url"] = source.asset_base_url(ref.source_id)
readme = await source.fetch_model_card(ref.source_id)
# Trim README to the section relevant to this model file
# (collection repos often have multiple models in one README).
if readme and raw_basename:
@@ -458,20 +501,14 @@ class AgentService:
@staticmethod
async def _fetch_readme(repo: str) -> str:
"""Fetch README.md from HuggingFace (tries ``main``, then ``master``)."""
async with aiohttp.ClientSession(
headers={"User-Agent": "ComfyUI-LoRA-Manager/1.0"},
timeout=aiohttp.ClientTimeout(total=30),
) as session:
for branch in ("main", "master"):
url = f"https://huggingface.co/{repo}/raw/{branch}/README.md"
try:
async with session.get(url) as resp:
if resp.status == 200:
return await resp.text()
except Exception as exc:
logger.debug("Failed to fetch README from %s: %s", url, exc)
return ""
"""Fetch a Hugging Face README (tries ``main``, then ``master``).
Kept for backward compatibility; new code should go through the
model-source registry so every supported site works.
"""
from ..model_sources import HuggingFaceSource
return await HuggingFaceSource().fetch_model_card(repo)
async def _emit_progress(
self,
py/services/agent/post_processor.py
+31 -19
View File
@@ -78,6 +78,7 @@ class PostProcessor:
download_preview,
refresh_cache,
)
from ..model_sources import get_source, has_external_source, resolve_source_ref
from .skills.enrich_hf_metadata.readme_processor import (
convert_readme_to_html,
extract_gallery_images,
@@ -85,17 +86,25 @@ class PostProcessor:
extract_relevant_section,
extract_simple_markdown_images,
extract_html_img_tags,
extract_repo_from_hf_url,
)
updated_fields: List[str] = []
preview_downloaded = False
# -- Determine whether this is an HF-sourced model -----------------
# Key off `hf_url` directly: `from_civitai` records provenance and can
# be true for a model that is also linked to HuggingFace (both sources
# coexist, see #1094), so it must not gate HF enrichment.
is_hf_model = bool(metadata.get("hf_url", ""))
# -- Determine whether this is an externally-sourced model ---------
# Key off the source fields directly: `from_civitai` records provenance
# and can be true for a model that is also linked to an external site
# (both sources coexist, see #1094), so it must not gate enrichment.
is_source_model = has_external_source(metadata)
source_ref = resolve_source_ref(metadata)
source = get_source(source_ref.platform) if source_ref else None
source_id = source_ref.source_id if source_ref else ""
asset_base_url = (
source.asset_base_url(source_id)
if source is not None and source_id
else None
)
# -- Collect updates -----------------------------------------------
updates: Dict[str, Any] = {}
@@ -103,7 +112,7 @@ class PostProcessor:
# base_model
new_base = (llm_output.get("base_model") or "").strip()
current_base = metadata.get("base_model", "") or ""
if new_base and self._should_overwrite(current_base, is_hf_model):
if new_base and self._should_overwrite(current_base, is_source_model):
updates["base_model"] = new_base
# trigger words → civitai.trainedWords
@@ -115,7 +124,7 @@ class PostProcessor:
trigger_words_empty = not cleaned
current_civitai = metadata.get("civitai") or {}
current_triggers = current_civitai.get("trainedWords") or []
if self._should_overwrite_list(current_triggers, is_hf_model):
if self._should_overwrite_list(current_triggers, is_source_model):
trig_civitai = dict(current_civitai)
if "civitai" in updates and isinstance(updates["civitai"], dict):
trig_civitai.update(updates["civitai"])
@@ -123,14 +132,14 @@ class PostProcessor:
updates["civitai"] = trig_civitai
# modelDescription — from raw README content (converted to HTML)
if readme_content and is_hf_model:
if readme_content and is_source_model:
converted = convert_readme_to_html(readme_content)
if converted:
updates["modelDescription"] = converted
# short_description → civitai.description (for "About this version")
short_desc = (llm_output.get("short_description") or "").strip()
if short_desc and is_hf_model:
if short_desc and is_source_model:
current_civitai = metadata.get("civitai") or {}
desc_civitai = dict(current_civitai)
if "civitai" in updates and isinstance(updates["civitai"], dict):
@@ -141,9 +150,8 @@ class PostProcessor:
# gallery images → civitai.images (from YAML frontmatter widget entries
# and Sample Gallery markdown tables in the README body)
gallery_images: List[Dict[str, Any]] = []
if readme_content and is_hf_model:
hf_url = metadata.get("hf_url", "") or ""
repo = extract_repo_from_hf_url(hf_url)
if readme_content and is_source_model:
repo = source_id
if repo:
rec_w = llm_output.get("recommended_width") or 0
rec_h = llm_output.get("recommended_height") or 0
@@ -152,6 +160,7 @@ class PostProcessor:
gallery = extract_gallery_images(
readme_content, repo,
default_width=rec_w, default_height=rec_h,
base_url=asset_base_url,
)
# 2. Sample Gallery table images (markdown body), deduplicated
@@ -160,6 +169,7 @@ class PostProcessor:
readme_content, repo,
existing_urls=existing_urls,
default_width=rec_w, default_height=rec_h,
base_url=asset_base_url,
)
existing_urls.update(img["url"] for img in table_images if img.get("url"))
@@ -168,6 +178,7 @@ class PostProcessor:
readme_content, repo,
existing_urls=existing_urls,
default_width=rec_w, default_height=rec_h,
base_url=asset_base_url,
)
existing_urls.update(img["url"] for img in simple_images if img.get("url"))
@@ -176,6 +187,7 @@ class PostProcessor:
readme_content, repo,
existing_urls=existing_urls,
default_width=rec_w, default_height=rec_h,
base_url=asset_base_url,
)
all_images = gallery + table_images + simple_images + html_images
@@ -193,7 +205,7 @@ class PostProcessor:
if isinstance(new_tags, list) and new_tags:
existing_tags = metadata.get("tags") or []
merged = self._merge_tags(existing_tags, new_tags)
if len(merged) > len(existing_tags) or is_hf_model:
if len(merged) > len(existing_tags) or is_source_model:
updates["tags"] = merged
# metadata_source & llm_enriched_at (always set)
@@ -222,7 +234,7 @@ class PostProcessor:
# README, find the first gallery image from the *model-specific
# section* of the README (not the repo-wide first image, which
# belongs to a different model in collection repos).
if not preview_remote_url and readme_content and is_hf_model:
if not preview_remote_url and readme_content and is_source_model:
model_basename = os.path.splitext(os.path.basename(model_path))[0]
relevant_section = extract_relevant_section(
readme_content, model_basename,
@@ -279,16 +291,16 @@ class PostProcessor:
# ------------------------------------------------------------------
@staticmethod
def _should_overwrite(current_value: str, is_hf_model: bool) -> bool:
def _should_overwrite(current_value: str, is_source_model: bool) -> bool:
"""Return ``True`` when a scalar field should be overwritten."""
return is_hf_model or not current_value or current_value.lower() in (
return is_source_model or not current_value or current_value.lower() in (
"", "unknown",
)
@staticmethod
def _should_overwrite_list(current_list: List[str], is_hf_model: bool) -> bool:
def _should_overwrite_list(current_list: List[str], is_source_model: bool) -> bool:
"""Return ``True`` when a list field should be overwritten."""
return is_hf_model or not current_list
return is_source_model or not current_list
@staticmethod
def _merge_tags(existing: List[str], new: List[str]) -> List[str]:
py/services/agent/skills/enrich_hf_metadata/prompt.md
+12 -9
View File
@@ -1,20 +1,23 @@
---
name: enrich_hf_metadata
title: "Enrich Metadata from HuggingFace"
title: "Enrich Metadata from Model Card"
description: >
Parse the HuggingFace model card via LLM to extract description, trigger
words, base model, tags, and preview image URL.
Parse the model card (README) from HuggingFace, ModelScope, or any other
supported model site via LLM to extract description, trigger words, base
model, tags, and preview image URL.
llm_required: true
---
You are an expert assistant for AI image generation models. Your task is to extract structured metadata from a HuggingFace model card (README.md).
You are an expert assistant for AI image generation models. Your task is to extract structured metadata from a model card (README).
## Model Information
- **Repository**: {{hf_url}}
- **Source site**: {{source_label}} ({{source_platform}})
- **Model page**: {{source_url}}
- **Model file path**: {{model_path}}
- **Model filename**: {{model_basename}}
- **Repository ID**: {{repo}}
- **Repository ID**: {{source_id}}
- **Repository raw-file base URL**: {{asset_base_url}}
## Current Metadata (may be incomplete)
@@ -39,7 +42,7 @@ name listed — do not invent aliases or modify variant suffixes.
{{base_models}}
## HuggingFace README Content
## Model Card Content
```
{{readme_content}}
@@ -92,7 +95,7 @@ The URL of the most suitable preview image from the README. Look for:
- The YAML frontmatter `widget:` section (which often has `output.url` fields)
- In collection repos: the sample images listed **under the section** for this specific model version
- Generic `![alt](url)` in the body
Choose the first image that appears to be a generation example (not a logo or diagram). Construct the absolute URL as `https://huggingface.co/{{repo}}/resolve/main/{filename}`. If no suitable image is found, return an empty string.
Choose the first image that appears to be a generation example (not a logo or diagram). Construct the absolute URL from the repository raw-file base URL (`{{asset_base_url}}`) plus the relative path. If no suitable image is found, return an empty string.
### notes
A plain-text summary of the model card's key practical usage information. Combine trigger words, style modifiers, recommended parameters (steps, CFG, resolution, sampler), and any setup tips into a readable paragraph. For collection repos, focus on the **specific model version** matching `{{model_basename}}`. Return empty string if the README has no useful usage info.
@@ -121,7 +124,7 @@ Your confidence level in the extracted data:
## Important: Handling Collection Repos (multiple model files)
Many HuggingFace repos contain **multiple model files** in a single repository
Many model repositories contain **multiple model files** in a single repository
(e.g. a "LoRA collection" with different styles/characters in separate files).
The model file currently being enriched is: **`{{model_basename}}`**
py/services/agent/skills/enrich_hf_metadata/readme_processor.py
+39 -15
View File
@@ -1,8 +1,15 @@
"""HF README processing for the ``enrich_hf_metadata`` skill.
"""Model card (README) processing for the ``enrich_hf_metadata`` skill.
Provides README cleaning for LLM injection, gallery/image extraction from
multiple formats (YAML widget, markdown, HTML ``<img>``, gallery tables),
and section-based README trimming for collection repos.
The extractors default to Hugging Face asset URLs, but every one of them
accepts an explicit ``base_url`` so the same parsing works for any model
source (ModelScope, ...). See :mod:`py.services.model_sources`.
This module deliberately has no package-relative imports: it is also loaded
standalone by the README-processing test harness.
"""
from __future__ import annotations
@@ -15,12 +22,25 @@ from typing import Any, List, Tuple
_REPO_URL_PATTERN = re.compile(r"https?://huggingface\.co/([^/]+/[^/]+)")
def resolve_asset_base_url(repo: str, base_url: str | None = None) -> str:
"""Return the base URL used to resolve repository-relative assets.
Falls back to the historical Hugging Face layout when *base_url* is not
supplied, so existing callers keep their behaviour.
"""
if base_url:
return base_url.rstrip("/")
return f"https://huggingface.co/{repo}/resolve/main"
def extract_simple_markdown_images(
markdown_text: str,
repo: str,
existing_urls: set[str] | None = None,
default_width: int = 512,
default_height: int = 512,
base_url: str | None = None,
) -> list[dict[str, Any]]:
"""Extract standalone markdown images from the README body.
@@ -32,10 +52,10 @@ def extract_simple_markdown_images(
Returns a list of dicts in the same ``civitai.images`` format as
:func:`extract_gallery_images`.
"""
if not markdown_text or not repo:
if not markdown_text or not (repo or base_url):
return []
base_url = f"https://huggingface.co/{repo}/resolve/main"
base_url = resolve_asset_base_url(repo, base_url)
images: list[dict[str, Any]] = []
seen_urls: set[str] = set(existing_urls) if existing_urls else set()
@@ -89,20 +109,21 @@ def extract_html_img_tags(
existing_urls: set[str] | None = None,
default_width: int = 512,
default_height: int = 512,
base_url: str | None = None,
) -> list[dict[str, Any]]:
"""Extract image URLs from HTML ``<img src=\"...\">`` tags in the README.
Many HF collection repos (e.g. ``deadman44/Z-Image_LoRA``) use raw HTML
``<img>`` tags exclusively for their sample images, with no markdown
``![]()`` equivalents. This function finds those tags and constructs
resolvable HF URLs.
resolvable URLs.
Returns a list of dicts in the ``civitai.images`` format.
"""
if not markdown_text or not repo:
if not markdown_text or not (repo or base_url):
return []
base_url = f"https://huggingface.co/{repo}/resolve/main"
base_url = resolve_asset_base_url(repo, base_url)
images: list[dict[str, Any]] = []
seen_urls: set[str] = set(existing_urls) if existing_urls else set()
@@ -166,7 +187,7 @@ def extract_html_img_tags(
def extract_repo_from_hf_url(hf_url: str) -> str:
"""Extract ``user/repo`` from a HuggingFace URL."""
m = _REPO_URL_PATTERN.match(hf_url)
m = _REPO_URL_PATTERN.match(hf_url or "")
return m.group(1) if m else ""
@@ -175,21 +196,23 @@ def extract_gallery_images(
repo: str,
default_width: int = 512,
default_height: int = 512,
base_url: str | None = None,
) -> List[dict[str, Any]]:
"""Extract widget/gallery images from the YAML frontmatter of a HF README.
"""Extract widget/gallery images from the YAML frontmatter of a README.
Args:
markdown_text: Raw README content.
repo: HF repo identifier (``user/repo``).
repo: Repository identifier (``user/repo``).
default_width: Fallback width when the README provides no dimension.
default_height: Fallback height when the README provides no dimension.
base_url: Overrides the asset base URL (defaults to Hugging Face).
Returns a list of dicts compatible with the ``civitai.images`` metadata
format, each containing ``url`` (absolute HF URL), ``meta.prompt``,
format, each containing ``url`` (absolute), ``meta.prompt``,
``width``, ``height``, and ``type``. Returns an empty list when no
widget entries are found or when *repo* is empty.
"""
if not markdown_text or not repo:
if not markdown_text or not (repo or base_url):
return []
frontmatter = _extract_frontmatter(markdown_text)
@@ -197,7 +220,7 @@ def extract_gallery_images(
return []
images: List[dict[str, Any]] = []
base_url = f"https://huggingface.co/{repo}/resolve/main"
base_url = resolve_asset_base_url(repo, base_url)
w = default_width or 512
h = default_height or 512
@@ -279,10 +302,11 @@ def extract_gallery_table_images(
existing_urls: set[str] | None = None,
default_width: int = 512,
default_height: int = 512,
base_url: str | None = None,
) -> list[dict[str, Any]]:
"""Extract images from ``| Preview | Prompt |`` markdown gallery tables.
Many HF READMEs include a sample-gallery table in the body (outside
Many READMEs include a sample-gallery table in the body (outside
the YAML frontmatter) that shows generation examples with their
prompts. This function parses those tables and merges results with
the widget-sourced images from :func:`extract_gallery_images`.
@@ -291,10 +315,10 @@ def extract_gallery_table_images(
:func:`extract_gallery_images`. Already-seen URLs (from *existing_urls*)
are skipped.
"""
if not markdown_text or not repo:
if not markdown_text or not (repo or base_url):
return []
base_url = f"https://huggingface.co/{repo}/resolve/main"
base_url = resolve_asset_base_url(repo, base_url)
images: list[dict[str, Any]] = []
seen_urls: set[str] = set(existing_urls) if existing_urls else set()
lines = markdown_text.split("\n")
py/services/base_model_service.py
+16 -12
View File
@@ -21,6 +21,7 @@ from .model_query import (
resolve_sub_type,
)
from .settings_manager import get_settings_manager
from .model_sources import source_group_key
from ..utils.civitai_utils import build_civitai_model_page_url
logger = logging.getLogger(__name__)
@@ -742,29 +743,32 @@ class BaseModelService(ABC):
@staticmethod
def _extract_hf_group_key(item: Dict[str, Any]) -> Optional[str]:
"""Extract `hf:{owner}/{repo}` from item's ``hf_url``, or None."""
hf_url = item.get("hf_url") if isinstance(item, dict) else None
if not hf_url or not isinstance(hf_url, str):
return None
m = re.match(
r"https?://huggingface\.co/([^/]+/[^/]+)", hf_url.strip()
)
if not m:
return None
return f"hf:{m.group(1)}"
key = BaseModelService._extract_source_group_key(item)
return key if key and key.startswith("hf:") else None
@staticmethod
def _extract_source_group_key(item: Dict[str, Any]) -> Optional[str]:
"""Return the external-source group key for *item*, or None.
Hugging Face keeps the historical ``hf:{owner}/{repo}`` shape; other
platforms use their own short prefix (``ms:`` / ``ta:``).
"""
return source_group_key(item)
@staticmethod
def _extract_group_key(item: Dict[str, Any]) -> Union[int, str, None]:
"""Return the group identity key: CivitAI modelId (int) or HF repo (str).
"""Return the group identity key.
Preference order:
1. CivitAI ``modelId`` (int)
2. HF repo identity ``hf:{owner}/{repo}`` (str)
2. External model source identity, e.g. ``hf:{owner}/{repo}``,
``ms:{owner}/{repo}``, ``ta:{model_id}`` (str)
3. ``None`` (no known grouping source)
"""
mid = BaseModelService._extract_model_id(item)
if mid is not None:
return mid
return BaseModelService._extract_hf_group_key(item)
return BaseModelService._extract_source_group_key(item)
@staticmethod
def _extract_model_id(item: Dict[str, Any]) -> Optional[int]:
py/services/checkpoint_service.py
+2
View File
@@ -67,6 +67,8 @@ class CheckpointService(BaseModelService):
"civitai": self.filter_civitai_data(model_data.get("civitai", {}), minimal=True),
"auto_tags": model_data.get("auto_tags") or extract_auto_tags(model_data),
"version_count": model_data.get("version_count"),
"source_platform": model_data.get("source_platform", ""),
"source_url": model_data.get("source_url", ""),
"hf_url": model_data.get("hf_url", ""),
}
py/services/embedding_service.py
+2
View File
@@ -67,6 +67,8 @@ class EmbeddingService(BaseModelService):
"civitai": self.filter_civitai_data(model_data.get("civitai", {}), minimal=True),
"auto_tags": model_data.get("auto_tags") or extract_auto_tags(model_data),
"version_count": model_data.get("version_count"),
"source_platform": model_data.get("source_platform", ""),
"source_url": model_data.get("source_url", ""),
"hf_url": model_data.get("hf_url", ""),
}
py/services/lora_service.py
+2
View File
@@ -79,6 +79,8 @@ class LoraService(BaseModelService):
),
"auto_tags": model_data.get("auto_tags") or extract_auto_tags(model_data),
"version_count": model_data.get("version_count"),
"source_platform": model_data.get("source_platform", ""),
"source_url": model_data.get("source_url", ""),
"hf_url": model_data.get("hf_url", ""),
}
py/services/metadata_sync_service.py
+4 -2
View File
@@ -14,6 +14,7 @@ from ..utils.model_utils import determine_base_model
from ..utils.models import autov3_from_civitai_files
from .connectivity_guard import OFFLINE_FRIENDLY_MESSAGE, is_expected_offline_error
from .errors import RateLimitError
from .model_sources import has_external_source
logger = logging.getLogger(__name__)
@@ -222,9 +223,10 @@ class MetadataSyncService:
error_msg = "CivitAI model is deleted and no archive provider is available"
return False, error_msg
else:
is_hf_source = bool(model_data.get("hf_url"))
is_hf_source = has_external_source(model_data)
if is_hf_source:
# HF-sourced model: only check CivitAI API directly.
# External-source model (Hugging Face / ModelScope /
# TensorArt): only check CivitAI API directly.
# CivArchive is almost guaranteed to have no record, and
# hitting it wastes rate-limit budget.
# Use a distinct provider name ("civitai_api" not None) so
py/services/model_scanner.py
+7
View File
@@ -15,6 +15,7 @@ from ..utils.civitai_utils import resolve_license_info
from .model_cache import ModelCache
from .model_hash_index import ModelHashIndex
from .model_lifecycle_service import delete_model_artifacts, _require_path_in_library_roots
from .model_sources import normalize_metadata_source
from .service_registry import ServiceRegistry
from .websocket_manager import ws_manager
from .persistent_model_cache import get_persistent_cache
@@ -387,8 +388,14 @@ class ModelScanner:
'civitai': civitai_slim,
'civitai_deleted': bool(get_value('civitai_deleted', False)),
'skip_metadata_refresh': bool(get_value('skip_metadata_refresh', False)),
# External model source (Hugging Face / ModelScope / TensorArt).
# `source_url` + `source_platform` are canonical; `hf_url` stays in
# sync as a legacy alias (normalised below).
'source_platform': get_value('source_platform', '') or '',
'source_url': get_value('source_url', '') or '',
'hf_url': get_value('hf_url', '') or '',
}
normalize_metadata_source(entry)
license_source: Dict[str, Any] = {}
if isinstance(civitai_full, Mapping):
py/services/model_sources/__init__.py
+61
View File
@@ -0,0 +1,61 @@
"""External model-source providers (Hugging Face, ModelScope, TensorArt).
This package is the single abstraction over "a site that hosts models and
a model card". See :mod:`py.services.model_sources.base` for the provider
protocol and :mod:`py.services.model_sources.registry` for the lookup and
metadata-normalisation helpers used across the codebase.
"""
from __future__ import annotations
from .base import (
GROUP_PREFIXES,
HTTP_TIMEOUT,
ModelSource,
SourceRef,
USER_AGENT,
clean_source_url,
fetch_text,
)
from .huggingface import HuggingFaceSource
from .modelscope import ModelScopeSource
from .registry import (
LEGACY_HF_URL_FIELD,
SOURCE_PLATFORM_FIELD,
SOURCE_URL_FIELD,
detect_source,
get_source,
get_source_platform,
has_external_source,
list_sources,
normalize_metadata_source,
resolve_source_ref,
source_group_key,
source_label,
)
from .tensorart import TensorArtSource
__all__ = [
"GROUP_PREFIXES",
"HTTP_TIMEOUT",
"LEGACY_HF_URL_FIELD",
"ModelSource",
"HuggingFaceSource",
"ModelScopeSource",
"SOURCE_PLATFORM_FIELD",
"SOURCE_URL_FIELD",
"SourceRef",
"TensorArtSource",
"USER_AGENT",
"clean_source_url",
"detect_source",
"fetch_text",
"get_source",
"get_source_platform",
"has_external_source",
"list_sources",
"normalize_metadata_source",
"resolve_source_ref",
"source_group_key",
"source_label",
]
py/services/model_sources/base.py
+186
View File
@@ -0,0 +1,186 @@
"""Base types for the external model-source provider abstraction.
A *model source* is a third-party site that hosts model files and a model
card (README) describing them — Hugging Face, ModelScope, TensorArt, and
whatever gets added later. Everything the rest of the codebase needs to
know about such a site is expressed by :class:`ModelSource`:
* how to recognise one of its URLs (:meth:`ModelSource.parse`)
* the canonical page URL for a source id (:meth:`ModelSource.canonical_url`)
* how to fetch the model card (:meth:`ModelSource.fetch_model_card`)
* how to turn repository-relative asset paths into absolute URLs
(:meth:`ModelSource.asset_base_url`)
* which capabilities the site actually supports
(``supports_enrichment`` / ``supports_download``)
Keeping this in one place means the agent pipeline, the scanners, and the
HTTP handlers never need site-specific branching.
"""
from __future__ import annotations
import logging
import re
from dataclasses import dataclass
from typing import Any, Optional
import aiohttp
logger = logging.getLogger(__name__)
#: Shared HTTP timeout for model-card fetches.
HTTP_TIMEOUT = 30
#: User agent used for all model-source HTTP requests.
USER_AGENT = "ComfyUI-LoRA-Manager/1.0"
#: Platform → short prefix used when building version-group keys.
#: ``huggingface`` keeps the historical ``hf:`` prefix for backward
#: compatibility with already-cached group keys.
GROUP_PREFIXES: dict[str, str] = {
"huggingface": "hf",
"modelscope": "ms",
"tensorart": "ta",
}
@dataclass(frozen=True)
class SourceRef:
"""A parsed reference to a model hosted on an external site."""
platform: str
"""Canonical platform id, e.g. ``"huggingface"``."""
source_id: str
"""Site-specific identity, e.g. ``"user/repo"`` or ``"827823520299086029"``."""
url: str
"""Canonical URL of the model page."""
async def fetch_text(url: str, *, timeout: int = HTTP_TIMEOUT) -> str:
"""Fetch *url* and return its body as text, or ``""`` on any failure.
Network problems are expected (offline installs, rate limits, dead
repos) and must never bubble up into the pipeline, so every error is
logged at debug level and normalised to an empty string.
"""
try:
async with aiohttp.ClientSession(
headers={"User-Agent": USER_AGENT},
timeout=aiohttp.ClientTimeout(total=timeout),
) as session:
async with session.get(url) as resp:
if resp.status == 200:
return await resp.text()
logger.debug("Fetch %s returned HTTP %s", url, resp.status)
except Exception as exc: # pragma: no cover - network dependent
logger.debug("Failed to fetch %s: %s", url, exc)
return ""
class ModelSource:
"""Description and I/O for one external model hosting site."""
#: Canonical platform id stored in metadata.
platform: str = ""
#: Human-readable name used in UI copy and prompts.
label: str = ""
#: Whether the agent skill can fetch a model card and run AI extraction.
supports_enrichment: bool = False
#: Whether models can be downloaded directly from this site.
supports_download: bool = False
#: Lenient pattern used to recognise URLs already stored in metadata.
#: Captures the site-specific source id in group ``id``.
url_pattern: re.Pattern[str] | None = None
#: Strict pattern used to validate user input. Must match the whole URL.
strict_url_pattern: re.Pattern[str] | None = None
# ------------------------------------------------------------------
# Parsing
# ------------------------------------------------------------------
def parse(self, url: str, *, strict: bool = False) -> Optional[str]:
"""Return the source id contained in *url*, or ``None``.
With ``strict=True`` the URL must match this site's canonical shape
exactly (used when validating what a user pasted); with
``strict=False`` sub-paths such as ``/resolve/main/file.bin`` are
tolerated (used when normalising already-stored values).
"""
if not url or not isinstance(url, str):
return None
candidate = url.strip()
if not candidate:
return None
pattern = self.strict_url_pattern if strict else self.url_pattern
if pattern is None:
return None
match = pattern.match(candidate)
return match.group("id") if match else None
def ref(self, url: str, *, strict: bool = False) -> Optional[SourceRef]:
"""Return a :class:`SourceRef` for *url*, or ``None`` if not ours."""
source_id = self.parse(url, strict=strict)
if not source_id:
return None
return SourceRef(
platform=self.platform,
source_id=source_id,
url=self.canonical_url(source_id),
)
# ------------------------------------------------------------------
# URLs and content
# ------------------------------------------------------------------
def canonical_url(self, source_id: str) -> str:
"""Return the canonical model-page URL for *source_id*."""
raise NotImplementedError
def asset_base_url(self, source_id: str, revision: str = "") -> str:
"""Base URL used to resolve repository-relative asset paths."""
return ""
def group_key(self, source_id: str) -> str:
"""Return the version-group key for *source_id*."""
prefix = GROUP_PREFIXES.get(self.platform, self.platform)
return f"{prefix}:{source_id}"
async def fetch_model_card(self, source_id: str) -> str:
"""Fetch the raw model card (README) markdown for *source_id*."""
return ""
def __repr__(self) -> str: # pragma: no cover - debugging aid
return f"<ModelSource {self.platform}>"
def clean_source_url(url: Any) -> str:
"""Normalise a stored source URL value into a stripped string."""
if not isinstance(url, str):
return ""
return url.strip()
__all__ = [
"GROUP_PREFIXES",
"HTTP_TIMEOUT",
"ModelSource",
"SourceRef",
"USER_AGENT",
"clean_source_url",
"fetch_text",
]
py/services/model_sources/huggingface.py
+49
View File
@@ -0,0 +1,49 @@
"""Hugging Face model source."""
from __future__ import annotations
import re
from .base import ModelSource, fetch_text
#: Lenient — used to normalise URLs already stored in metadata; tolerates
#: sub-paths such as ``/resolve/main/model.safetensors``.
_URL_PATTERN = re.compile(
r"https?://(?:www\.)?huggingface\.co/(?P<id>[^/?#\s]+/[^/?#\s]+)"
)
#: Strict — validates what the user pasted into the "link model" dialog.
_STRICT_URL_PATTERN = re.compile(
r"https?://(?:www\.)?huggingface\.co/(?P<id>[^/?#\s]+/[^/?#\s]+)/?$"
)
class HuggingFaceSource(ModelSource):
"""Hugging Face Hub (``huggingface.co``)."""
platform = "huggingface"
label = "Hugging Face"
supports_enrichment = True
supports_download = True
url_pattern = _URL_PATTERN
strict_url_pattern = _STRICT_URL_PATTERN
def canonical_url(self, source_id: str) -> str:
return f"https://huggingface.co/{source_id}"
def asset_base_url(self, source_id: str, revision: str = "") -> str:
return f"https://huggingface.co/{source_id}/resolve/{revision or 'main'}"
async def fetch_model_card(self, source_id: str) -> str:
"""Fetch ``README.md`` from Hugging Face (tries ``main``, then ``master``)."""
for branch in ("main", "master"):
text = await fetch_text(
f"https://huggingface.co/{source_id}/raw/{branch}/README.md"
)
if text:
return text
return ""
__all__ = ["HuggingFaceSource"]
py/services/model_sources/modelscope.py
+76
View File
@@ -0,0 +1,76 @@
"""ModelScope (魔搭社区) model source.
ModelScope exposes the same "model card as README.md" convention as
Hugging Face, including a YAML frontmatter block that often carries
``base_model:`` and ``trigger_words:``. Two public endpoints are used,
neither of which requires an API key for public models:
* ``/models/{owner}/{name}/resolve/{revision}/README.md`` — raw model card
* ``/api/v1/models/{owner}/{name}/repo?Revision=..&FilePath=README.md`` —
the same content through the API, used as a fallback when the resolve
URL is unavailable.
"""
from __future__ import annotations
import re
from .base import ModelSource, fetch_text
_URL_PATTERN = re.compile(
r"https?://(?:www\.)?modelscope\.(?:cn|com)/models/(?P<id>[^/?#\s]+/[^/?#\s]+)"
)
#: Trailing view segments the site appends to a model URL; accepted verbatim
#: when the user pastes a browser tab URL.
_VIEW_SEGMENTS = r"(?:summary|files|model-file|readme|community|evaluation)?"
_STRICT_URL_PATTERN = re.compile(
r"https?://(?:www\.)?modelscope\.(?:cn|com)/models/(?P<id>[^/?#\s]+/[^/?#\s]+)"
rf"/?{_VIEW_SEGMENTS}/?$"
)
#: ``master`` is ModelScope's default branch; ``main`` is tried as a fallback
#: for repos imported from Hugging Face.
_REVISIONS = ("master", "main")
class ModelScopeSource(ModelSource):
"""ModelScope (``modelscope.cn``)."""
platform = "modelscope"
label = "ModelScope"
supports_enrichment = True
supports_download = False
url_pattern = _URL_PATTERN
strict_url_pattern = _STRICT_URL_PATTERN
def canonical_url(self, source_id: str) -> str:
return f"https://modelscope.cn/models/{source_id}"
def asset_base_url(self, source_id: str, revision: str = "") -> str:
return f"https://modelscope.cn/models/{source_id}/resolve/{revision or 'master'}"
async def fetch_model_card(self, source_id: str) -> str:
"""Fetch the model card, preferring the raw resolve URL."""
for revision in _REVISIONS:
text = await fetch_text(
f"https://modelscope.cn/models/{source_id}/resolve/{revision}/README.md"
)
if text:
return text
# Fallback: the repo API proxies the same file and is reachable in
# environments where the CDN resolve host is blocked.
for revision in _REVISIONS:
text = await fetch_text(
"https://modelscope.cn/api/v1/models/"
f"{source_id}/repo?Revision={revision}&FilePath=README.md"
)
if text:
return text
return ""
__all__ = ["ModelScopeSource"]
py/services/model_sources/registry.py
+208
View File
@@ -0,0 +1,208 @@
"""Registry and metadata helpers for external model sources.
The registry is the single place the rest of the codebase asks "which site
is this URL from?", "what is this model's source?", and "can we enrich it?".
Import from :mod:`py.services.model_sources` rather than this module
directly.
"""
from __future__ import annotations
import logging
from typing import Any, Dict, Mapping, Optional
from .base import GROUP_PREFIXES, ModelSource, SourceRef, clean_source_url
from .huggingface import HuggingFaceSource
from .modelscope import ModelScopeSource
from .tensorart import TensorArtSource
logger = logging.getLogger(__name__)
#: Order matters only for disambiguation; the URL patterns are disjoint.
_SOURCES: tuple[ModelSource, ...] = (
HuggingFaceSource(),
ModelScopeSource(),
TensorArtSource(),
)
_BY_PLATFORM: Dict[str, ModelSource] = {s.platform: s for s in _SOURCES}
#: Metadata keys that carry the canonical external-source identity.
SOURCE_PLATFORM_FIELD = "source_platform"
SOURCE_URL_FIELD = "source_url"
#: Legacy field kept as a read/write alias for Hugging Face models so that
#: older sidecars, cached rows, and third-party consumers keep working.
LEGACY_HF_URL_FIELD = "hf_url"
def list_sources() -> list[ModelSource]:
"""Return every known model source."""
return list(_SOURCES)
def get_source(platform: Optional[str]) -> Optional[ModelSource]:
"""Return the source registered for *platform*, or ``None``."""
if not platform or not isinstance(platform, str):
return None
return _BY_PLATFORM.get(platform.strip().lower())
def source_label(platform: Optional[str], default: str = "") -> str:
"""Return the human-readable label for *platform*."""
source = get_source(platform)
return source.label if source else default
def detect_source(url: Optional[str], *, strict: bool = False) -> Optional[SourceRef]:
"""Return the :class:`SourceRef` for *url*, or ``None`` if unsupported."""
if not url or not isinstance(url, str):
return None
for source in _SOURCES:
ref = source.ref(url, strict=strict)
if ref is not None:
return ref
return None
def resolve_source_ref(metadata: Mapping[str, Any]) -> Optional[SourceRef]:
"""Return the source reference described by a model's metadata.
Handles all three storage states found in the wild:
1. ``source_url`` + ``source_platform`` (current format)
2. ``hf_url`` only (legacy Hugging Face storage)
3. ``hf_url`` plus a newer ``source_url`` (both written by older builds)
"""
if not isinstance(metadata, Mapping):
return None
platform = clean_source_url(metadata.get(SOURCE_PLATFORM_FIELD)).lower()
url = clean_source_url(metadata.get(SOURCE_URL_FIELD))
legacy = clean_source_url(metadata.get(LEGACY_HF_URL_FIELD))
source = get_source(platform)
if url:
if source is not None:
ref = source.ref(url)
if ref is not None:
return ref
ref = detect_source(url)
if ref is not None:
return ref
# Unknown platform but a URL is present: keep it addressable.
return SourceRef(platform=platform or "unknown", source_id="", url=url)
if legacy:
return detect_source(legacy)
return None
def normalize_metadata_source(metadata: Dict[str, Any]) -> Dict[str, Any]:
"""Normalise the external-source fields on *metadata* in place.
Guarantees that ``source_url``/``source_platform`` are present and
consistent, and that ``hf_url`` mirrors ``source_url`` for Hugging Face
models (never for other platforms, so a stale alias can't make a
ModelScope model look like a Hugging Face one).
Returns the same dict for convenient chaining.
"""
if not isinstance(metadata, dict):
return metadata
platform = clean_source_url(metadata.get(SOURCE_PLATFORM_FIELD)).lower()
url = clean_source_url(metadata.get(SOURCE_URL_FIELD))
legacy = clean_source_url(metadata.get(LEGACY_HF_URL_FIELD))
source = get_source(platform)
ref: Optional[SourceRef] = None
if url:
ref = source.ref(url) if source is not None else None
if ref is None:
ref = detect_source(url)
elif legacy:
ref = detect_source(legacy)
if ref is not None and ref.source_id:
platform = ref.platform
url = ref.url or url
if platform:
metadata[SOURCE_PLATFORM_FIELD] = platform
else:
metadata.setdefault(SOURCE_PLATFORM_FIELD, "")
metadata[SOURCE_URL_FIELD] = url
# Keep the legacy alias in sync, but only for Hugging Face.
if url and platform == "huggingface":
metadata[LEGACY_HF_URL_FIELD] = url
elif LEGACY_HF_URL_FIELD in metadata and platform and platform != "huggingface":
metadata[LEGACY_HF_URL_FIELD] = ""
elif legacy and not url:
metadata[LEGACY_HF_URL_FIELD] = legacy
return metadata
def has_external_source(item: Mapping[str, Any]) -> bool:
"""Return ``True`` when *item* is linked to any external model site."""
if not isinstance(item, Mapping):
return False
return bool(
clean_source_url(item.get(SOURCE_URL_FIELD))
or clean_source_url(item.get(LEGACY_HF_URL_FIELD))
)
def get_source_platform(item: Mapping[str, Any]) -> str:
"""Return the platform id stored on *item* (may be empty)."""
if not isinstance(item, Mapping):
return ""
platform = clean_source_url(item.get(SOURCE_PLATFORM_FIELD)).lower()
if platform:
return platform
ref = resolve_source_ref(item)
return ref.platform if ref else ""
def source_group_key(item: Mapping[str, Any]) -> Optional[str]:
"""Return the version-group key for *item*, or ``None``.
Hugging Face keeps the historical ``hf:{owner}/{repo}`` shape; other
platforms use their own short prefix (see :data:`GROUP_PREFIXES`).
"""
ref = resolve_source_ref(item)
if ref is None or not ref.source_id:
return None
source = get_source(ref.platform)
if source is None:
return None
return source.group_key(ref.source_id)
__all__ = [
"GROUP_PREFIXES",
"LEGACY_HF_URL_FIELD",
"SOURCE_PLATFORM_FIELD",
"SOURCE_URL_FIELD",
"detect_source",
"get_source",
"get_source_platform",
"has_external_source",
"list_sources",
"normalize_metadata_source",
"resolve_source_ref",
"source_group_key",
"source_label",
]
py/services/model_sources/tensorart.py
+56
View File
@@ -0,0 +1,56 @@
"""TensorArt model source (link / provenance only).
TensorArt support is intentionally limited to *linking* a model to its
TensorArt page. Automatic metadata extraction is not possible without a
user session:
* ``tensor.art`` sits behind a Cloudflare managed challenge, so plain
HTTP clients (aiohttp, requests, curl) receive ``403 "Just a moment..."``.
* Its internal API (``ap-east-1.tensorart.cloud`` / ``cn.tensorart.net``)
answers every ``/v1/model/*`` route with
``{"code":100002,"message":"invalid authorization header"}``.
* The official TAMS API requires an AccessKey/SecretKey pair and request
signatures, which is a poor fit for a "paste a URL" workflow.
``supports_enrichment`` is therefore ``False``: the agent pipeline skips
these models with an explicit reason instead of failing silently, and the
UI keeps showing the "View on TensorArt" link. ``tusi.cn`` is TensorArt's
Chinese mirror and is accepted as the same platform.
"""
from __future__ import annotations
import re
from .base import ModelSource
_DOMAINS = r"(?:tensor\.art|tusi\.cn)"
_URL_PATTERN = re.compile(
rf"https?://(?:www\.)?{_DOMAINS}/models/(?P<id>\d+)"
)
_STRICT_URL_PATTERN = re.compile(
rf"https?://(?:www\.)?{_DOMAINS}/models/(?P<id>\d+)(?:/[^/?#\s]+)?/?$"
)
class TensorArtSource(ModelSource):
"""TensorArt (``tensor.art``)."""
platform = "tensorart"
label = "TensorArt"
supports_enrichment = False
supports_download = False
url_pattern = _URL_PATTERN
strict_url_pattern = _STRICT_URL_PATTERN
def canonical_url(self, source_id: str) -> str:
return f"https://tensor.art/models/{source_id}"
def asset_base_url(self, source_id: str, revision: str = "") -> str:
# Unreachable today: enrichment is disabled for this platform.
return f"https://tensor.art/models/{source_id}"
__all__ = ["TensorArtSource"]
py/services/other_model_service.py
+2
View File
@@ -67,6 +67,8 @@ class OtherModelService(BaseModelService):
"civitai": self.filter_civitai_data(model_data.get("civitai", {}), minimal=True),
"auto_tags": model_data.get("auto_tags") or extract_auto_tags(model_data),
"version_count": model_data.get("version_count"),
"source_platform": model_data.get("source_platform", ""),
"source_url": model_data.get("source_url", ""),
"hf_url": model_data.get("hf_url", ""),
}
py/services/persistent_model_cache.py
+17
View File
@@ -7,6 +7,7 @@ from dataclasses import dataclass, field
from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple
from ..utils.cache_paths import CacheType, resolve_cache_path_with_migration
from .model_sources import normalize_metadata_source
logger = logging.getLogger(__name__)
@@ -62,6 +63,8 @@ class PersistentModelCache:
"db_checked",
"last_checked_at",
"hash_status",
"source_platform",
"source_url",
"hf_url",
)
_MODEL_UPDATE_COLUMNS: Tuple[str, ...] = _MODEL_COLUMNS[2:]
@@ -206,8 +209,13 @@ class PersistentModelCache:
"skip_metadata_refresh": bool(row["skip_metadata_refresh"]),
"license_flags": int(license_value),
"hash_status": row["hash_status"] or "completed",
"source_platform": row["source_platform"] or "",
"source_url": row["source_url"] or "",
"hf_url": row["hf_url"] or "",
}
# Legacy rows only carry `hf_url`; derive the canonical pair so
# every consumer sees the same shape.
normalize_metadata_source(item)
if row["autov3"] is not None:
item["autov3"] = (row["autov3"] or "").lower()
raw_data.append(item)
@@ -562,6 +570,8 @@ class PersistentModelCache:
db_checked INTEGER,
last_checked_at REAL,
hash_status TEXT,
source_platform TEXT DEFAULT '',
source_url TEXT DEFAULT '',
hf_url TEXT DEFAULT '',
PRIMARY KEY (model_type, file_path)
);
@@ -629,6 +639,8 @@ class PersistentModelCache:
# Persisting without explicit flags should assume CivitAI's documented defaults (0b111001 == 57).
"license_flags": f"INTEGER DEFAULT {DEFAULT_LICENSE_FLAGS}",
"hash_status": "TEXT DEFAULT 'completed'",
"source_platform": "TEXT DEFAULT ''",
"source_url": "TEXT DEFAULT ''",
"hf_url": "TEXT DEFAULT ''",
"autov3": "TEXT",
}
@@ -650,6 +662,9 @@ class PersistentModelCache:
return conn
def _prepare_model_row(self, model_type: str, item: Dict[str, Any]) -> Tuple[Any, ...]:
# Keep `source_*` and the legacy `hf_url` alias consistent no matter
# which caller populated the item.
normalize_metadata_source(item)
civitai = item.get("civitai") or {}
trained_words = civitai.get("trainedWords")
if isinstance(trained_words, str):
@@ -713,6 +728,8 @@ class PersistentModelCache:
1 if item.get("db_checked") else 0,
float(item.get("last_checked_at") or 0.0),
item.get("hash_status", "completed"),
item.get("source_platform") or "",
item.get("source_url") or "",
item.get("hf_url") or "",
)
py/services/use_cases/bulk_metadata_refresh_use_case.py
+6 -4
View File
@@ -7,6 +7,7 @@ import time
from typing import Any, Dict, List, Optional, Protocol, Sequence
from ..metadata_sync_service import MetadataSyncService
from ..model_sources import has_external_source
from ...utils.metadata_manager import MetadataManager
@@ -51,10 +52,11 @@ class BulkMetadataRefreshUseCase:
if not model.get("skip_metadata_refresh", False)
and not self._is_in_skip_path(model.get("folder", ""), skip_paths)
and (not model.get("civitai") or not model["civitai"].get("id"))
# Skip models downloaded from Hugging Face — they are not on
# CivitAI / CivArchive. Users can still refresh them individually
# via the right-click context menu.
and not model.get("hf_url", "")
# Skip models linked to an external model site (Hugging Face /
# ModelScope / TensorArt) — they are not on CivitAI / CivArchive.
# Users can still refresh them individually via the right-click
# context menu.
and not has_external_source(model)
and not (
# Skip models confirmed not on CivitAI when no need to retry
model.get("from_civitai") is False
static/js/components/ContextMenu/ModelContextMenuMixin.js
+78 -16
View File
@@ -7,6 +7,8 @@ import { MODEL_CONFIG } from '../../api/apiConfig.js';
import { translate } from '../../utils/i18nHelpers.js';
import { getNsfwLevelSelector } from '../shared/NsfwLevelSelector.js';
import { classifyModelRelinkUrl } from '../../utils/civitaiUtils.js';
import { parseModelSourceUrl, getModelSourceInfo } from '../../utils/modelSourceHelpers.js';
import { escapeHtml } from '../shared/utils.js';
// Mixin with shared functionality for LoraContextMenu and CheckpointContextMenu
export const ModelContextMenuMixin = {
@@ -211,7 +213,7 @@ export const ModelContextMenuMixin = {
setTimeout(() => urlInput.focus(), 50);
},
// HuggingFace linking methods
// External model source linking (Hugging Face / ModelScope / TensorArt)
showLinkHfModal() {
const filePath = this.currentCard.dataset.filepath;
if (!filePath) return;
@@ -225,15 +227,23 @@ export const ModelContextMenuMixin = {
}
this._boundLinkHfHandler = async () => {
const hfUrl = urlInput.value.trim();
if (!hfUrl) {
errorDiv.textContent = 'Please enter a HuggingFace repository URL.';
const rawUrl = urlInput.value.trim();
if (!rawUrl) {
errorDiv.textContent = translate(
'modals.linkModelSource.urlRequired',
{},
'Please enter a model page URL.'
);
return;
}
const hfPattern = /^https?:\/\/huggingface\.co\/([^/]+\/[^/]+)\/?$/;
if (!hfPattern.test(hfUrl)) {
errorDiv.textContent = 'Invalid URL format. Expected: https://huggingface.co/user/repo';
const sourceInfo = parseModelSourceUrl(rawUrl);
if (!sourceInfo) {
errorDiv.textContent = translate(
'modals.linkModelSource.invalidUrl',
{},
'Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.'
);
return;
}
@@ -241,12 +251,14 @@ export const ModelContextMenuMixin = {
modalManager.closeModal('linkHfModal');
try {
state.loadingManager.showSimpleLoading('Linking to HuggingFace...');
state.loadingManager.showSimpleLoading(
translate('modals.linkModelSource.linking', {}, 'Linking model source...')
);
const response = await fetch('/api/lm/set-hf-url', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ file_path: filePath, hf_url: hfUrl }),
body: JSON.stringify({ file_path: filePath, source_url: sourceInfo.url }),
});
if (!response.ok) {
@@ -262,7 +274,7 @@ export const ModelContextMenuMixin = {
throw new Error(data.error || 'Failed to link model');
}
} catch (error) {
console.error('Error linking model to HuggingFace:', error);
console.error('Error linking model source:', error);
showToast('toast.contextMenu.linkHfFailed', { message: error.message }, 'error');
} finally {
state.loadingManager.hide();
@@ -276,18 +288,68 @@ export const ModelContextMenuMixin = {
modalManager.showModal('linkHfModal');
this._renderSupportedSources();
setTimeout(() => urlInput.focus(), 50);
},
// HF metadata enrichment (AI agent) methods
/**
* Refresh the supported-site hints from the server so the dialog reflects
* whatever sources this backend build actually knows about. Falls back to
* the static markup in the template when the request fails.
*/
async _renderSupportedSources() {
const container = document.getElementById('hfSupportedSources');
if (!container) return;
try {
const response = await fetch('/api/lm/model-sources');
if (!response.ok) return;
const sources = await response.json();
if (!Array.isArray(sources) || sources.length === 0) return;
const examples = sources
.map((source) => source?.example_url)
.filter((url) => typeof url === 'string' && url);
if (examples.length === 0) return;
container.innerHTML = examples
.map((url) => `<strong>${escapeHtml(url)}</strong>`)
.join('<br>');
} catch (error) {
console.debug('Failed to load supported model sources:', error);
}
},
// Model metadata enrichment (AI agent) methods
updateEnrichMenuItem(card) {
const enrichItem = this.menu?.querySelector('[data-action="enrich-hf-llm"]');
if (!enrichItem) return;
const hasHfUrl = !!card.dataset.hf_url;
enrichItem.classList.toggle('disabled', !hasHfUrl);
enrichItem.title = hasHfUrl
? ''
: 'Link this model to a HuggingFace repo first (Link Model → Link to HuggingFace)';
const model = {
source_url: card.dataset.source_url || '',
source_platform: card.dataset.source_platform || '',
hf_url: card.dataset.hf_url || '',
};
const sourceInfo = getModelSourceInfo(model);
const canEnrich = Boolean(sourceInfo && sourceInfo.supportsEnrichment);
enrichItem.classList.toggle('disabled', !canEnrich);
if (canEnrich) {
enrichItem.title = '';
} else if (!sourceInfo) {
enrichItem.title = translate(
'toast.contextMenu.enrichNeedsSource',
{},
'Link this model to a model source first (Link Model → Link to Model Source)'
);
} else {
enrichItem.title = translate(
'toast.contextMenu.enrichUnsupportedSource',
{ source: sourceInfo.label },
`AI enrichment is not available for ${sourceInfo.label} models`
);
}
},
async enrichWithAgent(filePath) {
static/js/components/shared/ModelCard.js
+25 -12
View File
@@ -1,4 +1,5 @@
import { showToast, openCivitai, openHuggingFace, copyToClipboard, copyLoraSyntax, sendLoraToWorkflow, sendEmbeddingToWorkflow, openExampleImagesFolder, buildLoraSyntax, sendModelPathToWorkflow } from '../../utils/uiHelpers.js';
import { getModelSourceInfo, getModelSourceGroupKey, getModelSourceViewTitle, openModelSource } from '../../utils/modelSourceHelpers.js';
import { state, getCurrentPageState } from '../../state/index.js';
import { showModelModal } from './ModelModal.js';
import { hasCivitaiSource } from './utils.js';
@@ -65,12 +66,15 @@ function handleModelCardEvent_internal(event, modelType) {
if (event.target.closest('.fa-globe')) {
event.stopPropagation();
// CivitAI wins when the model actually has CivitAI data; otherwise fall
// back to HuggingFace. Relying on `from_civitai` here made the two
// sources mutually exclusive whenever one of them was (re)linked (#1094).
// back to the linked external source. Relying on `from_civitai` here
// made the two sources mutually exclusive whenever one of them was
// (re)linked (#1094).
if (card.dataset.has_civitai === 'true') {
openCivitai(card.dataset.filepath);
} else if (card.dataset.hf_url) {
} else if (card.dataset.source_platform === 'huggingface' && card.dataset.hf_url) {
openHuggingFace(card.dataset.hf_url);
} else if (card.dataset.source_url) {
openModelSource(card.dataset.source_url);
}
return true; // Stop propagation
}
@@ -337,6 +341,8 @@ async function showModelModalFromCard(card, modelType) {
modified: card.dataset.modified,
file_size: parseInt(card.dataset.file_size || '0'),
from_civitai: card.dataset.from_civitai === 'true',
source_platform: card.dataset.source_platform || '',
source_url: card.dataset.source_url || '',
hf_url: card.dataset.hf_url || '',
base_model: card.dataset.base_model,
notes: card.dataset.notes || '',
@@ -428,6 +434,8 @@ function showExampleAccessModal(card, modelType) {
modified: card.dataset.modified,
file_size: card.dataset.file_size,
from_civitai: card.dataset.from_civitai === 'true',
source_platform: card.dataset.source_platform || '',
source_url: card.dataset.source_url || '',
hf_url: card.dataset.hf_url || '',
base_model: card.dataset.base_model,
notes: card.dataset.notes,
@@ -490,7 +498,11 @@ export function createModelCard(model, modelType) {
card.dataset.base_model = model.base_model || 'Unknown';
card.dataset.favorite = model.favorite ? 'true' : 'false';
card.dataset.exclude = model.exclude ? 'true' : 'false';
card.dataset.hf_url = model.hf_url || '';
const modelSourceInfo = getModelSourceInfo(model);
card.dataset.source_url = modelSourceInfo?.url || '';
card.dataset.source_platform = modelSourceInfo?.platform || '';
// Legacy alias: only Hugging Face models expose `hf_url`.
card.dataset.hf_url = modelSourceInfo?.platform === 'huggingface' ? modelSourceInfo.url : '';
const hasUpdateAvailable = Boolean(model.update_available);
card.dataset.update_available = hasUpdateAvailable ? 'true' : 'false';
card.dataset.skip_metadata_refresh = model.skip_metadata_refresh ? 'true' : 'false';
@@ -508,11 +520,12 @@ export function createModelCard(model, modelType) {
const modelId = civitaiData?.modelId ?? civitaiData?.model_id;
if (modelId !== undefined && modelId !== null && modelId !== '') {
card.dataset.modelId = modelId;
} else if (model.hf_url) {
// For HF-only models, derive a group key from hf_url for version grouping
const match = model.hf_url.match(/https?:\/\/huggingface\.co\/([^/]+\/[^/]+)/);
if (match) {
card.dataset.modelId = 'hf:' + match[1];
} else {
// For externally-sourced models, derive a group key from the source
// URL for version grouping (hf:user/repo, ms:user/repo, ta:<id>).
const sourceGroupKey = getModelSourceGroupKey(model);
if (sourceGroupKey) {
card.dataset.modelId = sourceGroupKey;
}
}
@@ -610,10 +623,10 @@ export function createModelCard(model, modelType) {
const hasCivitai = hasCivitaiSource(model.civitai);
const globeTitle = hasCivitai ?
translate('modelCard.actions.viewOnCivitai', {}, 'View on Civitai') :
model.hf_url ?
translate('modelCard.actions.viewOnHuggingFace', {}, 'View on Hugging Face') :
modelSourceInfo ?
getModelSourceViewTitle(modelSourceInfo) :
translate('modelCard.actions.notAvailableFromCivitai', {}, 'Not available from Civitai');
const globeEnabled = hasCivitai || !!model.hf_url;
const globeEnabled = hasCivitai || !!modelSourceInfo;
let sendTitle;
let copyTitle;
if (modelType === MODEL_TYPES.LORA) {
static/js/components/shared/ModelModal.js
+18 -9
View File
@@ -1,4 +1,5 @@
import { showToast, openCivitai, sendLoraToWorkflow, sendEmbeddingToWorkflow, sendModelPathToWorkflow, buildLoraSyntax, copyToClipboard } from '../../utils/uiHelpers.js';
import { getModelSourceInfo, getModelSourceGroupKey, getModelSourceViewTitle, openModelSource } from '../../utils/modelSourceHelpers.js';
import { modalManager } from '../../managers/ModalManager.js';
import { MODEL_TYPES } from '../../api/apiConfig.js';
import {
@@ -397,10 +398,13 @@ export async function showModelModal(model, modelType) {
<div class="civitai-view" title="${translate('modals.model.actions.viewOnCivitai', {}, 'View on Civitai')}" data-action="view-civitai" data-filepath="${escapedFilePathAttr}">
<i class="fas fa-globe"></i> ${translate('modals.model.actions.viewOnCivitaiText', {}, 'View on Civitai')}
</div>`.trim() : '';
const escapedHfUrl = modelWithFullData.hf_url ? escapeAttribute(modelWithFullData.hf_url) : '';
const viewOnHuggingFaceAction = escapedHfUrl ? `
<div class="civitai-view" title="${translate('modals.model.actions.viewOnHuggingFace', {}, 'View on Hugging Face')}" data-action="view-huggingface" data-hf-url="${escapedHfUrl}">
<i class="fas fa-globe"></i> ${translate('modals.model.actions.viewOnHuggingFaceText', {}, 'View on Hugging Face')}
const sourceInfo = getModelSourceInfo(modelWithFullData);
const escapedSourceUrl = sourceInfo?.url ? escapeAttribute(sourceInfo.url) : '';
const isHuggingFaceSource = sourceInfo?.platform === 'huggingface';
const sourceTitle = sourceInfo ? getModelSourceViewTitle(sourceInfo) : '';
const viewOnHuggingFaceAction = escapedSourceUrl ? `
<div class="civitai-view" title="${escapeAttribute(sourceTitle)}" data-action="${isHuggingFaceSource ? 'view-huggingface' : 'view-model-source'}" ${isHuggingFaceSource ? 'data-hf-url' : 'data-source-url'}="${escapedSourceUrl}">
<i class="fas fa-globe"></i> ${escapeHtml(sourceTitle)}
</div>`.trim() : '';
const creatorInfoAction = modelWithFullData.civitai?.creator ? `
<div class="creator-info" data-username="${modelWithFullData.civitai.creator.username}" data-action="view-creator" title="${translate('modals.model.actions.viewCreatorProfile', {}, 'View Creator Profile')}">
@@ -520,12 +524,12 @@ export async function showModelModal(model, modelType) {
const loadingExamplesText = translate('modals.model.loading.examples', {}, 'Loading examples...');
const loadingVersionsText = translate('modals.model.loading.versions', {}, 'Loading versions...');
// Use CivitAI modelId, or derive HF group key for HF-only models
// Use CivitAI modelId, or derive a source group key for externally-linked models
let civitaiModelId = modelWithFullData.civitai?.modelId || '';
if (!civitaiModelId && modelWithFullData.hf_url) {
const match = modelWithFullData.hf_url.match(/https?:\/\/huggingface\.co\/([^/]+\/[^/]+)/);
if (match) {
civitaiModelId = 'hf:' + match[1];
if (!civitaiModelId) {
const sourceGroupKey = getModelSourceGroupKey(modelWithFullData);
if (sourceGroupKey) {
civitaiModelId = sourceGroupKey;
}
}
const civitaiVersionId = modelWithFullData.civitai?.id || '';
@@ -939,6 +943,11 @@ function setupEventHandlers(filePath, modelType) {
window.open(target.dataset.hfUrl, '_blank', 'noopener,noreferrer');
}
break;
case 'view-model-source':
if (target.dataset.sourceUrl) {
openModelSource(target.dataset.sourceUrl);
}
break;
case 'view-creator':
const username = target.dataset.username;
if (username) {
static/js/components/shared/ModelVersionsTab.js
+11 -9
View File
@@ -5,6 +5,7 @@ import { openCivitaiUrl, showToast } from '../../utils/uiHelpers.js';
import { translate } from '../../utils/i18nHelpers.js';
import { state } from '../../state/index.js';
import { buildCivitaiModelUrl } from '../../utils/civitaiUtils.js';
import { parseModelSourceGroupKey } from '../../utils/modelSourceHelpers.js';
import { formatFileSize } from './utils.js';
import { setSessionItem, removeSessionItem } from '../../utils/storageHelpers.js';
@@ -993,22 +994,23 @@ export function initVersionsTab({
renderErrorState(container, translate('modals.model.versions.missingModelId', {}, 'This model is missing a Civitai model id.'));
return;
}
// HF group keys (e.g. "hf:user/repo") are not real CivitAI model IDs —
// skip the remote API call and show a helpful message instead.
const isHfGroupKey = typeof modelId === 'string' && modelId.startsWith('hf:');
if (isHfGroupKey) {
// External source group keys (e.g. "hf:user/repo", "ms:user/repo",
// "ta:8278...") are not real CivitAI model IDs — skip the remote API
// call and show a helpful message instead.
const sourceGroup = parseModelSourceGroupKey(modelId);
if (sourceGroup) {
controller.isLoading = false;
controller.hasLoaded = true;
controller.record = null;
const hfMsg = translate(
'modals.model.versions.hfGroupInfo',
{},
'This is a HuggingFace model group. Open the library to see all versions in the grid.'
const sourceMsg = translate(
'modals.model.versions.sourceGroupInfo',
{ source: sourceGroup.label },
`This is a ${sourceGroup.label} model group. Open the library to see all versions in the grid.`
);
container.innerHTML = `
<div class="versions-empty-state">
<i class="fas fa-info-circle"></i>
<p>${escapeHtml(hfMsg)}</p>
<p>${escapeHtml(sourceMsg)}</p>
</div>
`;
return;
static/js/utils/modelSourceHelpers.js
+173
View File
@@ -0,0 +1,173 @@
/**
* External model source helpers (Hugging Face / ModelScope / TensorArt).
*
* Mirrors `py/services/model_sources/registry.py` so the frontend and the
* backend agree on URL recognition, version-group keys, and which sites
* support AI metadata enrichment.
*
* Models loaded from an older cache may only carry the legacy `hf_url`
* field; every helper here falls back to it, and to the legacy
* `hf:user/repo` group key shape.
*/
import { translate } from './i18nHelpers.js';
export const MODEL_SOURCES = [
{
platform: 'huggingface',
label: 'Hugging Face',
groupPrefix: 'hf',
supportsEnrichment: true,
supportsDownload: true,
exampleUrl: 'https://huggingface.co/user/repo',
placeholder: 'https://huggingface.co/user/repo',
pattern: /^https?:\/\/(?:www\.)?huggingface\.co\/([^/?#\s]+\/[^/?#\s]+)/i,
canonical: (id) => `https://huggingface.co/${id}`,
},
{
platform: 'modelscope',
label: 'ModelScope',
groupPrefix: 'ms',
supportsEnrichment: true,
supportsDownload: false,
exampleUrl: 'https://modelscope.cn/models/user/repo',
placeholder: 'https://modelscope.cn/models/user/repo',
pattern: /^https?:\/\/(?:www\.)?modelscope\.(?:cn|com)\/models\/([^/?#\s]+\/[^/?#\s]+)/i,
canonical: (id) => `https://modelscope.cn/models/${id}`,
},
{
platform: 'tensorart',
label: 'TensorArt',
groupPrefix: 'ta',
supportsEnrichment: false,
supportsDownload: false,
exampleUrl: 'https://tensor.art/models/827823520299086029',
placeholder: 'https://tensor.art/models/827823520299086029',
pattern: /^https?:\/\/(?:www\.)?(?:tensor\.art|tusi\.cn)\/models\/(\d+)/i,
canonical: (id) => `https://tensor.art/models/${id}`,
},
];
/** Return the source descriptor for a platform id, or null. */
export function getModelSource(platform) {
if (!platform || typeof platform !== 'string') return null;
const needle = platform.trim().toLowerCase();
return MODEL_SOURCES.find((source) => source.platform === needle) || null;
}
/**
* Parse any supported model URL.
* @returns {{platform: string, label: string, groupPrefix: string,
* supportsEnrichment: boolean, supportsDownload: boolean,
* sourceId: string, url: string}|null}
*/
export function parseModelSourceUrl(url) {
if (!url || typeof url !== 'string') return null;
const candidate = url.trim();
if (!candidate) return null;
for (const source of MODEL_SOURCES) {
const match = candidate.match(source.pattern);
if (match) {
return {
...source,
sourceId: match[1],
url: source.canonical(match[1]),
};
}
}
return null;
}
/** Return the stored source URL of a model (new field, then legacy). */
export function getModelSourceUrl(model) {
if (!model) return '';
const value = model.source_url || model.hf_url || '';
return typeof value === 'string' ? value.trim() : '';
}
/** Return the stored source platform of a model. */
export function getModelSourcePlatform(model) {
if (!model) return '';
const value = model.source_platform || '';
return typeof value === 'string' ? value.trim().toLowerCase() : '';
}
/**
* Resolve the full source descriptor for a model, tolerating models that
* predate the `source_*` fields.
*/
export function getModelSourceInfo(model) {
if (!model) return null;
const url = getModelSourceUrl(model);
const declared = getModelSource(getModelSourcePlatform(model));
const parsed = parseModelSourceUrl(url);
if (declared) {
return {
...declared,
sourceId: parsed ? parsed.sourceId : '',
url: parsed ? parsed.url : url,
};
}
return parsed;
}
/**
* Version-group key for a model, matching the backend's `_extract_group_key`.
* Returns `''` when the model has no external source.
*/
export function getModelSourceGroupKey(model) {
const info = getModelSourceInfo(model);
if (!info || !info.sourceId) return '';
return `${info.groupPrefix}:${info.sourceId}`;
}
/** Whether AI metadata enrichment can run for this model's source. */
export function canEnrichModelSource(model) {
const info = getModelSourceInfo(model);
return Boolean(info && info.supportsEnrichment);
}
/**
* Parse a version-group key such as `hf:user/repo`, `ms:user/repo`, or
* `ta:827823520299086029` back into its source descriptor.
*
* These keys are NOT CivitAI model ids, so callers must not send them to the
* CivitAI API.
*
* @returns {{platform: string, label: string, sourceId: string}|null}
*/
export function parseModelSourceGroupKey(groupKey) {
if (!groupKey || typeof groupKey !== 'string') return null;
const separator = groupKey.indexOf(':');
if (separator <= 0) return null;
const prefix = groupKey.slice(0, separator);
const source = MODEL_SOURCES.find((candidate) => candidate.groupPrefix === prefix);
if (!source) return null;
return {
platform: source.platform,
label: source.label,
sourceId: groupKey.slice(separator + 1),
};
}
/** Localised "View on X" title for the source globe icon. */
export function getModelSourceViewTitle(info) {
if (!info) return '';
if (info.platform === 'huggingface') {
return translate('modelCard.actions.viewOnHuggingFace', {}, 'View on Hugging Face');
}
return translate(
'modelCard.actions.viewOnSource',
{ source: info.label },
`View on ${info.label}`
);
}
/** Open a model page on its external site in a new tab. */
export function openModelSource(url) {
if (!url) return;
window.open(url, '_blank', 'noopener,noreferrer');
}
templates/checkpoints.html
+1 -1
View File
@@ -21,7 +21,7 @@
<i class="fas fa-external-link-alt"></i> <span>{{ t('loras.contextMenu.linkCivitai') }}</span>
</div>
<div class="context-menu-item" data-action="link-hf">
<i class="fas fa-robot"></i> <span>{{ t('loras.contextMenu.linkHuggingFace') }}</span>
<i class="fas fa-robot"></i> <span>{{ t('loras.contextMenu.linkModelSource') }}</span>
</div>
</div>
</div>
templates/components/context_menu.html
+1 -1
View File
@@ -21,7 +21,7 @@
<i class="fas fa-external-link-alt"></i> <span>{{ t('loras.contextMenu.linkCivitai') }}</span>
</div>
<div class="context-menu-item" data-action="link-hf">
<i class="fas fa-robot"></i> <span>{{ t('loras.contextMenu.linkHuggingFace') }}</span>
<i class="fas fa-robot"></i> <span>{{ t('loras.contextMenu.linkModelSource') }}</span>
</div>
</div>
</div>
templates/components/modals/link_hf_modal.html
+13 -8
View File
@@ -1,24 +1,29 @@
<!-- Link to HuggingFace Modal -->
<!-- Link to Model Source Modal -->
<div id="linkHfModal" class="modal">
<div class="modal-content">
<button class="close" onclick="modalManager.closeModal('linkHfModal')">&times;</button>
<h2>{{ t('modals.linkHuggingFace.title') }}</h2>
<h2>{{ t('modals.linkModelSource.title') }}</h2>
<div class="warning-box">
<i class="fas fa-info-circle"></i>
<p>{{ t('modals.linkHuggingFace.infoText') }}</p>
<p>{{ t('modals.linkModelSource.infoText') }}</p>
</div>
<div class="input-group">
<label for="hfModelUrl">{{ t('modals.linkHuggingFace.urlLabel') }}</label>
<input type="text" id="hfModelUrl" placeholder="{{ t('modals.linkHuggingFace.urlPlaceholder') }}" />
<label for="hfModelUrl">{{ t('modals.linkModelSource.urlLabel') }}</label>
<input type="text" id="hfModelUrl" placeholder="{{ t('modals.linkModelSource.urlPlaceholder') }}" />
<div class="input-error" id="hfModelUrlError"></div>
<div class="input-help">
{{ t('modals.linkHuggingFace.helpText') }}<br>
<strong>https://huggingface.co/user/repo</strong>
{{ t('modals.linkModelSource.helpText') }}
<div id="hfSupportedSources">
<strong>https://huggingface.co/user/repo</strong><br>
<strong>https://modelscope.cn/models/user/repo</strong><br>
<strong>https://tensor.art/models/827823520299086029</strong>
</div>
{{ t('modals.linkModelSource.enrichNote') }}
</div>
</div>
<div class="modal-actions">
<button class="cancel-btn" onclick="modalManager.closeModal('linkHfModal')">{{ t('common.actions.cancel') }}</button>
<button class="confirm-btn" id="confirmLinkHfBtn">{{ t('modals.linkHuggingFace.confirmAction') }}</button>
<button class="confirm-btn" id="confirmLinkHfBtn">{{ t('modals.linkModelSource.confirmAction') }}</button>
</div>
</div>
</div>
templates/embeddings.html
+1 -1
View File
@@ -21,7 +21,7 @@
<i class="fas fa-external-link-alt"></i> <span>{{ t('loras.contextMenu.linkCivitai') }}</span>
</div>
<div class="context-menu-item" data-action="link-hf">
<i class="fas fa-robot"></i> <span>{{ t('loras.contextMenu.linkHuggingFace') }}</span>
<i class="fas fa-robot"></i> <span>{{ t('loras.contextMenu.linkModelSource') }}</span>
</div>
</div>
</div>
templates/other.html
+1 -1
View File
@@ -80,7 +80,7 @@
<i class="fas fa-external-link-alt"></i> <span>{{ t('loras.contextMenu.linkCivitai') }}</span>
</div>
<div class="context-menu-item" data-action="link-hf">
<i class="fas fa-robot"></i> <span>{{ t('loras.contextMenu.linkHuggingFace') }}</span>
<i class="fas fa-robot"></i> <span>{{ t('loras.contextMenu.linkModelSource') }}</span>
</div>
</div>
</div>
tests/frontend/components/modelCard.sourceGlobe.test.js
+56
View File
@@ -191,4 +191,60 @@ describe('ModelCard source globe (#1094)', () => {
expect(openHuggingFace).toHaveBeenCalledWith('https://huggingface.co/user/repo');
expect(openCivitai).not.toHaveBeenCalled();
});
it('points the globe at ModelScope for a ModelScope-linked model', () => {
const card = mountCard(
createModelCard,
makeModel({
from_civitai: false,
civitai: {},
source_platform: 'modelscope',
source_url: 'https://modelscope.cn/models/user/repo',
})
);
expect(card.dataset.has_civitai).toBe('false');
expect(card.dataset.source_platform).toBe('modelscope');
expect(card.dataset.hf_url).toBe('');
expect(card.querySelector('.fa-globe').getAttribute('title')).toBe('View on ModelScope');
});
it('opens the ModelScope page when the globe is clicked', () => {
const openSpy = vi.spyOn(window, 'open').mockImplementation(() => {});
const card = mountCard(
createModelCard,
makeModel({
from_civitai: false,
civitai: {},
source_platform: 'modelscope',
source_url: 'https://modelscope.cn/models/user/repo',
})
);
setupModelCardEventDelegation('loras');
card.querySelector('.fa-globe').dispatchEvent(new MouseEvent('click', { bubbles: true }));
expect(openSpy).toHaveBeenCalledWith(
'https://modelscope.cn/models/user/repo',
'_blank',
'noopener,noreferrer'
);
expect(openCivitai).not.toHaveBeenCalled();
expect(openHuggingFace).not.toHaveBeenCalled();
openSpy.mockRestore();
});
it('points the globe at TensorArt for a TensorArt-linked model', () => {
const card = mountCard(
createModelCard,
makeModel({
from_civitai: false,
civitai: {},
source_platform: 'tensorart',
source_url: 'https://tensor.art/models/827823520299086029',
})
);
expect(card.querySelector('.fa-globe').getAttribute('title')).toBe('View on TensorArt');
});
});
tests/frontend/components/modelContextMenuMixin.test.js
+113 -1
View File
@@ -1,4 +1,4 @@
import { describe, expect, it } from 'vitest';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { ModelContextMenuMixin } from '../../../static/js/components/ContextMenu/ModelContextMenuMixin.js';
@@ -14,3 +14,115 @@ describe('ModelContextMenuMixin.getModelTypePrefix', () => {
expect(ModelContextMenuMixin.getModelTypePrefix.call({})).toBe('loras');
});
});
describe('ModelContextMenuMixin.updateEnrichMenuItem', () => {
function setupMenu() {
document.body.innerHTML = '<div id="menu"><div data-action="enrich-hf-llm"></div></div>';
return { menu: document.getElementById('menu') };
}
function cardWith(dataset) {
return { dataset };
}
it('enables enrichment for Hugging Face links', () => {
const context = setupMenu();
ModelContextMenuMixin.updateEnrichMenuItem.call(
context,
cardWith({ hf_url: 'https://huggingface.co/user/repo' })
);
const item = context.menu.querySelector('[data-action="enrich-hf-llm"]');
expect(item.classList.contains('disabled')).toBe(false);
expect(item.title).toBe('');
});
it('enables enrichment for ModelScope links', () => {
const context = setupMenu();
ModelContextMenuMixin.updateEnrichMenuItem.call(
context,
cardWith({
source_platform: 'modelscope',
source_url: 'https://modelscope.cn/models/user/repo',
})
);
const item = context.menu.querySelector('[data-action="enrich-hf-llm"]');
expect(item.classList.contains('disabled')).toBe(false);
});
it('disables enrichment for TensorArt and explains why', () => {
const context = setupMenu();
ModelContextMenuMixin.updateEnrichMenuItem.call(
context,
cardWith({
source_platform: 'tensorart',
source_url: 'https://tensor.art/models/827823520299086029',
})
);
const item = context.menu.querySelector('[data-action="enrich-hf-llm"]');
expect(item.classList.contains('disabled')).toBe(true);
expect(item.title).toContain('TensorArt');
});
it('disables enrichment when no source is linked', () => {
const context = setupMenu();
ModelContextMenuMixin.updateEnrichMenuItem.call(context, cardWith({}));
const item = context.menu.querySelector('[data-action="enrich-hf-llm"]');
expect(item.classList.contains('disabled')).toBe(true);
expect(item.title).toContain('Link this model to a model source');
});
});
describe('ModelContextMenuMixin._renderSupportedSources', () => {
const originalFetch = global.fetch;
beforeEach(() => {
document.body.innerHTML = '<div id="hfSupportedSources">static fallback</div>';
});
afterEach(() => {
global.fetch = originalFetch;
});
it('renders the server-provided example URLs', async () => {
global.fetch = vi.fn().mockResolvedValue({
ok: true,
json: async () => [
{ platform: 'huggingface', example_url: 'https://huggingface.co/user/repo' },
{ platform: 'modelscope', example_url: 'https://modelscope.cn/models/user/repo' },
{ platform: 'tensorart', example_url: 'https://tensor.art/models/123' },
],
});
await ModelContextMenuMixin._renderSupportedSources.call({});
const html = document.getElementById('hfSupportedSources').innerHTML;
expect(html).toContain('https://huggingface.co/user/repo');
expect(html).toContain('https://modelscope.cn/models/user/repo');
expect(html).toContain('https://tensor.art/models/123');
});
it('keeps the static fallback when the request fails', async () => {
global.fetch = vi.fn().mockRejectedValue(new Error('offline'));
await ModelContextMenuMixin._renderSupportedSources.call({});
expect(document.getElementById('hfSupportedSources').innerHTML).toBe('static fallback');
});
it('escapes markup from the server payload', async () => {
global.fetch = vi.fn().mockResolvedValue({
ok: true,
json: async () => [{ example_url: '<img src=x onerror=alert(1)>' }],
});
await ModelContextMenuMixin._renderSupportedSources.call({});
const html = document.getElementById('hfSupportedSources').innerHTML;
expect(html).not.toContain('<img');
expect(html).toContain('&lt;img');
});
});
tests/frontend/utils/modelSourceHelpers.test.js
+177
View File
@@ -0,0 +1,177 @@
import { describe, it, expect, vi } from 'vitest';
const { I18N_MODULE } = vi.hoisted(() => ({
I18N_MODULE: new URL('../../../static/js/utils/i18nHelpers.js', import.meta.url).pathname,
}));
vi.mock(I18N_MODULE, () => ({
translate: vi.fn((key, params, fallback) => (typeof fallback === 'string' ? fallback : key)),
}));
const {
MODEL_SOURCES,
parseModelSourceUrl,
getModelSource,
getModelSourceInfo,
getModelSourceUrl,
getModelSourceGroupKey,
canEnrichModelSource,
getModelSourceViewTitle,
parseModelSourceGroupKey,
openModelSource,
} = await import('../../../static/js/utils/modelSourceHelpers.js');
describe('modelSourceHelpers', () => {
it('exposes one descriptor per supported platform', () => {
expect(MODEL_SOURCES.map((s) => s.platform)).toEqual([
'huggingface',
'modelscope',
'tensorart',
]);
});
describe('parseModelSourceUrl', () => {
it('recognises Hugging Face URLs', () => {
const info = parseModelSourceUrl('https://huggingface.co/user/repo');
expect(info.platform).toBe('huggingface');
expect(info.sourceId).toBe('user/repo');
});
it('recognises ModelScope URLs with view sub-paths', () => {
const info = parseModelSourceUrl('https://modelscope.cn/models/user/repo/summary');
expect(info.platform).toBe('modelscope');
expect(info.sourceId).toBe('user/repo');
expect(info.url).toBe('https://modelscope.cn/models/user/repo');
});
it('recognises TensorArt URLs and keeps only the numeric id', () => {
const info = parseModelSourceUrl(
'https://tensor.art/models/827823520299086029/Vivid-Impressions-Storybook-Sstyle-V1.0'
);
expect(info.platform).toBe('tensorart');
expect(info.sourceId).toBe('827823520299086029');
expect(info.url).toBe('https://tensor.art/models/827823520299086029');
});
it('rejects unsupported URLs', () => {
expect(parseModelSourceUrl('https://example.com/x')).toBeNull();
expect(parseModelSourceUrl('')).toBeNull();
expect(parseModelSourceUrl(null)).toBeNull();
});
});
describe('getModelSourceInfo', () => {
it('falls back to the legacy hf_url field', () => {
const info = getModelSourceInfo({ hf_url: 'https://huggingface.co/user/repo' });
expect(info.platform).toBe('huggingface');
expect(info.sourceId).toBe('user/repo');
});
it('prefers the canonical source fields', () => {
const info = getModelSourceInfo({
source_platform: 'modelscope',
source_url: 'https://modelscope.cn/models/user/repo',
hf_url: 'https://huggingface.co/old/repo',
});
expect(info.platform).toBe('modelscope');
});
it('returns null when there is no source', () => {
expect(getModelSourceInfo({})).toBeNull();
expect(getModelSourceInfo({ hf_url: '' })).toBeNull();
});
});
describe('getModelSourceUrl', () => {
it('reads source_url then hf_url', () => {
expect(getModelSourceUrl({ source_url: 'https://a.example/1' })).toBe('https://a.example/1');
expect(getModelSourceUrl({ hf_url: 'https://huggingface.co/u/r' })).toBe(
'https://huggingface.co/u/r'
);
expect(getModelSourceUrl({})).toBe('');
});
});
describe('getModelSourceGroupKey', () => {
it('matches the backend group-key shapes', () => {
expect(getModelSourceGroupKey({ hf_url: 'https://huggingface.co/u/r' })).toBe('hf:u/r');
expect(
getModelSourceGroupKey({ source_url: 'https://modelscope.cn/models/u/r' })
).toBe('ms:u/r');
expect(getModelSourceGroupKey({ source_url: 'https://tensor.art/models/123' })).toBe(
'ta:123'
);
});
it('returns an empty string without a source', () => {
expect(getModelSourceGroupKey({})).toBe('');
});
});
describe('canEnrichModelSource', () => {
it('allows Hugging Face and ModelScope', () => {
expect(canEnrichModelSource({ hf_url: 'https://huggingface.co/u/r' })).toBe(true);
expect(
canEnrichModelSource({ source_url: 'https://modelscope.cn/models/u/r' })
).toBe(true);
});
it('disallows TensorArt and unlinked models', () => {
expect(canEnrichModelSource({ source_url: 'https://tensor.art/models/123' })).toBe(false);
expect(canEnrichModelSource({})).toBe(false);
});
});
describe('getModelSourceViewTitle', () => {
it('uses the branded label for non-HF sources', () => {
expect(getModelSourceViewTitle(getModelSource('modelscope'))).toBe('View on ModelScope');
expect(getModelSourceViewTitle(getModelSource('tensorart'))).toBe('View on TensorArt');
});
it('keeps the historical Hugging Face title', () => {
expect(getModelSourceViewTitle(getModelSource('huggingface'))).toBe(
'View on Hugging Face'
);
});
});
describe('parseModelSourceGroupKey', () => {
it('parses every external group-key prefix', () => {
expect(parseModelSourceGroupKey('hf:user/repo')).toEqual({
platform: 'huggingface',
label: 'Hugging Face',
sourceId: 'user/repo',
});
expect(parseModelSourceGroupKey('ms:user/repo').platform).toBe('modelscope');
expect(parseModelSourceGroupKey('ta:123').platform).toBe('tensorart');
});
it('rejects numeric CivitAI model ids and unknown prefixes', () => {
expect(parseModelSourceGroupKey(222)).toBeNull();
expect(parseModelSourceGroupKey('222')).toBeNull();
expect(parseModelSourceGroupKey('unknown:1')).toBeNull();
expect(parseModelSourceGroupKey('')).toBeNull();
expect(parseModelSourceGroupKey(null)).toBeNull();
});
});
describe('openModelSource', () => {
it('opens the URL in a new tab', () => {
const openSpy = vi.spyOn(window, 'open').mockImplementation(() => {});
openModelSource('https://modelscope.cn/models/u/r');
expect(openSpy).toHaveBeenCalledWith(
'https://modelscope.cn/models/u/r',
'_blank',
'noopener,noreferrer'
);
openSpy.mockRestore();
});
it('does nothing without a URL', () => {
const openSpy = vi.spyOn(window, 'open').mockImplementation(() => {});
openModelSource('');
expect(openSpy).not.toHaveBeenCalled();
openSpy.mockRestore();
});
});
});
tests/routes/test_hf_handlers.py
+152
View File
@@ -154,3 +154,155 @@ async def test_set_hf_url_rejects_non_repo_url(tmp_path, hf_env):
payload = _json_payload(response)
assert payload["success"] is False
hf_env["cache_write"].assert_not_awaited()
# ---------------------------------------------------------------------------
# Multi-source linking (ModelScope / TensorArt)
# ---------------------------------------------------------------------------
async def _write_plain_model(model_path, sha: str = "d" * 64) -> None:
await _write_model(
model_path,
{
"file_name": "model",
"model_name": "model",
"file_path": str(model_path),
"size": 32,
"modified": 1.0,
"sha256": sha,
"base_model": "Unknown",
"preview_url": "",
},
)
@pytest.mark.asyncio
async def test_set_hf_url_accepts_modelscope_and_stores_source_fields(tmp_path, hf_env):
model_path = tmp_path / "ms_model.safetensors"
await _write_plain_model(model_path)
response = await HfHandler().set_hf_url(
FakeRequest(
json_data={
"file_path": str(model_path),
"source_url": "https://modelscope.cn/models/jj3550945163/Krea-2-LORA",
}
)
)
assert response.status == 200
payload = _json_payload(response)
assert payload["source_platform"] == "modelscope"
assert payload["source_url"] == "https://modelscope.cn/models/jj3550945163/Krea-2-LORA"
saved = json.loads(open(_sidecar_path(model_path), encoding="utf-8").read())
assert saved["source_platform"] == "modelscope"
assert saved["source_url"] == "https://modelscope.cn/models/jj3550945163/Krea-2-LORA"
# No stale Hugging Face alias for a ModelScope model.
assert saved.get("hf_url", "") == ""
cached_metadata = hf_env["cache_write"].await_args.args[1]
assert cached_metadata["source_platform"] == "modelscope"
@pytest.mark.asyncio
async def test_set_hf_url_accepts_tensorart_url(tmp_path, hf_env):
model_path = tmp_path / "ta_model.safetensors"
await _write_plain_model(model_path, sha="e" * 64)
response = await HfHandler().set_hf_url(
FakeRequest(
json_data={
"file_path": str(model_path),
"source_url": (
"https://tensor.art/models/827823520299086029/"
"Vivid-Impressions-Storybook-Sstyle-V1.0"
),
}
)
)
assert response.status == 200
payload = _json_payload(response)
assert payload["source_platform"] == "tensorart"
# The canonical page URL is stored, without the slug.
assert payload["source_url"] == "https://tensor.art/models/827823520299086029"
@pytest.mark.asyncio
async def test_set_hf_url_canonicalises_modelscope_subpage(tmp_path, hf_env):
model_path = tmp_path / "ms_sub.safetensors"
await _write_plain_model(model_path, sha="f" * 64)
response = await HfHandler().set_hf_url(
FakeRequest(
json_data={
"file_path": str(model_path),
"source_url": "https://modelscope.cn/models/user/repo/summary",
}
)
)
assert response.status == 200
assert _json_payload(response)["source_url"] == "https://modelscope.cn/models/user/repo"
@pytest.mark.asyncio
async def test_set_hf_url_is_idempotent_for_modelscope(tmp_path, hf_env):
model_path = tmp_path / "ms_twice.safetensors"
await _write_plain_model(model_path, sha="1" * 64)
request = FakeRequest(
json_data={
"file_path": str(model_path),
"source_url": "https://modelscope.cn/models/user/repo",
}
)
await HfHandler().set_hf_url(request)
await HfHandler().set_hf_url(request)
# The second call short-circuits without rewriting the cache entry.
assert hf_env["cache_write"].await_count == 1
@pytest.mark.asyncio
async def test_set_hf_url_switching_source_clears_hf_alias(tmp_path, hf_env):
model_path = tmp_path / "switch.safetensors"
await _write_plain_model(model_path, sha="2" * 64)
await HfHandler().set_hf_url(
FakeRequest(
json_data={
"file_path": str(model_path),
"source_url": "https://huggingface.co/user/repo",
}
)
)
await HfHandler().set_hf_url(
FakeRequest(
json_data={
"file_path": str(model_path),
"source_url": "https://modelscope.cn/models/user/repo",
}
)
)
saved = json.loads(open(_sidecar_path(model_path), encoding="utf-8").read())
assert saved["source_platform"] == "modelscope"
assert saved.get("hf_url", "") == ""
@pytest.mark.asyncio
async def test_get_model_sources_lists_capabilities():
response = await HfHandler().get_model_sources(FakeRequest())
sources = _json_payload(response)
by_platform = {s["platform"]: s for s in sources}
assert set(by_platform) == {"huggingface", "modelscope", "tensorart"}
assert by_platform["huggingface"]["supports_enrichment"] is True
assert by_platform["modelscope"]["supports_enrichment"] is True
# TensorArt is link-only: no accessible model card for the backend.
assert by_platform["tensorart"]["supports_enrichment"] is False
assert by_platform["modelscope"]["supports_download"] is False
assert all(s["example_url"] for s in sources)
tests/services/test_agent_enrichment_source.py
+182
View File
@@ -0,0 +1,182 @@
"""Tests for source-aware AI enrichment orchestration.
Covers the fast-fail gate (:meth:`AgentService._enrichment_skip_reason`) and
the prompt-context builder for non-Hugging Face model sources.
"""
from __future__ import annotations
from unittest import mock
import pytest
from py.services.agent.agent_service import AgentService
class TestEnrichmentSkipReason:
def test_skips_when_no_source_linked(self):
reason = AgentService._enrichment_skip_reason({})
assert "source_url" in reason
def test_allows_huggingface(self):
assert (
AgentService._enrichment_skip_reason(
{"hf_url": "https://huggingface.co/user/repo"}
)
== ""
)
def test_allows_modelscope(self):
assert (
AgentService._enrichment_skip_reason(
{
"source_platform": "modelscope",
"source_url": "https://modelscope.cn/models/user/repo",
}
)
== ""
)
def test_skips_tensorart_with_reason(self):
reason = AgentService._enrichment_skip_reason(
{
"source_platform": "tensorart",
"source_url": "https://tensor.art/models/827823520299086029",
}
)
assert "TensorArt" in reason
assert "not available" in reason
def test_skips_unknown_platform(self):
reason = AgentService._enrichment_skip_reason(
{"source_platform": "somewhere", "source_url": "https://somewhere.example/m/1"}
)
assert "somewhere" in reason
class TestBuildPromptContext:
@pytest.mark.asyncio
async def test_modelscope_card_populates_source_variables(self):
service = AgentService()
readme = "---\nbase_model: krea/Krea-2-Turbo\n---\n# krea\n"
with (
mock.patch(
"py.services.model_sources.modelscope.ModelScopeSource.fetch_model_card",
new=mock.AsyncMock(return_value=readme),
) as mock_fetch,
mock.patch(
"py.metadata_ops.list_base_models",
new=mock.AsyncMock(return_value=["Krea 2 Turbo"]),
),
mock.patch(
"py.metadata_ops.identify_model_type",
new=mock.AsyncMock(return_value="lora"),
),
mock.patch(
"py.services.settings_manager.SettingsManager.get_priority_tag_config",
return_value={"lora": "style, subject"},
),
):
context = await service._build_prompt_context(
skill_name="enrich_hf_metadata",
model_path="/models/loras/krea.safetensors",
metadata={
"source_platform": "modelscope",
"source_url": "https://modelscope.cn/models/jj3550945163/Krea-2-LORA",
"file_name": "krea",
},
registry=mock.Mock(),
llm=mock.Mock(),
)
mock_fetch.assert_awaited_once_with("jj3550945163/Krea-2-LORA")
assert context["source_platform"] == "modelscope"
assert context["source_id"] == "jj3550945163/Krea-2-LORA"
assert context["source_label"] == "ModelScope"
assert (
context["asset_base_url"]
== "https://modelscope.cn/models/jj3550945163/Krea-2-LORA/resolve/master"
)
assert readme in context["readme_content_full"]
# Hugging Face aliases stay empty for a non-HF source.
assert context["hf_url"] == ""
assert context["repo"] == "jj3550945163/Krea-2-LORA"
@pytest.mark.asyncio
async def test_huggingface_keeps_legacy_aliases(self):
service = AgentService()
readme = "# card\n"
with (
mock.patch(
"py.services.model_sources.huggingface.HuggingFaceSource.fetch_model_card",
new=mock.AsyncMock(return_value=readme),
) as mock_fetch,
mock.patch(
"py.metadata_ops.list_base_models",
new=mock.AsyncMock(return_value=[]),
),
mock.patch(
"py.metadata_ops.identify_model_type",
new=mock.AsyncMock(return_value="lora"),
),
mock.patch(
"py.services.settings_manager.SettingsManager.get_priority_tag_config",
return_value={},
),
):
context = await service._build_prompt_context(
skill_name="enrich_hf_metadata",
model_path="/models/loras/thing.safetensors",
metadata={"hf_url": "https://huggingface.co/user/repo"},
registry=mock.Mock(),
llm=mock.Mock(),
)
mock_fetch.assert_awaited_once_with("user/repo")
assert context["source_platform"] == "huggingface"
assert context["hf_url"] == "https://huggingface.co/user/repo"
assert context["repo"] == "user/repo"
@pytest.mark.asyncio
async def test_tensorart_never_fetches_a_card(self):
service = AgentService()
with (
mock.patch(
"py.services.model_sources.huggingface.HuggingFaceSource.fetch_model_card",
new=mock.AsyncMock(),
) as hf_fetch,
mock.patch(
"py.services.model_sources.modelscope.ModelScopeSource.fetch_model_card",
new=mock.AsyncMock(),
) as ms_fetch,
mock.patch(
"py.metadata_ops.list_base_models",
new=mock.AsyncMock(return_value=[]),
),
mock.patch(
"py.metadata_ops.identify_model_type",
new=mock.AsyncMock(return_value="lora"),
),
mock.patch(
"py.services.settings_manager.SettingsManager.get_priority_tag_config",
return_value={},
),
):
context = await service._build_prompt_context(
skill_name="enrich_hf_metadata",
model_path="/models/loras/thing.safetensors",
metadata={
"source_platform": "tensorart",
"source_url": "https://tensor.art/models/827823520299086029",
},
registry=mock.Mock(),
llm=mock.Mock(),
)
hf_fetch.assert_not_awaited()
ms_fetch.assert_not_awaited()
assert context["readme_content"] == ""
assert context["source_platform"] == "tensorart"
tests/services/test_model_scanner.py
+2
View File
@@ -964,6 +964,8 @@ def _make_cache_entry(**overrides) -> Dict[str, Any]:
"civitai": {"id": 111, "modelId": 222, "name": "v1"},
"civitai_deleted": False,
"skip_metadata_refresh": False,
"source_platform": "",
"source_url": "",
"hf_url": "",
"license_flags": 113,
"hash_status": "completed",
tests/services/test_model_sources.py
+324
View File
@@ -0,0 +1,324 @@
"""Tests for the external model-source provider registry.
Covers URL recognition for Hugging Face / ModelScope / TensorArt, the
legacy ``hf_url`` → ``source_url`` normalisation, version-group keys, and
each provider's model-card fetching and capability flags.
"""
from __future__ import annotations
import pytest
from py.services.model_sources import (
HuggingFaceSource,
ModelScopeSource,
TensorArtSource,
detect_source,
get_source,
get_source_platform,
has_external_source,
list_sources,
normalize_metadata_source,
resolve_source_ref,
source_group_key,
source_label,
)
# ---------------------------------------------------------------------------
# URL recognition
# ---------------------------------------------------------------------------
class TestDetectSource:
@pytest.mark.parametrize(
("url", "platform", "source_id"),
[
("https://huggingface.co/user/repo", "huggingface", "user/repo"),
("https://www.huggingface.co/user/repo", "huggingface", "user/repo"),
(
"https://huggingface.co/user/repo/resolve/main/model.safetensors",
"huggingface",
"user/repo",
),
(
"https://modelscope.cn/models/jj3550945163/Krea-2-LORA",
"modelscope",
"jj3550945163/Krea-2-LORA",
),
(
"https://www.modelscope.cn/models/jj3550945163/Krea-2-LORA/summary",
"modelscope",
"jj3550945163/Krea-2-LORA",
),
(
"https://tensor.art/models/827823520299086029/Vivid-Impressions-Storybook-Sstyle-V1.0",
"tensorart",
"827823520299086029",
),
("https://tusi.cn/models/827823520299086029", "tensorart", "827823520299086029"),
],
)
def test_recognises_supported_urls(self, url, platform, source_id):
ref = detect_source(url)
assert ref is not None
assert ref.platform == platform
assert ref.source_id == source_id
@pytest.mark.parametrize(
"url",
[
"",
None,
"not-a-url",
"https://example.com/x",
"https://civitai.com/models/123",
],
)
def test_ignores_unsupported_urls(self, url):
assert detect_source(url) is None
def test_canonical_url_is_stable(self):
assert detect_source("https://huggingface.co/u/r").url == "https://huggingface.co/u/r"
assert (
detect_source("https://modelscope.cn/models/u/r/summary").url
== "https://modelscope.cn/models/u/r"
)
assert (
detect_source("https://tensor.art/models/123/some-slug").url
== "https://tensor.art/models/123"
)
class TestStrictParsing:
@pytest.mark.parametrize(
"url",
[
"https://huggingface.co/user/repo",
"https://huggingface.co/user/repo/",
"https://modelscope.cn/models/user/repo",
"https://modelscope.cn/models/user/repo/summary",
"https://tensor.art/models/827823520299086029",
"https://tensor.art/models/827823520299086029/Vivid-Impressions",
],
)
def test_accepts_user_facing_urls(self, url):
assert detect_source(url, strict=True) is not None
@pytest.mark.parametrize(
"url",
[
"https://huggingface.co/user/repo/resolve/main/model.safetensors",
"https://example.com/x",
"https://tensor.art/models/not-a-number",
],
)
def test_rejects_non_page_urls(self, url):
assert detect_source(url, strict=True) is None
# ---------------------------------------------------------------------------
# Capabilities
# ---------------------------------------------------------------------------
class TestCapabilities:
def test_huggingface_supports_everything(self):
source = get_source("huggingface")
assert source.supports_enrichment is True
assert source.supports_download is True
def test_modelscope_supports_enrichment_but_not_download(self):
source = get_source("modelscope")
assert source.supports_enrichment is True
assert source.supports_download is False
def test_tensorart_is_link_only(self):
source = get_source("tensorart")
assert source.supports_enrichment is False
assert source.supports_download is False
def test_registry_lists_every_source(self):
platforms = {s.platform for s in list_sources()}
assert platforms == {"huggingface", "modelscope", "tensorart"}
def test_labels_are_brand_names(self):
assert source_label("huggingface") == "Hugging Face"
assert source_label("modelscope") == "ModelScope"
assert source_label("tensorart") == "TensorArt"
assert source_label("unknown", "fallback") == "fallback"
# ---------------------------------------------------------------------------
# Metadata normalisation
# ---------------------------------------------------------------------------
class TestNormalizeMetadataSource:
def test_derives_source_fields_from_legacy_hf_url(self):
metadata = {"hf_url": "https://huggingface.co/user/repo"}
normalize_metadata_source(metadata)
assert metadata["source_platform"] == "huggingface"
assert metadata["source_url"] == "https://huggingface.co/user/repo"
assert metadata["hf_url"] == "https://huggingface.co/user/repo"
def test_canonicalises_modelscope_url_and_clears_hf_alias(self):
metadata = {
"source_platform": "modelscope",
"source_url": "https://modelscope.cn/models/user/repo/summary",
"hf_url": "https://huggingface.co/old/repo",
}
normalize_metadata_source(metadata)
assert metadata["source_platform"] == "modelscope"
assert metadata["source_url"] == "https://modelscope.cn/models/user/repo"
# A stale HF alias must not make a ModelScope model look like HF.
assert metadata["hf_url"] == ""
def test_preserves_unknown_url_for_unknown_platform(self):
metadata = {"source_url": "https://example.com/model/1", "source_platform": "other"}
normalize_metadata_source(metadata)
assert metadata["source_url"] == "https://example.com/model/1"
assert metadata["source_platform"] == "other"
def test_empty_metadata_gets_default_fields(self):
metadata: dict = {}
normalize_metadata_source(metadata)
assert metadata["source_platform"] == ""
assert metadata["source_url"] == ""
def test_infers_platform_from_url_when_missing(self):
metadata = {"source_url": "https://modelscope.cn/models/user/repo"}
normalize_metadata_source(metadata)
assert metadata["source_platform"] == "modelscope"
class TestResolveSourceRef:
def test_resolves_from_canonical_fields(self):
ref = resolve_source_ref(
{"source_platform": "modelscope", "source_url": "https://modelscope.cn/models/u/r"}
)
assert ref is not None
assert ref.platform == "modelscope"
assert ref.source_id == "u/r"
def test_resolves_from_legacy_hf_url(self):
ref = resolve_source_ref({"hf_url": "https://huggingface.co/u/r"})
assert ref is not None
assert ref.platform == "huggingface"
def test_returns_none_without_any_source(self):
assert resolve_source_ref({}) is None
assert resolve_source_ref({"hf_url": ""}) is None
class TestHelpers:
def test_has_external_source_accepts_both_field_shapes(self):
assert has_external_source({"hf_url": "https://huggingface.co/u/r"}) is True
assert has_external_source({"source_url": "https://modelscope.cn/models/u/r"}) is True
assert has_external_source({"source_url": ""}) is False
assert has_external_source({}) is False
def test_get_source_platform_infers_from_url(self):
assert get_source_platform({"hf_url": "https://huggingface.co/u/r"}) == "huggingface"
assert get_source_platform({"source_platform": "tensorart"}) == "tensorart"
assert get_source_platform({}) == ""
def test_group_keys_match_legacy_hf_shape(self):
assert source_group_key({"hf_url": "https://huggingface.co/u/r"}) == "hf:u/r"
assert (
source_group_key({"source_url": "https://modelscope.cn/models/u/r"}) == "ms:u/r"
)
assert (
source_group_key({"source_url": "https://tensor.art/models/123"}) == "ta:123"
)
def test_group_key_is_none_without_source(self):
assert source_group_key({}) is None
assert source_group_key({"hf_url": "https://example.com/x"}) is None
# ---------------------------------------------------------------------------
# Model card fetching
# ---------------------------------------------------------------------------
class TestFetchModelCard:
@pytest.mark.asyncio
async def test_huggingface_tries_main_then_master(self, monkeypatch):
calls: list[str] = []
async def fake_fetch_text(url: str, **_kwargs) -> str:
calls.append(url)
if url.endswith("/master/README.md"):
return "# card"
return ""
monkeypatch.setattr("py.services.model_sources.huggingface.fetch_text", fake_fetch_text)
card = await HuggingFaceSource().fetch_model_card("user/repo")
assert card == "# card"
assert calls == [
"https://huggingface.co/user/repo/raw/main/README.md",
"https://huggingface.co/user/repo/raw/master/README.md",
]
@pytest.mark.asyncio
async def test_modelscope_prefers_resolve_url(self, monkeypatch):
calls: list[str] = []
async def fake_fetch_text(url: str, **_kwargs) -> str:
calls.append(url)
return "---\nbase_model: krea/Krea-2-Turbo\n---\n# krea"
monkeypatch.setattr("py.services.model_sources.modelscope.fetch_text", fake_fetch_text)
card = await ModelScopeSource().fetch_model_card("u/r")
assert card.startswith("---")
assert calls == ["https://modelscope.cn/models/u/r/resolve/master/README.md"]
@pytest.mark.asyncio
async def test_modelscope_falls_back_to_repo_api(self, monkeypatch):
calls: list[str] = []
async def fake_fetch_text(url: str, **_kwargs) -> str:
calls.append(url)
if "/api/v1/models/" in url:
return "# from api"
return ""
monkeypatch.setattr("py.services.model_sources.modelscope.fetch_text", fake_fetch_text)
card = await ModelScopeSource().fetch_model_card("u/r")
assert card == "# from api"
assert "resolve/master/README.md" in calls[0]
assert (
"https://modelscope.cn/api/v1/models/u/r/repo?Revision=master&FilePath=README.md"
in calls
)
@pytest.mark.asyncio
async def test_tensorart_never_fetches(self):
# TensorArt enrichment is disabled: the provider must not issue any
# HTTP request, so it deliberately does not import `fetch_text`.
import importlib
module = importlib.import_module("py.services.model_sources.tensorart")
assert not hasattr(module, "fetch_text")
assert await TensorArtSource().fetch_model_card("123") == ""
class TestAssetBaseUrl:
def test_huggingface_uses_main_revision(self):
assert (
HuggingFaceSource().asset_base_url("u/r")
== "https://huggingface.co/u/r/resolve/main"
)
def test_modelscope_uses_master_revision(self):
assert (
ModelScopeSource().asset_base_url("u/r")
== "https://modelscope.cn/models/u/r/resolve/master"
)
tests/services/test_post_processor.py
+55
View File
@@ -292,6 +292,61 @@ Content
)
assert images[0]["meta"]["prompt"] == "a cat"
@pytest.mark.asyncio
async def test_gallery_images_use_modelscope_asset_base_url(self, processor):
"""A ModelScope-linked model resolves relative images against ModelScope."""
readme = """---
widget:
- text: "a cat"
output:
url: images/cat.png
---
Content
"""
with (
mock.patch("py.metadata_ops.apply_metadata_updates") as mock_apply,
mock.patch("py.metadata_ops.download_preview", return_value=None),
mock.patch("py.metadata_ops.refresh_cache"),
):
await processor.process(
skill_name="enrich_hf_metadata",
model_path="/p.safetensors",
llm_output=self.MIN_LLM_OUTPUT,
metadata={
"from_civitai": False,
"source_platform": "modelscope",
"source_url": "https://modelscope.cn/models/user/repo",
},
readme_content=readme,
)
applied = mock_apply.call_args[0][1]
images = applied.get("civitai", {}).get("images", [])
assert len(images) == 1
assert images[0]["url"] == (
"https://modelscope.cn/models/user/repo/resolve/master/images/cat.png"
)
@pytest.mark.asyncio
async def test_base_model_overwrites_existing_modelscope_model(self, processor):
"""ModelScope is an external source, so the LLM may overwrite base_model."""
llm = {**self.MIN_LLM_OUTPUT, "base_model": "Flux.1 D"}
with (
mock.patch("py.metadata_ops.apply_metadata_updates") as mock_apply,
mock.patch("py.metadata_ops.download_preview", return_value=False),
mock.patch("py.metadata_ops.refresh_cache"),
):
await processor.process(
skill_name="enrich_hf_metadata",
model_path="/p.safetensors",
llm_output=llm,
metadata={
"base_model": "SD 1.5",
"source_platform": "modelscope",
"source_url": "https://modelscope.cn/models/user/repo",
},
)
assert mock_apply.call_args[0][1]["base_model"] == "Flux.1 D"
@pytest.mark.asyncio
async def test_gallery_images_skipped_without_hf_url(self, processor):
"""Gallery images NOT extracted when the model has no HF source."""