feat(links): support ModelScope and TensorArt as model sources

A model file could only ever be linked to huggingface.co: `set_hf_url`
validated the URL with a huggingface-only regex, the agent fetched the card
from a hardcoded HF URL, and the readme processor built every relative image
path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the
same model-card convention (README.md + YAML frontmatter, often carrying
`base_model:` and `trigger_words:`) behind a public, key-less API, so the
enrichment pipeline could already serve it - it was the plumbing that was
HF-shaped, not the idea.

Make the external source a first-class, provider-driven concept:

- New `py/services/model_sources/` registry. A `ModelSource` owns URL
  recognition (lenient for stored values, strict for user input), the
  canonical page URL, model-card fetching, the asset base URL and the
  capability flags. `HuggingFaceSource` is the previous logic relocated;
  `ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md`
  and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is
  link-only on purpose: tensor.art answers plain HTTP clients with a
  Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud /
  cn.tensorart.net) rejects every /v1/model/* route with "invalid
  authorization header", so it declares supports_enrichment=False rather than
  failing silently later.
- Metadata gains `source_platform` + `source_url`; `hf_url` stays as a
  read/write alias, written only for Hugging Face, so existing sidecars,
  cached rows and third-party consumers keep working. Normalisation runs at
  the scanner, the persistent cache (both directions, plus two new columns
  behind an ALTER migration) and the linking handler - which is what stops a
  user who switches sources from leaving a stale `hf_url` on a ModelScope
  model.
- The agent pipeline keys off the provider instead of `hf_url`: the fast-fail
  gate now explains *why* a model is skipped (no source / unknown source /
  source without a reachable card), the prompt context exposes
  source_url/source_id/source_label/asset_base_url while still filling the
  legacy hf_url/repo aliases, and the four README image extractors take a
  base_url (defaulting to HF) so relative paths resolve against the right
  site. Version grouping generalises to hf: / ms: / ta: keys.
- `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but
  accepts `source_url`, validates against every provider and returns the
  platform. `GET /api/lm/model-sources` lets the UI render the supported-site
  list from the server.
- Frontend: a `modelSourceHelpers` mirror of the registry drives the link
  dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the
  version-group key and the enrichment gate; the versions tab no longer sends
  ms:/ta: keys to the CivitAI API.

TensorArt stays in the list because provenance is worth keeping even when the
card is unreadable - the dialog says so plainly ("Sites that don't expose one
(currently TensorArt) can only be linked") and the context menu disables
enrichment with a matching tooltip, instead of the user getting
"Unsupported URL".

Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a
1882-byte card whose frontmatter carries base_model/tags/trigger_words, and
relative images resolve to .../resolve/master/....

Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest
tests/i18n and a Jinja compile pass over templates/. The nine locales carry
[TODO: Translate] for the new strings, completed in the next commit.
This commit is contained in:
Will Miao
2026-09-14 07:24:08 +08:00
parent 84146b62fd
commit 5ab0e88abc
51 changed files with 2518 additions and 279 deletions
+17 -5
View File
@@ -62,13 +62,23 @@ Environment variable overrides: `LLM_API_KEY`, `LLM_MODEL`, `LLM_API_BASE`, `LLM
### enrich_hf_metadata ### enrich_hf_metadata
Enriches HuggingFace-downloaded models with metadata extracted by an LLM from the HF model card. Enriches models linked to an external model site with metadata extracted by an LLM from the site's model card (README).
**Entry point**: Right-click context menu → "Enrich Metadata (Agent)" **Entry point**: Right-click context menu → "Enrich Metadata with AI"
**Supported model sources**:
| Platform | Link | AI enrichment | Direct download |
| --- | --- | --- | --- |
| Hugging Face | yes | yes | yes |
| ModelScope | yes | yes | no |
| TensorArt | yes | no (see below) | no |
TensorArt is link-only: `tensor.art` sits behind a Cloudflare managed challenge and its internal API requires session authorization, so the backend cannot read its model pages. Linking still stores the canonical page URL and the "View on TensorArt" link works.
**What it does**: **What it does**:
1. Reads the model's `.metadata.json` to get the `hf_url` 1. Reads the model's `.metadata.json` to get the source (`source_platform` + `source_url`, or the legacy `hf_url`)
2. Fetches the README.md from the HuggingFace repository 2. Fetches the model card through the provider in `py/services/model_sources/`
3. Sends the README + local metadata to the LLM for structured extraction 3. Sends the README + local metadata to the LLM for structured extraction
4. Writes extracted fields to `.metadata.json`: 4. Writes extracted fields to `.metadata.json`:
- `base_model` — only if current value is empty - `base_model` — only if current value is empty
@@ -81,6 +91,8 @@ Enriches HuggingFace-downloaded models with metadata extracted by an LLM from th
6. Updates the scanner cache 6. Updates the scanner cache
7. Broadcasts WebSocket progress events 7. Broadcasts WebSocket progress events
Models with no source, an unknown source, or a source without model-card access (TensorArt) are skipped with an explicit reason and counted in the run summary.
**Model types**: LoRA, Checkpoint, Embedding **Model types**: LoRA, Checkpoint, Embedding
## Adding a New Skill ## Adding a New Skill
@@ -129,7 +141,7 @@ Use `{{variable}}` placeholders that will be replaced with data from the `prepar
```markdown ```markdown
You are an expert assistant... You are an expert assistant...
Model URL: {{hf_url}} Model URL: {{source_url}}
README content: README content:
{{readme_content}} {{readme_content}}
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "Auf CivitAI anzeigen", "viewOnCivitai": "Auf CivitAI anzeigen",
"notAvailableFromCivitai": "Nicht auf CivitAI verfügbar", "notAvailableFromCivitai": "Nicht auf CivitAI verfügbar",
"viewOnHuggingFace": "Auf Hugging Face ansehen", "viewOnHuggingFace": "Auf Hugging Face ansehen",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "An ComfyUI senden (Klick: Anhängen, Shift+Klick: Ersetzen)", "sendToWorkflow": "An ComfyUI senden (Klick: Anhängen, Shift+Klick: Ersetzen)",
"copyLoRASyntax": "LoRA-Syntax kopieren", "copyLoRASyntax": "LoRA-Syntax kopieren",
"checkpointNameCopied": "Checkpoint-Name kopiert", "checkpointNameCopied": "Checkpoint-Name kopiert",
@@ -874,7 +875,7 @@
"checkUpdates": "Updates prüfen", "checkUpdates": "Updates prüfen",
"linkModel": "Modell verknüpfen", "linkModel": "Modell verknüpfen",
"linkCivitai": "Mit CivitAI neu verknüpfen", "linkCivitai": "Mit CivitAI neu verknüpfen",
"linkHuggingFace": "Mit HuggingFace verknüpfen", "linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "LoRA-Syntax kopieren", "copySyntax": "LoRA-Syntax kopieren",
"copyFilename": "Modell-Dateiname kopieren", "copyFilename": "Modell-Dateiname kopieren",
"copyRecipeSyntax": "Rezept-Syntax kopieren", "copyRecipeSyntax": "Rezept-Syntax kopieren",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "Ordnerpfad eingeben oder aus Baum unten auswählen...", "pathPlaceholder": "Ordnerpfad eingeben oder aus Baum unten auswählen...",
"root": "Stammverzeichnis" "root": "Stammverzeichnis"
}, },
"linkHuggingFace": { "linkModelSource": {
"title": "Mit HuggingFace verknüpfen", "title": "[TODO: Translate] Link to Model Source",
"infoText": "Fügen Sie die HuggingFace-Repository-URL ein, um dieses Modell zuzuordnen. Dies ermöglicht die KI-gestützte Metadatenanreicherung.", "infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "HuggingFace-Repository-URL:", "urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "https://huggingface.co/user/repo", "urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "Geben Sie die vollständige URL des HuggingFace-Repositorys ein.", "helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"confirmAction": "Speichern & Verknüpfen" "enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
}, },
"relinkCivitai": { "relinkCivitai": {
"title": "Mit CivitAI neu verknüpfen", "title": "Mit CivitAI neu verknüpfen",
@@ -1847,7 +1852,7 @@
"empty": "Noch keine Versionshistorie für dieses Modell vorhanden.", "empty": "Noch keine Versionshistorie für dieses Modell vorhanden.",
"error": "Versionen konnten nicht geladen werden.", "error": "Versionen konnten nicht geladen werden.",
"missingModelId": "Für dieses Modell ist keine CivitAI-Model-ID vorhanden.", "missingModelId": "Für dieses Modell ist keine CivitAI-Model-ID vorhanden.",
"hfGroupInfo": "Dies ist eine HuggingFace-Modellgruppe. Öffnen Sie die Bibliothek, um alle Versionen im Raster zu sehen.", "sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": { "confirm": {
"delete": "Diese Version aus Ihrer Bibliothek löschen?" "delete": "Diese Version aus Ihrer Bibliothek löschen?"
}, },
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "Modell erfolgreich über CivitArchive neu verknüpft", "linkCivArchSuccess": "Modell erfolgreich über CivitArchive neu verknüpft",
"fetchMetadataFirst": "Bitte rufen Sie zuerst Metadaten von CivitAI ab", "fetchMetadataFirst": "Bitte rufen Sie zuerst Metadaten von CivitAI ab",
"noCivitaiInfo": "Keine CivitAI-Informationen verfügbar", "noCivitaiInfo": "Keine CivitAI-Informationen verfügbar",
"missingHash": "Modell-Hash nicht verfügbar" "missingHash": "Modell-Hash nicht verfügbar",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
}, },
"exampleImages": { "exampleImages": {
"pathUpdated": "Beispielbilder-Pfad erfolgreich aktualisiert", "pathUpdated": "Beispielbilder-Pfad erfolgreich aktualisiert",
+18 -11
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "View on CivitAI", "viewOnCivitai": "View on CivitAI",
"notAvailableFromCivitai": "Not available from CivitAI", "notAvailableFromCivitai": "Not available from CivitAI",
"viewOnHuggingFace": "View on Hugging Face", "viewOnHuggingFace": "View on Hugging Face",
"viewOnSource": "View on {source}",
"sendToWorkflow": "Send to ComfyUI (Click: Append, Shift+Click: Replace)", "sendToWorkflow": "Send to ComfyUI (Click: Append, Shift+Click: Replace)",
"copyLoRASyntax": "Copy LoRA Syntax", "copyLoRASyntax": "Copy LoRA Syntax",
"checkpointNameCopied": "Checkpoint name copied", "checkpointNameCopied": "Checkpoint name copied",
@@ -867,14 +868,14 @@
"complete": "Auto-organize complete", "complete": "Auto-organize complete",
"error": "Error: {error}" "error": "Error: {error}"
}, },
"enrichHfAgent": "Enrich HF Metadata (AI)" "enrichHfAgent": "Enrich Metadata with AI"
}, },
"contextMenu": { "contextMenu": {
"refreshMetadata": "Refresh CivitAI Data", "refreshMetadata": "Refresh CivitAI Data",
"checkUpdates": "Check Updates", "checkUpdates": "Check Updates",
"linkModel": "Link Model", "linkModel": "Link Model",
"linkCivitai": "Link to CivitAI", "linkCivitai": "Link to CivitAI",
"linkHuggingFace": "Link to HuggingFace", "linkModelSource": "Link to Model Source",
"copySyntax": "Copy LoRA Syntax", "copySyntax": "Copy LoRA Syntax",
"copyFilename": "Copy Model Filename", "copyFilename": "Copy Model Filename",
"copyRecipeSyntax": "Copy Recipe Syntax", "copyRecipeSyntax": "Copy Recipe Syntax",
@@ -896,7 +897,7 @@
"viewAllLoras": "View All LoRAs", "viewAllLoras": "View All LoRAs",
"downloadMissingLoras": "Download Missing LoRAs", "downloadMissingLoras": "Download Missing LoRAs",
"deleteRecipe": "Delete Recipe", "deleteRecipe": "Delete Recipe",
"enrichHfAgent": "Enrich HF Metadata (AI)" "enrichHfAgent": "Enrich Metadata with AI"
} }
}, },
"recipes": { "recipes": {
@@ -1596,12 +1597,16 @@
"pathPlaceholder": "Type folder path or select from tree below...", "pathPlaceholder": "Type folder path or select from tree below...",
"root": "Root" "root": "Root"
}, },
"linkHuggingFace": { "linkModelSource": {
"title": "Link to HuggingFace", "title": "Link to Model Source",
"infoText": "Paste the HuggingFace repository URL to associate this model with its source. This enables AI-powered metadata enrichment.", "infoText": "Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "HuggingFace Repository URL:", "urlLabel": "Model Page URL:",
"urlPlaceholder": "https://huggingface.co/user/repo", "urlPlaceholder": "https://huggingface.co/user/repo",
"helpText": "Enter the full URL of the HuggingFace repository.", "helpText": "Enter the full URL of the model page. Supported sites:",
"enrichNote": "AI enrichment needs a readable model card. Sites that don't expose one (currently TensorArt) can only be linked.",
"urlRequired": "Please enter a model page URL.",
"invalidUrl": "Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "Linking model source...",
"confirmAction": "Save & Link" "confirmAction": "Save & Link"
}, },
"relinkCivitai": { "relinkCivitai": {
@@ -1847,7 +1852,7 @@
"empty": "No version history available for this model yet.", "empty": "No version history available for this model yet.",
"error": "Failed to load versions.", "error": "Failed to load versions.",
"missingModelId": "This model is missing a CivitAI model id.", "missingModelId": "This model is missing a CivitAI model id.",
"hfGroupInfo": "This is a HuggingFace model group. Open the library to see all versions in the grid.", "sourceGroupInfo": "This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": { "confirm": {
"delete": "Delete this version from your library?" "delete": "Delete this version from your library?"
}, },
@@ -2478,12 +2483,14 @@
"contentRatingFailed": "Failed to set content rating: {message}", "contentRatingFailed": "Failed to set content rating: {message}",
"relinkSuccess": "Model successfully re-linked to CivitAI", "relinkSuccess": "Model successfully re-linked to CivitAI",
"relinkFailed": "Error: {message}", "relinkFailed": "Error: {message}",
"linkHfSuccess": "Model successfully linked to HuggingFace", "linkHfSuccess": "Model successfully linked to its model source",
"linkHfFailed": "Error: {message}", "linkHfFailed": "Error: {message}",
"linkCivArchSuccess": "Model successfully re-linked via CivitArchive", "linkCivArchSuccess": "Model successfully re-linked via CivitArchive",
"fetchMetadataFirst": "Please fetch metadata from CivitAI first", "fetchMetadataFirst": "Please fetch metadata from CivitAI first",
"noCivitaiInfo": "No CivitAI information available", "noCivitaiInfo": "No CivitAI information available",
"missingHash": "Model hash not available" "missingHash": "Model hash not available",
"enrichNeedsSource": "Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "AI enrichment is not available for {source} models"
}, },
"exampleImages": { "exampleImages": {
"pathUpdated": "Example images path updated successfully", "pathUpdated": "Example images path updated successfully",
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "Ver en CivitAI", "viewOnCivitai": "Ver en CivitAI",
"notAvailableFromCivitai": "No disponible en CivitAI", "notAvailableFromCivitai": "No disponible en CivitAI",
"viewOnHuggingFace": "Ver en Hugging Face", "viewOnHuggingFace": "Ver en Hugging Face",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "Enviar a ComfyUI (Clic: Añadir, Shift+Clic: Reemplazar)", "sendToWorkflow": "Enviar a ComfyUI (Clic: Añadir, Shift+Clic: Reemplazar)",
"copyLoRASyntax": "Copiar sintaxis de LoRA", "copyLoRASyntax": "Copiar sintaxis de LoRA",
"checkpointNameCopied": "Nombre del checkpoint copiado", "checkpointNameCopied": "Nombre del checkpoint copiado",
@@ -874,7 +875,7 @@
"checkUpdates": "Comprobar actualizaciones", "checkUpdates": "Comprobar actualizaciones",
"linkModel": "Vincular modelo", "linkModel": "Vincular modelo",
"linkCivitai": "Re-vincular a CivitAI", "linkCivitai": "Re-vincular a CivitAI",
"linkHuggingFace": "Vincular a HuggingFace", "linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "Copiar sintaxis de LoRA", "copySyntax": "Copiar sintaxis de LoRA",
"copyFilename": "Copiar nombre de archivo del modelo", "copyFilename": "Copiar nombre de archivo del modelo",
"copyRecipeSyntax": "Copiar sintaxis de receta", "copyRecipeSyntax": "Copiar sintaxis de receta",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "Escribe la ruta de la carpeta o selecciona del árbol de abajo...", "pathPlaceholder": "Escribe la ruta de la carpeta o selecciona del árbol de abajo...",
"root": "Raíz" "root": "Raíz"
}, },
"linkHuggingFace": { "linkModelSource": {
"title": "Vincular a HuggingFace", "title": "[TODO: Translate] Link to Model Source",
"infoText": "Pegue la URL del repositorio de HuggingFace para asociar este modelo. Esto permite el enriquecimiento de metadatos con IA.", "infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "URL del repositorio de HuggingFace:", "urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "https://huggingface.co/user/repo", "urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "Ingrese la URL completa del repositorio de HuggingFace.", "helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"confirmAction": "Guardar y vincular" "enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
}, },
"relinkCivitai": { "relinkCivitai": {
"title": "Re-vincular a CivitAI", "title": "Re-vincular a CivitAI",
@@ -1847,7 +1852,7 @@
"empty": "Aún no hay historial de versiones para este modelo.", "empty": "Aún no hay historial de versiones para este modelo.",
"error": "No se pudieron cargar las versiones.", "error": "No se pudieron cargar las versiones.",
"missingModelId": "Este modelo no tiene un ID de modelo de CivitAI.", "missingModelId": "Este modelo no tiene un ID de modelo de CivitAI.",
"hfGroupInfo": "Este es un grupo de modelos de HuggingFace. Abra la biblioteca para ver todas las versiones en la cuadrícula.", "sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": { "confirm": {
"delete": "¿Eliminar esta versión de tu biblioteca?" "delete": "¿Eliminar esta versión de tu biblioteca?"
}, },
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "Modelo re-vinculado exitosamente mediante CivitArchive", "linkCivArchSuccess": "Modelo re-vinculado exitosamente mediante CivitArchive",
"fetchMetadataFirst": "Por favor obtén metadatos de CivitAI primero", "fetchMetadataFirst": "Por favor obtén metadatos de CivitAI primero",
"noCivitaiInfo": "No hay información de CivitAI disponible", "noCivitaiInfo": "No hay información de CivitAI disponible",
"missingHash": "Hash del modelo no disponible" "missingHash": "Hash del modelo no disponible",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
}, },
"exampleImages": { "exampleImages": {
"pathUpdated": "Ruta de imágenes de ejemplo actualizada exitosamente", "pathUpdated": "Ruta de imágenes de ejemplo actualizada exitosamente",
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "Voir sur CivitAI", "viewOnCivitai": "Voir sur CivitAI",
"notAvailableFromCivitai": "Non disponible sur CivitAI", "notAvailableFromCivitai": "Non disponible sur CivitAI",
"viewOnHuggingFace": "Voir sur Hugging Face", "viewOnHuggingFace": "Voir sur Hugging Face",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "Envoyer vers ComfyUI (Clic: Ajouter, Maj+Clic: Remplacer)", "sendToWorkflow": "Envoyer vers ComfyUI (Clic: Ajouter, Maj+Clic: Remplacer)",
"copyLoRASyntax": "Copier la syntaxe LoRA", "copyLoRASyntax": "Copier la syntaxe LoRA",
"checkpointNameCopied": "Nom du checkpoint copié", "checkpointNameCopied": "Nom du checkpoint copié",
@@ -874,7 +875,7 @@
"checkUpdates": "Vérifier les mises à jour", "checkUpdates": "Vérifier les mises à jour",
"linkModel": "Lier le modèle", "linkModel": "Lier le modèle",
"linkCivitai": "Relier à nouveau à CivitAI", "linkCivitai": "Relier à nouveau à CivitAI",
"linkHuggingFace": "Lier à HuggingFace", "linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "Copier la syntaxe LoRA", "copySyntax": "Copier la syntaxe LoRA",
"copyFilename": "Copier le nom de fichier du modèle", "copyFilename": "Copier le nom de fichier du modèle",
"copyRecipeSyntax": "Copier la syntaxe de la recipe", "copyRecipeSyntax": "Copier la syntaxe de la recipe",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "Tapez le chemin du dossier ou sélectionnez dans l'arbre ci-dessous...", "pathPlaceholder": "Tapez le chemin du dossier ou sélectionnez dans l'arbre ci-dessous...",
"root": "Racine" "root": "Racine"
}, },
"linkHuggingFace": { "linkModelSource": {
"title": "Lier à HuggingFace", "title": "[TODO: Translate] Link to Model Source",
"infoText": "Collez l'URL du dépôt HuggingFace pour associer ce modèle à sa source. Cela permet l'enrichissement des métadonnées par IA.", "infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "URL du dépôt HuggingFace :", "urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "https://huggingface.co/user/repo", "urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "Entrez l'URL complète du dépôt HuggingFace.", "helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"confirmAction": "Enregistrer & lier" "enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
}, },
"relinkCivitai": { "relinkCivitai": {
"title": "Relier à nouveau à CivitAI", "title": "Relier à nouveau à CivitAI",
@@ -1847,7 +1852,7 @@
"empty": "Aucun historique de versions n'est disponible pour ce modèle pour le moment.", "empty": "Aucun historique de versions n'est disponible pour ce modèle pour le moment.",
"error": "Échec du chargement des versions.", "error": "Échec du chargement des versions.",
"missingModelId": "Ce modèle ne possède pas d'identifiant de modèle CivitAI.", "missingModelId": "Ce modèle ne possède pas d'identifiant de modèle CivitAI.",
"hfGroupInfo": "Ceci est un groupe de modèles HuggingFace. Ouvrez la bibliothèque pour voir toutes les versions dans la grille.", "sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": { "confirm": {
"delete": "Supprimer cette version de votre bibliothèque ?" "delete": "Supprimer cette version de votre bibliothèque ?"
}, },
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "Modèle relié via CivitArchive avec succès", "linkCivArchSuccess": "Modèle relié via CivitArchive avec succès",
"fetchMetadataFirst": "Veuillez d'abord récupérer les métadonnées depuis CivitAI", "fetchMetadataFirst": "Veuillez d'abord récupérer les métadonnées depuis CivitAI",
"noCivitaiInfo": "Aucune information CivitAI disponible", "noCivitaiInfo": "Aucune information CivitAI disponible",
"missingHash": "Hash du modèle non disponible" "missingHash": "Hash du modèle non disponible",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
}, },
"exampleImages": { "exampleImages": {
"pathUpdated": "Chemin des images d'exemple mis à jour avec succès", "pathUpdated": "Chemin des images d'exemple mis à jour avec succès",
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "הצג ב-CivitAI", "viewOnCivitai": "הצג ב-CivitAI",
"notAvailableFromCivitai": "לא זמין מ-CivitAI", "notAvailableFromCivitai": "לא זמין מ-CivitAI",
"viewOnHuggingFace": "צפייה ב-Hugging Face", "viewOnHuggingFace": "צפייה ב-Hugging Face",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "שלח ל-ComfyUI (לחיצה: הוסף, Shift+לחיצה: החלף)", "sendToWorkflow": "שלח ל-ComfyUI (לחיצה: הוסף, Shift+לחיצה: החלף)",
"copyLoRASyntax": "העתק תחביר LoRA", "copyLoRASyntax": "העתק תחביר LoRA",
"checkpointNameCopied": "שם Checkpoint הועתק", "checkpointNameCopied": "שם Checkpoint הועתק",
@@ -874,7 +875,7 @@
"checkUpdates": "בדוק עדכונים", "checkUpdates": "בדוק עדכונים",
"linkModel": "קישור מודל", "linkModel": "קישור מודל",
"linkCivitai": "קשר מחדש ל-CivitAI", "linkCivitai": "קשר מחדש ל-CivitAI",
"linkHuggingFace": "קישור ל-HuggingFace", "linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "העתק תחביר LoRA", "copySyntax": "העתק תחביר LoRA",
"copyFilename": "העתק שם קובץ מודל", "copyFilename": "העתק שם קובץ מודל",
"copyRecipeSyntax": "העתק תחביר מתכון", "copyRecipeSyntax": "העתק תחביר מתכון",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "הקלד נתיב תיקייה או בחר מהעץ למטה...", "pathPlaceholder": "הקלד נתיב תיקייה או בחר מהעץ למטה...",
"root": "שורש" "root": "שורש"
}, },
"linkHuggingFace": { "linkModelSource": {
"title": "קישור ל-HuggingFace", "title": "[TODO: Translate] Link to Model Source",
"infoText": "הדבק את כתובת ה-URL של מאגר HuggingFace כדי לשייך מודל זה למקורו. פעולה זו מאפשרת העשרת מטא-נתונים באמצעות AI.", "infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "כתובת URL של מאגר HuggingFace:", "urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "https://huggingface.co/user/repo", "urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "הזן את כתובת ה-URL המלאה של מאגר HuggingFace.", "helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"confirmAction": "שמור וקשר" "enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
}, },
"relinkCivitai": { "relinkCivitai": {
"title": "קשר מחדש ל-CivitAI", "title": "קשר מחדש ל-CivitAI",
@@ -1847,7 +1852,7 @@
"empty": "אין עדיין היסטוריית גרסאות למודל זה.", "empty": "אין עדיין היסטוריית גרסאות למודל זה.",
"error": "טעינת הגרסאות נכשלה.", "error": "טעינת הגרסאות נכשלה.",
"missingModelId": "למודל זה אין מזהה מודל של CivitAI.", "missingModelId": "למודל זה אין מזהה מודל של CivitAI.",
"hfGroupInfo": "זוהי קבוצת מודלים של HuggingFace. פתח את הספרייה כדי לראות את כל הגרסאות ברשת.", "sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": { "confirm": {
"delete": "למחוק גרסה זו מהספרייה שלך?" "delete": "למחוק גרסה זו מהספרייה שלך?"
}, },
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "המודל קושר מחדש דרך CivitArchive בהצלחה", "linkCivArchSuccess": "המודל קושר מחדש דרך CivitArchive בהצלחה",
"fetchMetadataFirst": "אנא אחזר מטא-נתונים מ-CivitAI תחילה", "fetchMetadataFirst": "אנא אחזר מטא-נתונים מ-CivitAI תחילה",
"noCivitaiInfo": "אין מידע מ-CivitAI זמין", "noCivitaiInfo": "אין מידע מ-CivitAI זמין",
"missingHash": "ה-hash של המודל אינו זמין" "missingHash": "ה-hash של המודל אינו זמין",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
}, },
"exampleImages": { "exampleImages": {
"pathUpdated": "נתיב תמונות הדוגמה עודכן בהצלחה", "pathUpdated": "נתיב תמונות הדוגמה עודכן בהצלחה",
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "CivitAIで表示", "viewOnCivitai": "CivitAIで表示",
"notAvailableFromCivitai": "CivitAIでは利用できません", "notAvailableFromCivitai": "CivitAIでは利用できません",
"viewOnHuggingFace": "Hugging Face で見る", "viewOnHuggingFace": "Hugging Face で見る",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "ComfyUIに送信(クリック:追加、Shift+クリック:置換)", "sendToWorkflow": "ComfyUIに送信(クリック:追加、Shift+クリック:置換)",
"copyLoRASyntax": "LoRA構文をコピー", "copyLoRASyntax": "LoRA構文をコピー",
"checkpointNameCopied": "Checkpointの名前をコピーしました", "checkpointNameCopied": "Checkpointの名前をコピーしました",
@@ -874,7 +875,7 @@
"checkUpdates": "更新確認", "checkUpdates": "更新確認",
"linkModel": "モデルをリンク", "linkModel": "モデルをリンク",
"linkCivitai": "CivitAI にリンク", "linkCivitai": "CivitAI にリンク",
"linkHuggingFace": "HuggingFace にリンク", "linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "LoRA構文をコピー", "copySyntax": "LoRA構文をコピー",
"copyFilename": "モデルファイル名をコピー", "copyFilename": "モデルファイル名をコピー",
"copyRecipeSyntax": "レシピ構文をコピー", "copyRecipeSyntax": "レシピ構文をコピー",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "フォルダパスを入力するか、下のツリーから選択...", "pathPlaceholder": "フォルダパスを入力するか、下のツリーから選択...",
"root": "ルート" "root": "ルート"
}, },
"linkHuggingFace": { "linkModelSource": {
"title": "HuggingFace にリンク", "title": "[TODO: Translate] Link to Model Source",
"infoText": "HuggingFace リポジトリの URL を貼り付けてモデルを関連付けます。AI によるメタデータ補完が有効になります。", "infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "HuggingFace リポジトリ URL", "urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "https://huggingface.co/user/repo", "urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "完全な HuggingFace リポジトリ URL を入力してください。", "helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"confirmAction": "保存&リンク" "enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
}, },
"relinkCivitai": { "relinkCivitai": {
"title": "CivitAIに再リンク", "title": "CivitAIに再リンク",
@@ -1847,7 +1852,7 @@
"empty": "このモデルにはまだバージョン履歴がありません。", "empty": "このモデルにはまだバージョン履歴がありません。",
"error": "バージョンの読み込みに失敗しました。", "error": "バージョンの読み込みに失敗しました。",
"missingModelId": "このモデルにはCivitAIのモデルIDがありません。", "missingModelId": "このモデルにはCivitAIのモデルIDがありません。",
"hfGroupInfo": "これは HuggingFace モデルグループです。ライブラリを開いてグリッドですべてのバージョンを表示してください。", "sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": { "confirm": {
"delete": "このバージョンをライブラリから削除しますか?" "delete": "このバージョンをライブラリから削除しますか?"
}, },
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "モデルがCivitArchive経由で正常に再リンクされました", "linkCivArchSuccess": "モデルがCivitArchive経由で正常に再リンクされました",
"fetchMetadataFirst": "最初にCivitAIからメタデータを取得してください", "fetchMetadataFirst": "最初にCivitAIからメタデータを取得してください",
"noCivitaiInfo": "CivitAI情報が利用できません", "noCivitaiInfo": "CivitAI情報が利用できません",
"missingHash": "モデルハッシュが利用できません" "missingHash": "モデルハッシュが利用できません",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
}, },
"exampleImages": { "exampleImages": {
"pathUpdated": "例画像パスが正常に更新されました", "pathUpdated": "例画像パスが正常に更新されました",
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "CivitAI에서 보기", "viewOnCivitai": "CivitAI에서 보기",
"notAvailableFromCivitai": "CivitAI에서 사용할 수 없음", "notAvailableFromCivitai": "CivitAI에서 사용할 수 없음",
"viewOnHuggingFace": "Hugging Face에서 보기", "viewOnHuggingFace": "Hugging Face에서 보기",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "ComfyUI로 전송 (클릭: 추가, Shift+클릭: 교체)", "sendToWorkflow": "ComfyUI로 전송 (클릭: 추가, Shift+클릭: 교체)",
"copyLoRASyntax": "LoRA 문법 복사", "copyLoRASyntax": "LoRA 문법 복사",
"checkpointNameCopied": "Checkpoint 이름 복사됨", "checkpointNameCopied": "Checkpoint 이름 복사됨",
@@ -874,7 +875,7 @@
"checkUpdates": "업데이트 확인", "checkUpdates": "업데이트 확인",
"linkModel": "모델 연결", "linkModel": "모델 연결",
"linkCivitai": "CivitAI에 연결", "linkCivitai": "CivitAI에 연결",
"linkHuggingFace": "HuggingFace에 연결", "linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "LoRA 문법 복사", "copySyntax": "LoRA 문법 복사",
"copyFilename": "모델 파일명 복사", "copyFilename": "모델 파일명 복사",
"copyRecipeSyntax": "레시피 문법 복사", "copyRecipeSyntax": "레시피 문법 복사",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "폴더 경로를 입력하거나 아래 트리에서 선택하세요...", "pathPlaceholder": "폴더 경로를 입력하거나 아래 트리에서 선택하세요...",
"root": "루트" "root": "루트"
}, },
"linkHuggingFace": { "linkModelSource": {
"title": "HuggingFace에 연결", "title": "[TODO: Translate] Link to Model Source",
"infoText": "HuggingFace 저장소 URL을 붙여넣어 모델을 연결합니다. AI 메타데이터 보강 기능을 사용할 수 있습니다.", "infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "HuggingFace 저장소 URL", "urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "https://huggingface.co/user/repo", "urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "전체 HuggingFace 저장소 URL을 입력하세요.", "helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"confirmAction": "저장 및 연결" "enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
}, },
"relinkCivitai": { "relinkCivitai": {
"title": "CivitAI에 다시 연결", "title": "CivitAI에 다시 연결",
@@ -1847,7 +1852,7 @@
"empty": "이 모델에는 아직 버전 기록이 없습니다.", "empty": "이 모델에는 아직 버전 기록이 없습니다.",
"error": "버전을 불러오지 못했습니다.", "error": "버전을 불러오지 못했습니다.",
"missingModelId": "이 모델에는 CivitAI 모델 ID가 없습니다.", "missingModelId": "이 모델에는 CivitAI 모델 ID가 없습니다.",
"hfGroupInfo": "HuggingFace 모델 그룹입니다. 라이브러리를 열어 그리드에서 모든 버전을 확인하세요.", "sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": { "confirm": {
"delete": "이 버전을 라이브러리에서 삭제하시겠습니까?" "delete": "이 버전을 라이브러리에서 삭제하시겠습니까?"
}, },
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "모델이 CivitArchive을 통해 성공적으로 다시 연결되었습니다", "linkCivArchSuccess": "모델이 CivitArchive을 통해 성공적으로 다시 연결되었습니다",
"fetchMetadataFirst": "먼저 CivitAI에서 메타데이터를 가져와주세요", "fetchMetadataFirst": "먼저 CivitAI에서 메타데이터를 가져와주세요",
"noCivitaiInfo": "사용 가능한 CivitAI 정보가 없습니다", "noCivitaiInfo": "사용 가능한 CivitAI 정보가 없습니다",
"missingHash": "모델 해시를 사용할 수 없습니다" "missingHash": "모델 해시를 사용할 수 없습니다",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
}, },
"exampleImages": { "exampleImages": {
"pathUpdated": "예시 이미지 경로가 성공적으로 업데이트되었습니다", "pathUpdated": "예시 이미지 경로가 성공적으로 업데이트되었습니다",
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "Посмотреть на CivitAI", "viewOnCivitai": "Посмотреть на CivitAI",
"notAvailableFromCivitai": "Недоступно на CivitAI", "notAvailableFromCivitai": "Недоступно на CivitAI",
"viewOnHuggingFace": "Открыть Hugging Face", "viewOnHuggingFace": "Открыть Hugging Face",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "Отправить в ComfyUI (Клик: Добавить, Shift+Клик: Заменить)", "sendToWorkflow": "Отправить в ComfyUI (Клик: Добавить, Shift+Клик: Заменить)",
"copyLoRASyntax": "Копировать синтаксис LoRA", "copyLoRASyntax": "Копировать синтаксис LoRA",
"checkpointNameCopied": "Имя checkpoint скопировано", "checkpointNameCopied": "Имя checkpoint скопировано",
@@ -874,7 +875,7 @@
"checkUpdates": "Проверить обновления", "checkUpdates": "Проверить обновления",
"linkModel": "Связать модель", "linkModel": "Связать модель",
"linkCivitai": "Пересвязать с CivitAI", "linkCivitai": "Пересвязать с CivitAI",
"linkHuggingFace": "Связать с HuggingFace", "linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "Копировать синтаксис LoRA", "copySyntax": "Копировать синтаксис LoRA",
"copyFilename": "Копировать имя файла модели", "copyFilename": "Копировать имя файла модели",
"copyRecipeSyntax": "Копировать синтаксис рецепта", "copyRecipeSyntax": "Копировать синтаксис рецепта",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "Введите путь к папке или выберите из дерева ниже...", "pathPlaceholder": "Введите путь к папке или выберите из дерева ниже...",
"root": "Корень" "root": "Корень"
}, },
"linkHuggingFace": { "linkModelSource": {
"title": "Связать с HuggingFace", "title": "[TODO: Translate] Link to Model Source",
"infoText": "Вставьте URL репозитория HuggingFace, чтобы связать эту модель с её источником. Это позволит обогащать метаданные с помощью ИИ.", "infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "URL репозитория HuggingFace:", "urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "https://huggingface.co/user/repo", "urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "Введите полный URL репозитория HuggingFace.", "helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"confirmAction": "Сохранить и связать" "enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
}, },
"relinkCivitai": { "relinkCivitai": {
"title": "Пересвязать с CivitAI", "title": "Пересвязать с CivitAI",
@@ -1847,7 +1852,7 @@
"empty": "Для этой модели пока нет истории версий.", "empty": "Для этой модели пока нет истории версий.",
"error": "Не удалось загрузить версии.", "error": "Не удалось загрузить версии.",
"missingModelId": "У этой модели отсутствует идентификатор модели CivitAI.", "missingModelId": "У этой модели отсутствует идентификатор модели CivitAI.",
"hfGroupInfo": "Это группа моделей HuggingFace. Откройте библиотеку, чтобы увидеть все версии в сетке.", "sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": { "confirm": {
"delete": "Удалить эту версию из библиотеки?" "delete": "Удалить эту версию из библиотеки?"
}, },
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "Модель успешно пересвязана через CivitArchive", "linkCivArchSuccess": "Модель успешно пересвязана через CivitArchive",
"fetchMetadataFirst": "Пожалуйста, сначала получите метаданные с CivitAI", "fetchMetadataFirst": "Пожалуйста, сначала получите метаданные с CivitAI",
"noCivitaiInfo": "Информация CivitAI недоступна", "noCivitaiInfo": "Информация CivitAI недоступна",
"missingHash": "Хеш модели недоступен" "missingHash": "Хеш модели недоступен",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
}, },
"exampleImages": { "exampleImages": {
"pathUpdated": "Путь к примерам изображений успешно обновлен", "pathUpdated": "Путь к примерам изображений успешно обновлен",
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "在 CivitAI 查看", "viewOnCivitai": "在 CivitAI 查看",
"notAvailableFromCivitai": "CivitAI 上不可用", "notAvailableFromCivitai": "CivitAI 上不可用",
"viewOnHuggingFace": "在 Hugging Face 查看", "viewOnHuggingFace": "在 Hugging Face 查看",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "发送到 ComfyUI(点击:追加,Shift+点击:替换)", "sendToWorkflow": "发送到 ComfyUI(点击:追加,Shift+点击:替换)",
"copyLoRASyntax": "复制 LoRA 语法", "copyLoRASyntax": "复制 LoRA 语法",
"checkpointNameCopied": "Checkpoint 名称已复制", "checkpointNameCopied": "Checkpoint 名称已复制",
@@ -874,7 +875,7 @@
"checkUpdates": "检查更新", "checkUpdates": "检查更新",
"linkModel": "链接模型", "linkModel": "链接模型",
"linkCivitai": "链接到 CivitAI", "linkCivitai": "链接到 CivitAI",
"linkHuggingFace": "链接到 HuggingFace", "linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "复制 LoRA 语法", "copySyntax": "复制 LoRA 语法",
"copyFilename": "复制模型文件名", "copyFilename": "复制模型文件名",
"copyRecipeSyntax": "复制配方语法", "copyRecipeSyntax": "复制配方语法",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "输入文件夹路径或从下方树中选择...", "pathPlaceholder": "输入文件夹路径或从下方树中选择...",
"root": "根目录" "root": "根目录"
}, },
"linkHuggingFace": { "linkModelSource": {
"title": "链接到 HuggingFace", "title": "[TODO: Translate] Link to Model Source",
"infoText": "粘贴 HuggingFace 仓库 URL 以关联此模型。关联后可启用 AI 元数据增强功能。", "infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "HuggingFace 仓库 URL", "urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "https://huggingface.co/user/repo", "urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "请输入完整的 HuggingFace 仓库 URL。", "helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"confirmAction": "保存并链接" "enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
}, },
"relinkCivitai": { "relinkCivitai": {
"title": "重新关联到 CivitAI", "title": "重新关联到 CivitAI",
@@ -1847,7 +1852,7 @@
"empty": "该模型还没有版本历史。", "empty": "该模型还没有版本历史。",
"error": "加载版本失败。", "error": "加载版本失败。",
"missingModelId": "该模型缺少 CivitAI 模型 ID。", "missingModelId": "该模型缺少 CivitAI 模型 ID。",
"hfGroupInfo": "这是一个 HuggingFace 模型组。打开库页面即可在网格中查看所有版本。", "sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": { "confirm": {
"delete": "从库中删除此版本?" "delete": "从库中删除此版本?"
}, },
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "模型已成功通过 CivitArchive 重新关联", "linkCivArchSuccess": "模型已成功通过 CivitArchive 重新关联",
"fetchMetadataFirst": "请先从 CivitAI 获取元数据", "fetchMetadataFirst": "请先从 CivitAI 获取元数据",
"noCivitaiInfo": "无 CivitAI 信息", "noCivitaiInfo": "无 CivitAI 信息",
"missingHash": "模型哈希不可用" "missingHash": "模型哈希不可用",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
}, },
"exampleImages": { "exampleImages": {
"pathUpdated": "示例图片路径更新成功", "pathUpdated": "示例图片路径更新成功",
+17 -10
View File
@@ -139,6 +139,7 @@
"viewOnCivitai": "在 CivitAI 查看", "viewOnCivitai": "在 CivitAI 查看",
"notAvailableFromCivitai": "CivitAI 不提供", "notAvailableFromCivitai": "CivitAI 不提供",
"viewOnHuggingFace": "在 Hugging Face 查看", "viewOnHuggingFace": "在 Hugging Face 查看",
"viewOnSource": "[TODO: Translate] View on {source}",
"sendToWorkflow": "傳送到 ComfyUI(點擊:附加,Shift+點擊:取代)", "sendToWorkflow": "傳送到 ComfyUI(點擊:附加,Shift+點擊:取代)",
"copyLoRASyntax": "複製 LoRA 語法", "copyLoRASyntax": "複製 LoRA 語法",
"checkpointNameCopied": "Checkpoint 名稱已複製", "checkpointNameCopied": "Checkpoint 名稱已複製",
@@ -874,7 +875,7 @@
"checkUpdates": "檢查更新", "checkUpdates": "檢查更新",
"linkModel": "連結模型", "linkModel": "連結模型",
"linkCivitai": "連結到 CivitAI", "linkCivitai": "連結到 CivitAI",
"linkHuggingFace": "連結到 HuggingFace", "linkModelSource": "[TODO: Translate] Link to Model Source",
"copySyntax": "複製 LoRA 語法", "copySyntax": "複製 LoRA 語法",
"copyFilename": "複製模型檔名", "copyFilename": "複製模型檔名",
"copyRecipeSyntax": "複製配方語法", "copyRecipeSyntax": "複製配方語法",
@@ -1596,13 +1597,17 @@
"pathPlaceholder": "輸入資料夾路徑或從下方樹狀結構選擇...", "pathPlaceholder": "輸入資料夾路徑或從下方樹狀結構選擇...",
"root": "根目錄" "root": "根目錄"
}, },
"linkHuggingFace": { "linkModelSource": {
"title": "連結到 HuggingFace", "title": "[TODO: Translate] Link to Model Source",
"infoText": "貼上 HuggingFace 倉庫 URL 以關聯此模型。關聯後可啟用 AI 中繼資料增強功能。", "infoText": "[TODO: Translate] Paste the model page URL to associate this model with its source. Linking enables AI-powered metadata enrichment for Hugging Face and ModelScope models.",
"urlLabel": "HuggingFace 倉庫 URL", "urlLabel": "[TODO: Translate] Model Page URL:",
"urlPlaceholder": "https://huggingface.co/user/repo", "urlPlaceholder": "[TODO: Translate] https://huggingface.co/user/repo",
"helpText": "請輸入完整的 HuggingFace 倉庫 URL。", "helpText": "[TODO: Translate] Enter the full URL of the model page. Supported sites:",
"confirmAction": "儲存並連結" "enrichNote": "[TODO: Translate] Note: AI metadata enrichment requires an accessible model card. TensorArt pages are not readable by the backend, so only the link is stored.",
"urlRequired": "[TODO: Translate] Please enter a model page URL.",
"invalidUrl": "[TODO: Translate] Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.",
"linking": "[TODO: Translate] Linking model source...",
"confirmAction": "[TODO: Translate] Save & Link"
}, },
"relinkCivitai": { "relinkCivitai": {
"title": "重新連結至 CivitAI", "title": "重新連結至 CivitAI",
@@ -1847,7 +1852,7 @@
"empty": "此模型尚無版本歷史。", "empty": "此模型尚無版本歷史。",
"error": "載入版本失敗。", "error": "載入版本失敗。",
"missingModelId": "此模型缺少 CivitAI 模型 ID。", "missingModelId": "此模型缺少 CivitAI 模型 ID。",
"hfGroupInfo": "這是一個 HuggingFace 模型組。打開庫頁面即可在網格中查看所有版本。", "sourceGroupInfo": "[TODO: Translate] This is a {source} model group. Open the library to see all versions in the grid.",
"confirm": { "confirm": {
"delete": "要從庫中刪除此版本嗎?" "delete": "要從庫中刪除此版本嗎?"
}, },
@@ -2483,7 +2488,9 @@
"linkCivArchSuccess": "模型已成功透過 CivitArchive 重新連結", "linkCivArchSuccess": "模型已成功透過 CivitArchive 重新連結",
"fetchMetadataFirst": "請先從 CivitAI 取得 metadata", "fetchMetadataFirst": "請先從 CivitAI 取得 metadata",
"noCivitaiInfo": "無 CivitAI 資訊", "noCivitaiInfo": "無 CivitAI 資訊",
"missingHash": "模型雜湊不可用" "missingHash": "模型雜湊不可用",
"enrichNeedsSource": "[TODO: Translate] Link this model to a model source first (Link Model → Link to Model Source)",
"enrichUnsupportedSource": "[TODO: Translate] AI enrichment is not available for {source} models"
}, },
"exampleImages": { "exampleImages": {
"pathUpdated": "範例圖片路徑已更新", "pathUpdated": "範例圖片路徑已更新",
+96 -20
View File
@@ -22,6 +22,11 @@ from ...services.downloader import (
get_downloader, get_downloader,
) )
from ...services.aria2_downloader import Aria2Downloader from ...services.aria2_downloader import Aria2Downloader
from ...services.model_sources import (
detect_source,
list_sources,
normalize_metadata_source,
)
from ...services.settings_manager import get_settings_manager from ...services.settings_manager import get_settings_manager
from ...services.service_registry import ServiceRegistry from ...services.service_registry import ServiceRegistry
from ...services.websocket_manager import ws_manager from ...services.websocket_manager import ws_manager
@@ -120,6 +125,8 @@ async def _save_hf_metadata(dest_path: str, repo: str, model_root: str) -> None:
# 2. Overlay HF-specific fields # 2. Overlay HF-specific fields
metadata._unknown_fields["hf_url"] = hf_url metadata._unknown_fields["hf_url"] = hf_url
metadata._unknown_fields["source_url"] = hf_url
metadata._unknown_fields["source_platform"] = "huggingface"
metadata.from_civitai = False # HF models are not from CivitAI metadata.from_civitai = False # HF models are not from CivitAI
# 3. Save metadata atomically # 3. Save metadata atomically
@@ -189,27 +196,72 @@ async def _add_to_scanner_cache(dest_path: str, metadata: dict[str, Any]) -> Non
class HfHandler: class HfHandler:
"""Handle Hugging Face model browsing and download.""" """Handle Hugging Face model browsing and download."""
async def get_model_sources(self, request: web.Request) -> web.Response:
"""List the external model sites the UI can link a model to.
Used by the "Link Model" dialog to validate URLs client-side and to
explain which sites support AI metadata enrichment.
"""
return web.json_response([
{
"platform": source.platform,
"label": source.label,
"supports_enrichment": source.supports_enrichment,
"supports_download": source.supports_download,
"example_url": source.canonical_url(
"user/repo" if source.platform != "tensorart" else "827823520299086029"
),
}
for source in list_sources()
])
async def set_hf_url(self, request: web.Request) -> web.Response: async def set_hf_url(self, request: web.Request) -> web.Response:
"""Link a model file to its page on an external model site.
Accepts ``source_url`` (preferred) or the legacy ``hf_url`` /
``url`` payload key. Hugging Face, ModelScope, and TensorArt URLs
are recognised; the platform is stored alongside the canonical URL.
TensorArt models can be linked and browsed, but not AI-enriched.
"""
try: try:
payload: dict[str, Any] = await request.json() payload: dict[str, Any] = await request.json()
except json.JSONDecodeError: except json.JSONDecodeError:
return web.json_response({"success": False, "error": "Invalid JSON"}, status=400) return web.json_response({"success": False, "error": "Invalid JSON"}, status=400)
file_path = (payload.get("file_path") or "").strip() file_path = (payload.get("file_path") or "").strip()
hf_url = (payload.get("hf_url") or "").strip() raw_url = (
payload.get("source_url")
if not file_path or not hf_url: or payload.get("hf_url")
return web.json_response( or payload.get("url")
{"success": False, "error": "Missing required fields: 'file_path' and 'hf_url'"}, or ""
status=400,
) )
source_url = raw_url.strip() if isinstance(raw_url, str) else ""
m = re.match(r"^https?://huggingface\.co/([^/]+/[^/]+)/?$", hf_url) if not file_path or not source_url:
if not m:
return web.json_response( return web.json_response(
{ {
"success": False, "success": False,
"error": "Invalid HuggingFace URL. Expected format: https://huggingface.co/user/repo", "error": "Missing required fields: 'file_path' and 'source_url'",
},
status=400,
)
ref = detect_source(source_url, strict=True)
if ref is None:
return web.json_response(
{
"success": False,
"error": (
"Unsupported model URL. Supported formats: "
+ ", ".join(
f"{s.label} ({s.canonical_url('user/repo')})"
if s.platform != "tensorart"
else f"{s.label} (https://tensor.art/models/<id>)"
for s in list_sources()
)
),
}, },
status=400, status=400,
) )
@@ -225,37 +277,61 @@ class HfHandler:
return web.json_response( return web.json_response(
{ {
"success": False, "success": False,
"error": "File is not within any configured model directory. Cannot link to HuggingFace.", "error": "File is not within any configured model directory. Cannot link to a model source.",
}, },
status=400, status=400,
) )
try: try:
existing = await MetadataManager.load_metadata_payload(file_path) existing = await MetadataManager.load_metadata_payload(file_path)
if existing.get("hf_url") == hf_url:
already_linked = (
(existing.get("source_url") or "").strip() == ref.url
and (existing.get("source_platform") or "").strip().lower()
== ref.platform
) or (
not existing.get("source_url")
and ref.platform == "huggingface"
and (existing.get("hf_url") or "").strip() == ref.url
)
if already_linked:
return web.json_response({ return web.json_response({
"success": True, "success": True,
"message": "hf_url already set", "message": "source_url already set",
"hf_url": hf_url, "source_url": ref.url,
"source_platform": ref.platform,
"hf_url": ref.url if ref.platform == "huggingface" else "",
}) })
existing["hf_url"] = hf_url existing["source_url"] = ref.url
existing["source_platform"] = ref.platform
if ref.platform == "huggingface":
existing["hf_url"] = ref.url
else:
existing.pop("hf_url", None)
normalize_metadata_source(existing)
# NOTE: deliberately do NOT touch `from_civitai` here. It records # NOTE: deliberately do NOT touch `from_civitai` here. It records
# where the metadata came from, and the UI must show the CivitAI # where the metadata came from, and the UI must show the CivitAI
# link whenever CivitAI data is present — linking HuggingFace must # link whenever CivitAI data is present — linking an external
# not hide it (#1094). HF provenance is tracked via `hf_url`. # source must not hide it (#1094). Source provenance is tracked
# via `source_platform` / `source_url`.
await MetadataManager.save_metadata(file_path, existing) await MetadataManager.save_metadata(file_path, existing)
await _add_to_scanner_cache(file_path, existing) await _add_to_scanner_cache(file_path, existing)
logger.info("Set hf_url=%s for %s", hf_url, file_path) logger.info(
"Linked %s to %s source (%s)", file_path, ref.platform, ref.url
)
return web.json_response({ return web.json_response({
"success": True, "success": True,
"message": f"hf_url set to {hf_url}", "message": f"Linked to {ref.url}",
"hf_url": hf_url, "source_url": ref.url,
"source_platform": ref.platform,
"hf_url": existing.get("hf_url", ""),
}) })
except Exception as exc: except Exception as exc:
logger.error("Failed to set hf_url for %s: %s", file_path, exc) logger.error("Failed to link %s to a model source: %s", file_path, exc)
return web.json_response( return web.json_response(
{"success": False, "error": str(exc)}, {"success": False, "error": str(exc)},
status=500, status=500,
+1
View File
@@ -4079,6 +4079,7 @@ class MiscHandlerSet:
"get_hf_repo_files": self.hf_handler.get_hf_repo_files, "get_hf_repo_files": self.hf_handler.get_hf_repo_files,
"download_hf_model": self.hf_handler.download_hf_model, "download_hf_model": self.hf_handler.download_hf_model,
"set_hf_url": self.hf_handler.set_hf_url, "set_hf_url": self.hf_handler.set_hf_url,
"get_model_sources": self.hf_handler.get_model_sources,
# Agent skill handlers # Agent skill handlers
"get_agent_skills": self.agent_handler.get_agent_skills, "get_agent_skills": self.agent_handler.get_agent_skills,
"execute_agent_skill": self.agent_handler.execute_agent_skill, "execute_agent_skill": self.agent_handler.execute_agent_skill,
+4
View File
@@ -113,6 +113,10 @@ MISC_ROUTE_DEFINITIONS: tuple[RouteDefinition, ...] = (
RouteDefinition( RouteDefinition(
"POST", "/api/lm/set-hf-url", "set_hf_url" "POST", "/api/lm/set-hf-url", "set_hf_url"
), ),
# Supported external model sites (Hugging Face / ModelScope / TensorArt)
RouteDefinition(
"GET", "/api/lm/model-sources", "get_model_sources"
),
# Agent skill endpoints # Agent skill endpoints
RouteDefinition( RouteDefinition(
"GET", "/api/lm/agent/skills", "get_agent_skills" "GET", "/api/lm/agent/skills", "get_agent_skills"
+65 -28
View File
@@ -19,16 +19,18 @@ from __future__ import annotations
import asyncio import asyncio
import json import json
import logging import logging
import os
import re import re
from dataclasses import dataclass, field from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional from typing import Any, Dict, List, Optional
import aiohttp
import os
from ...config import config from ...config import config
from ..llm_service import LLMService from ..llm_service import LLMService
from ..model_sources import (
get_source,
resolve_source_ref,
source_label,
)
from ..websocket_manager import ws_manager from ..websocket_manager import ws_manager
from .post_processor import PostProcessor from .post_processor import PostProcessor
from .skill_registry import SkillRegistry from .skill_registry import SkillRegistry
@@ -267,11 +269,14 @@ class AgentService:
from ...metadata_ops import read_metadata from ...metadata_ops import read_metadata
metadata = await read_metadata(model_path) metadata = await read_metadata(model_path)
# Fast-fail: enrich_hf_metadata requires hf_url to have HF README context # Fast-fail: enrich_hf_metadata needs an external model source
if skill_name == "enrich_hf_metadata" and not metadata.get("hf_url", ""): # that exposes an accessible model card.
if skill_name == "enrich_hf_metadata":
skip_reason = self._enrichment_skip_reason(metadata)
if skip_reason:
logger.info( logger.info(
"[%s] SKIP %sno hf_url in metadata", "[%s] SKIP %s%s",
skill_name, model_filename, skill_name, model_filename, skip_reason,
) )
skipped_count += 1 skipped_count += 1
skip_model = True skip_model = True
@@ -358,6 +363,28 @@ class AgentService:
# Base model grouping (keeps the prompt compact) # Base model grouping (keeps the prompt compact)
# ------------------------------------------------------------------ # ------------------------------------------------------------------
@staticmethod
def _enrichment_skip_reason(metadata: Dict[str, Any]) -> str:
"""Return why ``enrich_hf_metadata`` cannot run, or ``""`` if it can.
Distinguishes the three cases the user can act on: no source linked,
a source we don't know, and a known source whose model card is not
reachable from the backend (TensorArt).
"""
ref = resolve_source_ref(metadata)
if ref is None:
return "no model source linked (source_url missing)"
source = get_source(ref.platform)
if source is None:
return f"unsupported model source platform '{ref.platform}'"
if not source.supports_enrichment:
return (
f"{source.label} does not expose a model card to the backend; "
"AI metadata enrichment is not available for this source"
)
return ""
@staticmethod @staticmethod
def _format_base_models(models: List[str]) -> str: def _format_base_models(models: List[str]) -> str:
"""Format the base model list as a flat, one-per-line list. """Format the base model list as a flat, one-per-line list.
@@ -388,6 +415,14 @@ class AgentService:
context: Dict[str, Any] = { context: Dict[str, Any] = {
"model_path": model_path, "model_path": model_path,
"model_basename": "", "model_basename": "",
# Canonical external-source variables
"source_url": "",
"source_id": "",
"source_platform": "",
"source_label": "",
"asset_base_url": "",
# Legacy Hugging Face aliases (kept so older prompt templates and
# third-party skills keep rendering)
"hf_url": "", "hf_url": "",
"repo": "", "repo": "",
"readme_content": "", "readme_content": "",
@@ -411,12 +446,20 @@ class AgentService:
"size": metadata.get("size", 0), "size": metadata.get("size", 0),
} }
hf_url = metadata.get("hf_url", "") ref = resolve_source_ref(metadata)
context["hf_url"] = hf_url if ref is not None:
repo = self._extract_repo_from_url(hf_url) if hf_url else "" context["source_url"] = ref.url
context["repo"] = repo or "" context["source_id"] = ref.source_id
if repo: context["source_platform"] = ref.platform
readme = await self._fetch_readme(repo) context["source_label"] = source_label(ref.platform, ref.platform)
if ref.platform == "huggingface":
context["hf_url"] = ref.url
context["repo"] = ref.source_id
source = get_source(ref.platform) if ref is not None else None
if ref is not None and source is not None and source.supports_enrichment:
context["asset_base_url"] = source.asset_base_url(ref.source_id)
readme = await source.fetch_model_card(ref.source_id)
# Trim README to the section relevant to this model file # Trim README to the section relevant to this model file
# (collection repos often have multiple models in one README). # (collection repos often have multiple models in one README).
if readme and raw_basename: if readme and raw_basename:
@@ -458,20 +501,14 @@ class AgentService:
@staticmethod @staticmethod
async def _fetch_readme(repo: str) -> str: async def _fetch_readme(repo: str) -> str:
"""Fetch README.md from HuggingFace (tries ``main``, then ``master``).""" """Fetch a Hugging Face README (tries ``main``, then ``master``).
async with aiohttp.ClientSession(
headers={"User-Agent": "ComfyUI-LoRA-Manager/1.0"}, Kept for backward compatibility; new code should go through the
timeout=aiohttp.ClientTimeout(total=30), model-source registry so every supported site works.
) as session: """
for branch in ("main", "master"): from ..model_sources import HuggingFaceSource
url = f"https://huggingface.co/{repo}/raw/{branch}/README.md"
try: return await HuggingFaceSource().fetch_model_card(repo)
async with session.get(url) as resp:
if resp.status == 200:
return await resp.text()
except Exception as exc:
logger.debug("Failed to fetch README from %s: %s", url, exc)
return ""
async def _emit_progress( async def _emit_progress(
self, self,
+31 -19
View File
@@ -78,6 +78,7 @@ class PostProcessor:
download_preview, download_preview,
refresh_cache, refresh_cache,
) )
from ..model_sources import get_source, has_external_source, resolve_source_ref
from .skills.enrich_hf_metadata.readme_processor import ( from .skills.enrich_hf_metadata.readme_processor import (
convert_readme_to_html, convert_readme_to_html,
extract_gallery_images, extract_gallery_images,
@@ -85,17 +86,25 @@ class PostProcessor:
extract_relevant_section, extract_relevant_section,
extract_simple_markdown_images, extract_simple_markdown_images,
extract_html_img_tags, extract_html_img_tags,
extract_repo_from_hf_url,
) )
updated_fields: List[str] = [] updated_fields: List[str] = []
preview_downloaded = False preview_downloaded = False
# -- Determine whether this is an HF-sourced model ----------------- # -- Determine whether this is an externally-sourced model ---------
# Key off `hf_url` directly: `from_civitai` records provenance and can # Key off the source fields directly: `from_civitai` records provenance
# be true for a model that is also linked to HuggingFace (both sources # and can be true for a model that is also linked to an external site
# coexist, see #1094), so it must not gate HF enrichment. # (both sources coexist, see #1094), so it must not gate enrichment.
is_hf_model = bool(metadata.get("hf_url", "")) is_source_model = has_external_source(metadata)
source_ref = resolve_source_ref(metadata)
source = get_source(source_ref.platform) if source_ref else None
source_id = source_ref.source_id if source_ref else ""
asset_base_url = (
source.asset_base_url(source_id)
if source is not None and source_id
else None
)
# -- Collect updates ----------------------------------------------- # -- Collect updates -----------------------------------------------
updates: Dict[str, Any] = {} updates: Dict[str, Any] = {}
@@ -103,7 +112,7 @@ class PostProcessor:
# base_model # base_model
new_base = (llm_output.get("base_model") or "").strip() new_base = (llm_output.get("base_model") or "").strip()
current_base = metadata.get("base_model", "") or "" current_base = metadata.get("base_model", "") or ""
if new_base and self._should_overwrite(current_base, is_hf_model): if new_base and self._should_overwrite(current_base, is_source_model):
updates["base_model"] = new_base updates["base_model"] = new_base
# trigger words → civitai.trainedWords # trigger words → civitai.trainedWords
@@ -115,7 +124,7 @@ class PostProcessor:
trigger_words_empty = not cleaned trigger_words_empty = not cleaned
current_civitai = metadata.get("civitai") or {} current_civitai = metadata.get("civitai") or {}
current_triggers = current_civitai.get("trainedWords") or [] current_triggers = current_civitai.get("trainedWords") or []
if self._should_overwrite_list(current_triggers, is_hf_model): if self._should_overwrite_list(current_triggers, is_source_model):
trig_civitai = dict(current_civitai) trig_civitai = dict(current_civitai)
if "civitai" in updates and isinstance(updates["civitai"], dict): if "civitai" in updates and isinstance(updates["civitai"], dict):
trig_civitai.update(updates["civitai"]) trig_civitai.update(updates["civitai"])
@@ -123,14 +132,14 @@ class PostProcessor:
updates["civitai"] = trig_civitai updates["civitai"] = trig_civitai
# modelDescription — from raw README content (converted to HTML) # modelDescription — from raw README content (converted to HTML)
if readme_content and is_hf_model: if readme_content and is_source_model:
converted = convert_readme_to_html(readme_content) converted = convert_readme_to_html(readme_content)
if converted: if converted:
updates["modelDescription"] = converted updates["modelDescription"] = converted
# short_description → civitai.description (for "About this version") # short_description → civitai.description (for "About this version")
short_desc = (llm_output.get("short_description") or "").strip() short_desc = (llm_output.get("short_description") or "").strip()
if short_desc and is_hf_model: if short_desc and is_source_model:
current_civitai = metadata.get("civitai") or {} current_civitai = metadata.get("civitai") or {}
desc_civitai = dict(current_civitai) desc_civitai = dict(current_civitai)
if "civitai" in updates and isinstance(updates["civitai"], dict): if "civitai" in updates and isinstance(updates["civitai"], dict):
@@ -141,9 +150,8 @@ class PostProcessor:
# gallery images → civitai.images (from YAML frontmatter widget entries # gallery images → civitai.images (from YAML frontmatter widget entries
# and Sample Gallery markdown tables in the README body) # and Sample Gallery markdown tables in the README body)
gallery_images: List[Dict[str, Any]] = [] gallery_images: List[Dict[str, Any]] = []
if readme_content and is_hf_model: if readme_content and is_source_model:
hf_url = metadata.get("hf_url", "") or "" repo = source_id
repo = extract_repo_from_hf_url(hf_url)
if repo: if repo:
rec_w = llm_output.get("recommended_width") or 0 rec_w = llm_output.get("recommended_width") or 0
rec_h = llm_output.get("recommended_height") or 0 rec_h = llm_output.get("recommended_height") or 0
@@ -152,6 +160,7 @@ class PostProcessor:
gallery = extract_gallery_images( gallery = extract_gallery_images(
readme_content, repo, readme_content, repo,
default_width=rec_w, default_height=rec_h, default_width=rec_w, default_height=rec_h,
base_url=asset_base_url,
) )
# 2. Sample Gallery table images (markdown body), deduplicated # 2. Sample Gallery table images (markdown body), deduplicated
@@ -160,6 +169,7 @@ class PostProcessor:
readme_content, repo, readme_content, repo,
existing_urls=existing_urls, existing_urls=existing_urls,
default_width=rec_w, default_height=rec_h, default_width=rec_w, default_height=rec_h,
base_url=asset_base_url,
) )
existing_urls.update(img["url"] for img in table_images if img.get("url")) existing_urls.update(img["url"] for img in table_images if img.get("url"))
@@ -168,6 +178,7 @@ class PostProcessor:
readme_content, repo, readme_content, repo,
existing_urls=existing_urls, existing_urls=existing_urls,
default_width=rec_w, default_height=rec_h, default_width=rec_w, default_height=rec_h,
base_url=asset_base_url,
) )
existing_urls.update(img["url"] for img in simple_images if img.get("url")) existing_urls.update(img["url"] for img in simple_images if img.get("url"))
@@ -176,6 +187,7 @@ class PostProcessor:
readme_content, repo, readme_content, repo,
existing_urls=existing_urls, existing_urls=existing_urls,
default_width=rec_w, default_height=rec_h, default_width=rec_w, default_height=rec_h,
base_url=asset_base_url,
) )
all_images = gallery + table_images + simple_images + html_images all_images = gallery + table_images + simple_images + html_images
@@ -193,7 +205,7 @@ class PostProcessor:
if isinstance(new_tags, list) and new_tags: if isinstance(new_tags, list) and new_tags:
existing_tags = metadata.get("tags") or [] existing_tags = metadata.get("tags") or []
merged = self._merge_tags(existing_tags, new_tags) merged = self._merge_tags(existing_tags, new_tags)
if len(merged) > len(existing_tags) or is_hf_model: if len(merged) > len(existing_tags) or is_source_model:
updates["tags"] = merged updates["tags"] = merged
# metadata_source & llm_enriched_at (always set) # metadata_source & llm_enriched_at (always set)
@@ -222,7 +234,7 @@ class PostProcessor:
# README, find the first gallery image from the *model-specific # README, find the first gallery image from the *model-specific
# section* of the README (not the repo-wide first image, which # section* of the README (not the repo-wide first image, which
# belongs to a different model in collection repos). # belongs to a different model in collection repos).
if not preview_remote_url and readme_content and is_hf_model: if not preview_remote_url and readme_content and is_source_model:
model_basename = os.path.splitext(os.path.basename(model_path))[0] model_basename = os.path.splitext(os.path.basename(model_path))[0]
relevant_section = extract_relevant_section( relevant_section = extract_relevant_section(
readme_content, model_basename, readme_content, model_basename,
@@ -279,16 +291,16 @@ class PostProcessor:
# ------------------------------------------------------------------ # ------------------------------------------------------------------
@staticmethod @staticmethod
def _should_overwrite(current_value: str, is_hf_model: bool) -> bool: def _should_overwrite(current_value: str, is_source_model: bool) -> bool:
"""Return ``True`` when a scalar field should be overwritten.""" """Return ``True`` when a scalar field should be overwritten."""
return is_hf_model or not current_value or current_value.lower() in ( return is_source_model or not current_value or current_value.lower() in (
"", "unknown", "", "unknown",
) )
@staticmethod @staticmethod
def _should_overwrite_list(current_list: List[str], is_hf_model: bool) -> bool: def _should_overwrite_list(current_list: List[str], is_source_model: bool) -> bool:
"""Return ``True`` when a list field should be overwritten.""" """Return ``True`` when a list field should be overwritten."""
return is_hf_model or not current_list return is_source_model or not current_list
@staticmethod @staticmethod
def _merge_tags(existing: List[str], new: List[str]) -> List[str]: def _merge_tags(existing: List[str], new: List[str]) -> List[str]:
@@ -1,20 +1,23 @@
--- ---
name: enrich_hf_metadata name: enrich_hf_metadata
title: "Enrich Metadata from HuggingFace" title: "Enrich Metadata from Model Card"
description: > description: >
Parse the HuggingFace model card via LLM to extract description, trigger Parse the model card (README) from HuggingFace, ModelScope, or any other
words, base model, tags, and preview image URL. supported model site via LLM to extract description, trigger words, base
model, tags, and preview image URL.
llm_required: true llm_required: true
--- ---
You are an expert assistant for AI image generation models. Your task is to extract structured metadata from a HuggingFace model card (README.md). You are an expert assistant for AI image generation models. Your task is to extract structured metadata from a model card (README).
## Model Information ## Model Information
- **Repository**: {{hf_url}} - **Source site**: {{source_label}} ({{source_platform}})
- **Model page**: {{source_url}}
- **Model file path**: {{model_path}} - **Model file path**: {{model_path}}
- **Model filename**: {{model_basename}} - **Model filename**: {{model_basename}}
- **Repository ID**: {{repo}} - **Repository ID**: {{source_id}}
- **Repository raw-file base URL**: {{asset_base_url}}
## Current Metadata (may be incomplete) ## Current Metadata (may be incomplete)
@@ -39,7 +42,7 @@ name listed — do not invent aliases or modify variant suffixes.
{{base_models}} {{base_models}}
## HuggingFace README Content ## Model Card Content
``` ```
{{readme_content}} {{readme_content}}
@@ -92,7 +95,7 @@ The URL of the most suitable preview image from the README. Look for:
- The YAML frontmatter `widget:` section (which often has `output.url` fields) - The YAML frontmatter `widget:` section (which often has `output.url` fields)
- In collection repos: the sample images listed **under the section** for this specific model version - In collection repos: the sample images listed **under the section** for this specific model version
- Generic `![alt](url)` in the body - Generic `![alt](url)` in the body
Choose the first image that appears to be a generation example (not a logo or diagram). Construct the absolute URL as `https://huggingface.co/{{repo}}/resolve/main/{filename}`. If no suitable image is found, return an empty string. Choose the first image that appears to be a generation example (not a logo or diagram). Construct the absolute URL from the repository raw-file base URL (`{{asset_base_url}}`) plus the relative path. If no suitable image is found, return an empty string.
### notes ### notes
A plain-text summary of the model card's key practical usage information. Combine trigger words, style modifiers, recommended parameters (steps, CFG, resolution, sampler), and any setup tips into a readable paragraph. For collection repos, focus on the **specific model version** matching `{{model_basename}}`. Return empty string if the README has no useful usage info. A plain-text summary of the model card's key practical usage information. Combine trigger words, style modifiers, recommended parameters (steps, CFG, resolution, sampler), and any setup tips into a readable paragraph. For collection repos, focus on the **specific model version** matching `{{model_basename}}`. Return empty string if the README has no useful usage info.
@@ -121,7 +124,7 @@ Your confidence level in the extracted data:
## Important: Handling Collection Repos (multiple model files) ## Important: Handling Collection Repos (multiple model files)
Many HuggingFace repos contain **multiple model files** in a single repository Many model repositories contain **multiple model files** in a single repository
(e.g. a "LoRA collection" with different styles/characters in separate files). (e.g. a "LoRA collection" with different styles/characters in separate files).
The model file currently being enriched is: **`{{model_basename}}`** The model file currently being enriched is: **`{{model_basename}}`**
@@ -1,8 +1,15 @@
"""HF README processing for the ``enrich_hf_metadata`` skill. """Model card (README) processing for the ``enrich_hf_metadata`` skill.
Provides README cleaning for LLM injection, gallery/image extraction from Provides README cleaning for LLM injection, gallery/image extraction from
multiple formats (YAML widget, markdown, HTML ``<img>``, gallery tables), multiple formats (YAML widget, markdown, HTML ``<img>``, gallery tables),
and section-based README trimming for collection repos. and section-based README trimming for collection repos.
The extractors default to Hugging Face asset URLs, but every one of them
accepts an explicit ``base_url`` so the same parsing works for any model
source (ModelScope, ...). See :mod:`py.services.model_sources`.
This module deliberately has no package-relative imports: it is also loaded
standalone by the README-processing test harness.
""" """
from __future__ import annotations from __future__ import annotations
@@ -15,12 +22,25 @@ from typing import Any, List, Tuple
_REPO_URL_PATTERN = re.compile(r"https?://huggingface\.co/([^/]+/[^/]+)") _REPO_URL_PATTERN = re.compile(r"https?://huggingface\.co/([^/]+/[^/]+)")
def resolve_asset_base_url(repo: str, base_url: str | None = None) -> str:
"""Return the base URL used to resolve repository-relative assets.
Falls back to the historical Hugging Face layout when *base_url* is not
supplied, so existing callers keep their behaviour.
"""
if base_url:
return base_url.rstrip("/")
return f"https://huggingface.co/{repo}/resolve/main"
def extract_simple_markdown_images( def extract_simple_markdown_images(
markdown_text: str, markdown_text: str,
repo: str, repo: str,
existing_urls: set[str] | None = None, existing_urls: set[str] | None = None,
default_width: int = 512, default_width: int = 512,
default_height: int = 512, default_height: int = 512,
base_url: str | None = None,
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
"""Extract standalone markdown images from the README body. """Extract standalone markdown images from the README body.
@@ -32,10 +52,10 @@ def extract_simple_markdown_images(
Returns a list of dicts in the same ``civitai.images`` format as Returns a list of dicts in the same ``civitai.images`` format as
:func:`extract_gallery_images`. :func:`extract_gallery_images`.
""" """
if not markdown_text or not repo: if not markdown_text or not (repo or base_url):
return [] return []
base_url = f"https://huggingface.co/{repo}/resolve/main" base_url = resolve_asset_base_url(repo, base_url)
images: list[dict[str, Any]] = [] images: list[dict[str, Any]] = []
seen_urls: set[str] = set(existing_urls) if existing_urls else set() seen_urls: set[str] = set(existing_urls) if existing_urls else set()
@@ -89,20 +109,21 @@ def extract_html_img_tags(
existing_urls: set[str] | None = None, existing_urls: set[str] | None = None,
default_width: int = 512, default_width: int = 512,
default_height: int = 512, default_height: int = 512,
base_url: str | None = None,
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
"""Extract image URLs from HTML ``<img src=\"...\">`` tags in the README. """Extract image URLs from HTML ``<img src=\"...\">`` tags in the README.
Many HF collection repos (e.g. ``deadman44/Z-Image_LoRA``) use raw HTML Many HF collection repos (e.g. ``deadman44/Z-Image_LoRA``) use raw HTML
``<img>`` tags exclusively for their sample images, with no markdown ``<img>`` tags exclusively for their sample images, with no markdown
``![]()`` equivalents. This function finds those tags and constructs ``![]()`` equivalents. This function finds those tags and constructs
resolvable HF URLs. resolvable URLs.
Returns a list of dicts in the ``civitai.images`` format. Returns a list of dicts in the ``civitai.images`` format.
""" """
if not markdown_text or not repo: if not markdown_text or not (repo or base_url):
return [] return []
base_url = f"https://huggingface.co/{repo}/resolve/main" base_url = resolve_asset_base_url(repo, base_url)
images: list[dict[str, Any]] = [] images: list[dict[str, Any]] = []
seen_urls: set[str] = set(existing_urls) if existing_urls else set() seen_urls: set[str] = set(existing_urls) if existing_urls else set()
@@ -166,7 +187,7 @@ def extract_html_img_tags(
def extract_repo_from_hf_url(hf_url: str) -> str: def extract_repo_from_hf_url(hf_url: str) -> str:
"""Extract ``user/repo`` from a HuggingFace URL.""" """Extract ``user/repo`` from a HuggingFace URL."""
m = _REPO_URL_PATTERN.match(hf_url) m = _REPO_URL_PATTERN.match(hf_url or "")
return m.group(1) if m else "" return m.group(1) if m else ""
@@ -175,21 +196,23 @@ def extract_gallery_images(
repo: str, repo: str,
default_width: int = 512, default_width: int = 512,
default_height: int = 512, default_height: int = 512,
base_url: str | None = None,
) -> List[dict[str, Any]]: ) -> List[dict[str, Any]]:
"""Extract widget/gallery images from the YAML frontmatter of a HF README. """Extract widget/gallery images from the YAML frontmatter of a README.
Args: Args:
markdown_text: Raw README content. markdown_text: Raw README content.
repo: HF repo identifier (``user/repo``). repo: Repository identifier (``user/repo``).
default_width: Fallback width when the README provides no dimension. default_width: Fallback width when the README provides no dimension.
default_height: Fallback height when the README provides no dimension. default_height: Fallback height when the README provides no dimension.
base_url: Overrides the asset base URL (defaults to Hugging Face).
Returns a list of dicts compatible with the ``civitai.images`` metadata Returns a list of dicts compatible with the ``civitai.images`` metadata
format, each containing ``url`` (absolute HF URL), ``meta.prompt``, format, each containing ``url`` (absolute), ``meta.prompt``,
``width``, ``height``, and ``type``. Returns an empty list when no ``width``, ``height``, and ``type``. Returns an empty list when no
widget entries are found or when *repo* is empty. widget entries are found or when *repo* is empty.
""" """
if not markdown_text or not repo: if not markdown_text or not (repo or base_url):
return [] return []
frontmatter = _extract_frontmatter(markdown_text) frontmatter = _extract_frontmatter(markdown_text)
@@ -197,7 +220,7 @@ def extract_gallery_images(
return [] return []
images: List[dict[str, Any]] = [] images: List[dict[str, Any]] = []
base_url = f"https://huggingface.co/{repo}/resolve/main" base_url = resolve_asset_base_url(repo, base_url)
w = default_width or 512 w = default_width or 512
h = default_height or 512 h = default_height or 512
@@ -279,10 +302,11 @@ def extract_gallery_table_images(
existing_urls: set[str] | None = None, existing_urls: set[str] | None = None,
default_width: int = 512, default_width: int = 512,
default_height: int = 512, default_height: int = 512,
base_url: str | None = None,
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
"""Extract images from ``| Preview | Prompt |`` markdown gallery tables. """Extract images from ``| Preview | Prompt |`` markdown gallery tables.
Many HF READMEs include a sample-gallery table in the body (outside Many READMEs include a sample-gallery table in the body (outside
the YAML frontmatter) that shows generation examples with their the YAML frontmatter) that shows generation examples with their
prompts. This function parses those tables and merges results with prompts. This function parses those tables and merges results with
the widget-sourced images from :func:`extract_gallery_images`. the widget-sourced images from :func:`extract_gallery_images`.
@@ -291,10 +315,10 @@ def extract_gallery_table_images(
:func:`extract_gallery_images`. Already-seen URLs (from *existing_urls*) :func:`extract_gallery_images`. Already-seen URLs (from *existing_urls*)
are skipped. are skipped.
""" """
if not markdown_text or not repo: if not markdown_text or not (repo or base_url):
return [] return []
base_url = f"https://huggingface.co/{repo}/resolve/main" base_url = resolve_asset_base_url(repo, base_url)
images: list[dict[str, Any]] = [] images: list[dict[str, Any]] = []
seen_urls: set[str] = set(existing_urls) if existing_urls else set() seen_urls: set[str] = set(existing_urls) if existing_urls else set()
lines = markdown_text.split("\n") lines = markdown_text.split("\n")
+16 -12
View File
@@ -21,6 +21,7 @@ from .model_query import (
resolve_sub_type, resolve_sub_type,
) )
from .settings_manager import get_settings_manager from .settings_manager import get_settings_manager
from .model_sources import source_group_key
from ..utils.civitai_utils import build_civitai_model_page_url from ..utils.civitai_utils import build_civitai_model_page_url
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -742,29 +743,32 @@ class BaseModelService(ABC):
@staticmethod @staticmethod
def _extract_hf_group_key(item: Dict[str, Any]) -> Optional[str]: def _extract_hf_group_key(item: Dict[str, Any]) -> Optional[str]:
"""Extract `hf:{owner}/{repo}` from item's ``hf_url``, or None.""" """Extract `hf:{owner}/{repo}` from item's ``hf_url``, or None."""
hf_url = item.get("hf_url") if isinstance(item, dict) else None key = BaseModelService._extract_source_group_key(item)
if not hf_url or not isinstance(hf_url, str): return key if key and key.startswith("hf:") else None
return None
m = re.match( @staticmethod
r"https?://huggingface\.co/([^/]+/[^/]+)", hf_url.strip() def _extract_source_group_key(item: Dict[str, Any]) -> Optional[str]:
) """Return the external-source group key for *item*, or None.
if not m:
return None Hugging Face keeps the historical ``hf:{owner}/{repo}`` shape; other
return f"hf:{m.group(1)}" platforms use their own short prefix (``ms:`` / ``ta:``).
"""
return source_group_key(item)
@staticmethod @staticmethod
def _extract_group_key(item: Dict[str, Any]) -> Union[int, str, None]: def _extract_group_key(item: Dict[str, Any]) -> Union[int, str, None]:
"""Return the group identity key: CivitAI modelId (int) or HF repo (str). """Return the group identity key.
Preference order: Preference order:
1. CivitAI ``modelId`` (int) 1. CivitAI ``modelId`` (int)
2. HF repo identity ``hf:{owner}/{repo}`` (str) 2. External model source identity, e.g. ``hf:{owner}/{repo}``,
``ms:{owner}/{repo}``, ``ta:{model_id}`` (str)
3. ``None`` (no known grouping source) 3. ``None`` (no known grouping source)
""" """
mid = BaseModelService._extract_model_id(item) mid = BaseModelService._extract_model_id(item)
if mid is not None: if mid is not None:
return mid return mid
return BaseModelService._extract_hf_group_key(item) return BaseModelService._extract_source_group_key(item)
@staticmethod @staticmethod
def _extract_model_id(item: Dict[str, Any]) -> Optional[int]: def _extract_model_id(item: Dict[str, Any]) -> Optional[int]:
+2
View File
@@ -67,6 +67,8 @@ class CheckpointService(BaseModelService):
"civitai": self.filter_civitai_data(model_data.get("civitai", {}), minimal=True), "civitai": self.filter_civitai_data(model_data.get("civitai", {}), minimal=True),
"auto_tags": model_data.get("auto_tags") or extract_auto_tags(model_data), "auto_tags": model_data.get("auto_tags") or extract_auto_tags(model_data),
"version_count": model_data.get("version_count"), "version_count": model_data.get("version_count"),
"source_platform": model_data.get("source_platform", ""),
"source_url": model_data.get("source_url", ""),
"hf_url": model_data.get("hf_url", ""), "hf_url": model_data.get("hf_url", ""),
} }
+2
View File
@@ -67,6 +67,8 @@ class EmbeddingService(BaseModelService):
"civitai": self.filter_civitai_data(model_data.get("civitai", {}), minimal=True), "civitai": self.filter_civitai_data(model_data.get("civitai", {}), minimal=True),
"auto_tags": model_data.get("auto_tags") or extract_auto_tags(model_data), "auto_tags": model_data.get("auto_tags") or extract_auto_tags(model_data),
"version_count": model_data.get("version_count"), "version_count": model_data.get("version_count"),
"source_platform": model_data.get("source_platform", ""),
"source_url": model_data.get("source_url", ""),
"hf_url": model_data.get("hf_url", ""), "hf_url": model_data.get("hf_url", ""),
} }
+2
View File
@@ -79,6 +79,8 @@ class LoraService(BaseModelService):
), ),
"auto_tags": model_data.get("auto_tags") or extract_auto_tags(model_data), "auto_tags": model_data.get("auto_tags") or extract_auto_tags(model_data),
"version_count": model_data.get("version_count"), "version_count": model_data.get("version_count"),
"source_platform": model_data.get("source_platform", ""),
"source_url": model_data.get("source_url", ""),
"hf_url": model_data.get("hf_url", ""), "hf_url": model_data.get("hf_url", ""),
} }
+4 -2
View File
@@ -14,6 +14,7 @@ from ..utils.model_utils import determine_base_model
from ..utils.models import autov3_from_civitai_files from ..utils.models import autov3_from_civitai_files
from .connectivity_guard import OFFLINE_FRIENDLY_MESSAGE, is_expected_offline_error from .connectivity_guard import OFFLINE_FRIENDLY_MESSAGE, is_expected_offline_error
from .errors import RateLimitError from .errors import RateLimitError
from .model_sources import has_external_source
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -222,9 +223,10 @@ class MetadataSyncService:
error_msg = "CivitAI model is deleted and no archive provider is available" error_msg = "CivitAI model is deleted and no archive provider is available"
return False, error_msg return False, error_msg
else: else:
is_hf_source = bool(model_data.get("hf_url")) is_hf_source = has_external_source(model_data)
if is_hf_source: if is_hf_source:
# HF-sourced model: only check CivitAI API directly. # External-source model (Hugging Face / ModelScope /
# TensorArt): only check CivitAI API directly.
# CivArchive is almost guaranteed to have no record, and # CivArchive is almost guaranteed to have no record, and
# hitting it wastes rate-limit budget. # hitting it wastes rate-limit budget.
# Use a distinct provider name ("civitai_api" not None) so # Use a distinct provider name ("civitai_api" not None) so
+7
View File
@@ -15,6 +15,7 @@ from ..utils.civitai_utils import resolve_license_info
from .model_cache import ModelCache from .model_cache import ModelCache
from .model_hash_index import ModelHashIndex from .model_hash_index import ModelHashIndex
from .model_lifecycle_service import delete_model_artifacts, _require_path_in_library_roots from .model_lifecycle_service import delete_model_artifacts, _require_path_in_library_roots
from .model_sources import normalize_metadata_source
from .service_registry import ServiceRegistry from .service_registry import ServiceRegistry
from .websocket_manager import ws_manager from .websocket_manager import ws_manager
from .persistent_model_cache import get_persistent_cache from .persistent_model_cache import get_persistent_cache
@@ -387,8 +388,14 @@ class ModelScanner:
'civitai': civitai_slim, 'civitai': civitai_slim,
'civitai_deleted': bool(get_value('civitai_deleted', False)), 'civitai_deleted': bool(get_value('civitai_deleted', False)),
'skip_metadata_refresh': bool(get_value('skip_metadata_refresh', False)), 'skip_metadata_refresh': bool(get_value('skip_metadata_refresh', False)),
# External model source (Hugging Face / ModelScope / TensorArt).
# `source_url` + `source_platform` are canonical; `hf_url` stays in
# sync as a legacy alias (normalised below).
'source_platform': get_value('source_platform', '') or '',
'source_url': get_value('source_url', '') or '',
'hf_url': get_value('hf_url', '') or '', 'hf_url': get_value('hf_url', '') or '',
} }
normalize_metadata_source(entry)
license_source: Dict[str, Any] = {} license_source: Dict[str, Any] = {}
if isinstance(civitai_full, Mapping): if isinstance(civitai_full, Mapping):
+61
View File
@@ -0,0 +1,61 @@
"""External model-source providers (Hugging Face, ModelScope, TensorArt).
This package is the single abstraction over "a site that hosts models and
a model card". See :mod:`py.services.model_sources.base` for the provider
protocol and :mod:`py.services.model_sources.registry` for the lookup and
metadata-normalisation helpers used across the codebase.
"""
from __future__ import annotations
from .base import (
GROUP_PREFIXES,
HTTP_TIMEOUT,
ModelSource,
SourceRef,
USER_AGENT,
clean_source_url,
fetch_text,
)
from .huggingface import HuggingFaceSource
from .modelscope import ModelScopeSource
from .registry import (
LEGACY_HF_URL_FIELD,
SOURCE_PLATFORM_FIELD,
SOURCE_URL_FIELD,
detect_source,
get_source,
get_source_platform,
has_external_source,
list_sources,
normalize_metadata_source,
resolve_source_ref,
source_group_key,
source_label,
)
from .tensorart import TensorArtSource
__all__ = [
"GROUP_PREFIXES",
"HTTP_TIMEOUT",
"LEGACY_HF_URL_FIELD",
"ModelSource",
"HuggingFaceSource",
"ModelScopeSource",
"SOURCE_PLATFORM_FIELD",
"SOURCE_URL_FIELD",
"SourceRef",
"TensorArtSource",
"USER_AGENT",
"clean_source_url",
"detect_source",
"fetch_text",
"get_source",
"get_source_platform",
"has_external_source",
"list_sources",
"normalize_metadata_source",
"resolve_source_ref",
"source_group_key",
"source_label",
]
+186
View File
@@ -0,0 +1,186 @@
"""Base types for the external model-source provider abstraction.
A *model source* is a third-party site that hosts model files and a model
card (README) describing them Hugging Face, ModelScope, TensorArt, and
whatever gets added later. Everything the rest of the codebase needs to
know about such a site is expressed by :class:`ModelSource`:
* how to recognise one of its URLs (:meth:`ModelSource.parse`)
* the canonical page URL for a source id (:meth:`ModelSource.canonical_url`)
* how to fetch the model card (:meth:`ModelSource.fetch_model_card`)
* how to turn repository-relative asset paths into absolute URLs
(:meth:`ModelSource.asset_base_url`)
* which capabilities the site actually supports
(``supports_enrichment`` / ``supports_download``)
Keeping this in one place means the agent pipeline, the scanners, and the
HTTP handlers never need site-specific branching.
"""
from __future__ import annotations
import logging
import re
from dataclasses import dataclass
from typing import Any, Optional
import aiohttp
logger = logging.getLogger(__name__)
#: Shared HTTP timeout for model-card fetches.
HTTP_TIMEOUT = 30
#: User agent used for all model-source HTTP requests.
USER_AGENT = "ComfyUI-LoRA-Manager/1.0"
#: Platform → short prefix used when building version-group keys.
#: ``huggingface`` keeps the historical ``hf:`` prefix for backward
#: compatibility with already-cached group keys.
GROUP_PREFIXES: dict[str, str] = {
"huggingface": "hf",
"modelscope": "ms",
"tensorart": "ta",
}
@dataclass(frozen=True)
class SourceRef:
"""A parsed reference to a model hosted on an external site."""
platform: str
"""Canonical platform id, e.g. ``"huggingface"``."""
source_id: str
"""Site-specific identity, e.g. ``"user/repo"`` or ``"827823520299086029"``."""
url: str
"""Canonical URL of the model page."""
async def fetch_text(url: str, *, timeout: int = HTTP_TIMEOUT) -> str:
"""Fetch *url* and return its body as text, or ``""`` on any failure.
Network problems are expected (offline installs, rate limits, dead
repos) and must never bubble up into the pipeline, so every error is
logged at debug level and normalised to an empty string.
"""
try:
async with aiohttp.ClientSession(
headers={"User-Agent": USER_AGENT},
timeout=aiohttp.ClientTimeout(total=timeout),
) as session:
async with session.get(url) as resp:
if resp.status == 200:
return await resp.text()
logger.debug("Fetch %s returned HTTP %s", url, resp.status)
except Exception as exc: # pragma: no cover - network dependent
logger.debug("Failed to fetch %s: %s", url, exc)
return ""
class ModelSource:
"""Description and I/O for one external model hosting site."""
#: Canonical platform id stored in metadata.
platform: str = ""
#: Human-readable name used in UI copy and prompts.
label: str = ""
#: Whether the agent skill can fetch a model card and run AI extraction.
supports_enrichment: bool = False
#: Whether models can be downloaded directly from this site.
supports_download: bool = False
#: Lenient pattern used to recognise URLs already stored in metadata.
#: Captures the site-specific source id in group ``id``.
url_pattern: re.Pattern[str] | None = None
#: Strict pattern used to validate user input. Must match the whole URL.
strict_url_pattern: re.Pattern[str] | None = None
# ------------------------------------------------------------------
# Parsing
# ------------------------------------------------------------------
def parse(self, url: str, *, strict: bool = False) -> Optional[str]:
"""Return the source id contained in *url*, or ``None``.
With ``strict=True`` the URL must match this site's canonical shape
exactly (used when validating what a user pasted); with
``strict=False`` sub-paths such as ``/resolve/main/file.bin`` are
tolerated (used when normalising already-stored values).
"""
if not url or not isinstance(url, str):
return None
candidate = url.strip()
if not candidate:
return None
pattern = self.strict_url_pattern if strict else self.url_pattern
if pattern is None:
return None
match = pattern.match(candidate)
return match.group("id") if match else None
def ref(self, url: str, *, strict: bool = False) -> Optional[SourceRef]:
"""Return a :class:`SourceRef` for *url*, or ``None`` if not ours."""
source_id = self.parse(url, strict=strict)
if not source_id:
return None
return SourceRef(
platform=self.platform,
source_id=source_id,
url=self.canonical_url(source_id),
)
# ------------------------------------------------------------------
# URLs and content
# ------------------------------------------------------------------
def canonical_url(self, source_id: str) -> str:
"""Return the canonical model-page URL for *source_id*."""
raise NotImplementedError
def asset_base_url(self, source_id: str, revision: str = "") -> str:
"""Base URL used to resolve repository-relative asset paths."""
return ""
def group_key(self, source_id: str) -> str:
"""Return the version-group key for *source_id*."""
prefix = GROUP_PREFIXES.get(self.platform, self.platform)
return f"{prefix}:{source_id}"
async def fetch_model_card(self, source_id: str) -> str:
"""Fetch the raw model card (README) markdown for *source_id*."""
return ""
def __repr__(self) -> str: # pragma: no cover - debugging aid
return f"<ModelSource {self.platform}>"
def clean_source_url(url: Any) -> str:
"""Normalise a stored source URL value into a stripped string."""
if not isinstance(url, str):
return ""
return url.strip()
__all__ = [
"GROUP_PREFIXES",
"HTTP_TIMEOUT",
"ModelSource",
"SourceRef",
"USER_AGENT",
"clean_source_url",
"fetch_text",
]
+49
View File
@@ -0,0 +1,49 @@
"""Hugging Face model source."""
from __future__ import annotations
import re
from .base import ModelSource, fetch_text
#: Lenient — used to normalise URLs already stored in metadata; tolerates
#: sub-paths such as ``/resolve/main/model.safetensors``.
_URL_PATTERN = re.compile(
r"https?://(?:www\.)?huggingface\.co/(?P<id>[^/?#\s]+/[^/?#\s]+)"
)
#: Strict — validates what the user pasted into the "link model" dialog.
_STRICT_URL_PATTERN = re.compile(
r"https?://(?:www\.)?huggingface\.co/(?P<id>[^/?#\s]+/[^/?#\s]+)/?$"
)
class HuggingFaceSource(ModelSource):
"""Hugging Face Hub (``huggingface.co``)."""
platform = "huggingface"
label = "Hugging Face"
supports_enrichment = True
supports_download = True
url_pattern = _URL_PATTERN
strict_url_pattern = _STRICT_URL_PATTERN
def canonical_url(self, source_id: str) -> str:
return f"https://huggingface.co/{source_id}"
def asset_base_url(self, source_id: str, revision: str = "") -> str:
return f"https://huggingface.co/{source_id}/resolve/{revision or 'main'}"
async def fetch_model_card(self, source_id: str) -> str:
"""Fetch ``README.md`` from Hugging Face (tries ``main``, then ``master``)."""
for branch in ("main", "master"):
text = await fetch_text(
f"https://huggingface.co/{source_id}/raw/{branch}/README.md"
)
if text:
return text
return ""
__all__ = ["HuggingFaceSource"]
+76
View File
@@ -0,0 +1,76 @@
"""ModelScope (魔搭社区) model source.
ModelScope exposes the same "model card as README.md" convention as
Hugging Face, including a YAML frontmatter block that often carries
``base_model:`` and ``trigger_words:``. Two public endpoints are used,
neither of which requires an API key for public models:
* ``/models/{owner}/{name}/resolve/{revision}/README.md`` raw model card
* ``/api/v1/models/{owner}/{name}/repo?Revision=..&FilePath=README.md``
the same content through the API, used as a fallback when the resolve
URL is unavailable.
"""
from __future__ import annotations
import re
from .base import ModelSource, fetch_text
_URL_PATTERN = re.compile(
r"https?://(?:www\.)?modelscope\.(?:cn|com)/models/(?P<id>[^/?#\s]+/[^/?#\s]+)"
)
#: Trailing view segments the site appends to a model URL; accepted verbatim
#: when the user pastes a browser tab URL.
_VIEW_SEGMENTS = r"(?:summary|files|model-file|readme|community|evaluation)?"
_STRICT_URL_PATTERN = re.compile(
r"https?://(?:www\.)?modelscope\.(?:cn|com)/models/(?P<id>[^/?#\s]+/[^/?#\s]+)"
rf"/?{_VIEW_SEGMENTS}/?$"
)
#: ``master`` is ModelScope's default branch; ``main`` is tried as a fallback
#: for repos imported from Hugging Face.
_REVISIONS = ("master", "main")
class ModelScopeSource(ModelSource):
"""ModelScope (``modelscope.cn``)."""
platform = "modelscope"
label = "ModelScope"
supports_enrichment = True
supports_download = False
url_pattern = _URL_PATTERN
strict_url_pattern = _STRICT_URL_PATTERN
def canonical_url(self, source_id: str) -> str:
return f"https://modelscope.cn/models/{source_id}"
def asset_base_url(self, source_id: str, revision: str = "") -> str:
return f"https://modelscope.cn/models/{source_id}/resolve/{revision or 'master'}"
async def fetch_model_card(self, source_id: str) -> str:
"""Fetch the model card, preferring the raw resolve URL."""
for revision in _REVISIONS:
text = await fetch_text(
f"https://modelscope.cn/models/{source_id}/resolve/{revision}/README.md"
)
if text:
return text
# Fallback: the repo API proxies the same file and is reachable in
# environments where the CDN resolve host is blocked.
for revision in _REVISIONS:
text = await fetch_text(
"https://modelscope.cn/api/v1/models/"
f"{source_id}/repo?Revision={revision}&FilePath=README.md"
)
if text:
return text
return ""
__all__ = ["ModelScopeSource"]
+208
View File
@@ -0,0 +1,208 @@
"""Registry and metadata helpers for external model sources.
The registry is the single place the rest of the codebase asks "which site
is this URL from?", "what is this model's source?", and "can we enrich it?".
Import from :mod:`py.services.model_sources` rather than this module
directly.
"""
from __future__ import annotations
import logging
from typing import Any, Dict, Mapping, Optional
from .base import GROUP_PREFIXES, ModelSource, SourceRef, clean_source_url
from .huggingface import HuggingFaceSource
from .modelscope import ModelScopeSource
from .tensorart import TensorArtSource
logger = logging.getLogger(__name__)
#: Order matters only for disambiguation; the URL patterns are disjoint.
_SOURCES: tuple[ModelSource, ...] = (
HuggingFaceSource(),
ModelScopeSource(),
TensorArtSource(),
)
_BY_PLATFORM: Dict[str, ModelSource] = {s.platform: s for s in _SOURCES}
#: Metadata keys that carry the canonical external-source identity.
SOURCE_PLATFORM_FIELD = "source_platform"
SOURCE_URL_FIELD = "source_url"
#: Legacy field kept as a read/write alias for Hugging Face models so that
#: older sidecars, cached rows, and third-party consumers keep working.
LEGACY_HF_URL_FIELD = "hf_url"
def list_sources() -> list[ModelSource]:
"""Return every known model source."""
return list(_SOURCES)
def get_source(platform: Optional[str]) -> Optional[ModelSource]:
"""Return the source registered for *platform*, or ``None``."""
if not platform or not isinstance(platform, str):
return None
return _BY_PLATFORM.get(platform.strip().lower())
def source_label(platform: Optional[str], default: str = "") -> str:
"""Return the human-readable label for *platform*."""
source = get_source(platform)
return source.label if source else default
def detect_source(url: Optional[str], *, strict: bool = False) -> Optional[SourceRef]:
"""Return the :class:`SourceRef` for *url*, or ``None`` if unsupported."""
if not url or not isinstance(url, str):
return None
for source in _SOURCES:
ref = source.ref(url, strict=strict)
if ref is not None:
return ref
return None
def resolve_source_ref(metadata: Mapping[str, Any]) -> Optional[SourceRef]:
"""Return the source reference described by a model's metadata.
Handles all three storage states found in the wild:
1. ``source_url`` + ``source_platform`` (current format)
2. ``hf_url`` only (legacy Hugging Face storage)
3. ``hf_url`` plus a newer ``source_url`` (both written by older builds)
"""
if not isinstance(metadata, Mapping):
return None
platform = clean_source_url(metadata.get(SOURCE_PLATFORM_FIELD)).lower()
url = clean_source_url(metadata.get(SOURCE_URL_FIELD))
legacy = clean_source_url(metadata.get(LEGACY_HF_URL_FIELD))
source = get_source(platform)
if url:
if source is not None:
ref = source.ref(url)
if ref is not None:
return ref
ref = detect_source(url)
if ref is not None:
return ref
# Unknown platform but a URL is present: keep it addressable.
return SourceRef(platform=platform or "unknown", source_id="", url=url)
if legacy:
return detect_source(legacy)
return None
def normalize_metadata_source(metadata: Dict[str, Any]) -> Dict[str, Any]:
"""Normalise the external-source fields on *metadata* in place.
Guarantees that ``source_url``/``source_platform`` are present and
consistent, and that ``hf_url`` mirrors ``source_url`` for Hugging Face
models (never for other platforms, so a stale alias can't make a
ModelScope model look like a Hugging Face one).
Returns the same dict for convenient chaining.
"""
if not isinstance(metadata, dict):
return metadata
platform = clean_source_url(metadata.get(SOURCE_PLATFORM_FIELD)).lower()
url = clean_source_url(metadata.get(SOURCE_URL_FIELD))
legacy = clean_source_url(metadata.get(LEGACY_HF_URL_FIELD))
source = get_source(platform)
ref: Optional[SourceRef] = None
if url:
ref = source.ref(url) if source is not None else None
if ref is None:
ref = detect_source(url)
elif legacy:
ref = detect_source(legacy)
if ref is not None and ref.source_id:
platform = ref.platform
url = ref.url or url
if platform:
metadata[SOURCE_PLATFORM_FIELD] = platform
else:
metadata.setdefault(SOURCE_PLATFORM_FIELD, "")
metadata[SOURCE_URL_FIELD] = url
# Keep the legacy alias in sync, but only for Hugging Face.
if url and platform == "huggingface":
metadata[LEGACY_HF_URL_FIELD] = url
elif LEGACY_HF_URL_FIELD in metadata and platform and platform != "huggingface":
metadata[LEGACY_HF_URL_FIELD] = ""
elif legacy and not url:
metadata[LEGACY_HF_URL_FIELD] = legacy
return metadata
def has_external_source(item: Mapping[str, Any]) -> bool:
"""Return ``True`` when *item* is linked to any external model site."""
if not isinstance(item, Mapping):
return False
return bool(
clean_source_url(item.get(SOURCE_URL_FIELD))
or clean_source_url(item.get(LEGACY_HF_URL_FIELD))
)
def get_source_platform(item: Mapping[str, Any]) -> str:
"""Return the platform id stored on *item* (may be empty)."""
if not isinstance(item, Mapping):
return ""
platform = clean_source_url(item.get(SOURCE_PLATFORM_FIELD)).lower()
if platform:
return platform
ref = resolve_source_ref(item)
return ref.platform if ref else ""
def source_group_key(item: Mapping[str, Any]) -> Optional[str]:
"""Return the version-group key for *item*, or ``None``.
Hugging Face keeps the historical ``hf:{owner}/{repo}`` shape; other
platforms use their own short prefix (see :data:`GROUP_PREFIXES`).
"""
ref = resolve_source_ref(item)
if ref is None or not ref.source_id:
return None
source = get_source(ref.platform)
if source is None:
return None
return source.group_key(ref.source_id)
__all__ = [
"GROUP_PREFIXES",
"LEGACY_HF_URL_FIELD",
"SOURCE_PLATFORM_FIELD",
"SOURCE_URL_FIELD",
"detect_source",
"get_source",
"get_source_platform",
"has_external_source",
"list_sources",
"normalize_metadata_source",
"resolve_source_ref",
"source_group_key",
"source_label",
]
+56
View File
@@ -0,0 +1,56 @@
"""TensorArt model source (link / provenance only).
TensorArt support is intentionally limited to *linking* a model to its
TensorArt page. Automatic metadata extraction is not possible without a
user session:
* ``tensor.art`` sits behind a Cloudflare managed challenge, so plain
HTTP clients (aiohttp, requests, curl) receive ``403 "Just a moment..."``.
* Its internal API (``ap-east-1.tensorart.cloud`` / ``cn.tensorart.net``)
answers every ``/v1/model/*`` route with
``{"code":100002,"message":"invalid authorization header"}``.
* The official TAMS API requires an AccessKey/SecretKey pair and request
signatures, which is a poor fit for a "paste a URL" workflow.
``supports_enrichment`` is therefore ``False``: the agent pipeline skips
these models with an explicit reason instead of failing silently, and the
UI keeps showing the "View on TensorArt" link. ``tusi.cn`` is TensorArt's
Chinese mirror and is accepted as the same platform.
"""
from __future__ import annotations
import re
from .base import ModelSource
_DOMAINS = r"(?:tensor\.art|tusi\.cn)"
_URL_PATTERN = re.compile(
rf"https?://(?:www\.)?{_DOMAINS}/models/(?P<id>\d+)"
)
_STRICT_URL_PATTERN = re.compile(
rf"https?://(?:www\.)?{_DOMAINS}/models/(?P<id>\d+)(?:/[^/?#\s]+)?/?$"
)
class TensorArtSource(ModelSource):
"""TensorArt (``tensor.art``)."""
platform = "tensorart"
label = "TensorArt"
supports_enrichment = False
supports_download = False
url_pattern = _URL_PATTERN
strict_url_pattern = _STRICT_URL_PATTERN
def canonical_url(self, source_id: str) -> str:
return f"https://tensor.art/models/{source_id}"
def asset_base_url(self, source_id: str, revision: str = "") -> str:
# Unreachable today: enrichment is disabled for this platform.
return f"https://tensor.art/models/{source_id}"
__all__ = ["TensorArtSource"]
+2
View File
@@ -67,6 +67,8 @@ class OtherModelService(BaseModelService):
"civitai": self.filter_civitai_data(model_data.get("civitai", {}), minimal=True), "civitai": self.filter_civitai_data(model_data.get("civitai", {}), minimal=True),
"auto_tags": model_data.get("auto_tags") or extract_auto_tags(model_data), "auto_tags": model_data.get("auto_tags") or extract_auto_tags(model_data),
"version_count": model_data.get("version_count"), "version_count": model_data.get("version_count"),
"source_platform": model_data.get("source_platform", ""),
"source_url": model_data.get("source_url", ""),
"hf_url": model_data.get("hf_url", ""), "hf_url": model_data.get("hf_url", ""),
} }
+17
View File
@@ -7,6 +7,7 @@ from dataclasses import dataclass, field
from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple
from ..utils.cache_paths import CacheType, resolve_cache_path_with_migration from ..utils.cache_paths import CacheType, resolve_cache_path_with_migration
from .model_sources import normalize_metadata_source
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -62,6 +63,8 @@ class PersistentModelCache:
"db_checked", "db_checked",
"last_checked_at", "last_checked_at",
"hash_status", "hash_status",
"source_platform",
"source_url",
"hf_url", "hf_url",
) )
_MODEL_UPDATE_COLUMNS: Tuple[str, ...] = _MODEL_COLUMNS[2:] _MODEL_UPDATE_COLUMNS: Tuple[str, ...] = _MODEL_COLUMNS[2:]
@@ -206,8 +209,13 @@ class PersistentModelCache:
"skip_metadata_refresh": bool(row["skip_metadata_refresh"]), "skip_metadata_refresh": bool(row["skip_metadata_refresh"]),
"license_flags": int(license_value), "license_flags": int(license_value),
"hash_status": row["hash_status"] or "completed", "hash_status": row["hash_status"] or "completed",
"source_platform": row["source_platform"] or "",
"source_url": row["source_url"] or "",
"hf_url": row["hf_url"] or "", "hf_url": row["hf_url"] or "",
} }
# Legacy rows only carry `hf_url`; derive the canonical pair so
# every consumer sees the same shape.
normalize_metadata_source(item)
if row["autov3"] is not None: if row["autov3"] is not None:
item["autov3"] = (row["autov3"] or "").lower() item["autov3"] = (row["autov3"] or "").lower()
raw_data.append(item) raw_data.append(item)
@@ -562,6 +570,8 @@ class PersistentModelCache:
db_checked INTEGER, db_checked INTEGER,
last_checked_at REAL, last_checked_at REAL,
hash_status TEXT, hash_status TEXT,
source_platform TEXT DEFAULT '',
source_url TEXT DEFAULT '',
hf_url TEXT DEFAULT '', hf_url TEXT DEFAULT '',
PRIMARY KEY (model_type, file_path) PRIMARY KEY (model_type, file_path)
); );
@@ -629,6 +639,8 @@ class PersistentModelCache:
# Persisting without explicit flags should assume CivitAI's documented defaults (0b111001 == 57). # Persisting without explicit flags should assume CivitAI's documented defaults (0b111001 == 57).
"license_flags": f"INTEGER DEFAULT {DEFAULT_LICENSE_FLAGS}", "license_flags": f"INTEGER DEFAULT {DEFAULT_LICENSE_FLAGS}",
"hash_status": "TEXT DEFAULT 'completed'", "hash_status": "TEXT DEFAULT 'completed'",
"source_platform": "TEXT DEFAULT ''",
"source_url": "TEXT DEFAULT ''",
"hf_url": "TEXT DEFAULT ''", "hf_url": "TEXT DEFAULT ''",
"autov3": "TEXT", "autov3": "TEXT",
} }
@@ -650,6 +662,9 @@ class PersistentModelCache:
return conn return conn
def _prepare_model_row(self, model_type: str, item: Dict[str, Any]) -> Tuple[Any, ...]: def _prepare_model_row(self, model_type: str, item: Dict[str, Any]) -> Tuple[Any, ...]:
# Keep `source_*` and the legacy `hf_url` alias consistent no matter
# which caller populated the item.
normalize_metadata_source(item)
civitai = item.get("civitai") or {} civitai = item.get("civitai") or {}
trained_words = civitai.get("trainedWords") trained_words = civitai.get("trainedWords")
if isinstance(trained_words, str): if isinstance(trained_words, str):
@@ -713,6 +728,8 @@ class PersistentModelCache:
1 if item.get("db_checked") else 0, 1 if item.get("db_checked") else 0,
float(item.get("last_checked_at") or 0.0), float(item.get("last_checked_at") or 0.0),
item.get("hash_status", "completed"), item.get("hash_status", "completed"),
item.get("source_platform") or "",
item.get("source_url") or "",
item.get("hf_url") or "", item.get("hf_url") or "",
) )
@@ -7,6 +7,7 @@ import time
from typing import Any, Dict, List, Optional, Protocol, Sequence from typing import Any, Dict, List, Optional, Protocol, Sequence
from ..metadata_sync_service import MetadataSyncService from ..metadata_sync_service import MetadataSyncService
from ..model_sources import has_external_source
from ...utils.metadata_manager import MetadataManager from ...utils.metadata_manager import MetadataManager
@@ -51,10 +52,11 @@ class BulkMetadataRefreshUseCase:
if not model.get("skip_metadata_refresh", False) if not model.get("skip_metadata_refresh", False)
and not self._is_in_skip_path(model.get("folder", ""), skip_paths) and not self._is_in_skip_path(model.get("folder", ""), skip_paths)
and (not model.get("civitai") or not model["civitai"].get("id")) and (not model.get("civitai") or not model["civitai"].get("id"))
# Skip models downloaded from Hugging Face — they are not on # Skip models linked to an external model site (Hugging Face /
# CivitAI / CivArchive. Users can still refresh them individually # ModelScope / TensorArt) — they are not on CivitAI / CivArchive.
# via the right-click context menu. # Users can still refresh them individually via the right-click
and not model.get("hf_url", "") # context menu.
and not has_external_source(model)
and not ( and not (
# Skip models confirmed not on CivitAI when no need to retry # Skip models confirmed not on CivitAI when no need to retry
model.get("from_civitai") is False model.get("from_civitai") is False
@@ -7,6 +7,8 @@ import { MODEL_CONFIG } from '../../api/apiConfig.js';
import { translate } from '../../utils/i18nHelpers.js'; import { translate } from '../../utils/i18nHelpers.js';
import { getNsfwLevelSelector } from '../shared/NsfwLevelSelector.js'; import { getNsfwLevelSelector } from '../shared/NsfwLevelSelector.js';
import { classifyModelRelinkUrl } from '../../utils/civitaiUtils.js'; import { classifyModelRelinkUrl } from '../../utils/civitaiUtils.js';
import { parseModelSourceUrl, getModelSourceInfo } from '../../utils/modelSourceHelpers.js';
import { escapeHtml } from '../shared/utils.js';
// Mixin with shared functionality for LoraContextMenu and CheckpointContextMenu // Mixin with shared functionality for LoraContextMenu and CheckpointContextMenu
export const ModelContextMenuMixin = { export const ModelContextMenuMixin = {
@@ -211,7 +213,7 @@ export const ModelContextMenuMixin = {
setTimeout(() => urlInput.focus(), 50); setTimeout(() => urlInput.focus(), 50);
}, },
// HuggingFace linking methods // External model source linking (Hugging Face / ModelScope / TensorArt)
showLinkHfModal() { showLinkHfModal() {
const filePath = this.currentCard.dataset.filepath; const filePath = this.currentCard.dataset.filepath;
if (!filePath) return; if (!filePath) return;
@@ -225,15 +227,23 @@ export const ModelContextMenuMixin = {
} }
this._boundLinkHfHandler = async () => { this._boundLinkHfHandler = async () => {
const hfUrl = urlInput.value.trim(); const rawUrl = urlInput.value.trim();
if (!hfUrl) { if (!rawUrl) {
errorDiv.textContent = 'Please enter a HuggingFace repository URL.'; errorDiv.textContent = translate(
'modals.linkModelSource.urlRequired',
{},
'Please enter a model page URL.'
);
return; return;
} }
const hfPattern = /^https?:\/\/huggingface\.co\/([^/]+\/[^/]+)\/?$/; const sourceInfo = parseModelSourceUrl(rawUrl);
if (!hfPattern.test(hfUrl)) { if (!sourceInfo) {
errorDiv.textContent = 'Invalid URL format. Expected: https://huggingface.co/user/repo'; errorDiv.textContent = translate(
'modals.linkModelSource.invalidUrl',
{},
'Unsupported URL. Supported sites: Hugging Face, ModelScope, TensorArt.'
);
return; return;
} }
@@ -241,12 +251,14 @@ export const ModelContextMenuMixin = {
modalManager.closeModal('linkHfModal'); modalManager.closeModal('linkHfModal');
try { try {
state.loadingManager.showSimpleLoading('Linking to HuggingFace...'); state.loadingManager.showSimpleLoading(
translate('modals.linkModelSource.linking', {}, 'Linking model source...')
);
const response = await fetch('/api/lm/set-hf-url', { const response = await fetch('/api/lm/set-hf-url', {
method: 'POST', method: 'POST',
headers: { 'Content-Type': 'application/json' }, headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ file_path: filePath, hf_url: hfUrl }), body: JSON.stringify({ file_path: filePath, source_url: sourceInfo.url }),
}); });
if (!response.ok) { if (!response.ok) {
@@ -262,7 +274,7 @@ export const ModelContextMenuMixin = {
throw new Error(data.error || 'Failed to link model'); throw new Error(data.error || 'Failed to link model');
} }
} catch (error) { } catch (error) {
console.error('Error linking model to HuggingFace:', error); console.error('Error linking model source:', error);
showToast('toast.contextMenu.linkHfFailed', { message: error.message }, 'error'); showToast('toast.contextMenu.linkHfFailed', { message: error.message }, 'error');
} finally { } finally {
state.loadingManager.hide(); state.loadingManager.hide();
@@ -276,18 +288,68 @@ export const ModelContextMenuMixin = {
modalManager.showModal('linkHfModal'); modalManager.showModal('linkHfModal');
this._renderSupportedSources();
setTimeout(() => urlInput.focus(), 50); setTimeout(() => urlInput.focus(), 50);
}, },
// HF metadata enrichment (AI agent) methods /**
* Refresh the supported-site hints from the server so the dialog reflects
* whatever sources this backend build actually knows about. Falls back to
* the static markup in the template when the request fails.
*/
async _renderSupportedSources() {
const container = document.getElementById('hfSupportedSources');
if (!container) return;
try {
const response = await fetch('/api/lm/model-sources');
if (!response.ok) return;
const sources = await response.json();
if (!Array.isArray(sources) || sources.length === 0) return;
const examples = sources
.map((source) => source?.example_url)
.filter((url) => typeof url === 'string' && url);
if (examples.length === 0) return;
container.innerHTML = examples
.map((url) => `<strong>${escapeHtml(url)}</strong>`)
.join('<br>');
} catch (error) {
console.debug('Failed to load supported model sources:', error);
}
},
// Model metadata enrichment (AI agent) methods
updateEnrichMenuItem(card) { updateEnrichMenuItem(card) {
const enrichItem = this.menu?.querySelector('[data-action="enrich-hf-llm"]'); const enrichItem = this.menu?.querySelector('[data-action="enrich-hf-llm"]');
if (!enrichItem) return; if (!enrichItem) return;
const hasHfUrl = !!card.dataset.hf_url;
enrichItem.classList.toggle('disabled', !hasHfUrl); const model = {
enrichItem.title = hasHfUrl source_url: card.dataset.source_url || '',
? '' source_platform: card.dataset.source_platform || '',
: 'Link this model to a HuggingFace repo first (Link Model → Link to HuggingFace)'; hf_url: card.dataset.hf_url || '',
};
const sourceInfo = getModelSourceInfo(model);
const canEnrich = Boolean(sourceInfo && sourceInfo.supportsEnrichment);
enrichItem.classList.toggle('disabled', !canEnrich);
if (canEnrich) {
enrichItem.title = '';
} else if (!sourceInfo) {
enrichItem.title = translate(
'toast.contextMenu.enrichNeedsSource',
{},
'Link this model to a model source first (Link Model → Link to Model Source)'
);
} else {
enrichItem.title = translate(
'toast.contextMenu.enrichUnsupportedSource',
{ source: sourceInfo.label },
`AI enrichment is not available for ${sourceInfo.label} models`
);
}
}, },
async enrichWithAgent(filePath) { async enrichWithAgent(filePath) {
+25 -12
View File
@@ -1,4 +1,5 @@
import { showToast, openCivitai, openHuggingFace, copyToClipboard, copyLoraSyntax, sendLoraToWorkflow, sendEmbeddingToWorkflow, openExampleImagesFolder, buildLoraSyntax, sendModelPathToWorkflow } from '../../utils/uiHelpers.js'; import { showToast, openCivitai, openHuggingFace, copyToClipboard, copyLoraSyntax, sendLoraToWorkflow, sendEmbeddingToWorkflow, openExampleImagesFolder, buildLoraSyntax, sendModelPathToWorkflow } from '../../utils/uiHelpers.js';
import { getModelSourceInfo, getModelSourceGroupKey, getModelSourceViewTitle, openModelSource } from '../../utils/modelSourceHelpers.js';
import { state, getCurrentPageState } from '../../state/index.js'; import { state, getCurrentPageState } from '../../state/index.js';
import { showModelModal } from './ModelModal.js'; import { showModelModal } from './ModelModal.js';
import { hasCivitaiSource } from './utils.js'; import { hasCivitaiSource } from './utils.js';
@@ -65,12 +66,15 @@ function handleModelCardEvent_internal(event, modelType) {
if (event.target.closest('.fa-globe')) { if (event.target.closest('.fa-globe')) {
event.stopPropagation(); event.stopPropagation();
// CivitAI wins when the model actually has CivitAI data; otherwise fall // CivitAI wins when the model actually has CivitAI data; otherwise fall
// back to HuggingFace. Relying on `from_civitai` here made the two // back to the linked external source. Relying on `from_civitai` here
// sources mutually exclusive whenever one of them was (re)linked (#1094). // made the two sources mutually exclusive whenever one of them was
// (re)linked (#1094).
if (card.dataset.has_civitai === 'true') { if (card.dataset.has_civitai === 'true') {
openCivitai(card.dataset.filepath); openCivitai(card.dataset.filepath);
} else if (card.dataset.hf_url) { } else if (card.dataset.source_platform === 'huggingface' && card.dataset.hf_url) {
openHuggingFace(card.dataset.hf_url); openHuggingFace(card.dataset.hf_url);
} else if (card.dataset.source_url) {
openModelSource(card.dataset.source_url);
} }
return true; // Stop propagation return true; // Stop propagation
} }
@@ -337,6 +341,8 @@ async function showModelModalFromCard(card, modelType) {
modified: card.dataset.modified, modified: card.dataset.modified,
file_size: parseInt(card.dataset.file_size || '0'), file_size: parseInt(card.dataset.file_size || '0'),
from_civitai: card.dataset.from_civitai === 'true', from_civitai: card.dataset.from_civitai === 'true',
source_platform: card.dataset.source_platform || '',
source_url: card.dataset.source_url || '',
hf_url: card.dataset.hf_url || '', hf_url: card.dataset.hf_url || '',
base_model: card.dataset.base_model, base_model: card.dataset.base_model,
notes: card.dataset.notes || '', notes: card.dataset.notes || '',
@@ -428,6 +434,8 @@ function showExampleAccessModal(card, modelType) {
modified: card.dataset.modified, modified: card.dataset.modified,
file_size: card.dataset.file_size, file_size: card.dataset.file_size,
from_civitai: card.dataset.from_civitai === 'true', from_civitai: card.dataset.from_civitai === 'true',
source_platform: card.dataset.source_platform || '',
source_url: card.dataset.source_url || '',
hf_url: card.dataset.hf_url || '', hf_url: card.dataset.hf_url || '',
base_model: card.dataset.base_model, base_model: card.dataset.base_model,
notes: card.dataset.notes, notes: card.dataset.notes,
@@ -490,7 +498,11 @@ export function createModelCard(model, modelType) {
card.dataset.base_model = model.base_model || 'Unknown'; card.dataset.base_model = model.base_model || 'Unknown';
card.dataset.favorite = model.favorite ? 'true' : 'false'; card.dataset.favorite = model.favorite ? 'true' : 'false';
card.dataset.exclude = model.exclude ? 'true' : 'false'; card.dataset.exclude = model.exclude ? 'true' : 'false';
card.dataset.hf_url = model.hf_url || ''; const modelSourceInfo = getModelSourceInfo(model);
card.dataset.source_url = modelSourceInfo?.url || '';
card.dataset.source_platform = modelSourceInfo?.platform || '';
// Legacy alias: only Hugging Face models expose `hf_url`.
card.dataset.hf_url = modelSourceInfo?.platform === 'huggingface' ? modelSourceInfo.url : '';
const hasUpdateAvailable = Boolean(model.update_available); const hasUpdateAvailable = Boolean(model.update_available);
card.dataset.update_available = hasUpdateAvailable ? 'true' : 'false'; card.dataset.update_available = hasUpdateAvailable ? 'true' : 'false';
card.dataset.skip_metadata_refresh = model.skip_metadata_refresh ? 'true' : 'false'; card.dataset.skip_metadata_refresh = model.skip_metadata_refresh ? 'true' : 'false';
@@ -508,11 +520,12 @@ export function createModelCard(model, modelType) {
const modelId = civitaiData?.modelId ?? civitaiData?.model_id; const modelId = civitaiData?.modelId ?? civitaiData?.model_id;
if (modelId !== undefined && modelId !== null && modelId !== '') { if (modelId !== undefined && modelId !== null && modelId !== '') {
card.dataset.modelId = modelId; card.dataset.modelId = modelId;
} else if (model.hf_url) { } else {
// For HF-only models, derive a group key from hf_url for version grouping // For externally-sourced models, derive a group key from the source
const match = model.hf_url.match(/https?:\/\/huggingface\.co\/([^/]+\/[^/]+)/); // URL for version grouping (hf:user/repo, ms:user/repo, ta:<id>).
if (match) { const sourceGroupKey = getModelSourceGroupKey(model);
card.dataset.modelId = 'hf:' + match[1]; if (sourceGroupKey) {
card.dataset.modelId = sourceGroupKey;
} }
} }
@@ -610,10 +623,10 @@ export function createModelCard(model, modelType) {
const hasCivitai = hasCivitaiSource(model.civitai); const hasCivitai = hasCivitaiSource(model.civitai);
const globeTitle = hasCivitai ? const globeTitle = hasCivitai ?
translate('modelCard.actions.viewOnCivitai', {}, 'View on Civitai') : translate('modelCard.actions.viewOnCivitai', {}, 'View on Civitai') :
model.hf_url ? modelSourceInfo ?
translate('modelCard.actions.viewOnHuggingFace', {}, 'View on Hugging Face') : getModelSourceViewTitle(modelSourceInfo) :
translate('modelCard.actions.notAvailableFromCivitai', {}, 'Not available from Civitai'); translate('modelCard.actions.notAvailableFromCivitai', {}, 'Not available from Civitai');
const globeEnabled = hasCivitai || !!model.hf_url; const globeEnabled = hasCivitai || !!modelSourceInfo;
let sendTitle; let sendTitle;
let copyTitle; let copyTitle;
if (modelType === MODEL_TYPES.LORA) { if (modelType === MODEL_TYPES.LORA) {
+18 -9
View File
@@ -1,4 +1,5 @@
import { showToast, openCivitai, sendLoraToWorkflow, sendEmbeddingToWorkflow, sendModelPathToWorkflow, buildLoraSyntax, copyToClipboard } from '../../utils/uiHelpers.js'; import { showToast, openCivitai, sendLoraToWorkflow, sendEmbeddingToWorkflow, sendModelPathToWorkflow, buildLoraSyntax, copyToClipboard } from '../../utils/uiHelpers.js';
import { getModelSourceInfo, getModelSourceGroupKey, getModelSourceViewTitle, openModelSource } from '../../utils/modelSourceHelpers.js';
import { modalManager } from '../../managers/ModalManager.js'; import { modalManager } from '../../managers/ModalManager.js';
import { MODEL_TYPES } from '../../api/apiConfig.js'; import { MODEL_TYPES } from '../../api/apiConfig.js';
import { import {
@@ -397,10 +398,13 @@ export async function showModelModal(model, modelType) {
<div class="civitai-view" title="${translate('modals.model.actions.viewOnCivitai', {}, 'View on Civitai')}" data-action="view-civitai" data-filepath="${escapedFilePathAttr}"> <div class="civitai-view" title="${translate('modals.model.actions.viewOnCivitai', {}, 'View on Civitai')}" data-action="view-civitai" data-filepath="${escapedFilePathAttr}">
<i class="fas fa-globe"></i> ${translate('modals.model.actions.viewOnCivitaiText', {}, 'View on Civitai')} <i class="fas fa-globe"></i> ${translate('modals.model.actions.viewOnCivitaiText', {}, 'View on Civitai')}
</div>`.trim() : ''; </div>`.trim() : '';
const escapedHfUrl = modelWithFullData.hf_url ? escapeAttribute(modelWithFullData.hf_url) : ''; const sourceInfo = getModelSourceInfo(modelWithFullData);
const viewOnHuggingFaceAction = escapedHfUrl ? ` const escapedSourceUrl = sourceInfo?.url ? escapeAttribute(sourceInfo.url) : '';
<div class="civitai-view" title="${translate('modals.model.actions.viewOnHuggingFace', {}, 'View on Hugging Face')}" data-action="view-huggingface" data-hf-url="${escapedHfUrl}"> const isHuggingFaceSource = sourceInfo?.platform === 'huggingface';
<i class="fas fa-globe"></i> ${translate('modals.model.actions.viewOnHuggingFaceText', {}, 'View on Hugging Face')} const sourceTitle = sourceInfo ? getModelSourceViewTitle(sourceInfo) : '';
const viewOnHuggingFaceAction = escapedSourceUrl ? `
<div class="civitai-view" title="${escapeAttribute(sourceTitle)}" data-action="${isHuggingFaceSource ? 'view-huggingface' : 'view-model-source'}" ${isHuggingFaceSource ? 'data-hf-url' : 'data-source-url'}="${escapedSourceUrl}">
<i class="fas fa-globe"></i> ${escapeHtml(sourceTitle)}
</div>`.trim() : ''; </div>`.trim() : '';
const creatorInfoAction = modelWithFullData.civitai?.creator ? ` const creatorInfoAction = modelWithFullData.civitai?.creator ? `
<div class="creator-info" data-username="${modelWithFullData.civitai.creator.username}" data-action="view-creator" title="${translate('modals.model.actions.viewCreatorProfile', {}, 'View Creator Profile')}"> <div class="creator-info" data-username="${modelWithFullData.civitai.creator.username}" data-action="view-creator" title="${translate('modals.model.actions.viewCreatorProfile', {}, 'View Creator Profile')}">
@@ -520,12 +524,12 @@ export async function showModelModal(model, modelType) {
const loadingExamplesText = translate('modals.model.loading.examples', {}, 'Loading examples...'); const loadingExamplesText = translate('modals.model.loading.examples', {}, 'Loading examples...');
const loadingVersionsText = translate('modals.model.loading.versions', {}, 'Loading versions...'); const loadingVersionsText = translate('modals.model.loading.versions', {}, 'Loading versions...');
// Use CivitAI modelId, or derive HF group key for HF-only models // Use CivitAI modelId, or derive a source group key for externally-linked models
let civitaiModelId = modelWithFullData.civitai?.modelId || ''; let civitaiModelId = modelWithFullData.civitai?.modelId || '';
if (!civitaiModelId && modelWithFullData.hf_url) { if (!civitaiModelId) {
const match = modelWithFullData.hf_url.match(/https?:\/\/huggingface\.co\/([^/]+\/[^/]+)/); const sourceGroupKey = getModelSourceGroupKey(modelWithFullData);
if (match) { if (sourceGroupKey) {
civitaiModelId = 'hf:' + match[1]; civitaiModelId = sourceGroupKey;
} }
} }
const civitaiVersionId = modelWithFullData.civitai?.id || ''; const civitaiVersionId = modelWithFullData.civitai?.id || '';
@@ -939,6 +943,11 @@ function setupEventHandlers(filePath, modelType) {
window.open(target.dataset.hfUrl, '_blank', 'noopener,noreferrer'); window.open(target.dataset.hfUrl, '_blank', 'noopener,noreferrer');
} }
break; break;
case 'view-model-source':
if (target.dataset.sourceUrl) {
openModelSource(target.dataset.sourceUrl);
}
break;
case 'view-creator': case 'view-creator':
const username = target.dataset.username; const username = target.dataset.username;
if (username) { if (username) {
@@ -5,6 +5,7 @@ import { openCivitaiUrl, showToast } from '../../utils/uiHelpers.js';
import { translate } from '../../utils/i18nHelpers.js'; import { translate } from '../../utils/i18nHelpers.js';
import { state } from '../../state/index.js'; import { state } from '../../state/index.js';
import { buildCivitaiModelUrl } from '../../utils/civitaiUtils.js'; import { buildCivitaiModelUrl } from '../../utils/civitaiUtils.js';
import { parseModelSourceGroupKey } from '../../utils/modelSourceHelpers.js';
import { formatFileSize } from './utils.js'; import { formatFileSize } from './utils.js';
import { setSessionItem, removeSessionItem } from '../../utils/storageHelpers.js'; import { setSessionItem, removeSessionItem } from '../../utils/storageHelpers.js';
@@ -993,22 +994,23 @@ export function initVersionsTab({
renderErrorState(container, translate('modals.model.versions.missingModelId', {}, 'This model is missing a Civitai model id.')); renderErrorState(container, translate('modals.model.versions.missingModelId', {}, 'This model is missing a Civitai model id.'));
return; return;
} }
// HF group keys (e.g. "hf:user/repo") are not real CivitAI model IDs — // External source group keys (e.g. "hf:user/repo", "ms:user/repo",
// skip the remote API call and show a helpful message instead. // "ta:8278...") are not real CivitAI model IDs — skip the remote API
const isHfGroupKey = typeof modelId === 'string' && modelId.startsWith('hf:'); // call and show a helpful message instead.
if (isHfGroupKey) { const sourceGroup = parseModelSourceGroupKey(modelId);
if (sourceGroup) {
controller.isLoading = false; controller.isLoading = false;
controller.hasLoaded = true; controller.hasLoaded = true;
controller.record = null; controller.record = null;
const hfMsg = translate( const sourceMsg = translate(
'modals.model.versions.hfGroupInfo', 'modals.model.versions.sourceGroupInfo',
{}, { source: sourceGroup.label },
'This is a HuggingFace model group. Open the library to see all versions in the grid.' `This is a ${sourceGroup.label} model group. Open the library to see all versions in the grid.`
); );
container.innerHTML = ` container.innerHTML = `
<div class="versions-empty-state"> <div class="versions-empty-state">
<i class="fas fa-info-circle"></i> <i class="fas fa-info-circle"></i>
<p>${escapeHtml(hfMsg)}</p> <p>${escapeHtml(sourceMsg)}</p>
</div> </div>
`; `;
return; return;
+173
View File
@@ -0,0 +1,173 @@
/**
* External model source helpers (Hugging Face / ModelScope / TensorArt).
*
* Mirrors `py/services/model_sources/registry.py` so the frontend and the
* backend agree on URL recognition, version-group keys, and which sites
* support AI metadata enrichment.
*
* Models loaded from an older cache may only carry the legacy `hf_url`
* field; every helper here falls back to it, and to the legacy
* `hf:user/repo` group key shape.
*/
import { translate } from './i18nHelpers.js';
export const MODEL_SOURCES = [
{
platform: 'huggingface',
label: 'Hugging Face',
groupPrefix: 'hf',
supportsEnrichment: true,
supportsDownload: true,
exampleUrl: 'https://huggingface.co/user/repo',
placeholder: 'https://huggingface.co/user/repo',
pattern: /^https?:\/\/(?:www\.)?huggingface\.co\/([^/?#\s]+\/[^/?#\s]+)/i,
canonical: (id) => `https://huggingface.co/${id}`,
},
{
platform: 'modelscope',
label: 'ModelScope',
groupPrefix: 'ms',
supportsEnrichment: true,
supportsDownload: false,
exampleUrl: 'https://modelscope.cn/models/user/repo',
placeholder: 'https://modelscope.cn/models/user/repo',
pattern: /^https?:\/\/(?:www\.)?modelscope\.(?:cn|com)\/models\/([^/?#\s]+\/[^/?#\s]+)/i,
canonical: (id) => `https://modelscope.cn/models/${id}`,
},
{
platform: 'tensorart',
label: 'TensorArt',
groupPrefix: 'ta',
supportsEnrichment: false,
supportsDownload: false,
exampleUrl: 'https://tensor.art/models/827823520299086029',
placeholder: 'https://tensor.art/models/827823520299086029',
pattern: /^https?:\/\/(?:www\.)?(?:tensor\.art|tusi\.cn)\/models\/(\d+)/i,
canonical: (id) => `https://tensor.art/models/${id}`,
},
];
/** Return the source descriptor for a platform id, or null. */
export function getModelSource(platform) {
if (!platform || typeof platform !== 'string') return null;
const needle = platform.trim().toLowerCase();
return MODEL_SOURCES.find((source) => source.platform === needle) || null;
}
/**
* Parse any supported model URL.
* @returns {{platform: string, label: string, groupPrefix: string,
* supportsEnrichment: boolean, supportsDownload: boolean,
* sourceId: string, url: string}|null}
*/
export function parseModelSourceUrl(url) {
if (!url || typeof url !== 'string') return null;
const candidate = url.trim();
if (!candidate) return null;
for (const source of MODEL_SOURCES) {
const match = candidate.match(source.pattern);
if (match) {
return {
...source,
sourceId: match[1],
url: source.canonical(match[1]),
};
}
}
return null;
}
/** Return the stored source URL of a model (new field, then legacy). */
export function getModelSourceUrl(model) {
if (!model) return '';
const value = model.source_url || model.hf_url || '';
return typeof value === 'string' ? value.trim() : '';
}
/** Return the stored source platform of a model. */
export function getModelSourcePlatform(model) {
if (!model) return '';
const value = model.source_platform || '';
return typeof value === 'string' ? value.trim().toLowerCase() : '';
}
/**
* Resolve the full source descriptor for a model, tolerating models that
* predate the `source_*` fields.
*/
export function getModelSourceInfo(model) {
if (!model) return null;
const url = getModelSourceUrl(model);
const declared = getModelSource(getModelSourcePlatform(model));
const parsed = parseModelSourceUrl(url);
if (declared) {
return {
...declared,
sourceId: parsed ? parsed.sourceId : '',
url: parsed ? parsed.url : url,
};
}
return parsed;
}
/**
* Version-group key for a model, matching the backend's `_extract_group_key`.
* Returns `''` when the model has no external source.
*/
export function getModelSourceGroupKey(model) {
const info = getModelSourceInfo(model);
if (!info || !info.sourceId) return '';
return `${info.groupPrefix}:${info.sourceId}`;
}
/** Whether AI metadata enrichment can run for this model's source. */
export function canEnrichModelSource(model) {
const info = getModelSourceInfo(model);
return Boolean(info && info.supportsEnrichment);
}
/**
* Parse a version-group key such as `hf:user/repo`, `ms:user/repo`, or
* `ta:827823520299086029` back into its source descriptor.
*
* These keys are NOT CivitAI model ids, so callers must not send them to the
* CivitAI API.
*
* @returns {{platform: string, label: string, sourceId: string}|null}
*/
export function parseModelSourceGroupKey(groupKey) {
if (!groupKey || typeof groupKey !== 'string') return null;
const separator = groupKey.indexOf(':');
if (separator <= 0) return null;
const prefix = groupKey.slice(0, separator);
const source = MODEL_SOURCES.find((candidate) => candidate.groupPrefix === prefix);
if (!source) return null;
return {
platform: source.platform,
label: source.label,
sourceId: groupKey.slice(separator + 1),
};
}
/** Localised "View on X" title for the source globe icon. */
export function getModelSourceViewTitle(info) {
if (!info) return '';
if (info.platform === 'huggingface') {
return translate('modelCard.actions.viewOnHuggingFace', {}, 'View on Hugging Face');
}
return translate(
'modelCard.actions.viewOnSource',
{ source: info.label },
`View on ${info.label}`
);
}
/** Open a model page on its external site in a new tab. */
export function openModelSource(url) {
if (!url) return;
window.open(url, '_blank', 'noopener,noreferrer');
}
+1 -1
View File
@@ -21,7 +21,7 @@
<i class="fas fa-external-link-alt"></i> <span>{{ t('loras.contextMenu.linkCivitai') }}</span> <i class="fas fa-external-link-alt"></i> <span>{{ t('loras.contextMenu.linkCivitai') }}</span>
</div> </div>
<div class="context-menu-item" data-action="link-hf"> <div class="context-menu-item" data-action="link-hf">
<i class="fas fa-robot"></i> <span>{{ t('loras.contextMenu.linkHuggingFace') }}</span> <i class="fas fa-robot"></i> <span>{{ t('loras.contextMenu.linkModelSource') }}</span>
</div> </div>
</div> </div>
</div> </div>
+1 -1
View File
@@ -21,7 +21,7 @@
<i class="fas fa-external-link-alt"></i> <span>{{ t('loras.contextMenu.linkCivitai') }}</span> <i class="fas fa-external-link-alt"></i> <span>{{ t('loras.contextMenu.linkCivitai') }}</span>
</div> </div>
<div class="context-menu-item" data-action="link-hf"> <div class="context-menu-item" data-action="link-hf">
<i class="fas fa-robot"></i> <span>{{ t('loras.contextMenu.linkHuggingFace') }}</span> <i class="fas fa-robot"></i> <span>{{ t('loras.contextMenu.linkModelSource') }}</span>
</div> </div>
</div> </div>
</div> </div>
+13 -8
View File
@@ -1,24 +1,29 @@
<!-- Link to HuggingFace Modal --> <!-- Link to Model Source Modal -->
<div id="linkHfModal" class="modal"> <div id="linkHfModal" class="modal">
<div class="modal-content"> <div class="modal-content">
<button class="close" onclick="modalManager.closeModal('linkHfModal')">&times;</button> <button class="close" onclick="modalManager.closeModal('linkHfModal')">&times;</button>
<h2>{{ t('modals.linkHuggingFace.title') }}</h2> <h2>{{ t('modals.linkModelSource.title') }}</h2>
<div class="warning-box"> <div class="warning-box">
<i class="fas fa-info-circle"></i> <i class="fas fa-info-circle"></i>
<p>{{ t('modals.linkHuggingFace.infoText') }}</p> <p>{{ t('modals.linkModelSource.infoText') }}</p>
</div> </div>
<div class="input-group"> <div class="input-group">
<label for="hfModelUrl">{{ t('modals.linkHuggingFace.urlLabel') }}</label> <label for="hfModelUrl">{{ t('modals.linkModelSource.urlLabel') }}</label>
<input type="text" id="hfModelUrl" placeholder="{{ t('modals.linkHuggingFace.urlPlaceholder') }}" /> <input type="text" id="hfModelUrl" placeholder="{{ t('modals.linkModelSource.urlPlaceholder') }}" />
<div class="input-error" id="hfModelUrlError"></div> <div class="input-error" id="hfModelUrlError"></div>
<div class="input-help"> <div class="input-help">
{{ t('modals.linkHuggingFace.helpText') }}<br> {{ t('modals.linkModelSource.helpText') }}
<strong>https://huggingface.co/user/repo</strong> <div id="hfSupportedSources">
<strong>https://huggingface.co/user/repo</strong><br>
<strong>https://modelscope.cn/models/user/repo</strong><br>
<strong>https://tensor.art/models/827823520299086029</strong>
</div>
{{ t('modals.linkModelSource.enrichNote') }}
</div> </div>
</div> </div>
<div class="modal-actions"> <div class="modal-actions">
<button class="cancel-btn" onclick="modalManager.closeModal('linkHfModal')">{{ t('common.actions.cancel') }}</button> <button class="cancel-btn" onclick="modalManager.closeModal('linkHfModal')">{{ t('common.actions.cancel') }}</button>
<button class="confirm-btn" id="confirmLinkHfBtn">{{ t('modals.linkHuggingFace.confirmAction') }}</button> <button class="confirm-btn" id="confirmLinkHfBtn">{{ t('modals.linkModelSource.confirmAction') }}</button>
</div> </div>
</div> </div>
</div> </div>
+1 -1
View File
@@ -21,7 +21,7 @@
<i class="fas fa-external-link-alt"></i> <span>{{ t('loras.contextMenu.linkCivitai') }}</span> <i class="fas fa-external-link-alt"></i> <span>{{ t('loras.contextMenu.linkCivitai') }}</span>
</div> </div>
<div class="context-menu-item" data-action="link-hf"> <div class="context-menu-item" data-action="link-hf">
<i class="fas fa-robot"></i> <span>{{ t('loras.contextMenu.linkHuggingFace') }}</span> <i class="fas fa-robot"></i> <span>{{ t('loras.contextMenu.linkModelSource') }}</span>
</div> </div>
</div> </div>
</div> </div>
+1 -1
View File
@@ -80,7 +80,7 @@
<i class="fas fa-external-link-alt"></i> <span>{{ t('loras.contextMenu.linkCivitai') }}</span> <i class="fas fa-external-link-alt"></i> <span>{{ t('loras.contextMenu.linkCivitai') }}</span>
</div> </div>
<div class="context-menu-item" data-action="link-hf"> <div class="context-menu-item" data-action="link-hf">
<i class="fas fa-robot"></i> <span>{{ t('loras.contextMenu.linkHuggingFace') }}</span> <i class="fas fa-robot"></i> <span>{{ t('loras.contextMenu.linkModelSource') }}</span>
</div> </div>
</div> </div>
</div> </div>
@@ -191,4 +191,60 @@ describe('ModelCard source globe (#1094)', () => {
expect(openHuggingFace).toHaveBeenCalledWith('https://huggingface.co/user/repo'); expect(openHuggingFace).toHaveBeenCalledWith('https://huggingface.co/user/repo');
expect(openCivitai).not.toHaveBeenCalled(); expect(openCivitai).not.toHaveBeenCalled();
}); });
it('points the globe at ModelScope for a ModelScope-linked model', () => {
const card = mountCard(
createModelCard,
makeModel({
from_civitai: false,
civitai: {},
source_platform: 'modelscope',
source_url: 'https://modelscope.cn/models/user/repo',
})
);
expect(card.dataset.has_civitai).toBe('false');
expect(card.dataset.source_platform).toBe('modelscope');
expect(card.dataset.hf_url).toBe('');
expect(card.querySelector('.fa-globe').getAttribute('title')).toBe('View on ModelScope');
});
it('opens the ModelScope page when the globe is clicked', () => {
const openSpy = vi.spyOn(window, 'open').mockImplementation(() => {});
const card = mountCard(
createModelCard,
makeModel({
from_civitai: false,
civitai: {},
source_platform: 'modelscope',
source_url: 'https://modelscope.cn/models/user/repo',
})
);
setupModelCardEventDelegation('loras');
card.querySelector('.fa-globe').dispatchEvent(new MouseEvent('click', { bubbles: true }));
expect(openSpy).toHaveBeenCalledWith(
'https://modelscope.cn/models/user/repo',
'_blank',
'noopener,noreferrer'
);
expect(openCivitai).not.toHaveBeenCalled();
expect(openHuggingFace).not.toHaveBeenCalled();
openSpy.mockRestore();
});
it('points the globe at TensorArt for a TensorArt-linked model', () => {
const card = mountCard(
createModelCard,
makeModel({
from_civitai: false,
civitai: {},
source_platform: 'tensorart',
source_url: 'https://tensor.art/models/827823520299086029',
})
);
expect(card.querySelector('.fa-globe').getAttribute('title')).toBe('View on TensorArt');
});
}); });
@@ -1,4 +1,4 @@
import { describe, expect, it } from 'vitest'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { ModelContextMenuMixin } from '../../../static/js/components/ContextMenu/ModelContextMenuMixin.js'; import { ModelContextMenuMixin } from '../../../static/js/components/ContextMenu/ModelContextMenuMixin.js';
@@ -14,3 +14,115 @@ describe('ModelContextMenuMixin.getModelTypePrefix', () => {
expect(ModelContextMenuMixin.getModelTypePrefix.call({})).toBe('loras'); expect(ModelContextMenuMixin.getModelTypePrefix.call({})).toBe('loras');
}); });
}); });
describe('ModelContextMenuMixin.updateEnrichMenuItem', () => {
function setupMenu() {
document.body.innerHTML = '<div id="menu"><div data-action="enrich-hf-llm"></div></div>';
return { menu: document.getElementById('menu') };
}
function cardWith(dataset) {
return { dataset };
}
it('enables enrichment for Hugging Face links', () => {
const context = setupMenu();
ModelContextMenuMixin.updateEnrichMenuItem.call(
context,
cardWith({ hf_url: 'https://huggingface.co/user/repo' })
);
const item = context.menu.querySelector('[data-action="enrich-hf-llm"]');
expect(item.classList.contains('disabled')).toBe(false);
expect(item.title).toBe('');
});
it('enables enrichment for ModelScope links', () => {
const context = setupMenu();
ModelContextMenuMixin.updateEnrichMenuItem.call(
context,
cardWith({
source_platform: 'modelscope',
source_url: 'https://modelscope.cn/models/user/repo',
})
);
const item = context.menu.querySelector('[data-action="enrich-hf-llm"]');
expect(item.classList.contains('disabled')).toBe(false);
});
it('disables enrichment for TensorArt and explains why', () => {
const context = setupMenu();
ModelContextMenuMixin.updateEnrichMenuItem.call(
context,
cardWith({
source_platform: 'tensorart',
source_url: 'https://tensor.art/models/827823520299086029',
})
);
const item = context.menu.querySelector('[data-action="enrich-hf-llm"]');
expect(item.classList.contains('disabled')).toBe(true);
expect(item.title).toContain('TensorArt');
});
it('disables enrichment when no source is linked', () => {
const context = setupMenu();
ModelContextMenuMixin.updateEnrichMenuItem.call(context, cardWith({}));
const item = context.menu.querySelector('[data-action="enrich-hf-llm"]');
expect(item.classList.contains('disabled')).toBe(true);
expect(item.title).toContain('Link this model to a model source');
});
});
describe('ModelContextMenuMixin._renderSupportedSources', () => {
const originalFetch = global.fetch;
beforeEach(() => {
document.body.innerHTML = '<div id="hfSupportedSources">static fallback</div>';
});
afterEach(() => {
global.fetch = originalFetch;
});
it('renders the server-provided example URLs', async () => {
global.fetch = vi.fn().mockResolvedValue({
ok: true,
json: async () => [
{ platform: 'huggingface', example_url: 'https://huggingface.co/user/repo' },
{ platform: 'modelscope', example_url: 'https://modelscope.cn/models/user/repo' },
{ platform: 'tensorart', example_url: 'https://tensor.art/models/123' },
],
});
await ModelContextMenuMixin._renderSupportedSources.call({});
const html = document.getElementById('hfSupportedSources').innerHTML;
expect(html).toContain('https://huggingface.co/user/repo');
expect(html).toContain('https://modelscope.cn/models/user/repo');
expect(html).toContain('https://tensor.art/models/123');
});
it('keeps the static fallback when the request fails', async () => {
global.fetch = vi.fn().mockRejectedValue(new Error('offline'));
await ModelContextMenuMixin._renderSupportedSources.call({});
expect(document.getElementById('hfSupportedSources').innerHTML).toBe('static fallback');
});
it('escapes markup from the server payload', async () => {
global.fetch = vi.fn().mockResolvedValue({
ok: true,
json: async () => [{ example_url: '<img src=x onerror=alert(1)>' }],
});
await ModelContextMenuMixin._renderSupportedSources.call({});
const html = document.getElementById('hfSupportedSources').innerHTML;
expect(html).not.toContain('<img');
expect(html).toContain('&lt;img');
});
});
@@ -0,0 +1,177 @@
import { describe, it, expect, vi } from 'vitest';
const { I18N_MODULE } = vi.hoisted(() => ({
I18N_MODULE: new URL('../../../static/js/utils/i18nHelpers.js', import.meta.url).pathname,
}));
vi.mock(I18N_MODULE, () => ({
translate: vi.fn((key, params, fallback) => (typeof fallback === 'string' ? fallback : key)),
}));
const {
MODEL_SOURCES,
parseModelSourceUrl,
getModelSource,
getModelSourceInfo,
getModelSourceUrl,
getModelSourceGroupKey,
canEnrichModelSource,
getModelSourceViewTitle,
parseModelSourceGroupKey,
openModelSource,
} = await import('../../../static/js/utils/modelSourceHelpers.js');
describe('modelSourceHelpers', () => {
it('exposes one descriptor per supported platform', () => {
expect(MODEL_SOURCES.map((s) => s.platform)).toEqual([
'huggingface',
'modelscope',
'tensorart',
]);
});
describe('parseModelSourceUrl', () => {
it('recognises Hugging Face URLs', () => {
const info = parseModelSourceUrl('https://huggingface.co/user/repo');
expect(info.platform).toBe('huggingface');
expect(info.sourceId).toBe('user/repo');
});
it('recognises ModelScope URLs with view sub-paths', () => {
const info = parseModelSourceUrl('https://modelscope.cn/models/user/repo/summary');
expect(info.platform).toBe('modelscope');
expect(info.sourceId).toBe('user/repo');
expect(info.url).toBe('https://modelscope.cn/models/user/repo');
});
it('recognises TensorArt URLs and keeps only the numeric id', () => {
const info = parseModelSourceUrl(
'https://tensor.art/models/827823520299086029/Vivid-Impressions-Storybook-Sstyle-V1.0'
);
expect(info.platform).toBe('tensorart');
expect(info.sourceId).toBe('827823520299086029');
expect(info.url).toBe('https://tensor.art/models/827823520299086029');
});
it('rejects unsupported URLs', () => {
expect(parseModelSourceUrl('https://example.com/x')).toBeNull();
expect(parseModelSourceUrl('')).toBeNull();
expect(parseModelSourceUrl(null)).toBeNull();
});
});
describe('getModelSourceInfo', () => {
it('falls back to the legacy hf_url field', () => {
const info = getModelSourceInfo({ hf_url: 'https://huggingface.co/user/repo' });
expect(info.platform).toBe('huggingface');
expect(info.sourceId).toBe('user/repo');
});
it('prefers the canonical source fields', () => {
const info = getModelSourceInfo({
source_platform: 'modelscope',
source_url: 'https://modelscope.cn/models/user/repo',
hf_url: 'https://huggingface.co/old/repo',
});
expect(info.platform).toBe('modelscope');
});
it('returns null when there is no source', () => {
expect(getModelSourceInfo({})).toBeNull();
expect(getModelSourceInfo({ hf_url: '' })).toBeNull();
});
});
describe('getModelSourceUrl', () => {
it('reads source_url then hf_url', () => {
expect(getModelSourceUrl({ source_url: 'https://a.example/1' })).toBe('https://a.example/1');
expect(getModelSourceUrl({ hf_url: 'https://huggingface.co/u/r' })).toBe(
'https://huggingface.co/u/r'
);
expect(getModelSourceUrl({})).toBe('');
});
});
describe('getModelSourceGroupKey', () => {
it('matches the backend group-key shapes', () => {
expect(getModelSourceGroupKey({ hf_url: 'https://huggingface.co/u/r' })).toBe('hf:u/r');
expect(
getModelSourceGroupKey({ source_url: 'https://modelscope.cn/models/u/r' })
).toBe('ms:u/r');
expect(getModelSourceGroupKey({ source_url: 'https://tensor.art/models/123' })).toBe(
'ta:123'
);
});
it('returns an empty string without a source', () => {
expect(getModelSourceGroupKey({})).toBe('');
});
});
describe('canEnrichModelSource', () => {
it('allows Hugging Face and ModelScope', () => {
expect(canEnrichModelSource({ hf_url: 'https://huggingface.co/u/r' })).toBe(true);
expect(
canEnrichModelSource({ source_url: 'https://modelscope.cn/models/u/r' })
).toBe(true);
});
it('disallows TensorArt and unlinked models', () => {
expect(canEnrichModelSource({ source_url: 'https://tensor.art/models/123' })).toBe(false);
expect(canEnrichModelSource({})).toBe(false);
});
});
describe('getModelSourceViewTitle', () => {
it('uses the branded label for non-HF sources', () => {
expect(getModelSourceViewTitle(getModelSource('modelscope'))).toBe('View on ModelScope');
expect(getModelSourceViewTitle(getModelSource('tensorart'))).toBe('View on TensorArt');
});
it('keeps the historical Hugging Face title', () => {
expect(getModelSourceViewTitle(getModelSource('huggingface'))).toBe(
'View on Hugging Face'
);
});
});
describe('parseModelSourceGroupKey', () => {
it('parses every external group-key prefix', () => {
expect(parseModelSourceGroupKey('hf:user/repo')).toEqual({
platform: 'huggingface',
label: 'Hugging Face',
sourceId: 'user/repo',
});
expect(parseModelSourceGroupKey('ms:user/repo').platform).toBe('modelscope');
expect(parseModelSourceGroupKey('ta:123').platform).toBe('tensorart');
});
it('rejects numeric CivitAI model ids and unknown prefixes', () => {
expect(parseModelSourceGroupKey(222)).toBeNull();
expect(parseModelSourceGroupKey('222')).toBeNull();
expect(parseModelSourceGroupKey('unknown:1')).toBeNull();
expect(parseModelSourceGroupKey('')).toBeNull();
expect(parseModelSourceGroupKey(null)).toBeNull();
});
});
describe('openModelSource', () => {
it('opens the URL in a new tab', () => {
const openSpy = vi.spyOn(window, 'open').mockImplementation(() => {});
openModelSource('https://modelscope.cn/models/u/r');
expect(openSpy).toHaveBeenCalledWith(
'https://modelscope.cn/models/u/r',
'_blank',
'noopener,noreferrer'
);
openSpy.mockRestore();
});
it('does nothing without a URL', () => {
const openSpy = vi.spyOn(window, 'open').mockImplementation(() => {});
openModelSource('');
expect(openSpy).not.toHaveBeenCalled();
openSpy.mockRestore();
});
});
});
+152
View File
@@ -154,3 +154,155 @@ async def test_set_hf_url_rejects_non_repo_url(tmp_path, hf_env):
payload = _json_payload(response) payload = _json_payload(response)
assert payload["success"] is False assert payload["success"] is False
hf_env["cache_write"].assert_not_awaited() hf_env["cache_write"].assert_not_awaited()
# ---------------------------------------------------------------------------
# Multi-source linking (ModelScope / TensorArt)
# ---------------------------------------------------------------------------
async def _write_plain_model(model_path, sha: str = "d" * 64) -> None:
await _write_model(
model_path,
{
"file_name": "model",
"model_name": "model",
"file_path": str(model_path),
"size": 32,
"modified": 1.0,
"sha256": sha,
"base_model": "Unknown",
"preview_url": "",
},
)
@pytest.mark.asyncio
async def test_set_hf_url_accepts_modelscope_and_stores_source_fields(tmp_path, hf_env):
model_path = tmp_path / "ms_model.safetensors"
await _write_plain_model(model_path)
response = await HfHandler().set_hf_url(
FakeRequest(
json_data={
"file_path": str(model_path),
"source_url": "https://modelscope.cn/models/jj3550945163/Krea-2-LORA",
}
)
)
assert response.status == 200
payload = _json_payload(response)
assert payload["source_platform"] == "modelscope"
assert payload["source_url"] == "https://modelscope.cn/models/jj3550945163/Krea-2-LORA"
saved = json.loads(open(_sidecar_path(model_path), encoding="utf-8").read())
assert saved["source_platform"] == "modelscope"
assert saved["source_url"] == "https://modelscope.cn/models/jj3550945163/Krea-2-LORA"
# No stale Hugging Face alias for a ModelScope model.
assert saved.get("hf_url", "") == ""
cached_metadata = hf_env["cache_write"].await_args.args[1]
assert cached_metadata["source_platform"] == "modelscope"
@pytest.mark.asyncio
async def test_set_hf_url_accepts_tensorart_url(tmp_path, hf_env):
model_path = tmp_path / "ta_model.safetensors"
await _write_plain_model(model_path, sha="e" * 64)
response = await HfHandler().set_hf_url(
FakeRequest(
json_data={
"file_path": str(model_path),
"source_url": (
"https://tensor.art/models/827823520299086029/"
"Vivid-Impressions-Storybook-Sstyle-V1.0"
),
}
)
)
assert response.status == 200
payload = _json_payload(response)
assert payload["source_platform"] == "tensorart"
# The canonical page URL is stored, without the slug.
assert payload["source_url"] == "https://tensor.art/models/827823520299086029"
@pytest.mark.asyncio
async def test_set_hf_url_canonicalises_modelscope_subpage(tmp_path, hf_env):
model_path = tmp_path / "ms_sub.safetensors"
await _write_plain_model(model_path, sha="f" * 64)
response = await HfHandler().set_hf_url(
FakeRequest(
json_data={
"file_path": str(model_path),
"source_url": "https://modelscope.cn/models/user/repo/summary",
}
)
)
assert response.status == 200
assert _json_payload(response)["source_url"] == "https://modelscope.cn/models/user/repo"
@pytest.mark.asyncio
async def test_set_hf_url_is_idempotent_for_modelscope(tmp_path, hf_env):
model_path = tmp_path / "ms_twice.safetensors"
await _write_plain_model(model_path, sha="1" * 64)
request = FakeRequest(
json_data={
"file_path": str(model_path),
"source_url": "https://modelscope.cn/models/user/repo",
}
)
await HfHandler().set_hf_url(request)
await HfHandler().set_hf_url(request)
# The second call short-circuits without rewriting the cache entry.
assert hf_env["cache_write"].await_count == 1
@pytest.mark.asyncio
async def test_set_hf_url_switching_source_clears_hf_alias(tmp_path, hf_env):
model_path = tmp_path / "switch.safetensors"
await _write_plain_model(model_path, sha="2" * 64)
await HfHandler().set_hf_url(
FakeRequest(
json_data={
"file_path": str(model_path),
"source_url": "https://huggingface.co/user/repo",
}
)
)
await HfHandler().set_hf_url(
FakeRequest(
json_data={
"file_path": str(model_path),
"source_url": "https://modelscope.cn/models/user/repo",
}
)
)
saved = json.loads(open(_sidecar_path(model_path), encoding="utf-8").read())
assert saved["source_platform"] == "modelscope"
assert saved.get("hf_url", "") == ""
@pytest.mark.asyncio
async def test_get_model_sources_lists_capabilities():
response = await HfHandler().get_model_sources(FakeRequest())
sources = _json_payload(response)
by_platform = {s["platform"]: s for s in sources}
assert set(by_platform) == {"huggingface", "modelscope", "tensorart"}
assert by_platform["huggingface"]["supports_enrichment"] is True
assert by_platform["modelscope"]["supports_enrichment"] is True
# TensorArt is link-only: no accessible model card for the backend.
assert by_platform["tensorart"]["supports_enrichment"] is False
assert by_platform["modelscope"]["supports_download"] is False
assert all(s["example_url"] for s in sources)
@@ -0,0 +1,182 @@
"""Tests for source-aware AI enrichment orchestration.
Covers the fast-fail gate (:meth:`AgentService._enrichment_skip_reason`) and
the prompt-context builder for non-Hugging Face model sources.
"""
from __future__ import annotations
from unittest import mock
import pytest
from py.services.agent.agent_service import AgentService
class TestEnrichmentSkipReason:
def test_skips_when_no_source_linked(self):
reason = AgentService._enrichment_skip_reason({})
assert "source_url" in reason
def test_allows_huggingface(self):
assert (
AgentService._enrichment_skip_reason(
{"hf_url": "https://huggingface.co/user/repo"}
)
== ""
)
def test_allows_modelscope(self):
assert (
AgentService._enrichment_skip_reason(
{
"source_platform": "modelscope",
"source_url": "https://modelscope.cn/models/user/repo",
}
)
== ""
)
def test_skips_tensorart_with_reason(self):
reason = AgentService._enrichment_skip_reason(
{
"source_platform": "tensorart",
"source_url": "https://tensor.art/models/827823520299086029",
}
)
assert "TensorArt" in reason
assert "not available" in reason
def test_skips_unknown_platform(self):
reason = AgentService._enrichment_skip_reason(
{"source_platform": "somewhere", "source_url": "https://somewhere.example/m/1"}
)
assert "somewhere" in reason
class TestBuildPromptContext:
@pytest.mark.asyncio
async def test_modelscope_card_populates_source_variables(self):
service = AgentService()
readme = "---\nbase_model: krea/Krea-2-Turbo\n---\n# krea\n"
with (
mock.patch(
"py.services.model_sources.modelscope.ModelScopeSource.fetch_model_card",
new=mock.AsyncMock(return_value=readme),
) as mock_fetch,
mock.patch(
"py.metadata_ops.list_base_models",
new=mock.AsyncMock(return_value=["Krea 2 Turbo"]),
),
mock.patch(
"py.metadata_ops.identify_model_type",
new=mock.AsyncMock(return_value="lora"),
),
mock.patch(
"py.services.settings_manager.SettingsManager.get_priority_tag_config",
return_value={"lora": "style, subject"},
),
):
context = await service._build_prompt_context(
skill_name="enrich_hf_metadata",
model_path="/models/loras/krea.safetensors",
metadata={
"source_platform": "modelscope",
"source_url": "https://modelscope.cn/models/jj3550945163/Krea-2-LORA",
"file_name": "krea",
},
registry=mock.Mock(),
llm=mock.Mock(),
)
mock_fetch.assert_awaited_once_with("jj3550945163/Krea-2-LORA")
assert context["source_platform"] == "modelscope"
assert context["source_id"] == "jj3550945163/Krea-2-LORA"
assert context["source_label"] == "ModelScope"
assert (
context["asset_base_url"]
== "https://modelscope.cn/models/jj3550945163/Krea-2-LORA/resolve/master"
)
assert readme in context["readme_content_full"]
# Hugging Face aliases stay empty for a non-HF source.
assert context["hf_url"] == ""
assert context["repo"] == "jj3550945163/Krea-2-LORA"
@pytest.mark.asyncio
async def test_huggingface_keeps_legacy_aliases(self):
service = AgentService()
readme = "# card\n"
with (
mock.patch(
"py.services.model_sources.huggingface.HuggingFaceSource.fetch_model_card",
new=mock.AsyncMock(return_value=readme),
) as mock_fetch,
mock.patch(
"py.metadata_ops.list_base_models",
new=mock.AsyncMock(return_value=[]),
),
mock.patch(
"py.metadata_ops.identify_model_type",
new=mock.AsyncMock(return_value="lora"),
),
mock.patch(
"py.services.settings_manager.SettingsManager.get_priority_tag_config",
return_value={},
),
):
context = await service._build_prompt_context(
skill_name="enrich_hf_metadata",
model_path="/models/loras/thing.safetensors",
metadata={"hf_url": "https://huggingface.co/user/repo"},
registry=mock.Mock(),
llm=mock.Mock(),
)
mock_fetch.assert_awaited_once_with("user/repo")
assert context["source_platform"] == "huggingface"
assert context["hf_url"] == "https://huggingface.co/user/repo"
assert context["repo"] == "user/repo"
@pytest.mark.asyncio
async def test_tensorart_never_fetches_a_card(self):
service = AgentService()
with (
mock.patch(
"py.services.model_sources.huggingface.HuggingFaceSource.fetch_model_card",
new=mock.AsyncMock(),
) as hf_fetch,
mock.patch(
"py.services.model_sources.modelscope.ModelScopeSource.fetch_model_card",
new=mock.AsyncMock(),
) as ms_fetch,
mock.patch(
"py.metadata_ops.list_base_models",
new=mock.AsyncMock(return_value=[]),
),
mock.patch(
"py.metadata_ops.identify_model_type",
new=mock.AsyncMock(return_value="lora"),
),
mock.patch(
"py.services.settings_manager.SettingsManager.get_priority_tag_config",
return_value={},
),
):
context = await service._build_prompt_context(
skill_name="enrich_hf_metadata",
model_path="/models/loras/thing.safetensors",
metadata={
"source_platform": "tensorart",
"source_url": "https://tensor.art/models/827823520299086029",
},
registry=mock.Mock(),
llm=mock.Mock(),
)
hf_fetch.assert_not_awaited()
ms_fetch.assert_not_awaited()
assert context["readme_content"] == ""
assert context["source_platform"] == "tensorart"
+2
View File
@@ -964,6 +964,8 @@ def _make_cache_entry(**overrides) -> Dict[str, Any]:
"civitai": {"id": 111, "modelId": 222, "name": "v1"}, "civitai": {"id": 111, "modelId": 222, "name": "v1"},
"civitai_deleted": False, "civitai_deleted": False,
"skip_metadata_refresh": False, "skip_metadata_refresh": False,
"source_platform": "",
"source_url": "",
"hf_url": "", "hf_url": "",
"license_flags": 113, "license_flags": 113,
"hash_status": "completed", "hash_status": "completed",
+324
View File
@@ -0,0 +1,324 @@
"""Tests for the external model-source provider registry.
Covers URL recognition for Hugging Face / ModelScope / TensorArt, the
legacy ``hf_url`` ``source_url`` normalisation, version-group keys, and
each provider's model-card fetching and capability flags.
"""
from __future__ import annotations
import pytest
from py.services.model_sources import (
HuggingFaceSource,
ModelScopeSource,
TensorArtSource,
detect_source,
get_source,
get_source_platform,
has_external_source,
list_sources,
normalize_metadata_source,
resolve_source_ref,
source_group_key,
source_label,
)
# ---------------------------------------------------------------------------
# URL recognition
# ---------------------------------------------------------------------------
class TestDetectSource:
@pytest.mark.parametrize(
("url", "platform", "source_id"),
[
("https://huggingface.co/user/repo", "huggingface", "user/repo"),
("https://www.huggingface.co/user/repo", "huggingface", "user/repo"),
(
"https://huggingface.co/user/repo/resolve/main/model.safetensors",
"huggingface",
"user/repo",
),
(
"https://modelscope.cn/models/jj3550945163/Krea-2-LORA",
"modelscope",
"jj3550945163/Krea-2-LORA",
),
(
"https://www.modelscope.cn/models/jj3550945163/Krea-2-LORA/summary",
"modelscope",
"jj3550945163/Krea-2-LORA",
),
(
"https://tensor.art/models/827823520299086029/Vivid-Impressions-Storybook-Sstyle-V1.0",
"tensorart",
"827823520299086029",
),
("https://tusi.cn/models/827823520299086029", "tensorart", "827823520299086029"),
],
)
def test_recognises_supported_urls(self, url, platform, source_id):
ref = detect_source(url)
assert ref is not None
assert ref.platform == platform
assert ref.source_id == source_id
@pytest.mark.parametrize(
"url",
[
"",
None,
"not-a-url",
"https://example.com/x",
"https://civitai.com/models/123",
],
)
def test_ignores_unsupported_urls(self, url):
assert detect_source(url) is None
def test_canonical_url_is_stable(self):
assert detect_source("https://huggingface.co/u/r").url == "https://huggingface.co/u/r"
assert (
detect_source("https://modelscope.cn/models/u/r/summary").url
== "https://modelscope.cn/models/u/r"
)
assert (
detect_source("https://tensor.art/models/123/some-slug").url
== "https://tensor.art/models/123"
)
class TestStrictParsing:
@pytest.mark.parametrize(
"url",
[
"https://huggingface.co/user/repo",
"https://huggingface.co/user/repo/",
"https://modelscope.cn/models/user/repo",
"https://modelscope.cn/models/user/repo/summary",
"https://tensor.art/models/827823520299086029",
"https://tensor.art/models/827823520299086029/Vivid-Impressions",
],
)
def test_accepts_user_facing_urls(self, url):
assert detect_source(url, strict=True) is not None
@pytest.mark.parametrize(
"url",
[
"https://huggingface.co/user/repo/resolve/main/model.safetensors",
"https://example.com/x",
"https://tensor.art/models/not-a-number",
],
)
def test_rejects_non_page_urls(self, url):
assert detect_source(url, strict=True) is None
# ---------------------------------------------------------------------------
# Capabilities
# ---------------------------------------------------------------------------
class TestCapabilities:
def test_huggingface_supports_everything(self):
source = get_source("huggingface")
assert source.supports_enrichment is True
assert source.supports_download is True
def test_modelscope_supports_enrichment_but_not_download(self):
source = get_source("modelscope")
assert source.supports_enrichment is True
assert source.supports_download is False
def test_tensorart_is_link_only(self):
source = get_source("tensorart")
assert source.supports_enrichment is False
assert source.supports_download is False
def test_registry_lists_every_source(self):
platforms = {s.platform for s in list_sources()}
assert platforms == {"huggingface", "modelscope", "tensorart"}
def test_labels_are_brand_names(self):
assert source_label("huggingface") == "Hugging Face"
assert source_label("modelscope") == "ModelScope"
assert source_label("tensorart") == "TensorArt"
assert source_label("unknown", "fallback") == "fallback"
# ---------------------------------------------------------------------------
# Metadata normalisation
# ---------------------------------------------------------------------------
class TestNormalizeMetadataSource:
def test_derives_source_fields_from_legacy_hf_url(self):
metadata = {"hf_url": "https://huggingface.co/user/repo"}
normalize_metadata_source(metadata)
assert metadata["source_platform"] == "huggingface"
assert metadata["source_url"] == "https://huggingface.co/user/repo"
assert metadata["hf_url"] == "https://huggingface.co/user/repo"
def test_canonicalises_modelscope_url_and_clears_hf_alias(self):
metadata = {
"source_platform": "modelscope",
"source_url": "https://modelscope.cn/models/user/repo/summary",
"hf_url": "https://huggingface.co/old/repo",
}
normalize_metadata_source(metadata)
assert metadata["source_platform"] == "modelscope"
assert metadata["source_url"] == "https://modelscope.cn/models/user/repo"
# A stale HF alias must not make a ModelScope model look like HF.
assert metadata["hf_url"] == ""
def test_preserves_unknown_url_for_unknown_platform(self):
metadata = {"source_url": "https://example.com/model/1", "source_platform": "other"}
normalize_metadata_source(metadata)
assert metadata["source_url"] == "https://example.com/model/1"
assert metadata["source_platform"] == "other"
def test_empty_metadata_gets_default_fields(self):
metadata: dict = {}
normalize_metadata_source(metadata)
assert metadata["source_platform"] == ""
assert metadata["source_url"] == ""
def test_infers_platform_from_url_when_missing(self):
metadata = {"source_url": "https://modelscope.cn/models/user/repo"}
normalize_metadata_source(metadata)
assert metadata["source_platform"] == "modelscope"
class TestResolveSourceRef:
def test_resolves_from_canonical_fields(self):
ref = resolve_source_ref(
{"source_platform": "modelscope", "source_url": "https://modelscope.cn/models/u/r"}
)
assert ref is not None
assert ref.platform == "modelscope"
assert ref.source_id == "u/r"
def test_resolves_from_legacy_hf_url(self):
ref = resolve_source_ref({"hf_url": "https://huggingface.co/u/r"})
assert ref is not None
assert ref.platform == "huggingface"
def test_returns_none_without_any_source(self):
assert resolve_source_ref({}) is None
assert resolve_source_ref({"hf_url": ""}) is None
class TestHelpers:
def test_has_external_source_accepts_both_field_shapes(self):
assert has_external_source({"hf_url": "https://huggingface.co/u/r"}) is True
assert has_external_source({"source_url": "https://modelscope.cn/models/u/r"}) is True
assert has_external_source({"source_url": ""}) is False
assert has_external_source({}) is False
def test_get_source_platform_infers_from_url(self):
assert get_source_platform({"hf_url": "https://huggingface.co/u/r"}) == "huggingface"
assert get_source_platform({"source_platform": "tensorart"}) == "tensorart"
assert get_source_platform({}) == ""
def test_group_keys_match_legacy_hf_shape(self):
assert source_group_key({"hf_url": "https://huggingface.co/u/r"}) == "hf:u/r"
assert (
source_group_key({"source_url": "https://modelscope.cn/models/u/r"}) == "ms:u/r"
)
assert (
source_group_key({"source_url": "https://tensor.art/models/123"}) == "ta:123"
)
def test_group_key_is_none_without_source(self):
assert source_group_key({}) is None
assert source_group_key({"hf_url": "https://example.com/x"}) is None
# ---------------------------------------------------------------------------
# Model card fetching
# ---------------------------------------------------------------------------
class TestFetchModelCard:
@pytest.mark.asyncio
async def test_huggingface_tries_main_then_master(self, monkeypatch):
calls: list[str] = []
async def fake_fetch_text(url: str, **_kwargs) -> str:
calls.append(url)
if url.endswith("/master/README.md"):
return "# card"
return ""
monkeypatch.setattr("py.services.model_sources.huggingface.fetch_text", fake_fetch_text)
card = await HuggingFaceSource().fetch_model_card("user/repo")
assert card == "# card"
assert calls == [
"https://huggingface.co/user/repo/raw/main/README.md",
"https://huggingface.co/user/repo/raw/master/README.md",
]
@pytest.mark.asyncio
async def test_modelscope_prefers_resolve_url(self, monkeypatch):
calls: list[str] = []
async def fake_fetch_text(url: str, **_kwargs) -> str:
calls.append(url)
return "---\nbase_model: krea/Krea-2-Turbo\n---\n# krea"
monkeypatch.setattr("py.services.model_sources.modelscope.fetch_text", fake_fetch_text)
card = await ModelScopeSource().fetch_model_card("u/r")
assert card.startswith("---")
assert calls == ["https://modelscope.cn/models/u/r/resolve/master/README.md"]
@pytest.mark.asyncio
async def test_modelscope_falls_back_to_repo_api(self, monkeypatch):
calls: list[str] = []
async def fake_fetch_text(url: str, **_kwargs) -> str:
calls.append(url)
if "/api/v1/models/" in url:
return "# from api"
return ""
monkeypatch.setattr("py.services.model_sources.modelscope.fetch_text", fake_fetch_text)
card = await ModelScopeSource().fetch_model_card("u/r")
assert card == "# from api"
assert "resolve/master/README.md" in calls[0]
assert (
"https://modelscope.cn/api/v1/models/u/r/repo?Revision=master&FilePath=README.md"
in calls
)
@pytest.mark.asyncio
async def test_tensorart_never_fetches(self):
# TensorArt enrichment is disabled: the provider must not issue any
# HTTP request, so it deliberately does not import `fetch_text`.
import importlib
module = importlib.import_module("py.services.model_sources.tensorart")
assert not hasattr(module, "fetch_text")
assert await TensorArtSource().fetch_model_card("123") == ""
class TestAssetBaseUrl:
def test_huggingface_uses_main_revision(self):
assert (
HuggingFaceSource().asset_base_url("u/r")
== "https://huggingface.co/u/r/resolve/main"
)
def test_modelscope_uses_master_revision(self):
assert (
ModelScopeSource().asset_base_url("u/r")
== "https://modelscope.cn/models/u/r/resolve/master"
)
+55
View File
@@ -292,6 +292,61 @@ Content
) )
assert images[0]["meta"]["prompt"] == "a cat" assert images[0]["meta"]["prompt"] == "a cat"
@pytest.mark.asyncio
async def test_gallery_images_use_modelscope_asset_base_url(self, processor):
"""A ModelScope-linked model resolves relative images against ModelScope."""
readme = """---
widget:
- text: "a cat"
output:
url: images/cat.png
---
Content
"""
with (
mock.patch("py.metadata_ops.apply_metadata_updates") as mock_apply,
mock.patch("py.metadata_ops.download_preview", return_value=None),
mock.patch("py.metadata_ops.refresh_cache"),
):
await processor.process(
skill_name="enrich_hf_metadata",
model_path="/p.safetensors",
llm_output=self.MIN_LLM_OUTPUT,
metadata={
"from_civitai": False,
"source_platform": "modelscope",
"source_url": "https://modelscope.cn/models/user/repo",
},
readme_content=readme,
)
applied = mock_apply.call_args[0][1]
images = applied.get("civitai", {}).get("images", [])
assert len(images) == 1
assert images[0]["url"] == (
"https://modelscope.cn/models/user/repo/resolve/master/images/cat.png"
)
@pytest.mark.asyncio
async def test_base_model_overwrites_existing_modelscope_model(self, processor):
"""ModelScope is an external source, so the LLM may overwrite base_model."""
llm = {**self.MIN_LLM_OUTPUT, "base_model": "Flux.1 D"}
with (
mock.patch("py.metadata_ops.apply_metadata_updates") as mock_apply,
mock.patch("py.metadata_ops.download_preview", return_value=False),
mock.patch("py.metadata_ops.refresh_cache"),
):
await processor.process(
skill_name="enrich_hf_metadata",
model_path="/p.safetensors",
llm_output=llm,
metadata={
"base_model": "SD 1.5",
"source_platform": "modelscope",
"source_url": "https://modelscope.cn/models/user/repo",
},
)
assert mock_apply.call_args[0][1]["base_model"] == "Flux.1 D"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_gallery_images_skipped_without_hf_url(self, processor): async def test_gallery_images_skipped_without_hf_url(self, processor):
"""Gallery images NOT extracted when the model has no HF source.""" """Gallery images NOT extracted when the model has no HF source."""