Compare commits

...

11 Commits

Author SHA1 Message Date
Will Miao
8b344ea39f feat(ui): add View on Hugging Face button, plumb hf_url through full cache pipeline 2026-07-01 08:38:16 +08:00
Will Miao
8348a0cef8 fix(download): harden HF download path validation, fix WebSocket leak, add URL detection tests (#965, #977)
Security hardening:
- Validate repo format with strict regex (reject .. traversal)
- Validate filename rejects path separators and ..
- Validate relative_path rejects absolute paths and ..
- Verify model_root is within configured scanner roots using
  realpath + os.sep guard to prevent prefix-match bypass
- Add realpath-based escape detection for final dest_path

Bug fixes:
- Fix WebSocket leak in _downloadHfSingle: wrap ws.close() in
  try/finally so it closes even if downloadHfModel() throws
- Same fix for batch HF download per-file WebSocket loop

Frontend hardening:
- Tighten HF repo regex: require huggingface.co for full URLs,
  reject bare .. patterns
- Add 12 unit tests for detectUrlType() covering HF resolve,
  HF repo, CivitAI, CivArchive, direct HTTP, edge cases
2026-07-01 05:51:58 +08:00
Will Miao
7cf785b72f fix(ui): unify HF file selection UI, remove cloud icon, add select-all, cleanup dead code (#965, #977)
- Unify single-URL and multi-URL HF repo flows to use the same batch
  preview interface (remove separate repoFileStep)
- Remove unnecessary cloud icon from HF batch preview items
- Use formatFileSize() instead of hardcoded MB text
- Change default selection to unchecked (no preselected files)
- Add select all / deselect all checkbox with dynamic Next button
- Clean up dead CSS, HTML template, and JS methods from removed
  repoFileStep
- Add selectAll i18n key with translations for all 10 locales
- Fix batch progress bar name fallback for HF items
2026-06-30 23:28:35 +08:00
Will Miao
e8913f4481 feat(ui): dynamically populate base model dropdown from CivitAI API, add Krea 2 constants (#1001) 2026-06-30 22:41:17 +08:00
Will Miao
f9c3d8dc97 fix(metadata): demote CivArchive hash lookup failure from ERROR to DEBUG
A model not being found on CivArchive by hash is a routine case (the
model simply isn't published there), not an error. The callers already
log the outcome at WARNING (bulk_metadata_refresh) or DEBUG
(metadata_sync_service) with full context, making this ERROR-level log
both misleading and redundant.
2026-06-30 19:42:30 +08:00
Will Miao
09ca91fc0e feat(download): add Hugging Face model download to standalone UI wizard (#965, #977)
Integrate HF model downloading into the existing CivitAI-style wizard flow:
- URL type detection (civitai / hf-resolve / hf-repo / direct-http)
- Repo file explorer with checkbox-based file selection
- Batch/queue download with per-file WebSocket progress
- Aria2 backend support (respects download_backend setting)
- Scanner cache integration via create_default_metadata + add_model_to_cache
- i18n updates for all 10 locales
2026-06-30 19:36:12 +08:00
Will Miao
16f5222efd fix(cache): prevent corrupted cache rows from breaking model listings (#730)
Cache corruption (NULL model_name/file_name from legacy DB rows or partial
writes) caused format_response to raise KeyError/AttributeError, failing the
entire /loras/list request and showing no models in the UI.

Fix across three layers:
- format_response (lora/checkpoint/embedding): replace direct dict[] access
  with .get() fallbacks; return None for entries missing file_path
- handlers: filter None entries from list/excluded/fetch/duplicate/conflict
  endpoints instead of letting them crash or appear as null in responses
- model_scanner: always use validate_batch repaired copies (previously
  discarded when no invalid entries, leaving None values in raw_data)
- persistent_model_cache: add or-empty-string guards on read and write for
  nullable TEXT columns (model_name, file_name, folder, base_model, etc.)
2026-06-30 09:02:42 +08:00
Will Miao
28e7c04b37 fix(settings): migrate all settings subdirectories on portable mode switch 2026-06-29 21:40:37 +08:00
Will Miao
28f99c46d3 fix(update): preserve user data dirs during Git-based update via git clean -e excludes
git clean -fd in _perform_git_update deleted untracked, non-ignored
directories (wildcards, stats, backups, civitai, caches, logs) during
portable-mode updates, since released tags do not list them in .gitignore.
Add -e excludes for all user-managed paths to both nightly and stable
update branches. Add regression tests for both paths.
2026-06-29 21:10:38 +08:00
Will Miao
205194f4e6 chore: add stats, wildcards, backups, and logs dirs to .gitignore 2026-06-29 19:46:04 +08:00
willmiao
402d8b07cf docs: auto-update supporters list in README 2026-06-28 14:17:19 +00:00
43 changed files with 21029 additions and 19334 deletions

4
.gitignore vendored
View File

@@ -7,6 +7,10 @@ py/run_test.py
.vscode/ .vscode/
cache/ cache/
civitai/ civitai/
stats/
wildcards/
backups/
logs/
node_modules/ node_modules/
coverage/ coverage/
.coverage .coverage

File diff suppressed because one or more lines are too long

File diff suppressed because it is too large Load Diff

View File

@@ -105,6 +105,7 @@
"removeFromFavorites": "Remove from favorites", "removeFromFavorites": "Remove from favorites",
"viewOnCivitai": "View on Civitai", "viewOnCivitai": "View on Civitai",
"notAvailableFromCivitai": "Not available from Civitai", "notAvailableFromCivitai": "Not available from Civitai",
"viewOnHuggingFace": "View on Hugging Face",
"sendToWorkflow": "Send to ComfyUI (Click: Append, Shift+Click: Replace)", "sendToWorkflow": "Send to ComfyUI (Click: Append, Shift+Click: Replace)",
"copyLoRASyntax": "Copy LoRA Syntax", "copyLoRASyntax": "Copy LoRA Syntax",
"checkpointNameCopied": "Checkpoint name copied", "checkpointNameCopied": "Checkpoint name copied",
@@ -1134,7 +1135,10 @@
"titleWithType": "Download {type} from URL", "titleWithType": "Download {type} from URL",
"civitaiUrl": "Civitai URL(s):", "civitaiUrl": "Civitai URL(s):",
"placeholder": "https://civitai.com/models/...", "placeholder": "https://civitai.com/models/...",
"urlHint": "Enter one CivitAI or CivArchive URL per line. Supports multiple URLs for batch download.", "urlHint": "Enter one CivitAI, CivArchive, or Hugging Face URL per line. Supports multiple URLs for batch download.",
"selectHfFiles": "Select file(s) to download from this repository:",
"selectAll": "Select All",
"fetchingRepoFiles": "Fetching repository files...",
"locationPreview": "Download Location Preview", "locationPreview": "Download Location Preview",
"useDefaultPath": "Use Default Path", "useDefaultPath": "Use Default Path",
"useDefaultPathTooltip": "When enabled, files are automatically organized using configured path templates", "useDefaultPathTooltip": "When enabled, files are automatically organized using configured path templates",
@@ -1163,7 +1167,9 @@
}, },
"errors": { "errors": {
"invalidUrl": "Invalid Civitai URL format", "invalidUrl": "Invalid Civitai URL format",
"noVersions": "No versions available for this model" "noVersions": "No versions available for this model",
"mixedSources": "Cannot mix CivitAI and Hugging Face URLs in the same batch.",
"noModelFiles": "No model files found in this repository."
}, },
"status": { "status": {
"preparing": "Preparing download...", "preparing": "Preparing download...",
@@ -1314,6 +1320,8 @@
"editVersionName": "Edit version name", "editVersionName": "Edit version name",
"viewOnCivitai": "View on Civitai", "viewOnCivitai": "View on Civitai",
"viewOnCivitaiText": "View on Civitai", "viewOnCivitaiText": "View on Civitai",
"viewOnHuggingFace": "View on Hugging Face",
"viewOnHuggingFaceText": "View on Hugging Face",
"viewCreatorProfile": "View Creator Profile", "viewCreatorProfile": "View Creator Profile",
"openFileLocation": "Open File Location", "openFileLocation": "Open File Location",
"sendToWorkflow": "Send to ComfyUI", "sendToWorkflow": "Send to ComfyUI",

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,409 @@
"""Handlers for Hugging Face model listing and download.
Minimal MVP implementation — uses direct HTTP to the HF API for file
listing and the project's existing aiohttp-based Downloader for
downloading. No huggingface_hub dependency required.
"""
from __future__ import annotations
import json
import logging
import os
import re
from typing import Any
import aiohttp
from aiohttp import web
from ...config import config
from ...services.downloader import (
DownloadProgress,
get_downloader,
)
from ...services.aria2_downloader import Aria2Downloader
from ...services.settings_manager import get_settings_manager
from ...services.service_registry import ServiceRegistry
from ...services.websocket_manager import ws_manager
from ...utils.constants import MODEL_FILE_EXTENSIONS
from ...utils.metadata_manager import MetadataManager
from ...utils.models import LoraMetadata, CheckpointMetadata, EmbeddingMetadata
logger = logging.getLogger(__name__)
_DEFAULT_MODEL_CLASS = LoraMetadata
_DEFAULT_SCANNER_GETTER = "get_lora_scanner"
# Shared aiohttp session for HF API calls (created on first use)
_hf_api_session: aiohttp.ClientSession | None = None
async def _get_hf_api_session() -> aiohttp.ClientSession:
"""Get or create the shared aiohttp session for HF API calls."""
global _hf_api_session # needed because we reassign the module-level name
if _hf_api_session is None or _hf_api_session.closed:
_hf_api_session = aiohttp.ClientSession(
headers={"User-Agent": "ComfyUI-LoRA-Manager/1.0"},
timeout=aiohttp.ClientTimeout(total=30),
)
return _hf_api_session
def _infer_model_type(model_root: str) -> tuple[Any, str]:
"""Determine model class and scanner by matching ``model_root`` against the
configured root paths for each model type (from ``Config``).
The ``model_root`` value comes from the frontend's model-root dropdown,
which is populated from the current page's scanner roots. By checking
which scanner's root list it belongs to, we avoid fragile heuristics
like substring-matching path names.
"""
norm = os.path.normpath(model_root).replace(os.sep, "/")
# LoRA roots
for p in (config.loras_roots or []) + (config.extra_loras_roots or []):
if os.path.normpath(p).replace(os.sep, "/") == norm:
return LoraMetadata, "get_lora_scanner"
# Checkpoint / UNet roots
for p in (
(config.checkpoints_roots or [])
+ (config.extra_checkpoints_roots or [])
+ (config.unet_roots or [])
+ (config.extra_unet_roots or [])
):
if os.path.normpath(p).replace(os.sep, "/") == norm:
return CheckpointMetadata, "get_checkpoint_scanner"
# Embedding roots
for p in (config.embeddings_roots or []) + (config.extra_embeddings_roots or []):
if os.path.normpath(p).replace(os.sep, "/") == norm:
return EmbeddingMetadata, "get_embedding_scanner"
# Fallback — should not happen in normal use
logger.warning(
"Could not determine model type for root '%s'; defaulting to LoRA",
model_root,
)
return _DEFAULT_MODEL_CLASS, _DEFAULT_SCANNER_GETTER
async def _save_hf_metadata(dest_path: str, repo: str, model_root: str) -> None:
"""Create a proper .metadata.json and add the model to the scanner cache.
Uses ``MetadataManager.create_default_metadata()`` which computes the
SHA256 hash, extracts safetensors header metadata (base_model), and
produces a fully-populated ``LoraMetadata`` (or ``CheckpointMetadata`` /
``EmbeddingMetadata``) object. We then overlay HF-specific fields and
register the model in the in-memory scanner cache so it appears
immediately without a full filesystem walk.
"""
try:
hf_url = f"https://huggingface.co/{repo}"
model_class, scanner_getter_name = _infer_model_type(model_root)
# 1. Create proper metadata (computes SHA256, reads safetensors headers)
metadata = await MetadataManager.create_default_metadata(
dest_path, model_class=model_class
)
if metadata is None:
logger.warning("create_default_metadata returned None for %s", dest_path)
return
# 2. Overlay HF-specific fields
metadata._unknown_fields["hf_url"] = hf_url
metadata.from_civitai = False # HF models are not from CivitAI
# 3. Save metadata atomically
await MetadataManager.save_metadata(dest_path, metadata)
logger.info("Saved HF metadata (with hf_url) for %s", dest_path)
# 4. Determine relative folder path for cache
# model_root is an absolute path; dest_path is under it
folder = ""
if os.path.isabs(model_root) and dest_path.startswith(model_root):
rel = os.path.relpath(os.path.dirname(dest_path), model_root)
folder = rel.replace(os.sep, "/") if rel != "." else ""
# 5. Add to scanner cache (same as CivitAI's _execute_download does)
scanner_getter = getattr(ServiceRegistry, scanner_getter_name, None)
if scanner_getter is not None:
scanner = await scanner_getter()
if scanner is not None:
metadata_dict = metadata.to_dict()
metadata_dict["hf_url"] = hf_url
await scanner.add_model_to_cache(metadata_dict, folder)
logger.info("Added %s to scanner cache (folder=%s)", dest_path, folder)
except Exception as exc:
logger.warning("Failed to save HF metadata for %s: %s", dest_path, exc)
class HfHandler:
"""Handle Hugging Face model browsing and download."""
async def get_hf_repo_files(self, request: web.Request) -> web.Response:
"""List model-weight files from a HF repo with real file sizes.
Uses the HF tree API endpoint which returns accurate file sizes
(including LFS-tracked files), unlike the model info endpoint.
"""
repo = request.query.get("repo", "").strip()
if not repo or "/" not in repo:
return web.json_response(
{"error": "Missing or invalid 'repo' parameter (expected user/repo)"},
status=400,
)
url = f"https://huggingface.co/api/models/{repo}/tree/main"
try:
session = await _get_hf_api_session()
async with session.get(url) as resp:
if resp.status == 404:
return web.json_response(
{"error": f"Repo '{repo}' not found"}, status=404
)
if resp.status != 200:
text = await resp.text()
return web.json_response(
{"error": f"HF API error {resp.status}: {text[:200]}"},
status=resp.status,
)
tree: list[dict[str, Any]] = await resp.json()
except Exception as exc:
logger.error("Failed to fetch HF repo files: %s", exc)
return web.json_response({"error": str(exc)}, status=502)
files: list[dict[str, Any]] = []
for entry in tree:
path: str = entry.get("path", "")
ext = os.path.splitext(path)[1].lower()
if ext not in MODEL_FILE_EXTENSIONS:
continue
size = entry.get("size", 0) or 0
if size == 0 and "lfs" in entry:
size = entry["lfs"].get("size", 0) or 0
files.append({
"filename": path,
"size": size,
})
files.sort(key=lambda f: f["size"], reverse=True)
return web.json_response(files)
async def download_hf_model(self, request: web.Request) -> web.Response:
"""Download a single file from Hugging Face into the model directory.
POST JSON body::
{
"repo": "dx8152/Flux2-Klein-9B-Consistency",
"filename": "Flux2-Klein-9B-consistency-V2.safetensors",
"revision": "main",
"model_root": "loras",
"relative_path": "",
"use_default_paths": false,
"download_id": "optional-batch-id"
}
If ``download_id`` is provided, real-time progress (bytes, speed,
percentage) is broadcast via the WebSocket progress system, matching
the CivitAI download experience.
Respects the ``download_backend`` setting (``aria2`` or ``default``).
"""
try:
payload: dict[str, Any] = await request.json()
except json.JSONDecodeError:
return web.json_response({"error": "Invalid JSON"}, status=400)
repo = (payload.get("repo") or "").strip()
filename = (payload.get("filename") or "").strip()
revision = (payload.get("revision") or "main").strip()
model_root = (payload.get("model_root") or "").strip()
relative_path = (payload.get("relative_path") or "").strip()
use_default_paths = bool(payload.get("use_default_paths", False))
download_id: str | None = payload.get("download_id")
logger.info(
"download_hf_model: repo=%s file=%s root=%s download_id=%s",
repo, filename, model_root, download_id,
)
if not repo or not filename:
return web.json_response(
{"error": "Missing required fields: 'repo' and 'filename'"}, status=400
)
# Validate repo format — must be user/repo_name
if repo.count("/") != 1 or not re.match(r"^[a-zA-Z0-9_.-]+/[a-zA-Z0-9_.-]+$", repo):
return web.json_response({"error": f"Invalid repo format: {repo}"}, status=400)
author, repo_name = repo.split("/", 1)
if ".." in (author, repo_name) or "." in (author, repo_name):
return web.json_response({"error": f"Invalid repo format: {repo}"}, status=400)
# Validate filename — must not contain path separators or ..
if "/" in filename or "\\" in filename or ".." in filename:
return web.json_response({"error": "Invalid filename"}, status=400)
# Validate relative_path — must not be absolute or escape base directory
if relative_path:
if os.path.isabs(relative_path):
return web.json_response({"error": "relative_path must not be absolute"}, status=400)
if ".." in relative_path.split("/") or "\\" in relative_path:
return web.json_response({"error": "Invalid relative_path"}, status=400)
# Validate model_root — must not contain path traversal
if not os.path.isabs(model_root):
# For relative model_root, check it doesn't escape
resolved_model_root = os.path.realpath(
os.path.join(os.getcwd(), "models", model_root)
)
else:
resolved_model_root = os.path.realpath(model_root)
# Verify model_root is within a configured scanner root
allowed_roots = set()
for root_list in (
config.loras_roots or [],
config.extra_loras_roots or [],
config.checkpoints_roots or [],
config.extra_checkpoints_roots or [],
config.unet_roots or [],
config.extra_unet_roots or [],
config.embeddings_roots or [],
config.extra_embeddings_roots or [],
):
for r in root_list:
allowed_roots.add(os.path.realpath(r))
if not any(resolved_model_root == root or resolved_model_root.startswith(root + os.sep) for root in allowed_roots):
logger.warning("Invalid model_root rejected: %s", model_root)
return web.json_response({"error": f"Invalid model_root: {model_root}"}, status=400)
base_dir = resolved_model_root
if use_default_paths:
target_dir = os.path.join(base_dir, "huggingface", author, repo_name)
elif relative_path:
target_dir = os.path.join(base_dir, relative_path)
else:
target_dir = base_dir
os.makedirs(target_dir, exist_ok=True)
dest_path = os.path.join(target_dir, filename)
# Resolve symlinks and check for path traversal escape
real_dest = os.path.realpath(dest_path)
real_base = os.path.realpath(target_dir)
if not real_dest.startswith(real_base + os.sep):
logger.warning("Path traversal blocked: %s -> %s", dest_path, real_dest)
return web.json_response({"error": "Path traversal detected"}, status=400)
# Check if already exists (simple skip)
if os.path.exists(dest_path) and os.path.getsize(dest_path) > 0:
logger.info("download_hf_model: file already exists, skipping — %s", dest_path)
return web.json_response({
"success": True,
"message": f"File already exists: {dest_path}",
"path": dest_path,
})
# Build HF resolve URL
resolve_url = (
f"https://huggingface.co/{repo}/resolve/{revision}/{filename}"
)
# Set up progress callback if download_id is provided
progress_callback = None
if download_id:
async def _progress_callback(
progress: float | DownloadProgress,
snapshot: DownloadProgress | None = None,
) -> None:
percent = 0.0
metrics = snapshot if isinstance(snapshot, DownloadProgress) else None
if isinstance(progress, DownloadProgress):
percent = progress.percent_complete
metrics = progress
elif isinstance(snapshot, DownloadProgress):
percent = snapshot.percent_complete
else:
percent = float(progress)
broadcast: dict[str, Any] = {
"status": "progress",
"progress": round(percent),
}
if metrics:
broadcast["bytes_downloaded"] = metrics.bytes_downloaded
broadcast["total_bytes"] = metrics.total_bytes
broadcast["bytes_per_second"] = metrics.bytes_per_second
await ws_manager.broadcast_download_progress(download_id, broadcast)
progress_callback = _progress_callback
# Respect download backend setting (aria2 vs default)
download_backend = (
get_settings_manager().get("download_backend", "default")
)
if download_backend == "aria2":
aria2 = await Aria2Downloader.get_instance()
aid = download_id or f"hf_{repo}_{filename}"
try:
hf_success, hf_result = await aria2.download_file(
url=resolve_url,
save_path=dest_path,
download_id=aid,
progress_callback=progress_callback,
)
if hf_success:
await _save_hf_metadata(dest_path, repo, model_root)
return web.json_response({
"success": True,
"message": f"Downloaded to {dest_path}",
"path": dest_path,
})
else:
return web.json_response(
{"success": False, "error": hf_result or "aria2 download failed"},
status=500,
)
except Exception as exc:
logger.error("HF download (aria2) failed: %s", exc)
return web.json_response(
{"success": False, "error": str(exc)}, status=500
)
# Default: use built-in aiohttp Downloader
downloader = await get_downloader()
try:
success, result = await downloader.download_file(
url=resolve_url,
save_path=dest_path,
use_auth=False,
allow_resume=True,
progress_callback=progress_callback,
)
if success:
await _save_hf_metadata(dest_path, repo, model_root)
return web.json_response({
"success": True,
"message": f"Downloaded to {result}",
"path": result,
})
else:
return web.json_response(
{"success": False, "error": result or "Download failed"},
status=500,
)
except Exception as exc:
logger.error("HF download failed: %s", exc)
return web.json_response(
{"success": False, "error": str(exc)}, status=500
)

View File

@@ -48,6 +48,7 @@ from ...utils.constants import (
SUPPORTED_MEDIA_EXTENSIONS, SUPPORTED_MEDIA_EXTENSIONS,
VALID_LORA_TYPES, VALID_LORA_TYPES,
) )
from .hf_handlers import HfHandler
from ...utils.civitai_utils import rewrite_preview_url from ...utils.civitai_utils import rewrite_preview_url
from ...utils.example_images_paths import ( from ...utils.example_images_paths import (
find_non_compliant_items_in_example_images_root, find_non_compliant_items_in_example_images_root,
@@ -3315,6 +3316,7 @@ class MiscHandlerSet:
doctor: DoctorHandler, doctor: DoctorHandler,
example_workflows: ExampleWorkflowsHandler, example_workflows: ExampleWorkflowsHandler,
base_model: BaseModelHandlerSet, base_model: BaseModelHandlerSet,
hf_handler: HfHandler | None = None,
) -> None: ) -> None:
self.health = health self.health = health
self.settings = settings self.settings = settings
@@ -3333,6 +3335,7 @@ class MiscHandlerSet:
self.doctor = doctor self.doctor = doctor
self.example_workflows = example_workflows self.example_workflows = example_workflows
self.base_model = base_model self.base_model = base_model
self.hf_handler = hf_handler
def to_route_mapping( def to_route_mapping(
self, self,
@@ -3378,6 +3381,9 @@ class MiscHandlerSet:
"get_supporters": self.supporters.get_supporters, "get_supporters": self.supporters.get_supporters,
"get_example_workflows": self.example_workflows.get_example_workflows, "get_example_workflows": self.example_workflows.get_example_workflows,
"get_example_workflow": self.example_workflows.get_example_workflow, "get_example_workflow": self.example_workflows.get_example_workflow,
# Hugging Face handlers
"get_hf_repo_files": self.hf_handler.get_hf_repo_files,
"download_hf_model": self.hf_handler.download_hf_model,
# Base model handlers # Base model handlers
"get_base_models": self.base_model.get_base_models, "get_base_models": self.base_model.get_base_models,
"refresh_base_models": self.base_model.refresh_base_models, "refresh_base_models": self.base_model.refresh_base_models,

View File

@@ -203,11 +203,17 @@ class ModelListingHandler:
result = await self._service.get_paginated_data(**params) result = await self._service.get_paginated_data(**params)
format_start = time.perf_counter() format_start = time.perf_counter()
formatted_raw = [
await self._service.format_response(entry)
for entry in result["items"]
]
# Filter out None entries returned for corrupted cache rows (issue #730).
# Note: "total" intentionally remains the pre-filter count to reflect
# the true number of models in the cache; corrupted entries are rare
# and adjusting total would cause pagination drift on every page.
formatted_items = [item for item in formatted_raw if item is not None]
formatted_result = { formatted_result = {
"items": [ "items": formatted_items,
await self._service.format_response(item)
for item in result["items"]
],
"total": result["total"], "total": result["total"],
"page": result["page"], "page": result["page"],
"page_size": result["page_size"], "page_size": result["page_size"],
@@ -238,11 +244,15 @@ class ModelListingHandler:
result = await self._service.get_excluded_paginated_data(**params) result = await self._service.get_excluded_paginated_data(**params)
format_start = time.perf_counter() format_start = time.perf_counter()
formatted_raw = [
await self._service.format_response(entry)
for entry in result["items"]
]
# Filter out None entries returned for corrupted cache rows (issue #730).
# "total" stays at the pre-filter count; see get_models for rationale.
formatted_items = [item for item in formatted_raw if item is not None]
formatted_result = { formatted_result = {
"items": [ "items": formatted_items,
await self._service.format_response(item)
for item in result["items"]
],
"total": result["total"], "total": result["total"],
"page": result["page"], "page": result["page"],
"page_size": result["page_size"], "page_size": result["page_size"],
@@ -533,8 +543,13 @@ class ModelManagementHandler:
if not success: if not success:
return web.json_response({"success": False, "error": error}) return web.json_response({"success": False, "error": error})
formatted_metadata = await self._service.format_response(model_data) formatted = await self._service.format_response(model_data)
return web.json_response({"success": True, "metadata": formatted_metadata}) if formatted is None:
return web.json_response(
{"success": False, "error": "Model entry is corrupted (missing file_path)"},
status=500,
)
return web.json_response({"success": True, "metadata": formatted})
except Exception as exc: except Exception as exc:
if is_expected_offline_error(str(exc)): if is_expected_offline_error(str(exc)):
return web.json_response( return web.json_response(
@@ -1091,10 +1106,12 @@ class ModelQueryHandler:
# Sort: originals first, copies last # Sort: originals first, copies last
sorted_models = self._sort_duplicate_group(filtered) sorted_models = self._sort_duplicate_group(filtered)
# Format response # Format response, filtering out corrupted entries (issue #730)
group = {"hash": sha256, "models": []} group = {"hash": sha256, "models": []}
for model in sorted_models: for model in sorted_models:
group["models"].append(await self._service.format_response(model)) formatted = await self._service.format_response(model)
if formatted is not None:
group["models"].append(formatted)
# Only include groups with 2+ models after filtering # Only include groups with 2+ models after filtering
if len(group["models"]) > 1: if len(group["models"]) > 1:
@@ -1211,9 +1228,9 @@ class ModelQueryHandler:
(m for m in cache.raw_data if m["file_path"] == path), None (m for m in cache.raw_data if m["file_path"] == path), None
) )
if model: if model:
group["models"].append( formatted = await self._service.format_response(model)
await self._service.format_response(model) if formatted is not None:
) group["models"].append(formatted)
hash_val = self._service.scanner.get_hash_by_filename(filename) hash_val = self._service.scanner.get_hash_by_filename(filename)
if hash_val: if hash_val:
main_path = self._service.get_path_by_hash(hash_val) main_path = self._service.get_path_by_hash(hash_val)
@@ -1223,9 +1240,9 @@ class ModelQueryHandler:
None, None,
) )
if main_model: if main_model:
group["models"].insert( formatted = await self._service.format_response(main_model)
0, await self._service.format_response(main_model) if formatted is not None:
) group["models"].insert(0, formatted)
if group["models"]: if group["models"]:
result.append(group) result.append(group)
return web.json_response( return web.json_response(

View File

@@ -94,6 +94,13 @@ MISC_ROUTE_DEFINITIONS: tuple[RouteDefinition, ...] = (
RouteDefinition( RouteDefinition(
"GET", "/api/lm/delete-model-version", "delete_model_version" "GET", "/api/lm/delete-model-version", "delete_model_version"
), ),
# Hugging Face model endpoints
RouteDefinition(
"GET", "/api/lm/hf-repo-files", "get_hf_repo_files"
),
RouteDefinition(
"POST", "/api/lm/download-hf-model", "download_hf_model"
),
) )

View File

@@ -39,6 +39,7 @@ from .handlers.misc_handlers import (
build_service_registry_adapter, build_service_registry_adapter,
) )
from .handlers.base_model_handlers import BaseModelHandlerSet from .handlers.base_model_handlers import BaseModelHandlerSet
from .handlers.hf_handlers import HfHandler
from .misc_route_registrar import MiscRouteRegistrar from .misc_route_registrar import MiscRouteRegistrar
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -136,6 +137,7 @@ class MiscRoutes:
doctor = DoctorHandler(settings_service=self._settings) doctor = DoctorHandler(settings_service=self._settings)
example_workflows = ExampleWorkflowsHandler() example_workflows = ExampleWorkflowsHandler()
base_model = BaseModelHandlerSet() base_model = BaseModelHandlerSet()
hf_handler = HfHandler()
return self._handler_set_factory( return self._handler_set_factory(
health=health, health=health,
@@ -155,6 +157,7 @@ class MiscRoutes:
doctor=doctor, doctor=doctor,
example_workflows=example_workflows, example_workflows=example_workflows,
base_model=base_model, base_model=base_model,
hf_handler=hf_handler,
) )

View File

@@ -16,6 +16,27 @@ logger = logging.getLogger(__name__)
NETWORK_EXCEPTIONS = (ClientError, OSError, asyncio.TimeoutError) NETWORK_EXCEPTIONS = (ClientError, OSError, asyncio.TimeoutError)
# User-managed directories that live inside the plugin folder (portable
# mode) and must survive a Git-based update. ``git clean -fd`` would
# otherwise delete them because they are untracked and, in released tags,
# not listed in ``.gitignore``. ``-e`` excludes a path from cleaning
# regardless of whether it is ignored.
_PRESERVE_DIRS = ('settings.json', 'civitai', 'wildcards', 'backups', 'stats', 'logs', 'cache', 'model_cache')
def _clean_excludes() -> List[str]:
"""Build the ``-e`` arguments for ``git clean`` from :data:`_PRESERVE_DIRS`."""
excludes: List[str] = []
for name in _PRESERVE_DIRS:
excludes.append('-e')
excludes.append(name)
# For directories, also exclude nested matches explicitly
# (``-e dir`` alone matches the dir entry; ``-e dir/**`` guards
# contents under all git versions as defense-in-depth).
excludes.append('-e')
excludes.append(f'{name}/**')
return excludes
class UpdateRoutes: class UpdateRoutes:
"""Routes for handling plugin update checks""" """Routes for handling plugin update checks"""
@@ -365,6 +386,8 @@ class UpdateRoutes:
) )
return False, "" return False, ""
clean_excludes = _clean_excludes()
try: try:
# Open the Git repository # Open the Git repository
repo = git.Repo(plugin_root) repo = git.Repo(plugin_root)
@@ -376,8 +399,9 @@ class UpdateRoutes:
if nightly: if nightly:
# Reset to discard any local changes # Reset to discard any local changes
repo.git.reset('--hard') repo.git.reset('--hard')
# Clean untracked files # Clean untracked files, but preserve user-managed directories
repo.git.clean('-fd') # (wildcards, backups, stats, civitai, caches, settings.json).
repo.git.clean('-fd', *clean_excludes)
# Switch to main branch and pull latest # Switch to main branch and pull latest
main_branch = 'main' main_branch = 'main'
@@ -394,8 +418,9 @@ class UpdateRoutes:
else: else:
# Reset to discard any local changes # Reset to discard any local changes
repo.git.reset('--hard') repo.git.reset('--hard')
# Clean untracked files # Clean untracked files, but preserve user-managed directories
repo.git.clean('-fd') # (wildcards, backups, stats, civitai, caches, settings.json).
repo.git.clean('-fd', *clean_excludes)
# Get latest release tag # Get latest release tag
tags = sorted(repo.tags, key=lambda t: t.commit.committed_datetime, reverse=True) tags = sorted(repo.tags, key=lambda t: t.commit.committed_datetime, reverse=True)

View File

@@ -791,8 +791,12 @@ class BaseModelService(ABC):
} }
@abstractmethod @abstractmethod
async def format_response(self, model_data: Dict) -> Dict: async def format_response(self, model_data: Dict) -> Optional[Dict]:
"""Format model data for API response - must be implemented by subclasses""" """Format model data for API response - must be implemented by subclasses.
Subclasses should return None for corrupted entries so the handler
layer can filter them out. See issue #730.
"""
pass pass
# Common service methods that delegate to scanner # Common service methods that delegate to scanner

View File

@@ -1,6 +1,6 @@
import os import os
import logging import logging
from typing import Dict from typing import Dict, Optional
from .base_model_service import BaseModelService from .base_model_service import BaseModelService
from .auto_tag_service import extract_auto_tags from .auto_tag_service import extract_auto_tags
@@ -21,20 +21,37 @@ class CheckpointService(BaseModelService):
""" """
super().__init__("checkpoint", scanner, CheckpointMetadata, update_service=update_service) super().__init__("checkpoint", scanner, CheckpointMetadata, update_service=update_service)
async def format_response(self, checkpoint_data: Dict) -> Dict: async def format_response(self, checkpoint_data: Dict) -> Optional[Dict]:
"""Format Checkpoint data for API response""" """Format Checkpoint data for API response.
Returns None when the entry is missing critical fields (corrupted cache
row), so the handler layer can filter it out. See issue #730.
"""
# Guard against corrupted cache entries missing critical fields
file_path = checkpoint_data.get("file_path")
if not file_path or not isinstance(file_path, str):
logger.warning(
"Skipping corrupted checkpoint entry (missing file_path): %s",
checkpoint_data.get("file_name", "<unknown>"),
)
return None
# Get sub_type from cache entry (new canonical field) # Get sub_type from cache entry (new canonical field)
sub_type = checkpoint_data.get("sub_type", "checkpoint") sub_type = checkpoint_data.get("sub_type", "checkpoint")
file_name = checkpoint_data.get("file_name") or ""
model_name = checkpoint_data.get("model_name") or file_name
folder = checkpoint_data.get("folder") or ""
return { return {
"model_name": checkpoint_data["model_name"], "model_name": model_name,
"file_name": checkpoint_data["file_name"], "file_name": file_name,
"preview_url": config.get_preview_static_url(checkpoint_data.get("preview_url", "")), "preview_url": config.get_preview_static_url(checkpoint_data.get("preview_url", "")),
"preview_nsfw_level": checkpoint_data.get("preview_nsfw_level", 0), "preview_nsfw_level": checkpoint_data.get("preview_nsfw_level", 0),
"base_model": checkpoint_data.get("base_model", ""), "base_model": checkpoint_data.get("base_model", ""),
"folder": checkpoint_data["folder"], "folder": folder,
"sha256": checkpoint_data.get("sha256", ""), "sha256": checkpoint_data.get("sha256", ""),
"file_path": checkpoint_data["file_path"].replace(os.sep, "/"), "file_path": file_path.replace(os.sep, "/"),
"file_size": checkpoint_data.get("size", 0), "file_size": checkpoint_data.get("size", 0),
"modified": checkpoint_data.get("modified", ""), "modified": checkpoint_data.get("modified", ""),
"tags": checkpoint_data.get("tags", []), "tags": checkpoint_data.get("tags", []),
@@ -49,6 +66,7 @@ class CheckpointService(BaseModelService):
"civitai": self.filter_civitai_data(checkpoint_data.get("civitai", {}), minimal=True), "civitai": self.filter_civitai_data(checkpoint_data.get("civitai", {}), minimal=True),
"auto_tags": checkpoint_data.get("auto_tags") or extract_auto_tags(checkpoint_data), "auto_tags": checkpoint_data.get("auto_tags") or extract_auto_tags(checkpoint_data),
"version_count": checkpoint_data.get("version_count"), "version_count": checkpoint_data.get("version_count"),
"hf_url": checkpoint_data.get("hf_url", ""),
} }
def find_duplicate_hashes(self) -> Dict: def find_duplicate_hashes(self) -> Dict:

View File

@@ -327,7 +327,7 @@ class CivArchiveClient:
if resolved: if resolved:
return resolved, None return resolved, None
logger.error("Error fetching version of CivArchive model by hash %s", model_hash[:10]) logger.debug("Error fetching version of CivArchive model by hash %s", model_hash[:10])
return None, "No version data found" return None, "No version data found"
except RateLimitError: except RateLimitError:

View File

@@ -196,6 +196,7 @@ class CivitaiBaseModelService:
"ernie": "ERNI", "ernie": "ERNI",
"ernie turbo": "ETRB", "ernie turbo": "ETRB",
"nucleus": "NUCL", "nucleus": "NUCL",
"krea 2": "KR2",
"svd": "SVD", "svd": "SVD",
"ltxv": "LTXV", "ltxv": "LTXV",
"ltxv2": "LTV2", "ltxv2": "LTV2",
@@ -424,6 +425,7 @@ class CivitaiBaseModelService:
"Ernie", "Ernie",
"Ernie Turbo", "Ernie Turbo",
"Nucleus", "Nucleus",
"Krea 2",
], ],
} }

View File

@@ -1,6 +1,6 @@
import os import os
import logging import logging
from typing import Dict from typing import Dict, Optional
from .base_model_service import BaseModelService from .base_model_service import BaseModelService
from .auto_tag_service import extract_auto_tags from .auto_tag_service import extract_auto_tags
@@ -21,20 +21,37 @@ class EmbeddingService(BaseModelService):
""" """
super().__init__("embedding", scanner, EmbeddingMetadata, update_service=update_service) super().__init__("embedding", scanner, EmbeddingMetadata, update_service=update_service)
async def format_response(self, embedding_data: Dict) -> Dict: async def format_response(self, embedding_data: Dict) -> Optional[Dict]:
"""Format Embedding data for API response""" """Format Embedding data for API response.
Returns None when the entry is missing critical fields (corrupted cache
row), so the handler layer can filter it out. See issue #730.
"""
# Guard against corrupted cache entries missing critical fields
file_path = embedding_data.get("file_path")
if not file_path or not isinstance(file_path, str):
logger.warning(
"Skipping corrupted embedding entry (missing file_path): %s",
embedding_data.get("file_name", "<unknown>"),
)
return None
# Get sub_type from cache entry (new canonical field) # Get sub_type from cache entry (new canonical field)
sub_type = embedding_data.get("sub_type", "embedding") sub_type = embedding_data.get("sub_type", "embedding")
file_name = embedding_data.get("file_name") or ""
model_name = embedding_data.get("model_name") or file_name
folder = embedding_data.get("folder") or ""
return { return {
"model_name": embedding_data["model_name"], "model_name": model_name,
"file_name": embedding_data["file_name"], "file_name": file_name,
"preview_url": config.get_preview_static_url(embedding_data.get("preview_url", "")), "preview_url": config.get_preview_static_url(embedding_data.get("preview_url", "")),
"preview_nsfw_level": embedding_data.get("preview_nsfw_level", 0), "preview_nsfw_level": embedding_data.get("preview_nsfw_level", 0),
"base_model": embedding_data.get("base_model", ""), "base_model": embedding_data.get("base_model", ""),
"folder": embedding_data["folder"], "folder": folder,
"sha256": embedding_data.get("sha256", ""), "sha256": embedding_data.get("sha256", ""),
"file_path": embedding_data["file_path"].replace(os.sep, "/"), "file_path": file_path.replace(os.sep, "/"),
"file_size": embedding_data.get("size", 0), "file_size": embedding_data.get("size", 0),
"modified": embedding_data.get("modified", ""), "modified": embedding_data.get("modified", ""),
"tags": embedding_data.get("tags", []), "tags": embedding_data.get("tags", []),
@@ -49,6 +66,7 @@ class EmbeddingService(BaseModelService):
"civitai": self.filter_civitai_data(embedding_data.get("civitai", {}), minimal=True), "civitai": self.filter_civitai_data(embedding_data.get("civitai", {}), minimal=True),
"auto_tags": embedding_data.get("auto_tags") or extract_auto_tags(embedding_data), "auto_tags": embedding_data.get("auto_tags") or extract_auto_tags(embedding_data),
"version_count": embedding_data.get("version_count"), "version_count": embedding_data.get("version_count"),
"hf_url": embedding_data.get("hf_url", ""),
} }
def find_duplicate_hashes(self) -> Dict: def find_duplicate_hashes(self) -> Dict:

View File

@@ -24,23 +24,41 @@ class LoraService(BaseModelService):
""" """
super().__init__("lora", scanner, LoraMetadata, update_service=update_service) super().__init__("lora", scanner, LoraMetadata, update_service=update_service)
async def format_response(self, lora_data: Dict) -> Dict: async def format_response(self, lora_data: Dict) -> Optional[Dict]:
"""Format LoRA data for API response""" """Format LoRA data for API response.
Returns None when the entry is missing critical fields (corrupted cache
row), so the handler layer can filter it out instead of crashing the
whole listing request. See issue #730.
"""
# Guard against corrupted cache entries missing critical fields
file_path = lora_data.get("file_path")
if not file_path or not isinstance(file_path, str):
logger.warning(
"Skipping corrupted LoRA entry (missing file_path): %s",
lora_data.get("file_name", "<unknown>"),
)
return None
# Resolve sub_type using priority: sub_type > model_type > civitai.model.type > default # Resolve sub_type using priority: sub_type > model_type > civitai.model.type > default
# Normalize to lowercase for consistent API responses # Normalize to lowercase for consistent API responses
sub_type = resolve_sub_type(lora_data).lower() sub_type = resolve_sub_type(lora_data).lower()
file_name = lora_data.get("file_name") or ""
model_name = lora_data.get("model_name") or file_name
folder = lora_data.get("folder") or ""
return { return {
"model_name": lora_data["model_name"], "model_name": model_name,
"file_name": lora_data["file_name"], "file_name": file_name,
"preview_url": config.get_preview_static_url( "preview_url": config.get_preview_static_url(
lora_data.get("preview_url", "") lora_data.get("preview_url", "")
), ),
"preview_nsfw_level": lora_data.get("preview_nsfw_level", 0), "preview_nsfw_level": lora_data.get("preview_nsfw_level", 0),
"base_model": lora_data.get("base_model", ""), "base_model": lora_data.get("base_model", ""),
"folder": lora_data["folder"], "folder": folder,
"sha256": lora_data.get("sha256", ""), "sha256": lora_data.get("sha256", ""),
"file_path": lora_data["file_path"].replace(os.sep, "/"), "file_path": file_path.replace(os.sep, "/"),
"file_size": lora_data.get("size", 0), "file_size": lora_data.get("size", 0),
"modified": lora_data.get("modified", ""), "modified": lora_data.get("modified", ""),
"tags": lora_data.get("tags", []), "tags": lora_data.get("tags", []),
@@ -60,6 +78,7 @@ class LoraService(BaseModelService):
), ),
"auto_tags": lora_data.get("auto_tags") or extract_auto_tags(lora_data), "auto_tags": lora_data.get("auto_tags") or extract_auto_tags(lora_data),
"version_count": lora_data.get("version_count"), "version_count": lora_data.get("version_count"),
"hf_url": lora_data.get("hf_url", ""),
} }
async def _apply_specific_filters(self, data: List[Dict], **kwargs) -> List[Dict]: async def _apply_specific_filters(self, data: List[Dict], **kwargs) -> List[Dict]:

View File

@@ -248,6 +248,7 @@ class ModelScanner:
'civitai': civitai_slim, 'civitai': civitai_slim,
'civitai_deleted': bool(get_value('civitai_deleted', False)), 'civitai_deleted': bool(get_value('civitai_deleted', False)),
'skip_metadata_refresh': bool(get_value('skip_metadata_refresh', False)), 'skip_metadata_refresh': bool(get_value('skip_metadata_refresh', False)),
'hf_url': get_value('hf_url', '') or '',
} }
license_source: Dict[str, Any] = {} license_source: Dict[str, Any] = {}
@@ -476,11 +477,20 @@ class ModelScanner:
for tag in adjusted_item.get('tags') or []: for tag in adjusted_item.get('tags') or []:
tags_count[tag] = tags_count.get(tag, 0) + 1 tags_count[tag] = tags_count.get(tag, 0) + 1
# Validate cache entries and check health # Validate cache entries and check health.
# Always use the validated/repaired entries — even when there are no
# invalid entries, auto_repair may have filled in missing optional
# fields (model_name, file_name, folder) with safe defaults on a copied
# working_entry. Without this unconditional replacement the repaired
# copies are discarded and None values propagate to format_response.
# See issue #730.
valid_entries, invalid_entries = CacheEntryValidator.validate_batch( valid_entries, invalid_entries = CacheEntryValidator.validate_batch(
adjusted_raw_data, auto_repair=True adjusted_raw_data, auto_repair=True
) )
# Always use the validated entries (repaired copies)
adjusted_raw_data = valid_entries
if invalid_entries: if invalid_entries:
monitor = CacheHealthMonitor() monitor = CacheHealthMonitor()
report = monitor.check_health(adjusted_raw_data, auto_repair=True) report = monitor.check_health(adjusted_raw_data, auto_repair=True)

View File

@@ -57,6 +57,7 @@ class PersistentModelCache:
"db_checked", "db_checked",
"last_checked_at", "last_checked_at",
"hash_status", "hash_status",
"hf_url",
) )
_MODEL_UPDATE_COLUMNS: Tuple[str, ...] = _MODEL_COLUMNS[2:] _MODEL_UPDATE_COLUMNS: Tuple[str, ...] = _MODEL_COLUMNS[2:]
_instances: Dict[str, "PersistentModelCache"] = {} _instances: Dict[str, "PersistentModelCache"] = {}
@@ -165,8 +166,8 @@ class PersistentModelCache:
item = { item = {
"file_path": file_path, "file_path": file_path,
"file_name": row["file_name"], "file_name": row["file_name"] or "",
"model_name": row["model_name"], "model_name": row["model_name"] or "",
"folder": row["folder"] or "", "folder": row["folder"] or "",
"size": row["size"] or 0, "size": row["size"] or 0,
"modified": row["modified"] or 0.0, "modified": row["modified"] or 0.0,
@@ -188,6 +189,7 @@ class PersistentModelCache:
"skip_metadata_refresh": bool(row["skip_metadata_refresh"]), "skip_metadata_refresh": bool(row["skip_metadata_refresh"]),
"license_flags": int(license_value), "license_flags": int(license_value),
"hash_status": row["hash_status"] or "completed", "hash_status": row["hash_status"] or "completed",
"hf_url": row["hf_url"] or "",
} }
raw_data.append(item) raw_data.append(item)
@@ -452,6 +454,7 @@ class PersistentModelCache:
db_checked INTEGER, db_checked INTEGER,
last_checked_at REAL, last_checked_at REAL,
hash_status TEXT, hash_status TEXT,
hf_url TEXT DEFAULT '',
PRIMARY KEY (model_type, file_path) PRIMARY KEY (model_type, file_path)
); );
@@ -500,6 +503,7 @@ class PersistentModelCache:
# Persisting without explicit flags should assume CivitAI's documented defaults (0b111001 == 57). # Persisting without explicit flags should assume CivitAI's documented defaults (0b111001 == 57).
"license_flags": f"INTEGER DEFAULT {DEFAULT_LICENSE_FLAGS}", "license_flags": f"INTEGER DEFAULT {DEFAULT_LICENSE_FLAGS}",
"hash_status": "TEXT DEFAULT 'completed'", "hash_status": "TEXT DEFAULT 'completed'",
"hf_url": "TEXT DEFAULT ''",
} }
for column, definition in required_columns.items(): for column, definition in required_columns.items():
@@ -548,19 +552,19 @@ class PersistentModelCache:
return ( return (
model_type, model_type,
item.get("file_path"), item.get("file_path"),
item.get("file_name"), item.get("file_name") or "",
item.get("model_name"), item.get("model_name") or "",
item.get("folder"), item.get("folder") or "",
int(item.get("size") or 0), int(item.get("size") or 0),
float(item.get("modified") or 0.0), float(item.get("modified") or 0.0),
(item.get("sha256") or "").lower() or None, (item.get("sha256") or "").lower() or None,
item.get("base_model"), item.get("base_model") or "",
item.get("preview_url"), item.get("preview_url") or "",
int(item.get("preview_nsfw_level") or 0), int(item.get("preview_nsfw_level") or 0),
1 if item.get("from_civitai", True) else 0, 1 if item.get("from_civitai", True) else 0,
1 if item.get("favorite") else 0, 1 if item.get("favorite") else 0,
item.get("notes"), item.get("notes") or "",
item.get("usage_tips"), item.get("usage_tips") or "",
metadata_source, metadata_source,
civitai.get("id"), civitai.get("id"),
civitai.get("modelId"), civitai.get("modelId"),
@@ -575,6 +579,7 @@ class PersistentModelCache:
1 if item.get("db_checked") else 0, 1 if item.get("db_checked") else 0,
float(item.get("last_checked_at") or 0.0), float(item.get("last_checked_at") or 0.0),
item.get("hash_status", "completed"), item.get("hash_status", "completed"),
item.get("hf_url") or "",
) )
def _insert_model_sql(self) -> str: def _insert_model_sql(self) -> str:

View File

@@ -1568,7 +1568,7 @@ class SettingsManager:
previous_dir = os.path.dirname(previous_path) or target_dir previous_dir = os.path.dirname(previous_path) or target_dir
if os.path.abspath(previous_path) != os.path.abspath(target_path): if os.path.abspath(previous_path) != os.path.abspath(target_path):
self._copy_model_cache_directory(previous_dir, target_dir) self._migrate_settings_directory_content(previous_dir, target_dir)
logger.info("Switching settings file to: %s", target_path) logger.info("Switching settings file to: %s", target_path)
self._pending_portable_switch = {"other_path": other_path} self._pending_portable_switch = {"other_path": other_path}
@@ -1603,46 +1603,52 @@ class SettingsManager:
finally: finally:
self._pending_portable_switch = None self._pending_portable_switch = None
def _copy_model_cache_directory(self, source_dir: str, target_dir: str) -> None: def _migrate_settings_directory_content(
"""Copy model_cache artifacts when switching storage locations.""" self, source_dir: str, target_dir: str
) -> None:
"""Migrate settings directory subdirectories when switching storage locations.
Copies the canonical subdirectories (cache, backups, logs, stats, wildcards)
from the old settings directory to the new one. Legacy cache artifacts
(model_cache, recipe_cache, etc.) are migrated lazily by
``resolve_cache_path_with_migration`` on first access.
Args:
source_dir: The previous settings directory path.
target_dir: The new settings directory path.
"""
if not source_dir or not target_dir: if not source_dir or not target_dir:
return return
source_cache_dir = os.path.join(source_dir, "model_cache") def _copy_dir(name: str) -> None:
target_cache_dir = os.path.join(target_dir, "model_cache") source = os.path.join(source_dir, name)
if os.path.isdir(source_cache_dir) and os.path.abspath( target = os.path.join(target_dir, name)
source_cache_dir if os.path.isdir(source) and os.path.abspath(source) != os.path.abspath(
) != os.path.abspath(target_cache_dir): target
try: ):
shutil.copytree( try:
source_cache_dir, shutil.copytree(
target_cache_dir, source,
dirs_exist_ok=True, target,
ignore=shutil.ignore_patterns("*.sqlite-shm", "*.sqlite-wal"), dirs_exist_ok=True,
) ignore=shutil.ignore_patterns("*.sqlite-shm", "*.sqlite-wal"),
except Exception as exc: )
logger.warning( except Exception as exc:
"Failed to copy model_cache directory from %s to %s: %s", logger.warning(
source_cache_dir, "Failed to copy directory %s from %s to %s: %s",
target_cache_dir, name,
exc, source,
) target,
exc,
)
source_cache_file = os.path.join(source_dir, "model_cache.sqlite") # Managed subdirectories under settings_dir
target_cache_file = os.path.join(target_dir, "model_cache.sqlite") _copy_dir("cache")
if os.path.isfile(source_cache_file) and os.path.abspath( _copy_dir("backups")
source_cache_file _copy_dir("logs")
) != os.path.abspath(target_cache_file): _copy_dir("stats")
try: _copy_dir("wildcards")
shutil.copy2(source_cache_file, target_cache_file)
except Exception as exc:
logger.warning(
"Failed to copy model_cache.sqlite from %s to %s: %s",
source_cache_file,
target_cache_file,
exc,
)
def _get_user_config_directory(self) -> str: def _get_user_config_directory(self) -> str:
"""Return the user configuration directory, falling back to ~/.config.""" """Return the user configuration directory, falling back to ~/.config."""

View File

@@ -47,6 +47,20 @@ SUPPORTED_MEDIA_EXTENSIONS = {
"videos": [".mp4", ".webm"], "videos": [".mp4", ".webm"],
} }
# Model weight file extensions recognised by scanners.
# This is the union of all scanner extensions (lora, checkpoint, embedding).
MODEL_FILE_EXTENSIONS = {
".safetensors",
".ckpt",
".pt",
".pt2",
".bin",
".pth",
".pkl",
".sft",
".gguf",
}
# Valid sub-types for each scanner type # Valid sub-types for each scanner type
VALID_LORA_SUB_TYPES = ["lora", "locon", "dora"] VALID_LORA_SUB_TYPES = ["lora", "locon", "dora"]
VALID_CHECKPOINT_SUB_TYPES = ["checkpoint", "diffusion_model"] VALID_CHECKPOINT_SUB_TYPES = ["checkpoint", "diffusion_model"]
@@ -215,5 +229,6 @@ SUPPORTED_DOWNLOAD_SKIP_BASE_MODELS = frozenset(
"Ernie", "Ernie",
"Ernie Turbo", "Ernie Turbo",
"Nucleus", "Nucleus",
"Krea 2",
] ]
) )

View File

@@ -821,4 +821,66 @@
[data-theme="dark"] .batch-preview-item { [data-theme="dark"] .batch-preview-item {
background: var(--lora-surface); background: var(--lora-surface);
} }
.hf-badge {
display: inline-block;
padding: 1px 6px;
border-radius: 8px;
background: oklch(0.55 0.12 250 / 0.15);
color: oklch(0.7 0.12 250);
font-size: 0.75em;
font-weight: 600;
margin-left: 4px;
}
/* Checkbox inside HF batch preview items */
.batch-preview-checkbox {
width: 18px;
height: 18px;
cursor: pointer;
accent-color: var(--lora-accent);
flex-shrink: 0;
padding: 0;
border: none;
margin: 0;
}
/* Select All toolbar in batch preview */
.batch-preview-select-all {
display: flex;
align-items: center;
gap: 8px;
padding: 8px 12px;
border-bottom: 1px solid var(--border-color);
background: var(--lora-surface);
cursor: pointer;
position: sticky;
top: 0;
z-index: 1;
}
.batch-preview-select-all input[type="checkbox"] {
width: 18px;
height: 18px;
cursor: pointer;
accent-color: var(--lora-accent);
flex-shrink: 0;
padding: 0;
border: none;
margin: 0;
}
.batch-preview-select-all label {
cursor: pointer;
font-size: 0.9em;
color: var(--text-color);
font-weight: 500;
margin: 0;
user-select: none;
}
[data-theme="dark"] .batch-preview-select-all {
background: var(--lora-surface);
}

View File

@@ -190,6 +190,12 @@ export const DOWNLOAD_ENDPOINTS = {
exampleImages: '/api/lm/force-download-example-images' // New endpoint for downloading example images exampleImages: '/api/lm/force-download-example-images' // New endpoint for downloading example images
}; };
// Hugging Face API endpoints
export const HF_ENDPOINTS = {
repoFiles: '/api/lm/hf-repo-files',
download: '/api/lm/download-hf-model',
};
// WebSocket endpoints // WebSocket endpoints
export const WS_ENDPOINTS = { export const WS_ENDPOINTS = {
fetchProgress: '/ws/fetch-progress' fetchProgress: '/ws/fetch-progress'

View File

@@ -7,6 +7,7 @@ import {
getCurrentModelType, getCurrentModelType,
isValidModelType, isValidModelType,
DOWNLOAD_ENDPOINTS, DOWNLOAD_ENDPOINTS,
HF_ENDPOINTS,
WS_ENDPOINTS WS_ENDPOINTS
} from './apiConfig.js'; } from './apiConfig.js';
import { resetAndReload } from './modelApiFactory.js'; import { resetAndReload } from './modelApiFactory.js';
@@ -1243,6 +1244,48 @@ export class BaseModelApiClient {
} }
} }
async fetchHfRepoFiles(repo, revision = 'main') {
try {
const params = new URLSearchParams({ repo, revision });
const response = await fetch(`${HF_ENDPOINTS.repoFiles}?${params}`);
if (!response.ok) {
const err = await response.json().catch(() => ({}));
throw new Error(err.error || 'Failed to fetch HF repo files');
}
return await response.json();
} catch (error) {
console.error('Error fetching HF repo files:', error);
throw error;
}
}
async downloadHfModel({ repo, filename, revision, modelRoot, relativePath, useDefaultPaths, download_id }) {
try {
const response = await fetch(HF_ENDPOINTS.download, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
repo,
filename,
revision: revision || 'main',
model_root: modelRoot,
relative_path: relativePath || '',
use_default_paths: useDefaultPaths || false,
...(download_id ? { download_id } : {}),
})
});
if (!response.ok) {
throw new Error(await response.text());
}
return await response.json();
} catch (error) {
console.error('Error downloading HF model:', error);
throw error;
}
}
_buildQueryParams(baseParams, pageState) { _buildQueryParams(baseParams, pageState) {
const params = new URLSearchParams(baseParams); const params = new URLSearchParams(baseParams);
const isExcludedView = pageState.viewMode === 'excluded'; const isExcludedView = pageState.viewMode === 'excluded';

View File

@@ -1,4 +1,4 @@
import { showToast, openCivitai, copyToClipboard, copyLoraSyntax, sendLoraToWorkflow, sendEmbeddingToWorkflow, openExampleImagesFolder, buildLoraSyntax, sendModelPathToWorkflow } from '../../utils/uiHelpers.js'; import { showToast, openCivitai, openHuggingFace, copyToClipboard, copyLoraSyntax, sendLoraToWorkflow, sendEmbeddingToWorkflow, openExampleImagesFolder, buildLoraSyntax, sendModelPathToWorkflow } from '../../utils/uiHelpers.js';
import { state, getCurrentPageState } from '../../state/index.js'; import { state, getCurrentPageState } from '../../state/index.js';
import { showModelModal } from './ModelModal.js'; import { showModelModal } from './ModelModal.js';
import { toggleShowcase } from './showcase/ShowcaseView.js'; import { toggleShowcase } from './showcase/ShowcaseView.js';
@@ -66,6 +66,8 @@ function handleModelCardEvent_internal(event, modelType) {
event.stopPropagation(); event.stopPropagation();
if (card.dataset.from_civitai === 'true') { if (card.dataset.from_civitai === 'true') {
openCivitai(card.dataset.filepath); openCivitai(card.dataset.filepath);
} else if (card.dataset.hf_url) {
openHuggingFace(card.dataset.hf_url);
} }
return true; // Stop propagation return true; // Stop propagation
} }
@@ -313,6 +315,7 @@ async function showModelModalFromCard(card, modelType) {
modified: card.dataset.modified, modified: card.dataset.modified,
file_size: parseInt(card.dataset.file_size || '0'), file_size: parseInt(card.dataset.file_size || '0'),
from_civitai: card.dataset.from_civitai === 'true', from_civitai: card.dataset.from_civitai === 'true',
hf_url: card.dataset.hf_url || '',
base_model: card.dataset.base_model, base_model: card.dataset.base_model,
notes: card.dataset.notes || '', notes: card.dataset.notes || '',
favorite: card.dataset.favorite === 'true', favorite: card.dataset.favorite === 'true',
@@ -401,6 +404,7 @@ function showExampleAccessModal(card, modelType) {
modified: card.dataset.modified, modified: card.dataset.modified,
file_size: card.dataset.file_size, file_size: card.dataset.file_size,
from_civitai: card.dataset.from_civitai === 'true', from_civitai: card.dataset.from_civitai === 'true',
hf_url: card.dataset.hf_url || '',
base_model: card.dataset.base_model, base_model: card.dataset.base_model,
notes: card.dataset.notes, notes: card.dataset.notes,
favorite: card.dataset.favorite === 'true', favorite: card.dataset.favorite === 'true',
@@ -467,6 +471,7 @@ export function createModelCard(model, modelType) {
card.dataset.base_model = model.base_model || 'Unknown'; card.dataset.base_model = model.base_model || 'Unknown';
card.dataset.favorite = model.favorite ? 'true' : 'false'; card.dataset.favorite = model.favorite ? 'true' : 'false';
card.dataset.exclude = model.exclude ? 'true' : 'false'; card.dataset.exclude = model.exclude ? 'true' : 'false';
card.dataset.hf_url = model.hf_url || '';
const hasUpdateAvailable = Boolean(model.update_available); const hasUpdateAvailable = Boolean(model.update_available);
card.dataset.update_available = hasUpdateAvailable ? 'true' : 'false'; card.dataset.update_available = hasUpdateAvailable ? 'true' : 'false';
card.dataset.skip_metadata_refresh = model.skip_metadata_refresh ? 'true' : 'false'; card.dataset.skip_metadata_refresh = model.skip_metadata_refresh ? 'true' : 'false';
@@ -578,7 +583,10 @@ export function createModelCard(model, modelType) {
translate('modelCard.actions.addToFavorites', {}, 'Add to favorites'); translate('modelCard.actions.addToFavorites', {}, 'Add to favorites');
const globeTitle = model.from_civitai ? const globeTitle = model.from_civitai ?
translate('modelCard.actions.viewOnCivitai', {}, 'View on Civitai') : translate('modelCard.actions.viewOnCivitai', {}, 'View on Civitai') :
translate('modelCard.actions.notAvailableFromCivitai', {}, 'Not available from Civitai'); model.hf_url ?
translate('modelCard.actions.viewOnHuggingFace', {}, 'View on Hugging Face') :
translate('modelCard.actions.notAvailableFromCivitai', {}, 'Not available from Civitai');
const globeEnabled = model.from_civitai || !!model.hf_url;
let sendTitle; let sendTitle;
let copyTitle; let copyTitle;
if (modelType === MODEL_TYPES.LORA) { if (modelType === MODEL_TYPES.LORA) {
@@ -603,7 +611,7 @@ export function createModelCard(model, modelType) {
</i> </i>
<i class="fas fa-globe" <i class="fas fa-globe"
title="${globeTitle}" title="${globeTitle}"
${!model.from_civitai ? 'style="opacity: 0.5; cursor: not-allowed"' : ''}> ${!globeEnabled ? 'style="opacity: 0.5; cursor: not-allowed"' : ''}>
</i> </i>
<i class="fas fa-paper-plane" <i class="fas fa-paper-plane"
title="${sendTitle}"> title="${sendTitle}">

View File

@@ -3,7 +3,7 @@
* Handles model metadata editing functionality - General version * Handles model metadata editing functionality - General version
*/ */
import { BASE_MODEL_CATEGORIES } from '../../utils/constants.js'; import { BASE_MODEL_CATEGORIES, getMergedBaseModels } from '../../utils/constants.js';
import { showToast } from '../../utils/uiHelpers.js'; import { showToast } from '../../utils/uiHelpers.js';
import { getModelApiClient } from '../../api/modelApiFactory.js'; import { getModelApiClient } from '../../api/modelApiFactory.js';
@@ -267,6 +267,7 @@ export function setupBaseModelEditing(filePath) {
// Add options from BASE_MODEL_CATEGORIES constants // Add options from BASE_MODEL_CATEGORIES constants
const baseModelCategories = BASE_MODEL_CATEGORIES; const baseModelCategories = BASE_MODEL_CATEGORIES;
const categorizedModels = new Set();
// Create option groups for better organization // Create option groups for better organization
Object.entries(baseModelCategories).forEach(([category, models]) => { Object.entries(baseModelCategories).forEach(([category, models]) => {
@@ -277,13 +278,30 @@ export function setupBaseModelEditing(filePath) {
const option = document.createElement('option'); const option = document.createElement('option');
option.value = model; option.value = model;
option.textContent = model; option.textContent = model;
option.selected = model === currentValue; if (model === currentValue) option.selected = true;
categorizedModels.add(model);
group.appendChild(option); group.appendChild(option);
}); });
dropdown.appendChild(group); dropdown.appendChild(group);
}); });
// Check for dynamic base models from API that aren't in any category
const mergedModels = getMergedBaseModels();
const uncategorizedModels = mergedModels.filter(model => !categorizedModels.has(model));
if (uncategorizedModels.length > 0) {
const group = document.createElement('optgroup');
group.label = 'Other (API)';
uncategorizedModels.forEach(model => {
const option = document.createElement('option');
option.value = model;
option.textContent = model;
if (model === currentValue) option.selected = true;
group.appendChild(option);
});
dropdown.appendChild(group);
}
// Replace content with dropdown // Replace content with dropdown
baseModelContent.style.display = 'none'; baseModelContent.style.display = 'none';
baseModelDisplay.insertBefore(dropdown, editBtn); baseModelDisplay.insertBefore(dropdown, editBtn);

View File

@@ -360,6 +360,11 @@ export async function showModelModal(model, modelType) {
const viewOnCivitaiAction = modelWithFullData.from_civitai ? ` const viewOnCivitaiAction = modelWithFullData.from_civitai ? `
<div class="civitai-view" title="${translate('modals.model.actions.viewOnCivitai', {}, 'View on Civitai')}" data-action="view-civitai" data-filepath="${escapedFilePathAttr}"> <div class="civitai-view" title="${translate('modals.model.actions.viewOnCivitai', {}, 'View on Civitai')}" data-action="view-civitai" data-filepath="${escapedFilePathAttr}">
<i class="fas fa-globe"></i> ${translate('modals.model.actions.viewOnCivitaiText', {}, 'View on Civitai')} <i class="fas fa-globe"></i> ${translate('modals.model.actions.viewOnCivitaiText', {}, 'View on Civitai')}
</div>`.trim() : '';
const escapedHfUrl = modelWithFullData.hf_url ? escapeAttribute(modelWithFullData.hf_url) : '';
const viewOnHuggingFaceAction = escapedHfUrl ? `
<div class="civitai-view" title="${translate('modals.model.actions.viewOnHuggingFace', {}, 'View on Hugging Face')}" data-action="view-huggingface" data-hf-url="${escapedHfUrl}">
<i class="fas fa-globe"></i> ${translate('modals.model.actions.viewOnHuggingFaceText', {}, 'View on Hugging Face')}
</div>`.trim() : ''; </div>`.trim() : '';
const creatorInfoAction = modelWithFullData.civitai?.creator ? ` const creatorInfoAction = modelWithFullData.civitai?.creator ? `
<div class="creator-info" data-username="${modelWithFullData.civitai.creator.username}" data-action="view-creator" title="${translate('modals.model.actions.viewCreatorProfile', {}, 'View Creator Profile')}"> <div class="creator-info" data-username="${modelWithFullData.civitai.creator.username}" data-action="view-creator" title="${translate('modals.model.actions.viewCreatorProfile', {}, 'View Creator Profile')}">
@@ -377,6 +382,9 @@ export async function showModelModal(model, modelType) {
if (viewOnCivitaiAction) { if (viewOnCivitaiAction) {
creatorActionItems.push(indentMarkup(viewOnCivitaiAction, 24)); creatorActionItems.push(indentMarkup(viewOnCivitaiAction, 24));
} }
if (viewOnHuggingFaceAction) {
creatorActionItems.push(indentMarkup(viewOnHuggingFaceAction, 24));
}
if (creatorInfoAction) { if (creatorInfoAction) {
creatorActionItems.push(indentMarkup(creatorInfoAction, 24)); creatorActionItems.push(indentMarkup(creatorInfoAction, 24));
} }
@@ -869,6 +877,11 @@ function setupEventHandlers(filePath, modelType) {
case 'view-civitai': case 'view-civitai':
openCivitai(target.dataset.filepath); openCivitai(target.dataset.filepath);
break; break;
case 'view-huggingface':
if (target.dataset.hfUrl) {
window.open(target.dataset.hfUrl, '_blank', 'noopener,noreferrer');
}
break;
case 'view-creator': case 'view-creator':
const username = target.dataset.username; const username = target.dataset.username;
if (username) { if (username) {

View File

@@ -7,6 +7,7 @@ import { getStorageItem, setStorageItem } from '../utils/storageHelpers.js';
import { FolderTreeManager } from '../components/FolderTreeManager.js'; import { FolderTreeManager } from '../components/FolderTreeManager.js';
import { translate } from '../utils/i18nHelpers.js'; import { translate } from '../utils/i18nHelpers.js';
import { extractCivitaiModelUrlParts } from '../utils/civitaiUtils.js'; import { extractCivitaiModelUrlParts } from '../utils/civitaiUtils.js';
import { formatFileSize } from '../utils/formatters.js';
export class DownloadManager { export class DownloadManager {
constructor() { constructor() {
@@ -27,6 +28,10 @@ export class DownloadManager {
this.isBatchMode = false; this.isBatchMode = false;
this.editingBatchIndex = -1; this.editingBatchIndex = -1;
// HF download state
this.hfRepoId = null;
this.hfSelectedFiles = [];
this.loadingManager = new LoadingManager(); this.loadingManager = new LoadingManager();
this.folderTreeManager = new FolderTreeManager(); this.folderTreeManager = new FolderTreeManager();
this.folderClickHandler = null; this.folderClickHandler = null;
@@ -44,6 +49,8 @@ export class DownloadManager {
this.handleToggleDefaultPath = this.toggleDefaultPath.bind(this); this.handleToggleDefaultPath = this.toggleDefaultPath.bind(this);
this.handleBackToUrlFromBatch = this.backToUrlFromBatch.bind(this); this.handleBackToUrlFromBatch = this.backToUrlFromBatch.bind(this);
this.handleNextFromBatch = this.nextFromBatch.bind(this); this.handleNextFromBatch = this.nextFromBatch.bind(this);
} }
showDownloadModal() { showDownloadModal() {
@@ -99,6 +106,8 @@ export class DownloadManager {
// Default path toggle handler // Default path toggle handler
document.getElementById('useDefaultPath').addEventListener('change', this.handleToggleDefaultPath); document.getElementById('useDefaultPath').addEventListener('change', this.handleToggleDefaultPath);
} }
updateModalLabels() { updateModalLabels() {
@@ -160,6 +169,10 @@ export class DownloadManager {
// Reset default path toggle // Reset default path toggle
this.loadDefaultPathSetting(); this.loadDefaultPathSetting();
// Reset HF state
this.hfRepoId = null;
this.hfSelectedFiles = [];
} }
async retrieveVersionsForModel(modelId, source = null) { async retrieveVersionsForModel(modelId, source = null) {
@@ -180,6 +193,29 @@ export class DownloadManager {
return; return;
} }
// Detect URL types — all URLs must share the same source type
const urlTypes = urls.map(u => DownloadManager.detectUrlType(u));
const isHf = urlTypes.every(t => t && (t.type === 'hf-resolve' || t.type === 'hf-repo'));
const isCivitai = urlTypes.every(t => t && t.type === 'civitai');
if (!isHf && !isCivitai) {
const allValid = urlTypes.every(t => t !== null);
if (!allValid) {
errorElement.textContent = translate('modals.download.errors.invalidUrl');
return;
}
// Mixed sources not supported in one batch
if (urls.length > 1) {
errorElement.textContent = translate('modals.download.errors.mixedSources');
return;
}
}
if (isHf) {
return this._validateAndFetchHf(urls, errorElement);
}
// --- Original CivitAI flow below ---
if (urls.length === 1) { if (urls.length === 1) {
this.isBatchMode = false; this.isBatchMode = false;
try { try {
@@ -271,6 +307,112 @@ export class DownloadManager {
this.showBatchPreviewStep(); this.showBatchPreviewStep();
} }
// ---- Hugging Face download flow ----
async _validateAndFetchHf(urls, errorElement) {
if (urls.length === 1) {
const info = DownloadManager.detectUrlType(urls[0]);
// Direct file resolve URL → skip file selection, go to location
if (info.type === 'hf-resolve') {
this.isBatchMode = false;
this.hfRepoId = info.repo;
this.hfSelectedFiles = [info.filename];
this.source = 'huggingface';
this.proceedToLocation();
return;
}
// Repo URL → fetch file list and convert to batch items
try {
this.loadingManager.showSimpleLoading(translate('modals.download.fetchingRepoFiles'));
const files = await this.apiClient.fetchHfRepoFiles(info.repo);
if (!files || files.length === 0) {
throw new Error(translate('modals.download.errors.noModelFiles'));
}
this.isBatchMode = true;
this.batchModels = [];
this.source = 'huggingface';
for (const file of files) {
this.batchModels.push({
url: urls[0],
source: 'huggingface',
repo: info.repo,
filename: file.filename,
revision: 'main',
displayName: file.filename,
fileSizeBytes: file.size,
selectedVersion: true,
versions: [],
checked: false,
error: null,
});
}
this.showBatchPreviewStep();
} catch (err) {
errorElement.textContent = err.message;
} finally {
this.loadingManager.hide();
}
return;
}
// Multiple HF URLs → batch mode: flatten all files from all repos
this.isBatchMode = true;
this.batchModels = [];
this.source = 'huggingface';
this.loadingManager.showSimpleLoading(translate('modals.download.fetchingRepoFiles'));
for (const url of urls) {
const info = DownloadManager.detectUrlType(url);
if (!info) {
this.batchModels.push({ url, error: 'Invalid URL', versions: [], selectedVersion: null });
continue;
}
if (info.type === 'hf-resolve') {
this.batchModels.push({
url,
source: 'huggingface',
repo: info.repo,
filename: info.filename,
revision: info.revision || 'main',
displayName: info.filename,
selectedVersion: true,
versions: [],
checked: false,
error: null,
});
} else if (info.type === 'hf-repo') {
try {
const files = await this.apiClient.fetchHfRepoFiles(info.repo);
if (!files || files.length === 0) {
this.batchModels.push({ url, error: 'No model files found', versions: [], selectedVersion: null });
continue;
}
// Flatten: create one batch item per file, all checked by default
for (const file of files) {
this.batchModels.push({
url,
source: 'huggingface',
repo: info.repo,
filename: file.filename,
revision: 'main',
displayName: file.filename,
fileSizeBytes: file.size,
selectedVersion: true,
versions: [],
checked: false,
error: null,
});
}
} catch (err) {
this.batchModels.push({ url, error: err.message, versions: [], selectedVersion: null });
}
}
}
this.loadingManager.hide();
this.showBatchPreviewStep();
}
async fetchVersionsForCurrentModel() { async fetchVersionsForCurrentModel() {
const errorElement = document.getElementById('urlError'); const errorElement = document.getElementById('urlError');
if (errorElement) { if (errorElement) {
@@ -311,6 +453,60 @@ export class DownloadManager {
return { modelId: null, modelVersionId: null, source: null }; return { modelId: null, modelVersionId: null, source: null };
} }
/**
* Detect the source type of a download URL.
* @param {string} url
* @returns {{ type: string, repo?: string, filename?: string, revision?: string } | null}
* type: 'civitai' | 'civarchive' | 'hf-resolve' | 'hf-repo' | 'direct-http'
*/
static detectUrlType(url) {
const trimmed = url.trim();
if (!trimmed) return null;
// CivitAI
if (/civitai\.com\/models\//i.test(trimmed) || /civitaiarchive|civarchive/i.test(trimmed)) {
// Will be parsed by existing CivitAI logic
return { type: 'civitai' };
}
// Hugging Face resolve URL → direct file
const hfResolveMatch = trimmed.match(/huggingface\.co\/([^/\s]+\/[^/\s]+)\/resolve\/([^/\s]+)\/(.+)/i);
if (hfResolveMatch) {
return {
type: 'hf-resolve',
repo: hfResolveMatch[1],
revision: hfResolveMatch[2],
filename: hfResolveMatch[3],
};
}
// Hugging Face repo URL (huggingface.co/user/repo or bare user/repo path)
// Require huggingface.co prefix for full URLs; bare user/repo only without ://
const hfRepoMatch = trimmed.match(
trimmed.includes('://')
? /^https?:\/\/huggingface\.co\/([a-zA-Z0-9_.-]+\/[a-zA-Z0-9_.-]+)(?:\/?$|$)/
: /^([a-zA-Z0-9_.-]+\/[a-zA-Z0-9_.-]+)$/
);
if (hfRepoMatch) {
// Reject path-traversal patterns like "../.." or "user/.."
const parts = hfRepoMatch[1].split('/');
if (parts.some(p => p === '.' || p === '..')) {
return null;
}
return {
type: 'hf-repo',
repo: hfRepoMatch[1],
};
}
// Direct HTTP(S) URL (non-HF)
if (/^https?:\/\//i.test(trimmed)) {
return { type: 'direct-http' };
}
return null;
}
extractModelId(url) { extractModelId(url) {
const result = DownloadManager.parseModelUrl(url); const result = DownloadManager.parseModelUrl(url);
this.modelVersionId = result.modelVersionId; this.modelVersionId = result.modelVersionId;
@@ -559,8 +755,8 @@ export class DownloadManager {
return; return;
} }
// In single-URL mode, validate version selection // In single-URL mode, validate version selection (skip for HF)
if (!this.isBatchMode) { if (!this.isBatchMode && this.source !== 'huggingface') {
if (!this.currentVersion) { if (!this.currentVersion) {
showToast('toast.loras.pleaseSelectVersion', {}, 'error'); showToast('toast.loras.pleaseSelectVersion', {}, 'error');
return; return;
@@ -784,6 +980,77 @@ export class DownloadManager {
} }
} }
async _downloadHfSingle({ modelRoot, targetFolder, useDefaultPaths }) {
modalManager.closeModal('downloadModal');
this.loadingManager.restoreProgressBar();
const totalFiles = this.hfSelectedFiles.length;
const updateProgress = this.loadingManager.showDownloadProgress(totalFiles);
try {
let completedDownloads = 0;
for (let i = 0; i < totalFiles; i++) {
const filename = this.hfSelectedFiles[i];
updateProgress(0, completedDownloads, filename);
this.loadingManager.setStatus(`Downloading ${filename}...`);
const downloadId = Date.now().toString() + '_' + i;
const wsProtocol = window.location.protocol === 'https:' ? 'wss://' : 'ws://';
const ws = new WebSocket(`${wsProtocol}${window.location.host}/ws/download-progress?id=${downloadId}`);
try {
await new Promise((resolve, reject) => {
ws.onopen = resolve;
ws.onerror = reject;
});
// Capture completed count at WS creation time so progress
// updates arriving after completedDownloads increments still
// show the correct "N / total" position.
const snapshotCompleted = completedDownloads;
ws.onmessage = (event) => {
const data = JSON.parse(event.data);
if (data.status === 'progress') {
const metrics = {
bytesDownloaded: data.bytes_downloaded,
totalBytes: data.total_bytes,
bytesPerSecond: data.bytes_per_second,
};
updateProgress(data.progress, snapshotCompleted, filename, metrics);
}
};
const response = await this.apiClient.downloadHfModel({
repo: this.hfRepoId,
filename,
revision: 'main',
modelRoot,
relativePath: targetFolder,
useDefaultPaths,
download_id: downloadId,
});
if (response?.success) {
completedDownloads++;
updateProgress(100, completedDownloads, filename);
}
} finally {
ws.close();
}
}
showToast('toast.loras.downloadCompleted', {}, 'success');
// Reload page data — model is already in scanner cache via backend
await resetAndReload(true);
return true;
} catch (error) {
console.error('Failed to download HF model:', error);
showToast('toast.downloads.downloadError', { message: error?.message }, 'error');
return false;
} finally {
this.loadingManager.hide();
}
}
updatePathSelectionUI() { updatePathSelectionUI() {
const manualSelection = document.getElementById('manualPathSelection'); const manualSelection = document.getElementById('manualPathSelection');
@@ -812,13 +1079,19 @@ export class DownloadManager {
document.querySelectorAll('.download-step').forEach(step => step.style.display = 'none'); document.querySelectorAll('.download-step').forEach(step => step.style.display = 'none');
document.getElementById('batchPreviewStep').style.display = 'block'; document.getElementById('batchPreviewStep').style.display = 'block';
const validCount = this.batchModels.filter(m => !m.error && m.selectedVersion).length; const validCount = this.batchModels.filter(m => {
if (m.error) return false;
if (m.source === 'huggingface') return m.checked !== false;
return m.selectedVersion;
}).length;
document.getElementById('downloadModalTitle').textContent = document.getElementById('downloadModalTitle').textContent =
translate('modals.download.titleWithType', { type: this.apiClient.apiConfig.config.displayName }) + translate('modals.download.titleWithType', { type: this.apiClient.apiConfig.config.displayName }) +
` (${validCount})`; ` (${validCount})`;
const list = document.getElementById('batchPreviewList'); const list = document.getElementById('batchPreviewList');
list.innerHTML = this.batchModels.map((item, index) => { const hasHfItems = this.batchModels.some(m => m.source === 'huggingface' && !m.error);
let itemsHtml = this.batchModels.map((item, index) => {
if (item.error) { if (item.error) {
return ` return `
<div class="batch-preview-item batch-preview-error" data-index="${index}"> <div class="batch-preview-item batch-preview-error" data-index="${index}">
@@ -837,6 +1110,30 @@ export class DownloadManager {
} }
const ver = item.selectedVersion; const ver = item.selectedVersion;
// HF batch item rendering with checkbox
if (item.source === 'huggingface') {
const hfSize = item.fileSizeBytes
? formatFileSize(item.fileSizeBytes)
: '?';
return `
<div class="batch-preview-item" data-index="${index}">
<input type="checkbox" class="batch-preview-checkbox"
data-index="${index}" ${item.checked !== false ? 'checked' : ''} />
<div class="batch-preview-info">
<div class="batch-preview-name">${item.displayName || item.filename || `HF #${index}`} <span class="hf-badge">HF</span></div>
<div class="batch-preview-meta">
<span>${hfSize}</span>
<span>${item.repo || ''}</span>
</div>
</div>
<button class="batch-preview-remove" data-index="${index}" title="${translate('common.actions.remove', {}, 'Remove')}">
<i class="fas fa-times"></i>
</button>
</div>
`;
}
const firstImage = ver?.images?.find(img => !img.url.endsWith('.mp4')); const firstImage = ver?.images?.find(img => !img.url.endsWith('.mp4'));
const thumbnailUrl = firstImage ? firstImage.url : '/loras_static/images/no-preview.png'; const thumbnailUrl = firstImage ? firstImage.url : '/loras_static/images/no-preview.png';
const fileSize = ver?.modelSizeKB const fileSize = ver?.modelSizeKB
@@ -866,6 +1163,21 @@ export class DownloadManager {
`; `;
}).join(''); }).join('');
// Prepend select-all toolbar if there are HF items with checkboxes
if (hasHfItems) {
const allChecked = this.batchModels
.filter(m => m.source === 'huggingface' && !m.error)
.every(m => m.checked !== false);
itemsHtml = `
<div class="batch-preview-select-all">
<input type="checkbox" id="batchSelectAll" ${allChecked ? 'checked' : ''} />
<label for="batchSelectAll">${translate('modals.download.selectAll', {}, 'Select All')}</label>
</div>
` + itemsHtml;
}
list.innerHTML = itemsHtml;
list.onclick = (e) => { list.onclick = (e) => {
const removeBtn = e.target.closest('.batch-preview-remove'); const removeBtn = e.target.closest('.batch-preview-remove');
if (removeBtn) { if (removeBtn) {
@@ -881,6 +1193,59 @@ export class DownloadManager {
} }
}; };
// Checkbox handler for HF batch items
const checkboxes = list.querySelectorAll('.batch-preview-checkbox');
checkboxes.forEach(cb => {
cb.addEventListener('change', (e) => {
const idx = parseInt(e.target.dataset.index);
if (this.batchModels[idx]) {
this.batchModels[idx].checked = e.target.checked;
}
// Update valid count in title and Next button
const checkedCount = this.batchModels.filter(
m => !m.error && m.checked !== false
).length;
document.getElementById('downloadModalTitle').textContent =
translate('modals.download.titleWithType', { type: this.apiClient.apiConfig.config.displayName }) +
` (${checkedCount})`;
const nextBtn = document.getElementById('nextFromBatchBtn');
nextBtn.disabled = checkedCount === 0;
nextBtn.classList.toggle('disabled', checkedCount === 0);
// Update select-all checkbox state
const selectAll = document.getElementById('batchSelectAll');
if (selectAll) {
const hfItems = this.batchModels.filter(m => m.source === 'huggingface' && !m.error);
selectAll.checked = hfItems.length > 0 && hfItems.every(m => m.checked !== false);
}
});
});
// Select-all handler
const selectAll = document.getElementById('batchSelectAll');
if (selectAll) {
selectAll.addEventListener('change', (e) => {
const checked = e.target.checked;
const hfCheckboxes = list.querySelectorAll('.batch-preview-checkbox');
hfCheckboxes.forEach(cb => {
cb.checked = checked;
const idx = parseInt(cb.dataset.index);
if (this.batchModels[idx]) {
this.batchModels[idx].checked = checked;
}
});
// Update valid count in title and Next button
const checkedCount = this.batchModels.filter(
m => !m.error && m.checked !== false
).length;
document.getElementById('downloadModalTitle').textContent =
translate('modals.download.titleWithType', { type: this.apiClient.apiConfig.config.displayName }) +
` (${checkedCount})`;
const nextBtn = document.getElementById('nextFromBatchBtn');
nextBtn.disabled = checkedCount === 0;
nextBtn.classList.toggle('disabled', checkedCount === 0);
});
}
const nextBtn = document.getElementById('nextFromBatchBtn'); const nextBtn = document.getElementById('nextFromBatchBtn');
nextBtn.disabled = validCount === 0; nextBtn.disabled = validCount === 0;
nextBtn.classList.toggle('disabled', validCount === 0); nextBtn.classList.toggle('disabled', validCount === 0);
@@ -903,7 +1268,12 @@ export class DownloadManager {
} }
nextFromBatch() { nextFromBatch() {
const validModels = this.batchModels.filter(m => !m.error && m.selectedVersion); // For HF items, respect the checked flag; for CivitAI items, use selectedVersion
const validModels = this.batchModels.filter(m => {
if (m.error) return false;
if (m.source === 'huggingface') return m.checked !== false;
return m.selectedVersion;
});
if (validModels.length === 0) return; if (validModels.length === 0) return;
this.proceedToLocation(); this.proceedToLocation();
} }
@@ -953,6 +1323,15 @@ export class DownloadManager {
targetFolder = this.folderTreeManager.getSelectedPath(); targetFolder = this.folderTreeManager.getSelectedPath();
} }
if (!this.isBatchMode) { if (!this.isBatchMode) {
// Single-item download
if (this.source === 'huggingface') {
return this._downloadHfSingle({
modelRoot,
targetFolder,
useDefaultPaths,
});
}
const fileParams = this.selectedFile ? { const fileParams = this.selectedFile ? {
type: this.selectedFile.type || 'Model', type: this.selectedFile.type || 'Model',
format: this.selectedFile.metadata?.format || 'SafeTensor', format: this.selectedFile.metadata?.format || 'SafeTensor',
@@ -974,7 +1353,13 @@ export class DownloadManager {
} }
// Batch download mode // Batch download mode
const downloadItems = this.batchModels.filter(m => !m.error && m.selectedVersion && !m.selectedVersion.existsLocally); const downloadItems = this.batchModels.filter(m => {
if (m.error) return false;
if (!m.selectedVersion) return false;
// HF items have selectedVersion as a boolean marker + checked flag
if (m.source === 'huggingface') return m.checked !== false;
return !m.selectedVersion.existsLocally;
});
if (downloadItems.length === 0) { if (downloadItems.length === 0) {
showToast('toast.loras.downloadCompleted', {}, 'info'); showToast('toast.loras.downloadCompleted', {}, 'info');
modalManager.closeModal('downloadModal'); modalManager.closeModal('downloadModal');
@@ -999,7 +1384,7 @@ export class DownloadManager {
if (data.status === 'progress' && data.download_id?.startsWith(batchDownloadId)) { if (data.status === 'progress' && data.download_id?.startsWith(batchDownloadId)) {
const current = downloadItems[completedDownloads + failedDownloads]; const current = downloadItems[completedDownloads + failedDownloads];
const name = current?.selectedVersion?.name || `#${completedDownloads + failedDownloads + 1}`; const name = current?.selectedVersion?.name || current?.displayName || current?.filename || `#${completedDownloads + failedDownloads + 1}`;
const metrics = { const metrics = {
bytesDownloaded: data.bytes_downloaded, bytesDownloaded: data.bytes_downloaded,
totalBytes: data.total_bytes, totalBytes: data.total_bytes,
@@ -1016,22 +1401,59 @@ export class DownloadManager {
for (let i = 0; i < downloadItems.length; i++) { for (let i = 0; i < downloadItems.length; i++) {
const item = downloadItems[i]; const item = downloadItems[i];
const ver = item.selectedVersion; const name = item.displayName || item.filename || (item.selectedVersion?.name || `Model #${item.modelId}`);
const name = ver?.name || `Model #${item.modelId}`; const isHf = item.source === 'huggingface';
updateProgress(0, completedDownloads, name); updateProgress(0, completedDownloads, name);
loadingManager.setStatus(`${i + 1}/${downloadItems.length}: ${name}`); loadingManager.setStatus(`${i + 1}/${downloadItems.length}: ${name}`);
try { try {
const response = await this.apiClient.downloadModel( let response;
item.modelId, if (isHf) {
ver.id, // Per-file WebSocket for real-time progress
modelRoot, const downloadId = Date.now().toString() + '_hf_' + i;
targetFolder, const wsHf = new WebSocket(`${wsProtocol}${window.location.host}/ws/download-progress?id=${downloadId}`);
useDefaultPaths, try {
batchDownloadId, await new Promise((resolve, reject) => {
item.source wsHf.onopen = resolve;
); wsHf.onerror = reject;
});
const snapshotCompleted = completedDownloads;
wsHf.onmessage = (event) => {
const data = JSON.parse(event.data);
if (data.status === 'progress') {
const metrics = {
bytesDownloaded: data.bytes_downloaded,
totalBytes: data.total_bytes,
bytesPerSecond: data.bytes_per_second,
};
updateProgress(data.progress, snapshotCompleted, name, metrics);
}
};
response = await this.apiClient.downloadHfModel({
repo: item.repo,
filename: item.filename,
revision: item.revision || 'main',
modelRoot,
relativePath: targetFolder,
useDefaultPaths,
download_id: downloadId,
});
} finally {
wsHf.close();
}
} else {
response = await this.apiClient.downloadModel(
item.modelId,
item.selectedVersion.id,
modelRoot,
targetFolder,
useDefaultPaths,
batchDownloadId,
item.source
);
}
if (!response.success) { if (!response.success) {
failedDownloads++; failedDownloads++;

View File

@@ -70,6 +70,7 @@ export const BASE_MODELS = {
ERNIE_TURBO: "Ernie Turbo", ERNIE_TURBO: "Ernie Turbo",
NUCLEUS: "Nucleus", NUCLEUS: "Nucleus",
PONY_V7: "Pony V7", PONY_V7: "Pony V7",
KREA_2: "Krea 2",
// Default // Default
UNKNOWN: "Other" UNKNOWN: "Other"
}; };
@@ -197,6 +198,7 @@ export const BASE_MODEL_ABBREVIATIONS = {
[BASE_MODELS.ERNIE]: 'ERNI', [BASE_MODELS.ERNIE]: 'ERNI',
[BASE_MODELS.ERNIE_TURBO]: 'ETRB', [BASE_MODELS.ERNIE_TURBO]: 'ETRB',
[BASE_MODELS.NUCLEUS]: 'NUCL', [BASE_MODELS.NUCLEUS]: 'NUCL',
[BASE_MODELS.KREA_2]: 'KR2',
// Default // Default
[BASE_MODELS.UNKNOWN]: 'OTH' [BASE_MODELS.UNKNOWN]: 'OTH'
@@ -401,6 +403,7 @@ export const BASE_MODEL_CATEGORIES = {
BASE_MODELS.PIXART_A, BASE_MODELS.PIXART_E, BASE_MODELS.HUNYUAN_1, BASE_MODELS.PIXART_A, BASE_MODELS.PIXART_E, BASE_MODELS.HUNYUAN_1,
BASE_MODELS.LUMINA, BASE_MODELS.KOLORS, BASE_MODELS.NOOBAI, BASE_MODELS.ANIMA, BASE_MODELS.LUMINA, BASE_MODELS.KOLORS, BASE_MODELS.NOOBAI, BASE_MODELS.ANIMA,
BASE_MODELS.ERNIE, BASE_MODELS.ERNIE_TURBO, BASE_MODELS.NUCLEUS, BASE_MODELS.ERNIE, BASE_MODELS.ERNIE_TURBO, BASE_MODELS.NUCLEUS,
BASE_MODELS.KREA_2,
BASE_MODELS.UNKNOWN BASE_MODELS.UNKNOWN
] ]
}; };

View File

@@ -319,6 +319,15 @@ export function openCivitai(filePath) {
openCivitaiByMetadata(civitaiId, versionId, modelName); openCivitaiByMetadata(civitaiId, versionId, modelName);
} }
/**
* Open a Hugging Face model page in a new tab
* @param {string} hfUrl - The Hugging Face URL
*/
export function openHuggingFace(hfUrl) {
if (!hfUrl) return;
window.open(hfUrl, '_blank', 'noopener,noreferrer');
}
/** /**
* Dynamically positions the search options panel and filter panel * Dynamically positions the search options panel and filter panel
* based on the current layout and folder tags container height * based on the current layout and folder tags container height

View File

@@ -14,7 +14,7 @@
<div class="error-message" id="urlError"></div> <div class="error-message" id="urlError"></div>
<div class="input-hint"> <div class="input-hint">
<i class="fas fa-info-circle"></i> <i class="fas fa-info-circle"></i>
<span>{{ t('modals.download.urlHint') }}</span> <span id="urlHint">{{ t('modals.download.urlHint') }}</span>
</div> </div>
</div> </div>
<div class="modal-actions"> <div class="modal-actions">

View File

@@ -0,0 +1,103 @@
import { describe, it, expect } from 'vitest';
import { DownloadManager } from '../../../static/js/managers/DownloadManager.js';
describe('DownloadManager.detectUrlType — HF URL detection', () => {
it('detects HF resolve URL with file', () => {
const result = DownloadManager.detectUrlType(
'https://huggingface.co/dx8152/Flux2-Klein-9B-Consistency/resolve/main/Flux2-Klein-9B-consistency-V2.safetensors'
);
expect(result).toEqual({
type: 'hf-resolve',
repo: 'dx8152/Flux2-Klein-9B-Consistency',
revision: 'main',
filename: 'Flux2-Klein-9B-consistency-V2.safetensors',
});
});
it('detects HF resolve URL with subdirectory file', () => {
const result = DownloadManager.detectUrlType(
'https://huggingface.co/user/repo/resolve/main/subdir/model.safetensors'
);
expect(result).toEqual({
type: 'hf-resolve',
repo: 'user/repo',
revision: 'main',
filename: 'subdir/model.safetensors',
});
});
it('detects HF repo URL (full URL)', () => {
const result = DownloadManager.detectUrlType(
'https://huggingface.co/dx8152/Flux2-Klein-9B-Consistency'
);
expect(result).toEqual({
type: 'hf-repo',
repo: 'dx8152/Flux2-Klein-9B-Consistency',
});
});
it('detects HF repo URL (bare user/repo)', () => {
const result = DownloadManager.detectUrlType('dx8152/Flux2-Klein-9B-Consistency');
expect(result).toEqual({
type: 'hf-repo',
repo: 'dx8152/Flux2-Klein-9B-Consistency',
});
});
it('detects HF repo URL with trailing slash', () => {
const result = DownloadManager.detectUrlType(
'https://huggingface.co/user/repo/'
);
expect(result).toEqual({
type: 'hf-repo',
repo: 'user/repo',
});
});
it('detects CivitAI URL', () => {
const result = DownloadManager.detectUrlType(
'https://civitai.com/models/123/some-model'
);
expect(result).toEqual({ type: 'civitai' });
});
it('detects CivArchive URL', () => {
const result = DownloadManager.detectUrlType(
'https://civarchive.com/models/456'
);
expect(result).toEqual({ type: 'civitai' });
});
it('detects direct HTTP URL', () => {
const result = DownloadManager.detectUrlType(
'https://example.com/file.zip'
);
expect(result).toEqual({ type: 'direct-http' });
});
it('returns null for invalid input', () => {
expect(DownloadManager.detectUrlType('')).toBeNull();
expect(DownloadManager.detectUrlType(' ')).toBeNull();
});
it('returns null for unrecognized path', () => {
expect(DownloadManager.detectUrlType('justrandomtext')).toBeNull();
});
it('prefers HF resolve over repo when both match', () => {
const result = DownloadManager.detectUrlType(
'https://huggingface.co/user/repo/resolve/main/file.safetensors'
);
expect(result?.type).toBe('hf-resolve');
});
it('prefers CivitAI over HF when both match', () => {
// CivitAI check comes first in detectUrlType
// This URL should be detected as CivitAI, not HF
const result = DownloadManager.detectUrlType(
'https://civitai.com/models/123?huggingface.co/test/repo'
);
expect(result?.type).toBe('civitai');
});
});

View File

@@ -201,6 +201,45 @@ def test_list_models_returns_formatted_items(mock_service, mock_scanner):
asyncio.run(scenario()) asyncio.run(scenario())
def test_list_models_filters_out_corrupted_entries(mock_service, mock_scanner):
"""Corrupted cache entries (format_response returns None) must not appear
in the response items nor cause a 500. See issue #730.
"""
mock_service.paginated_items = [
{"file_path": "/tmp/good.safetensors", "name": "Good"},
{"file_path": None, "name": "Corrupted"}, # triggers None from format_response
{"file_path": "/tmp/also_good.safetensors", "name": "AlsoGood"},
]
# Override format_response to return None for corrupted entries
original_format = mock_service.format_response
async def conditional_format(item):
if item.get("file_path") is None:
return None
return await original_format(item)
mock_service.format_response = conditional_format
async def scenario():
client = await create_test_client(mock_service)
try:
response = await client.get("/api/lm/test-models/list")
payload = await response.json()
assert response.status == 200
# Only the 2 non-corrupted entries should appear
assert len(payload["items"]) == 2
assert payload["items"][0]["name"] == "Good"
assert payload["items"][1]["name"] == "AlsoGood"
# None should never appear in the items list
assert None not in payload["items"]
finally:
await client.close()
asyncio.run(scenario())
def test_model_types_endpoint_returns_counts(mock_service, mock_scanner): def test_model_types_endpoint_returns_counts(mock_service, mock_scanner):
mock_service.model_types = [ mock_service.model_types = [
{"type": "LoRa", "count": 3}, {"type": "LoRa", "count": 3},

View File

@@ -59,3 +59,180 @@ async def test_get_nightly_version_network_error_logs_warning(monkeypatch, caplo
assert changelog == [] assert changelog == []
assert "Unable to reach GitHub for nightly version" in caplog.text assert "Unable to reach GitHub for nightly version" in caplog.text
assert "Traceback" not in caplog.text assert "Traceback" not in caplog.text
def test_clean_excludes_covers_user_data_dirs():
"""git clean must receive -e excludes for every user-managed dir."""
excludes = update_routes._clean_excludes()
assert "-e" in excludes # at least one exclude flag present
for name in update_routes._PRESERVE_DIRS:
assert name in excludes
assert f"{name}/**" in excludes
@pytest.mark.asyncio
async def test_perform_git_update_preserves_user_dirs(monkeypatch, tmp_path):
"""``git clean`` must be called with -e excludes for user data dirs.
Regression test for portable-mode updates wiping wildcards/, stats/,
backups/, etc. because ``git clean -fd`` removed untracked, non-ignored
directories.
"""
calls = []
class FakeGit:
def reset(self, *args, **kwargs):
calls.append(("reset", args))
def clean(self, *args, **kwargs):
calls.append(("clean", args))
def checkout(self, *args, **kwargs):
calls.append(("checkout", args))
class FakeRemote:
def fetch(self):
calls.append(("fetch", ()))
def pull(self, *args, **kwargs):
calls.append(("pull", args))
class FakeRemotes:
origin = FakeRemote()
class FakeCommit:
hexsha = "abcdef123456"
class FakeHeads:
def __getitem__(self, name):
class Head:
def checkout(self_inner):
calls.append(("head-checkout", (name,)))
return Head()
class FakeBranches:
names = ["main"]
def __iter__(self):
class B:
name = "main"
return iter([B()])
class FakeRepo:
def __init__(self, path):
calls.append(("repo", (path,)))
git = FakeGit()
remotes = FakeRemotes()
head = type("H", (), {"commit": FakeCommit()})()
branches = FakeBranches()
heads = FakeHeads()
def create_head(self, name, ref):
calls.append(("create_head", (name, ref)))
class FakeGitModule:
class Repo:
def __new__(cls, path):
return FakeRepo(path)
class exc:
class GitError(Exception):
pass
import builtins
real_import = builtins.__import__
def fake_import(name, *args, **kwargs):
if name == "git":
return FakeGitModule
return real_import(name, *args, **kwargs)
monkeypatch.setattr(builtins, "__import__", fake_import)
success, version = await update_routes.UpdateRoutes._perform_git_update(
str(tmp_path), nightly=True
)
assert success is True
clean_calls = [c for c in calls if c[0] == "clean"]
assert len(clean_calls) == 1
clean_args = clean_calls[0][1]
# Every preserved dir must be excluded via -e
for name in update_routes._PRESERVE_DIRS:
assert name in clean_args, f"{name} missing from git clean excludes"
assert f"{name}/**" in clean_args, f"{name}/** missing from git clean excludes"
# Ensure there's an -e before each name occurrence
idx = clean_args.index(name)
assert clean_args[idx - 1] == "-e"
@pytest.mark.asyncio
async def test_perform_git_update_stable_preserves_user_dirs(monkeypatch, tmp_path):
"""Stable (tag) update path must also pass -e excludes to git clean."""
calls = []
class FakeGit:
def reset(self, *args, **kwargs):
calls.append(("reset", args))
def clean(self, *args, **kwargs):
calls.append(("clean", args))
def checkout(self, *args, **kwargs):
calls.append(("checkout", args))
class FakeRemote:
def fetch(self):
calls.append(("fetch", ()))
class FakeRemotes:
origin = FakeRemote()
class FakeCommit:
committed_datetime = "2026-01-01"
class FakeTag:
name = "v9.9.9"
commit = FakeCommit()
class FakeRepo:
def __init__(self, path):
calls.append(("repo", (path,)))
git = FakeGit()
remotes = FakeRemotes()
tags = [FakeTag()]
class FakeGitModule:
class Repo:
def __new__(cls, path):
return FakeRepo(path)
class exc:
class GitError(Exception):
pass
import builtins
real_import = builtins.__import__
def fake_import(name, *args, **kwargs):
if name == "git":
return FakeGitModule
return real_import(name, *args, **kwargs)
monkeypatch.setattr(builtins, "__import__", fake_import)
success, version = await update_routes.UpdateRoutes._perform_git_update(
str(tmp_path), nightly=False
)
assert success is True
assert version == "v9.9.9"
clean_calls = [c for c in calls if c[0] == "clean"]
assert len(clean_calls) == 1
clean_args = clean_calls[0][1]
for name in update_routes._PRESERVE_DIRS:
assert name in clean_args, f"{name} missing from git clean excludes (stable)"

View File

@@ -199,8 +199,107 @@ class TestEmbeddingServiceFormatResponse:
"from_civitai": True, "from_civitai": True,
"civitai": {}, "civitai": {},
} }
result = await embedding_service.format_response(embedding_data) result = await embedding_service.format_response(embedding_data)
assert result["sub_type"] == "embedding" assert result["sub_type"] == "embedding"
assert "model_type" not in result # Removed in refactoring assert "model_type" not in result # Removed in refactoring
class TestFormatResponseCorruptedEntries:
"""Test format_response handles corrupted cache entries gracefully (issue #730).
When cache rows have None/missing critical fields (e.g. from a partially
written or legacy DB), format_response must NOT raise KeyError/AttributeError.
Instead it returns None so the handler layer can filter the bad entry out
instead of failing the entire listing request.
"""
@pytest.fixture
def mock_scanner(self):
scanner = MagicMock()
scanner._hash_index = MagicMock()
return scanner
@pytest.fixture
def lora_service(self, mock_scanner):
return LoraService(mock_scanner)
@pytest.fixture
def checkpoint_service(self, mock_scanner):
return CheckpointService(mock_scanner)
@pytest.fixture
def embedding_service(self, mock_scanner):
return EmbeddingService(mock_scanner)
@pytest.mark.asyncio
async def test_lora_returns_none_on_missing_file_path(self, lora_service):
"""format_response returns None when file_path is missing (corrupted row)."""
lora_data = {
"model_name": "Test LoRA",
"file_name": "test_lora",
"file_path": None, # corrupted: missing file_path
"folder": "",
"sha256": "abc123",
"tags": [],
"from_civitai": True,
"civitai": {},
}
result = await lora_service.format_response(lora_data)
assert result is None
@pytest.mark.asyncio
async def test_lora_handles_none_model_name_gracefully(self, lora_service):
"""format_response should not crash when model_name is None (legacy DB row)."""
lora_data = {
"model_name": None, # NULL from old DB row
"file_name": "test_lora",
"file_path": "/models/test_lora.safetensors",
"folder": "",
"sha256": "abc123",
"tags": [],
"from_civitai": True,
"civitai": {},
}
result = await lora_service.format_response(lora_data)
# Should not raise; model_name falls back to file_name
assert result is not None
assert result["model_name"] == "test_lora"
@pytest.mark.asyncio
async def test_checkpoint_returns_none_on_missing_file_path(self, checkpoint_service):
"""format_response returns None when file_path is missing (corrupted row)."""
checkpoint_data = {
"model_name": "Test",
"file_name": "test",
"file_path": "", # empty string == corrupted
"folder": "",
"sha256": "abc",
"tags": [],
"from_civitai": True,
"civitai": {},
"sub_type": "checkpoint",
}
result = await checkpoint_service.format_response(checkpoint_data)
assert result is None
@pytest.mark.asyncio
async def test_embedding_handles_none_fields_gracefully(self, embedding_service):
"""format_response should not crash when optional fields are None."""
embedding_data = {
"model_name": None,
"file_name": None,
"file_path": "/models/test.pt",
"folder": None,
"sha256": "abc",
"tags": [],
"from_civitai": True,
"civitai": {},
"sub_type": "embedding",
}
result = await embedding_service.format_response(embedding_data)
assert result is not None
assert result["file_path"] == "/models/test.pt"
# model_name falls back to file_name which falls back to ""
assert result["model_name"] == ""

View File

@@ -200,52 +200,97 @@ def _setup_storage_paths(tmp_path, monkeypatch):
return project_root, user_dir, user_settings_path return project_root, user_dir, user_settings_path
def _populate_cache(root_dir, marker_name, db_text): def _populate_settings_dir(root_dir):
cache_dir = root_dir / "model_cache" """Create test data for all managed subdirectories under a settings directory."""
cache_dir.mkdir(exist_ok=True) (root_dir / "cache" / "symlink").mkdir(parents=True, exist_ok=True)
marker_file = cache_dir / marker_name (root_dir / "cache" / "symlink" / "symlink_map.json").write_text(
marker_file.write_text(marker_name, encoding="utf-8") '{"migrated": true}', encoding="utf-8"
(root_dir / "model_cache.sqlite").write_text(db_text, encoding="utf-8") )
(root_dir / "backups").mkdir(parents=True, exist_ok=True)
(root_dir / "backups" / "backup_test.zip").write_text(
"backup", encoding="utf-8"
)
(root_dir / "logs").mkdir(parents=True, exist_ok=True)
(root_dir / "logs" / "session.log").write_text("log", encoding="utf-8")
(root_dir / "stats").mkdir(parents=True, exist_ok=True)
(root_dir / "stats" / "stats.json").write_text(
'{"stats": true}', encoding="utf-8"
)
(root_dir / "wildcards").mkdir(parents=True, exist_ok=True)
(root_dir / "wildcards" / "test.txt").write_text("wildcard", encoding="utf-8")
def test_switch_to_portable_mode_copies_cache(tmp_path, monkeypatch): def test_switch_to_portable_mode_copies_subdirectories(tmp_path, monkeypatch):
project_root, user_dir, user_settings = _setup_storage_paths(tmp_path, monkeypatch) project_root, user_dir, user_settings = _setup_storage_paths(tmp_path, monkeypatch)
_populate_cache(user_dir, "user_marker.txt", "user_db") _populate_settings_dir(user_dir)
manager = SettingsManager() manager = SettingsManager()
manager.set("use_portable_settings", True) manager.set("use_portable_settings", True)
assert manager.settings_file == str(project_root / "settings.json") assert manager.settings_file == str(project_root / "settings.json")
marker_copy = project_root / "model_cache" / "user_marker.txt" # Managed subdirectories should all be migrated
assert marker_copy.read_text(encoding="utf-8") == "user_marker.txt" assert (
assert (project_root / "model_cache.sqlite").read_text( project_root / "cache" / "symlink" / "symlink_map.json"
).read_text(encoding="utf-8") == '{"migrated": true}'
assert (
project_root / "backups" / "backup_test.zip"
).read_text(encoding="utf-8") == "backup"
assert (project_root / "logs" / "session.log").read_text(
encoding="utf-8" encoding="utf-8"
) == "user_db" ) == "log"
assert (project_root / "stats" / "stats.json").read_text(
encoding="utf-8"
) == '{"stats": true}'
assert (project_root / "wildcards" / "test.txt").read_text(
encoding="utf-8"
) == "wildcard"
assert user_settings.exists() assert user_settings.exists()
def test_switching_back_to_user_config_moves_cache(tmp_path, monkeypatch): def test_switching_back_to_user_config_moves_subdirectories(tmp_path, monkeypatch):
project_root, user_dir, user_settings = _setup_storage_paths(tmp_path, monkeypatch) project_root, user_dir, user_settings = _setup_storage_paths(tmp_path, monkeypatch)
_populate_cache(user_dir, "user_marker.txt", "user_db") _populate_settings_dir(user_dir)
manager = SettingsManager() manager = SettingsManager()
manager.set("use_portable_settings", True) manager.set("use_portable_settings", True)
project_cache_dir = project_root / "model_cache" # Populate project-root managed subdirectories
project_cache_dir.mkdir(exist_ok=True) (project_root / "cache" / "model").mkdir(parents=True, exist_ok=True)
(project_cache_dir / "project_marker.txt").write_text( (project_root / "cache" / "model" / "default.sqlite").write_text(
"project_marker", encoding="utf-8" "project_db", encoding="utf-8"
)
(project_root / "backups" / "project_backup.zip").write_text(
"project_backup", encoding="utf-8"
)
(project_root / "logs" / "project.log").write_text(
"project_log", encoding="utf-8"
)
(project_root / "stats" / "project_stats.json").write_text(
'{"project": true}', encoding="utf-8"
)
(project_root / "wildcards" / "project.txt").write_text(
"project_wildcard", encoding="utf-8"
) )
(project_root / "model_cache.sqlite").write_text("project_db", encoding="utf-8")
manager.set("use_portable_settings", False) manager.set("use_portable_settings", False)
assert manager.settings_file == str(user_settings) assert manager.settings_file == str(user_settings)
assert (user_dir / "model_cache" / "project_marker.txt").read_text( assert (user_dir / "cache" / "model" / "default.sqlite").read_text(
encoding="utf-8" encoding="utf-8"
) == "project_marker" ) == "project_db"
assert (user_dir / "model_cache.sqlite").read_text(encoding="utf-8") == "project_db" assert (user_dir / "backups" / "project_backup.zip").read_text(
encoding="utf-8"
) == "project_backup"
assert (user_dir / "logs" / "project.log").read_text(
encoding="utf-8"
) == "project_log"
assert (user_dir / "stats" / "project_stats.json").read_text(
encoding="utf-8"
) == '{"project": true}'
assert (user_dir / "wildcards" / "project.txt").read_text(
encoding="utf-8"
) == "project_wildcard"
def test_download_path_template_parses_json_string(manager): def test_download_path_template_parses_json_string(manager):