mirror of
https://github.com/willmiao/ComfyUI-Lora-Manager.git
synced 2026-09-28 22:44:09 -03:00
Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b90d60f043 | ||
|
|
c3a9350155 |
@@ -20,18 +20,21 @@ By default, `.metadata.json` sidecars and preview images live **alongside** thei
|
||||
| `sidecar_storage_mode` | `"alongside"` \| `"centralized"` | `"alongside"` |
|
||||
| `sidecar_storage_path` | Absolute path string; empty = `<settings dir>/sidecars` | `""` |
|
||||
|
||||
In centralized mode, sidecars and previews mirror the library-relative directory structure:
|
||||
In centralized mode, sidecars and previews mirror each model root's directory structure:
|
||||
|
||||
```
|
||||
<sidecar_root>/<library>/<root_basename-roothash>/<rel_dir>/<name>.metadata.json
|
||||
<sidecar_root>/<root_component>/<rel_dir>/<name>.metadata.json
|
||||
```
|
||||
|
||||
- `<library>` is the active library name and `<rel_dir>` the model's directory relative to the model root containing the file. `<root_basename-roothash>` combines the root's basename with a short hash of its full path so two roots sharing a basename (e.g. `/mnt/a/loras` and `/mnt/b/loras`) never collide. Each component is sanitized to filesystem-safe characters.
|
||||
- `<rel_dir>` is the model's directory relative to the model root containing the file; the longest matching root wins, so nested roots mirror under the most specific root.
|
||||
- `<root_component>` identifies the model root and **survives the root being moved or renamed**. It starts as the deterministic `<sanitized basename>-<path digest>` — so mirrors created by older builds, and mirrors left behind by a relocated sidecar root, still resolve — and is then pinned in `<sidecar_root>/.lm-sidecar-roots.json` alongside the root's last known path and a few sample subdirectories. Two roots sharing a basename (e.g. `/mnt/a/loras` and `/mnt/b/loras`) always get distinct components and never collide. Each path component is sanitized to filesystem-safe characters.
|
||||
- **Moving or renaming a model root does not strand its sidecars.** On the next run the mirror identity is re-anchored to the root's new path (matched by basename and recorded sample directories), so favorites, notes, tags and usage tips keep resolving. An existing hash-named mirror from an older build is adopted as-is on first use.
|
||||
- If an identity cannot be re-anchored unambiguously (e.g. two same-named candidate roots), nothing is guessed: the mirror stays on disk untouched and surfaces as an orphan in **Doctor → Centralized Sidecars** (and in the log). Restoring the original root path re-links it automatically.
|
||||
- `.civitai.info` files always stay next to the model file, in both modes.
|
||||
- Changing the mode does **not** move existing files automatically — run the migration (`POST /api/lm/sidecars/migrate` with `{"direction": "to_centralized" | "to_alongside"}`, or the "Migrate Sidecars Now" button in settings). The migration covers excluded (hidden) models too, so un-excluding one later never strands its sidecar in the old layout. The result payload includes a `sidecar_root` field with the resolved centralized root, and the settings UI shows the outcome counters plus an "Open Folder" shortcut.
|
||||
- Changing `sidecar_storage_path` while centralized likewise needs a root relocation: `{"direction": "relocate_root", "old_root": "<previous path>"}` moves the whole mirror tree to the new root (the settings UI offers this automatically).
|
||||
- Changing `sidecar_storage_path` while centralized likewise needs a root relocation: `{"direction": "relocate_root", "old_root": "<previous path>"}` moves the whole mirror tree to the new root (the settings UI offers this automatically). The identity map travels with the tree, and its entries win over any map the destination acquired beforehand — so a mirror that was re-anchored earlier keeps its name even if something resolved against the new path before the relocation ran.
|
||||
- The settings UI always shows the resolved effective storage root (via the `sidecar_storage_root*` fields in `GET /api/lm/settings`), with `POST /api/lm/sidecars/open-location` opening it in the file manager. When the resolved root lies inside the plugin installation folder (portable settings mode), the UI warns: reinstalling or clean-updating the plugin would delete the sidecars, so an explicit path outside the installation folder is recommended. The repo `.gitignore` excludes the portable-mode default (`/sidecars/`).
|
||||
- All sidecar/preview path derivation goes through the helpers in `py/utils/sidecar_paths.py`; never construct paths inline.
|
||||
- All sidecar/preview path derivation goes through the helpers in `py/utils/sidecar_paths.py`; never construct paths inline. In the default `alongside` mode these helpers do no extra I/O at all — the identity map is only loaded and reconciled when centralized storage is actually in use.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -2769,6 +2769,9 @@
|
||||
"filename_conflicts": {
|
||||
"title": "Konflikte durch doppelte Dateinamen"
|
||||
},
|
||||
"sidecar_mirror_orphans": {
|
||||
"title": "[TODO: Translate] Centralized Sidecars"
|
||||
},
|
||||
"ui_version": {
|
||||
"title": "UI-Version"
|
||||
}
|
||||
|
||||
@@ -2769,6 +2769,9 @@
|
||||
"filename_conflicts": {
|
||||
"title": "Duplicate Filename Conflicts"
|
||||
},
|
||||
"sidecar_mirror_orphans": {
|
||||
"title": "Centralized Sidecars"
|
||||
},
|
||||
"ui_version": {
|
||||
"title": "UI Version"
|
||||
}
|
||||
|
||||
@@ -2769,6 +2769,9 @@
|
||||
"filename_conflicts": {
|
||||
"title": "Conflictos de nombres de archivo duplicados"
|
||||
},
|
||||
"sidecar_mirror_orphans": {
|
||||
"title": "[TODO: Translate] Centralized Sidecars"
|
||||
},
|
||||
"ui_version": {
|
||||
"title": "Versión de la interfaz"
|
||||
}
|
||||
|
||||
@@ -2769,6 +2769,9 @@
|
||||
"filename_conflicts": {
|
||||
"title": "Conflits de noms de fichiers en double"
|
||||
},
|
||||
"sidecar_mirror_orphans": {
|
||||
"title": "[TODO: Translate] Centralized Sidecars"
|
||||
},
|
||||
"ui_version": {
|
||||
"title": "Version de l'interface"
|
||||
}
|
||||
|
||||
@@ -2769,6 +2769,9 @@
|
||||
"filename_conflicts": {
|
||||
"title": "התנגשויות שמות קבצים כפולים"
|
||||
},
|
||||
"sidecar_mirror_orphans": {
|
||||
"title": "[TODO: Translate] Centralized Sidecars"
|
||||
},
|
||||
"ui_version": {
|
||||
"title": "גרסת הממשק"
|
||||
}
|
||||
|
||||
@@ -2769,6 +2769,9 @@
|
||||
"filename_conflicts": {
|
||||
"title": "ファイル名重複競合"
|
||||
},
|
||||
"sidecar_mirror_orphans": {
|
||||
"title": "[TODO: Translate] Centralized Sidecars"
|
||||
},
|
||||
"ui_version": {
|
||||
"title": "UI バージョン"
|
||||
}
|
||||
|
||||
@@ -2769,6 +2769,9 @@
|
||||
"filename_conflicts": {
|
||||
"title": "파일명 중복 충돌"
|
||||
},
|
||||
"sidecar_mirror_orphans": {
|
||||
"title": "[TODO: Translate] Centralized Sidecars"
|
||||
},
|
||||
"ui_version": {
|
||||
"title": "UI 버전"
|
||||
}
|
||||
|
||||
@@ -2769,6 +2769,9 @@
|
||||
"filename_conflicts": {
|
||||
"title": "Конфликты дублирующихся имён файлов"
|
||||
},
|
||||
"sidecar_mirror_orphans": {
|
||||
"title": "[TODO: Translate] Centralized Sidecars"
|
||||
},
|
||||
"ui_version": {
|
||||
"title": "Версия интерфейса"
|
||||
}
|
||||
|
||||
@@ -2769,6 +2769,9 @@
|
||||
"filename_conflicts": {
|
||||
"title": "文件名重复冲突"
|
||||
},
|
||||
"sidecar_mirror_orphans": {
|
||||
"title": "[TODO: Translate] Centralized Sidecars"
|
||||
},
|
||||
"ui_version": {
|
||||
"title": "UI 版本"
|
||||
}
|
||||
|
||||
@@ -2769,6 +2769,9 @@
|
||||
"filename_conflicts": {
|
||||
"title": "檔案名稱重複衝突"
|
||||
},
|
||||
"sidecar_mirror_orphans": {
|
||||
"title": "[TODO: Translate] Centralized Sidecars"
|
||||
},
|
||||
"ui_version": {
|
||||
"title": "UI 版本"
|
||||
}
|
||||
|
||||
@@ -75,6 +75,8 @@ from ...utils.sidecar_paths import (
|
||||
get_configured_sidecar_root,
|
||||
get_metadata_path,
|
||||
get_preview_dir,
|
||||
get_storage_mode,
|
||||
get_unmatched_sidecar_components,
|
||||
)
|
||||
from ...utils.usage_stats import UsageStats
|
||||
from .base_model_handlers import BaseModelHandlerSet
|
||||
@@ -806,6 +808,7 @@ class DoctorHandler:
|
||||
await self._check_civitai_api_key(),
|
||||
await self._check_cache_health(),
|
||||
await self._check_filename_conflicts(),
|
||||
self._check_sidecar_mirror_orphans(),
|
||||
self._check_ui_version(client_version, app_version),
|
||||
]
|
||||
|
||||
@@ -1042,6 +1045,71 @@ class DoctorHandler:
|
||||
logger.error("Error exporting doctor bundle: %s", exc, exc_info=True)
|
||||
return web.json_response({"success": False, "error": str(exc)}, status=500)
|
||||
|
||||
def _check_sidecar_mirror_orphans(self) -> dict[str, Any]:
|
||||
"""Flag centralized sidecars stranded by a moved/removed model root.
|
||||
|
||||
Centralized sidecars live under a per-root mirror directory. A root
|
||||
that was moved, renamed, or dropped from the configuration leaves its
|
||||
mirror behind; without this check the loss is silent, because the
|
||||
scanner simply rebuilds default metadata at the new location.
|
||||
"""
|
||||
|
||||
actions = [{"id": "open-settings", "label": "Open Settings"}]
|
||||
try:
|
||||
mode = get_storage_mode()
|
||||
except Exception as exc: # pragma: no cover - defensive fallback
|
||||
logger.debug("Doctor: sidecar mode lookup failed: %s", exc)
|
||||
mode = "alongside"
|
||||
|
||||
if mode != "centralized":
|
||||
return {
|
||||
"id": "sidecar_mirror_orphans",
|
||||
"title": "Centralized Sidecars",
|
||||
"status": "ok",
|
||||
"summary": "Sidecar metadata is stored alongside the models.",
|
||||
"details": [],
|
||||
"actions": actions,
|
||||
}
|
||||
|
||||
try:
|
||||
orphans = get_unmatched_sidecar_components()
|
||||
except Exception as exc: # pragma: no cover - defensive fallback
|
||||
logger.warning("Doctor: sidecar orphan check failed: %s", exc)
|
||||
orphans = []
|
||||
|
||||
if not orphans:
|
||||
return {
|
||||
"id": "sidecar_mirror_orphans",
|
||||
"title": "Centralized Sidecars",
|
||||
"status": "ok",
|
||||
"summary": "Every mirrored sidecar directory is linked to a model root.",
|
||||
"details": [f"Root: {describe_sidecar_root().get('root', '')}"],
|
||||
"actions": actions,
|
||||
}
|
||||
|
||||
details = [
|
||||
"Metadata (favorites, notes, tags, usage tips) for these models is on disk but is not being read.",
|
||||
"This usually means a model root was moved, renamed, or removed. Restore the original root path in Settings; the mirror is re-linked automatically.",
|
||||
]
|
||||
details.extend(
|
||||
f"{item['component']} — last known root: {item['last_path'] or 'unknown'}"
|
||||
for item in orphans[:5]
|
||||
)
|
||||
if len(orphans) > 5:
|
||||
details.append(f"… and {len(orphans) - 5} more")
|
||||
return {
|
||||
"id": "sidecar_mirror_orphans",
|
||||
"title": "Centralized Sidecars",
|
||||
"status": "warning",
|
||||
"summary": (
|
||||
f"{len(orphans)} sidecar "
|
||||
f"director{'y' if len(orphans) == 1 else 'ies'} could not be "
|
||||
"linked to a configured model root."
|
||||
),
|
||||
"details": details,
|
||||
"actions": actions,
|
||||
}
|
||||
|
||||
async def _check_civitai_api_key(self) -> dict[str, Any]:
|
||||
api_key = (self._settings.get("civitai_api_key", "") or "").strip()
|
||||
if not api_key:
|
||||
|
||||
@@ -1284,21 +1284,6 @@ class RecipeManagementHandler:
|
||||
_original_image_url,
|
||||
) = await self._download_remote_media(image_url)
|
||||
|
||||
# CivitAI's optimized rendition is re-encoded and metadata-free, so an
|
||||
# embedded ComfyUI workflow only exists in the original. Fetch it
|
||||
# lazily: unlike the URL import path (which needs the original for
|
||||
# metadata parsing anyway), this path would download it purely for the
|
||||
# workflow, so it is skipped unless the API reports one.
|
||||
original_workflow = None
|
||||
if _original_image_url and self._meta_indicates_comfy_workflow(
|
||||
civitai_meta_raw
|
||||
):
|
||||
_raw_original, original_workflow = await self._fetch_original_media(
|
||||
_original_image_url
|
||||
)
|
||||
if original_workflow:
|
||||
metadata["workflow"] = original_workflow
|
||||
|
||||
# Build a version-cached map of local model hashes to cache items so
|
||||
# CivitaiApiMetadataParser can skip CivitAI API calls for models that
|
||||
# exist on disk. Built once and shared by every parse pass below.
|
||||
@@ -1821,10 +1806,6 @@ class RecipeManagementHandler:
|
||||
if recipe_scanner is None:
|
||||
raise RuntimeError("Recipe scanner unavailable")
|
||||
|
||||
# Opt-in workflow embedding. The widget historically POSTs with no
|
||||
# body at all, so a missing/empty body is not an error.
|
||||
workflow = await self._read_optional_json_field(request, "workflow")
|
||||
|
||||
analysis = await self._analysis_service.analyze_widget_metadata(
|
||||
recipe_scanner=recipe_scanner
|
||||
)
|
||||
@@ -1837,7 +1818,6 @@ class RecipeManagementHandler:
|
||||
recipe_scanner=recipe_scanner,
|
||||
metadata=metadata,
|
||||
image_bytes=image_bytes,
|
||||
workflow=workflow,
|
||||
)
|
||||
return web.json_response(result.payload, status=result.status)
|
||||
except RecipeValidationError as exc:
|
||||
@@ -1902,24 +1882,6 @@ class RecipeManagementHandler:
|
||||
return []
|
||||
return [tag.strip() for tag in tag_text.split(",") if tag.strip()]
|
||||
|
||||
async def _read_optional_json_field(
|
||||
self, request: web.Request, field: str
|
||||
) -> Any:
|
||||
"""Read one field from an optional JSON request body.
|
||||
|
||||
Some callers (notably the widget's long-standing "Save Recipe" action)
|
||||
POST with no body at all, and a stale cached extension may still do so
|
||||
after a body is introduced. A missing, empty or malformed body is
|
||||
therefore treated as "no value" rather than a request error.
|
||||
"""
|
||||
if not request.can_read_body:
|
||||
return None
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception:
|
||||
return None
|
||||
return data.get(field) if isinstance(data, dict) else None
|
||||
|
||||
async def _count_recipe_loras(
|
||||
self, recipe_scanner: Any, recipe_id: Optional[str]
|
||||
) -> Optional[int]:
|
||||
@@ -2128,90 +2090,6 @@ class RecipeManagementHandler:
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
def _read_embedded_workflow(self, image_path: Optional[str]) -> Optional[str]:
|
||||
"""Return a ComfyUI workflow embedded in ``image_path``, if any.
|
||||
|
||||
``ExifUtils.extract_image_metadata`` stops at the generation
|
||||
parameters, so the UI-format workflow has to be read through the
|
||||
structured metadata reader. Failures map to ``None``.
|
||||
"""
|
||||
if not image_path or not os.path.exists(image_path):
|
||||
return None
|
||||
try:
|
||||
metadata = ExifUtils._load_structured_metadata(image_path)
|
||||
except Exception as exc:
|
||||
self._logger.debug(
|
||||
"Failed to read embedded workflow from %s: %s", image_path, exc
|
||||
)
|
||||
return None
|
||||
workflow = metadata.get("workflow") if isinstance(metadata, dict) else None
|
||||
return workflow if isinstance(workflow, str) and workflow else None
|
||||
|
||||
@staticmethod
|
||||
def _meta_indicates_comfy_workflow(civitai_meta_raw: Any) -> bool:
|
||||
"""Whether CivitAI reports an embedded ComfyUI workflow for an image.
|
||||
|
||||
``meta.comfy`` is the payload CivitAI captured from the original image,
|
||||
so its presence is the signal that fetching the original is worth the
|
||||
bandwidth when the caller does not already need it for metadata
|
||||
parsing.
|
||||
"""
|
||||
if not isinstance(civitai_meta_raw, dict):
|
||||
return False
|
||||
inner = civitai_meta_raw.get("meta")
|
||||
if isinstance(inner, dict) and inner.get("comfy"):
|
||||
return True
|
||||
return bool(civitai_meta_raw.get("comfy"))
|
||||
|
||||
async def _fetch_original_media(
|
||||
self, original_image_url: Optional[str]
|
||||
) -> tuple[Optional[str], Optional[str]]:
|
||||
"""Download the original rendition and read its embedded media.
|
||||
|
||||
CivitAI's optimized renditions are re-encoded and carry no metadata, so
|
||||
the original is the only source for embedded generation metadata and
|
||||
for the UI-format ComfyUI workflow (the raw extractor's fallback chain
|
||||
ends at ``workflow`` only when no prompt is present).
|
||||
|
||||
Returns ``(raw_metadata, workflow)``; either element is ``None`` when
|
||||
unavailable. Failures never raise — imports keep working with the
|
||||
optimized rendition when the original cannot be fetched.
|
||||
"""
|
||||
if not original_image_url:
|
||||
return None, None
|
||||
|
||||
with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as temp_file:
|
||||
temp_path = temp_file.name
|
||||
try:
|
||||
downloader = await self._downloader_factory()
|
||||
success, _result = await downloader.download_file(
|
||||
original_image_url, temp_path, use_auth=False
|
||||
)
|
||||
if not success:
|
||||
self._logger.warning(
|
||||
"Failed to download original rendition: %s", original_image_url
|
||||
)
|
||||
return None, None
|
||||
|
||||
raw_metadata = await asyncio.to_thread(
|
||||
ExifUtils.extract_image_metadata, temp_path
|
||||
)
|
||||
workflow = await asyncio.to_thread(
|
||||
self._read_embedded_workflow, temp_path
|
||||
)
|
||||
return raw_metadata, workflow
|
||||
except Exception as exc:
|
||||
self._logger.warning(
|
||||
"Failed to read original rendition %s: %s", original_image_url, exc
|
||||
)
|
||||
return None, None
|
||||
finally:
|
||||
try:
|
||||
if os.path.exists(temp_path):
|
||||
os.unlink(temp_path)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def _safe_int(self, value: Any) -> int:
|
||||
try:
|
||||
return int(value)
|
||||
@@ -2417,7 +2295,6 @@ class RecipeManagementHandler:
|
||||
"Failed to extract embedded metadata: %s", exc
|
||||
)
|
||||
|
||||
original_workflow: Optional[str] = None
|
||||
if not parsed_embedded and original_image_url:
|
||||
self._logger.debug(
|
||||
"Optimized image has no embedded metadata, "
|
||||
@@ -2425,32 +2302,48 @@ class RecipeManagementHandler:
|
||||
original_image_url,
|
||||
)
|
||||
try:
|
||||
raw_orig, original_workflow = await self._fetch_original_media(
|
||||
original_image_url
|
||||
)
|
||||
diagnostics["exif_present"] = bool(raw_orig) or bool(
|
||||
diagnostics.get("exif_present")
|
||||
)
|
||||
if raw_orig:
|
||||
parser = (
|
||||
self._analysis_service._recipe_parser_factory.create_parser(
|
||||
raw_orig
|
||||
)
|
||||
downloader = await self._downloader_factory()
|
||||
with tempfile.NamedTemporaryFile(
|
||||
suffix=".png", delete=False
|
||||
) as tmp:
|
||||
orig_tmp_path = tmp.name
|
||||
try:
|
||||
success, _ = await downloader.download_file(
|
||||
original_image_url, orig_tmp_path, use_auth=False
|
||||
)
|
||||
if parser:
|
||||
diagnostics["exif_parser"] = parser.__class__.__name__
|
||||
if isinstance(parser, CivitaiApiMetadataParser):
|
||||
parsed_embedded = await parser.parse_metadata(
|
||||
raw_orig,
|
||||
recipe_scanner=recipe_scanner,
|
||||
local_cache=local_cache,
|
||||
if success:
|
||||
raw_orig = await asyncio.to_thread(
|
||||
ExifUtils.extract_image_metadata, orig_tmp_path
|
||||
)
|
||||
diagnostics["exif_present"] = bool(raw_orig)
|
||||
if raw_orig:
|
||||
parser = (
|
||||
self._analysis_service._recipe_parser_factory.create_parser(
|
||||
raw_orig
|
||||
)
|
||||
)
|
||||
else:
|
||||
parsed_embedded = await parser.parse_metadata(
|
||||
raw_orig, recipe_scanner=recipe_scanner
|
||||
)
|
||||
if parsed_embedded and "gen_params" in parsed_embedded:
|
||||
embedded_gen_params = parsed_embedded["gen_params"]
|
||||
if parser:
|
||||
diagnostics["exif_parser"] = parser.__class__.__name__
|
||||
if isinstance(parser, CivitaiApiMetadataParser):
|
||||
parsed_embedded = await parser.parse_metadata(
|
||||
raw_orig,
|
||||
recipe_scanner=recipe_scanner,
|
||||
local_cache=local_cache,
|
||||
)
|
||||
else:
|
||||
parsed_embedded = await parser.parse_metadata(
|
||||
raw_orig, recipe_scanner=recipe_scanner
|
||||
)
|
||||
if (
|
||||
parsed_embedded
|
||||
and "gen_params" in parsed_embedded
|
||||
):
|
||||
embedded_gen_params = parsed_embedded[
|
||||
"gen_params"
|
||||
]
|
||||
finally:
|
||||
if os.path.exists(orig_tmp_path):
|
||||
os.unlink(orig_tmp_path)
|
||||
except Exception as exc:
|
||||
self._logger.warning(
|
||||
"Failed to extract metadata from original image: %s", exc
|
||||
@@ -2498,8 +2391,6 @@ class RecipeManagementHandler:
|
||||
"gen_params": embedded_gen_params or {},
|
||||
"source_path": image_url,
|
||||
}
|
||||
if original_workflow:
|
||||
metadata["workflow"] = original_workflow
|
||||
|
||||
# Extract preview_nsfw_level from the CivitAI API response
|
||||
# (injected into civitai_meta_raw by _download_remote_media).
|
||||
|
||||
@@ -645,11 +645,6 @@ class BatchImportService:
|
||||
if payload.get("checkpoint"):
|
||||
metadata["checkpoint"] = payload["checkpoint"]
|
||||
|
||||
# A workflow recovered from the source's original rendition
|
||||
# travels as metadata and is embedded into the stored image.
|
||||
if payload.get("workflow"):
|
||||
metadata["workflow"] = payload["workflow"]
|
||||
|
||||
nsfw = payload.get("preview_nsfw_level")
|
||||
if isinstance(nsfw, int) and nsfw > 0:
|
||||
metadata["preview_nsfw_level"] = nsfw
|
||||
|
||||
@@ -117,10 +117,6 @@ class RecipeAnalysisService:
|
||||
image_info: Optional[dict[str, Any]] = None
|
||||
is_video = False
|
||||
extension = ".jpg" # Default
|
||||
# Workflow recovered from the image. CivitAI's optimized renditions are
|
||||
# re-encoded and carry no metadata, so for those the workflow only
|
||||
# exists in the original rendition, fetched below for EXIF extraction.
|
||||
recovered_workflow: Optional[str] = None
|
||||
# Diagnostics collected during analysis; surfaced in the payload so
|
||||
# callers can persist an import_info block explaining empty LoRA lists.
|
||||
diagnostics: dict[str, Any] = {"channel": "url"}
|
||||
@@ -242,9 +238,6 @@ class RecipeAnalysisService:
|
||||
exif_metadata = await asyncio.to_thread(
|
||||
self._exif_utils.extract_image_metadata, temp_path
|
||||
)
|
||||
recovered_workflow = await asyncio.to_thread(
|
||||
self._read_embedded_workflow, temp_path
|
||||
)
|
||||
|
||||
# Fallback: try the original (non-optimized) image for EXIF data
|
||||
if not exif_metadata and civitai_image_id and image_info:
|
||||
@@ -262,16 +255,6 @@ class RecipeAnalysisService:
|
||||
self._exif_utils.extract_image_metadata,
|
||||
orig_temp_path,
|
||||
)
|
||||
# The original is also the only place a ComfyUI
|
||||
# workflow survives; carry it so the save step can
|
||||
# embed it even though the stored preview stays the
|
||||
# small, metadata-free optimized rendition.
|
||||
recovered_workflow = (
|
||||
await asyncio.to_thread(
|
||||
self._read_embedded_workflow, orig_temp_path
|
||||
)
|
||||
or recovered_workflow
|
||||
)
|
||||
finally:
|
||||
self._safe_cleanup(orig_temp_path)
|
||||
|
||||
@@ -375,8 +358,6 @@ class RecipeAnalysisService:
|
||||
|
||||
diagnostics["is_video"] = is_video
|
||||
result.payload["diagnostics"] = diagnostics
|
||||
if recovered_workflow:
|
||||
result.payload["workflow"] = recovered_workflow
|
||||
return result
|
||||
finally:
|
||||
if temp_path:
|
||||
@@ -564,25 +545,6 @@ class RecipeAnalysisService:
|
||||
if not success:
|
||||
raise RecipeDownloadError(f"Failed to download image from URL: {result}")
|
||||
|
||||
def _read_embedded_workflow(self, image_path: Optional[str]) -> Optional[str]:
|
||||
"""Return a ComfyUI workflow embedded in ``image_path``, if any.
|
||||
|
||||
The raw metadata string extractor stops at the generation parameters
|
||||
(``prompt``/``parameters``), so the UI-format workflow has to be read
|
||||
through the structured metadata reader. Failures map to ``None``.
|
||||
"""
|
||||
if not image_path or not os.path.exists(image_path):
|
||||
return None
|
||||
try:
|
||||
metadata = self._exif_utils._load_structured_metadata(image_path)
|
||||
except Exception as exc:
|
||||
self._logger.debug(
|
||||
"Failed to read embedded workflow from %s: %s", image_path, exc
|
||||
)
|
||||
return None
|
||||
workflow = metadata.get("workflow") if isinstance(metadata, dict) else None
|
||||
return workflow if isinstance(workflow, str) and workflow else None
|
||||
|
||||
def _metadata_not_found_response(self, path: str) -> AnalysisResult:
|
||||
payload: dict[str, Any] = {
|
||||
"error": "No metadata found in this image",
|
||||
|
||||
@@ -18,7 +18,6 @@ from ...utils.base_model import (
|
||||
RELATION_INCOMPATIBLE,
|
||||
base_model_relation,
|
||||
)
|
||||
from ...utils.constants import MAX_WORKFLOW_EMBED_BYTES
|
||||
from ...utils.utils import calculate_recipe_fingerprint
|
||||
from ..pending_delete_service import get_pending_delete_service
|
||||
from .errors import RecipeNotFoundError, RecipeValidationError
|
||||
@@ -74,11 +73,6 @@ class RecipePersistenceService:
|
||||
byte-level EXIF update that leaves the pixels untouched). Used
|
||||
by local re-import, where the source is the recipe's own
|
||||
already-optimized preview image.
|
||||
|
||||
``metadata`` may carry a ``workflow`` entry (JSON string, dict or
|
||||
list) recovered from the source's original rendition; it is embedded
|
||||
into the stored image so the recipe reports ``has_workflow`` and can
|
||||
send the workflow back to ComfyUI.
|
||||
"""
|
||||
|
||||
missing_fields = []
|
||||
@@ -93,13 +87,6 @@ class RecipePersistenceService:
|
||||
|
||||
assert metadata is not None
|
||||
|
||||
# A workflow recovered from a higher-fidelity source (CivitAI's
|
||||
# original rendition — its optimized preview is re-encoded and carries
|
||||
# no metadata) travels as data instead of as image bytes. It is
|
||||
# embedded below so ``has_workflow`` and the "send workflow to ComfyUI"
|
||||
# action work for imports whose preview pixels are metadata-free.
|
||||
workflow = metadata.get("workflow")
|
||||
|
||||
resolved_image_bytes = self._resolve_image_bytes(image_bytes, image_base64)
|
||||
recipes_dir = target_dir or recipe_scanner.recipes_dir
|
||||
os.makedirs(recipes_dir, exist_ok=True)
|
||||
@@ -121,7 +108,6 @@ class RecipePersistenceService:
|
||||
format="webp",
|
||||
quality=85,
|
||||
preserve_metadata=True,
|
||||
workflow=workflow,
|
||||
)
|
||||
|
||||
image_filename = f"{recipe_id}{extension}"
|
||||
@@ -130,12 +116,6 @@ class RecipePersistenceService:
|
||||
with open(normalized_image_path, "wb") as file_obj:
|
||||
file_obj.write(optimized_image)
|
||||
|
||||
# The optimization branch above embeds the workflow while re-encoding;
|
||||
# the verbatim (skip_optimize) branch still needs it added, and this is
|
||||
# also the safety net when re-encoding dropped it.
|
||||
if workflow and not is_video:
|
||||
self._exif_utils.embed_workflow(normalized_image_path, workflow)
|
||||
|
||||
current_time = time.time()
|
||||
loras_data = [self._normalise_lora_entry(lora) for lora in (metadata.get("loras") or [])]
|
||||
checkpoint_entry = self._sanitize_checkpoint_entry(self._extract_checkpoint_entry(metadata))
|
||||
@@ -875,15 +855,8 @@ class RecipePersistenceService:
|
||||
recipe_scanner,
|
||||
metadata: dict[str, Any],
|
||||
image_bytes: bytes,
|
||||
workflow: Any = None,
|
||||
) -> PersistenceResult:
|
||||
"""Save a recipe constructed from widget metadata.
|
||||
|
||||
``workflow`` is the caller's ComfyUI graph (UI or API format) to embed
|
||||
in the stored preview. Embedding is opt-in because the graph is by far
|
||||
the largest metadata field and its widget values may contain sensitive
|
||||
data; an oversized graph is dropped rather than inflating the preview.
|
||||
"""
|
||||
"""Save a recipe constructed from widget metadata."""
|
||||
|
||||
if not metadata:
|
||||
raise RecipeValidationError("No generation metadata found")
|
||||
@@ -892,25 +865,12 @@ class RecipePersistenceService:
|
||||
os.makedirs(recipes_dir, exist_ok=True)
|
||||
|
||||
recipe_id = str(uuid.uuid4())
|
||||
|
||||
workflow_json = self._exif_utils.normalise_workflow(workflow)
|
||||
workflow_skipped: Optional[str] = None
|
||||
if workflow_json and len(workflow_json.encode("utf-8")) > MAX_WORKFLOW_EMBED_BYTES:
|
||||
self._logger.warning(
|
||||
"Widget workflow is %d bytes (limit %d); saving recipe without it",
|
||||
len(workflow_json),
|
||||
MAX_WORKFLOW_EMBED_BYTES,
|
||||
)
|
||||
workflow_json = None
|
||||
workflow_skipped = "too_large"
|
||||
|
||||
optimized_image, extension = self._exif_utils.optimize_image(
|
||||
image_data=image_bytes,
|
||||
target_width=self._card_preview_width,
|
||||
format="webp",
|
||||
quality=85,
|
||||
preserve_metadata=True,
|
||||
workflow=workflow_json,
|
||||
)
|
||||
image_filename = f"{recipe_id}{extension}"
|
||||
image_path = os.path.join(recipes_dir, image_filename)
|
||||
@@ -964,9 +924,9 @@ class RecipePersistenceService:
|
||||
if key not in ["checkpoint", "loras"]
|
||||
},
|
||||
"loras_stack": lora_stack,
|
||||
# Set by detection below: the workflow is embedded during
|
||||
# re-encoding only when the caller opted in and it fit the cap.
|
||||
"has_workflow": self._detect_has_workflow(image_path),
|
||||
# Widget saves re-encode an in-memory tensor to PNG/WebP with no
|
||||
# embedded metadata chunks, so a workflow can never be present.
|
||||
"has_workflow": False,
|
||||
# Widget saves read LoRAs straight from the current workflow; an
|
||||
# empty list means the workflow used no LoRAs.
|
||||
"import_info": build_import_info(CHANNEL_WIDGET, None, loras_data),
|
||||
@@ -982,17 +942,15 @@ class RecipePersistenceService:
|
||||
self._exif_utils.append_recipe_metadata(image_path, recipe_data)
|
||||
await recipe_scanner.add_recipe(recipe_data)
|
||||
|
||||
payload: dict[str, Any] = {
|
||||
"success": True,
|
||||
"recipe_id": recipe_id,
|
||||
"image_path": image_path,
|
||||
"json_path": json_path,
|
||||
"recipe_name": recipe_name,
|
||||
"has_workflow": recipe_data["has_workflow"],
|
||||
}
|
||||
if workflow_skipped:
|
||||
payload["workflow_skipped"] = workflow_skipped
|
||||
return PersistenceResult(payload)
|
||||
return PersistenceResult(
|
||||
{
|
||||
"success": True,
|
||||
"recipe_id": recipe_id,
|
||||
"image_path": image_path,
|
||||
"json_path": json_path,
|
||||
"recipe_name": recipe_name,
|
||||
}
|
||||
)
|
||||
|
||||
# Helper methods ---------------------------------------------------
|
||||
|
||||
|
||||
@@ -5,7 +5,8 @@ Two storage layouts exist (see :mod:`py.utils.sidecar_paths`):
|
||||
- ``alongside``: ``<model_dir>/<name>.metadata.json`` and preview files live
|
||||
next to the model file.
|
||||
- ``centralized``: the same files live under the configured sidecar root,
|
||||
mirroring the library-relative directory structure.
|
||||
mirroring each model root's directory structure under a per-root identity
|
||||
component (see :func:`py.utils.sidecar_paths.root_mirror_component`).
|
||||
|
||||
This use case moves the ``.metadata.json`` sidecar and preview files for every
|
||||
known model from one layout to the other. Model files themselves NEVER move.
|
||||
@@ -51,10 +52,12 @@ from ...utils.file_utils import find_preview_file, get_preview_extension
|
||||
from ...utils.metadata_manager import MetadataManager
|
||||
from ...utils.sidecar_paths import (
|
||||
METADATA_SUFFIX,
|
||||
ROOT_MAP_FILENAME,
|
||||
STORAGE_MODE_CENTRALIZED,
|
||||
get_configured_sidecar_root,
|
||||
get_sidecar_root,
|
||||
get_storage_mode,
|
||||
relocate_root_map,
|
||||
resolve_centralized_dir_for_dir,
|
||||
)
|
||||
|
||||
@@ -199,14 +202,19 @@ class SidecarMigrationUseCase:
|
||||
)
|
||||
|
||||
files: List[Tuple[str, str]] = []
|
||||
source_map_path = os.path.join(old, ROOT_MAP_FILENAME)
|
||||
if os.path.isdir(old):
|
||||
for dirpath, _dirnames, filenames in os.walk(old):
|
||||
rel = os.path.relpath(dirpath, old)
|
||||
target_dir = new_root if rel == os.curdir else os.path.join(new_root, rel)
|
||||
for filename in filenames:
|
||||
files.append(
|
||||
(os.path.join(dirpath, filename), os.path.join(target_dir, filename))
|
||||
)
|
||||
source = os.path.join(dirpath, filename)
|
||||
# The identity map is handled by relocate_root_map below:
|
||||
# _transfer's keep-newer rule would let a destination map
|
||||
# written before the relocation displace it.
|
||||
if source == source_map_path:
|
||||
continue
|
||||
files.append((source, os.path.join(target_dir, filename)))
|
||||
|
||||
errors: List[Dict[str, str]] = []
|
||||
counters: Dict[str, Any] = {"moved": 0, "conflicts": 0}
|
||||
@@ -242,6 +250,23 @@ class SidecarMigrationUseCase:
|
||||
errors.append({"model": os.path.basename(src), "error": str(exc)})
|
||||
await emit("processing", processed=index, current=os.path.basename(src))
|
||||
|
||||
# The identity map names the directories just moved, so it travels with
|
||||
# them and wins over any map the destination acquired beforehand.
|
||||
# A failure here strands the moved metadata, so it is a real error.
|
||||
try:
|
||||
if not relocate_root_map(old, new_root):
|
||||
errors.append(
|
||||
{
|
||||
"model": ROOT_MAP_FILENAME,
|
||||
"error": "sidecar root map could not be written to the new root",
|
||||
}
|
||||
)
|
||||
except Exception as exc:
|
||||
self._logger.error(
|
||||
"Sidecar root relocation failed for the root map: %s", exc, exc_info=True
|
||||
)
|
||||
errors.append({"model": ROOT_MAP_FILENAME, "error": str(exc)})
|
||||
|
||||
old_prefix = old.replace(os.sep, "/").rstrip("/") + "/"
|
||||
new_prefix = new_root.replace(os.sep, "/").rstrip("/") + "/"
|
||||
for sidecar in moved_sidecars:
|
||||
|
||||
@@ -41,14 +41,6 @@ PREVIEW_EXTENSIONS = [
|
||||
# Card preview image width
|
||||
CARD_PREVIEW_WIDTH = 480
|
||||
|
||||
# Upper bound for a ComfyUI workflow embedded into a recipe preview on the
|
||||
# opt-in widget save path. The workflow is by far the largest metadata field
|
||||
# (tens of KB for a simple graph), so an anomalous graph — e.g. one carrying
|
||||
# base64 blobs in widget values — is skipped instead of inflating the preview.
|
||||
# Imports are deliberately not capped: their workflow comes from an image the
|
||||
# user already chose, and preserving it is the point.
|
||||
MAX_WORKFLOW_EMBED_BYTES = 256 * 1024
|
||||
|
||||
# Width for optimized example images
|
||||
EXAMPLE_IMAGE_WIDTH = 832
|
||||
|
||||
|
||||
+23
-134
@@ -341,125 +341,29 @@ class ExifUtils:
|
||||
|
||||
metadata_fields = ExifUtils._load_structured_metadata(image_path)
|
||||
metadata_fields["parameters"] = metadata
|
||||
return ExifUtils._write_structured_metadata(image_path, metadata_fields)
|
||||
|
||||
with Image.open(image_path) as img:
|
||||
img_format = img.format
|
||||
|
||||
if img_format == "PNG":
|
||||
png_info = ExifUtils._build_pnginfo(img, metadata_fields)
|
||||
img.save(image_path, format="PNG", pnginfo=png_info)
|
||||
return image_path
|
||||
|
||||
exif_bytes = ExifUtils._build_exif_bytes(
|
||||
metadata_fields, img.info.get("exif")
|
||||
)
|
||||
save_kwargs: dict[str, Any] = {"exif": exif_bytes}
|
||||
if img_format == "WEBP":
|
||||
save_kwargs["quality"] = 85
|
||||
|
||||
img.save(image_path, format=img_format, **save_kwargs)
|
||||
|
||||
return image_path
|
||||
except Exception as e:
|
||||
logger.error(f"Error updating metadata in {image_path}: {e}")
|
||||
return image_path
|
||||
|
||||
@staticmethod
|
||||
def _write_structured_metadata(
|
||||
image_path: str, metadata_fields: dict[str, Optional[str]]
|
||||
) -> str:
|
||||
"""Write structured metadata fields back into an image.
|
||||
|
||||
PNG keeps them as text chunks (``parameters``/``prompt``/``workflow``);
|
||||
every other supported container stores them in EXIF, where the workflow
|
||||
travels in ``ImageDescription`` behind a ``Workflow:`` prefix (see
|
||||
:meth:`_build_exif_bytes`).
|
||||
"""
|
||||
with Image.open(image_path) as img:
|
||||
img_format = img.format
|
||||
|
||||
if img_format == "PNG":
|
||||
png_info = ExifUtils._build_pnginfo(img, metadata_fields)
|
||||
img.save(image_path, format="PNG", pnginfo=png_info)
|
||||
return image_path
|
||||
|
||||
exif_bytes = ExifUtils._build_exif_bytes(
|
||||
metadata_fields, img.info.get("exif")
|
||||
)
|
||||
save_kwargs: dict[str, Any] = {"exif": exif_bytes}
|
||||
if img_format == "WEBP":
|
||||
save_kwargs["quality"] = 85
|
||||
|
||||
img.save(image_path, format=img_format, **save_kwargs)
|
||||
|
||||
return image_path
|
||||
|
||||
@staticmethod
|
||||
def normalise_workflow(workflow: Any) -> Optional[str]:
|
||||
"""Coerce a workflow payload into the JSON string metadata form.
|
||||
|
||||
Accepts the string form stored in image chunks as well as already
|
||||
decoded dict/list payloads; anything else yields ``None``.
|
||||
"""
|
||||
if isinstance(workflow, str):
|
||||
return workflow or None
|
||||
if isinstance(workflow, (dict, list)):
|
||||
try:
|
||||
return json.dumps(workflow)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _merge_workflow(
|
||||
metadata_fields: Optional[dict[str, Optional[str]]], workflow: Any
|
||||
) -> Optional[dict[str, Optional[str]]]:
|
||||
"""Add a caller-supplied workflow to extracted metadata fields.
|
||||
|
||||
Returns ``metadata_fields`` untouched when there is nothing to add, and
|
||||
never overwrites a workflow the source image already carries.
|
||||
"""
|
||||
workflow_json = ExifUtils.normalise_workflow(workflow)
|
||||
if not workflow_json:
|
||||
return metadata_fields
|
||||
if metadata_fields is None:
|
||||
metadata_fields = {
|
||||
"parameters": None,
|
||||
"prompt": None,
|
||||
"workflow": None,
|
||||
"comment": None,
|
||||
}
|
||||
if not metadata_fields.get("workflow"):
|
||||
metadata_fields["workflow"] = workflow_json
|
||||
return metadata_fields
|
||||
|
||||
@staticmethod
|
||||
def embed_workflow(image_path: str, workflow: Any) -> str:
|
||||
"""Embed a ComfyUI workflow into an image that does not carry one.
|
||||
|
||||
Recipe imports recover the workflow from the source's original
|
||||
rendition (CivitAI's optimized preview is re-encoded and metadata-free)
|
||||
and hand it over as data rather than as image bytes. Images that
|
||||
already embed a workflow are left untouched.
|
||||
|
||||
WebP files are patched at the byte level so preview pixels are not
|
||||
re-encoded a second time.
|
||||
"""
|
||||
workflow_json = ExifUtils.normalise_workflow(workflow)
|
||||
if not image_path or not workflow_json:
|
||||
return image_path
|
||||
|
||||
ext = os.path.splitext(image_path)[1].lower()
|
||||
if ext in ['.mp4', '.webm', '.avif', '.jxl']:
|
||||
return image_path
|
||||
|
||||
try:
|
||||
metadata_fields = ExifUtils._load_structured_metadata(image_path)
|
||||
if metadata_fields.get("workflow"):
|
||||
return image_path
|
||||
metadata_fields["workflow"] = workflow_json
|
||||
|
||||
if ext == '.webp':
|
||||
try:
|
||||
exif_bytes = ExifUtils._build_exif_bytes(metadata_fields)
|
||||
with open(image_path, "rb") as file_obj:
|
||||
image_bytes = file_obj.read()
|
||||
updated = ExifUtils._replace_webp_exif(image_bytes, exif_bytes)
|
||||
with open(image_path, "wb") as file_obj:
|
||||
file_obj.write(updated)
|
||||
return image_path
|
||||
except ValueError:
|
||||
# Container without an EXIF chunk: fall through to a full
|
||||
# rewrite so the workflow is still embedded.
|
||||
pass
|
||||
|
||||
return ExifUtils._write_structured_metadata(image_path, metadata_fields)
|
||||
except Exception as e:
|
||||
logger.error(f"Error embedding workflow in {image_path}: {e}")
|
||||
return image_path
|
||||
|
||||
|
||||
@staticmethod
|
||||
def append_recipe_metadata(image_path, recipe_data, pixel_preserving=False) -> str:
|
||||
"""Append recipe metadata to an image's EXIF data
|
||||
@@ -646,7 +550,7 @@ class ExifUtils:
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def optimize_image(image_data, target_width=250, format='webp', quality=85, preserve_metadata=False, workflow=None):
|
||||
def optimize_image(image_data, target_width=250, format='webp', quality=85, preserve_metadata=False):
|
||||
"""
|
||||
Optimize an image by resizing and converting to WebP format
|
||||
|
||||
@@ -656,19 +560,10 @@ class ExifUtils:
|
||||
format: Output format (default: webp)
|
||||
quality: Output quality (0-100)
|
||||
preserve_metadata: Whether to preserve EXIF metadata
|
||||
workflow: Optional ComfyUI workflow (JSON string, dict or list) to
|
||||
embed when the source image does not carry one. Used by import
|
||||
paths that recover the workflow from a higher-fidelity source
|
||||
(e.g. CivitAI's original rendition) while the preview pixels
|
||||
come from a metadata-free optimized rendition.
|
||||
|
||||
Returns:
|
||||
Tuple of (optimized_image_data, extension)
|
||||
"""
|
||||
# A supplied workflow can only survive when metadata is embedded, so
|
||||
# treat it as an implicit request for preservation.
|
||||
if workflow is not None:
|
||||
preserve_metadata = True
|
||||
try:
|
||||
if isinstance(image_data, str) and os.path.exists(image_data):
|
||||
ext = os.path.splitext(image_data)[1].lower()
|
||||
@@ -732,12 +627,6 @@ class ExifUtils:
|
||||
logger.warning(f"Failed to extract metadata, continuing without it: {e}")
|
||||
# Continue without metadata
|
||||
|
||||
# Merge in a workflow recovered elsewhere (e.g. from CivitAI's
|
||||
# original rendition). The source image wins when it already has
|
||||
# one, and this is what lets the metadata-free optimized preview
|
||||
# still end up with the workflow embedded.
|
||||
metadata_fields = ExifUtils._merge_workflow(metadata_fields, workflow)
|
||||
|
||||
# Calculate new height to maintain aspect ratio
|
||||
width, height = img.size
|
||||
new_height = int(height * (target_width / width))
|
||||
@@ -797,8 +686,8 @@ class ExifUtils:
|
||||
temp_file.write(optimized_data)
|
||||
|
||||
try:
|
||||
ExifUtils._write_structured_metadata(
|
||||
temp_path, metadata_fields
|
||||
ExifUtils.update_image_metadata(
|
||||
temp_path, metadata_fields.get("parameters") or ""
|
||||
)
|
||||
# Read back the file
|
||||
with open(temp_path, 'rb') as f:
|
||||
|
||||
+767
-34
@@ -11,21 +11,39 @@ setting:
|
||||
file, the historical layout other tools may rely on.
|
||||
- ``centralized``: sidecars and previews live under a configurable root
|
||||
(``sidecar_storage_path`` setting, default ``<settings_dir>/sidecars``),
|
||||
mirroring the library-relative directory structure:
|
||||
``<root>/<library>/<root_basename-roothash>/<rel_dir>/<name>.metadata.json``.
|
||||
mirroring each model root's directory structure::
|
||||
|
||||
All helpers are pure path computations: no directory scans and no file I/O
|
||||
on the hot path. Settings lookups go through ``SettingsManager.get`` (a dict
|
||||
read); config roots come from the already-initialized ``config`` singleton.
|
||||
<sidecar_root>/<root_component>/<rel_dir>/<name>.metadata.json
|
||||
|
||||
``<root_component>`` identifies the model root and **survives the root being
|
||||
moved or renamed**. It starts as the deterministic ``<sanitized
|
||||
basename>-<path digest>`` (so pre-existing mirrors, and mirrors left behind
|
||||
by a relocated sidecar root, still resolve) and is then pinned in
|
||||
``<sidecar_root>/.lm-sidecar-roots.json`` together with the root's last
|
||||
known path and a few sample subdirectories. Re-anchoring a remembered
|
||||
identity to a new path keeps every sidecar under it usable; computing a new
|
||||
name instead would strand them and silently rebuild default metadata
|
||||
(losing favorites, notes, tags).
|
||||
|
||||
Hot-path behaviour: path resolution is a settings read plus a dict lookup.
|
||||
The persistent map is loaded and reconciled lazily, at most once per change
|
||||
of the configured model-root set, and only when centralized storage is
|
||||
actually in use (``get_sidecar_dir`` short-circuits in alongside mode), so
|
||||
default installs never read, write, or scan anything new. The root map file
|
||||
and its directory are only created when there is something to remember.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from typing import List, Optional
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from typing import Dict, Iterable, List, Optional, Set, Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -36,6 +54,21 @@ STORAGE_MODE_CENTRALIZED = "centralized"
|
||||
|
||||
_VALID_MODES = frozenset({STORAGE_MODE_ALONGSIDE, STORAGE_MODE_CENTRALIZED})
|
||||
|
||||
# Persistent root-identity map, stored inside the sidecar root so it travels
|
||||
# with the mirror tree it describes.
|
||||
ROOT_MAP_FILENAME = ".lm-sidecar-roots.json"
|
||||
_ROOT_MAP_VERSION = 1
|
||||
|
||||
# Bounds for the re-anchor heuristic and the deferred sample persistence.
|
||||
_MAX_SAMPLE_REL_DIRS = 16
|
||||
_MAX_MATCH_PROBES = 64
|
||||
_SAMPLE_SAVE_INTERVAL_SECONDS = 30.0
|
||||
|
||||
# Mirror directory names are ``<sanitized basename>-<8 hex>``; the suffix is a
|
||||
# root id (or the legacy path digest) and is what distinguishes a component
|
||||
# directory from a pre-identity-map library prefix directory.
|
||||
_COMPONENT_PATTERN = re.compile(r"^.+-(?:[0-9a-f]{8})$")
|
||||
|
||||
|
||||
def _get_settings_value(key: str, default=None):
|
||||
"""Read a setting defensively; never fail path resolution on settings errors."""
|
||||
@@ -184,16 +217,262 @@ def _iter_model_roots() -> List[str]:
|
||||
|
||||
|
||||
def _normalize_for_match(path: str) -> str:
|
||||
"""Normalize a path for identity comparisons.
|
||||
|
||||
Business paths are preserved (no ``realpath``): symlinks are not resolved,
|
||||
matching the rest of the codebase.
|
||||
"""
|
||||
|
||||
return os.path.normpath(os.path.abspath(path))
|
||||
|
||||
|
||||
def root_mirror_component(root_path: str) -> str:
|
||||
"""Return the mirror path component identifying a model root.
|
||||
def _normalized_roots(raw_roots: Iterable[str]) -> Tuple[str, ...]:
|
||||
"""Return the deduplicated, normalized, order-stable model roots."""
|
||||
|
||||
``<sanitized basename>-<hash>`` where the hash is a short digest of the
|
||||
normalized absolute root path. Two roots sharing a basename (e.g.
|
||||
``/mnt/a/loras`` and ``/mnt/b/loras``) would otherwise map to the same
|
||||
mirror directory and overwrite each other's sidecars.
|
||||
seen: Dict[str, None] = {}
|
||||
for candidate in raw_roots:
|
||||
if not isinstance(candidate, str) or not candidate.strip():
|
||||
continue
|
||||
seen.setdefault(_normalize_for_match(candidate), None)
|
||||
return tuple(seen)
|
||||
|
||||
|
||||
def _best_root_for(
|
||||
normalized_dir: str, normalized_roots: Tuple[str, ...]
|
||||
) -> Optional[str]:
|
||||
"""Return the most specific configured root containing ``normalized_dir``."""
|
||||
|
||||
best: Optional[str] = None
|
||||
for normalized in normalized_roots:
|
||||
if normalized_dir == normalized or normalized_dir.startswith(
|
||||
normalized + os.sep
|
||||
):
|
||||
if best is None or len(normalized) > len(best):
|
||||
best = normalized
|
||||
return best
|
||||
|
||||
|
||||
class _RootMapState:
|
||||
"""In-memory view of one sidecar root's identity map."""
|
||||
|
||||
__slots__ = (
|
||||
"signature",
|
||||
"components",
|
||||
"root_ids",
|
||||
"entries",
|
||||
"unmatched",
|
||||
"loaded",
|
||||
"dirty_samples",
|
||||
"last_save",
|
||||
"persist_disabled",
|
||||
)
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.signature: Tuple[str, ...] = ()
|
||||
self.components: Dict[str, str] = {}
|
||||
self.root_ids: Dict[str, str] = {}
|
||||
self.entries: Dict[str, Dict[str, object]] = {}
|
||||
self.unmatched: List[str] = []
|
||||
self.loaded = False
|
||||
self.dirty_samples = False
|
||||
self.last_save = 0.0
|
||||
self.persist_disabled = False
|
||||
|
||||
|
||||
_ROOT_MAPS: Dict[str, _RootMapState] = {}
|
||||
_ROOT_MAPS_LOCK = threading.RLock()
|
||||
|
||||
|
||||
def reset_root_map_cache() -> None:
|
||||
"""Forget every cached root-identity map (tests, storage relocation)."""
|
||||
|
||||
with _ROOT_MAPS_LOCK:
|
||||
_ROOT_MAPS.clear()
|
||||
|
||||
|
||||
def _root_map_path(sidecar_root: str) -> str:
|
||||
return os.path.join(sidecar_root, ROOT_MAP_FILENAME)
|
||||
|
||||
|
||||
def _load_root_map(sidecar_root: str) -> Dict[str, Dict[str, object]]:
|
||||
"""Read the persistent map; unreadable or malformed files degrade to empty."""
|
||||
|
||||
path = _root_map_path(sidecar_root)
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8") as handle:
|
||||
payload = json.load(handle)
|
||||
except FileNotFoundError:
|
||||
return {}
|
||||
except (OSError, ValueError) as exc:
|
||||
logger.warning("sidecar_paths: ignoring unreadable root map %s: %s", path, exc)
|
||||
return {}
|
||||
|
||||
if not isinstance(payload, dict):
|
||||
return {}
|
||||
raw_roots = payload.get("roots")
|
||||
if not isinstance(raw_roots, dict):
|
||||
return {}
|
||||
|
||||
entries: Dict[str, Dict[str, object]] = {}
|
||||
for root_id, raw_entry in raw_roots.items():
|
||||
if not isinstance(root_id, str) or not root_id:
|
||||
continue
|
||||
if not isinstance(raw_entry, dict):
|
||||
continue
|
||||
component = raw_entry.get("component")
|
||||
if not isinstance(component, str) or not component:
|
||||
continue
|
||||
samples = raw_entry.get("sample_rel_dirs")
|
||||
entries[root_id] = {
|
||||
"component": component,
|
||||
"basename": raw_entry.get("basename") or "",
|
||||
"last_path": raw_entry.get("last_path") or "",
|
||||
"sample_rel_dirs": (
|
||||
[item for item in samples if isinstance(item, str)][
|
||||
:_MAX_SAMPLE_REL_DIRS
|
||||
]
|
||||
if isinstance(samples, list)
|
||||
else []
|
||||
),
|
||||
}
|
||||
return entries
|
||||
|
||||
|
||||
def _persistable(sidecar_root: str) -> bool:
|
||||
"""Return True when the root map could realistically be written.
|
||||
|
||||
Walks up to the nearest existing ancestor (the sidecar root itself is
|
||||
created lazily, alongside the first sidecar) and checks write access. Used
|
||||
to decide whether identities can be remembered across restarts: when they
|
||||
cannot, resolution falls back to the deterministic path-derived component
|
||||
rather than handing out identifiers that would be forgotten.
|
||||
"""
|
||||
|
||||
probe = os.path.abspath(sidecar_root)
|
||||
while probe and not os.path.exists(probe):
|
||||
parent = os.path.dirname(probe)
|
||||
if parent == probe:
|
||||
return False
|
||||
probe = parent
|
||||
return bool(probe) and os.access(probe, os.W_OK)
|
||||
|
||||
|
||||
def _write_root_map(sidecar_root: str, entries: Dict[str, Dict[str, object]]) -> bool:
|
||||
"""Atomically persist ``entries`` as the root map for ``sidecar_root``."""
|
||||
|
||||
path = _root_map_path(sidecar_root)
|
||||
# Snapshot before serializing: sample directories are appended from the
|
||||
# resolution path, which is not serialized with the map lock.
|
||||
payload = {
|
||||
"version": _ROOT_MAP_VERSION,
|
||||
"roots": {
|
||||
root_id: {
|
||||
"component": entry.get("component", ""),
|
||||
"basename": entry.get("basename", ""),
|
||||
"last_path": entry.get("last_path", ""),
|
||||
"sample_rel_dirs": list(entry.get("sample_rel_dirs") or []),
|
||||
}
|
||||
for root_id, entry in entries.items()
|
||||
},
|
||||
}
|
||||
temp_path = f"{path}.tmp"
|
||||
try:
|
||||
os.makedirs(sidecar_root, exist_ok=True)
|
||||
with open(temp_path, "w", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle, indent=2, ensure_ascii=False)
|
||||
os.replace(temp_path, path)
|
||||
except OSError as exc:
|
||||
logger.warning("sidecar_paths: cannot persist the root map %s: %s", path, exc)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _save_root_map(sidecar_root: str, state: _RootMapState) -> bool:
|
||||
"""Persist a reconciled state; disables persistence when it cannot write."""
|
||||
|
||||
if state.persist_disabled:
|
||||
return False
|
||||
|
||||
if not _write_root_map(sidecar_root, state.entries):
|
||||
state.persist_disabled = True
|
||||
logger.warning(
|
||||
"sidecar_paths: mirror directory names fall back to path-derived "
|
||||
"components for %s",
|
||||
sidecar_root,
|
||||
)
|
||||
return False
|
||||
state.dirty_samples = False
|
||||
state.last_save = time.monotonic()
|
||||
return True
|
||||
|
||||
|
||||
def relocate_root_map(source_root: str, destination_root: str) -> bool:
|
||||
"""Carry the root map from a relocated sidecar root to its destination.
|
||||
|
||||
Call this after the mirror tree itself has been moved. Entries recorded
|
||||
under ``source_root`` win over any identity the destination picked up on
|
||||
its own: resolving against the new sidecar path *before* the relocation
|
||||
writes a map that names mirrors after the current model-root path, while
|
||||
the directories actually being moved are still named after the pinned
|
||||
identity. Destination-only entries are preserved, and the source file is
|
||||
always removed so the emptied tree can be pruned.
|
||||
|
||||
Returns False only when a source map existed but could not be written to
|
||||
the destination — the caller must surface that, since the moved metadata
|
||||
would otherwise be unreachable. Cached state is dropped either way so the
|
||||
next resolution reloads the merged map.
|
||||
"""
|
||||
|
||||
source_path = _root_map_path(source_root)
|
||||
source_entries = _load_root_map(source_root)
|
||||
|
||||
with _ROOT_MAPS_LOCK:
|
||||
if source_entries:
|
||||
destination_path = _root_map_path(destination_root)
|
||||
merged: Dict[str, Dict[str, object]] = {}
|
||||
if os.path.exists(destination_path):
|
||||
merged.update(_load_root_map(destination_root))
|
||||
source_paths = {
|
||||
_normalize_for_match(str(entry["last_path"]))
|
||||
for entry in source_entries.values()
|
||||
if entry.get("last_path")
|
||||
}
|
||||
preserved = {
|
||||
root_id: entry
|
||||
for root_id, entry in merged.items()
|
||||
if not entry.get("last_path")
|
||||
or _normalize_for_match(str(entry.get("last_path"))) not in source_paths
|
||||
}
|
||||
preserved.update(source_entries)
|
||||
if not _write_root_map(destination_root, preserved):
|
||||
return False
|
||||
|
||||
if os.path.exists(source_path):
|
||||
try:
|
||||
os.remove(source_path)
|
||||
except OSError as exc: # pragma: no cover - defensive cleanup
|
||||
logger.debug(
|
||||
"sidecar_paths: cannot remove relocated root map %s: %s",
|
||||
source_path,
|
||||
exc,
|
||||
)
|
||||
|
||||
reset_root_map_cache()
|
||||
return True
|
||||
|
||||
|
||||
def _new_root_id() -> str:
|
||||
return uuid.uuid4().hex[:8]
|
||||
|
||||
|
||||
def _legacy_component(root_path: str) -> str:
|
||||
"""Return the deterministic path-derived component for a root.
|
||||
|
||||
``<sanitized basename>-<digest of the normalized absolute path>``. This is
|
||||
both the name mirrors created by older builds already use and the initial
|
||||
identity for a newly seen root, so a mirror is found again even when the
|
||||
root map is missing. Once an identity is recorded, the map keeps the name
|
||||
pinned across root moves.
|
||||
"""
|
||||
|
||||
normalized = _normalize_for_match(root_path)
|
||||
@@ -201,12 +480,444 @@ def root_mirror_component(root_path: str) -> str:
|
||||
return f"{sanitize_path_component(os.path.basename(normalized))}-{digest}"
|
||||
|
||||
|
||||
def _list_component_dirs(sidecar_root: str) -> Set[str]:
|
||||
"""Return the mirror component directories present under ``sidecar_root``.
|
||||
|
||||
Components are normally immediate children. Pre-identity-map builds nested
|
||||
them one level deeper under a library name
|
||||
(``<sidecar_root>/<library>/<component>``); those are reported as
|
||||
``<library>/<component>`` so the legacy layout keeps resolving — the legacy
|
||||
prefix is treated purely as part of the component name and is never
|
||||
interpreted as a library.
|
||||
"""
|
||||
|
||||
try:
|
||||
names = os.listdir(sidecar_root)
|
||||
except OSError:
|
||||
return set()
|
||||
|
||||
components: Set[str] = set()
|
||||
unqualified: List[str] = []
|
||||
for name in names:
|
||||
if name == ROOT_MAP_FILENAME or name.endswith(".tmp"):
|
||||
continue
|
||||
if not os.path.isdir(os.path.join(sidecar_root, name)):
|
||||
continue
|
||||
if _looks_like_component(name):
|
||||
components.add(name)
|
||||
else:
|
||||
unqualified.append(name)
|
||||
|
||||
for prefix in unqualified:
|
||||
try:
|
||||
nested = os.listdir(os.path.join(sidecar_root, prefix))
|
||||
except OSError:
|
||||
continue
|
||||
for name in nested:
|
||||
if not _looks_like_component(name):
|
||||
continue
|
||||
if os.path.isdir(os.path.join(sidecar_root, prefix, name)):
|
||||
components.add(f"{prefix}/{name}")
|
||||
return components
|
||||
|
||||
|
||||
def _looks_like_component(name: str) -> bool:
|
||||
"""True for ``<basename>-<8 hex>`` mirror directory names."""
|
||||
|
||||
return bool(_COMPONENT_PATTERN.match(name))
|
||||
|
||||
|
||||
def _subdir_names(directory: str) -> Set[str]:
|
||||
"""Sanitized names of ``directory``'s immediate subdirectories (bounded)."""
|
||||
|
||||
names: Set[str] = set()
|
||||
try:
|
||||
with os.scandir(directory) as iterator:
|
||||
for index, dir_entry in enumerate(iterator):
|
||||
if index >= _MAX_MATCH_PROBES:
|
||||
break
|
||||
try:
|
||||
if dir_entry.is_dir():
|
||||
names.add(sanitize_path_component(dir_entry.name))
|
||||
except OSError:
|
||||
continue
|
||||
except OSError:
|
||||
return set()
|
||||
return names
|
||||
|
||||
|
||||
def _mirror_dir_overlap(sidecar_root: str, component: str, root: str) -> int:
|
||||
"""Count sibling directories shared by a mirror component and a root.
|
||||
|
||||
Last-resort signal for re-anchoring when the entry carries no recorded
|
||||
sample directories (e.g. the map was lost): both sides are sanitized the
|
||||
same way, so renamed-but-similar layouts still overlap.
|
||||
"""
|
||||
|
||||
if not component:
|
||||
return 0
|
||||
mirror_names = _subdir_names(os.path.join(sidecar_root, component))
|
||||
if not mirror_names:
|
||||
return 0
|
||||
root_names = _subdir_names(root)
|
||||
if not root_names:
|
||||
return 0
|
||||
return len(mirror_names & root_names)
|
||||
|
||||
|
||||
def _match_score(sidecar_root: str, entry: Dict[str, object], root: str) -> int:
|
||||
"""Score how likely ``entry`` describes the mirror of ``root``."""
|
||||
|
||||
score = 0
|
||||
if str(entry.get("basename") or "") == os.path.basename(root):
|
||||
score += 2
|
||||
|
||||
samples = entry.get("sample_rel_dirs") or []
|
||||
if isinstance(samples, list):
|
||||
for rel in samples[:_MAX_MATCH_PROBES]:
|
||||
if not isinstance(rel, str) or not rel:
|
||||
continue
|
||||
parts = [part for part in rel.split("/") if part not in ("", ".")]
|
||||
if parts and os.path.isdir(os.path.join(root, *parts)):
|
||||
score += 1
|
||||
if not samples:
|
||||
score += _mirror_dir_overlap(
|
||||
sidecar_root, str(entry.get("component") or ""), root
|
||||
)
|
||||
return score
|
||||
|
||||
|
||||
def _rank_scores(
|
||||
assignments: List[Tuple[int, str, str]],
|
||||
) -> Tuple[Dict[str, int], Dict[str, int], Dict[str, int], Dict[str, int]]:
|
||||
"""Return (best/runner-up score per entry id, best/runner-up per root)."""
|
||||
|
||||
best_id: Dict[str, int] = {}
|
||||
runner_id: Dict[str, int] = {}
|
||||
best_root: Dict[str, int] = {}
|
||||
runner_root: Dict[str, int] = {}
|
||||
for score, root_id, root in assignments:
|
||||
if score > best_id.get(root_id, 0):
|
||||
runner_id[root_id] = best_id.get(root_id, 0)
|
||||
best_id[root_id] = score
|
||||
elif score > runner_id.get(root_id, 0):
|
||||
runner_id[root_id] = score
|
||||
if score > best_root.get(root, 0):
|
||||
runner_root[root] = best_root.get(root, 0)
|
||||
best_root[root] = score
|
||||
elif score > runner_root.get(root, 0):
|
||||
runner_root[root] = score
|
||||
return best_id, runner_id, best_root, runner_root
|
||||
|
||||
|
||||
def _due_for_sample_save(state: _RootMapState, now: float) -> bool:
|
||||
return now - state.last_save >= _SAMPLE_SAVE_INTERVAL_SECONDS
|
||||
|
||||
|
||||
def _reconcile_root_map(
|
||||
sidecar_root: str,
|
||||
normalized_roots: Tuple[str, ...],
|
||||
state: _RootMapState,
|
||||
) -> None:
|
||||
"""Match remembered root identities against the configured roots.
|
||||
|
||||
Order of preference per root: the identity previously recorded for the
|
||||
same path, then a deterministic path-derived component already on disk
|
||||
(pre-identity-map installs), then an orphaned identity re-anchored by
|
||||
basename/sample-directory scoring, then a unique unclaimed component whose
|
||||
name matches, and finally a brand-new identity.
|
||||
"""
|
||||
|
||||
entries = (
|
||||
state.entries
|
||||
if state.loaded and state.entries
|
||||
else _load_root_map(sidecar_root)
|
||||
)
|
||||
# Copy so partial mutation cannot leak between reconciliation attempts.
|
||||
entries = {root_id: dict(entry) for root_id, entry in entries.items()}
|
||||
|
||||
on_disk = _list_component_dirs(sidecar_root)
|
||||
persistable = _persistable(sidecar_root)
|
||||
if not persistable:
|
||||
state.persist_disabled = True
|
||||
|
||||
components: Dict[str, str] = {}
|
||||
root_ids: Dict[str, str] = {}
|
||||
claimed: Set[str] = set()
|
||||
changed = False
|
||||
|
||||
by_path: Dict[str, str] = {}
|
||||
for root_id, entry in entries.items():
|
||||
last_path = entry.get("last_path")
|
||||
if isinstance(last_path, str) and last_path.strip():
|
||||
by_path.setdefault(_normalize_for_match(last_path), root_id)
|
||||
|
||||
# 1. Same path as last time: reuse the remembered identity verbatim.
|
||||
pending: List[str] = []
|
||||
for root in normalized_roots:
|
||||
root_id = by_path.get(root)
|
||||
if root_id is not None:
|
||||
component = str(entries[root_id]["component"])
|
||||
if component not in claimed:
|
||||
components[root] = component
|
||||
root_ids[root] = root_id
|
||||
claimed.add(component)
|
||||
continue
|
||||
pending.append(root)
|
||||
|
||||
if not persistable:
|
||||
# Identities created here could never be remembered; keep the layout
|
||||
# deterministic instead of stranding sidecars on the next restart.
|
||||
for root in normalized_roots:
|
||||
components.setdefault(root, _legacy_component(root))
|
||||
state.entries = {}
|
||||
state.root_ids = {}
|
||||
state.components = components
|
||||
state.unmatched = sorted(
|
||||
component for component in on_disk if component not in set(components.values())
|
||||
)
|
||||
state.signature = normalized_roots
|
||||
state.loaded = True
|
||||
_warn_unmatched(sidecar_root, state.unmatched)
|
||||
return
|
||||
|
||||
# 2. Adopt a deterministic path-derived component already on disk. Older
|
||||
# builds nested it under the library name, hence the last-segment match.
|
||||
still_pending: List[str] = []
|
||||
for root in pending:
|
||||
legacy = _legacy_component(root)
|
||||
matches = [
|
||||
component
|
||||
for component in on_disk
|
||||
if component == legacy
|
||||
or component.rsplit("/", 1)[-1] == legacy
|
||||
]
|
||||
component = matches[0] if len(matches) == 1 and matches[0] not in claimed else None
|
||||
if component is not None:
|
||||
root_id = _new_root_id()
|
||||
entries[root_id] = {
|
||||
"component": component,
|
||||
"basename": os.path.basename(root),
|
||||
"last_path": root,
|
||||
"sample_rel_dirs": [],
|
||||
}
|
||||
components[root] = component
|
||||
root_ids[root] = root_id
|
||||
claimed.add(component)
|
||||
changed = True
|
||||
else:
|
||||
still_pending.append(root)
|
||||
|
||||
# 3. Re-anchor identities whose recorded path no longer exists.
|
||||
configured = set(normalized_roots)
|
||||
assignments: List[Tuple[int, str, str]] = []
|
||||
for root_id, entry in entries.items():
|
||||
last_path = str(entry.get("last_path") or "")
|
||||
component = str(entry["component"])
|
||||
if not last_path.strip() or component in claimed:
|
||||
continue
|
||||
if _normalize_for_match(last_path) in configured:
|
||||
continue
|
||||
for root in still_pending:
|
||||
score = _match_score(sidecar_root, entry, root)
|
||||
if score > 0:
|
||||
assignments.append((score, root_id, root))
|
||||
|
||||
if assignments:
|
||||
best_id, runner_id, best_root, runner_root = _rank_scores(assignments)
|
||||
assignments.sort(key=lambda item: (-item[0], item[1], item[2]))
|
||||
used_ids: Set[str] = set()
|
||||
used_roots: Set[str] = set()
|
||||
for score, root_id, root in assignments:
|
||||
if root_id in used_ids or root in used_roots:
|
||||
continue
|
||||
# Never guess: the pair must be the unique best on both sides.
|
||||
if score < best_id[root_id] or score < best_root[root]:
|
||||
continue
|
||||
if runner_id.get(root_id, 0) == score or runner_root.get(root, 0) == score:
|
||||
continue
|
||||
entries[root_id]["last_path"] = root
|
||||
components[root] = str(entries[root_id]["component"])
|
||||
root_ids[root] = root_id
|
||||
claimed.add(str(entries[root_id]["component"]))
|
||||
used_ids.add(root_id)
|
||||
used_roots.add(root)
|
||||
changed = True
|
||||
still_pending = [root for root in still_pending if root not in used_roots]
|
||||
|
||||
# 4/5. Remaining roots: adopt a uniquely matching unclaimed component,
|
||||
# otherwise mint a fresh identity.
|
||||
referenced = {str(entry["component"]) for entry in entries.values()}
|
||||
free_components = [
|
||||
component
|
||||
for component in on_disk
|
||||
if component not in claimed and component not in referenced
|
||||
]
|
||||
for root in still_pending:
|
||||
prefix = sanitize_path_component(os.path.basename(root)) + "-"
|
||||
candidates = [
|
||||
component
|
||||
for component in free_components
|
||||
if component.rsplit("/", 1)[-1].startswith(prefix)
|
||||
]
|
||||
# Default to the deterministic path-derived name. It is stable across
|
||||
# restarts even if the map is lost, and it matches what a sidecar-root
|
||||
# relocation or a mirror created by an older build already used; the
|
||||
# map only overrides it later, to keep the name across a root move.
|
||||
component = (
|
||||
candidates[0] if len(candidates) == 1 else _legacy_component(root)
|
||||
)
|
||||
if len(candidates) == 1:
|
||||
free_components.remove(component)
|
||||
root_id = _new_root_id()
|
||||
entries[root_id] = {
|
||||
"component": component,
|
||||
"basename": os.path.basename(root),
|
||||
"last_path": root,
|
||||
"sample_rel_dirs": [],
|
||||
}
|
||||
components[root] = component
|
||||
root_ids[root] = root_id
|
||||
claimed.add(component)
|
||||
changed = True
|
||||
|
||||
state.entries = entries
|
||||
state.components = components
|
||||
state.root_ids = root_ids
|
||||
state.unmatched = sorted(
|
||||
component for component in on_disk if component not in claimed
|
||||
)
|
||||
state.signature = normalized_roots
|
||||
state.loaded = True
|
||||
|
||||
_warn_unmatched(sidecar_root, state.unmatched)
|
||||
|
||||
if changed and not _save_root_map(sidecar_root, state):
|
||||
_degrade_to_legacy(state, normalized_roots)
|
||||
|
||||
|
||||
def _warn_unmatched(sidecar_root: str, unmatched: List[str]) -> None:
|
||||
"""Surface mirror directories no configured root claims (never silent)."""
|
||||
|
||||
if not unmatched:
|
||||
return
|
||||
preview = ", ".join(unmatched[:5])
|
||||
logger.warning(
|
||||
"sidecar_paths: %d mirrored sidecar dir(s) under %s are not linked to any "
|
||||
"configured model root (%s%s). Their metadata stays on disk but is not "
|
||||
"used until the model root is restored or re-anchored.",
|
||||
len(unmatched),
|
||||
sidecar_root,
|
||||
preview,
|
||||
", ..." if len(unmatched) > 5 else "",
|
||||
)
|
||||
|
||||
|
||||
def _degrade_to_legacy(state: _RootMapState, normalized_roots: Tuple[str, ...]) -> None:
|
||||
"""Fall back to deterministic components after a failed persist."""
|
||||
|
||||
state.components = {root: _legacy_component(root) for root in normalized_roots}
|
||||
state.root_ids = {}
|
||||
state.entries = {}
|
||||
state.unmatched = []
|
||||
state.dirty_samples = False
|
||||
state.persist_disabled = True
|
||||
|
||||
|
||||
def _ensure_root_map(
|
||||
sidecar_root: str, normalized_roots: Tuple[str, ...]
|
||||
) -> _RootMapState:
|
||||
"""Return the reconciled identity map, reusing the cached one when current."""
|
||||
|
||||
key = _normalize_for_match(sidecar_root)
|
||||
state = _ROOT_MAPS.get(key)
|
||||
if state is not None and state.loaded and state.signature == normalized_roots:
|
||||
return state
|
||||
|
||||
with _ROOT_MAPS_LOCK:
|
||||
state = _ROOT_MAPS.get(key)
|
||||
if state is not None and state.loaded and state.signature == normalized_roots:
|
||||
return state
|
||||
if state is None:
|
||||
state = _RootMapState()
|
||||
_ROOT_MAPS[key] = state
|
||||
_reconcile_root_map(sidecar_root, normalized_roots, state)
|
||||
return state
|
||||
|
||||
|
||||
def _component_for(
|
||||
best_root: str,
|
||||
sidecar_root: str,
|
||||
normalized_roots: Tuple[str, ...],
|
||||
rel_dir: str,
|
||||
) -> Optional[str]:
|
||||
"""Return the pinned mirror component for ``best_root``."""
|
||||
|
||||
state = _ensure_root_map(sidecar_root, normalized_roots)
|
||||
if not state.components.get(best_root):
|
||||
return _legacy_component(best_root)
|
||||
_record_sample(sidecar_root, state, best_root, rel_dir)
|
||||
# Re-read: a failed sample persist degrades the state to deterministic
|
||||
# names, and this call must agree with every later one.
|
||||
return state.components.get(best_root) or _legacy_component(best_root)
|
||||
|
||||
|
||||
def _record_sample(
|
||||
sidecar_root: str, state: _RootMapState, root: str, rel_dir: str
|
||||
) -> None:
|
||||
"""Remember a root-relative directory to make future re-anchoring precise."""
|
||||
|
||||
root_id = state.root_ids.get(root)
|
||||
if not root_id:
|
||||
return
|
||||
entry = state.entries.get(root_id)
|
||||
if entry is None:
|
||||
return
|
||||
|
||||
normalized = (rel_dir or "").replace(os.sep, "/").strip("/")
|
||||
if not normalized or normalized == ".":
|
||||
return
|
||||
|
||||
samples = entry.get("sample_rel_dirs")
|
||||
if not isinstance(samples, list):
|
||||
samples = []
|
||||
entry["sample_rel_dirs"] = samples
|
||||
if normalized in samples or len(samples) >= _MAX_SAMPLE_REL_DIRS:
|
||||
return
|
||||
|
||||
samples.append(normalized)
|
||||
state.dirty_samples = True
|
||||
if state.persist_disabled or not _due_for_sample_save(state, time.monotonic()):
|
||||
return
|
||||
with _ROOT_MAPS_LOCK:
|
||||
if state.persist_disabled or not _due_for_sample_save(state, time.monotonic()):
|
||||
return
|
||||
if not _save_root_map(sidecar_root, state):
|
||||
_degrade_to_legacy(state, state.signature)
|
||||
|
||||
|
||||
def root_mirror_component(root_path: str, *, sidecar_root: Optional[str] = None) -> str:
|
||||
"""Return the mirror component identifying a model root.
|
||||
|
||||
The deterministic ``<sanitized basename>-<path digest>`` until the root map
|
||||
pins a different name for it — which happens only when a remembered root
|
||||
was moved or renamed and had to be re-anchored. Two roots sharing a
|
||||
basename always get distinct components, so their mirrors cannot collide.
|
||||
"""
|
||||
|
||||
resolved_root = sidecar_root if sidecar_root is not None else get_configured_sidecar_root()
|
||||
normalized = _normalize_for_match(root_path)
|
||||
if not resolved_root:
|
||||
return _legacy_component(normalized)
|
||||
state = _ensure_root_map(resolved_root, _normalized_roots(_iter_model_roots()))
|
||||
return state.components.get(normalized) or _legacy_component(normalized)
|
||||
|
||||
|
||||
def resolve_centralized_dir(model_path: str) -> Optional[str]:
|
||||
"""Return the centralized mirror directory for ``model_path``.
|
||||
|
||||
The mirror layout is
|
||||
``<sidecar_root>/<library>/<root_basename-roothash>/<rel_dir>``
|
||||
where ``rel_dir`` is the model's directory relative to the model root that
|
||||
The mirror layout is ``<sidecar_root>/<root_component>/<rel_dir>`` where
|
||||
``rel_dir`` is the model's directory relative to the model root that
|
||||
contains it. The longest matching root wins so nested roots resolve to the
|
||||
most specific mirror. Returns ``None`` when centralized storage is inactive
|
||||
or the path is not under any configured model root.
|
||||
@@ -237,33 +948,55 @@ def resolve_centralized_dir_for_dir(
|
||||
return None
|
||||
|
||||
normalized_dir = _normalize_for_match(model_dir)
|
||||
|
||||
best_root: Optional[str] = None
|
||||
for candidate in _iter_model_roots():
|
||||
if not candidate:
|
||||
continue
|
||||
normalized = _normalize_for_match(candidate)
|
||||
if normalized_dir == normalized or normalized_dir.startswith(normalized + os.sep):
|
||||
if best_root is None or len(normalized) > len(best_root):
|
||||
best_root = normalized
|
||||
|
||||
normalized_roots = _normalized_roots(_iter_model_roots())
|
||||
best_root = _best_root_for(normalized_dir, normalized_roots)
|
||||
if best_root is None:
|
||||
return None
|
||||
|
||||
try:
|
||||
from ..services.settings_manager import get_settings_manager
|
||||
|
||||
library = get_settings_manager().get_active_library_name() or "default"
|
||||
except Exception: # pragma: no cover - defensive fallback
|
||||
library = "default"
|
||||
|
||||
rel_dir = os.path.relpath(normalized_dir, best_root)
|
||||
parts = [root, sanitize_path_component(library), root_mirror_component(best_root)]
|
||||
component = _component_for(best_root, root, normalized_roots, rel_dir)
|
||||
if not component:
|
||||
return None
|
||||
|
||||
parts = [root, component]
|
||||
if rel_dir and rel_dir != os.curdir:
|
||||
parts.extend(sanitize_path_component(part) for part in rel_dir.split(os.sep) if part not in ("", os.curdir))
|
||||
parts.extend(
|
||||
sanitize_path_component(part)
|
||||
for part in rel_dir.split(os.sep)
|
||||
if part not in ("", os.curdir)
|
||||
)
|
||||
return os.path.join(*parts)
|
||||
|
||||
|
||||
def get_unmatched_sidecar_components() -> List[Dict[str, str]]:
|
||||
"""Describe mirrored sidecar directories no configured root claims.
|
||||
|
||||
Empty unless centralized storage is active. Used by diagnostics to surface
|
||||
metadata stranded by a moved model root (see :func:`_warn_unmatched`).
|
||||
"""
|
||||
|
||||
sidecar_root = get_sidecar_root()
|
||||
if not sidecar_root:
|
||||
return []
|
||||
|
||||
state = _ensure_root_map(sidecar_root, _normalized_roots(_iter_model_roots()))
|
||||
by_component: Dict[str, Dict[str, object]] = {}
|
||||
for entry in state.entries.values():
|
||||
by_component.setdefault(str(entry.get("component") or ""), entry)
|
||||
|
||||
unmatched: List[Dict[str, str]] = []
|
||||
for component in state.unmatched:
|
||||
entry = by_component.get(component) or {}
|
||||
unmatched.append(
|
||||
{
|
||||
"component": component,
|
||||
"basename": str(entry.get("basename") or ""),
|
||||
"last_path": str(entry.get("last_path") or ""),
|
||||
}
|
||||
)
|
||||
return unmatched
|
||||
|
||||
|
||||
def get_sidecar_dir(model_path: str) -> str:
|
||||
"""Return the directory holding the model's sidecar/preview assets.
|
||||
|
||||
|
||||
+1
-1
@@ -81,7 +81,7 @@ body {
|
||||
--badge-skip-refresh-glow: var(--color-skip-refresh-glow);
|
||||
}
|
||||
|
||||
:root[data-theme="dark"] {
|
||||
[data-theme="dark"] {
|
||||
--bg-color: var(--bg-base);
|
||||
--text-color: var(--text-primary);
|
||||
--text-muted: var(--text-secondary);
|
||||
|
||||
@@ -572,7 +572,7 @@
|
||||
}
|
||||
|
||||
/* Dark mode: use each preset's dark-mode accent lightness for visibility.
|
||||
These match the --color-accent-l values from :root[data-theme="dark"][data-theme-preset="..."]
|
||||
These match the --color-accent-l values from [data-theme="dark"][data-theme-preset="..."]
|
||||
in tokens/colors.css so the swatch accurately previews what the theme looks like. */
|
||||
|
||||
[data-theme="dark"] .preset-swatch-default {
|
||||
|
||||
@@ -78,7 +78,7 @@
|
||||
--favorite-glow: oklch(65% 0.15 85 / 0.5);
|
||||
}
|
||||
|
||||
:root[data-theme="dark"] {
|
||||
[data-theme="dark"] {
|
||||
--bg-base: #1a1a1a;
|
||||
--bg-elevated: oklch(25% 0.02 256 / 0.98);
|
||||
--bg-overlay: oklch(0% 0 0 / 0.75);
|
||||
@@ -118,7 +118,7 @@
|
||||
|
||||
/* ── Preset: Nord ──────────────────────────────────────────── */
|
||||
|
||||
:root[data-theme-preset="nord"] {
|
||||
[data-theme-preset="nord"] {
|
||||
--color-accent-h: 213;
|
||||
--color-accent-c: 0.18;
|
||||
--color-accent-l: 62%;
|
||||
@@ -152,7 +152,7 @@
|
||||
--favorite-glow: oklch(72% 0.14 85 / 0.5);
|
||||
}
|
||||
|
||||
:root[data-theme="dark"][data-theme-preset="nord"] {
|
||||
[data-theme="dark"][data-theme-preset="nord"] {
|
||||
--color-accent-h: 213;
|
||||
--color-accent-c: 0.18;
|
||||
--color-accent-l: 68%;
|
||||
@@ -188,7 +188,7 @@
|
||||
|
||||
/* ── Preset: Midnight ───────────────────────────────────────── */
|
||||
|
||||
:root[data-theme-preset="midnight"] {
|
||||
[data-theme-preset="midnight"] {
|
||||
--color-accent-h: 300;
|
||||
--color-accent-c: 0.15;
|
||||
--color-accent-l: 52%;
|
||||
@@ -222,7 +222,7 @@
|
||||
--favorite-glow: oklch(72% 0.16 85 / 0.5);
|
||||
}
|
||||
|
||||
:root[data-theme="dark"][data-theme-preset="midnight"] {
|
||||
[data-theme="dark"][data-theme-preset="midnight"] {
|
||||
--color-accent-h: 300;
|
||||
--color-accent-c: 0.14;
|
||||
--color-accent-l: 68%;
|
||||
@@ -258,7 +258,7 @@
|
||||
|
||||
/* ── Preset: Monokai ───────────────────────────────────────── */
|
||||
|
||||
:root[data-theme-preset="monokai"] {
|
||||
[data-theme-preset="monokai"] {
|
||||
--color-accent-h: 190;
|
||||
--color-accent-c: 0.24;
|
||||
--color-accent-l: 72%;
|
||||
@@ -291,7 +291,7 @@
|
||||
--favorite-glow: oklch(72% 0.16 85 / 0.5);
|
||||
}
|
||||
|
||||
:root[data-theme="dark"][data-theme-preset="monokai"] {
|
||||
[data-theme="dark"][data-theme-preset="monokai"] {
|
||||
--color-accent-h: 190;
|
||||
--color-accent-c: 0.24;
|
||||
--color-accent-l: 72%;
|
||||
@@ -326,7 +326,7 @@
|
||||
|
||||
/* ── Preset: Dracula ───────────────────────────────────────── */
|
||||
|
||||
:root[data-theme-preset="dracula"] {
|
||||
[data-theme-preset="dracula"] {
|
||||
--color-accent-h: 265;
|
||||
--color-accent-c: 0.24;
|
||||
--color-accent-l: 68%;
|
||||
@@ -359,7 +359,7 @@
|
||||
--favorite-glow: oklch(72% 0.16 85 / 0.5);
|
||||
}
|
||||
|
||||
:root[data-theme="dark"][data-theme-preset="dracula"] {
|
||||
[data-theme="dark"][data-theme-preset="dracula"] {
|
||||
--color-accent-h: 265;
|
||||
--color-accent-c: 0.24;
|
||||
--color-accent-l: 72%;
|
||||
@@ -394,7 +394,7 @@
|
||||
|
||||
/* ── Preset: Solarized ─────────────────────────────────────── */
|
||||
|
||||
:root[data-theme-preset="solarized"] {
|
||||
[data-theme-preset="solarized"] {
|
||||
--color-accent-h: 175;
|
||||
--color-accent-c: 0.18;
|
||||
--color-accent-l: 55%;
|
||||
@@ -429,7 +429,7 @@
|
||||
--favorite-glow: oklch(68% 0.16 75 / 0.5);
|
||||
}
|
||||
|
||||
:root[data-theme="dark"][data-theme-preset="solarized"] {
|
||||
[data-theme="dark"][data-theme-preset="solarized"] {
|
||||
--color-accent-h: 175;
|
||||
--color-accent-c: 0.18;
|
||||
--color-accent-l: 60%;
|
||||
|
||||
@@ -72,15 +72,6 @@ export class DownloadManager {
|
||||
completeMetadata.diagnostics = diagnostics;
|
||||
}
|
||||
|
||||
// A ComfyUI workflow recovered from the source's original
|
||||
// rendition: CivitAI's optimized preview is re-encoded and
|
||||
// metadata-free, so the workflow travels as data and the
|
||||
// backend embeds it into the stored image.
|
||||
const workflow = this.importManager.recipeData.workflow;
|
||||
if (workflow) {
|
||||
completeMetadata.workflow = workflow;
|
||||
}
|
||||
|
||||
// Preserve preview_nsfw_level from analysis so the saved
|
||||
// recipe applies the correct NSFW blur on the preview image.
|
||||
const nsfwLevel = this.importManager.recipeData.preview_nsfw_level;
|
||||
|
||||
@@ -1,79 +0,0 @@
|
||||
import { describe, it, beforeEach, afterEach, expect, vi } from 'vitest';
|
||||
|
||||
const { EVENTS_MODULE, API_MODULE, APP_MODULE, COMPONENTS_MODULE, UTILS_MODULE } =
|
||||
vi.hoisted(() => ({
|
||||
EVENTS_MODULE: new URL('../../../web/comfyui/loras_widget_events.js', import.meta.url)
|
||||
.pathname,
|
||||
API_MODULE: new URL('../../../scripts/api.js', import.meta.url).pathname,
|
||||
APP_MODULE: new URL('../../../scripts/app.js', import.meta.url).pathname,
|
||||
COMPONENTS_MODULE: new URL('../../../web/comfyui/loras_widget_components.js', import.meta.url)
|
||||
.pathname,
|
||||
UTILS_MODULE: new URL('../../../web/comfyui/loras_widget_utils.js', import.meta.url)
|
||||
.pathname,
|
||||
}));
|
||||
|
||||
const saveRecipeDirectly = vi.fn();
|
||||
|
||||
vi.mock(API_MODULE, () => ({ api: {} }));
|
||||
vi.mock(APP_MODULE, () => ({ app: {} }));
|
||||
|
||||
vi.mock(COMPONENTS_MODULE, () => ({
|
||||
createMenuItem: (text, icon, onClick) => {
|
||||
const el = document.createElement('div');
|
||||
el.className = 'lm-lora-menu-item';
|
||||
el.textContent = text;
|
||||
if (onClick) el.addEventListener('click', onClick);
|
||||
return el;
|
||||
},
|
||||
createDropIndicator: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock(UTILS_MODULE, () => ({
|
||||
parseLoraValue: vi.fn(() => []),
|
||||
formatLoraValue: vi.fn((value) => value),
|
||||
syncClipStrengthIfCollapsed: vi.fn(),
|
||||
saveRecipeDirectly,
|
||||
copyToClipboard: vi.fn(),
|
||||
showToast: vi.fn(),
|
||||
moveLoraByDirection: vi.fn(),
|
||||
getDropTargetIndex: vi.fn(),
|
||||
getLoraStrengthRange: vi.fn(),
|
||||
applyStrengthRangeCue: vi.fn(),
|
||||
}));
|
||||
|
||||
function findMenuItem(label) {
|
||||
return Array.from(document.querySelectorAll('.lm-lora-menu-item')).find(
|
||||
(item) => item.textContent === label
|
||||
);
|
||||
}
|
||||
|
||||
describe('LoRA widget context menu save options', () => {
|
||||
beforeEach(() => {
|
||||
document.body.innerHTML = '';
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
document.body.innerHTML = '';
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
it('offers workflow embedding as a separate, opt-in action', async () => {
|
||||
const { createContextMenu } = await import(EVENTS_MODULE);
|
||||
const widget = { value: [], callback: vi.fn() };
|
||||
|
||||
createContextMenu(10, 10, 'lora-a', widget, null, vi.fn());
|
||||
|
||||
const plain = findMenuItem('Save Recipe');
|
||||
const withWorkflow = findMenuItem('Save Recipe with Workflow');
|
||||
expect(plain).toBeTruthy();
|
||||
expect(withWorkflow).toBeTruthy();
|
||||
|
||||
plain.click();
|
||||
expect(saveRecipeDirectly).toHaveBeenLastCalledWith();
|
||||
|
||||
// Re-open: the first click removed the menu.
|
||||
createContextMenu(10, 10, 'lora-a', widget, null, vi.fn());
|
||||
findMenuItem('Save Recipe with Workflow').click();
|
||||
expect(saveRecipeDirectly).toHaveBeenLastCalledWith({ embedWorkflow: true });
|
||||
});
|
||||
});
|
||||
@@ -1,101 +0,0 @@
|
||||
import { describe, it, beforeEach, afterEach, expect, vi } from 'vitest';
|
||||
|
||||
const { showToastMock, translateMock } = vi.hoisted(() => ({
|
||||
showToastMock: vi.fn(),
|
||||
translateMock: vi.fn((key, params, fallback) =>
|
||||
typeof fallback === 'string' ? fallback : key
|
||||
),
|
||||
}));
|
||||
|
||||
vi.mock('../../../static/js/utils/uiHelpers.js', () => ({
|
||||
showToast: showToastMock,
|
||||
}));
|
||||
|
||||
vi.mock('../../../static/js/utils/i18nHelpers.js', () => ({
|
||||
translate: translateMock,
|
||||
}));
|
||||
|
||||
vi.mock('../../../static/js/api/modelApiFactory.js', () => ({
|
||||
getModelApiClient: vi.fn(() => ({})),
|
||||
}));
|
||||
|
||||
vi.mock('../../../static/js/api/apiConfig.js', () => ({
|
||||
MODEL_TYPES: { LORA: 'loras', CHECKPOINT: 'checkpoints', EMBEDDING: 'embeddings' },
|
||||
}));
|
||||
|
||||
vi.mock('../../../static/js/utils/storageHelpers.js', () => ({
|
||||
getStorageItem: vi.fn(() => null),
|
||||
}));
|
||||
|
||||
vi.mock('../../../static/js/state/index.js', () => ({
|
||||
state: { virtualScroller: null },
|
||||
}));
|
||||
|
||||
import { DownloadManager } from '../../../static/js/managers/import/DownloadManager.js';
|
||||
|
||||
function buildImportManager(recipeData) {
|
||||
return {
|
||||
recipeId: null,
|
||||
recipeName: 'Test Recipe',
|
||||
recipeImage: null,
|
||||
recipeTags: [],
|
||||
downloadableLoRAs: [],
|
||||
importMode: 'url',
|
||||
recipeData,
|
||||
loadingManager: { showSimpleLoading: vi.fn(), hide: vi.fn() },
|
||||
};
|
||||
}
|
||||
|
||||
async function saveAndReadMetadata(recipeData) {
|
||||
let capturedBody = null;
|
||||
const fetchMock = vi.fn(async (url, options) => {
|
||||
capturedBody = options?.body ?? null;
|
||||
return { ok: true, json: async () => ({ success: true }) };
|
||||
});
|
||||
globalThis.fetch = fetchMock;
|
||||
window.fetch = fetchMock;
|
||||
|
||||
const manager = new DownloadManager(buildImportManager(recipeData));
|
||||
await manager.saveRecipe(true);
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledWith('/api/lm/recipes/save', expect.anything());
|
||||
expect(capturedBody).toBeInstanceOf(FormData);
|
||||
return JSON.parse(capturedBody.get('metadata'));
|
||||
}
|
||||
|
||||
describe('recipe import save payload', () => {
|
||||
beforeEach(() => {
|
||||
globalThis.modalManager = { closeModal: vi.fn() };
|
||||
globalThis.window.recipeManager = { loadRecipes: vi.fn() };
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
delete globalThis.modalManager;
|
||||
delete globalThis.window.recipeManager;
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
it('forwards a workflow recovered from the original rendition', async () => {
|
||||
const workflow = '{"nodes": [{"id": 1}]}';
|
||||
const metadata = await saveAndReadMetadata({
|
||||
image_base64: 'AAAA',
|
||||
base_model: 'sd',
|
||||
loras: [],
|
||||
gen_params: {},
|
||||
workflow,
|
||||
});
|
||||
|
||||
expect(metadata.workflow).toBe(workflow);
|
||||
});
|
||||
|
||||
it('omits the workflow key when analysis recovered none', async () => {
|
||||
const metadata = await saveAndReadMetadata({
|
||||
image_base64: 'AAAA',
|
||||
base_model: 'sd',
|
||||
loras: [],
|
||||
gen_params: {},
|
||||
});
|
||||
|
||||
expect('workflow' in metadata).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -1,68 +0,0 @@
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { readFileSync } from 'fs';
|
||||
import path from 'path';
|
||||
|
||||
// Regression guard: theme palette tokens may only be declared on the root element.
|
||||
//
|
||||
// `applyTheme()` (static/js/utils/uiHelpers.js, and Header.setThemeMode) mirrors
|
||||
// the active mode onto <body> as `data-theme="dark"`, but the theme *preset* is
|
||||
// only ever written to <html> (`data-theme-preset`). While the token blocks in
|
||||
// tokens/colors.css used the bare attribute selector `[data-theme="dark"]`, the
|
||||
// body matched that block on its own and re-declared the DEFAULT dark palette
|
||||
// (#1a1a1a / #2d2d2d), shadowing the preset palette it inherited from <html>.
|
||||
// The page painted the selected preset's accent but the default preset's
|
||||
// backgrounds/surfaces/text, and flipped into that state ~200ms after load when
|
||||
// initTheme() first touched <body> — the accent-tinted background flash on
|
||||
// reload and nav-tab switches, visible under every preset except "default".
|
||||
// Root-scoping the token blocks keeps <body>'s data-theme inert.
|
||||
describe('Theme token scope', () => {
|
||||
const repoRoot = path.resolve(__dirname, '../../..');
|
||||
const read = (rel) => readFileSync(path.join(repoRoot, rel), 'utf-8');
|
||||
const stripComments = (css) => css.replace(/\/\*[\s\S]*?\*\//g, '');
|
||||
|
||||
const COLOR_TOKENS = read('static/css/tokens/colors.css');
|
||||
const BASE_CSS = read('static/css/base.css');
|
||||
|
||||
// Selectors that declare a palette token, e.g. `--bg-base:` / `--lora-surface:`.
|
||||
const tokenDeclaringSelectors = (css) => {
|
||||
const selectors = [];
|
||||
const ruleRe = /([^{}]+)\{([^{}]*)\}/g;
|
||||
let match;
|
||||
while ((match = ruleRe.exec(stripComments(css)))) {
|
||||
const selector = match[1].trim();
|
||||
const declaresPaletteToken = /(^|[;\s])--(?:bg|surface|text|border|color|favorite|lora|badge|card)-[\w-]+\s*:/.test(
|
||||
match[2]
|
||||
);
|
||||
if (declaresPaletteToken) selectors.push(selector);
|
||||
}
|
||||
return selectors;
|
||||
};
|
||||
|
||||
const isRootScoped = (selector) =>
|
||||
selector.split(',').every((part) => /^\s*(:root|html)\b/.test(part));
|
||||
|
||||
it('declares every colors.css palette token on :root', () => {
|
||||
const selectors = tokenDeclaringSelectors(COLOR_TOKENS);
|
||||
expect(selectors.length).toBeGreaterThan(0);
|
||||
expect(selectors.filter((selector) => !isRootScoped(selector))).toEqual([]);
|
||||
});
|
||||
|
||||
it('declares every base.css palette alias on :root', () => {
|
||||
const selectors = tokenDeclaringSelectors(BASE_CSS);
|
||||
expect(selectors.length).toBeGreaterThan(0);
|
||||
expect(selectors.filter((selector) => !isRootScoped(selector))).toEqual([]);
|
||||
});
|
||||
|
||||
it('keeps dark/preset token blocks anchored to the root element', () => {
|
||||
const css = stripComments(COLOR_TOKENS);
|
||||
expect(css).toContain(':root[data-theme="dark"] {');
|
||||
for (const preset of ['nord', 'midnight', 'monokai', 'dracula', 'solarized']) {
|
||||
expect(css).toContain(`:root[data-theme-preset="${preset}"] {`);
|
||||
expect(css).toContain(`:root[data-theme="dark"][data-theme-preset="${preset}"] {`);
|
||||
}
|
||||
// No bare attribute selector may open a rule: <body> carries data-theme
|
||||
// without the preset, so such a block would re-declare the default palette.
|
||||
expect(css).not.toMatch(/(^|\n)\s*\[data-theme/);
|
||||
expect(stripComments(BASE_CSS)).not.toMatch(/(^|\n)\s*\[data-theme/);
|
||||
});
|
||||
});
|
||||
@@ -1,92 +0,0 @@
|
||||
import { describe, it, beforeEach, afterEach, expect, vi } from 'vitest';
|
||||
|
||||
const { UTILS_MODULE, APP_MODULE, API_MODULE, BASE_PATH_MODULE } = vi.hoisted(() => ({
|
||||
UTILS_MODULE: new URL('../../../web/comfyui/loras_widget_utils.js', import.meta.url).pathname,
|
||||
APP_MODULE: new URL('../../../scripts/app.js', import.meta.url).pathname,
|
||||
API_MODULE: new URL('../../../scripts/api.js', import.meta.url).pathname,
|
||||
BASE_PATH_MODULE: new URL('../../../web/comfyui/base_path.js', import.meta.url).pathname,
|
||||
}));
|
||||
|
||||
const toastAdd = vi.fn();
|
||||
const graphToPrompt = vi.fn();
|
||||
|
||||
vi.mock(APP_MODULE, () => ({
|
||||
app: {
|
||||
graphToPrompt,
|
||||
extensionManager: { toast: { add: toastAdd } },
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock(API_MODULE, () => ({
|
||||
api: { fetchApi: vi.fn() },
|
||||
}));
|
||||
|
||||
vi.mock(BASE_PATH_MODULE, () => ({
|
||||
lmUrl: (path) => `/lm${path}`,
|
||||
}));
|
||||
|
||||
async function runSave(options, responseBody) {
|
||||
let captured = null;
|
||||
globalThis.fetch = vi.fn(async (url, init) => {
|
||||
captured = { url, init };
|
||||
return { json: async () => responseBody };
|
||||
});
|
||||
|
||||
const { saveRecipeDirectly } = await import(UTILS_MODULE);
|
||||
await saveRecipeDirectly(options);
|
||||
return captured;
|
||||
}
|
||||
|
||||
describe('saveRecipeDirectly', () => {
|
||||
beforeEach(() => {
|
||||
graphToPrompt.mockResolvedValue({
|
||||
workflow: { nodes: [{ id: 1 }], last_node_id: 1 },
|
||||
output: { 1: { class_type: 'KSampler' } },
|
||||
});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
delete globalThis.fetch;
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
it('posts no workflow by default', async () => {
|
||||
const captured = await runSave(undefined, { success: true, has_workflow: false });
|
||||
|
||||
expect(captured.init.body).toBe('{}');
|
||||
expect(JSON.parse(captured.init.body)).not.toHaveProperty('workflow');
|
||||
});
|
||||
|
||||
it('embeds the UI-format graph when asked', async () => {
|
||||
const captured = await runSave(
|
||||
{ embedWorkflow: true },
|
||||
{ success: true, has_workflow: true }
|
||||
);
|
||||
|
||||
const body = JSON.parse(captured.init.body);
|
||||
expect(body.workflow).toEqual({ nodes: [{ id: 1 }], last_node_id: 1 });
|
||||
expect(captured.init.headers['Content-Type']).toBe('application/json');
|
||||
});
|
||||
|
||||
it('reports a skipped oversized workflow as a warning', async () => {
|
||||
await runSave(
|
||||
{ embedWorkflow: true },
|
||||
{ success: true, has_workflow: false, workflow_skipped: 'too_large' }
|
||||
);
|
||||
|
||||
const lastToast = toastAdd.mock.calls.at(-1)[0];
|
||||
expect(lastToast.severity).toBe('warn');
|
||||
expect(lastToast.summary).toBe('Recipe Saved without Workflow');
|
||||
expect(lastToast.detail).toContain('too large');
|
||||
});
|
||||
|
||||
it('reports a successful embed distinctly from a plain save', async () => {
|
||||
await runSave(
|
||||
{ embedWorkflow: true },
|
||||
{ success: true, has_workflow: true }
|
||||
);
|
||||
|
||||
const lastToast = toastAdd.mock.calls.at(-1)[0];
|
||||
expect(lastToast.summary).toBe('Recipe Saved with Workflow');
|
||||
});
|
||||
});
|
||||
@@ -12,6 +12,7 @@ import pytest
|
||||
from aiohttp import web
|
||||
|
||||
from py.services.model_hash_index import ModelHashIndex
|
||||
from py.routes.handlers import misc_handlers
|
||||
from py.routes.handlers.misc_handlers import (
|
||||
BackupHandler,
|
||||
DoctorHandler,
|
||||
@@ -209,6 +210,108 @@ async def test_doctor_handler_reports_key_cache_and_ui_issues():
|
||||
assert diagnostic_map["ui_version"]["status"] == "warning"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_doctor_handler_flags_orphaned_centralized_sidecars(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
):
|
||||
async def civitai_factory():
|
||||
return DummyCivitaiClient()
|
||||
|
||||
handler = DoctorHandler(
|
||||
settings_service=DummySettings({"civitai_api_key": "token"}),
|
||||
civitai_client_factory=civitai_factory,
|
||||
scanner_factories=(),
|
||||
)
|
||||
monkeypatch.setattr(misc_handlers, "get_storage_mode", lambda: "centralized")
|
||||
monkeypatch.setattr(
|
||||
misc_handlers,
|
||||
"get_unmatched_sidecar_components",
|
||||
lambda: [
|
||||
{
|
||||
"component": "loras-abc12345",
|
||||
"basename": "loras",
|
||||
"last_path": "/mnt/old/loras",
|
||||
},
|
||||
{"component": "loras-def67890", "basename": "loras", "last_path": ""},
|
||||
],
|
||||
)
|
||||
|
||||
response = await handler.get_doctor_diagnostics(
|
||||
FakeRequest(method="GET") # pyright: ignore[reportArgumentType]
|
||||
)
|
||||
payload = _json_payload(response)
|
||||
item = {entry["id"]: entry for entry in payload["diagnostics"]}[
|
||||
"sidecar_mirror_orphans"
|
||||
]
|
||||
|
||||
assert item["status"] == "warning"
|
||||
assert "2 sidecar directories" in item["summary"]
|
||||
assert any("/mnt/old/loras" in line for line in item["details"])
|
||||
assert any("unknown" in line for line in item["details"])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_doctor_handler_sidecar_check_ok_when_all_linked(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
):
|
||||
async def civitai_factory():
|
||||
return DummyCivitaiClient()
|
||||
|
||||
handler = DoctorHandler(
|
||||
settings_service=DummySettings({"civitai_api_key": "token"}),
|
||||
civitai_client_factory=civitai_factory,
|
||||
scanner_factories=(),
|
||||
)
|
||||
monkeypatch.setattr(misc_handlers, "get_storage_mode", lambda: "centralized")
|
||||
monkeypatch.setattr(misc_handlers, "get_unmatched_sidecar_components", lambda: [])
|
||||
monkeypatch.setattr(
|
||||
misc_handlers, "describe_sidecar_root", lambda: {"root": "/sidecars"}
|
||||
)
|
||||
|
||||
response = await handler.get_doctor_diagnostics(
|
||||
FakeRequest(method="GET") # pyright: ignore[reportArgumentType]
|
||||
)
|
||||
payload = _json_payload(response)
|
||||
item = {entry["id"]: entry for entry in payload["diagnostics"]}[
|
||||
"sidecar_mirror_orphans"
|
||||
]
|
||||
|
||||
assert item["status"] == "ok"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_doctor_handler_sidecar_check_skipped_alongside(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
):
|
||||
async def civitai_factory():
|
||||
return DummyCivitaiClient()
|
||||
|
||||
handler = DoctorHandler(
|
||||
settings_service=DummySettings({"civitai_api_key": "token"}),
|
||||
civitai_client_factory=civitai_factory,
|
||||
scanner_factories=(),
|
||||
)
|
||||
monkeypatch.setattr(misc_handlers, "get_storage_mode", lambda: "alongside")
|
||||
|
||||
def _unexpected():
|
||||
raise AssertionError("orphan lookup must not run in alongside mode")
|
||||
|
||||
monkeypatch.setattr(
|
||||
misc_handlers, "get_unmatched_sidecar_components", _unexpected
|
||||
)
|
||||
|
||||
response = await handler.get_doctor_diagnostics(
|
||||
FakeRequest(method="GET") # pyright: ignore[reportArgumentType]
|
||||
)
|
||||
payload = _json_payload(response)
|
||||
item = {entry["id"]: entry for entry in payload["diagnostics"]}[
|
||||
"sidecar_mirror_orphans"
|
||||
]
|
||||
|
||||
assert item["status"] == "ok"
|
||||
assert "alongside" in item["summary"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_doctor_handler_can_repair_cache():
|
||||
scanner = DummyDoctorScanner(model_type="lora", raw_data=[])
|
||||
|
||||
@@ -1,262 +0,0 @@
|
||||
"""Workflow preservation for remote recipe imports.
|
||||
|
||||
CivitAI serves a re-encoded, metadata-free ``optimized`` rendition as the
|
||||
recipe preview, so an embedded ComfyUI workflow only exists in the
|
||||
``original=true`` image. These tests pin the recovery and transport of that
|
||||
workflow through the remote import paths.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from PIL import Image, PngImagePlugin
|
||||
|
||||
from py.routes.handlers.recipe_handlers import RecipeManagementHandler
|
||||
from py.services.recipes.persistence_service import PersistenceResult
|
||||
from py.utils.exif_utils import ExifUtils
|
||||
|
||||
|
||||
async def _noop_ensure() -> None:
|
||||
return None
|
||||
|
||||
|
||||
class CapturingPersistence:
|
||||
"""Persistence service double recording the save payload."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.calls: list[dict[str, Any]] = []
|
||||
|
||||
async def save_recipe(self, **kwargs: Any) -> PersistenceResult:
|
||||
self.calls.append(kwargs)
|
||||
return PersistenceResult({"success": True, "recipe_id": "recipe-1"})
|
||||
|
||||
|
||||
class StubScanner:
|
||||
"""Scanner double exposing only what the remote import paths touch."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.recipes_dir = "/tmp/recipes"
|
||||
|
||||
async def build_local_hash_cache(self) -> dict[str, Any]:
|
||||
return {}
|
||||
|
||||
async def get_local_lora(self, name, base_model=None):
|
||||
return None
|
||||
|
||||
|
||||
def _make_handler(
|
||||
persistence: CapturingPersistence,
|
||||
*,
|
||||
downloader_factory=None,
|
||||
) -> RecipeManagementHandler:
|
||||
async def default_downloader_factory():
|
||||
class Downloader:
|
||||
async def download_file(self, url, path, use_auth=False):
|
||||
Path(path).write_bytes(b"downloaded")
|
||||
return True, "ok"
|
||||
|
||||
return Downloader()
|
||||
|
||||
analysis_service = SimpleNamespace(
|
||||
_recipe_parser_factory=SimpleNamespace(create_parser=lambda metadata: None)
|
||||
)
|
||||
|
||||
return RecipeManagementHandler(
|
||||
ensure_dependencies_ready=_noop_ensure,
|
||||
recipe_scanner_getter=lambda: StubScanner(),
|
||||
logger=logging.getLogger(__name__),
|
||||
persistence_service=persistence, # pyright: ignore[reportArgumentType]
|
||||
analysis_service=analysis_service, # pyright: ignore[reportArgumentType]
|
||||
downloader_factory=downloader_factory or default_downloader_factory,
|
||||
civitai_client_getter=lambda: None,
|
||||
)
|
||||
|
||||
|
||||
def _meta_with_comfy() -> dict[str, Any]:
|
||||
return {
|
||||
"id": 143518055,
|
||||
"meta": {"prompt": "p", "comfy": '{"prompt": {"1": {"class_type": "KSampler"}}}'},
|
||||
}
|
||||
|
||||
|
||||
def test_meta_indicates_comfy_workflow() -> None:
|
||||
assert RecipeManagementHandler._meta_indicates_comfy_workflow(
|
||||
{"meta": {"comfy": "{}"}}
|
||||
)
|
||||
assert RecipeManagementHandler._meta_indicates_comfy_workflow({"comfy": "{}"})
|
||||
assert not RecipeManagementHandler._meta_indicates_comfy_workflow({"meta": {}})
|
||||
assert not RecipeManagementHandler._meta_indicates_comfy_workflow(
|
||||
{"meta": {"comfy": None}}
|
||||
)
|
||||
assert not RecipeManagementHandler._meta_indicates_comfy_workflow(None)
|
||||
assert not RecipeManagementHandler._meta_indicates_comfy_workflow("comfy")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_original_media_reads_workflow_and_cleans_up(tmp_path, monkeypatch):
|
||||
workflow = json.dumps({"nodes": [{"id": 1}], "last_node_id": 1})
|
||||
source = tmp_path / "original.png"
|
||||
png_info = PngImagePlugin.PngInfo()
|
||||
png_info.add_text("workflow", workflow)
|
||||
png_info.add_text("prompt", '{"1": {"class_type": "KSampler"}}')
|
||||
Image.new("RGB", (32, 32), color="red").save(source, pnginfo=png_info)
|
||||
|
||||
written: list[str] = []
|
||||
|
||||
async def downloader_factory():
|
||||
class Downloader:
|
||||
async def download_file(self, url, path, use_auth=False):
|
||||
written.append(str(path))
|
||||
Path(path).write_bytes(source.read_bytes())
|
||||
return True, "ok"
|
||||
|
||||
return Downloader()
|
||||
|
||||
handler = _make_handler(CapturingPersistence(), downloader_factory=downloader_factory)
|
||||
|
||||
raw_metadata, recovered = await handler._fetch_original_media(
|
||||
"https://image.civitai.com/x/original=true/x.png"
|
||||
)
|
||||
|
||||
assert recovered == workflow
|
||||
# extract_image_metadata prefers the prompt chunk over the workflow.
|
||||
assert raw_metadata is not None and "class_type" in raw_metadata
|
||||
assert written and not os.path.exists(written[0])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_original_media_degrades_on_download_failure():
|
||||
async def downloader_factory():
|
||||
class Downloader:
|
||||
async def download_file(self, url, path, use_auth=False):
|
||||
return False, "boom"
|
||||
|
||||
return Downloader()
|
||||
|
||||
handler = _make_handler(CapturingPersistence(), downloader_factory=downloader_factory)
|
||||
|
||||
assert await handler._fetch_original_media("https://image.civitai.com/x.png") == (
|
||||
None,
|
||||
None,
|
||||
)
|
||||
assert await handler._fetch_original_media(None) == (None, None)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_remote_import_transports_workflow_to_save(monkeypatch):
|
||||
workflow = json.dumps({"nodes": [{"id": 4}]})
|
||||
persistence = CapturingPersistence()
|
||||
handler = _make_handler(persistence)
|
||||
|
||||
async def fake_download_remote_media(image_url):
|
||||
return (
|
||||
b"optimized-preview",
|
||||
".jpg",
|
||||
_meta_with_comfy(),
|
||||
12345,
|
||||
"https://image.civitai.com/x/original=true/x.png",
|
||||
)
|
||||
|
||||
fetched: list[str] = []
|
||||
|
||||
async def fake_fetch_original_media(original_url):
|
||||
fetched.append(original_url)
|
||||
return None, workflow
|
||||
|
||||
handler._download_remote_media = fake_download_remote_media # type: ignore[method-assign]
|
||||
handler._fetch_original_media = fake_fetch_original_media # type: ignore[method-assign]
|
||||
monkeypatch.setattr(
|
||||
ExifUtils, "extract_image_metadata", staticmethod(lambda path: None)
|
||||
)
|
||||
|
||||
response = await handler._do_import_remote_recipe(
|
||||
image_url="https://civitai.red/images/143518055",
|
||||
name="Recipe",
|
||||
lora_entries=[],
|
||||
checkpoint_entry=None,
|
||||
gen_params_request={},
|
||||
tags=[],
|
||||
base_model="Krea 2",
|
||||
source_path="https://civitai.red/images/143518055",
|
||||
)
|
||||
|
||||
assert response.status == 200
|
||||
assert fetched == ["https://image.civitai.com/x/original=true/x.png"]
|
||||
assert persistence.calls[0]["metadata"]["workflow"] == workflow
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_remote_import_skips_original_without_comfy_meta(monkeypatch):
|
||||
persistence = CapturingPersistence()
|
||||
handler = _make_handler(persistence)
|
||||
|
||||
async def fake_download_remote_media(image_url):
|
||||
return (
|
||||
b"optimized-preview",
|
||||
".jpg",
|
||||
{"id": 1, "meta": {"prompt": "p"}},
|
||||
None,
|
||||
"https://image.civitai.com/x/original=true/x.png",
|
||||
)
|
||||
|
||||
async def fail_fetch(original_url): # pragma: no cover - must not be called
|
||||
raise AssertionError("original rendition should not be fetched")
|
||||
|
||||
handler._download_remote_media = fake_download_remote_media # type: ignore[method-assign]
|
||||
handler._fetch_original_media = fail_fetch # type: ignore[method-assign]
|
||||
monkeypatch.setattr(
|
||||
ExifUtils, "extract_image_metadata", staticmethod(lambda path: None)
|
||||
)
|
||||
|
||||
response = await handler._do_import_remote_recipe(
|
||||
image_url="https://civitai.red/images/1",
|
||||
name="Recipe",
|
||||
lora_entries=[],
|
||||
checkpoint_entry=None,
|
||||
gen_params_request={},
|
||||
tags=[],
|
||||
base_model="SDXL 1.0",
|
||||
source_path="https://civitai.red/images/1",
|
||||
)
|
||||
|
||||
assert response.status == 200
|
||||
assert "workflow" not in persistence.calls[0]["metadata"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_url_import_transports_workflow_to_save(monkeypatch):
|
||||
workflow = json.dumps({"nodes": [{"id": 5}]})
|
||||
persistence = CapturingPersistence()
|
||||
handler = _make_handler(persistence)
|
||||
|
||||
async def fake_download_remote_media(image_url):
|
||||
return (
|
||||
b"optimized-preview",
|
||||
".jpg",
|
||||
{"id": 9, "meta": {"prompt": "p"}},
|
||||
None,
|
||||
"https://image.civitai.com/x/original=true/x.png",
|
||||
)
|
||||
|
||||
async def fake_fetch_original_media(original_url):
|
||||
return None, workflow
|
||||
|
||||
handler._download_remote_media = fake_download_remote_media # type: ignore[method-assign]
|
||||
handler._fetch_original_media = fake_fetch_original_media # type: ignore[method-assign]
|
||||
monkeypatch.setattr(
|
||||
ExifUtils, "extract_image_metadata", staticmethod(lambda path: None)
|
||||
)
|
||||
|
||||
response = await handler._do_import_from_url(
|
||||
"https://civitai.red/images/143518055", StubScanner()
|
||||
)
|
||||
|
||||
assert response.status == 200
|
||||
assert persistence.calls[0]["metadata"]["workflow"] == workflow
|
||||
@@ -208,11 +208,6 @@ class StubAnalysisService:
|
||||
self.remote_calls: List[Optional[str]] = []
|
||||
self.local_calls: List[Optional[str]] = []
|
||||
self.local_ignore_recipe_metadata_calls: List[bool] = []
|
||||
self.widget_analysis_calls: List[Any] = []
|
||||
self.widget_result = SimpleNamespace(
|
||||
payload={"metadata": {"loras": ""}, "image_bytes": b"widget-image"},
|
||||
status=200,
|
||||
)
|
||||
self.result = SimpleNamespace(payload={"loras": []}, status=200)
|
||||
self._recipe_parser_factory: Any = None
|
||||
StubAnalysisService.instances.append(self)
|
||||
@@ -247,8 +242,7 @@ class StubAnalysisService:
|
||||
return self.result
|
||||
|
||||
async def analyze_widget_metadata(self, *, recipe_scanner) -> SimpleNamespace:
|
||||
self.widget_analysis_calls.append(recipe_scanner)
|
||||
return self.widget_result
|
||||
return SimpleNamespace(payload={"metadata": {}, "image_bytes": b""}, status=200)
|
||||
|
||||
|
||||
class StubPersistenceService:
|
||||
@@ -258,7 +252,6 @@ class StubPersistenceService:
|
||||
|
||||
def __init__(self, **_: Any) -> None:
|
||||
self.save_calls: List[Dict[str, Any]] = []
|
||||
self.widget_calls: List[Dict[str, Any]] = []
|
||||
self.delete_calls: List[str] = []
|
||||
self.move_calls: List[Dict[str, str]] = []
|
||||
self.update_calls: List[Dict[str, Any]] = []
|
||||
@@ -366,24 +359,9 @@ class StubPersistenceService:
|
||||
)
|
||||
|
||||
async def save_recipe_from_widget(
|
||||
self,
|
||||
*,
|
||||
recipe_scanner,
|
||||
metadata: Dict[str, Any],
|
||||
image_bytes: bytes,
|
||||
workflow: Any = None,
|
||||
self, *, recipe_scanner, metadata: Dict[str, Any], image_bytes: bytes
|
||||
) -> SimpleNamespace: # pragma: no cover
|
||||
self.widget_calls.append(
|
||||
{
|
||||
"recipe_scanner": recipe_scanner,
|
||||
"metadata": metadata,
|
||||
"image_bytes": image_bytes,
|
||||
"workflow": workflow,
|
||||
}
|
||||
)
|
||||
return SimpleNamespace(
|
||||
payload={"success": True, "has_workflow": workflow is not None}, status=200
|
||||
)
|
||||
return SimpleNamespace(payload={"success": True}, status=200)
|
||||
|
||||
|
||||
class StubSharingService:
|
||||
@@ -503,33 +481,6 @@ async def recipe_harness(
|
||||
StubSharingService.instances.clear()
|
||||
|
||||
|
||||
async def test_save_from_widget_forwards_workflow_body(monkeypatch, tmp_path: Path) -> None:
|
||||
"""The opt-in workflow arrives through a real JSON body and is handed to
|
||||
the persistence layer; the response reports whether it was embedded."""
|
||||
async with recipe_harness(monkeypatch, tmp_path) as harness:
|
||||
workflow = {"nodes": [{"id": 1}], "last_node_id": 1}
|
||||
|
||||
response = await harness.client.post(
|
||||
"/api/lm/recipes/save-from-widget", json={"workflow": workflow}
|
||||
)
|
||||
payload = await response.json()
|
||||
|
||||
assert response.status == 200
|
||||
assert payload["has_workflow"] is True
|
||||
assert harness.persistence.widget_calls[0]["workflow"] == workflow
|
||||
|
||||
|
||||
async def test_save_from_widget_without_body_still_saves(monkeypatch, tmp_path: Path) -> None:
|
||||
"""The long-standing body-less POST must keep working unchanged."""
|
||||
async with recipe_harness(monkeypatch, tmp_path) as harness:
|
||||
response = await harness.client.post("/api/lm/recipes/save-from-widget")
|
||||
payload = await response.json()
|
||||
|
||||
assert response.status == 200
|
||||
assert payload["has_workflow"] is False
|
||||
assert harness.persistence.widget_calls[0]["workflow"] is None
|
||||
|
||||
|
||||
async def test_list_recipes_provides_file_urls(monkeypatch, tmp_path: Path) -> None:
|
||||
async with recipe_harness(monkeypatch, tmp_path) as harness:
|
||||
recipe_path = harness.tmp_dir / "recipes" / "demo.png"
|
||||
|
||||
@@ -1,129 +0,0 @@
|
||||
"""Handler tests for the widget "Save Recipe" endpoint.
|
||||
|
||||
Covers the opt-in workflow body: the endpoint historically received no body at
|
||||
all, so a missing, empty or malformed body must degrade to "no workflow"
|
||||
rather than failing the save.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from py.routes.handlers.recipe_handlers import RecipeManagementHandler
|
||||
|
||||
|
||||
async def _noop_ensure() -> None:
|
||||
return None
|
||||
|
||||
|
||||
class FakeRequest:
|
||||
"""Minimal request double exposing the optional-body contract."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
body: Any = None,
|
||||
can_read_body: bool = True,
|
||||
json_raises: bool = False,
|
||||
) -> None:
|
||||
self._body = body
|
||||
self.can_read_body = can_read_body
|
||||
self._json_raises = json_raises
|
||||
|
||||
async def json(self) -> Any:
|
||||
if self._json_raises or self._body is None:
|
||||
raise ValueError("no JSON body")
|
||||
return self._body
|
||||
|
||||
|
||||
class CapturingPersistence:
|
||||
def __init__(self) -> None:
|
||||
self.calls: list[dict[str, Any]] = []
|
||||
|
||||
async def save_recipe_from_widget(self, **kwargs: Any) -> SimpleNamespace:
|
||||
self.calls.append(kwargs)
|
||||
return SimpleNamespace(
|
||||
payload={"success": True, "has_workflow": bool(kwargs.get("workflow"))},
|
||||
status=200,
|
||||
)
|
||||
|
||||
|
||||
def _make_handler(persistence: CapturingPersistence) -> RecipeManagementHandler:
|
||||
analysis_service = SimpleNamespace(
|
||||
analyze_widget_metadata=lambda **kwargs: _analysis_result()
|
||||
)
|
||||
|
||||
return RecipeManagementHandler(
|
||||
ensure_dependencies_ready=_noop_ensure,
|
||||
recipe_scanner_getter=lambda: object(),
|
||||
logger=logging.getLogger(__name__),
|
||||
persistence_service=persistence, # pyright: ignore[reportArgumentType]
|
||||
analysis_service=analysis_service, # pyright: ignore[reportArgumentType]
|
||||
downloader_factory=lambda: None,
|
||||
civitai_client_getter=lambda: None,
|
||||
)
|
||||
|
||||
|
||||
async def _analysis_result() -> SimpleNamespace:
|
||||
return SimpleNamespace(
|
||||
payload={"metadata": {"loras": ""}, "image_bytes": b"image"}
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_widget_save_forwards_workflow_from_json_body():
|
||||
persistence = CapturingPersistence()
|
||||
handler = _make_handler(persistence)
|
||||
workflow = {"nodes": [{"id": 1}]}
|
||||
|
||||
response = await handler.save_recipe_from_widget(
|
||||
FakeRequest(body={"workflow": workflow}) # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
assert response.status == 200
|
||||
assert persistence.calls[0]["workflow"] == workflow
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_widget_save_without_body_passes_no_workflow():
|
||||
persistence = CapturingPersistence()
|
||||
handler = _make_handler(persistence)
|
||||
|
||||
response = await handler.save_recipe_from_widget(
|
||||
FakeRequest(can_read_body=False) # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
assert response.status == 200
|
||||
assert persistence.calls[0]["workflow"] is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_widget_save_tolerates_malformed_body():
|
||||
persistence = CapturingPersistence()
|
||||
handler = _make_handler(persistence)
|
||||
|
||||
await handler.save_recipe_from_widget(
|
||||
FakeRequest(json_raises=True) # type: ignore[arg-type]
|
||||
)
|
||||
await handler.save_recipe_from_widget(
|
||||
FakeRequest(body=["not", "an", "object"]) # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
assert [call["workflow"] for call in persistence.calls] == [None, None]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_widget_save_reports_embedded_workflow_in_response():
|
||||
persistence = CapturingPersistence()
|
||||
handler = _make_handler(persistence)
|
||||
|
||||
response = await handler.save_recipe_from_widget(
|
||||
FakeRequest(body={"workflow": {"nodes": []}}) # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
assert json.loads(response.text)["has_workflow"] is True
|
||||
@@ -589,49 +589,6 @@ class TestBatchImportServiceEdgeCases:
|
||||
assert "batch-import" in persistence_service.saved_recipes[0]["tags"]
|
||||
assert "test" in persistence_service.saved_recipes[0]["tags"]
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_workflow_from_analysis_is_passed_to_persistence(self, tmp_path):
|
||||
"""A workflow recovered from the source's original rendition travels in
|
||||
the analysis payload and must reach save_recipe as metadata."""
|
||||
workflow = '{"nodes": [{"id": 1}]}'
|
||||
ws_manager = MockWebSocketManager()
|
||||
analysis_service = MockAnalysisService(
|
||||
{
|
||||
"https://civitai.red/images/1": MockAnalysisResult(
|
||||
{
|
||||
"loras": [{"name": "test-lora"}],
|
||||
"workflow": workflow,
|
||||
}
|
||||
),
|
||||
}
|
||||
)
|
||||
persistence_service = MockPersistenceService()
|
||||
logger = logging.getLogger("test")
|
||||
|
||||
service = BatchImportService(
|
||||
analysis_service=analysis_service, # pyright: ignore[reportArgumentType]
|
||||
persistence_service=persistence_service, # pyright: ignore[reportArgumentType]
|
||||
ws_manager=ws_manager,
|
||||
logger=logger,
|
||||
)
|
||||
|
||||
recipe_scanner_getter = lambda: SimpleNamespace(
|
||||
find_recipes_by_fingerprint=lambda x: [],
|
||||
)
|
||||
civitai_client_getter = lambda: SimpleNamespace()
|
||||
|
||||
await service.start_batch_import(
|
||||
recipe_scanner_getter=recipe_scanner_getter,
|
||||
civitai_client_getter=civitai_client_getter,
|
||||
items=[{"source": "https://civitai.red/images/1"}],
|
||||
tags=[],
|
||||
)
|
||||
|
||||
await asyncio.sleep(0.3)
|
||||
|
||||
assert persistence_service.saved_recipes
|
||||
assert persistence_service.saved_recipes[0]["metadata"]["workflow"] == workflow
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_skip_duplicates_parameter(self, service):
|
||||
recipe_scanner_getter = lambda: SimpleNamespace()
|
||||
|
||||
@@ -19,13 +19,22 @@ from py.services.model_lifecycle_service import (
|
||||
from py.services.pending_delete_service import PendingDeleteService
|
||||
from py.services.settings_manager import get_settings_manager
|
||||
from py.utils.metadata_manager import MetadataManager
|
||||
from py.utils.sidecar_paths import root_mirror_component
|
||||
from py.utils.sidecar_paths import reset_root_map_cache, root_mirror_component
|
||||
|
||||
|
||||
def _normalize(path) -> str:
|
||||
return str(path).replace(os.sep, "/")
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate_root_map():
|
||||
"""Root identities are cached per sidecar root; keep tests independent."""
|
||||
|
||||
reset_root_map_cache()
|
||||
yield
|
||||
reset_root_map_cache()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def library_root(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
|
||||
"""Single checkpoint root under tmp_path; every other root emptied."""
|
||||
@@ -59,11 +68,10 @@ def centralized(library_root: Path, tmp_path: Path) -> Path:
|
||||
|
||||
|
||||
def _mirror_dir(library_root: Path, sidecar_root: Path, *rel: str) -> Path:
|
||||
"""Expected mirror directory for a library-relative path."""
|
||||
"""Expected mirror directory for a root-relative path."""
|
||||
|
||||
library = get_settings_manager().get_active_library_name()
|
||||
component = root_mirror_component(str(library_root))
|
||||
return sidecar_root.joinpath(library, component, *rel)
|
||||
component = root_mirror_component(str(library_root), sidecar_root=str(sidecar_root))
|
||||
return sidecar_root.joinpath(component, *rel)
|
||||
|
||||
|
||||
def _write_sidecar(
|
||||
|
||||
@@ -27,36 +27,14 @@ class DummyExifUtils:
|
||||
self.appended = None
|
||||
self.optimized_calls = 0
|
||||
self.workflow_value = None
|
||||
self.optimized_workflow = None
|
||||
self.embedded_workflows = []
|
||||
|
||||
def optimize_image(
|
||||
self,
|
||||
image_data,
|
||||
target_width,
|
||||
format,
|
||||
quality,
|
||||
preserve_metadata,
|
||||
workflow=None,
|
||||
):
|
||||
def optimize_image(self, image_data, target_width, format, quality, preserve_metadata):
|
||||
self.optimized_calls += 1
|
||||
self.optimized_workflow = workflow
|
||||
return image_data, ".webp"
|
||||
|
||||
def append_recipe_metadata(self, image_path, recipe_data, pixel_preserving=False):
|
||||
self.appended = (image_path, recipe_data, pixel_preserving)
|
||||
|
||||
def embed_workflow(self, image_path, workflow):
|
||||
self.embedded_workflows.append((image_path, workflow))
|
||||
return image_path
|
||||
|
||||
def normalise_workflow(self, workflow):
|
||||
if isinstance(workflow, str):
|
||||
return workflow or None
|
||||
if isinstance(workflow, (dict, list)):
|
||||
return json.dumps(workflow)
|
||||
return None
|
||||
|
||||
def extract_image_metadata(self, path):
|
||||
return {}
|
||||
|
||||
@@ -153,55 +131,6 @@ async def test_save_recipe_skip_optimize_preserves_image_bytes(tmp_path):
|
||||
assert exif_utils.appended[2] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_save_recipe_skip_optimize_still_embeds_recovered_workflow(tmp_path):
|
||||
"""The verbatim branch bypasses optimize_image, so the recovered workflow
|
||||
has to be embedded by the explicit safety net."""
|
||||
image_buffer = BytesIO()
|
||||
Image.new("RGB", (96, 48), color="olive").save(
|
||||
image_buffer, format="WEBP", quality=85
|
||||
)
|
||||
|
||||
class DummyScanner:
|
||||
def __init__(self, root):
|
||||
self.recipes_dir = str(root / "recipes")
|
||||
|
||||
async def add_recipe(self, recipe_data):
|
||||
return None
|
||||
|
||||
async def find_recipes_by_fingerprint(self, fingerprint):
|
||||
return []
|
||||
|
||||
service = RecipePersistenceService(
|
||||
exif_utils=ExifUtils,
|
||||
card_preview_width=512,
|
||||
logger=logging.getLogger("test"),
|
||||
)
|
||||
|
||||
workflow = {"nodes": [{"id": 8}]}
|
||||
result = await service.save_recipe(
|
||||
recipe_scanner=DummyScanner(tmp_path),
|
||||
image_bytes=image_buffer.getvalue(),
|
||||
image_base64=None,
|
||||
name="Verbatim Workflow",
|
||||
tags=[],
|
||||
metadata={"base_model": "sd", "loras": [], "workflow": workflow},
|
||||
extension=".webp",
|
||||
skip_optimize=True,
|
||||
)
|
||||
|
||||
image_path = Path(result.payload["image_path"])
|
||||
with Image.open(image_path) as img:
|
||||
assert img.size == (96, 48)
|
||||
assert img.format == "WEBP"
|
||||
assert ExifUtils._load_structured_metadata(str(image_path))["workflow"] == (
|
||||
json.dumps(workflow)
|
||||
)
|
||||
|
||||
stored = json.loads(Path(result.payload["json_path"]).read_text())
|
||||
assert stored["has_workflow"] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_save_recipe_skip_optimize_default_optimizes(tmp_path):
|
||||
"""Normal saves must keep optimizing; only re-import opts out."""
|
||||
@@ -721,88 +650,6 @@ async def test_save_recipe_preserves_workflow_when_png_is_converted_to_webp(tmp_
|
||||
assert "Recipe metadata:" in decoded_comment
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_save_recipe_embeds_workflow_recovered_from_source(tmp_path):
|
||||
"""Import paths hand the workflow over as metadata when their preview bytes
|
||||
are metadata-free (CivitAI's optimized rendition); save_recipe must embed
|
||||
it so the recipe reports has_workflow and can send it to ComfyUI."""
|
||||
class DummyScanner:
|
||||
def __init__(self, root):
|
||||
self.recipes_dir = str(root)
|
||||
|
||||
async def find_recipes_by_fingerprint(self, fingerprint):
|
||||
return []
|
||||
|
||||
async def add_recipe(self, recipe_data):
|
||||
return None
|
||||
|
||||
image_buffer = BytesIO()
|
||||
Image.new("RGB", (96, 48), color="teal").save(
|
||||
image_buffer, format="WEBP", quality=85
|
||||
)
|
||||
|
||||
service = RecipePersistenceService(
|
||||
exif_utils=ExifUtils,
|
||||
card_preview_width=64,
|
||||
logger=logging.getLogger("test"),
|
||||
)
|
||||
|
||||
workflow = {"nodes": [{"id": 1}], "last_node_id": 1}
|
||||
result = await service.save_recipe(
|
||||
recipe_scanner=DummyScanner(tmp_path),
|
||||
image_bytes=image_buffer.getvalue(),
|
||||
image_base64=None,
|
||||
name="Recovered Workflow",
|
||||
tags=["workflow"],
|
||||
metadata={"base_model": "sd", "loras": [], "workflow": workflow},
|
||||
extension=".webp",
|
||||
)
|
||||
|
||||
image_path = Path(result.payload["image_path"])
|
||||
assert ExifUtils._load_structured_metadata(str(image_path))["workflow"] == (
|
||||
json.dumps(workflow)
|
||||
)
|
||||
|
||||
stored = json.loads(Path(result.payload["json_path"]).read_text())
|
||||
assert stored["has_workflow"] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_save_recipe_passes_recovered_workflow_to_optimizer(tmp_path):
|
||||
"""The workflow travels through optimize_image (single encode pass) rather
|
||||
than being patched in afterwards."""
|
||||
exif_utils = DummyExifUtils()
|
||||
|
||||
class DummyScanner:
|
||||
def __init__(self, root):
|
||||
self.recipes_dir = str(root)
|
||||
|
||||
async def find_recipes_by_fingerprint(self, fingerprint):
|
||||
return []
|
||||
|
||||
async def add_recipe(self, recipe_data):
|
||||
return None
|
||||
|
||||
workflow = '{"nodes": [{"id": 2}]}'
|
||||
service = RecipePersistenceService(
|
||||
exif_utils=exif_utils,
|
||||
card_preview_width=512,
|
||||
logger=logging.getLogger("test"),
|
||||
)
|
||||
|
||||
await service.save_recipe(
|
||||
recipe_scanner=DummyScanner(tmp_path),
|
||||
image_bytes=b"image-bytes",
|
||||
image_base64=None,
|
||||
name="Recovered Workflow",
|
||||
tags=[],
|
||||
metadata={"base_model": "sd", "loras": [], "workflow": workflow},
|
||||
extension=".webp",
|
||||
)
|
||||
|
||||
assert exif_utils.optimized_workflow == workflow
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_save_recipe_strips_checkpoint_local_fields(tmp_path):
|
||||
exif_utils = DummyExifUtils()
|
||||
@@ -988,137 +835,6 @@ async def test_save_recipe_from_widget_enriches_checkpoint_from_local_cache(tmp_
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_save_recipe_from_widget_embeds_opted_in_workflow(tmp_path):
|
||||
"""Opt-in widget saves embed the live graph so the recipe can send its
|
||||
workflow back to ComfyUI, mirroring imported recipes."""
|
||||
class DummyScanner:
|
||||
def __init__(self, root):
|
||||
self.recipes_dir = str(root)
|
||||
self.added = []
|
||||
|
||||
async def get_local_lora(self, name): # pragma: no cover - no loras
|
||||
return None
|
||||
|
||||
async def add_recipe(self, recipe_data):
|
||||
self.added.append(recipe_data)
|
||||
|
||||
image_buffer = BytesIO()
|
||||
Image.new("RGB", (96, 48), color="navy").save(
|
||||
image_buffer, format="PNG"
|
||||
)
|
||||
|
||||
scanner = DummyScanner(tmp_path)
|
||||
service = RecipePersistenceService(
|
||||
exif_utils=ExifUtils,
|
||||
card_preview_width=64,
|
||||
logger=logging.getLogger("test"),
|
||||
)
|
||||
|
||||
workflow = {"nodes": [{"id": 1}], "last_node_id": 1}
|
||||
result = await service.save_recipe_from_widget(
|
||||
recipe_scanner=scanner,
|
||||
metadata={"loras": "", "prompt": "a calm scene"},
|
||||
image_bytes=image_buffer.getvalue(),
|
||||
workflow=workflow,
|
||||
)
|
||||
|
||||
assert result.payload["has_workflow"] is True
|
||||
assert "workflow_skipped" not in result.payload
|
||||
|
||||
stored = json.loads(Path(result.payload["json_path"]).read_text())
|
||||
assert stored["has_workflow"] is True
|
||||
assert ExifUtils._load_structured_metadata(result.payload["image_path"])[
|
||||
"workflow"
|
||||
] == json.dumps(workflow)
|
||||
assert scanner.added[0]["has_workflow"] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_save_recipe_from_widget_without_workflow_stays_unflagged(tmp_path):
|
||||
"""The default action must keep saving a workflow-free preview."""
|
||||
|
||||
class DummyScanner:
|
||||
def __init__(self, root):
|
||||
self.recipes_dir = str(root)
|
||||
|
||||
async def get_local_lora(self, name): # pragma: no cover - no loras
|
||||
return None
|
||||
|
||||
async def add_recipe(self, recipe_data):
|
||||
return None
|
||||
|
||||
image_buffer = BytesIO()
|
||||
Image.new("RGB", (96, 48), color="navy").save(
|
||||
image_buffer, format="PNG"
|
||||
)
|
||||
|
||||
service = RecipePersistenceService(
|
||||
exif_utils=ExifUtils,
|
||||
card_preview_width=64,
|
||||
logger=logging.getLogger("test"),
|
||||
)
|
||||
|
||||
result = await service.save_recipe_from_widget(
|
||||
recipe_scanner=DummyScanner(tmp_path),
|
||||
metadata={"loras": "", "prompt": "a calm scene"},
|
||||
image_bytes=image_buffer.getvalue(),
|
||||
)
|
||||
|
||||
assert result.payload["has_workflow"] is False
|
||||
assert ExifUtils._load_structured_metadata(result.payload["image_path"])[
|
||||
"workflow"
|
||||
] is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_save_recipe_from_widget_skips_oversized_workflow(
|
||||
tmp_path, monkeypatch
|
||||
):
|
||||
"""A pathological graph is dropped instead of inflating the preview."""
|
||||
monkeypatch.setattr(
|
||||
"py.services.recipes.persistence_service.MAX_WORKFLOW_EMBED_BYTES", 32
|
||||
)
|
||||
|
||||
class DummyScanner:
|
||||
def __init__(self, root):
|
||||
self.recipes_dir = str(root)
|
||||
self.added = []
|
||||
|
||||
async def get_local_lora(self, name): # pragma: no cover - no loras
|
||||
return None
|
||||
|
||||
async def add_recipe(self, recipe_data):
|
||||
self.added.append(recipe_data)
|
||||
|
||||
image_buffer = BytesIO()
|
||||
Image.new("RGB", (96, 48), color="navy").save(
|
||||
image_buffer, format="PNG"
|
||||
)
|
||||
|
||||
scanner = DummyScanner(tmp_path)
|
||||
service = RecipePersistenceService(
|
||||
exif_utils=ExifUtils,
|
||||
card_preview_width=64,
|
||||
logger=logging.getLogger("test"),
|
||||
)
|
||||
|
||||
workflow = {"nodes": [{"id": index} for index in range(20)]}
|
||||
result = await service.save_recipe_from_widget(
|
||||
recipe_scanner=scanner,
|
||||
metadata={"loras": "", "prompt": "a calm scene"},
|
||||
image_bytes=image_buffer.getvalue(),
|
||||
workflow=workflow,
|
||||
)
|
||||
|
||||
assert result.payload["workflow_skipped"] == "too_large"
|
||||
assert result.payload["has_workflow"] is False
|
||||
assert ExifUtils._load_structured_metadata(result.payload["image_path"])[
|
||||
"workflow"
|
||||
] is None
|
||||
assert scanner.added[0]["has_workflow"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_recipe_updates_paths(tmp_path):
|
||||
exif_utils = DummyExifUtils()
|
||||
@@ -1355,83 +1071,6 @@ async def test_analyze_remote_image_supports_civitai_red():
|
||||
assert result.payload["loras"] == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_analyze_remote_image_returns_workflow_from_original_rendition():
|
||||
"""CivitAI's optimized rendition is re-encoded and metadata-free, so an
|
||||
embedded workflow only exists in the original. Analysis must surface it so
|
||||
the save step can embed it while the stored preview stays the optimized
|
||||
image."""
|
||||
workflow = json.dumps({"nodes": [{"id": 1}], "last_node_id": 1})
|
||||
|
||||
class FakeExif:
|
||||
def extract_image_metadata(self, path):
|
||||
# The optimized rendition carries no metadata at all.
|
||||
return None
|
||||
|
||||
def _load_structured_metadata(self, path):
|
||||
# Only the original rendition (fetched to a .png temp file)
|
||||
# carries the embedded workflow.
|
||||
return {
|
||||
"parameters": None,
|
||||
"prompt": None,
|
||||
"workflow": workflow if str(path).endswith(".png") else None,
|
||||
"comment": None,
|
||||
}
|
||||
|
||||
downloaded: list[str] = []
|
||||
|
||||
async def downloader_factory():
|
||||
class Downloader:
|
||||
async def download_file(self, url, path, use_auth=False):
|
||||
downloaded.append(url)
|
||||
Path(path).write_bytes(b"fake-image")
|
||||
return True, "success"
|
||||
|
||||
return Downloader()
|
||||
|
||||
class DummyFactory:
|
||||
def create_parser(self, metadata):
|
||||
async def parse_metadata(m, recipe_scanner=None, civitai_client=None):
|
||||
return {"loras": [], "gen_params": {"prompt": "p"}}
|
||||
|
||||
return SimpleNamespace(parse_metadata=parse_metadata)
|
||||
|
||||
service = RecipeAnalysisService(
|
||||
exif_utils=FakeExif(),
|
||||
recipe_parser_factory=DummyFactory(),
|
||||
downloader_factory=downloader_factory,
|
||||
metadata_collector=None,
|
||||
metadata_processor_cls=None,
|
||||
metadata_registry_cls=None,
|
||||
standalone_mode=False,
|
||||
logger=logging.getLogger("test"),
|
||||
)
|
||||
|
||||
class DummyClient:
|
||||
async def get_image_info(self, image_id, source_url=None):
|
||||
return {
|
||||
"url": "https://image.civitai.com/x/original=true/sample.jpeg",
|
||||
"type": "image",
|
||||
"meta": {"prompt": "p"},
|
||||
}
|
||||
|
||||
class DummyScanner:
|
||||
async def find_recipes_by_fingerprint(self, fingerprint):
|
||||
return []
|
||||
|
||||
result = await service.analyze_remote_image(
|
||||
url="https://civitai.red/images/143518055",
|
||||
recipe_scanner=DummyScanner(),
|
||||
civitai_client=DummyClient(),
|
||||
)
|
||||
|
||||
assert result.payload["workflow"] == workflow
|
||||
# The optimized rendition is used as the preview, the original only as the
|
||||
# metadata/workflow fallback.
|
||||
assert any("width=450,optimized=true" in url for url in downloaded)
|
||||
assert any("original=true" in url for url in downloaded)
|
||||
|
||||
|
||||
def _exif_utils_returning(metadata):
|
||||
class MetadataExifUtils(DummyExifUtils):
|
||||
def extract_image_metadata(self, path):
|
||||
|
||||
@@ -4,6 +4,7 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
@@ -12,13 +13,26 @@ import pytest
|
||||
from py.config import config
|
||||
from py.services.settings_manager import get_settings_manager
|
||||
from py.services.use_cases.sidecar_migration_use_case import SidecarMigrationUseCase
|
||||
from py.utils.sidecar_paths import root_mirror_component
|
||||
from py.utils.sidecar_paths import (
|
||||
get_metadata_path,
|
||||
reset_root_map_cache,
|
||||
root_mirror_component,
|
||||
)
|
||||
|
||||
|
||||
def _normalize(path) -> str:
|
||||
return str(path).replace(os.sep, "/")
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate_root_map():
|
||||
"""Root identities are cached per sidecar root; keep tests independent."""
|
||||
|
||||
reset_root_map_cache()
|
||||
yield
|
||||
reset_root_map_cache()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def library_root(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
|
||||
"""Single lora root under tmp_path; every other root emptied."""
|
||||
@@ -54,11 +68,10 @@ def _set_mode(mode: str) -> None:
|
||||
|
||||
|
||||
def _mirror_dir(library_root: Path, sidecar_root: Path, *rel: str) -> Path:
|
||||
"""Expected mirror directory for a library-relative path."""
|
||||
"""Expected mirror directory for a root-relative path."""
|
||||
|
||||
library = get_settings_manager().get_active_library_name()
|
||||
component = root_mirror_component(str(library_root))
|
||||
return sidecar_root.joinpath(library, component, *rel)
|
||||
component = root_mirror_component(str(library_root), sidecar_root=str(sidecar_root))
|
||||
return sidecar_root.joinpath(component, *rel)
|
||||
|
||||
|
||||
def _write_model(directory: Path, stem: str) -> Path:
|
||||
@@ -482,6 +495,61 @@ async def test_migrate_root_relocates_tree_and_reconciles(
|
||||
assert not old_root.exists()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_migrate_root_preserves_a_reanchored_identity(
|
||||
library_root: Path, sidecar_root: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
):
|
||||
"""Relocating the sidecar root must not displace a pinned mirror name.
|
||||
|
||||
A root that moved earlier is mirrored under a name derived from its *old*
|
||||
path. If anything resolves against the destination sidecar root before the
|
||||
relocation runs, it writes a map naming the mirror after the *current*
|
||||
path; the relocation must keep the pinned entry, because the directories
|
||||
actually being moved are named after it.
|
||||
"""
|
||||
|
||||
_set_mode("centralized")
|
||||
first_sidecars = tmp_path / "first_sidecars"
|
||||
get_settings_manager().set("sidecar_storage_path", str(first_sidecars))
|
||||
reset_root_map_cache()
|
||||
|
||||
model = _write_model(library_root / "sub", "model")
|
||||
pinned = root_mirror_component(str(library_root))
|
||||
original = Path(get_metadata_path(str(model)))
|
||||
original.parent.mkdir(parents=True, exist_ok=True)
|
||||
original.write_text(json.dumps({"favorite": True}), encoding="utf-8")
|
||||
|
||||
# The model root moves; the recorded identity is re-anchored to it.
|
||||
moved_root = tmp_path / "relocated" / "loras"
|
||||
shutil.copytree(library_root, moved_root)
|
||||
monkeypatch.setattr(config, "loras_roots", [str(moved_root)], raising=False)
|
||||
reset_root_map_cache()
|
||||
moved_model = moved_root / "sub" / "model.safetensors"
|
||||
assert Path(get_metadata_path(str(moved_model))) == original
|
||||
assert root_mirror_component(str(moved_root)) == pinned
|
||||
|
||||
# Settings now point at the destination and something resolves first: the
|
||||
# destination map is written naming the mirror after the *current* path.
|
||||
get_settings_manager().set("sidecar_storage_path", str(sidecar_root))
|
||||
reset_root_map_cache()
|
||||
before_move = Path(get_metadata_path(str(moved_model)))
|
||||
assert before_move.parent.parent.name != pinned
|
||||
assert not before_move.exists()
|
||||
|
||||
use_case = _make_use_case([str(moved_model)])
|
||||
summary = await use_case.migrate_root(str(first_sidecars), force=True)
|
||||
assert summary["success"] is True
|
||||
|
||||
# The metadata travelled with the tree and is still found under the
|
||||
# pinned identity that names it.
|
||||
relocated = sidecar_root / original.relative_to(first_sidecars)
|
||||
assert relocated.exists()
|
||||
assert json.loads(relocated.read_text(encoding="utf-8")) == {"favorite": True}
|
||||
|
||||
reset_root_map_cache()
|
||||
assert Path(get_metadata_path(str(moved_model))) == relocated
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_migrate_root_guards(
|
||||
library_root: Path, sidecar_root: Path, tmp_path: Path
|
||||
|
||||
@@ -211,104 +211,6 @@ def test_update_image_metadata_preserves_png_workflow(tmp_path):
|
||||
)
|
||||
|
||||
|
||||
def test_optimize_image_embeds_supplied_workflow_when_source_has_none(tmp_path):
|
||||
"""Import paths hand the workflow over as data when the preview source is
|
||||
metadata-free (CivitAI's optimized rendition); optimize_image must embed
|
||||
it while re-encoding, otherwise the recipe loses has_workflow."""
|
||||
image_path = tmp_path / "optimized.webp"
|
||||
Image.new("RGB", (64, 32), color="red").save(image_path, format="WEBP", quality=85)
|
||||
|
||||
workflow = {"nodes": [{"id": 1}], "last_node_id": 1}
|
||||
optimized_data, extension = ExifUtils.optimize_image(
|
||||
str(image_path),
|
||||
target_width=32,
|
||||
format="webp",
|
||||
quality=85,
|
||||
preserve_metadata=True,
|
||||
workflow=workflow,
|
||||
)
|
||||
|
||||
optimized_path = tmp_path / f"embedded{extension}"
|
||||
optimized_path.write_bytes(optimized_data)
|
||||
|
||||
metadata = ExifUtils._load_structured_metadata(str(optimized_path))
|
||||
assert metadata["workflow"] == json.dumps(workflow)
|
||||
|
||||
|
||||
def test_optimize_image_keeps_source_workflow_over_supplied(tmp_path):
|
||||
image_path = tmp_path / "source.png"
|
||||
png_info = PngImagePlugin.PngInfo()
|
||||
png_info.add_text("workflow", '{"nodes":[{"id":7}]}')
|
||||
Image.new("RGB", (64, 32), color="red").save(image_path, pnginfo=png_info)
|
||||
|
||||
optimized_data, extension = ExifUtils.optimize_image(
|
||||
str(image_path),
|
||||
target_width=32,
|
||||
format="webp",
|
||||
quality=85,
|
||||
preserve_metadata=True,
|
||||
workflow={"nodes": [{"id": 1}]},
|
||||
)
|
||||
|
||||
optimized_path = tmp_path / f"sourcewins{extension}"
|
||||
optimized_path.write_bytes(optimized_data)
|
||||
|
||||
metadata = ExifUtils._load_structured_metadata(str(optimized_path))
|
||||
assert metadata["workflow"] == '{"nodes":[{"id":7}]}'
|
||||
|
||||
|
||||
def test_embed_workflow_adds_workflow_to_metadata_free_webp(tmp_path):
|
||||
image_path = tmp_path / "preview.webp"
|
||||
Image.new("RGB", (32, 32), color="blue").save(image_path, format="WEBP", quality=85)
|
||||
|
||||
workflow = json.dumps({"nodes": [{"id": 1}]})
|
||||
returned = ExifUtils.embed_workflow(str(image_path), workflow)
|
||||
|
||||
assert returned == str(image_path)
|
||||
metadata = ExifUtils._load_structured_metadata(str(image_path))
|
||||
assert metadata["workflow"] == workflow
|
||||
with Image.open(image_path) as img:
|
||||
assert img.size == (32, 32)
|
||||
|
||||
|
||||
def test_embed_workflow_adds_workflow_to_metadata_free_png(tmp_path):
|
||||
image_path = tmp_path / "preview.png"
|
||||
Image.new("RGB", (32, 32), color="blue").save(image_path)
|
||||
|
||||
workflow = {"nodes": [{"id": 3}]}
|
||||
ExifUtils.embed_workflow(str(image_path), workflow)
|
||||
|
||||
metadata = ExifUtils._load_structured_metadata(str(image_path))
|
||||
assert metadata["workflow"] == json.dumps(workflow)
|
||||
|
||||
|
||||
def test_embed_workflow_leaves_existing_workflow_untouched(tmp_path):
|
||||
image_path = tmp_path / "preview.png"
|
||||
png_info = PngImagePlugin.PngInfo()
|
||||
png_info.add_text("workflow", '{"nodes":[{"id":9}]}')
|
||||
Image.new("RGB", (32, 32), color="green").save(image_path, pnginfo=png_info)
|
||||
|
||||
ExifUtils.embed_workflow(str(image_path), {"nodes": [{"id": 1}]})
|
||||
|
||||
with Image.open(image_path) as img:
|
||||
assert img.info["workflow"] == '{"nodes":[{"id":9}]}'
|
||||
|
||||
|
||||
def test_embed_workflow_ignores_unsupported_payloads_and_containers(tmp_path):
|
||||
image_path = tmp_path / "preview.webp"
|
||||
Image.new("RGB", (16, 16), color="black").save(image_path, format="WEBP")
|
||||
|
||||
# Nothing to embed / unsupported payload types are no-ops.
|
||||
assert ExifUtils.embed_workflow(str(image_path), None) == str(image_path)
|
||||
assert ExifUtils.embed_workflow(str(image_path), "") == str(image_path)
|
||||
assert ExifUtils.embed_workflow(str(image_path), 123) == str(image_path)
|
||||
assert ExifUtils._load_structured_metadata(str(image_path))["workflow"] is None
|
||||
|
||||
video_path = tmp_path / "clip.mp4"
|
||||
video_path.write_bytes(b"video")
|
||||
assert ExifUtils.embed_workflow(str(video_path), {"nodes": []}) == str(video_path)
|
||||
|
||||
|
||||
# --- ISOBMFF / brotli extraction tests ---
|
||||
|
||||
import struct
|
||||
|
||||
@@ -2,7 +2,10 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
@@ -11,14 +14,17 @@ from py.services.settings_manager import get_settings_manager
|
||||
from py.utils import sidecar_paths
|
||||
from py.utils.sidecar_paths import (
|
||||
METADATA_SUFFIX,
|
||||
ROOT_MAP_FILENAME,
|
||||
get_configured_sidecar_root,
|
||||
get_metadata_path,
|
||||
get_preview_dir,
|
||||
get_sidecar_dir,
|
||||
get_sidecar_root,
|
||||
get_storage_mode,
|
||||
get_unmatched_sidecar_components,
|
||||
is_centralized,
|
||||
is_metadata_path,
|
||||
relocate_root_map,
|
||||
resolve_centralized_dir,
|
||||
resolve_centralized_dir_for_dir,
|
||||
resolve_metadata_path,
|
||||
@@ -27,10 +33,36 @@ from py.utils.sidecar_paths import (
|
||||
)
|
||||
|
||||
|
||||
def _write_map(root: Path, entries: dict) -> Path:
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
path = root / ROOT_MAP_FILENAME
|
||||
path.write_text(
|
||||
json.dumps({"version": 1, "roots": entries}), encoding="utf-8"
|
||||
)
|
||||
return path
|
||||
|
||||
|
||||
def _normalize(path: Path) -> str:
|
||||
return str(path).replace(os.sep, "/")
|
||||
|
||||
|
||||
def _legacy_component(root) -> str:
|
||||
"""Independent reimplementation of the pre-identity-map component name."""
|
||||
|
||||
normalized = os.path.normpath(os.path.abspath(str(root)))
|
||||
digest = hashlib.sha256(normalized.encode("utf-8")).hexdigest()[:8]
|
||||
return f"{sanitize_path_component(os.path.basename(normalized))}-{digest}"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_root_map_cache():
|
||||
"""Identities are cached per sidecar root; keep tests order-independent."""
|
||||
|
||||
sidecar_paths.reset_root_map_cache()
|
||||
yield
|
||||
sidecar_paths.reset_root_map_cache()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def model_roots(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> dict:
|
||||
"""Point every config model root the sidecar module reads at tmp_path."""
|
||||
@@ -68,6 +100,17 @@ def centralized(model_roots: dict, tmp_path: Path) -> Path:
|
||||
return sidecar_root
|
||||
|
||||
|
||||
def _write_sidecar(model_path: Path, payload: dict | None = None) -> Path:
|
||||
"""Create the model file and its sidecar at the module-resolved path."""
|
||||
|
||||
model_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
model_path.write_bytes(b"weights")
|
||||
sidecar = Path(get_metadata_path(str(model_path)))
|
||||
sidecar.parent.mkdir(parents=True, exist_ok=True)
|
||||
sidecar.write_text(json.dumps(payload or {"favorite": True}), encoding="utf-8")
|
||||
return sidecar
|
||||
|
||||
|
||||
class TestAlongsideMode:
|
||||
def test_default_mode_is_alongside(self):
|
||||
assert get_storage_mode() == "alongside"
|
||||
@@ -86,6 +129,26 @@ class TestAlongsideMode:
|
||||
assert get_sidecar_dir(str(model)) == expected
|
||||
assert get_preview_dir(str(model)) == expected
|
||||
|
||||
def test_alongside_resolution_touches_nothing(
|
||||
self, model_roots: dict, tmp_path: Path
|
||||
):
|
||||
"""The default branch must not load, create, or scan the mirror tree."""
|
||||
|
||||
sidecar_root = tmp_path / "sidecars"
|
||||
settings = get_settings_manager()
|
||||
settings.set("sidecar_storage_mode", "alongside")
|
||||
settings.set("sidecar_storage_path", str(sidecar_root))
|
||||
|
||||
model = model_roots["loras"] / "sub" / "model.safetensors"
|
||||
assert get_metadata_path(str(model)) == os.path.join(
|
||||
str(model_roots["loras"]), "sub", "model" + METADATA_SUFFIX
|
||||
)
|
||||
assert get_preview_dir(str(model)) == os.path.join(
|
||||
str(model_roots["loras"]), "sub"
|
||||
)
|
||||
assert not sidecar_root.exists()
|
||||
assert sidecar_paths._ROOT_MAPS == {}
|
||||
|
||||
|
||||
class TestPathPredicates:
|
||||
def test_is_metadata_path(self):
|
||||
@@ -117,13 +180,12 @@ class TestSanitizePathComponent:
|
||||
class TestCentralizedMode:
|
||||
def test_mirror_layout(self, model_roots: dict, centralized: Path):
|
||||
model = model_roots["loras"] / "styles" / "anime" / "model.safetensors"
|
||||
library = get_settings_manager().get_active_library_name()
|
||||
root_component = root_mirror_component(str(model_roots["loras"]))
|
||||
|
||||
metadata_path = get_metadata_path(str(model))
|
||||
|
||||
expected = os.path.join(
|
||||
str(centralized), library, root_component, "styles", "anime", "model" + METADATA_SUFFIX
|
||||
str(centralized), root_component, "styles", "anime", "model" + METADATA_SUFFIX
|
||||
)
|
||||
assert metadata_path == expected
|
||||
assert get_preview_dir(str(model)) == os.path.dirname(expected)
|
||||
@@ -155,7 +217,7 @@ class TestCentralizedMode:
|
||||
assert root_mirror_component(str(model_roots["loras"])) != root_mirror_component(
|
||||
str(other_root)
|
||||
)
|
||||
# Same root always maps to the same component (stable hash).
|
||||
# Same root always maps to the same component (pinned identity).
|
||||
assert root_mirror_component(str(model_roots["loras"])) == root_mirror_component(
|
||||
str(model_roots["loras"]) + os.sep
|
||||
)
|
||||
@@ -171,11 +233,10 @@ class TestCentralizedMode:
|
||||
[str(model_roots["loras"]), str(nested)],
|
||||
raising=False,
|
||||
)
|
||||
library = get_settings_manager().get_active_library_name()
|
||||
|
||||
model = nested / "model.safetensors"
|
||||
assert get_metadata_path(str(model)) == os.path.join(
|
||||
str(centralized), library, root_mirror_component(str(nested)), "model" + METADATA_SUFFIX
|
||||
str(centralized), root_mirror_component(str(nested)), "model" + METADATA_SUFFIX
|
||||
)
|
||||
|
||||
def test_outside_roots_falls_back_to_alongside(
|
||||
@@ -204,10 +265,8 @@ class TestCentralizedMode:
|
||||
def test_resolve_centralized_dir_for_dir_root_maps_to_mirror_base(
|
||||
self, model_roots: dict, centralized: Path
|
||||
):
|
||||
library = get_settings_manager().get_active_library_name()
|
||||
|
||||
assert resolve_centralized_dir_for_dir(str(model_roots["loras"])) == os.path.join(
|
||||
str(centralized), library, root_mirror_component(str(model_roots["loras"]))
|
||||
str(centralized), root_mirror_component(str(model_roots["loras"]))
|
||||
)
|
||||
|
||||
def test_empty_path_uses_default_sidecar_root(self, model_roots: dict, tmp_path: Path):
|
||||
@@ -221,6 +280,293 @@ class TestCentralizedMode:
|
||||
assert is_centralized()
|
||||
|
||||
|
||||
class TestRootIdentityMap:
|
||||
"""The root component is a remembered identity, not a path hash."""
|
||||
|
||||
def test_component_is_pinned_in_the_map_file(
|
||||
self, model_roots: dict, centralized: Path
|
||||
):
|
||||
model = model_roots["loras"] / "sub" / "model.safetensors"
|
||||
first = get_metadata_path(str(model))
|
||||
|
||||
map_path = centralized / ROOT_MAP_FILENAME
|
||||
assert map_path.exists()
|
||||
payload = json.loads(map_path.read_text(encoding="utf-8"))
|
||||
persisted = {entry["component"] for entry in payload["roots"].values()}
|
||||
assert root_mirror_component(str(model_roots["loras"])) in persisted
|
||||
|
||||
# Restarting the process must resolve the exact same location.
|
||||
sidecar_paths.reset_root_map_cache()
|
||||
assert get_metadata_path(str(model)) == first
|
||||
|
||||
def test_component_is_deterministic_and_survives_a_lost_map(
|
||||
self, model_roots: dict, centralized: Path
|
||||
):
|
||||
"""Deleting the map must not strand a mirror whose root has not moved."""
|
||||
|
||||
model = model_roots["loras"] / "sub" / "model.safetensors"
|
||||
first = _write_sidecar(model)
|
||||
assert Path(get_metadata_path(str(model))) == first
|
||||
|
||||
(centralized / ROOT_MAP_FILENAME).unlink()
|
||||
sidecar_paths.reset_root_map_cache()
|
||||
|
||||
assert Path(get_metadata_path(str(model))) == first
|
||||
assert root_mirror_component(str(model_roots["loras"])) == _legacy_component(
|
||||
model_roots["loras"]
|
||||
)
|
||||
|
||||
def test_legacy_path_derived_component_is_adopted(
|
||||
self, model_roots: dict, centralized: Path
|
||||
):
|
||||
"""Upgrading from the hash-named layout keeps existing mirrors usable."""
|
||||
|
||||
legacy = _legacy_component(model_roots["loras"])
|
||||
|
||||
sidecar = centralized / legacy / "sub" / ("model" + METADATA_SUFFIX)
|
||||
sidecar.parent.mkdir(parents=True)
|
||||
sidecar.write_text(json.dumps({"favorite": True}), encoding="utf-8")
|
||||
|
||||
model = model_roots["loras"] / "sub" / "model.safetensors"
|
||||
assert get_metadata_path(str(model)) == str(sidecar)
|
||||
assert root_mirror_component(str(model_roots["loras"])) == legacy
|
||||
|
||||
def test_pre_identity_map_library_nested_component_is_adopted(
|
||||
self, model_roots: dict, centralized: Path
|
||||
):
|
||||
"""Mirrors created by the pre-identity-map build nested under a library."""
|
||||
|
||||
legacy = _legacy_component(model_roots["loras"])
|
||||
sidecar = (
|
||||
centralized / "comfyui" / legacy / "sub" / ("model" + METADATA_SUFFIX)
|
||||
)
|
||||
sidecar.parent.mkdir(parents=True)
|
||||
sidecar.write_text(json.dumps({"favorite": True}), encoding="utf-8")
|
||||
|
||||
model = model_roots["loras"] / "sub" / "model.safetensors"
|
||||
assert get_metadata_path(str(model)) == str(sidecar)
|
||||
assert root_mirror_component(str(model_roots["loras"])) == f"comfyui/{legacy}"
|
||||
# The adopted tree counts as linked, not orphaned.
|
||||
assert get_unmatched_sidecar_components() == []
|
||||
|
||||
def test_moved_root_reuses_its_mirror(
|
||||
self, model_roots: dict, centralized: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
):
|
||||
"""A relocated (and reconfigured) root keeps its remembered identity."""
|
||||
|
||||
from py.config import config
|
||||
|
||||
model = model_roots["loras"] / "sub" / "model.safetensors"
|
||||
before = _write_sidecar(model)
|
||||
|
||||
moved_root = tmp_path / "relocated" / "loras"
|
||||
shutil.copytree(model_roots["loras"], moved_root)
|
||||
monkeypatch.setattr(config, "loras_roots", [str(moved_root)], raising=False)
|
||||
sidecar_paths.reset_root_map_cache()
|
||||
|
||||
moved_model = moved_root / "sub" / "model.safetensors"
|
||||
after = Path(get_metadata_path(str(moved_model)))
|
||||
|
||||
assert after == before
|
||||
assert json.loads(after.read_text(encoding="utf-8")) == {"favorite": True}
|
||||
|
||||
def test_renamed_root_reanchors_via_directory_overlap(
|
||||
self, model_roots: dict, centralized: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
):
|
||||
"""A rename that changes the basename still re-anchors when the tree matches."""
|
||||
|
||||
from py.config import config
|
||||
|
||||
model = model_roots["loras"] / "sub" / "model.safetensors"
|
||||
before = _write_sidecar(model)
|
||||
|
||||
renamed_root = tmp_path / "brand-new-name"
|
||||
shutil.copytree(model_roots["loras"], renamed_root)
|
||||
monkeypatch.setattr(config, "loras_roots", [str(renamed_root)], raising=False)
|
||||
sidecar_paths.reset_root_map_cache()
|
||||
|
||||
renamed_model = renamed_root / "sub" / "model.safetensors"
|
||||
assert Path(get_metadata_path(str(renamed_model))) == before
|
||||
|
||||
def test_ambiguous_reanchor_reports_orphans_instead_of_guessing(
|
||||
self, model_roots: dict, centralized: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
):
|
||||
from py.config import config
|
||||
|
||||
model = model_roots["loras"] / "sub" / "model.safetensors"
|
||||
before = _write_sidecar(model)
|
||||
component = before.parent.parent.name
|
||||
|
||||
# Two equally plausible candidates: never silently pick one.
|
||||
candidate_a = tmp_path / "a" / "loras"
|
||||
candidate_b = tmp_path / "b" / "loras"
|
||||
shutil.copytree(model_roots["loras"], candidate_a)
|
||||
shutil.copytree(model_roots["loras"], candidate_b)
|
||||
monkeypatch.setattr(
|
||||
config, "loras_roots", [str(candidate_a), str(candidate_b)], raising=False
|
||||
)
|
||||
sidecar_paths.reset_root_map_cache()
|
||||
|
||||
for candidate in (candidate_a, candidate_b):
|
||||
resolved = Path(
|
||||
get_metadata_path(str(candidate / "sub" / "model.safetensors"))
|
||||
)
|
||||
assert resolved != before
|
||||
assert component not in str(resolved)
|
||||
|
||||
orphans = get_unmatched_sidecar_components()
|
||||
assert [item["component"] for item in orphans] == [component]
|
||||
|
||||
def test_removed_root_surfaces_its_component_as_orphan(
|
||||
self, model_roots: dict, centralized: Path, monkeypatch: pytest.MonkeyPatch
|
||||
):
|
||||
from py.config import config
|
||||
|
||||
model = model_roots["loras"] / "sub" / "model.safetensors"
|
||||
sidecar = _write_sidecar(model)
|
||||
component = sidecar.parent.parent.name
|
||||
|
||||
monkeypatch.setattr(config, "loras_roots", [], raising=False)
|
||||
sidecar_paths.reset_root_map_cache()
|
||||
|
||||
assert [item["component"] for item in get_unmatched_sidecar_components()] == [
|
||||
component
|
||||
]
|
||||
|
||||
def test_unmatched_helper_is_empty_in_alongside_mode(
|
||||
self, model_roots: dict, tmp_path: Path
|
||||
):
|
||||
settings = get_settings_manager()
|
||||
settings.set("sidecar_storage_mode", "alongside")
|
||||
settings.set("sidecar_storage_path", str(tmp_path / "sidecars"))
|
||||
|
||||
assert get_unmatched_sidecar_components() == []
|
||||
assert not (tmp_path / "sidecars").exists()
|
||||
|
||||
def test_relocated_sidecar_root_resolves_after_map_written_first(
|
||||
self, model_roots: dict, tmp_path: Path
|
||||
):
|
||||
"""A resolve before the mirror tree is relocated must not rename it."""
|
||||
|
||||
settings = get_settings_manager()
|
||||
old_root = tmp_path / "old-sidecars"
|
||||
new_root = tmp_path / "new-sidecars"
|
||||
settings.set("sidecar_storage_mode", "centralized")
|
||||
settings.set("sidecar_storage_path", str(old_root))
|
||||
|
||||
model = model_roots["loras"] / "sub" / "model.safetensors"
|
||||
sidecar = _write_sidecar(model)
|
||||
|
||||
# The path setting changes first, and something resolves against the
|
||||
# new root before the user runs the relocation.
|
||||
settings.set("sidecar_storage_path", str(new_root))
|
||||
sidecar_paths.reset_root_map_cache()
|
||||
expected = new_root / sidecar.relative_to(old_root)
|
||||
assert Path(get_metadata_path(str(model))) == expected
|
||||
assert not expected.exists()
|
||||
|
||||
# Now the mirror tree moves, exactly as migrate_root does.
|
||||
for dirpath, _dirnames, filenames in os.walk(old_root):
|
||||
rel = os.path.relpath(dirpath, old_root)
|
||||
target_dir = new_root if rel == os.curdir else new_root / rel
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
for filename in filenames:
|
||||
shutil.move(str(Path(dirpath) / filename), str(target_dir / filename))
|
||||
|
||||
sidecar_paths.reset_root_map_cache()
|
||||
assert Path(get_metadata_path(str(model))) == expected
|
||||
assert json.loads(expected.read_text(encoding="utf-8")) == {"favorite": True}
|
||||
|
||||
def test_relocate_root_map_prefers_the_source_entries(self, tmp_path: Path):
|
||||
"""A map written at the destination before a relocation must not win."""
|
||||
|
||||
source = tmp_path / "old-sidecars"
|
||||
destination = tmp_path / "new-sidecars"
|
||||
_write_map(
|
||||
source,
|
||||
{
|
||||
"aaaa1111": {
|
||||
"component": "loras-pinned",
|
||||
"basename": "loras",
|
||||
"last_path": "/models/loras",
|
||||
"sample_rel_dirs": [],
|
||||
}
|
||||
},
|
||||
)
|
||||
_write_map(
|
||||
destination,
|
||||
{
|
||||
"bbbb2222": {
|
||||
"component": "loras-recomputed",
|
||||
"basename": "loras",
|
||||
"last_path": "/models/loras",
|
||||
"sample_rel_dirs": [],
|
||||
},
|
||||
"cccc3333": {
|
||||
"component": "vae-other",
|
||||
"basename": "vae",
|
||||
"last_path": "/models/vae",
|
||||
"sample_rel_dirs": [],
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
assert relocate_root_map(str(source), str(destination)) is True
|
||||
|
||||
merged = json.loads(
|
||||
(destination / ROOT_MAP_FILENAME).read_text(encoding="utf-8")
|
||||
)["roots"]
|
||||
assert merged["aaaa1111"]["component"] == "loras-pinned"
|
||||
assert "bbbb2222" not in merged # superseded for the same root path
|
||||
assert merged["cccc3333"]["component"] == "vae-other"
|
||||
assert not (source / ROOT_MAP_FILENAME).exists()
|
||||
|
||||
def test_relocate_root_map_without_a_source_map_is_a_noop(self, tmp_path: Path):
|
||||
source = tmp_path / "old-sidecars"
|
||||
destination = tmp_path / "new-sidecars"
|
||||
_write_map(
|
||||
destination,
|
||||
{
|
||||
"bbbb2222": {
|
||||
"component": "loras-keep",
|
||||
"basename": "loras",
|
||||
"last_path": "/models/loras",
|
||||
"sample_rel_dirs": [],
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
assert relocate_root_map(str(source), str(destination)) is True
|
||||
|
||||
kept = json.loads(
|
||||
(destination / ROOT_MAP_FILENAME).read_text(encoding="utf-8")
|
||||
)["roots"]
|
||||
assert kept["bbbb2222"]["component"] == "loras-keep"
|
||||
|
||||
def test_corrupt_map_file_is_ignored_and_rebuilt(
|
||||
self, model_roots: dict, centralized: Path
|
||||
):
|
||||
model = model_roots["loras"] / "model.safetensors"
|
||||
map_path = centralized / ROOT_MAP_FILENAME
|
||||
|
||||
get_metadata_path(str(model))
|
||||
map_path.write_text("{not json", encoding="utf-8")
|
||||
sidecar_paths.reset_root_map_cache()
|
||||
|
||||
path_after = get_metadata_path(str(model))
|
||||
assert json.loads(map_path.read_text(encoding="utf-8"))["roots"]
|
||||
assert os.path.basename(os.path.dirname(path_after)).startswith("loras-")
|
||||
|
||||
def test_root_map_file_is_not_mistaken_for_a_component(
|
||||
self, model_roots: dict, centralized: Path
|
||||
):
|
||||
model = model_roots["loras"] / "model.safetensors"
|
||||
get_metadata_path(str(model))
|
||||
|
||||
assert (centralized / ROOT_MAP_FILENAME).is_file()
|
||||
assert get_unmatched_sidecar_components() == []
|
||||
|
||||
|
||||
class TestModeIndependentResolution:
|
||||
"""Migration tooling resolves the mirror layout regardless of active mode."""
|
||||
|
||||
@@ -248,17 +594,18 @@ class TestModeIndependentResolution:
|
||||
sidecar_root = tmp_path / "sidecars"
|
||||
settings = get_settings_manager()
|
||||
settings.set("sidecar_storage_mode", "alongside")
|
||||
library = settings.get_active_library_name()
|
||||
settings.set("sidecar_storage_path", str(sidecar_root))
|
||||
|
||||
model_dir = model_roots["loras"] / "sub"
|
||||
component = root_mirror_component(
|
||||
str(model_roots["loras"]), sidecar_root=str(sidecar_root)
|
||||
)
|
||||
|
||||
# Alongside mode: no root resolves without the override.
|
||||
assert resolve_centralized_dir_for_dir(str(model_dir)) is None
|
||||
assert resolve_centralized_dir_for_dir(
|
||||
str(model_dir), sidecar_root=str(sidecar_root)
|
||||
) == os.path.join(
|
||||
str(sidecar_root), library, root_mirror_component(str(model_roots["loras"])), "sub"
|
||||
)
|
||||
) == os.path.join(str(sidecar_root), component, "sub")
|
||||
|
||||
|
||||
class TestSettingsValidation:
|
||||
|
||||
@@ -873,19 +873,6 @@ export function createContextMenu(x, y, loraName, widget, previewTooltip, render
|
||||
}
|
||||
);
|
||||
|
||||
// Save recipe with the current graph embedded. Kept opt-in rather than
|
||||
// folded into "Save Recipe": the workflow dwarfs every other metadata field
|
||||
// and can carry sensitive widget values, so it stays an explicit choice.
|
||||
const saveWithWorkflowOption = createMenuItem(
|
||||
'Save Recipe with Workflow',
|
||||
'<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"><circle cx="18" cy="5" r="3"></circle><circle cx="6" cy="12" r="3"></circle><circle cx="18" cy="19" r="3"></circle><line x1="8.59" y1="13.51" x2="15.42" y2="17.49"></line><line x1="15.41" y1="6.51" x2="8.59" y2="10.49"></line></svg>',
|
||||
() => {
|
||||
menu.remove();
|
||||
document.removeEventListener('click', closeMenu);
|
||||
saveRecipeDirectly({ embedWorkflow: true });
|
||||
}
|
||||
);
|
||||
|
||||
// Move Up option with arrow up icon
|
||||
const moveUpOption = createMenuItem(
|
||||
'Move Up',
|
||||
@@ -995,7 +982,6 @@ export function createContextMenu(x, y, loraName, widget, previewTooltip, render
|
||||
menu.appendChild(copyTriggerWordsOption);
|
||||
menu.appendChild(separator2);
|
||||
menu.appendChild(saveOption);
|
||||
menu.appendChild(saveWithWorkflowOption);
|
||||
|
||||
document.body.appendChild(menu);
|
||||
|
||||
|
||||
@@ -460,29 +460,15 @@ export function syncClipStrengthIfCollapsed(loraData) {
|
||||
}
|
||||
|
||||
// Function to directly save the recipe without dialog
|
||||
export async function saveRecipeDirectly({ embedWorkflow = false } = {}) {
|
||||
export async function saveRecipeDirectly() {
|
||||
try {
|
||||
const prompt = await app.graphToPrompt();
|
||||
console.log('Prompt:', prompt); // for debugging purposes
|
||||
|
||||
// Embedding the graph is opt-in: it is by far the largest metadata field
|
||||
// and its widget values can contain sensitive data (paths, API keys). The
|
||||
// UI-format graph is sent rather than the API prompt so node layout and
|
||||
// groups survive — that is what "Send Workflow to ComfyUI" restores.
|
||||
const requestBody = {};
|
||||
if (embedWorkflow) {
|
||||
if (prompt && prompt.workflow) {
|
||||
requestBody.workflow = prompt.workflow;
|
||||
} else {
|
||||
showToast('No workflow available to embed; saving the recipe without it', 'warning');
|
||||
}
|
||||
}
|
||||
|
||||
// Show loading toast
|
||||
if (app && app.extensionManager && app.extensionManager.toast) {
|
||||
app.extensionManager.toast.add({
|
||||
severity: 'info',
|
||||
summary: embedWorkflow ? 'Saving Recipe with Workflow' : 'Saving Recipe',
|
||||
summary: 'Saving Recipe',
|
||||
detail: 'Please wait...',
|
||||
life: 2000
|
||||
});
|
||||
@@ -490,9 +476,7 @@ export async function saveRecipeDirectly({ embedWorkflow = false } = {}) {
|
||||
|
||||
// Send the request to the backend API
|
||||
const response = await fetch(lmUrl('/api/lm/recipes/save-from-widget'), {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(requestBody)
|
||||
method: 'POST'
|
||||
});
|
||||
|
||||
const result = await response.json();
|
||||
@@ -500,23 +484,12 @@ export async function saveRecipeDirectly({ embedWorkflow = false } = {}) {
|
||||
// Show result toast
|
||||
if (app && app.extensionManager && app.extensionManager.toast) {
|
||||
if (result.success) {
|
||||
let severity = 'success';
|
||||
let summary = embedWorkflow ? 'Recipe Saved with Workflow' : 'Recipe Saved';
|
||||
let detail = embedWorkflow
|
||||
? 'Recipe and the current workflow have been saved'
|
||||
: 'Recipe has been saved successfully';
|
||||
|
||||
if (embedWorkflow && result.workflow_skipped === 'too_large') {
|
||||
severity = 'warn';
|
||||
summary = 'Recipe Saved without Workflow';
|
||||
detail = 'The workflow is too large to embed; the recipe was saved without it';
|
||||
} else if (embedWorkflow && result.has_workflow !== true) {
|
||||
severity = 'warn';
|
||||
summary = 'Recipe Saved without Workflow';
|
||||
detail = 'The workflow could not be embedded in the recipe image';
|
||||
}
|
||||
|
||||
app.extensionManager.toast.add({ severity, summary, detail, life: 5000 });
|
||||
app.extensionManager.toast.add({
|
||||
severity: 'success',
|
||||
summary: 'Recipe Saved',
|
||||
detail: 'Recipe has been saved successfully',
|
||||
life: 3000
|
||||
});
|
||||
} else {
|
||||
app.extensionManager.toast.add({
|
||||
severity: 'error',
|
||||
|
||||
Reference in New Issue
Block a user