Compare commits

..

11 Commits

Author SHA1 Message Date
Will Miao
4199c30fec fix(metadata-sync): downgrade "Model not found" to INFO and replace model_name with file+sha256 in log 2026-06-17 00:06:43 +08:00
Will Miao
4a8084cdbc feat(save-image): support %NodeTitle.WidgetName% placeholders and fix %seed% None fallback (#314) 2026-06-16 23:48:44 +08:00
Will Miao
6263e6848c fix: move posix_fadvise(DONTNEED) after read loop so it actually evicts pages (#985) 2026-06-16 23:12:02 +08:00
Will Miao
58c266ad07 fix(scanner): respect lazy hash for checkpoints, add posix_fadvise, cancel on shutdown (#985) 2026-06-16 23:00:23 +08:00
Will Miao
2939813e1a feat(metadata-fetch): add result summary modal with i18n, fix contrast and counting bugs (#38) 2026-06-16 22:38:50 +08:00
Will Miao
a9e5ee7e79 fix: follow-up nits for AVIF/JXL brotli support
- Fix JXL container ftyp size check (==20 → >=16) to accept
  wider range of valid JXL files
- Add brotli decompression size limit (2 MB) to prevent OOM
- Add trailing newline to requirements.txt
- Add unit tests for new ISOBMFF/brotli extraction paths:
  JXL/AVIF happy paths, missing brob, corrupt payload,
  non-ISOBMFF fallthrough, write-skip on AVIF/JXL,
  JSON dict/list fields, and oversized decompression
2026-06-16 16:27:56 +08:00
Will Miao
a17b0e9901 Merge pull request #982 from koloved/main
Add AVIF and JXL image support with brotli metadata decompression
2026-06-16 16:24:30 +08:00
s.ivanov
8f23d966bf Update requirements.txt 2026-06-16 07:27:32 +02:00
Will Miao
7a76fc72d0 fix(rate-limit): continue to next provider on CivArchive 429 to prevent bulk refresh from freezing (#983)
When CivArchive returns HTTP 429 with a large retry_after, the bulk
metadata refresh would block for hours because:

1. FallbackMetadataProvider raised RateLimitError instead of continuing
   to the next provider (e.g., SQLite archive was never reached).

2. _RateLimitRetryHelper retried long-rate-limit 429s 3 times — all
   futile since the hourly cap hasn't reset.

3. The batch loop had no awareness of persistent rate-limiting,
   causing 192+ models to each hammer the same rate-limited endpoint.

Changes:
- FallbackMetadataProvider: all 6 methods now continue to next provider
  on RateLimitError instead of raising (model_metadata_provider.py)
- fetch_and_update_model: deleted-model path also continues on
  RateLimitError so sqlite provider gets a chance (metadata_sync_service.py)
- _RateLimitRetryHelper: when retry_after >= 120s, only 1 attempt is
  made — retries are futile for hour-scale rate limits
- BulkMetadataRefreshUseCase: tracks consecutive rate-limit failures
  and aborts early after 3 (bulk_metadata_refresh_use_case.py)

Tests: updated test_fallback_respects_retry_limit for new continue
behavior; added tests for large/small retry_after thresholds.
2026-06-16 13:08:34 +08:00
Will Miao
518a4dd5ee chore: add reasonix.toml and .codegraph/ to .gitignore 2026-06-16 13:05:11 +08:00
s.ivanov
2b6d4e5d8b Add AVIF and JXL image support with brotli metadata decompression 2026-06-15 09:28:49 +02:00
32 changed files with 1274 additions and 85 deletions

4
.gitignore vendored
View File

@@ -12,12 +12,14 @@ coverage/
.coverage .coverage
model_cache/ model_cache/
# agent # agent / dev tooling
.opencode/ .opencode/
.claude/ .claude/
.sisyphus/ .sisyphus/
.codex .codex
.omo .omo
reasonix.toml
.codegraph/
# Vue widgets development cache (but keep build output) # Vue widgets development cache (but keep build output)
vue-widgets/node_modules/ vue-widgets/node_modules/

View File

@@ -185,12 +185,25 @@ The Save Image Node supports dynamic filename generation using pattern codes. Yo
#### Available Pattern Codes #### Available Pattern Codes
##### Cross-Node Placeholders (ComfyUI Standard)
- `%NodeTitle.WidgetName%` - Reference any widget value from any node in your workflow, for example:
- `%KSampler.seed%` - The seed from a KSampler node
- `%Empty Latent Image.width%` - The width from an Empty Latent Image node
- `%KSampler.steps%` - The steps value from a KSampler node
- Nodes are matched by their "Node name for S&R" property, then by their title
##### Generation Metadata Placeholders (LoRA Manager)
- `%seed%` - Inserts the generation seed number - `%seed%` - Inserts the generation seed number
- `%width%` - Inserts the image width - `%width%` - Inserts the image width
- `%height%` - Inserts the image height - `%height%` - Inserts the image height
- `%pprompt:N%` - Inserts the positive prompt (limited to N characters) - `%pprompt:N%` - Inserts the positive prompt (limited to N characters)
- `%nprompt:N%` - Inserts the negative prompt (limited to N characters) - `%nprompt:N%` - Inserts the negative prompt (limited to N characters)
- `%model:N%` - Inserts the model/checkpoint name (limited to N characters) - `%model:N%` - Inserts the model/checkpoint name (limited to N characters)
##### Date/Time Placeholders
- `%date%` - Inserts current date/time as "yyyyMMddhhmmss" - `%date%` - Inserts current date/time as "yyyyMMddhhmmss"
- `%date:FORMAT%` - Inserts date using custom format with: - `%date:FORMAT%` - Inserts date using custom format with:
- `yyyy` - 4-digit year - `yyyy` - 4-digit year
@@ -209,8 +222,25 @@ The Save Image Node supports dynamic filename generation using pattern codes. Yo
- `%date:yyyy-MM-dd%``2025-04-28` - `%date:yyyy-MM-dd%``2025-04-28`
- `%pprompt:20%_%seed%``beautiful landscape_1234567890` - `%pprompt:20%_%seed%``beautiful landscape_1234567890`
- `%model%_%date:yyMMdd%_%seed%``dreamshaper_v8_250428_1234567890` - `%model%_%date:yyMMdd%_%seed%``dreamshaper_v8_250428_1234567890`
- `%KSampler.seed%``1234567890` (resolved from the KSampler node's widget)
- `%Empty Latent Image.width%x%Empty Latent Image.height%``512x768`
- `%KSampler.seed%_%KSampler.steps%``1234567890_25`
You can combine multiple patterns to create detailed, organized filenames for your generated images. You can combine multiple patterns to create detailed, organized filenames for your generated images. Cross-node and metadata placeholders can be mixed freely — for example: `%KSampler.seed%_%model%_%date:yyyyMMdd%`.
##### Organizing Images into Subdirectories
Including a path separator (`/` on all platforms) in the filename prefix creates subdirectories automatically, which is especially powerful when combined with placeholders:
| Pattern | Result |
|---|---|
| `%date:yyyy-MM-dd%/%seed%` | Saves to `2025-04-28/1234567890.png` |
| `%model%/%date:yyMMdd%_%seed%` | Saves to `dreamshaper_v8/250428_1234567890.png` |
| `%KSampler.seed%/%model%` | Saves to `1234567890/dreamshaper_v8.png` |
| `%date:yyyy/MM/dd%/%seed%` | Saves to `2025/04/28/1234567890.png` (nested year/month/day) |
| `%model%/training/%seed%` | Saves to `dreamshaper_v8/training/1234567890.png` |
> **Note**: The subdirectory is created relative to your ComfyUI output directory (configurable via `--output-directory`). Characters invalid for folder names are automatically replaced with underscores.
### Standalone Mode ### Standalone Mode

View File

@@ -1398,6 +1398,21 @@
"versionDeleted": "Version gelöscht" "versionDeleted": "Version gelöscht"
} }
} }
},
"metadataFetchSummary": {
"title": "Metadaten abrufen — Zusammenfassung",
"statSuccess": "Erfolgreich",
"statFailed": "Fehlgeschlagen",
"statSkipped": "Übersprungen",
"statTotal": "Gesamt geprüft",
"statDuration": "Dauer",
"successMessage": "Alle {count} {type}s erfolgreich aktualisiert!",
"failedItems": "Fehlgeschlagene Elemente ({count})",
"close": "Schließen",
"copyReport": "Bericht kopieren",
"downloadCsv": "CSV herunterladen",
"columnModelName": "Modellname",
"columnError": "Fehler"
} }
}, },
"modelTags": { "modelTags": {
@@ -1957,7 +1972,9 @@
"bulkMoveSuccess": "{successCount} {type}s erfolgreich verschoben", "bulkMoveSuccess": "{successCount} {type}s erfolgreich verschoben",
"exampleImagesDownloadSuccess": "Beispielbilder erfolgreich heruntergeladen!", "exampleImagesDownloadSuccess": "Beispielbilder erfolgreich heruntergeladen!",
"exampleImagesDownloadFailed": "Fehler beim Herunterladen der Beispielbilder: {message}", "exampleImagesDownloadFailed": "Fehler beim Herunterladen der Beispielbilder: {message}",
"moveFailed": "Failed to move item: {message}" "moveFailed": "Failed to move item: {message}",
"copiedToClipboard": "In die Zwischenablage kopiert",
"downloadStarted": "Download gestartet"
} }
}, },
"doctor": { "doctor": {

View File

@@ -1398,6 +1398,21 @@
"versionDeleted": "Version deleted" "versionDeleted": "Version deleted"
} }
} }
},
"metadataFetchSummary": {
"title": "Metadata Fetch Summary",
"statSuccess": "Success",
"statFailed": "Failed",
"statSkipped": "Skipped",
"statTotal": "Total Scanned",
"statDuration": "Duration",
"successMessage": "All {count} {type}s updated successfully!",
"failedItems": "Failed Items ({count})",
"close": "Close",
"copyReport": "Copy Report",
"downloadCsv": "Download CSV",
"columnModelName": "Model Name",
"columnError": "Error"
} }
}, },
"modelTags": { "modelTags": {
@@ -1957,7 +1972,9 @@
"bulkMoveSuccess": "Successfully moved {successCount} {type}s", "bulkMoveSuccess": "Successfully moved {successCount} {type}s",
"exampleImagesDownloadSuccess": "Successfully downloaded example images!", "exampleImagesDownloadSuccess": "Successfully downloaded example images!",
"exampleImagesDownloadFailed": "Failed to download example images: {message}", "exampleImagesDownloadFailed": "Failed to download example images: {message}",
"moveFailed": "Failed to move item: {message}" "moveFailed": "Failed to move item: {message}",
"copiedToClipboard": "Copied to clipboard",
"downloadStarted": "Download started"
} }
}, },
"doctor": { "doctor": {
@@ -2052,4 +2069,4 @@
"retry": "Retry" "retry": "Retry"
} }
} }
} }

View File

@@ -1398,6 +1398,21 @@
"versionDeleted": "Versión eliminada" "versionDeleted": "Versión eliminada"
} }
} }
},
"metadataFetchSummary": {
"title": "Resumen de obtención de metadatos",
"statSuccess": "Éxito",
"statFailed": "Fallido",
"statSkipped": "Omitido",
"statTotal": "Total escaneado",
"statDuration": "Duración",
"successMessage": "¡Todos los {count} {type}s actualizados correctamente!",
"failedItems": "Elementos fallidos ({count})",
"close": "Cerrar",
"copyReport": "Copiar informe",
"downloadCsv": "Descargar CSV",
"columnModelName": "Nombre del modelo",
"columnError": "Error"
} }
}, },
"modelTags": { "modelTags": {
@@ -1957,7 +1972,9 @@
"bulkMoveSuccess": "Movidos exitosamente {successCount} {type}s", "bulkMoveSuccess": "Movidos exitosamente {successCount} {type}s",
"exampleImagesDownloadSuccess": "¡Imágenes de ejemplo descargadas exitosamente!", "exampleImagesDownloadSuccess": "¡Imágenes de ejemplo descargadas exitosamente!",
"exampleImagesDownloadFailed": "Error al descargar imágenes de ejemplo: {message}", "exampleImagesDownloadFailed": "Error al descargar imágenes de ejemplo: {message}",
"moveFailed": "Failed to move item: {message}" "moveFailed": "Failed to move item: {message}",
"copiedToClipboard": "Copiado al portapapeles",
"downloadStarted": "Descarga iniciada"
} }
}, },
"doctor": { "doctor": {

View File

@@ -1398,6 +1398,21 @@
"versionDeleted": "Version supprimée" "versionDeleted": "Version supprimée"
} }
} }
},
"metadataFetchSummary": {
"title": "Récapitulatif de la récupération des métadonnées",
"statSuccess": "Réussi",
"statFailed": "Échoué",
"statSkipped": "Ignoré",
"statTotal": "Total scanné",
"statDuration": "Durée",
"successMessage": "Tous les {count} {type}s mis à jour avec succès !",
"failedItems": "Éléments échoués ({count})",
"close": "Fermer",
"copyReport": "Copier le rapport",
"downloadCsv": "Télécharger CSV",
"columnModelName": "Nom du modèle",
"columnError": "Erreur"
} }
}, },
"modelTags": { "modelTags": {
@@ -1957,7 +1972,9 @@
"bulkMoveSuccess": "{successCount} {type}s déplacés avec succès", "bulkMoveSuccess": "{successCount} {type}s déplacés avec succès",
"exampleImagesDownloadSuccess": "Images d'exemple téléchargées avec succès !", "exampleImagesDownloadSuccess": "Images d'exemple téléchargées avec succès !",
"exampleImagesDownloadFailed": "Échec du téléchargement des images d'exemple : {message}", "exampleImagesDownloadFailed": "Échec du téléchargement des images d'exemple : {message}",
"moveFailed": "Failed to move item: {message}" "moveFailed": "Failed to move item: {message}",
"copiedToClipboard": "Copié dans le presse-papiers",
"downloadStarted": "Téléchargement démarré"
} }
}, },
"doctor": { "doctor": {

View File

@@ -1398,6 +1398,21 @@
"versionDeleted": "הגרסה נמחקה" "versionDeleted": "הגרסה נמחקה"
} }
} }
},
"metadataFetchSummary": {
"title": "סיכום שליפת מטא-דאטה",
"statSuccess": "הצלחה",
"statFailed": "נכשל",
"statSkipped": "דולג",
"statTotal": "סה\"כ נסרק",
"statDuration": "משך",
"successMessage": "כל {count} {type}s עודכנו בהצלחה!",
"failedItems": "פריטים נכשלים ({count})",
"close": "סגור",
"copyReport": "העתק דוח",
"downloadCsv": "הורד CSV",
"columnModelName": "שם המודל",
"columnError": "שגיאה"
} }
}, },
"modelTags": { "modelTags": {
@@ -1957,7 +1972,9 @@
"bulkMoveSuccess": "הועברו בהצלחה {successCount} {type}s", "bulkMoveSuccess": "הועברו בהצלחה {successCount} {type}s",
"exampleImagesDownloadSuccess": "תמונות הדוגמה הורדו בהצלחה!", "exampleImagesDownloadSuccess": "תמונות הדוגמה הורדו בהצלחה!",
"exampleImagesDownloadFailed": "הורדת תמונות הדוגמה נכשלה: {message}", "exampleImagesDownloadFailed": "הורדת תמונות הדוגמה נכשלה: {message}",
"moveFailed": "Failed to move item: {message}" "moveFailed": "Failed to move item: {message}",
"copiedToClipboard": "הועתק ללוח",
"downloadStarted": "ההורדה החלה"
} }
}, },
"doctor": { "doctor": {

View File

@@ -1398,6 +1398,21 @@
"versionDeleted": "バージョンを削除しました" "versionDeleted": "バージョンを削除しました"
} }
} }
},
"metadataFetchSummary": {
"title": "メタデータ取得サマリー",
"statSuccess": "成功",
"statFailed": "失敗",
"statSkipped": "スキップ",
"statTotal": "スキャン合計",
"statDuration": "所要時間",
"successMessage": "すべての{count}件の{type}を正常に更新しました",
"failedItems": "失敗したアイテム ({count})",
"close": "閉じる",
"copyReport": "レポートをコピー",
"downloadCsv": "CSVをダウンロード",
"columnModelName": "モデル名",
"columnError": "エラー"
} }
}, },
"modelTags": { "modelTags": {
@@ -1957,7 +1972,9 @@
"bulkMoveSuccess": "{successCount} {type}が正常に移動されました", "bulkMoveSuccess": "{successCount} {type}が正常に移動されました",
"exampleImagesDownloadSuccess": "例画像が正常にダウンロードされました!", "exampleImagesDownloadSuccess": "例画像が正常にダウンロードされました!",
"exampleImagesDownloadFailed": "例画像のダウンロードに失敗しました:{message}", "exampleImagesDownloadFailed": "例画像のダウンロードに失敗しました:{message}",
"moveFailed": "Failed to move item: {message}" "moveFailed": "Failed to move item: {message}",
"copiedToClipboard": "クリップボードにコピーしました",
"downloadStarted": "ダウンロードを開始しました"
} }
}, },
"doctor": { "doctor": {

View File

@@ -1398,6 +1398,21 @@
"versionDeleted": "버전이 삭제되었습니다" "versionDeleted": "버전이 삭제되었습니다"
} }
} }
},
"metadataFetchSummary": {
"title": "메타데이터 가져오기 요약",
"statSuccess": "성공",
"statFailed": "실패",
"statSkipped": "건너뜀",
"statTotal": "총 스캔",
"statDuration": "소요 시간",
"successMessage": "모든 {count}개 {type}이(가) 성공적으로 업데이트되었습니다",
"failedItems": "실패한 항목 ({count})",
"close": "닫기",
"copyReport": "보고서 복사",
"downloadCsv": "CSV 다운로드",
"columnModelName": "모델 이름",
"columnError": "오류"
} }
}, },
"modelTags": { "modelTags": {
@@ -1957,7 +1972,9 @@
"bulkMoveSuccess": "{successCount}개 {type}이(가) 성공적으로 이동되었습니다", "bulkMoveSuccess": "{successCount}개 {type}이(가) 성공적으로 이동되었습니다",
"exampleImagesDownloadSuccess": "예시 이미지가 성공적으로 다운로드되었습니다!", "exampleImagesDownloadSuccess": "예시 이미지가 성공적으로 다운로드되었습니다!",
"exampleImagesDownloadFailed": "예시 이미지 다운로드 실패: {message}", "exampleImagesDownloadFailed": "예시 이미지 다운로드 실패: {message}",
"moveFailed": "Failed to move item: {message}" "moveFailed": "Failed to move item: {message}",
"copiedToClipboard": "클립보드에 복사됨",
"downloadStarted": "다운로드 시작됨"
} }
}, },
"doctor": { "doctor": {

View File

@@ -1398,6 +1398,21 @@
"versionDeleted": "Версия удалена" "versionDeleted": "Версия удалена"
} }
} }
},
"metadataFetchSummary": {
"title": "Сводка получения метаданных",
"statSuccess": "Успешно",
"statFailed": "Ошибка",
"statSkipped": "Пропущено",
"statTotal": "Всего проверено",
"statDuration": "Длительность",
"successMessage": "Все {count} {type}s успешно обновлены",
"failedItems": "Ошибочные элементы ({count})",
"close": "Закрыть",
"copyReport": "Копировать отчет",
"downloadCsv": "Скачать CSV",
"columnModelName": "Имя модели",
"columnError": "Ошибка"
} }
}, },
"modelTags": { "modelTags": {
@@ -1957,7 +1972,9 @@
"bulkMoveSuccess": "Успешно перемещено {successCount} {type}s", "bulkMoveSuccess": "Успешно перемещено {successCount} {type}s",
"exampleImagesDownloadSuccess": "Примеры изображений успешно загружены!", "exampleImagesDownloadSuccess": "Примеры изображений успешно загружены!",
"exampleImagesDownloadFailed": "Не удалось загрузить примеры изображений: {message}", "exampleImagesDownloadFailed": "Не удалось загрузить примеры изображений: {message}",
"moveFailed": "Failed to move item: {message}" "moveFailed": "Failed to move item: {message}",
"copiedToClipboard": "Скопировано в буфер обмена",
"downloadStarted": "Загрузка начата"
} }
}, },
"doctor": { "doctor": {

View File

@@ -1398,6 +1398,21 @@
"versionDeleted": "版本已删除" "versionDeleted": "版本已删除"
} }
} }
},
"metadataFetchSummary": {
"title": "元数据获取摘要",
"statSuccess": "成功",
"statFailed": "失败",
"statSkipped": "已跳过",
"statTotal": "总计扫描",
"statDuration": "耗时",
"successMessage": "全部 {count} 个 {type} 更新成功!",
"failedItems": "失败项目 ({count})",
"close": "关闭",
"copyReport": "复制报告",
"downloadCsv": "下载 CSV",
"columnModelName": "模型名称",
"columnError": "错误"
} }
}, },
"modelTags": { "modelTags": {
@@ -1957,7 +1972,9 @@
"bulkMoveSuccess": "成功移动 {successCount} 个 {type}", "bulkMoveSuccess": "成功移动 {successCount} 个 {type}",
"exampleImagesDownloadSuccess": "示例图片下载成功!", "exampleImagesDownloadSuccess": "示例图片下载成功!",
"exampleImagesDownloadFailed": "示例图片下载失败:{message}", "exampleImagesDownloadFailed": "示例图片下载失败:{message}",
"moveFailed": "Failed to move item: {message}" "moveFailed": "Failed to move item: {message}",
"copiedToClipboard": "已复制到剪贴板",
"downloadStarted": "下载已开始"
} }
}, },
"doctor": { "doctor": {

View File

@@ -1398,6 +1398,21 @@
"versionDeleted": "已刪除此版本" "versionDeleted": "已刪除此版本"
} }
} }
},
"metadataFetchSummary": {
"title": "元資料獲取摘要",
"statSuccess": "成功",
"statFailed": "失敗",
"statSkipped": "已跳過",
"statTotal": "總計掃描",
"statDuration": "耗時",
"successMessage": "全部 {count} 個 {type} 更新成功!",
"failedItems": "失敗項目 ({count})",
"close": "關閉",
"copyReport": "複製報告",
"downloadCsv": "下載 CSV",
"columnModelName": "模型名稱",
"columnError": "錯誤"
} }
}, },
"modelTags": { "modelTags": {
@@ -1957,7 +1972,9 @@
"bulkMoveSuccess": "已成功移動 {successCount} 個 {type}", "bulkMoveSuccess": "已成功移動 {successCount} 個 {type}",
"exampleImagesDownloadSuccess": "範例圖片下載成功!", "exampleImagesDownloadSuccess": "範例圖片下載成功!",
"exampleImagesDownloadFailed": "下載範例圖片失敗:{message}", "exampleImagesDownloadFailed": "下載範例圖片失敗:{message}",
"moveFailed": "Failed to move item: {message}" "moveFailed": "Failed to move item: {message}",
"copiedToClipboard": "已複製到剪貼簿",
"downloadStarted": "下載已開始"
} }
}, },
"doctor": { "doctor": {

View File

@@ -436,5 +436,14 @@ class LoraManager:
try: try:
logger.info("LoRA Manager: Cleaning up services") logger.info("LoRA Manager: Cleaning up services")
# Cancel any in-flight scanner initialization tasks so thread-pool
# workers (e.g. _initialize_cache_sync) can break out of their loops
# when the server shuts down (e.g. Ctrl+C on WSL).
for name in ("lora_scanner", "checkpoint_scanner", "embedding_scanner"):
scanner = ServiceRegistry.get_service_sync(name)
if scanner is not None and hasattr(scanner, "cancel_task"):
scanner.cancel_task()
logger.debug("LoRA Manager: Cancelled %s", name)
except Exception as e: except Exception as e:
logger.error(f"Error during cleanup: {e}", exc_info=True) logger.error(f"Error during cleanup: {e}", exc_info=True)

View File

@@ -16,6 +16,8 @@ IMG_EXTENSIONS = (
".tif", ".tif",
".tiff", ".tiff",
".webp", ".webp",
".avif",
".jxl",
".mp4" ".mp4"
) )

View File

@@ -298,7 +298,12 @@ class SaveImageLM:
key = parts[0] key = parts[0]
if key == "seed" and "seed" in metadata_dict: if key == "seed" and "seed" in metadata_dict:
filename = filename.replace(segment, str(metadata_dict.get("seed", ""))) seed_value = metadata_dict.get("seed")
if seed_value is not None:
filename = filename.replace(segment, str(seed_value))
else:
# Fallback if seed was not captured by metadata collector
filename = filename.replace(segment, "0")
elif key == "width" and "size" in metadata_dict: elif key == "width" and "size" in metadata_dict:
size = metadata_dict.get("size", "x") size = metadata_dict.get("size", "x")
w = size.split("x")[0] if isinstance(size, str) else size[0] w = size.split("x")[0] if isinstance(size, str) else size[0]

View File

@@ -216,13 +216,19 @@ class MetadataSyncService:
provider_used: Optional[str] = None provider_used: Optional[str] = None
last_error: Optional[str] = None last_error: Optional[str] = None
civitai_api_not_found = False civitai_api_not_found = False
any_rate_limited = False
for provider_name, provider in provider_attempts: for provider_name, provider in provider_attempts:
try: try:
civitai_metadata_candidate, error = await provider.get_model_by_hash(sha256) civitai_metadata_candidate, error = await provider.get_model_by_hash(sha256)
except RateLimitError as exc: except RateLimitError as exc:
exc.provider = exc.provider or (provider_name or provider.__class__.__name__) logger.warning(
raise "Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
provider_name or provider.__class__.__name__,
exc.retry_after or 0,
)
any_rate_limited = True
continue
except Exception as exc: # pragma: no cover - defensive logging except Exception as exc: # pragma: no cover - defensive logging
logger.error("Provider %s failed for hash %s: %s", provider_name, sha256, exc) logger.error("Provider %s failed for hash %s: %s", provider_name, sha256, exc)
civitai_metadata_candidate, error = None, str(exc) civitai_metadata_candidate, error = None, str(exc)
@@ -276,14 +282,17 @@ class MetadataSyncService:
) )
resolved_error = last_error or default_error resolved_error = last_error or default_error
if any_rate_limited and "Rate limited" not in resolved_error:
resolved_error = "Rate limited"
if is_expected_offline_error(resolved_error): if is_expected_offline_error(resolved_error):
resolved_error = OFFLINE_FRIENDLY_MESSAGE resolved_error = OFFLINE_FRIENDLY_MESSAGE
error_msg = ( error_msg = (
f"Error fetching metadata: {resolved_error} " f"Error fetching metadata: {resolved_error} "
f"(model_name={model_data.get('model_name', '')})" f"(file={os.path.basename(file_path)}, sha256={sha256})"
) )
if is_expected_offline_error(resolved_error): is_model_not_found = "Model not found" in resolved_error
if is_expected_offline_error(resolved_error) or is_model_not_found:
logger.info(error_msg) logger.info(error_msg)
else: else:
logger.error(error_msg) logger.error(error_msg)

View File

@@ -65,7 +65,14 @@ class _RateLimitRetryHelper:
return await func(*args, **kwargs) return await func(*args, **kwargs)
except RateLimitError as exc: except RateLimitError as exc:
attempt += 1 attempt += 1
if attempt >= self._retry_limit:
# Determine effective retry limit based on rate-limit magnitude
effective_retry_limit = self._retry_limit # default: 3
if exc.retry_after is not None and exc.retry_after >= 120.0:
# Long rate-limit window (>=2 min) — retries are futile
effective_retry_limit = 1 # total 1 attempt = 0 retries
if attempt >= effective_retry_limit:
exc.provider = exc.provider or label exc.provider = exc.provider or label
raise raise
@@ -478,8 +485,12 @@ class FallbackMetadataProvider(ModelMetadataProvider):
if result: if result:
return result, error return result, error
except RateLimitError as exc: except RateLimitError as exc:
exc.provider = exc.provider or label logger.warning(
raise exc "Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
label,
exc.retry_after or 0,
)
continue
except Exception as e: except Exception as e:
logger.debug("Provider %s failed for get_model_by_hash: %s", label, e) logger.debug("Provider %s failed for get_model_by_hash: %s", label, e)
continue continue
@@ -497,16 +508,12 @@ class FallbackMetadataProvider(ModelMetadataProvider):
if result: if result:
return result return result
except RateLimitError as exc: except RateLimitError as exc:
if not_found_confirmed: logger.warning(
logger.debug( "Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
"Suppressing rate limit from %s for model %s: " label,
"already confirmed as not found by another provider", exc.retry_after or 0,
label, )
model_id, continue
)
return None
exc.provider = exc.provider or label
raise exc
except ResourceNotFoundError: except ResourceNotFoundError:
not_found_confirmed = True not_found_confirmed = True
logger.debug( logger.debug(
@@ -532,8 +539,12 @@ class FallbackMetadataProvider(ModelMetadataProvider):
if result: if result:
return result return result
except RateLimitError as exc: except RateLimitError as exc:
exc.provider = exc.provider or label logger.warning(
raise exc "Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
label,
exc.retry_after or 0,
)
continue
except Exception as e: except Exception as e:
logger.debug("Provider %s failed for get_model_version: %s", label, e) logger.debug("Provider %s failed for get_model_version: %s", label, e)
continue continue
@@ -550,8 +561,12 @@ class FallbackMetadataProvider(ModelMetadataProvider):
if result: if result:
return result, error return result, error
except RateLimitError as exc: except RateLimitError as exc:
exc.provider = exc.provider or label logger.warning(
raise exc "Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
label,
exc.retry_after or 0,
)
continue
except Exception as e: except Exception as e:
logger.debug("Provider %s failed for get_model_version_info: %s", label, e) logger.debug("Provider %s failed for get_model_version_info: %s", label, e)
continue continue
@@ -572,8 +587,12 @@ class FallbackMetadataProvider(ModelMetadataProvider):
except NotImplementedError: except NotImplementedError:
continue continue
except RateLimitError as exc: except RateLimitError as exc:
exc.provider = exc.provider or label logger.warning(
raise exc "Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
label,
exc.retry_after or 0,
)
continue
except Exception as e: except Exception as e:
logger.debug( logger.debug(
"Provider %s failed for get_model_versions_by_hashes: %s", "Provider %s failed for get_model_versions_by_hashes: %s",
@@ -594,8 +613,12 @@ class FallbackMetadataProvider(ModelMetadataProvider):
if result is not None: if result is not None:
return result return result
except RateLimitError as exc: except RateLimitError as exc:
exc.provider = exc.provider or label logger.warning(
raise exc "Provider %s is rate-limited (retry_after=%.0fs); skipping to next provider",
label,
exc.retry_after or 0,
)
continue
except Exception as e: except Exception as e:
logger.debug("Provider %s failed for get_user_models: %s", label, e) logger.debug("Provider %s failed for get_user_models: %s", label, e)
continue continue

View File

@@ -1067,8 +1067,11 @@ class ModelScanner:
model_data = self._build_cache_entry(metadata, folder=normalized_folder) model_data = self._build_cache_entry(metadata, folder=normalized_folder)
# Compute SHA256 hash when metadata provided none (e.g., CivitAI API response has empty hashes) # Compute SHA256 hash when metadata provided none (e.g., CivitAI API response has empty hashes).
if not model_data.get('sha256') and file_path: # Respect hash_status='pending' (set by CheckpointScanner for large models) to defer
# hash calculation until on-demand — avoids reading entire checkpoint files at startup.
hash_status = model_data.get('hash_status', '')
if not model_data.get('sha256') and hash_status != 'pending' and file_path:
try: try:
logger.info(f"Computing SHA256 hash for {file_path} (was empty from metadata)") logger.info(f"Computing SHA256 hash for {file_path} (was empty from metadata)")
sha256 = await calculate_sha256(file_path) sha256 = await calculate_sha256(file_path)

View File

@@ -3,6 +3,7 @@
from __future__ import annotations from __future__ import annotations
import logging import logging
import time
from typing import Any, Dict, List, Optional, Protocol, Sequence from typing import Any, Dict, List, Optional, Protocol, Sequence
from ..metadata_sync_service import MetadataSyncService from ..metadata_sync_service import MetadataSyncService
@@ -62,26 +63,48 @@ class BulkMetadataRefreshUseCase:
] ]
total_to_process = len(to_process) total_to_process = len(to_process)
initial_skipped = total_models - total_to_process # models excluded from fetch queue
processed = 0 processed = 0
success = 0 success = 0
skipped_count = initial_skipped
handled_count = initial_skipped
needs_resort = False needs_resort = False
start_time = time.monotonic()
failures: List[Dict[str, str]] = []
self._service.scanner.reset_cancellation() self._service.scanner.reset_cancellation()
async def emit(status: str, **extra: Any) -> None: async def emit(status: str, **extra: Any) -> None:
if progress_callback is None: if progress_callback is None:
return return
payload = {"status": status, "total": total_to_process, "processed": processed, "success": success} payload = {
"status": status,
"total": total_models,
"processed": processed,
"success": success,
"failure_count": len(failures),
"skipped_count": skipped_count,
"handled": handled_count,
"elapsed_seconds": int(time.monotonic() - start_time),
}
# Only include full failure details in terminal emits (completed,
# cancelled, rate_limited) to avoid serializing the list on every
# per-model progress update.
if failures and status in ("completed", "cancelled", "rate_limited"):
payload["failures"] = failures
payload.update(extra) payload.update(extra)
await progress_callback.on_progress(payload) await progress_callback.on_progress(payload)
await emit("started") await emit("started")
RATE_LIMIT_ABORT_THRESHOLD = 3
consecutive_rate_limits = 0
for model in to_process: for model in to_process:
if self._service.scanner.is_cancelled(): if self._service.scanner.is_cancelled():
self._logger.info("Bulk metadata refresh cancelled by user") self._logger.info("Bulk metadata refresh cancelled by user")
await emit("cancelled", processed=processed, success=success) await emit("cancelled", processed=processed, success=success)
return {"success": False, "message": "Operation cancelled", "processed": processed, "updated": success, "total": total_models} return {"success": False, "message": "Operation cancelled", "processed": processed, "updated": success, "total": total_models, "failures": failures, "failure_count": len(failures), "skipped_count": skipped_count, "elapsed_seconds": int(time.monotonic() - start_time)}
try: try:
original_name = model.get("model_name") original_name = model.get("model_name")
@@ -101,31 +124,76 @@ class BulkMetadataRefreshUseCase:
model["hash_status"] = "completed" model["hash_status"] = "completed"
else: else:
self._logger.error(f"Failed to calculate hash for {file_path}") self._logger.error(f"Failed to calculate hash for {file_path}")
failures.append({"name": model.get("model_name", file_path or "Unknown"), "error": "Failed to calculate hash"})
processed += 1 processed += 1
handled_count += 1
continue continue
else: else:
self._logger.warning(f"Scanner does not support lazy hash calculation for {file_path}") self._logger.warning(f"Scanner does not support lazy hash calculation for {file_path}")
skipped_count += 1
processed += 1 processed += 1
handled_count += 1
continue continue
# Skip models without valid hash # Skip models without valid hash
if not model.get("sha256"): if not model.get("sha256"):
self._logger.warning(f"Skipping model without hash: {file_path}") self._logger.warning(f"Skipping model without hash: {file_path}")
skipped_count += 1
processed += 1 processed += 1
handled_count += 1
continue continue
await MetadataManager.hydrate_model_data(model) await MetadataManager.hydrate_model_data(model)
result, _ = await self._metadata_sync.fetch_and_update_model( result, error_msg = await self._metadata_sync.fetch_and_update_model(
sha256=model["sha256"], sha256=model["sha256"],
file_path=model["file_path"], file_path=model["file_path"],
model_data=model, model_data=model,
update_cache_func=self._service.scanner.update_single_model_cache, update_cache_func=self._service.scanner.update_single_model_cache,
) )
if not result and error_msg and "Rate limited" in error_msg:
consecutive_rate_limits += 1
else:
consecutive_rate_limits = 0
if not result:
current_name = model.get("model_name", file_path or "Unknown")
failures.append({"name": current_name, "error": error_msg or "Unknown error"})
self._logger.warning("Failed to fetch metadata for %s: %s", current_name, error_msg)
if consecutive_rate_limits >= RATE_LIMIT_ABORT_THRESHOLD:
# The current model was attempted and failed due to rate limiting;
# count it before aborting so the summary is consistent.
processed += 1
handled_count += 1
self._logger.warning(
"Bulk metadata refresh aborted: %d consecutive rate limits detected. "
"Processed %d/%d models.",
consecutive_rate_limits,
processed,
total_to_process,
)
await emit(
"rate_limited",
)
return {
"success": False,
"message": f"Rate limit detected; {total_to_process - processed} models skipped",
"processed": processed,
"updated": success,
"total": total_models,
"failures": failures,
"failure_count": len(failures),
"skipped_count": skipped_count,
"elapsed_seconds": int(time.monotonic() - start_time),
}
if result: if result:
success += 1 success += 1
if original_name != model.get("model_name"): if original_name != model.get("model_name"):
needs_resort = True needs_resort = True
processed += 1 processed += 1
handled_count += 1
await emit( await emit(
"processing", "processing",
processed=processed, processed=processed,
@@ -134,6 +202,9 @@ class BulkMetadataRefreshUseCase:
) )
except Exception as exc: # pragma: no cover - logging path except Exception as exc: # pragma: no cover - logging path
processed += 1 processed += 1
handled_count += 1
current_name = model.get("model_name", model.get("file_path", "Unknown"))
failures.append({"name": current_name, "error": str(exc)})
self._logger.error( self._logger.error(
"Error fetching CivitAI data for %s: %s", "Error fetching CivitAI data for %s: %s",
model.get("file_path"), model.get("file_path"),
@@ -150,7 +221,7 @@ class BulkMetadataRefreshUseCase:
f"{success} of {processed} processed {self._service.model_type}s (total: {total_models})" f"{success} of {processed} processed {self._service.model_type}s (total: {total_models})"
) )
return {"success": True, "message": message, "processed": processed, "updated": success, "total": total_models} return {"success": True, "message": message, "processed": processed, "updated": success, "total": total_models, "failures": failures, "failure_count": len(failures), "skipped_count": skipped_count, "elapsed_seconds": int(time.monotonic() - start_time)}
@staticmethod @staticmethod
def _is_in_skip_path(folder: str, skip_paths: List[str]) -> bool: def _is_in_skip_path(folder: str, skip_paths: List[str]) -> bool:

View File

@@ -31,6 +31,8 @@ PREVIEW_EXTENSIONS = [
".mp4", ".mp4",
".gif", ".gif",
".webm", ".webm",
".avif",
".jxl",
] ]
# Card preview image width # Card preview image width
@@ -41,7 +43,7 @@ EXAMPLE_IMAGE_WIDTH = 832
# Supported media extensions for example downloads # Supported media extensions for example downloads
SUPPORTED_MEDIA_EXTENSIONS = { SUPPORTED_MEDIA_EXTENSIONS = {
"images": [".jpg", ".jpeg", ".png", ".webp", ".gif"], "images": [".jpg", ".jpeg", ".png", ".webp", ".gif", ".avif", ".jxl"],
"videos": [".mp4", ".webm"], "videos": [".mp4", ".webm"],
} }

View File

@@ -62,6 +62,10 @@ class ExampleImagesProcessor:
return '.gif' return '.gif'
elif content.startswith(b'RIFF') and b'WEBP' in content[:12]: elif content.startswith(b'RIFF') and b'WEBP' in content[:12]:
return '.webp' return '.webp'
elif len(content) >= 12 and content[4:8] == b'ftyp' and b'avif' in content[8:24]:
return '.avif'
elif content.startswith(b'\x00\x00\x00\x0cJXL \x0d\x0a\x87\x0a'):
return '.jxl'
elif content.startswith(b'\x00\x00\x00\x18ftypmp4') or content.startswith(b'\x00\x00\x00\x20ftypmp4'): elif content.startswith(b'\x00\x00\x00\x18ftypmp4') or content.startswith(b'\x00\x00\x00\x20ftypmp4'):
return '.mp4' return '.mp4'
elif content.startswith(b'\x1A\x45\xDF\xA3'): elif content.startswith(b'\x1A\x45\xDF\xA3'):
@@ -75,6 +79,8 @@ class ExampleImagesProcessor:
'image/png': '.png', 'image/png': '.png',
'image/gif': '.gif', 'image/gif': '.gif',
'image/webp': '.webp', 'image/webp': '.webp',
'image/avif': '.avif',
'image/jxl': '.jxl',
'video/mp4': '.mp4', 'video/mp4': '.mp4',
'video/webm': '.webm', 'video/webm': '.webm',
'video/quicktime': '.mov' 'video/quicktime': '.mov'

View File

@@ -1,17 +1,125 @@
import json import json
import logging import logging
import os import os
import struct
from io import BytesIO from io import BytesIO
from typing import Any, Optional from typing import Any, Optional
import piexif import piexif
from PIL import Image, PngImagePlugin from PIL import Image, PngImagePlugin
try:
import brotli
_BROTLI_AVAILABLE = True
except ImportError:
brotli = None
_BROTLI_AVAILABLE = False
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
class ExifUtils: class ExifUtils:
"""Utility functions for working with EXIF data in images""" """Utility functions for working with EXIF data in images"""
@staticmethod
def _parse_isobmff_boxes(data: bytes, offset: int = 0) -> list[dict]:
boxes = []
while offset + 8 <= len(data):
size = struct.unpack('>I', data[offset:offset + 4])[0]
box_type = data[offset + 4:offset + 8]
if size == 0:
break
if size < 8 or offset + size > len(data):
break
box_data = data[offset + 8:offset + size]
boxes.append({'type': box_type, 'data': box_data, 'size': size})
offset += size
return boxes
@staticmethod
def _is_jxl_container(data: bytes) -> bool:
if len(data) < 32:
return False
return (
struct.unpack('>I', data[:4])[0] == 12
and data[4:8] == b'JXL '
and data[8:12] == bytes([0x0d, 0x0a, 0x87, 0x0a])
and struct.unpack('>I', data[12:16])[0] >= 16
and data[16:20] == b'ftyp'
and data[20:24] == b'jxl '
)
@staticmethod
def _is_avif_container(data: bytes) -> bool:
if len(data) < 16:
return False
for box in ExifUtils._parse_isobmff_boxes(data):
if box['type'] == b'ftyp' and b'avif' in box['data']:
return True
return False
# Max decompressed size for brotli metadata (2 MB)
_BROTLI_MAX_DECOMPRESSED = 2 * 1024 * 1024
@staticmethod
def _extract_isobmff_brotli(image_path: str) -> Optional[dict]:
try:
with open(image_path, 'rb') as f:
data = f.read()
except Exception:
return None
if ExifUtils._is_jxl_container(data):
boxes = ExifUtils._parse_isobmff_boxes(data, offset=12)
elif ExifUtils._is_avif_container(data):
boxes = ExifUtils._parse_isobmff_boxes(data)
else:
return None
brob = None
for box in boxes:
if box['type'] == b'brob':
brob = box
break
if brob is None:
return None
payload = brob['data']
if payload[:4] != b'comf':
return None
compressed = payload[4:]
if _BROTLI_AVAILABLE:
try:
decompressed = brotli.decompress(compressed)
if len(decompressed) > ExifUtils._BROTLI_MAX_DECOMPRESSED:
logger.warning(
"Brotli metadata too large (%d bytes, max %d), ignoring",
len(decompressed),
ExifUtils._BROTLI_MAX_DECOMPRESSED,
)
decompressed = None
except Exception:
decompressed = None
else:
decompressed = None
raw = decompressed if decompressed is not None else compressed
try:
meta = json.loads(raw.decode('utf-8'))
except Exception:
return None
result = {"parameters": None, "prompt": None, "workflow": None, "comment": None}
if isinstance(meta.get("prompt"), (dict, list)):
result["prompt"] = json.dumps(meta["prompt"])
elif isinstance(meta.get("prompt"), str):
result["prompt"] = meta["prompt"]
if isinstance(meta.get("workflow"), (dict, list)):
result["workflow"] = json.dumps(meta["workflow"])
elif isinstance(meta.get("workflow"), str):
result["workflow"] = meta["workflow"]
return result
@staticmethod @staticmethod
def _decode_user_comment(user_comment: Any) -> Optional[str]: def _decode_user_comment(user_comment: Any) -> Optional[str]:
if user_comment is None: if user_comment is None:
@@ -43,6 +151,12 @@ class ExifUtils:
"comment": None, "comment": None,
} }
ext = os.path.splitext(image_path)[1].lower()
if ext in ('.avif', '.jxl'):
brotli_meta = ExifUtils._extract_isobmff_brotli(image_path)
if brotli_meta:
return brotli_meta
with Image.open(image_path) as img: with Image.open(image_path) as img:
info = getattr(img, "info", {}) or {} info = getattr(img, "info", {}) or {}
@@ -149,7 +263,6 @@ class ExifUtils:
Optional[str]: Extracted metadata or None if not found Optional[str]: Extracted metadata or None if not found
""" """
try: try:
# Skip for video files
if image_path: if image_path:
ext = os.path.splitext(image_path)[1].lower() ext = os.path.splitext(image_path)[1].lower()
if ext in ['.mp4', '.webm']: if ext in ['.mp4', '.webm']:
@@ -177,10 +290,9 @@ class ExifUtils:
str: Path to the updated image str: Path to the updated image
""" """
try: try:
# Skip for video files
if image_path: if image_path:
ext = os.path.splitext(image_path)[1].lower() ext = os.path.splitext(image_path)[1].lower()
if ext in ['.mp4', '.webm']: if ext in ['.mp4', '.webm', '.avif', '.jxl']:
return image_path return image_path
metadata_fields = ExifUtils._load_structured_metadata(image_path) metadata_fields = ExifUtils._load_structured_metadata(image_path)
@@ -212,10 +324,9 @@ class ExifUtils:
def append_recipe_metadata(image_path, recipe_data) -> str: def append_recipe_metadata(image_path, recipe_data) -> str:
"""Append recipe metadata to an image's EXIF data""" """Append recipe metadata to an image's EXIF data"""
try: try:
# Skip for video files
if image_path: if image_path:
ext = os.path.splitext(image_path)[1].lower() ext = os.path.splitext(image_path)[1].lower()
if ext in ['.mp4', '.webm']: if ext in ['.mp4', '.webm', '.avif', '.jxl']:
return image_path return image_path
# First, extract existing metadata # First, extract existing metadata
@@ -327,10 +438,9 @@ class ExifUtils:
Tuple of (optimized_image_data, extension) Tuple of (optimized_image_data, extension)
""" """
try: try:
# Skip for video files early if it's a file path
if isinstance(image_data, str) and os.path.exists(image_data): if isinstance(image_data, str) and os.path.exists(image_data):
ext = os.path.splitext(image_data)[1].lower() ext = os.path.splitext(image_data)[1].lower()
if ext in ['.mp4', '.webm']: if ext in ['.mp4', '.webm', '.avif', '.jxl']:
try: try:
with open(image_data, 'rb') as f: with open(image_data, 'rb') as f:
return f.read(), ext return f.read(), ext

View File

@@ -34,12 +34,21 @@ def _get_hash_chunk_size_bytes() -> int:
async def calculate_sha256(file_path: str) -> str: async def calculate_sha256(file_path: str) -> str:
"""Calculate SHA256 hash of a file (full file content).""" """Calculate SHA256 hash of a file (full file content).
Uses ``posix_fadvise`` with ``POSIX_FADV_DONTNEED`` to avoid polluting the OS page
cache — critical on WSL where cached file pages live inside the VM and are not
accounted for in guest ``used`` memory, causing VmmemWSL to balloon.
"""
sha256_hash = hashlib.sha256() sha256_hash = hashlib.sha256()
chunk_size = _get_hash_chunk_size_bytes() chunk_size = _get_hash_chunk_size_bytes()
with open(file_path, "rb") as f: with open(file_path, "rb") as f:
fd = f.fileno()
for byte_block in iter(lambda: f.read(chunk_size), b""): for byte_block in iter(lambda: f.read(chunk_size), b""):
sha256_hash.update(byte_block) sha256_hash.update(byte_block)
# Evict pages after reading so the data doesn't linger in the kernel page
# cache — on WSL this otherwise appears as unreclaimable VmmemWSL growth.
os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED)
return sha256_hash.hexdigest() return sha256_hash.hexdigest()

View File

@@ -13,3 +13,5 @@ aiosqlite
beautifulsoup4 beautifulsoup4
platformdirs platformdirs
pyyaml pyyaml
# brotli — ISOBMFF (AVIF/JXL) metadata decompression
brotli>=1.2.0

View File

@@ -0,0 +1,196 @@
/* Metadata Refresh Result Modal — component styles only */
.metadata-refresh-result-modal {
max-width: 700px;
}
.refresh-summary-stats {
display: flex;
flex-wrap: wrap;
gap: var(--space-2);
margin-bottom: var(--space-3);
}
.stat-card {
display: flex;
align-items: center;
gap: var(--space-2);
padding: var(--space-2) var(--space-3);
border-radius: var(--border-radius-sm);
background: var(--surface-subtle);
border-left: 4px solid transparent;
font-size: var(--text-sm);
flex: 1;
min-width: 130px;
}
.stat-card > i {
font-size: 1.25em;
flex-shrink: 0;
}
.stat-card-body {
display: flex;
flex-direction: column;
min-width: 0;
}
.stat-card-label {
font-size: var(--text-xs);
color: var(--text-secondary);
line-height: var(--leading-tight);
}
.stat-card-value {
font-weight: var(--weight-bold);
font-size: var(--text-lg);
color: var(--lora-text);
line-height: var(--leading-tight);
}
.stat-card-success {
border-left-color: var(--color-success);
}
.stat-card-success > i {
color: var(--color-success);
}
.stat-card-failure {
border-left-color: var(--color-error);
}
.stat-card-failure > i {
color: var(--color-error);
}
.stat-card-skipped {
border-left-color: var(--color-warning);
}
.stat-card-skipped > i {
color: var(--color-warning);
}
.stat-card-total {
border-left-color: var(--color-info);
}
.stat-card-total > i {
color: var(--color-info);
}
.stat-card-time {
border-left-color: var(--color-accent);
}
.stat-card-time > i {
color: var(--color-accent);
}
.refresh-failures-section {
margin-bottom: var(--space-3);
}
.refresh-failures-section h4 {
margin: 0 0 var(--space-2) 0;
font-size: var(--text-base);
color: var(--color-error);
display: flex;
align-items: center;
gap: var(--space-1);
}
.refresh-failures-section h4 i {
font-size: 0.9em;
}
.failure-table-wrapper {
max-height: 300px;
overflow-y: auto;
border: 1px solid var(--lora-border);
border-radius: var(--border-radius-sm);
}
.failure-table {
width: 100%;
border-collapse: collapse;
font-size: var(--text-sm);
}
.failure-table th {
position: sticky;
top: 0;
background: var(--lora-surface);
border-bottom: 2px solid var(--lora-border);
padding: var(--space-1) var(--space-2);
text-align: left;
font-weight: var(--weight-semibold);
color: var(--text-secondary);
z-index: 1;
}
.failure-table td {
padding: var(--space-1) var(--space-2);
border-bottom: 1px solid var(--lora-border);
vertical-align: top;
}
.failure-table tr:last-child td {
border-bottom: none;
}
.failure-table tr:hover td {
background: var(--surface-subtle);
}
.failure-index {
width: 30px;
text-align: center;
color: var(--text-secondary);
}
.failure-name {
max-width: 300px;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
font-family: var(--font-mono);
font-size: var(--text-xs);
}
.failure-error {
color: var(--color-error);
font-size: var(--text-xs);
}
.refresh-success-message {
display: flex;
align-items: center;
gap: var(--space-2);
padding: var(--space-3);
margin-bottom: var(--space-3);
background: var(--surface-subtle);
border-left: 4px solid var(--color-success);
color: var(--lora-text);
border-radius: var(--border-radius-sm);
font-weight: var(--weight-medium);
}
.refresh-success-message i {
font-size: 1.2em;
flex-shrink: 0;
color: var(--color-success);
}
[data-theme="dark"] .failure-table th {
background: var(--lora-surface);
}
[data-theme="dark"] .failure-table td {
border-bottom-color: var(--lora-border);
}
[data-theme="dark"] .failure-table tr:hover td {
background: var(--surface-subtle);
}

View File

@@ -40,6 +40,7 @@
@import 'components/statistics.css'; /* Add statistics component */ @import 'components/statistics.css'; /* Add statistics component */
@import 'components/sidebar.css'; /* Add sidebar component */ @import 'components/sidebar.css'; /* Add sidebar component */
@import 'components/media-viewer.css'; @import 'components/media-viewer.css';
@import 'components/metadata-refresh-result.css';
.initialization-notice { .initialization-notice {
display: flex; display: flex;

View File

@@ -547,6 +547,14 @@ export class BaseModelApiClient {
const wsProtocol = window.location.protocol === 'https:' ? 'wss://' : 'ws://'; const wsProtocol = window.location.protocol === 'https:' ? 'wss://' : 'ws://';
ws = new WebSocket(`${wsProtocol}${window.location.host}${WS_ENDPOINTS.fetchProgress}`); ws = new WebSocket(`${wsProtocol}${window.location.host}${WS_ENDPOINTS.fetchProgress}`);
// Wait for WebSocket connection to establish
await new Promise((resolve, reject) => {
ws.onopen = resolve;
ws.onerror = reject;
});
// Now that we're connected, set up the message/error handlers
// for the actual operation (separate from connection errors)
const operationComplete = new Promise((resolve, reject) => { const operationComplete = new Promise((resolve, reject) => {
ws.onmessage = (event) => { ws.onmessage = (event) => {
const data = JSON.parse(event.data); const data = JSON.parse(event.data);
@@ -556,25 +564,39 @@ export class BaseModelApiClient {
loading.setStatus('Starting metadata fetch...'); loading.setStatus('Starting metadata fetch...');
break; break;
case 'processing': case 'processing': {
const percent = ((data.processed / data.total) * 100).toFixed(1); const handled = data.handled || data.processed;
const percent = ((handled / data.total) * 100).toFixed(1);
loading.setProgress(percent); loading.setProgress(percent);
loading.setStatus( let statusText = `Processing (${handled}/${data.total}) ${data.current_name || ''}`;
`Processing (${data.processed}/${data.total}) ${data.current_name}` if (data.failure_count > 0) {
); statusText += ` | ❌ ${data.failure_count} failed`;
}
if (data.skipped_count > 0) {
statusText += ` | ⏭️ ${data.skipped_count} skipped`;
}
loading.setStatus(statusText);
break; break;
}
case 'completed': case 'completed': {
loading.setProgress(100); loading.setProgress(100);
loading.setStatus( let summaryText = `Completed: Updated ${data.success} of ${data.processed} ${this.apiConfig.config.displayName}s`;
`Completed: Updated ${data.success} of ${data.processed} ${this.apiConfig.config.displayName}s` if (data.failure_count > 0) {
); summaryText += ` | ❌ ${data.failure_count} failed`;
}
if (data.skipped_count > 0) {
summaryText += ` | ⏭️ ${data.skipped_count} skipped`;
}
summaryText += ` (⏱ ${data.elapsed_seconds || '?'}s)`;
loading.setStatus(summaryText);
resolve(data); resolve(data);
break; break;
}
case 'cancelled': case 'cancelled':
loading.setStatus('Operation cancelled by user'); loading.setStatus('Operation cancelled by user');
resolve(data); // Consider it complete but marked as cancelled resolve(data);
break; break;
case 'error': case 'error':
@@ -588,12 +610,6 @@ export class BaseModelApiClient {
}; };
}); });
// Wait for WebSocket connection to establish
await new Promise((resolve, reject) => {
ws.onopen = resolve;
ws.onerror = reject;
});
const response = await fetch(this.apiConfig.endpoints.fetchAllCivitai, { const response = await fetch(this.apiConfig.endpoints.fetchAllCivitai, {
method: 'POST', method: 'POST',
headers: { 'Content-Type': 'application/json' }, headers: { 'Content-Type': 'application/json' },
@@ -608,10 +624,10 @@ export class BaseModelApiClient {
const finalData = await operationComplete; const finalData = await operationComplete;
resetAndReload(false); resetAndReload(false);
if (finalData && finalData.status === 'cancelled') {
showToast('toast.api.operationCancelledPartial', { success: finalData.success, total: finalData.total }, 'info'); // Show result summary with failure details
} else { if (finalData) {
showToast('toast.api.metadataUpdateComplete', {}, 'success'); this._showMetadataRefreshResult(finalData);
} }
} catch (error) { } catch (error) {
console.error('Error fetching metadata:', error); console.error('Error fetching metadata:', error);
@@ -627,6 +643,210 @@ export class BaseModelApiClient {
}); });
} }
_showMetadataRefreshResult(data) {
const { success, total } = data;
if (data.status === 'cancelled') {
showToast('toast.api.operationCancelledPartial', { success, total }, 'info');
return;
}
this._showFailureDetailsModal(data);
}
_showFailureDetailsModal(data) {
const { failures = [], success, processed, total, failure_count, skipped_count, elapsed_seconds } = data;
// Build failure list HTML
const failureRows = failures.map((f, i) =>
`<tr>
<td class="failure-index">${i + 1}</td>
<td class="failure-name" title="${this._escapeHtml(f.name)}">${this._escapeHtml(f.name)}</td>
<td class="failure-error">${this._escapeHtml(f.error || 'Unknown')}</td>
</tr>`
).join('');
const modalHtml = `
<div id="metadataRefreshResultModal" class="modal" style="display: block;">
<div class="modal-content metadata-refresh-result-modal">
<button class="close" data-action="close-modal">&times;</button>
<h2><i class="fas fa-sync-alt"></i> ${translate('modals.metadataFetchSummary.title', {}, 'Metadata Fetch Summary')}</h2>
<div class="refresh-summary-stats">
<div class="stat-card stat-card-success">
<i class="fas fa-check-circle"></i>
<div class="stat-card-body">
<span class="stat-card-label">${translate('modals.metadataFetchSummary.statSuccess', {}, 'Success')}</span>
<span class="stat-card-value">${success}</span>
</div>
</div>
<div class="stat-card stat-card-failure">
<i class="fas fa-times-circle"></i>
<div class="stat-card-body">
<span class="stat-card-label">${translate('modals.metadataFetchSummary.statFailed', {}, 'Failed')}</span>
<span class="stat-card-value">${failure_count}</span>
</div>
</div>
<div class="stat-card stat-card-skipped">
<i class="fas fa-forward"></i>
<div class="stat-card-body">
<span class="stat-card-label">${translate('modals.metadataFetchSummary.statSkipped', {}, 'Skipped')}</span>
<span class="stat-card-value">${skipped_count}</span>
</div>
</div>
<div class="stat-card stat-card-total">
<i class="fas fa-database"></i>
<div class="stat-card-body">
<span class="stat-card-label">${translate('modals.metadataFetchSummary.statTotal', {}, 'Total Scanned')}</span>
<span class="stat-card-value">${total || processed}</span>
</div>
</div>
<div class="stat-card stat-card-time">
<i class="fas fa-clock"></i>
<div class="stat-card-body">
<span class="stat-card-label">${translate('modals.metadataFetchSummary.statDuration', {}, 'Duration')}</span>
<span class="stat-card-value">${elapsed_seconds}s</span>
</div>
</div>
</div>
${failure_count > 0 ? `
<div class="refresh-failures-section">
<h4><i class="fas fa-exclamation-triangle"></i> ${translate('modals.metadataFetchSummary.failedItems', { count: failure_count }, 'Failed Items (' + failure_count + ')')}</h4>
<div class="failure-table-wrapper">
<table class="failure-table">
<thead>
<tr>
<th>#</th>
<th>${translate('modals.metadataFetchSummary.columnModelName', {}, 'Model Name')}</th>
<th>${translate('modals.metadataFetchSummary.columnError', {}, 'Error')}</th>
</tr>
</thead>
<tbody>${failureRows}</tbody>
</table>
</div>
</div>
` : `
<div class="refresh-success-message">
<i class="fas fa-check-circle"></i> ${translate('modals.metadataFetchSummary.successMessage', { count: success, type: this.apiConfig.config.displayName }, 'All ' + success + ' ' + this.apiConfig.config.displayName + 's updated successfully!')}
</div>
`}
<div class="modal-actions">
<button class="cancel-btn" data-action="close-modal">${translate('modals.metadataFetchSummary.close', {}, 'Close')}</button>
${failure_count > 0 ? `
<button class="secondary-btn" data-action="copy-report"><i class="fas fa-copy"></i> ${translate('modals.metadataFetchSummary.copyReport', {}, 'Copy Report')}</button>
<button class="secondary-btn" data-action="download-csv"><i class="fas fa-download"></i> ${translate('modals.metadataFetchSummary.downloadCsv', {}, 'Download CSV')}</button>
` : ''}
</div>
</div>
</div>
`;
const existing = document.getElementById('metadataRefreshResultModal');
if (existing) existing.remove();
const container = document.createElement('div');
container.innerHTML = modalHtml;
const modal = container.firstElementChild;
document.body.appendChild(modal);
modal.addEventListener('click', (e) => {
const action = e.target.closest('[data-action]')?.dataset.action;
if (!action) return;
e.preventDefault();
switch (action) {
case 'close-modal':
modal.remove();
break;
case 'copy-report':
BaseModelApiClient._copyRefreshReport(e.target.closest('[data-action]'), data);
break;
case 'download-csv':
BaseModelApiClient._downloadRefreshReport(data);
break;
}
});
}
_escapeHtml(str) {
if (!str) return '';
const div = document.createElement('div');
div.textContent = str;
return div.innerHTML;
}
static _copyRefreshReport(btn, data) {
const { failures = [], success, processed, total, failure_count, skipped_count, elapsed_seconds } = data;
const lines = [
'=== Metadata Refresh Report ===',
`Date: ${new Date().toLocaleString()}`,
`Duration: ${elapsed_seconds}s`,
`Total scanned: ${total || processed}`,
`Successfully updated: ${success}`,
`Failed: ${failure_count}`,
`Skipped: ${skipped_count}`,
'',
];
if (failure_count > 0) {
lines.push('--- Failed Items ---');
failures.forEach((f, i) => {
lines.push(`${i + 1}. ${f.name || 'Unknown'}${f.error || 'Unknown error'}`);
});
lines.push('');
}
lines.push('====================');
const text = lines.join('\n');
navigator.clipboard.writeText(text).then(() => {
showToast('toast.api.copiedToClipboard', {}, 'success');
if (btn) {
const origHTML = btn.innerHTML;
btn.innerHTML = '<i class="fas fa-check"></i> Copied!';
setTimeout(() => { btn.innerHTML = origHTML; }, 2000);
}
}).catch(() => {
// Fallback
const textarea = document.createElement('textarea');
textarea.value = text;
document.body.appendChild(textarea);
textarea.select();
document.execCommand('copy');
document.body.removeChild(textarea);
showToast('toast.api.copiedToClipboard', {}, 'success');
});
}
static _downloadRefreshReport(data) {
const { failures = [], success, processed, total, failure_count, skipped_count, elapsed_seconds } = data;
// CSV header
let csv = 'Model Name,Error\n';
failures.forEach(f => {
const name = (f.name || 'Unknown').replace(/"/g, '""');
const error = (f.error || 'Unknown').replace(/"/g, '""');
csv += `"${name}","${error}"\n`;
});
// Add summary as trailing comments
csv += `\n# Summary: ${success} success, ${failure_count} failed, ${skipped_count} skipped, ${elapsed_seconds}s\n`;
csv += `# Total scanned: ${total || processed}\n`;
const blob = new Blob([csv], { type: 'text/csv;charset=utf-8;' });
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = `metadata-refresh-failures-${Date.now()}.csv`;
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
URL.revokeObjectURL(url);
showToast('toast.api.downloadStarted', {}, 'success');
}
async refreshBulkModelMetadata(filePaths) { async refreshBulkModelMetadata(filePaths) {
if (!filePaths || filePaths.length === 0) { if (!filePaths || filePaths.length === 0) {
throw new Error('No file paths provided'); throw new Error('No file paths provided');

View File

@@ -355,9 +355,9 @@ function renderImportInterface(isEmpty) {
<button class="select-files-btn" id="selectExampleFilesBtn"> <button class="select-files-btn" id="selectExampleFilesBtn">
<i class="fas fa-folder-open"></i> Select Files <i class="fas fa-folder-open"></i> Select Files
</button> </button>
<p class="import-formats">Supported formats: jpg, png, gif, webp, mp4, webm</p> <p class="import-formats">Supported formats: jpg, png, gif, webp, avif, jxl, mp4, webm</p>
</div> </div>
<input type="file" id="exampleFilesInput" multiple accept="image/*,video/mp4,video/webm" style="display: none;"> <input type="file" id="exampleFilesInput" multiple accept="image/*,image/avif,image/jxl,video/mp4,video/webm" style="display: none;">
<div class="import-progress-container" style="display: none;"> <div class="import-progress-container" style="display: none;">
<div class="import-progress"> <div class="import-progress">
<div class="progress-bar"></div> <div class="progress-bar"></div>
@@ -473,7 +473,7 @@ export function initExampleImport(modelHash, container) {
*/ */
async function handleImportFiles(files, modelHash, importContainer) { async function handleImportFiles(files, modelHash, importContainer) {
// Filter for supported file types // Filter for supported file types
const supportedImages = ['.jpg', '.jpeg', '.png', '.gif', '.webp']; const supportedImages = ['.jpg', '.jpeg', '.png', '.gif', '.webp', '.avif', '.jxl'];
const supportedVideos = ['.mp4', '.webm']; const supportedVideos = ['.mp4', '.webm'];
const supportedExtensions = [...supportedImages, ...supportedVideos]; const supportedExtensions = [...supportedImages, ...supportedVideos];

View File

@@ -441,7 +441,6 @@ async def test_fetch_and_update_model_returns_rate_limit_error(tmp_path):
assert ok is False assert ok is False
assert error is not None and "Rate limited" in error assert error is not None and "Rate limited" in error
assert "7" in error
helpers.metadata_manager.save_metadata.assert_not_awaited() helpers.metadata_manager.save_metadata.assert_not_awaited()
update_cache.assert_not_awaited() update_cache.assert_not_awaited()
helpers.provider_selector.assert_not_awaited() helpers.provider_selector.assert_not_awaited()

View File

@@ -63,7 +63,8 @@ async def test_fallback_retries_same_provider_on_rate_limit(monkeypatch):
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_fallback_respects_retry_limit(monkeypatch): async def test_fallback_continues_to_next_provider_on_rate_limit(monkeypatch):
"""After exhausting retries on primary, fallback should continue to secondary."""
sleep_mock = AsyncMock() sleep_mock = AsyncMock()
monkeypatch.setattr(provider_module.asyncio, "sleep", sleep_mock) monkeypatch.setattr(provider_module.asyncio, "sleep", sleep_mock)
monkeypatch.setattr(provider_module.random, "uniform", lambda *_: 0.0) monkeypatch.setattr(provider_module.random, "uniform", lambda *_: 0.0)
@@ -76,13 +77,13 @@ async def test_fallback_respects_retry_limit(monkeypatch):
rate_limit_retry_limit=2, rate_limit_retry_limit=2,
) )
with pytest.raises(RateLimitError) as exc_info: # After Change A: no longer raises; falls through to secondary
await fallback.get_model_by_hash("abc") result, error = await fallback.get_model_by_hash("abc")
assert exc_info.value.provider == "primary" assert error is None
assert primary.calls == 2 assert result == {"id": "secondary"}
assert secondary.calls == 0 assert primary.calls == 2 # retry_limit exhausted on primary
sleep_mock.assert_awaited_once() assert secondary.calls == 1 # secondary IS called now
@pytest.mark.asyncio @pytest.mark.asyncio
@@ -117,3 +118,40 @@ async def test_rate_limit_retrying_provider_respects_limit(monkeypatch):
assert exc_info.value.provider == "inner" assert exc_info.value.provider == "inner"
assert inner.calls == 2 assert inner.calls == 2
sleep_mock.assert_awaited_once() sleep_mock.assert_awaited_once()
@pytest.mark.asyncio
async def test_retry_helper_limits_retries_for_large_retry_after():
"""With retry_after >= 120s, _RateLimitRetryHelper should only attempt once (no retries)."""
calls = 0
async def failing():
nonlocal calls
calls += 1
raise RateLimitError("limited", retry_after=1500.0)
helper = provider_module._RateLimitRetryHelper(retry_limit=3)
with pytest.raises(RateLimitError):
await helper.run("test", failing)
assert calls == 1 # No retries for large retry_after
@pytest.mark.asyncio
async def test_retry_helper_retries_normally_for_small_retry_after(monkeypatch):
"""With retry_after < 120s, _RateLimitRetryHelper should retry normally (up to limit)."""
sleep_mock = AsyncMock()
monkeypatch.setattr(provider_module.asyncio, "sleep", sleep_mock)
calls = 0
async def succeeding():
nonlocal calls
calls += 1
if calls == 1:
raise RateLimitError("limited", retry_after=30.0)
return {"ok": True}, None
helper = provider_module._RateLimitRetryHelper(retry_limit=3)
result, _ = await helper.run("test", succeeding)
assert result == {"ok": True}
assert calls == 2 # Retried once (small retry_after)

View File

@@ -141,3 +141,150 @@ def test_update_image_metadata_preserves_png_workflow(tmp_path):
img.info["parameters"] img.info["parameters"]
== 'prompt text\nRecipe metadata: {"title":"recipe"}' == 'prompt text\nRecipe metadata: {"title":"recipe"}'
) )
# --- ISOBMFF / brotli extraction tests ---
import struct
import brotli
def _build_jxl_with_brob(payload_json: dict) -> bytes:
"""Build a minimal JXL container with a brob box containing brotli-compressed JSON."""
# ISOBMFF box 1: JXL signature box (size=12, type='JXL ', signature)
box1 = struct.pack(">I", 12) + b"JXL " + bytes([0x0d, 0x0a, 0x87, 0x0a])
# ISOBMFF box 2: ftyp (size=16, type='ftyp', major='jxl ', minor=0)
box2 = struct.pack(">I", 16) + b"ftyp" + b"jxl " + struct.pack(">I", 0)
# ISOBMFF box 3: brob — payload is b'comf' + brotli(json)
compressed = brotli.compress(json.dumps(payload_json).encode("utf-8"))
brob_payload = b"comf" + compressed
box3 = struct.pack(">I", 8 + len(brob_payload)) + b"brob" + brob_payload
return box1 + box2 + box3
def _build_avif_with_brob(payload_json: dict) -> bytes:
"""Build a minimal AVIF container with a brob box containing brotli-compressed JSON."""
compressed = brotli.compress(json.dumps(payload_json).encode("utf-8"))
brob_payload = b"comf" + compressed
ftyp_box = struct.pack(">I", 20) + b"ftyp" + b"avif" + struct.pack(">I", 0) + b"avif"
brob_box = struct.pack(">I", 8 + len(brob_payload)) + b"brob" + brob_payload
return ftyp_box + brob_box
class TestIsobmffBrotliExtraction:
"""Tests for ISOBMFF brotli metadata extraction in ExifUtils."""
def test_extract_jxl_brotli_happy_path(self, tmp_path):
"""JXL container with valid brob box extracts prompt and workflow."""
payload = {"prompt": "a cute cat", "workflow": {"nodes": [{"id": 1}]}}
data = _build_jxl_with_brob(payload)
path = tmp_path / "test.jxl"
path.write_bytes(data)
result = ExifUtils._load_structured_metadata(str(path))
assert result["prompt"] == "a cute cat"
assert result["workflow"] == '{"nodes": [{"id": 1}]}'
assert result["parameters"] is None
assert result["comment"] is None
def test_extract_avif_brotli_happy_path(self, tmp_path):
"""AVIF container with valid brob box extracts prompt and workflow."""
payload = {"prompt": "landscape", "workflow": {"nodes": []}}
data = _build_avif_with_brob(payload)
path = tmp_path / "test.avif"
path.write_bytes(data)
result = ExifUtils._load_structured_metadata(str(path))
assert result["prompt"] == "landscape"
assert result["workflow"] == '{"nodes": []}'
def test_extract_no_brob_box_returns_none(self, tmp_path):
"""JXL container without a brob box returns None from _extract_isobmff_brotli."""
# Only JXL signature + ftyp, no brob
box1 = struct.pack(">I", 12) + b"JXL " + bytes([0x0d, 0x0a, 0x87, 0x0a])
box2 = struct.pack(">I", 16) + b"ftyp" + b"jxl " + struct.pack(">I", 0)
path = tmp_path / "test.jxl"
path.write_bytes(box1 + box2)
# The low-level extraction should return None (no brob box)
result = ExifUtils._extract_isobmff_brotli(str(path))
assert result is None
def test_extract_corrupt_brob_returns_none(self, tmp_path):
"""Broken brob box payload gracefully returns None."""
box1 = struct.pack(">I", 12) + b"JXL " + bytes([0x0d, 0x0a, 0x87, 0x0a])
box2 = struct.pack(">I", 16) + b"ftyp" + b"jxl " + struct.pack(">I", 0)
# brob with garbage payload that doesn't start with b'comf'
garbage = b"\xff\xff\xff\xff" * 32
box3 = struct.pack(">I", 8 + len(garbage)) + b"brob" + garbage
path = tmp_path / "test.jxl"
path.write_bytes(box1 + box2 + box3)
result = ExifUtils._extract_isobmff_brotli(str(path))
assert result is None
def test_extract_non_isobmff_file_falls_through(self, tmp_path):
"""A regular PNG file is not processed as ISOBMFF and returns PIL metadata."""
png_info = PngImagePlugin.PngInfo()
png_info.add_text("prompt", "from png")
path = tmp_path / "test.png"
Image.new("RGB", (4, 4), color="red").save(path, pnginfo=png_info)
result = ExifUtils._load_structured_metadata(str(path))
assert result["prompt"] == "from png"
def test_extract_skip_on_update_and_optimize(self, tmp_path):
"""AVIF/JXL files are skipped for write operations (update/append/optimize)."""
path = tmp_path / "test.avif"
path.write_bytes(b"fake avif data")
# update_image_metadata should return the path unchanged
result = ExifUtils.update_image_metadata(str(path), "some metadata")
assert result == str(path)
# append_recipe_metadata should also skip
result = ExifUtils.append_recipe_metadata(str(path), {"title": "test"})
assert result == str(path)
# optimize_image should passthrough for AVIF/JXL paths
result_data, ext = ExifUtils.optimize_image(str(path))
assert ext == ".avif"
assert result_data == b"fake avif data"
def test_extract_prompt_as_dict(self, tmp_path):
"""prompt field as dict is JSON-serialized."""
payload = {"prompt": {"text": "hello", "negative": "bad"}}
data = _build_jxl_with_brob(payload)
path = tmp_path / "test.jxl"
path.write_bytes(data)
result = ExifUtils._load_structured_metadata(str(path))
assert json.loads(result["prompt"]) == {"text": "hello", "negative": "bad"}
def test_extract_workflow_as_list(self, tmp_path):
"""workflow field as list is JSON-serialized."""
payload = {"workflow": [{"id": 1}, {"id": 2}]}
data = _build_avif_with_brob(payload)
path = tmp_path / "test.avif"
path.write_bytes(data)
result = ExifUtils._load_structured_metadata(str(path))
assert json.loads(result["workflow"]) == [{"id": 1}, {"id": 2}]
def test_over_decompressed_size_limit(self, tmp_path, monkeypatch):
"""Decompressed data exceeding _BROTLI_MAX_DECOMPRESSED is rejected."""
# Monkey-patch the limit to a small value to avoid large test data
monkeypatch.setattr(ExifUtils, "_BROTLI_MAX_DECOMPRESSED", 100)
large_content = "x" * 200
payload = {"prompt": large_content}
data = _build_jxl_with_brob(payload)
path = tmp_path / "test.jxl"
path.write_bytes(data)
# Direct extraction should return None because decompressed size exceeds limit
result = ExifUtils._extract_isobmff_brotli(str(path))
assert result is None

View File

@@ -0,0 +1,135 @@
import { app } from "../../scripts/app.js";
import { chainCallback, getAllGraphNodes, getWidgetByName } from "./utils.js";
/**
* Format a date string using the given pattern (e.g. "yyyy-MM-dd").
* Supports: yyyy, yy, MM, M, dd, d, hh, h, mm, m, ss, s
*/
function formatDate(text, date) {
const pad = (n, len) => n.toString().padStart(len, "0");
// Order matters: longer patterns first to avoid partial substring matches.
// The original ComfyUI frontend uses the same ordered-alternation approach.
return text
.replace(/yyyy/g, () => date.getFullYear().toString())
.replace(/yy/g, () => pad(date.getFullYear() % 100, 2))
.replace(/MM/g, () => pad(date.getMonth() + 1, 2))
.replace(/M/g, () => (date.getMonth() + 1).toString())
.replace(/dd/g, () => pad(date.getDate(), 2))
.replace(/d/g, () => date.getDate().toString())
.replace(/hh/g, () => pad(date.getHours(), 2))
.replace(/h/g, () => date.getHours().toString())
.replace(/mm/g, () => pad(date.getMinutes(), 2))
.replace(/m/g, () => date.getMinutes().toString())
.replace(/ss/g, () => pad(date.getSeconds(), 2))
.replace(/s/g, () => date.getSeconds().toString());
}
/**
* Resolve %NodeTitle.WidgetName% placeholders in a string using the current graph.
*
* Patterns supported:
* %NodeTitle.WidgetName% widget value from a node (by title or "Node name for S&R")
* %date:format% current date/time formatted (e.g. %date:yyyy-MM-dd%)
* %width%, %height% left as-is, handled by the backend
*
* All other %text% patterns are passed through unchanged (they may be handled by
* the backend's format_filename, e.g. %seed%, %model%, %pprompt%).
*/
function applyTextReplacements(value) {
if (!value || typeof value !== "string" || !value.includes("%")) {
return value;
}
// Collect all nodes from the entire graph hierarchy (including subgraphs)
const allNodes = getAllGraphNodes(app.graph);
return value.replace(/%([^%]+)%/g, function (match, text) {
const split = text.split(".");
if (split.length !== 2) {
// Handle %date:format% patterns
if (split[0].startsWith("date:")) {
return formatDate(split[0].substring(5), new Date());
}
// %width% and %height% are left for the backend to handle
if (text !== "width" && text !== "height") {
console.warn(
"[Save Image (LoraManager)] Unknown placeholder: %" + text + "%"
);
}
return match;
}
// Try finding the node by its "Node name for S&R" property first
let nodes = allNodes
.filter((n) => n.node.properties?.["Node name for S&R"] === split[0])
.map((n) => n.node);
// Fall back to matching by node title
if (!nodes.length) {
nodes = allNodes
.filter((n) => n.node.title === split[0])
.map((n) => n.node);
}
if (!nodes.length) {
console.warn(
"[Save Image (LoraManager)] Node not found: " + split[0]
);
return match;
}
if (nodes.length > 1) {
console.warn(
"[Save Image (LoraManager)] Multiple nodes matched '" +
split[0] +
"', using first match"
);
}
const node = nodes[0];
const widget = node.widgets?.find((w) => w.name === split[1]);
if (!widget) {
console.warn(
"[Save Image (LoraManager)] Widget '" +
split[1] +
"' not found on node " +
split[0]
);
return match;
}
// Sanitize the value: replace characters invalid for filenames
// eslint-disable-next-line no-control-regex
return ((widget.value ?? "") + "").replaceAll(
/[/?<>\\:*|"\x00-\x1F\x7F]/g,
"_"
);
});
}
app.registerExtension({
name: "LoraManager.SaveImageExtraOutput",
async beforeRegisterNodeDef(nodeType, nodeData) {
if (nodeData.name !== "Save Image (LoraManager)") {
return;
}
chainCallback(nodeType.prototype, "onNodeCreated", function () {
// Find the filename_prefix widget
const widget = getWidgetByName(this, "filename_prefix");
if (!widget) {
console.warn(
"[Save Image (LoraManager)] filename_prefix widget not found"
);
return;
}
// Override serialization to resolve %NodeTitle.WidgetName% placeholders
widget.serializeValue = () => {
return applyTextReplacements(widget.value);
};
});
},
});