Commit Graph

243 Commits

Author SHA1 Message Date
Will Miao
ede97f3f3e Fix calculate_recipe_fingerprint to handle non-string hash and invalid strength values
- Handle non-string hash values by converting to string before lower()
- Add try-except for strength conversion to handle invalid values like empty strings
- Fixes hypothesis test failures when random data generates unexpected types
2026-02-25 00:11:38 +08:00
Will Miao
70398ed985 feat(lora-loader): Load LoRAs using lower-level API to bypass folder_paths validation
- Add get_lora_info_absolute() function to return absolute file paths
- Replace LoraLoader().load_lora() with comfy.utils.load_torch_file() +
  comfy.sd.load_lora_for_models() to enable loading LoRAs from any path
- This allows LoRA Manager to load LoRAs from non-standard paths (multi-library support)
- Fixes #805
2026-02-23 18:06:15 +08:00
Will Miao
26b139884c perf(usage-stats): prevent unnecessary writes when idle
- Add is_dirty flag to track if statistics have changed
- Only write stats file when data actually changes
- Add enable_usage_statistics setting in ComfyUI settings
- Skip backend requests when usage statistics is disabled
- Fix standalone mode compatibility for MetadataRegistry

Fixes #826
2026-02-23 14:00:00 +08:00
Will Miao
b9516c6b62 fix: Handle missing Civitai API response fields gracefully
Fix KeyError when 'hashes', 'name', or 'model' fields are missing from
Civitai API responses. Use .get() with defaults instead of direct dict
access in:

- LoraMetadata.from_civitai_info()
- CheckpointMetadata.from_civitai_info()
- EmbeddingMetadata.from_civitai_info()
- RecipeScanner._get_hash_from_civitai()
- DownloadManager._process_download()

Fixes #820
2026-02-18 12:02:48 +08:00
Will Miao
6b1e3f06ed refactor(example-images): minimize async lock contention by moving I/O outside critical sections
- Extract progress file loading to async methods to run in executor
- Refactor start_download to reduce lock time by pre-loading data before entering lock
- Improve check_pending_models efficiency with single-pass model collection and async loading
- Add type hints to get_status method
- Add tests for download task callback execution and error handling
2026-02-11 09:24:00 +08:00
Will Miao
765c1c42a9 feat: enhance skip metadata refresh with smart UI and subtle badges, #790 2026-02-09 09:56:18 +08:00
Will Miao
c12aefa82a fix(recipes): detect duplicates for remote imports using modelVersionId and Civitai URL, #750
- Use modelVersionId as fallback for all loras in fingerprint calculation (not just deleted)
- Add URL-based duplicate detection using source_path field
- Combine both fingerprint and URL-based duplicate detection in API response
- Fix _download_remote_media return type and unbound variable issue
2026-02-03 21:32:15 +08:00
Will Miao
1da476d858 feat(example-images): add check pending models endpoint and improve async handling
- Add /api/example-images/check-pending endpoint to quickly check models needing downloads
- Improve DownloadManager.start_download() to return immediately without blocking
- Add _handle_download_task_done callback for proper error handling and progress saving
- Add check_pending_models() method for lightweight pre-download validation
- Update frontend ExampleImagesManager to use new check-pending endpoint
- Add comprehensive tests for new functionality
2026-02-02 12:31:07 +08:00
Will Miao
e252e44403 refactor(logging): replace print statements with logger for consistency 2026-02-02 10:47:17 +08:00
Will Miao
ffcfe5ea3e fix(metadata): rename model_type to sub_type and add embedding subtype, see #797
- Change `model_type` field to `sub_type` for checkpoint models to improve naming consistency
- Add `sub_type="embedding"` for embedding models to properly categorize model subtypes
- Maintain backward compatibility with existing metadata structure
2026-01-31 22:54:53 +08:00
Will Miao
719e18adb6 feat(media): add media type hint support for file extension detection, fixes #795 and fixes #751
- Add optional `media_type_hint` parameter to `_get_file_extension_from_content_or_headers` method
- When `media_type_hint` is "video" and no extension can be determined from content/headers/URL, default to `.mp4`
- Pass image metadata type as hint in both `process_example_images` and `process_example_images_batch` methods
- Add unit tests to verify media type hint behavior and priority
2026-01-31 19:39:37 +08:00
Will Miao
5e91073476 refactor: unify model_type semantics by introducing sub_type field
This commit resolves the semantic confusion around the model_type field by
clearly distinguishing between:
- scanner_type: architecture-level (lora/checkpoint/embedding)
- sub_type: business-level subtype (lora/locon/dora/checkpoint/diffusion_model/embedding)

Backend Changes:
- Rename model_type to sub_type in CheckpointMetadata and EmbeddingMetadata
- Add resolve_sub_type() and normalize_sub_type() in model_query.py
- Update checkpoint_scanner to use _resolve_sub_type()
- Update service format_response to include both sub_type and model_type
- Add VALID_*_SUB_TYPES constants with backward compatible aliases

Frontend Changes:
- Add MODEL_SUBTYPE_DISPLAY_NAMES constants
- Keep MODEL_TYPE_DISPLAY_NAMES as backward compatible alias

Testing:
- Add 43 new tests covering sub_type resolution and API response

Documentation:
- Add refactoring todo document to docs/technical/

BREAKING CHANGE: None - full backward compatibility maintained
2026-01-30 06:56:10 +08:00
Will Miao
7b0c6c8bab refactor(cache): reorganize cache directory structure with automatic legacy cleanup
- Centralize cache path resolution in new py/utils/cache_paths.py module
- Migrate legacy cache files to organized structure: {settings_dir}/cache/{model|recipe|fts|symlink}/
- Automatically clean up legacy files after successful migration with integrity verification
- Update Config symlink cache to use new path and migrate from old location
- Simplify service classes (PersistentModelCache, PersistentRecipeCache, RecipeFTSIndex, TagFTSIndex) to use centralized migration logic
- Add comprehensive test coverage for cache paths and automatic cleanup
2026-01-26 16:12:08 +08:00
Will Miao
1dee7f5cf9 feat(constants): standardize formatting and expand diffusion model list
- Normalize string quotes to double quotes across all constants for consistency
- Add trailing commas in dictionaries and lists to improve diff readability
- Expand DIFFUSION_MODEL_BASE_MODELS with additional Wan Video and Qwen models
- Fix comment spacing in NSFW_LEVELS dictionary
- Maintain all existing functionality while improving code style
2026-01-19 01:20:46 +08:00
Will Miao
b0f0158f98 feat(example-images): add force parameter to retry failed downloads
When force=true is passed via API, models in failed_models set are
re-downloaded instead of being skipped. On successful download, model is
removed from failed_models set.

This provides a manual batch repair mechanism for users when CivitAI
media server is temporarily down and causes empty folders.

Changes:
- Backend: Add force parameter to start_download(), _download_all_example_images(), _process_model()
- Backend: Skip failed_models check when force=true
- Backend: Remove model from failed_models on successful force retry
- Frontend: GlobalContextMenu now calls API with force=true directly
- Tests: Update mock to accept force parameter
2026-01-18 21:58:12 +08:00
Will Miao
dad549f65f feat(download): auto-route diffusion models to unet folder based on baseModel, see #770
CivitAI does not distinguish between checkpoint and diffusion model types -
both are labeled as "checkpoint". For certain base model types like
"ZImageTurbo", all models are actually diffusion models and should be
saved to the unet/diffusion model folder instead of the checkpoint folder.

- Add DIFFUSION_MODEL_BASE_MODELS constant for known diffusion model types
- Add default_unet_root setting with auto-set logic
- Route downloads to unet folder when baseModel matches known diffusion types

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 11:58:20 +08:00
Will Miao
aab1797269 Revert "feat: add automatic retry for failed example image downloads"
This reverts commit cb460fcdb0.
2026-01-18 10:55:30 +08:00
Will Miao
cb460fcdb0 feat: add automatic retry for failed example image downloads
- Add failed_model_timestamps to track when models fail
- Retry failed models after 24-hour cooldown period
- Skip retry if example folder already has files
- Skip retry if failure was less than 24 hours ago
- Log count of failed models with retry message
- Fix unbound snapshot variable in exception path
- Remove duplicate/unreachable directory check code
- Update string quotes to double quotes (PEP 8)

This fixes the issue where failed models were permanently skipped in
auto-download mode, even when their example folders were empty.
2026-01-18 08:55:49 +08:00
Will Miao
fa063ba1ce fix: Reprocess example images with missing folders, improve error handling, and add new tests. Fixes #760 2026-01-08 00:25:36 +08:00
Will Miao
f13f22c949 chore: Improve logging by adding SHA256 hash calculation timing and clarifying metadata creation messages. 2026-01-03 08:56:34 +08:00
Will Miao
4317b06049 fix: Prevent EXIF metadata operations on video files by adding type checks. 2026-01-02 20:18:03 +08:00
Will Miao
953117efa1 feat: add logging setup and standalone mode detection to LoRA Manager
- Initialize logging configuration via `setup_logging()` when not in standalone mode
- Detect standalone mode using environment variables `LORA_MANAGER_STANDALONE` and `HF_HUB_DISABLE_TELEMETRY`
- Remove redundant `STANDALONE_MODE` variable that previously checked `sys.modules`
2025-12-31 18:52:44 +08:00
Will Miao
a6e23a7630 feat(example-images): add NSFW level setting endpoint
Add new POST endpoint `/api/lm/example-images/set-nsfw-level` to allow updating NSFW classification for individual example images. The endpoint supports both regular and custom images, validates required parameters, and updates the corresponding model metadata. This enables users to manually adjust NSFW ratings for better content filtering.
2025-12-09 20:37:16 +08:00
Will Miao
22ee37b817 feat: parse aggregate commercial use values, see #708
Add support for parsing comma-separated and JSON-style commercial use permission values in both Python backend and JavaScript frontend. Implement helper functions to split aggregated values into individual permissions while preserving original values when no aggregation is detected.

Added comprehensive test coverage for the new parsing functionality to ensure correct handling of various input formats including strings, arrays, and iterable objects with aggregated commercial use values.
2025-11-30 17:18:28 +08:00
Will Miao
df93670598 feat: add checkpoint metadata to EXIF recipe data
Add support for storing checkpoint information in image EXIF metadata. The checkpoint data is simplified and includes fields like model ID, version, name, hash, and base model. This allows for better tracking of AI model checkpoints used in image generation workflows.
2025-11-29 08:46:38 +08:00
Will Miao
4557da8b63 feat(metadata): return tuple with metadata and success flag
Change `load_metadata` method to return a tuple containing both the metadata object and a boolean success flag instead of just the metadata object. This provides clearer error handling and allows callers to distinguish between successful loads with null metadata versus actual load failures.
2025-11-11 11:18:33 +08:00
Will Miao
415fc5720c feat(settings): add configurable hash chunk size 2025-11-10 10:15:01 +08:00
Will Miao
ba6e2eadba feat: update license flag handling and default permissions
- Update DEFAULT_LICENSE_FLAGS from 57 to 127 to enable all commercial modes by default
- Replace CommercialUseLevel enum with bitwise commercial permission handling
- Simplify commercial value normalization and validation using allowed values set
- Adjust bit shifting in license flag construction to accommodate new commercial bits structure
- Remove CommercialUseLevel from exports and update tests accordingly
- Improve handling of empty commercial use values with proper type checking

The changes streamline commercial permission processing and align with CivitAI's default license configuration while maintaining backward compatibility.
2025-11-06 22:14:36 +08:00
Will Miao
ddf9e33961 feat: add license information handling for Civitai models
Add license resolution utilities and integrate license information into model metadata processing. The changes include:

- Add `resolve_license_payload` function to extract license data from Civitai model responses
- Integrate license information into model metadata in CivitaiClient and MetadataSyncService
- Add license flags support in model scanning and caching
- Implement CommercialUseLevel enum for standardized license classification
- Update model scanner to handle unknown fields when extracting metadata values

This ensures proper license attribution and compliance when working with Civitai models.
2025-11-06 17:05:54 +08:00
Will Miao
3d6bb432c4 feat: normalize tags to lowercase for Windows compatibility, see #637
Convert all tags to lowercase in tag processing logic to prevent case sensitivity issues on Windows filesystems. This ensures consistent tag matching and prevents duplicate tags with different cases from being created.

Changes include:
- TagUpdateService now converts tags to lowercase before comparison
- Utils function converts model tags to lowercase before priority resolution
- Test cases updated to reflect lowercase tag expectations
2025-11-04 12:54:09 +08:00
pixelpaws
c2db5eb6df Merge pull request #634 from willmiao/codex/refactor-config-to-handle-default-library
fix: drop template default library when saving ComfyUI paths
2025-11-02 09:12:26 +08:00
pixelpaws
ef0bcc6cf1 fix(config): remove template default library before saving paths 2025-11-02 09:09:36 +08:00
pixelpaws
285428ad3a fix(portable): use project root for settings storage 2025-11-02 09:07:57 +08:00
pixelpaws
ce42d83ce9 feat(settings): add portable settings toggle 2025-10-30 14:08:21 +08:00
pixelpaws
5326fa2970 Merge pull request #607 from willmiao/codex/fix-404-errors-for-example-image-links, see #590
Preserve local previews when CivitAI images 404
2025-10-27 19:09:41 +08:00
pixelpaws
4753206c52 fix(example-images): accept legacy library folders 2025-10-27 18:21:43 +08:00
pixelpaws
613aa3b1c3 fix(example-images): preserve local previews on 404 2025-10-27 18:21:14 +08:00
pixelpaws
807425f12a feat(preview): respect blur mature content setting 2025-10-25 06:43:03 +08:00
pixelpaws
f7a856349a fix(utils): sanitize path template folder names 2025-10-18 16:20:47 +08:00
Will Miao
10f5588e4a feat: add model_name and version_name placeholders to download paths, #552
Add support for {model_name} and {version_name} placeholders in download path templates. These new placeholders allow for more flexible and descriptive file organization by including the actual model name and version name in the download directory structure.

Changes include:
- Updated download_manager.py and utils.py to handle new placeholders
- Added placeholders to constants.js for UI reference
- Updated settings modal template to show available placeholders
- Added comprehensive tests to verify placeholder functionality

This enhancement provides users with more control over how downloaded models are organized on their file system.
2025-10-17 16:01:06 +08:00
pixelpaws
52e671638b feat(example-images): add stop control for download panel 2025-10-15 08:46:03 +08:00
Will Miao
3d98572a62 feat: improve civitai data handling and type safety, fixes #565
- Replace setdefault with get and explicit dict initialization in MetadataUpdater
- Change civitai field type from Optional[Dict] to Dict[str, Any] with default_factory
- Add None check and dict initialization in BaseModelMetadata.__post_init__
- Ensures civitai data is always a dictionary, preventing type errors and improving code reliability
2025-10-14 16:03:33 +08:00
pixelpaws
7be1a2bd65 Merge pull request #557 from willmiao/civarc-api-support
CivArchive API support
2025-10-11 18:10:06 +08:00
Will Miao
1dc189eb39 feat(metadata): implement fallback provider strategy for deleted models
Refactor metadata sync service to use a prioritized provider fallback system when handling deleted CivitAI models. The new approach:

1. Attempts civarchive_api provider first for deleted models
2. Falls back to sqlite provider if archive DB is enabled
3. Maintains existing default provider behavior for non-deleted models
4. Tracks provider attempts and errors for better debugging

This improves reliability when fetching metadata for deleted models by trying multiple sources before giving up, and provides clearer error messages based on which providers were attempted.
2025-10-11 17:44:38 +08:00
pixelpaws
6120922204 chore(priority-tags): add newline terminator 2025-10-11 17:38:20 +08:00
Will Miao
e275968553 feat(civitai): remove debug print statement from rewrite_preview_url function 2025-10-10 08:18:23 +08:00
Will Miao
c9a65c7347 feat(metadata): implement model data hydration and enhance metadata handling across services, fixes #547 2025-10-09 22:15:07 +08:00
Will Miao
f542ade628 feat(civitai): implement URL rewriting for Civitai previews and enhance download handling, fixes #499 2025-10-09 17:54:37 +08:00
pixelpaws
8cf762ffd3 feat(misc): add civitai user model lookup 2025-10-09 11:49:41 +08:00
pixelpaws
a232997a79 fix(utils): respect metadata sync overrides 2025-10-08 15:52:15 +08:00