Commit Graph

933 Commits

Author SHA1 Message Date
Will Miao
c9e5ea42cb Fix null-safety issues and apply code formatting
Bug fixes:
- Add null guards for base_models_roots/embeddings_roots in backup cleanup
- Fix null-safety initialization of extra_unet_roots

Formatting:
- Apply consistent code style across Python files
- Fix line wrapping, quote consistency, and trailing commas
- Add type ignore comments for dynamic/platform-specific code
2026-02-28 21:38:41 +08:00
Will Miao
78b55d10ba refactor: move supporters loading to separate API endpoint
- Add SupportersHandler in misc_handlers.py to serve /api/lm/supporters
- Register new endpoint in misc_route_registrar.py
- Remove supporters from page load template context in model_handlers.py
- Create supportersService.js for frontend data fetching
- Update Header.js to fetch supporters when support modal opens
- Modify support_modal.html to use client-side rendering

This change improves page load performance by loading supporters data
on-demand instead of during initial page render.
2026-02-28 20:14:20 +08:00
Will Miao
77a2215e62 Fix lazy hash calculation for checkpoints in extra paths
- Allow empty sha256 when hash_status is 'pending' in cache entry validator
- Add on-demand hash calculation during bulk metadata refresh for checkpoints
  with pending hash status
- Add comprehensive tests for both fixes

Fixes issue where checkpoints in extra paths were not visible in UI and
not processed during bulk metadata refresh due to empty sha256.
2026-02-27 19:19:16 +08:00
pixelpaws
31901f1f0e Merge pull request #829 from willmiao/feature/lazy-hash-checkpoints
feat: lazy hash calculation for checkpoints
2026-02-27 11:02:39 +08:00
Will Miao
12a789ef96 fix(extra-folder-paths): fix extra folder paths support for checkpoint and unet roots
- Fix config.py: save and restore main paths when processing extra folder paths to prevent
  _prepare_checkpoint_paths from overwriting checkpoints_roots and unet_roots
- Fix lora_manager.py: apply library settings during initialization to load extra folder paths
  in ComfyUI plugin mode
- Fix checkpoint_routes.py: merge checkpoints/unet roots with extra paths in API endpoints
- Add logging for extra folder paths

Fixes issue where extra folder paths were not recognized for checkpoints and unet models.
2026-02-27 10:37:15 +08:00
Will Miao
d50bbe71c2 fix(extra-folder-paths): fix extra folder paths support for checkpoint and unet roots
- Fix config.py: save and restore main paths when processing extra folder paths to prevent
  _prepare_checkpoint_paths from overwriting checkpoints_roots and unet_roots
- Fix lora_manager.py: apply library settings during initialization to load extra folder paths
  in ComfyUI plugin mode
- Fix checkpoint_routes.py: merge checkpoints/unet roots with extra paths in API endpoints
- Add logging for extra folder paths

Fixes issue where extra folder paths were not recognized for checkpoints and unet models.
2026-02-27 10:27:29 +08:00
Will Miao
40d9f8d0aa feat: lazy hash calculation for checkpoints
Checkpoints are typically large (10GB+). This change delays SHA256
hash calculation until metadata fetch from Civitai is requested,
significantly improving initial scan performance.

- Add hash_status field to BaseModelMetadata
- CheckpointScanner skips hash during initial scan
- On-demand hash calculation during Civitai fetch
- Background bulk hash calculation support
2026-02-26 22:41:44 +08:00
Will Miao
87b462192b feat: Add extra folder paths support for LoRA Manager
Introduce extra_folder_paths feature to allow users to add additional
model roots that are managed by LoRA Manager but not shared with ComfyUI.

Changes:
- Add extra_folder_paths support in SettingsManager (stored per library)
- Add extra path attributes in Config class (extra_loras_roots, etc.)
- Merge folder_paths with extra_folder_paths when applying library settings
- Update LoraScanner, CheckpointScanner, EmbeddingScanner to include
  extra paths in their model roots
- Add comprehensive tests for the new functionality

This enables users to manage models from additional directories without
modifying ComfyUI's model folder configuration.
2026-02-25 18:16:17 +08:00
Will Miao
ede97f3f3e Fix calculate_recipe_fingerprint to handle non-string hash and invalid strength values
- Handle non-string hash values by converting to string before lower()
- Add try-except for strength conversion to handle invalid values like empty strings
- Fixes hypothesis test failures when random data generates unexpected types
2026-02-25 00:11:38 +08:00
Will Miao
fc98c752dc Fix Windows FileNotFoundError when loading LoRAs from lora_stack
lora_stack stores relative paths (e.g., 'Illustrious/style/file.safetensors'),
but comfy.utils.load_torch_file requires absolute paths. Previously, when
loading LoRAs from lora_stack, the relative path was passed directly to the
low-level API, causing FileNotFoundError on Windows.

This fix extracts the lora name from the relative path and uses
get_lora_info_absolute() to resolve the full absolute path before passing
it to load_torch_file(). This maintains compatibility with the lora_stack
format while ensuring correct file loading across all platforms.

Fixes: FileNotFoundError for relative paths in LoraLoaderLM and LoraTextLoaderLM
when processing lora_stack input.
2026-02-25 00:01:41 +08:00
Will Miao
70398ed985 feat(lora-loader): Load LoRAs using lower-level API to bypass folder_paths validation
- Add get_lora_info_absolute() function to return absolute file paths
- Replace LoraLoader().load_lora() with comfy.utils.load_torch_file() +
  comfy.sd.load_lora_for_models() to enable loading LoRAs from any path
- This allows LoRA Manager to load LoRAs from non-standard paths (multi-library support)
- Fixes #805
2026-02-23 18:06:15 +08:00
pixelpaws
7a04cec08d Merge pull request #825 from RanKaze/main
feat: filter node with mode:0
2026-02-23 16:39:45 +08:00
Will Miao
26b139884c perf(usage-stats): prevent unnecessary writes when idle
- Add is_dirty flag to track if statistics have changed
- Only write stats file when data actually changes
- Add enable_usage_statistics setting in ComfyUI settings
- Skip backend requests when usage statistics is disabled
- Fix standalone mode compatibility for MetadataRegistry

Fixes #826
2026-02-23 14:00:00 +08:00
K1einB1ue
60324c1299 feat: filter node with mode:0 2026-02-22 07:19:08 +08:00
Will Miao
773adb27c9 feat(model_download): add file_params JSON parsing to download handler
- Parse optional file_params query parameter as JSON in ModelDownloadHandler
- Add error handling for invalid JSON with warning log
- Maintain backward compatibility with existing download parameters
2026-02-22 04:26:38 +08:00
Will Miao
9117ee60dd feat(download): add file_params support for precise file selection 2026-02-20 15:32:48 +08:00
Will Miao
879588e252 refactor(settings): invert sync logic from whitelist to blacklist
Replace _SYNC_KEYS (37 keys) with _NO_SYNC_KEYS (5 keys) in SettingsHandler.
New settings automatically sync to frontend unless explicitly excluded.

Changes:
- SettingsHandler now syncs all settings except those in _NO_SYNC_KEYS
- Added keys() method to SettingsManager for iteration
- Updated tests to use new behavior

Benefits:
- No more missing keys when adding new settings
- Reduced maintenance burden
- Explicit exclusions for sensitive/internal settings only

Fixes: #86
2026-02-20 12:14:50 +08:00
Will Miao
67869f19ff feat(early-access): implement EA filtering and UI improvements
Add Early Access version support with filtering and improved UI:

Backend:
- Add is_early_access and early_access_ends_at fields to ModelVersionRecord
- Implement two-phase EA detection (bulk API + single API enrichment)
- Add hide_early_access_updates setting to filter EA updates
- Update has_update() and has_updates_bulk() to respect EA filter setting
- Add _enrich_early_access_details() for precise EA time fetching
- Fix setting propagation through base_model_service and model_update_service

Frontend:
- Add smart relative time display for EA (in Xh, in Xd, or date)
- Replace EA label with clock icon in metadata (fa-clock)
- Show Download button with bolt icon for EA versions (fa-bolt)
- Change EA badge color to #F59F00 (CivitAI Buzz theme)
- Fix toggle UI for hide_early_access_updates setting
- Add translation keys for EA time formatting

Tests:
- Update all tests to pass with new EA functionality
- Add test coverage for EA filtering logic

Closes #815
2026-02-20 10:32:51 +08:00
Will Miao
e8b37365a6 fix: Show all tags in LoRA Pool without limit (#819)
- Backend: Support limit=0 to return all tags in top-tags API
- Frontend: Remove tags limit setting and fetch all tags by default
- UI: Implement virtual scrolling in TagsModal for performance
  - Initial display 200 tags, load more on scroll
  - Show all results when searching
- Remove lora_pool_tags_limit setting to simplify UX

Fixes #819
2026-02-19 09:59:08 +08:00
Will Miao
b9516c6b62 fix: Handle missing Civitai API response fields gracefully
Fix KeyError when 'hashes', 'name', or 'model' fields are missing from
Civitai API responses. Use .get() with defaults instead of direct dict
access in:

- LoraMetadata.from_civitai_info()
- CheckpointMetadata.from_civitai_info()
- EmbeddingMetadata.from_civitai_info()
- RecipeScanner._get_hash_from_civitai()
- DownloadManager._process_download()

Fixes #820
2026-02-18 12:02:48 +08:00
Will Miao
16c52877ad feat: add dynamic trigger_words inputs to PromptLM node
- Backend: Add _AllContainer for dynamic input validation bypass
- Backend: Modify INPUT_TYPES to support trigger_words1, trigger_words2, etc.
- Backend: Update encode() to collect all trigger_words* from kwargs
- Frontend: Create prompt_dynamic_inputs.js extension
- Frontend: Implement onConnectionsChange to auto-add/remove input slots
- Frontend: Renumber inputs sequentially on connect/disconnect

Based on Impact Pack's Switch (Any) node dynamic input pattern.
2026-02-18 07:18:12 +08:00
Will Miao
6b1e3f06ed refactor(example-images): minimize async lock contention by moving I/O outside critical sections
- Extract progress file loading to async methods to run in executor
- Refactor start_download to reduce lock time by pre-loading data before entering lock
- Improve check_pending_models efficiency with single-pass model collection and async loading
- Add type hints to get_status method
- Add tests for download task callback execution and error handling
2026-02-11 09:24:00 +08:00
Will Miao
94edde7744 feat(settings): add metadata_refresh_skip_paths to sync keys for UI update 2026-02-09 10:09:53 +08:00
Will Miao
024dfff021 feat: add metadata refresh skip paths setting, #790 2026-02-09 09:56:19 +08:00
Will Miao
765c1c42a9 feat: enhance skip metadata refresh with smart UI and subtle badges, #790 2026-02-09 09:56:18 +08:00
Will Miao
b4ad03c9bf fix: improve example image upload reliability and error handling, #804
- Sequential per-file upload to avoid client_max_size limits
- Add backend exception handler with proper 500 responses
- Increase standalone server upload limit to 256MB
- Add partial success localization support
2026-02-08 09:17:19 +08:00
Will Miao
fa3625ff72 feat(filter): add tag logic toggle (OR/AND) for include tags filtering
Add a segmented toggle in the Filter Panel to switch between 'Any' (OR)
and 'All' (AND) logic when filtering by multiple include tags.

Changes:
- Backend: Add tag_logic field to FilterCriteria and ModelFilterSet
- Backend: Parse tag_logic parameter in model handlers
- Frontend: Add segmented toggle UI in filter panel header
- Frontend: Add interaction logic and state management for tag logic
- Add translations for all supported languages
- Add comprehensive tests for the new feature

Closes #802
2026-02-05 22:36:30 +08:00
Will Miao
895d13dc96 feat(settings): clean up default values from settings.json
Add automatic cleanup of default values from settings.json to keep configuration files minimal and focused on user customizations. Introduces a threshold-based cleanup that only removes default values when the file contains a significant number of them (10+), preserving small template-based configurations while cleaning up legacy bloated files.

Key changes:
- Add DEFAULT_KEYS_CLEANUP_THRESHOLD constant to control cleanup aggressiveness
- Implement _cleanup_default_values_from_disk() method that removes default values from disk while keeping them available in memory
- Modify _ensure_default_settings() to only save when existing values are updated, not when defaults are inserted
- Update _serialize_settings_for_disk() to only persist settings that differ from defaults
- Add cleanup call during initialization for existing settings files

This reduces file size and noise in settings.json while maintaining full functionality at runtime.
2026-02-05 08:40:27 +08:00
Will Miao
b7e0821f66 feat(duplicates): add filter support for duplicate model finding, #783 2026-02-04 20:47:30 +08:00
Will Miao
7bcf4e4491 feat(config): discover deep symlinks dynamically when accessing previews 2026-02-04 00:16:59 +08:00
Will Miao
c12aefa82a fix(recipes): detect duplicates for remote imports using modelVersionId and Civitai URL, #750
- Use modelVersionId as fallback for all loras in fingerprint calculation (not just deleted)
- Add URL-based duplicate detection using source_path field
- Combine both fingerprint and URL-based duplicate detection in API response
- Fix _download_remote_media return type and unbound variable issue
2026-02-03 21:32:15 +08:00
Will Miao
655d3cab71 fix(config): prioritize checkpoints over unet when paths overlap, #799
When checkpoints and unet folders point to the same physical location
(via symlinks), prioritize checkpoints for backward compatibility.

This prevents the 'Failed to load Checkpoint root' error that users
experience when they have incorrectly configured their ComfyUI paths.

Changes:
- Detect overlapping real paths between checkpoints and unet
- Log warning to inform users of the configuration issue
- Remove overlapping paths from unet_map, keeping checkpoints

Fixes #<issue-number>
2026-02-03 18:27:42 +08:00
Will Miao
358e658459 fix(trigger_word_toggle): add trigger word normalization method
Introduce a new private method `_normalize_trigger_words` to handle consistent splitting and cleaning of trigger word strings. This method splits input by both single and double commas, strips whitespace, and filters out empty strings, returning a set of normalized words. It is now used in `process_trigger_words` to compare trigger word overrides, ensuring accurate detection of changes by comparing normalized sets instead of raw strings.
2026-02-03 15:42:09 +08:00
Will Miao
f5dbd6b8e8 fix(metadata): auto-disable archive_db setting when database file is missing 2026-02-03 08:36:27 +08:00
Will Miao
2c026a2646 fix(metadata-sync): persist db_checked flag for deleted models
When a deleted model is checked against the SQLite archive and not found, the `db_checked` flag was set in memory but never saved to disk. This occurred because the save operation was only triggered when `civitai_api_not_found` was True, which is not the case for deleted models (since the CivitAI API is not attempted). As a result, deleted models would be rechecked on every refresh instead of being skipped.

Changes:
- Introduce a `needs_save` flag to track when metadata state is updated
- Save metadata whenever `db_checked` is set to True, regardless of API status
- Ensure `last_checked_at` is set for SQLite-only attempts
- Add regression test to verify the fix
2026-02-03 07:34:41 +08:00
Will Miao
820afe9319 feat(recipe_scanner): ensure cache initialization and improve type safety
- Initialize RecipeCache in scan_recipes to prevent None reference errors
- Import PersistedRecipeData directly instead of using string annotation
- Remove redundant import inside _reconcile_recipe_cache method
2026-02-02 21:57:44 +08:00
Will Miao
5a97f4bc75 feat(recipe_scanner): optimize recipe lookup performance
Refactor recipe lookup logic to improve efficiency from O(n²) to O(n + m):
- Build recipe_by_id dictionary for O(1) recipe ID lookups
- Simplify persisted_by_path construction using recipe_id extraction
- Add fallback lookup by recipe_id when path lookup fails
- Maintain same functionality while reducing computational complexity
2026-02-02 19:37:06 +08:00
Will Miao
94da404cc5 fix: skip confirmed not-found models in bulk metadata refresh
When enable_metadata_archive_db=True, the previous filter logic would
repeatedly try to fetch metadata for models that were already confirmed
to not exist on CivitAI (from_civitai=False, civitai_deleted=True).

The fix adds a skip condition to exclude models that:
1. Are confirmed not from CivitAI (from_civitai=False)
2. Are marked as deleted/not found on CivitAI (civitai_deleted=True)
3. Either have no archive DB enabled, or have already been checked (db_checked=True)

This prevents unnecessary API calls to CivArchive for user-trained models
or models from non-CivitAI sources.

Fixes repeated "Error fetching version of CivArchive model by hash" logs
for models that will never be found on CivitAI/CivArchive.
2026-02-02 13:27:18 +08:00
Will Miao
1da476d858 feat(example-images): add check pending models endpoint and improve async handling
- Add /api/example-images/check-pending endpoint to quickly check models needing downloads
- Improve DownloadManager.start_download() to return immediately without blocking
- Add _handle_download_task_done callback for proper error handling and progress saving
- Add check_pending_models() method for lightweight pre-download validation
- Update frontend ExampleImagesManager to use new check-pending endpoint
- Add comprehensive tests for new functionality
2026-02-02 12:31:07 +08:00
Will Miao
e252e44403 refactor(logging): replace print statements with logger for consistency 2026-02-02 10:47:17 +08:00
Will Miao
778ad8abd2 feat(cache): add cache health monitoring and validation system, see #730
- Add cache entry validator service for data integrity checks
- Add cache health monitor service for periodic health checks
- Enhance model cache and scanner with validation support
- Update websocket manager for health status broadcasting
- Add initialization banner service for cache health alerts
- Add comprehensive test coverage for new services
- Update translations across all locales
- Refactor sync translation keys script
2026-02-02 08:30:59 +08:00
Will Miao
db1aec94e5 refactor(logging): replace print statements with logger in metadata_collector 2026-02-01 15:41:41 +08:00
Will Miao
553e1868e1 perf(config): limit symlink scan to first level for faster startup
Replace recursive directory traversal with first-level-only symlink scanning
to fix severe performance issues on large model collections (220K+ files).

- Rename _scan_directory_links to _scan_first_level_symlinks
- Only scan symlinks directly under each root directory
- Skip traversal of normal subdirectories entirely
- Update tests to reflect first-level behavior
- Add test_deep_symlink_not_scanned to document intentional limitation

Startup time reduced from 15+ minutes to seconds for affected users.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-01 12:42:35 +08:00
Will Miao
c0f03b79a8 feat(settings): change model card footer action default to replace_preview 2026-02-01 07:38:04 +08:00
Will Miao
e17d6c8ebf feat(testing): enhance test configuration and add Vue component tests
- Update package.json test script to run both JS and Vue tests
- Simplify LoraCyclerLM output by removing redundant lora name fallback
- Extend Vitest config to include TypeScript test files
- Add Vue testing dependencies and setup for component testing
- Implement comprehensive test suite for BatchQueueSimulator component
- Add test setup file with global mocks for ComfyUI modules
2026-02-01 00:59:50 +08:00
Will Miao
ffcfe5ea3e fix(metadata): rename model_type to sub_type and add embedding subtype, see #797
- Change `model_type` field to `sub_type` for checkpoint models to improve naming consistency
- Add `sub_type="embedding"` for embedding models to properly categorize model subtypes
- Maintain backward compatibility with existing metadata structure
2026-01-31 22:54:53 +08:00
Will Miao
719e18adb6 feat(media): add media type hint support for file extension detection, fixes #795 and fixes #751
- Add optional `media_type_hint` parameter to `_get_file_extension_from_content_or_headers` method
- When `media_type_hint` is "video" and no extension can be determined from content/headers/URL, default to `.mp4`
- Pass image metadata type as hint in both `process_example_images` and `process_example_images_batch` methods
- Add unit tests to verify media type hint behavior and priority
2026-01-31 19:39:37 +08:00
Will Miao
84c62f2954 refactor(model-type): complete phase 5 cleanup by removing deprecated model_type field
- Remove backward compatibility code for `model_type` in `ModelScanner._build_cache_entry()`
- Update `CheckpointScanner` to only handle `sub_type` in `adjust_metadata()` and `adjust_cached_entry()`
- Delete deprecated aliases `resolve_civitai_model_type` and `normalize_civitai_model_type` from `model_query.py`
- Update frontend components (`RecipeModal.js`, `ModelCard.js`, etc.) to use `sub_type` instead of `model_type`
- Update API response format to return only `sub_type`, removing `model_type` from service responses
- Revise technical documentation to mark Phase 5 as completed and remove outdated TODO items

All cleanup tasks for the model type refactoring are now complete, ensuring consistent use of `sub_type` across the codebase.
2026-01-30 07:48:31 +08:00
Will Miao
5e91073476 refactor: unify model_type semantics by introducing sub_type field
This commit resolves the semantic confusion around the model_type field by
clearly distinguishing between:
- scanner_type: architecture-level (lora/checkpoint/embedding)
- sub_type: business-level subtype (lora/locon/dora/checkpoint/diffusion_model/embedding)

Backend Changes:
- Rename model_type to sub_type in CheckpointMetadata and EmbeddingMetadata
- Add resolve_sub_type() and normalize_sub_type() in model_query.py
- Update checkpoint_scanner to use _resolve_sub_type()
- Update service format_response to include both sub_type and model_type
- Add VALID_*_SUB_TYPES constants with backward compatible aliases

Frontend Changes:
- Add MODEL_SUBTYPE_DISPLAY_NAMES constants
- Keep MODEL_TYPE_DISPLAY_NAMES as backward compatible alias

Testing:
- Add 43 new tests covering sub_type resolution and API response

Documentation:
- Add refactoring todo document to docs/technical/

BREAKING CHANGE: None - full backward compatibility maintained
2026-01-30 06:56:10 +08:00
Will Miao
08267cdb48 refactor(filter): extract preset management logic into FilterPresetManager
Move filter preset creation, deletion, application, and storage logic
from FilterManager into a dedicated FilterPresetManager class to
improve separation of concerns and maintainability.

- Add FilterPresetManager with preset CRUD operations
- Update FilterManager to use preset manager via composition
- Handle EMPTY_WILDCARD_MARKER for wildcard base model filters
- Add preset-related translations to all locale files
- Update filter preset UI styling and interactions
2026-01-29 16:32:41 +08:00