Commit Graph

953 Commits

Author SHA1 Message Date
dogatech
8dd849892d Allow for empty lora (no loras option) in Lora Pool 2026-03-19 09:23:03 +08:00
Will Miao
fefcaa4a45 fix: improve Civitai recipe import by extracting EXIF when API metadata is empty
- Add validation to check if Civitai API metadata contains recipe fields
- Fall back to EXIF extraction when API returns empty metadata (meta.meta=null)
- Improve error messages to distinguish between missing metadata and unsupported format
- Add _has_recipe_fields() helper method to validate metadata content

This fixes import failures for Civitai images where the API returns
metadata wrapper but no actual generation parameters (e.g., images
edited in Photoshop that lost their original generation metadata)
2026-03-18 22:30:36 +08:00
Will Miao
701a6a6c44 refactor: remove GGUF loading logic from CheckpointLoaderLM
GGUF models are pure Unet models and should be handled by UNETLoaderLM.
2026-03-18 21:36:07 +08:00
Will Miao
0ef414d17e feat: standardize Checkpoint/Unet loader names and use OS-native path separators
- Rename nodes to 'Checkpoint Loader (LoraManager)' and 'Unet Loader (LoraManager)'\n- Use os.sep for relative path formatting in model COMBO inputs\n- Update path matching to be robust across OS separators\n- Update docstrings and comments
2026-03-18 21:33:19 +08:00
Will Miao
75dccaef87 test: fix cache validator tests to account for new hash_status field and side effects 2026-03-18 21:10:56 +08:00
Will Miao
7e87ec9521 fix: persist hash_status in model cache to support lazy hashing on restart 2026-03-18 21:07:40 +08:00
Will Miao
46522edb1b refactor: simplify GGUF import helper with dynamic path detection
- Add _get_gguf_path() to dynamically derive ComfyUI-GGUF path from current file location
- Remove Strategy 2 and 3, keeping only Strategy 1 (sys.modules path-based lookup)
- Remove hard-coded absolute paths
- Streamline logging output
- Code cleanup: reduced from 235 to 154 lines
2026-03-18 19:55:54 +08:00
Will Miao
2dae4c1291 fix: isolate extra unet paths from checkpoints to prevent type misclassification
Refactor _prepare_checkpoint_paths() to return a tuple instead of having
side effects on instance variables. This prevents extra unet paths from
being incorrectly classified as checkpoints when processing extra paths.

- Changed return type from List[str] to Tuple[List[str], List[str], List[str]]
  (all_paths, checkpoint_roots, unet_roots)
- Updated _init_checkpoint_paths() and _apply_library_paths() callers
- Fixed extra paths processing to properly isolate main and extra roots
- Updated test_checkpoint_path_overlap.py tests for new API

This ensures models in extra unet paths are correctly identified as
diffusion_model type and don't appear in checkpoints list.
2026-03-17 22:03:57 +08:00
Will Miao
70c150bd80 fix(services): implement stable sorting for model and recipe caches
Add file_path as a tie-breaker for all sort modes in ModelCache, BaseModelService, LoraService, and RecipeCache to ensure deterministic ordering when primary keys are identical. Resolves issue #859.
2026-03-17 14:20:23 +08:00
Will Miao
9e81c33f8a fix(utils): make sanitize_folder_name idempotent by combining strip/rstrip calls 2026-03-17 11:24:59 +08:00
Will Miao
b5a0725d2c fix(autocomplete): improve tag search ranking with popularity-based sorting
- Add LOG10(post_count) weighting to BM25 score for better relevance ranking
- Prioritize tag_name prefix matches above alias matches using CASE statement
- Remove frontend re-scoring logic to trust backend排序 results
- Fix pagination consistency: page N+1 scores <= page N minimum score

Key improvements:
- '1girl' (6M posts) now ranks #1 instead of #149 for search '1'
- tag_name prefix matches always appear before alias matches
- Popular tags rank higher than obscure ones with same prefix
- Consistent ordering across pagination boundaries

Test coverage:
- Add test_search_tag_name_prefix_match_priority
- Add test_search_ranks_popular_tags_higher
- Add test_search_pagination_ordering_consistency
- Add test_search_rank_score_includes_popularity_weight
- Update test data with 15 tags starting with '1'

Fixes issues with autocomplete dropdown showing inconsistent results
when scrolling through paginated search results.
2026-03-16 19:09:07 +08:00
Will Miao
ee466113d5 feat: implement batch import recipe functionality (frontend + backend fixes)
Backend fixes:
- Add missing API route for /api/lm/recipes/batch-import/progress (GET)
- Add missing API route for /api/lm/recipes/batch-import/directory (POST)
- Add missing API route for /api/lm/recipes/browse-directory (POST)
- Register WebSocket endpoint for batch import progress
- Fix skip_no_metadata default value (True -> False) to allow no-LoRA imports
- Add items array to BatchImportProgress.to_dict() for detailed results

Frontend implementation:
- Create BatchImportManager.js with complete batch import workflow
- Add directory browser UI for selecting folders
- Add batch import modal with URL list and directory input modes
- Implement real-time progress tracking (WebSocket + HTTP polling)
- Add results summary with success/failed/skipped statistics
- Add expandable details view showing individual item status
- Auto-refresh recipe list after import completion

UI improvements:
- Add spinner animation for importing status
- Simplify results summary UI to match progress stats styling
- Fix current item text alignment
- Fix dark theme styling for directory browser button
- Fix batch import button styling consistency

Translations:
- Add batch import related i18n keys to all locale files
- Run sync_translation_keys.py to sync all translations

Fixes:
- Batch import now allows images without LoRAs (matches single import behavior)
- Progress endpoint now returns complete items array with status details
- Results view correctly displays skipped items with error messages
2026-03-16 09:41:58 +08:00
Will Miao
f86651652c feat(batch-import): implement backend batch import service with adaptive concurrency
- Add BatchImportService with concurrent execution using asyncio.gather
- Implement AdaptiveConcurrencyController with dynamic adjustment
- Add input validation for URLs and local paths
- Support duplicate detection via skip_duplicates parameter
- Add WebSocket progress broadcasting for real-time updates
- Create comprehensive unit tests for batch import functionality
- Update API handlers and route registrations
- Add i18n translation keys for batch import UI
2026-03-16 09:41:58 +08:00
Will Miao
c89d4dae85 fix(extra-paths): support trigger words for LoRAs in extra folder paths, fixes #860
- Update get_lora_info() to check both loras_roots and extra_loras_roots
- Add fallback logic to return trigger words even if path not in recognized roots
- Ensure Trigger Word Toggle node displays trigger words for LoRAs from extra folder paths

Fixes issue where LoRAs added from extra folder paths would not show their trigger words in connected Trigger Word Toggle nodes.
2026-03-16 09:38:21 +08:00
Will Miao
cda271890a feat(workflow-template): add new tab template workflow with auto-zoom
- Add GET /api/lm/example-workflows endpoint to list available templates
- Add GET /api/lm/example-workflows/{filename} to retrieve specific workflow
- Add 'New Tab Template Workflow' setting in LoRA Manager settings
- Automatically apply 80% zoom level when loading template workflows
- Override workflow's saved view settings to prevent visual zoom flicker

The feature allows users to select a template workflow from example_workflows/
directory to load when creating new workflow tabs, with a hardcoded 0.8 zoom
level for better initial view experience.
2026-03-08 21:03:14 +08:00
Will Miao
43f6bfab36 fix(autocomplete): strip file extensions from model names in search suggestions
Remove .safetensors/.ckpt/.pt/.bin extensions from model names in autocomplete
suggestions to improve UX and search relevance:

Frontend (web/comfyui/autocomplete.js):
- Add _getDisplayText() helper to strip extensions from model paths
- Update _matchItem() to match against filename without extension
- Update render() and createItemElement() to display clean names

Backend (py/services/base_model_service.py):
- Add _remove_model_extension() helper method
- Update _relative_path_matches_tokens() to ignore extensions in matching
- Update _relative_path_sort_key() to sort based on names without extensions

Tests (tests/services/test_relative_path_search.py):
- Add tests to verify 's' and 'safe' queries don't match all .safetensors files

Fixes issue where typing 's' would match all .safetensors files and cluttered
suggestions with redundant extension names.
2026-03-07 23:07:10 +08:00
Will Miao
a802a89ff9 feat(autocomplete): implement virtual scrolling and pagination
- Add virtual scrolling with configurable visible items (default: 15)
- Implement pagination with offset/limit for backend APIs
- Support loading more items on scroll
- Fix width calculation for suggestions dropdown
- Update backend services to support offset parameter

Files modified:
- web/comfyui/autocomplete.js (virtual scroll, pagination)
- py/services/base_model_service.py (offset support)
- py/services/custom_words_service.py (offset support)
- py/services/tag_fts_index.py (offset support)
- py/routes/handlers/model_handlers.py (offset param)
- py/routes/handlers/misc_handlers.py (offset param)
2026-03-07 22:17:26 +08:00
Will Miao
4d8113464c perf(recipe_scanner): eliminate event loop blocking during cache rebuild
Refactor force_refresh path to use thread pool execution instead of blocking
the event loop shared with ComfyUI. Key changes:

- Fix 1: Route force_refresh through _initialize_recipe_cache_sync() in thread pool
- Fix 2: Add GIL release points (time.sleep(0)) every 100 files in sync loops
- Fix 3: Move RecipeCache.resort() to thread pool via run_in_executor
- Fix 4: Persist cache automatically after force_refresh
- Fix 5: Increase yield frequency in _enrich_cache_metadata (every recipe)

This eliminates the ~5 minute freeze when rebuilding 30K recipe cache.

Fixes performance issue where ComfyUI became unresponsive during recipe
scanning due to shared Python event loop blocking.
2026-03-04 15:10:46 +08:00
Will Miao
8c5d5a8ca0 feat(stats): implement infinite scrolling and paginated model usage lists (fixes #812)
- Add get_model_usage_list API endpoint for paginated stats
- Replace static rendering with client-side infinite scroll logic
- Add scrollbars and max-height to model usage lists
2026-03-03 15:00:01 +08:00
Will Miao
bde11b153f fix(preview): resolve CORS error when setting CivitAI remote media as preview
- Add new endpoint POST /api/lm/{prefix}/set-preview-from-url to handle
  remote image downloads server-side, avoiding CORS issues
- Use rewrite_preview_url() to download optimized smaller images (450px width)
- Use Downloader service for reliable downloads with retry logic and proxy support
- Update frontend to call new endpoint instead of fetching images in browser

fixes #837
2026-03-02 13:21:18 +08:00
Will Miao
c9e5ea42cb Fix null-safety issues and apply code formatting
Bug fixes:
- Add null guards for base_models_roots/embeddings_roots in backup cleanup
- Fix null-safety initialization of extra_unet_roots

Formatting:
- Apply consistent code style across Python files
- Fix line wrapping, quote consistency, and trailing commas
- Add type ignore comments for dynamic/platform-specific code
2026-02-28 21:38:41 +08:00
Will Miao
78b55d10ba refactor: move supporters loading to separate API endpoint
- Add SupportersHandler in misc_handlers.py to serve /api/lm/supporters
- Register new endpoint in misc_route_registrar.py
- Remove supporters from page load template context in model_handlers.py
- Create supportersService.js for frontend data fetching
- Update Header.js to fetch supporters when support modal opens
- Modify support_modal.html to use client-side rendering

This change improves page load performance by loading supporters data
on-demand instead of during initial page render.
2026-02-28 20:14:20 +08:00
Will Miao
77a2215e62 Fix lazy hash calculation for checkpoints in extra paths
- Allow empty sha256 when hash_status is 'pending' in cache entry validator
- Add on-demand hash calculation during bulk metadata refresh for checkpoints
  with pending hash status
- Add comprehensive tests for both fixes

Fixes issue where checkpoints in extra paths were not visible in UI and
not processed during bulk metadata refresh due to empty sha256.
2026-02-27 19:19:16 +08:00
pixelpaws
31901f1f0e Merge pull request #829 from willmiao/feature/lazy-hash-checkpoints
feat: lazy hash calculation for checkpoints
2026-02-27 11:02:39 +08:00
Will Miao
12a789ef96 fix(extra-folder-paths): fix extra folder paths support for checkpoint and unet roots
- Fix config.py: save and restore main paths when processing extra folder paths to prevent
  _prepare_checkpoint_paths from overwriting checkpoints_roots and unet_roots
- Fix lora_manager.py: apply library settings during initialization to load extra folder paths
  in ComfyUI plugin mode
- Fix checkpoint_routes.py: merge checkpoints/unet roots with extra paths in API endpoints
- Add logging for extra folder paths

Fixes issue where extra folder paths were not recognized for checkpoints and unet models.
2026-02-27 10:37:15 +08:00
Will Miao
d50bbe71c2 fix(extra-folder-paths): fix extra folder paths support for checkpoint and unet roots
- Fix config.py: save and restore main paths when processing extra folder paths to prevent
  _prepare_checkpoint_paths from overwriting checkpoints_roots and unet_roots
- Fix lora_manager.py: apply library settings during initialization to load extra folder paths
  in ComfyUI plugin mode
- Fix checkpoint_routes.py: merge checkpoints/unet roots with extra paths in API endpoints
- Add logging for extra folder paths

Fixes issue where extra folder paths were not recognized for checkpoints and unet models.
2026-02-27 10:27:29 +08:00
Will Miao
40d9f8d0aa feat: lazy hash calculation for checkpoints
Checkpoints are typically large (10GB+). This change delays SHA256
hash calculation until metadata fetch from Civitai is requested,
significantly improving initial scan performance.

- Add hash_status field to BaseModelMetadata
- CheckpointScanner skips hash during initial scan
- On-demand hash calculation during Civitai fetch
- Background bulk hash calculation support
2026-02-26 22:41:44 +08:00
Will Miao
87b462192b feat: Add extra folder paths support for LoRA Manager
Introduce extra_folder_paths feature to allow users to add additional
model roots that are managed by LoRA Manager but not shared with ComfyUI.

Changes:
- Add extra_folder_paths support in SettingsManager (stored per library)
- Add extra path attributes in Config class (extra_loras_roots, etc.)
- Merge folder_paths with extra_folder_paths when applying library settings
- Update LoraScanner, CheckpointScanner, EmbeddingScanner to include
  extra paths in their model roots
- Add comprehensive tests for the new functionality

This enables users to manage models from additional directories without
modifying ComfyUI's model folder configuration.
2026-02-25 18:16:17 +08:00
Will Miao
ede97f3f3e Fix calculate_recipe_fingerprint to handle non-string hash and invalid strength values
- Handle non-string hash values by converting to string before lower()
- Add try-except for strength conversion to handle invalid values like empty strings
- Fixes hypothesis test failures when random data generates unexpected types
2026-02-25 00:11:38 +08:00
Will Miao
fc98c752dc Fix Windows FileNotFoundError when loading LoRAs from lora_stack
lora_stack stores relative paths (e.g., 'Illustrious/style/file.safetensors'),
but comfy.utils.load_torch_file requires absolute paths. Previously, when
loading LoRAs from lora_stack, the relative path was passed directly to the
low-level API, causing FileNotFoundError on Windows.

This fix extracts the lora name from the relative path and uses
get_lora_info_absolute() to resolve the full absolute path before passing
it to load_torch_file(). This maintains compatibility with the lora_stack
format while ensuring correct file loading across all platforms.

Fixes: FileNotFoundError for relative paths in LoraLoaderLM and LoraTextLoaderLM
when processing lora_stack input.
2026-02-25 00:01:41 +08:00
Will Miao
70398ed985 feat(lora-loader): Load LoRAs using lower-level API to bypass folder_paths validation
- Add get_lora_info_absolute() function to return absolute file paths
- Replace LoraLoader().load_lora() with comfy.utils.load_torch_file() +
  comfy.sd.load_lora_for_models() to enable loading LoRAs from any path
- This allows LoRA Manager to load LoRAs from non-standard paths (multi-library support)
- Fixes #805
2026-02-23 18:06:15 +08:00
pixelpaws
7a04cec08d Merge pull request #825 from RanKaze/main
feat: filter node with mode:0
2026-02-23 16:39:45 +08:00
Will Miao
26b139884c perf(usage-stats): prevent unnecessary writes when idle
- Add is_dirty flag to track if statistics have changed
- Only write stats file when data actually changes
- Add enable_usage_statistics setting in ComfyUI settings
- Skip backend requests when usage statistics is disabled
- Fix standalone mode compatibility for MetadataRegistry

Fixes #826
2026-02-23 14:00:00 +08:00
K1einB1ue
60324c1299 feat: filter node with mode:0 2026-02-22 07:19:08 +08:00
Will Miao
773adb27c9 feat(model_download): add file_params JSON parsing to download handler
- Parse optional file_params query parameter as JSON in ModelDownloadHandler
- Add error handling for invalid JSON with warning log
- Maintain backward compatibility with existing download parameters
2026-02-22 04:26:38 +08:00
Will Miao
9117ee60dd feat(download): add file_params support for precise file selection 2026-02-20 15:32:48 +08:00
Will Miao
879588e252 refactor(settings): invert sync logic from whitelist to blacklist
Replace _SYNC_KEYS (37 keys) with _NO_SYNC_KEYS (5 keys) in SettingsHandler.
New settings automatically sync to frontend unless explicitly excluded.

Changes:
- SettingsHandler now syncs all settings except those in _NO_SYNC_KEYS
- Added keys() method to SettingsManager for iteration
- Updated tests to use new behavior

Benefits:
- No more missing keys when adding new settings
- Reduced maintenance burden
- Explicit exclusions for sensitive/internal settings only

Fixes: #86
2026-02-20 12:14:50 +08:00
Will Miao
67869f19ff feat(early-access): implement EA filtering and UI improvements
Add Early Access version support with filtering and improved UI:

Backend:
- Add is_early_access and early_access_ends_at fields to ModelVersionRecord
- Implement two-phase EA detection (bulk API + single API enrichment)
- Add hide_early_access_updates setting to filter EA updates
- Update has_update() and has_updates_bulk() to respect EA filter setting
- Add _enrich_early_access_details() for precise EA time fetching
- Fix setting propagation through base_model_service and model_update_service

Frontend:
- Add smart relative time display for EA (in Xh, in Xd, or date)
- Replace EA label with clock icon in metadata (fa-clock)
- Show Download button with bolt icon for EA versions (fa-bolt)
- Change EA badge color to #F59F00 (CivitAI Buzz theme)
- Fix toggle UI for hide_early_access_updates setting
- Add translation keys for EA time formatting

Tests:
- Update all tests to pass with new EA functionality
- Add test coverage for EA filtering logic

Closes #815
2026-02-20 10:32:51 +08:00
Will Miao
e8b37365a6 fix: Show all tags in LoRA Pool without limit (#819)
- Backend: Support limit=0 to return all tags in top-tags API
- Frontend: Remove tags limit setting and fetch all tags by default
- UI: Implement virtual scrolling in TagsModal for performance
  - Initial display 200 tags, load more on scroll
  - Show all results when searching
- Remove lora_pool_tags_limit setting to simplify UX

Fixes #819
2026-02-19 09:59:08 +08:00
Will Miao
b9516c6b62 fix: Handle missing Civitai API response fields gracefully
Fix KeyError when 'hashes', 'name', or 'model' fields are missing from
Civitai API responses. Use .get() with defaults instead of direct dict
access in:

- LoraMetadata.from_civitai_info()
- CheckpointMetadata.from_civitai_info()
- EmbeddingMetadata.from_civitai_info()
- RecipeScanner._get_hash_from_civitai()
- DownloadManager._process_download()

Fixes #820
2026-02-18 12:02:48 +08:00
Will Miao
16c52877ad feat: add dynamic trigger_words inputs to PromptLM node
- Backend: Add _AllContainer for dynamic input validation bypass
- Backend: Modify INPUT_TYPES to support trigger_words1, trigger_words2, etc.
- Backend: Update encode() to collect all trigger_words* from kwargs
- Frontend: Create prompt_dynamic_inputs.js extension
- Frontend: Implement onConnectionsChange to auto-add/remove input slots
- Frontend: Renumber inputs sequentially on connect/disconnect

Based on Impact Pack's Switch (Any) node dynamic input pattern.
2026-02-18 07:18:12 +08:00
Will Miao
6b1e3f06ed refactor(example-images): minimize async lock contention by moving I/O outside critical sections
- Extract progress file loading to async methods to run in executor
- Refactor start_download to reduce lock time by pre-loading data before entering lock
- Improve check_pending_models efficiency with single-pass model collection and async loading
- Add type hints to get_status method
- Add tests for download task callback execution and error handling
2026-02-11 09:24:00 +08:00
Will Miao
94edde7744 feat(settings): add metadata_refresh_skip_paths to sync keys for UI update 2026-02-09 10:09:53 +08:00
Will Miao
024dfff021 feat: add metadata refresh skip paths setting, #790 2026-02-09 09:56:19 +08:00
Will Miao
765c1c42a9 feat: enhance skip metadata refresh with smart UI and subtle badges, #790 2026-02-09 09:56:18 +08:00
Will Miao
b4ad03c9bf fix: improve example image upload reliability and error handling, #804
- Sequential per-file upload to avoid client_max_size limits
- Add backend exception handler with proper 500 responses
- Increase standalone server upload limit to 256MB
- Add partial success localization support
2026-02-08 09:17:19 +08:00
Will Miao
fa3625ff72 feat(filter): add tag logic toggle (OR/AND) for include tags filtering
Add a segmented toggle in the Filter Panel to switch between 'Any' (OR)
and 'All' (AND) logic when filtering by multiple include tags.

Changes:
- Backend: Add tag_logic field to FilterCriteria and ModelFilterSet
- Backend: Parse tag_logic parameter in model handlers
- Frontend: Add segmented toggle UI in filter panel header
- Frontend: Add interaction logic and state management for tag logic
- Add translations for all supported languages
- Add comprehensive tests for the new feature

Closes #802
2026-02-05 22:36:30 +08:00
Will Miao
895d13dc96 feat(settings): clean up default values from settings.json
Add automatic cleanup of default values from settings.json to keep configuration files minimal and focused on user customizations. Introduces a threshold-based cleanup that only removes default values when the file contains a significant number of them (10+), preserving small template-based configurations while cleaning up legacy bloated files.

Key changes:
- Add DEFAULT_KEYS_CLEANUP_THRESHOLD constant to control cleanup aggressiveness
- Implement _cleanup_default_values_from_disk() method that removes default values from disk while keeping them available in memory
- Modify _ensure_default_settings() to only save when existing values are updated, not when defaults are inserted
- Update _serialize_settings_for_disk() to only persist settings that differ from defaults
- Add cleanup call during initialization for existing settings files

This reduces file size and noise in settings.json while maintaining full functionality at runtime.
2026-02-05 08:40:27 +08:00
Will Miao
b7e0821f66 feat(duplicates): add filter support for duplicate model finding, #783 2026-02-04 20:47:30 +08:00
Will Miao
7bcf4e4491 feat(config): discover deep symlinks dynamically when accessing previews 2026-02-04 00:16:59 +08:00