Refactor force_refresh path to use thread pool execution instead of blocking
the event loop shared with ComfyUI. Key changes:
- Fix 1: Route force_refresh through _initialize_recipe_cache_sync() in thread pool
- Fix 2: Add GIL release points (time.sleep(0)) every 100 files in sync loops
- Fix 3: Move RecipeCache.resort() to thread pool via run_in_executor
- Fix 4: Persist cache automatically after force_refresh
- Fix 5: Increase yield frequency in _enrich_cache_metadata (every recipe)
This eliminates the ~5 minute freeze when rebuilding 30K recipe cache.
Fixes performance issue where ComfyUI became unresponsive during recipe
scanning due to shared Python event loop blocking.
Fix KeyError when 'hashes', 'name', or 'model' fields are missing from
Civitai API responses. Use .get() with defaults instead of direct dict
access in:
- LoraMetadata.from_civitai_info()
- CheckpointMetadata.from_civitai_info()
- EmbeddingMetadata.from_civitai_info()
- RecipeScanner._get_hash_from_civitai()
- DownloadManager._process_download()
Fixes#820
- Use modelVersionId as fallback for all loras in fingerprint calculation (not just deleted)
- Add URL-based duplicate detection using source_path field
- Combine both fingerprint and URL-based duplicate detection in API response
- Fix _download_remote_media return type and unbound variable issue
Refactor recipe lookup logic to improve efficiency from O(n²) to O(n + m):
- Build recipe_by_id dictionary for O(1) recipe ID lookups
- Simplify persisted_by_path construction using recipe_id extraction
- Add fallback lookup by recipe_id when path lookup fails
- Maintain same functionality while reducing computational complexity
Introduce a new PersistentRecipeCache service that stores recipe metadata in an SQLite database to significantly reduce application startup time. The cache eliminates the need to walk directories and parse JSON files on each launch by persisting recipe data between sessions.
Key features:
- Thread-safe singleton implementation with library-specific instances
- Automatic schema initialization and migration support
- JSON serialization for complex recipe fields (LoRAs, checkpoints, generation parameters, tags)
- File system monitoring with mtime/size validation for cache invalidation
- Environment variable toggle (LORA_MANAGER_DISABLE_PERSISTENT_CACHE) for debugging
- Comprehensive test suite covering save/load cycles, cache invalidation, and edge cases
The cache improves user experience by enabling near-instantaneous recipe loading after the initial cache population, while maintaining data consistency through file change detection.
Fixes a critical bug in FTS query building where multi-word searches
with field restrictions incorrectly used OR between all word+field
combinations instead of requiring ALL words to match within at least
one field.
Example: searching "cute cat" in {title, tags} previously produced:
title:cute* OR title:cat* OR tags:cute* OR tags:cat*
Which matched recipes with ANY word in ANY field.
Now produces:
(title:cute* title:cat*) OR (tags:cute* tags:cat*)
Which requires ALL words to match within at least one field.
Also adds fallback to fuzzy search when FTS returns empty results,
improving search reliability.
Co-Authored-By: Claude <noreply@anthropic.com>
Introduce a new RecipeFTSIndex class that provides fast prefix-based search across recipe fields (title, tags, LoRA names/models, prompts) using SQLite's FTS5 extension. The implementation supports sub-100ms search times for large datasets (20k+ recipes) and includes asynchronous indexing, incremental updates, and comprehensive unit tests.
Updated image path resolution logic to prioritize local sibling images in the same directory as recipes. When a stored image path differs from a local sibling, the system now automatically updates the recipe file to use the local path and persists this repair. This improves reliability when recipe assets are moved or reorganized, ensuring images remain accessible even if original paths become invalid.
- Add GET /api/lm/recipes/roots endpoint to retrieve recipe root directories
- Add POST /api/lm/recipe/move endpoint to move recipes between directories
- Register new endpoints in route definitions
- Implement error handling for both new endpoints with proper status codes
- Enable recipe management operations for better file organization
- Add new API endpoints for folder operations: get_folders, get_folder_tree, and get_unified_folder_tree
- Extend recipe listing handler to support folder and recursive filtering parameters
- Register new folder-related routes in route definitions
- Enable users to organize and browse recipes using folder structures
Add _normalize_checkpoint_entry method to handle legacy checkpoint data formats (strings, tuples) by converting them to dictionaries. This prevents errors during enrichment when checkpoint data is not in the expected dictionary format. Invalid checkpoint entries are now removed instead of causing processing failures.
- Update get_paginated_data and get_recipe_by_id methods to use normalization
- Add test cases for legacy string and tuple checkpoint formats
- Ensure backward compatibility with existing checkpoint handling
Add _normalize_checkpoint_entry method to handle legacy and malformed checkpoint data by:
- Converting string entries to structured dict format
- Handling single-element lists/tuples recursively
- Dropping invalid entries with appropriate warnings
- Maintaining backward compatibility while improving data consistency
Add test case to verify string checkpoint conversion works correctly.
- Add CheckpointScanner dependency to RecipeScanner singleton
- Implement checkpoint enrichment in recipe data processing
- Add _enrich_checkpoint_entry method to enhance checkpoint metadata
- Update recipe formatting to include checkpoint information
- Extend get_instance, __new__, and __init__ methods to support checkpoint scanner
- Add _get_checkpoint_from_version_index method for cache lookup
This enables recipe scanner to handle checkpoint models alongside existing LoRA support, providing complete model metadata for recipes.
Add _normalize_preview_url method to ensure preview URLs are properly formatted for browser access. The method handles absolute paths by converting them to static URLs via config.get_preview_static_url, while preserving API paths and other valid URLs. This ensures consistent preview image display across different URL formats.
Update _enrich_lora_entry to apply URL normalization to preview URLs obtained from both hash-based lookups and version entries. Add comprehensive test coverage for absolute path normalization scenarios.
Add _get_lora_from_version_index method to fetch cached LoRA entries by modelVersionId when hash is unavailable. This improves LoRA enrichment by using version index as fallback when hash is missing, ensuring proper library status, file paths, and preview URLs are set even without hash values.
Update test suite to include version_index in stub cache and add test coverage for version-based lookup functionality.
- Update frontend tag filter to cycle through include/exclude/clear states
- Add backend support for tag_include and tag_exclude query parameters
- Maintain backward compatibility with legacy tag parameter
- Store tag states as dictionary with 'include'/'exclude' values
- Update test matrix documentation to reflect new tag behavior
The changes enable more granular tag filtering where users can now explicitly include or exclude specific tags, rather than just adding tags to a simple inclusion list. This provides better control over search results and improves the filtering user experience.
- Replaced direct usage of Civitai client with a fallback metadata provider across all recipe parsers.
- Updated metadata service to improve initialization and error handling.
- Enhanced download manager to utilize a downloader service for file operations.
- Improved recipe scanner to fetch model information through the new metadata provider.
- Updated utility functions to streamline image downloading and processing.
- Added comprehensive logging and error handling for better debugging and reliability.
- Introduced `get_default_metadata_provider()` for simplified access to the default provider.
- Ensured backward compatibility with existing APIs and workflows.
- Added support for filtering LoRAs by hash in both API and UI components.
- Implemented session storage management for custom filter states when navigating between recipes and LoRAs.
- Introduced a new button in the recipe modal to view associated LoRAs, enhancing user navigation.
- Updated CSS styles for new UI elements, including a custom filter indicator and LoRA view button.
- Refactored existing JavaScript components to streamline the handling of filter parameters and improve maintainability.
- Implemented a new API endpoint to fetch recipes associated with a specific Lora by its hash.
- Enhanced the recipe scanning logic to support filtering by Lora hash and bypassing other filters.
- Added a new method to retrieve a recipe by its ID with formatted metadata.
- Created a new RecipeTab component to display recipes in the Lora modal.
- Introduced session storage utilities for managing custom filter states.
- Updated the UI to include a custom filter indicator and loading/error states for recipes.
- Refactored existing recipe management logic to accommodate new features and improve maintainability.
- Changed warning logs to debug logs in CivitaiClient and RecipeScanner for better log granularity.
- Updated the mapper registration function name for clarity and adjusted related logging messages.
- Enhanced extension loading process to automatically register mappers from NODE_MAPPERS_EXT, improving modularity and maintainability.
- Introduced a new API endpoint to update recipe metadata, allowing users to modify recipe titles and tags.
- Enhanced RecipeModal to support inline editing of recipe titles and tags, improving user interaction.
- Updated RecipeCard to reflect changes in recipe metadata, ensuring consistency across the application.
- Improved error handling for metadata updates to provide clearer feedback to users.
- Introduced a new API endpoint to rename LoRA files, including validation and error handling for file paths and names.
- Updated the RecipeScanner to reflect changes in LoRA filenames across recipe files and cache.
- Enhanced the LoraModal UI to allow inline editing of file names with improved user interaction and validation.
- Added CSS styles for the editing interface to improve visual feedback during file name editing.
- Removed the custom async_timeout function and replaced it with direct usage of the initialization lock.
- Simplified the cache initialization process by eliminating the dependency on the lora scanner.
- Enhanced error handling during cache initialization to ensure a fallback to an empty cache on failure.
