Commit Graph

399 Commits

Author SHA1 Message Date
Will Miao
edbcca9bbd refactor: simplify symlink cache invalidation by removing background rescan and noise_mtime in favor of a root-path-only fingerprint. 2026-01-03 19:29:53 +08:00
Will Miao
f13f22c949 chore: Improve logging by adding SHA256 hash calculation timing and clarifying metadata creation messages. 2026-01-03 08:56:34 +08:00
Will Miao
ab85ba54a9 feat: Implement recipe repair cancellation with UI support and refactor LoadingManager to a singleton. 2026-01-02 20:03:27 +08:00
Will Miao
837c32c42f feat: implement task cancellation for model scanning and bulk operations 2026-01-02 18:48:28 +08:00
Will Miao
afa5533145 feat: reduce log verbosity by changing info logs to debug
Changed logging level from INFO to DEBUG for performance-related messages in model management service. This reduces noise in production logs while maintaining debugging capability for performance analysis.
2025-12-31 16:14:24 +08:00
Will Miao
102defe29c feat: add open settings location endpoint
- Add `open_settings_location` method to `FileSystemHandler` to open OS file explorer at settings file location
- Register new POST route `/api/lm/settings/open-location` for settings file access
- Inject `SettingsManager` dependency into `FileSystemHandler` constructor
- Add cross-platform support for Windows, macOS, and Linux file explorers
- Include error handling for missing settings files and system exceptions
2025-12-31 16:09:23 +08:00
Will Miao
8120716cd8 feat: enhance model move functionality with cache entry updates
- Return cache entry data from model move operations for immediate UI updates
- Add recalculate_type parameter to update_single_model_cache for proper type adjustment
- Propagate cache entry through API layer to frontend MoveManager
- Enable virtual scroller to update moved items with new cache data
2025-12-31 10:33:22 +08:00
Will Miao
a59c31bc06 fix: Synchronize aiohttp session creation and refresh with an asyncio lock to prevent race conditions. 2025-12-30 19:47:30 +08:00
Will Miao
5d5a2a998a feat: Implement model move, import, and download functionalities with corresponding UI and API updates. 2025-12-28 21:18:27 +08:00
Will Miao
d9ce2c56c0 feat: Add 'times used' translation keys and implement usage-based sorting in the model service. 2025-12-26 22:39:51 +08:00
pixelpaws
1329294981 Merge branch 'sort-by-usage-count' into main 2025-12-26 22:17:03 +08:00
Will Miao
475906a25e perf: Add detailed performance logging to model retrieval, filtering, and sorting operations. see #711 2025-12-25 22:36:24 +08:00
Will Miao
91cd88f1df feat: Add recursive root folder scanning with API and UI updates. fixes #737 2025-12-25 21:07:52 +08:00
Will Miao
a552f07448 feat: Refactor checkpoint metadata to use Civitai API naming conventions and remove gen_params checkpoint syncing. 2025-12-24 20:25:39 +08:00
Will Miao
6486107ca2 feat: Introduce recipe management with data models, scanning, enrichment, and repair for generation configurations. 2025-12-24 20:02:20 +08:00
Will Miao
6330c65d41 feat: Add recipe metadata repair functionality with UI, API, and progress tracking. 2025-12-23 21:50:58 +08:00
Will Miao
00e6904664 feat: Introduce "No tags" filter option for models and recipes. fixes #728 2025-12-23 18:48:35 +08:00
Will Miao
b044b329fc feat: Update recipes page with default descending date sort, refactor state properties for search/filters, and add new localization strings. 2025-12-23 11:57:25 +08:00
Will Miao
bc9dd317f7 feat: Add prompt search filter for recipes and fix 'Favorites' localization across multiple languages. 2025-12-23 10:52:12 +08:00
Will Miao
db7f09797b feat: Introduce recipe favoriting with star icon toggle and filter options. 2025-12-23 10:07:09 +08:00
Will Miao
6e64f97e2b feat: add bulk move recipes endpoint
Add new move_recipes_bulk endpoint to handle moving multiple recipes simultaneously. This improves efficiency when reorganizing recipe collections by allowing batch operations instead of individual moves.

- Add move_recipes_bulk handler method with proper error handling
- Register new POST /api/lm/recipes/move-bulk route
- Implement bulk move logic in persistence service
- Validate required parameters (recipe_ids and target_path)
- Handle common error cases including validation, not found, and server errors
2025-12-23 09:15:07 +08:00
Will Miao
3f646aa0c9 feat: add recipe root directory and move recipe endpoints
- Add GET /api/lm/recipes/roots endpoint to retrieve recipe root directories
- Add POST /api/lm/recipe/move endpoint to move recipes between directories
- Register new endpoints in route definitions
- Implement error handling for both new endpoints with proper status codes
- Enable recipe management operations for better file organization
2025-12-23 09:13:57 +08:00
Will Miao
67fb205b43 feat: add folder-based recipe organization and navigation
- Add new API endpoints for folder operations: get_folders, get_folder_tree, and get_unified_folder_tree
- Extend recipe listing handler to support folder and recursive filtering parameters
- Register new folder-related routes in route definitions
- Enable users to organize and browse recipes using folder structures
2025-12-23 09:12:27 +08:00
Will Miao
3ba5c4c2ab refactor: improve update_lora_filename_by_hash logic and add a test to verify recipe updates. 2025-12-22 17:58:04 +08:00
Will Miao
7caca0163e feat: Add support for remote video analysis and preview for recipe imports. see #420 2025-12-21 21:42:28 +08:00
Will Miao
30fd0470de feat: Add support for video recipe previews by conditionally optimizing media during persistence and updating UI components to display videos. 2025-12-21 20:00:44 +08:00
Will Miao
ca6bb43406 feat: remove path separator normalization for cross-platform compatibility
Removed the forced normalization of path separators to forward slashes in BaseModelService to maintain platform-specific separators. Updated test cases to use os.sep for constructing expected paths, ensuring tests work correctly across different operating systems while preserving native path representations.
2025-12-17 19:07:08 +08:00
Will Miao
099a71b2cc feat(config): seed root symlink mappings before deep scanning
Add `_seed_root_symlink_mappings` method to ensure symlinked root folders are recorded before deep scanning, preventing them from being missed during directory traversal. This ensures that root symlinks are properly captured in the path mappings.

Additionally, normalize separators in relative paths for cross-platform consistency in `BaseModelService`, and update tests to verify root symlinks are preserved in the cache.
2025-12-16 22:05:40 +08:00
Will Miao
3382d83aee feat: remove prewarm cache and improve recipe scanner initialization
- Remove prewarm_cache startup hook from BaseRecipeRoutes
- Add post-scan task management to RecipeScanner for proper cleanup
- Ensure LoRA scanner initialization completes before recipe enrichment
- Schedule post-scan enrichment after cache initialization
- Improve error handling and task cancellation during shutdown
2025-12-16 21:00:04 +08:00
stone9k
56143eb170 feat(usage_count): sorting by usage_count + usage_count on ModelCard 2025-12-12 16:39:24 +01:00
Will Miao
3fc72d6bc1 feat(config): replace symlink scanning with cached mapping system
- Add `get_settings_dir` import for cache directory resolution
- Replace `_scan_symbolic_links` and `_rebuild_preview_roots` with unified `_initialize_symlink_mappings` method
- Implement fingerprint-based cache validation using root mtimes, inodes, and noise-aware timestamps
- Add helper methods for path normalization, cache location, and symlink root aggregation
- Improve performance by avoiding redundant symlink traversal when directory structure is unchanged
2025-12-09 19:51:01 +08:00
Will Miao
5000478991 feat(download): support multiple model file extensions in archive extraction
- Add `_get_supported_extensions_for_type` method to return allowed extensions per model type
- Rename `_extract_safetensors_from_archive` to `_extract_model_files_from_archive` and extend to filter by allowed extensions
- Update error message to list supported extensions when archive contains no valid files
- Add test for extracting .pt embedding files from zip archives
2025-12-07 09:00:47 +08:00
Will Miao
f8b9fa9b20 fix(civitai): improve metadata parsing for nested structures, see #700
- Refactor metadata detection to handle nested "meta" objects
- Add support for lowercase "lora:" hash keys
- Extract metadata from nested "meta" field when present
- Update tests to verify nested metadata parsing
- Handle case-insensitive LORA hash detection

The changes ensure proper parsing of Civitai image metadata that may be wrapped in nested structures, improving compatibility with different API response formats.
2025-11-26 13:46:08 +08:00
Will Miao
9198a23ba9 feat: normalize and validate checkpoint entries before enrichment
Add _normalize_checkpoint_entry method to handle legacy checkpoint data formats (strings, tuples) by converting them to dictionaries. This prevents errors during enrichment when checkpoint data is not in the expected dictionary format. Invalid checkpoint entries are now removed instead of causing processing failures.

- Update get_paginated_data and get_recipe_by_id methods to use normalization
- Add test cases for legacy string and tuple checkpoint formats
- Ensure backward compatibility with existing checkpoint handling
2025-11-21 23:36:32 +08:00
Will Miao
02bac7edfb feat: normalize and validate checkpoint entries in recipes
Add _normalize_checkpoint_entry method to handle legacy and malformed checkpoint data by:
- Converting string entries to structured dict format
- Handling single-element lists/tuples recursively
- Dropping invalid entries with appropriate warnings
- Maintaining backward compatibility while improving data consistency

Add test case to verify string checkpoint conversion works correctly.
2025-11-21 23:00:02 +08:00
Will Miao
ea1d1a49c9 feat: enhance search with include/exclude tokens and improved sorting
- Add token parsing to support include/exclude search terms using "-" prefix
- Implement token-based matching logic for relative path searches
- Improve search result sorting by prioritizing prefix matches and match position
- Add frontend test for multi-token highlighting with exclusion support
2025-11-21 19:48:43 +08:00
Will Miao
1971881537 feat: add checkpoint scanner integration to recipe scanner
- Add CheckpointScanner dependency to RecipeScanner singleton
- Implement checkpoint enrichment in recipe data processing
- Add _enrich_checkpoint_entry method to enhance checkpoint metadata
- Update recipe formatting to include checkpoint information
- Extend get_instance, __new__, and __init__ methods to support checkpoint scanner
- Add _get_checkpoint_from_version_index method for cache lookup

This enables recipe scanner to handle checkpoint models alongside existing LoRA support, providing complete model metadata for recipes.
2025-11-21 15:36:54 +08:00
Will Miao
4eb46a8d3e feat: consolidate checkpoint metadata handling
- Extract checkpoint entry from multiple metadata locations using helper method
- Sanitize checkpoint metadata by removing transient/local-only fields
- Remove checkpoint duplication from generation parameters to store only at top level
- Update frontend to properly populate checkpoint metadata during import
- Add tests for new checkpoint handling functionality

This ensures consistent checkpoint metadata structure and prevents data duplication across different storage locations.
2025-11-21 14:55:45 +08:00
Will Miao
36f28b3c65 feat: normalize LoRA preview URLs for browser accessibility
Add _normalize_preview_url method to ensure preview URLs are properly formatted for browser access. The method handles absolute paths by converting them to static URLs via config.get_preview_static_url, while preserving API paths and other valid URLs. This ensures consistent preview image display across different URL formats.

Update _enrich_lora_entry to apply URL normalization to preview URLs obtained from both hash-based lookups and version entries. Add comprehensive test coverage for absolute path normalization scenarios.
2025-11-21 12:31:23 +08:00
Will Miao
e24621a0af feat(recipe-scanner): add version index fallback for LoRA enrichment
Add _get_lora_from_version_index method to fetch cached LoRA entries by modelVersionId when hash is unavailable. This improves LoRA enrichment by using version index as fallback when hash is missing, ensuring proper library status, file paths, and preview URLs are set even without hash values.

Update test suite to include version_index in stub cache and add test coverage for version-based lookup functionality.
2025-11-21 11:27:09 +08:00
Will Miao
7173a2b9d6 feat: add remote recipe import functionality
Add support for importing recipes from remote sources by:
- Adding import_remote_recipe endpoint to RecipeHandlerSet
- Injecting downloader_factory and civitai_client_getter dependencies
- Implementing image download and resource parsing logic
- Supporting Civitai resource payloads with checkpoints and LoRAs
- Adding required imports for regex and temporary file handling

This enables users to import recipes directly from external sources like Civitai without manual file downloads.
2025-11-21 11:12:58 +08:00
Will Miao
9952721e76 feat(downloads): support safetensors zips and previews 2025-11-20 19:41:31 +08:00
Will Miao
26e4895807 feat(auto-organize): improve exclusion handling and progress reporting
- Add auto_organize_exclusions to settings handler proxy keys
- Refactor model file service to handle exclusions relative to model roots
- Improve auto-organize progress reporting for empty operations
- Fix exclusion pattern matching to consider relative paths within model roots
- Ensure proper validation when no model roots are configured
- Add comprehensive cleanup reporting for empty auto-organize operations
2025-11-20 18:33:48 +08:00
pixelpaws
07721af87c feat(settings): add auto-organize exclusions 2025-11-20 16:08:32 +08:00
Will Miao
3849f7eef9 feat(metadata): improve civitai deletion detection logic
- Track when Civitai API returns "Model not found" for default provider
- Use dedicated flag instead of error string comparison for deletion detection
- Ensure archive-sourced models don't get marked as deleted
- Add test coverage for archive source deletion flag behavior
- Fix deletion flag logic to properly handle provider fallback scenarios
2025-11-19 19:16:40 +08:00
Will Miao
51b5261f40 fix(model): align rename extension detection 2025-11-19 11:20:09 +08:00
Will Miao
963f6b1383 fix(model): preserve original extension on rename 2025-11-19 11:08:08 +08:00
Will Miao
b75baa1d1a fix: support GGUF model deletion in model lifecycle service
- Add optional main_extension parameter to delete_model_artifacts function
- Extract file extension from model filename to handle different file types
- Update model scanner to pass file extension when deleting models
- Add test case for GGUF file deletion to ensure proper cleanup
- Maintain backward compatibility with existing safetensors models

This change allows the model lifecycle service to properly delete GGUF model files along with their associated metadata and preview files, expanding support beyond just safetensors format.
2025-11-19 10:36:03 +08:00
Will Miao
1427dc8e38 feat(model-lifecycle): integrate model update service for deletion sync
Add ModelUpdateService dependency to ModelLifecycleService to enable synchronization during model deletion. The service is now passed through BaseModelRoutes initialization and used in delete_model to trigger updates when a model is removed. This ensures external systems stay in sync with local model state changes.

Key changes:
- Inject update_service into ModelLifecycleService constructor
- Extract model ID from metadata during deletion
- Call update service sync method after successful deletion
- Add proper type hints and TYPE_CHECKING imports
2025-11-18 21:02:39 +08:00
Will Miao
e9d55fe146 feat(filters): add model type filter 2025-11-18 16:43:44 +08:00