Commit Graph

230 Commits

Author SHA1 Message Date
Will Miao
3ba5c4c2ab refactor: improve update_lora_filename_by_hash logic and add a test to verify recipe updates. 2025-12-22 17:58:04 +08:00
Will Miao
7caca0163e feat: Add support for remote video analysis and preview for recipe imports. see #420 2025-12-21 21:42:28 +08:00
Will Miao
30fd0470de feat: Add support for video recipe previews by conditionally optimizing media during persistence and updating UI components to display videos. 2025-12-21 20:00:44 +08:00
Will Miao
63b087fc80 feat: Implement cache busting for static assets, remove client-side version mismatch banner, and add project overview documentation. 2025-12-19 22:40:36 +08:00
Will Miao
c8a179488a feat(metadata): enhance primary sampler detection and workflow tracing
- Add support for `basic_pipe` nodes in metadata processor to handle pipeline nodes like FromBasicPipe
- Optimize `find_primary_checkpoint` by accepting optional `primary_sampler_id` to avoid redundant calculations
- Update `get_workflow_trace` to pass known primary sampler ID for improved efficiency
2025-12-18 22:30:41 +08:00
Will Miao
ca6bb43406 feat: remove path separator normalization for cross-platform compatibility
Removed the forced normalization of path separators to forward slashes in BaseModelService to maintain platform-specific separators. Updated test cases to use os.sep for constructing expected paths, ensuring tests work correctly across different operating systems while preserving native path representations.
2025-12-17 19:07:08 +08:00
Will Miao
a07720a3bf feat: Add model path tracing to accurately identify the primary checkpoint in workflows and include new tests. 2025-12-17 12:52:52 +08:00
Will Miao
099a71b2cc feat(config): seed root symlink mappings before deep scanning
Add `_seed_root_symlink_mappings` method to ensure symlinked root folders are recorded before deep scanning, preventing them from being missed during directory traversal. This ensures that root symlinks are properly captured in the path mappings.

Additionally, normalize separators in relative paths for cross-platform consistency in `BaseModelService`, and update tests to verify root symlinks are preserved in the cache.
2025-12-16 22:05:40 +08:00
Will Miao
3382d83aee feat: remove prewarm cache and improve recipe scanner initialization
- Remove prewarm_cache startup hook from BaseRecipeRoutes
- Add post-scan task management to RecipeScanner for proper cleanup
- Ensure LoRA scanner initialization completes before recipe enrichment
- Schedule post-scan enrichment after cache initialization
- Improve error handling and task cancellation during shutdown
2025-12-16 21:00:04 +08:00
Will Miao
2494fa19a6 feat(config): add background symlink rescan and simplify cache validation
- Added threading import and optional `_rescan_thread` for background operations
- Simplified `_load_symlink_cache` to only validate path mappings, removing fingerprint checks
- Updated `_initialize_symlink_mappings` to rebuild preview roots and schedule rescan when cache is loaded
- Added `_schedule_symlink_rescan` method to perform background validation of symlinks
- Cleared `_path_mappings` at start of `_scan_symbolic_links` to prevent stale entries
- Background rescan improves performance by deferring symlink validation after cache load
2025-12-15 18:46:23 +08:00
Will Miao
817de3a0ae test: improve vi.mock calls to preserve original module exports
Updated vi.mock calls in test files to use async importOriginal pattern, ensuring original module exports are preserved while mocking specific functions. This prevents unintended side effects and maintains better test isolation.
2025-12-11 18:27:55 +08:00
Will Miao
a6e23a7630 feat(example-images): add NSFW level setting endpoint
Add new POST endpoint `/api/lm/example-images/set-nsfw-level` to allow updating NSFW classification for individual example images. The endpoint supports both regular and custom images, validates required parameters, and updates the corresponding model metadata. This enables users to manually adjust NSFW ratings for better content filtering.
2025-12-09 20:37:16 +08:00
Will Miao
3fc72d6bc1 feat(config): replace symlink scanning with cached mapping system
- Add `get_settings_dir` import for cache directory resolution
- Replace `_scan_symbolic_links` and `_rebuild_preview_roots` with unified `_initialize_symlink_mappings` method
- Implement fingerprint-based cache validation using root mtimes, inodes, and noise-aware timestamps
- Add helper methods for path normalization, cache location, and symlink root aggregation
- Improve performance by avoiding redundant symlink traversal when directory structure is unchanged
2025-12-09 19:51:01 +08:00
Will Miao
a3a00bbeed feat(trigger_word_toggle): refactor trigger word filtering logic, fixes #718 and fixes #285
- Simplify and consolidate the logic for processing trigger words and groups
- Remove redundant code paths and improve maintainability
- Ensure consistent behavior between list and string trigger data inputs
- Preserve existing functionality for strength adjustment and group mode
2025-12-09 14:16:56 +08:00
Will Miao
74bfd397aa feat: add CSP middleware to allow remote media previews, fixes #710, see #715
Introduce `relax_csp_for_remote_media` middleware that modifies Content Security Policy headers to permit loading media from trusted external domains (Civitai and Genur). This is necessary for LoRA Manager UI previews when ComfyUI runs with `--disable-api-nodes`, which otherwise blocks remote images and videos. The middleware is inserted after ComfyUI's `block_external_middleware` to properly extend the restrictive CSP header.
2025-12-09 10:37:35 +08:00
Will Miao
5000478991 feat(download): support multiple model file extensions in archive extraction
- Add `_get_supported_extensions_for_type` method to return allowed extensions per model type
- Rename `_extract_safetensors_from_archive` to `_extract_model_files_from_archive` and extend to filter by allowed extensions
- Update error message to list supported extensions when archive contains no valid files
- Add test for extracting .pt embedding files from zip archives
2025-12-07 09:00:47 +08:00
Will Miao
22ee37b817 feat: parse aggregate commercial use values, see #708
Add support for parsing comma-separated and JSON-style commercial use permission values in both Python backend and JavaScript frontend. Implement helper functions to split aggregated values into individual permissions while preserving original values when no aggregation is detected.

Added comprehensive test coverage for the new parsing functionality to ensure correct handling of various input formats including strings, arrays, and iterable objects with aggregated commercial use values.
2025-11-30 17:18:28 +08:00
Will Miao
df93670598 feat: add checkpoint metadata to EXIF recipe data
Add support for storing checkpoint information in image EXIF metadata. The checkpoint data is simplified and includes fields like model ID, version, name, hash, and base model. This allows for better tracking of AI model checkpoints used in image generation workflows.
2025-11-29 08:46:38 +08:00
Will Miao
073fb3a94a feat(recipe-parser): enhance LoRA metadata with local file matching
Add comprehensive local file matching for LoRA entries in recipe metadata:
- Add modelVersionId-based lookup via new _get_lora_from_version_index method
- Extend LoRA entry with additional fields: existsLocally, inLibrary, localPath, thumbnailUrl, size
- Improve local file detection by checking both SHA256 hash and modelVersionId
- Set default thumbnail URL and size values for missing LoRA files
- Add proper typing with Optional imports for better code clarity

This provides more accurate local file status and metadata for LoRA entries in recipes.
2025-11-29 08:29:05 +08:00
Will Miao
53c4165d82 feat(parser): enhance model metadata extraction in Automatic1111 parser
- Add MODEL_NAME_PATTERN regex to extract model names from parameters
- Extract model hash from parsed hashes when available in metadata
- Add checkpoint model hash and name extraction from parameters section
- Implement checkpoint resource processing from Civitai metadata
- Improve model information completeness for better recipe tracking
2025-11-29 08:13:55 +08:00
Will Miao
2b2e4fefab feat(tests): restructure test HTML to nest elements under model modal
Refactor the test HTML structure to properly nest all model metadata elements within the model modal container. This improves test accuracy by matching the actual DOM structure used in the application, ensuring that element selection and event handling work correctly during testing.
2025-11-27 20:44:05 +08:00
pixelpaws
8a628f0bd0 Merge pull request #703 from willmiao/fix/showcase-listener-leaks
fix(showcase): tear down modal listeners
2025-11-27 20:09:45 +08:00
Will Miao
b67c8598d6 feat(metadata): clear stale cache entries when metadata is empty
Update metadata registry to remove cache entries when node metadata becomes empty instead of keeping stale data. This prevents accumulation of unused cache entries and ensures cache only contains valid metadata. Added test case to verify cache behavior when LoRA configurations are removed.
2025-11-27 20:04:38 +08:00
Will Miao
0254c9d0e9 fix(showcase): tear down modal listeners 2025-11-27 18:00:59 +08:00
Will Miao
f8b9fa9b20 fix(civitai): improve metadata parsing for nested structures, see #700
- Refactor metadata detection to handle nested "meta" objects
- Add support for lowercase "lora:" hash keys
- Extract metadata from nested "meta" field when present
- Update tests to verify nested metadata parsing
- Handle case-insensitive LORA hash detection

The changes ensure proper parsing of Civitai image metadata that may be wrapped in nested structures, improving compatibility with different API response formats.
2025-11-26 13:46:08 +08:00
Will Miao
9198a23ba9 feat: normalize and validate checkpoint entries before enrichment
Add _normalize_checkpoint_entry method to handle legacy checkpoint data formats (strings, tuples) by converting them to dictionaries. This prevents errors during enrichment when checkpoint data is not in the expected dictionary format. Invalid checkpoint entries are now removed instead of causing processing failures.

- Update get_paginated_data and get_recipe_by_id methods to use normalization
- Add test cases for legacy string and tuple checkpoint formats
- Ensure backward compatibility with existing checkpoint handling
2025-11-21 23:36:32 +08:00
Will Miao
02bac7edfb feat: normalize and validate checkpoint entries in recipes
Add _normalize_checkpoint_entry method to handle legacy and malformed checkpoint data by:
- Converting string entries to structured dict format
- Handling single-element lists/tuples recursively
- Dropping invalid entries with appropriate warnings
- Maintaining backward compatibility while improving data consistency

Add test case to verify string checkpoint conversion works correctly.
2025-11-21 23:00:02 +08:00
Will Miao
ea1d1a49c9 feat: enhance search with include/exclude tokens and improved sorting
- Add token parsing to support include/exclude search terms using "-" prefix
- Implement token-based matching logic for relative path searches
- Improve search result sorting by prioritizing prefix matches and match position
- Add frontend test for multi-token highlighting with exclusion support
2025-11-21 19:48:43 +08:00
Will Miao
9a789f8f08 feat: add checkpoint hash filtering and navigation
- Add checkpoint hash parameter parsing to backend routes
- Implement checkpoint hash filtering in frontend API client
- Add click navigation from recipe modal to checkpoints page
- Update checkpoint items to use pointer cursor for better UX

Checkpoint items in recipe modal are now clickable and will navigate to the checkpoints page with appropriate hash filtering applied. This improves user workflow when wanting to view checkpoint details from recipes.
2025-11-21 16:17:01 +08:00
Will Miao
4eb46a8d3e feat: consolidate checkpoint metadata handling
- Extract checkpoint entry from multiple metadata locations using helper method
- Sanitize checkpoint metadata by removing transient/local-only fields
- Remove checkpoint duplication from generation parameters to store only at top level
- Update frontend to properly populate checkpoint metadata during import
- Add tests for new checkpoint handling functionality

This ensures consistent checkpoint metadata structure and prevents data duplication across different storage locations.
2025-11-21 14:55:45 +08:00
Will Miao
36f28b3c65 feat: normalize LoRA preview URLs for browser accessibility
Add _normalize_preview_url method to ensure preview URLs are properly formatted for browser access. The method handles absolute paths by converting them to static URLs via config.get_preview_static_url, while preserving API paths and other valid URLs. This ensures consistent preview image display across different URL formats.

Update _enrich_lora_entry to apply URL normalization to preview URLs obtained from both hash-based lookups and version entries. Add comprehensive test coverage for absolute path normalization scenarios.
2025-11-21 12:31:23 +08:00
Will Miao
2452cc4df1 feat(recipes): resolve base model from checkpoint metadata
Add metadata service integration to automatically resolve base model information from checkpoint metadata during recipe import. This replaces the previous approach of relying solely on request parameters and provides more accurate base model information.

- Add _resolve_base_model_from_checkpoint method to fetch base model from metadata provider
- Update recipe import logic to use resolved base model when available
- Add comprehensive tests for base model resolution with fallback behavior
- Remove debug print statement from import parameters
2025-11-21 12:12:27 +08:00
Will Miao
e24621a0af feat(recipe-scanner): add version index fallback for LoRA enrichment
Add _get_lora_from_version_index method to fetch cached LoRA entries by modelVersionId when hash is unavailable. This improves LoRA enrichment by using version index as fallback when hash is missing, ensuring proper library status, file paths, and preview URLs are set even without hash values.

Update test suite to include version_index in stub cache and add test coverage for version-based lookup functionality.
2025-11-21 11:27:09 +08:00
Will Miao
7173a2b9d6 feat: add remote recipe import functionality
Add support for importing recipes from remote sources by:
- Adding import_remote_recipe endpoint to RecipeHandlerSet
- Injecting downloader_factory and civitai_client_getter dependencies
- Implementing image download and resource parsing logic
- Supporting Civitai resource payloads with checkpoints and LoRAs
- Adding required imports for regex and temporary file handling

This enables users to import recipes directly from external sources like Civitai without manual file downloads.
2025-11-21 11:12:58 +08:00
Will Miao
9952721e76 feat(downloads): support safetensors zips and previews 2025-11-20 19:41:31 +08:00
Will Miao
c533a8e7bf feat: enhance Civitai metadata handling and image URL processing
- Import rewrite_preview_url utility for optimized image URL handling
- Update thumbnail URL processing for both LoRA and checkpoint entries to use rewritten URLs
- Expand checkpoint metadata with modelId, file size, SHA256 hash, and file name
- Improve error handling and data validation for Civitai API responses
- Maintain backward compatibility with existing data structures
2025-11-20 16:31:48 +08:00
pixelpaws
07721af87c feat(settings): add auto-organize exclusions 2025-11-20 16:08:32 +08:00
Will Miao
3849f7eef9 feat(metadata): improve civitai deletion detection logic
- Track when Civitai API returns "Model not found" for default provider
- Use dedicated flag instead of error string comparison for deletion detection
- Ensure archive-sourced models don't get marked as deleted
- Add test coverage for archive source deletion flag behavior
- Fix deletion flag logic to properly handle provider fallback scenarios
2025-11-19 19:16:40 +08:00
Will Miao
51b5261f40 fix(model): align rename extension detection 2025-11-19 11:20:09 +08:00
Will Miao
963f6b1383 fix(model): preserve original extension on rename 2025-11-19 11:08:08 +08:00
Will Miao
b75baa1d1a fix: support GGUF model deletion in model lifecycle service
- Add optional main_extension parameter to delete_model_artifacts function
- Extract file extension from model filename to handle different file types
- Update model scanner to pass file extension when deleting models
- Add test case for GGUF file deletion to ensure proper cleanup
- Maintain backward compatibility with existing safetensors models

This change allows the model lifecycle service to properly delete GGUF model files along with their associated metadata and preview files, expanding support beyond just safetensors format.
2025-11-19 10:36:03 +08:00
Will Miao
1427dc8e38 feat(model-lifecycle): integrate model update service for deletion sync
Add ModelUpdateService dependency to ModelLifecycleService to enable synchronization during model deletion. The service is now passed through BaseModelRoutes initialization and used in delete_model to trigger updates when a model is removed. This ensures external systems stay in sync with local model state changes.

Key changes:
- Inject update_service into ModelLifecycleService constructor
- Extract model ID from metadata during deletion
- Call update service sync method after successful deletion
- Add proper type hints and TYPE_CHECKING imports
2025-11-18 21:02:39 +08:00
Will Miao
e9d55fe146 feat(filters): add model type filter 2025-11-18 16:43:44 +08:00
Will Miao
57f369a6de feat(model): add model type filtering support
- Add model_types parameter to ModelListingHandler to support filtering by model type
- Implement get_model_types endpoint in ModelQueryHandler to retrieve available model types
- Register new /api/lm/{prefix}/model-types route for model type queries
- Extend BaseModelService to handle model type filtering in queries
- Support both model_type and civitai_model_type query parameters for backward compatibility

This enables users to filter models by specific types, improving model discovery and organization capabilities.
2025-11-18 15:36:01 +08:00
Will Miao
831a9da9d7 feat: update version badge logic for same-base update strategy, see #676
- Remove unused isNewer variable calculation
- Use dividerThresholdVersionId instead of latestLibraryVersionId for badge logic
- Add test case to verify newer version badge appears with same-base strategy
- Ensures correct badge display when filtering by same base model versions
2025-11-18 11:18:32 +08:00
Will Miao
655157434e feat(versions): add base filter toggle UI and styling
Add CSS classes and JavaScript logic for the base filter toggle button in the versions toolbar. The filter allows users to switch between showing all versions or only versions matching the current base model. Includes styling for different states (active, hover, disabled) and accessibility features like screen reader support.
2025-11-18 06:47:07 +08:00
Will Miao
0e73db0669 feat: implement same_base update strategy for model annotations
Add support for configurable update flag strategy with new "same_base" mode that considers base model versions when determining update availability. The strategy is controlled by the "update_flag_strategy" setting.

When strategy is set to "same_base":
- Uses get_records_bulk instead of has_updates_bulk
- Compares model versions against highest local versions per base model
- Provides more granular update detection based on base model relationships

Fallback to existing bulk or individual update checks when:
- Strategy is not "same_base"
- Bulk operations fail
- Records are unavailable

This enables more precise update flagging for models sharing common bases.
2025-11-17 19:26:41 +08:00
Will Miao
354cf03bbc fix(settings): sync portable mode toggle 2025-11-16 17:36:52 +08:00
pixelpaws
cc8e1c5049 fix(recipes): allow widget save without lora matches 2025-11-13 15:52:31 +08:00
Will Miao
29bb85359e feat(context-menu): refresh missing license metadata 2025-11-11 14:24:59 +08:00