Commit Graph

29 Commits

Author SHA1 Message Date
Will Miao
5e91073476 refactor: unify model_type semantics by introducing sub_type field
This commit resolves the semantic confusion around the model_type field by
clearly distinguishing between:
- scanner_type: architecture-level (lora/checkpoint/embedding)
- sub_type: business-level subtype (lora/locon/dora/checkpoint/diffusion_model/embedding)

Backend Changes:
- Rename model_type to sub_type in CheckpointMetadata and EmbeddingMetadata
- Add resolve_sub_type() and normalize_sub_type() in model_query.py
- Update checkpoint_scanner to use _resolve_sub_type()
- Update service format_response to include both sub_type and model_type
- Add VALID_*_SUB_TYPES constants with backward compatible aliases

Frontend Changes:
- Add MODEL_SUBTYPE_DISPLAY_NAMES constants
- Keep MODEL_TYPE_DISPLAY_NAMES as backward compatible alias

Testing:
- Add 43 new tests covering sub_type resolution and API response

Documentation:
- Add refactoring todo document to docs/technical/

BREAKING CHANGE: None - full backward compatibility maintained
2026-01-30 06:56:10 +08:00
Will Miao
7b0c6c8bab refactor(cache): reorganize cache directory structure with automatic legacy cleanup
- Centralize cache path resolution in new py/utils/cache_paths.py module
- Migrate legacy cache files to organized structure: {settings_dir}/cache/{model|recipe|fts|symlink}/
- Automatically clean up legacy files after successful migration with integrity verification
- Update Config symlink cache to use new path and migrate from old location
- Simplify service classes (PersistentModelCache, PersistentRecipeCache, RecipeFTSIndex, TagFTSIndex) to use centralized migration logic
- Add comprehensive test coverage for cache paths and automatic cleanup
2026-01-26 16:12:08 +08:00
Will Miao
b0f0158f98 feat(example-images): add force parameter to retry failed downloads
When force=true is passed via API, models in failed_models set are
re-downloaded instead of being skipped. On successful download, model is
removed from failed_models set.

This provides a manual batch repair mechanism for users when CivitAI
media server is temporarily down and causes empty folders.

Changes:
- Backend: Add force parameter to start_download(), _download_all_example_images(), _process_model()
- Backend: Skip failed_models check when force=true
- Backend: Remove model from failed_models on successful force retry
- Frontend: GlobalContextMenu now calls API with force=true directly
- Tests: Update mock to accept force parameter
2026-01-18 21:58:12 +08:00
Will Miao
aab1797269 Revert "feat: add automatic retry for failed example image downloads"
This reverts commit cb460fcdb0.
2026-01-18 10:55:30 +08:00
Will Miao
cb460fcdb0 feat: add automatic retry for failed example image downloads
- Add failed_model_timestamps to track when models fail
- Retry failed models after 24-hour cooldown period
- Skip retry if example folder already has files
- Skip retry if failure was less than 24 hours ago
- Log count of failed models with retry message
- Fix unbound snapshot variable in exception path
- Remove duplicate/unreachable directory check code
- Update string quotes to double quotes (PEP 8)

This fixes the issue where failed models were permanently skipped in
auto-download mode, even when their example folders were empty.
2026-01-18 08:55:49 +08:00
Will Miao
22ee37b817 feat: parse aggregate commercial use values, see #708
Add support for parsing comma-separated and JSON-style commercial use permission values in both Python backend and JavaScript frontend. Implement helper functions to split aggregated values into individual permissions while preserving original values when no aggregation is detected.

Added comprehensive test coverage for the new parsing functionality to ensure correct handling of various input formats including strings, arrays, and iterable objects with aggregated commercial use values.
2025-11-30 17:18:28 +08:00
Will Miao
df93670598 feat: add checkpoint metadata to EXIF recipe data
Add support for storing checkpoint information in image EXIF metadata. The checkpoint data is simplified and includes fields like model ID, version, name, hash, and base model. This allows for better tracking of AI model checkpoints used in image generation workflows.
2025-11-29 08:46:38 +08:00
Will Miao
ba6e2eadba feat: update license flag handling and default permissions
- Update DEFAULT_LICENSE_FLAGS from 57 to 127 to enable all commercial modes by default
- Replace CommercialUseLevel enum with bitwise commercial permission handling
- Simplify commercial value normalization and validation using allowed values set
- Adjust bit shifting in license flag construction to accommodate new commercial bits structure
- Remove CommercialUseLevel from exports and update tests accordingly
- Improve handling of empty commercial use values with proper type checking

The changes streamline commercial permission processing and align with CivitAI's default license configuration while maintaining backward compatibility.
2025-11-06 22:14:36 +08:00
Will Miao
ddf9e33961 feat: add license information handling for Civitai models
Add license resolution utilities and integrate license information into model metadata processing. The changes include:

- Add `resolve_license_payload` function to extract license data from Civitai model responses
- Integrate license information into model metadata in CivitaiClient and MetadataSyncService
- Add license flags support in model scanning and caching
- Implement CommercialUseLevel enum for standardized license classification
- Update model scanner to handle unknown fields when extracting metadata values

This ensures proper license attribution and compliance when working with Civitai models.
2025-11-06 17:05:54 +08:00
Will Miao
3d6bb432c4 feat: normalize tags to lowercase for Windows compatibility, see #637
Convert all tags to lowercase in tag processing logic to prevent case sensitivity issues on Windows filesystems. This ensures consistent tag matching and prevents duplicate tags with different cases from being created.

Changes include:
- TagUpdateService now converts tags to lowercase before comparison
- Utils function converts model tags to lowercase before priority resolution
- Test cases updated to reflect lowercase tag expectations
2025-11-04 12:54:09 +08:00
pixelpaws
4753206c52 fix(example-images): accept legacy library folders 2025-10-27 18:21:43 +08:00
pixelpaws
807425f12a feat(preview): respect blur mature content setting 2025-10-25 06:43:03 +08:00
pixelpaws
f7a856349a fix(utils): sanitize path template folder names 2025-10-18 16:20:47 +08:00
Will Miao
10f5588e4a feat: add model_name and version_name placeholders to download paths, #552
Add support for {model_name} and {version_name} placeholders in download path templates. These new placeholders allow for more flexible and descriptive file organization by including the actual model name and version name in the download directory structure.

Changes include:
- Updated download_manager.py and utils.py to handle new placeholders
- Added placeholders to constants.js for UI reference
- Updated settings modal template to show available placeholders
- Added comprehensive tests to verify placeholder functionality

This enhancement provides users with more control over how downloaded models are organized on their file system.
2025-10-17 16:01:06 +08:00
pixelpaws
52e671638b feat(example-images): add stop control for download panel 2025-10-15 08:46:03 +08:00
Will Miao
3d98572a62 feat: improve civitai data handling and type safety, fixes #565
- Replace setdefault with get and explicit dict initialization in MetadataUpdater
- Change civitai field type from Optional[Dict] to Dict[str, Any] with default_factory
- Add None check and dict initialization in BaseModelMetadata.__post_init__
- Ensures civitai data is always a dictionary, preventing type errors and improving code reliability
2025-10-14 16:03:33 +08:00
Will Miao
c9a65c7347 feat(metadata): implement model data hydration and enhance metadata handling across services, fixes #547 2025-10-09 22:15:07 +08:00
pixelpaws
c53fa8692b refactor(settings): lazily initialize manager 2025-10-08 08:56:57 +08:00
pixelpaws
3539c453d3 test(utils): cover example images migrations 2025-10-05 22:19:29 +08:00
Will Miao
9ac4203b1c test(example-images): allow monkeypatching os.startfile on linux 2025-10-05 16:30:44 +08:00
Will Miao
413444500e refactor(model_scanner): normalize path comparisons for model roots
fix(example_images_download_manager): re-raise specific exception on download error

refactor(usage_stats): define constants locally to avoid conditional imports

test(example_images_download_manager): update exception handling in download tests

test(example_images_file_manager): differentiate between os.startfile and subprocess.Popen in tests

test(example_images_paths): ensure valid example images root with single-library mode

test(usage_stats): use string literals for metadata payload to avoid conditional imports
2025-10-05 15:48:50 +08:00
pixelpaws
e21d5835ec Merge pull request #524 from willmiao/codex/add-async-tests-for-websocketmanager
Add websocket broadcast and usage stats tests
2025-10-05 15:19:40 +08:00
pixelpaws
b195d4569c test(usage-stats): allow metadata registry monkeypatch 2025-10-05 15:13:20 +08:00
pixelpaws
1da0434454 Merge pull request #526 from willmiao/codex/add-tests-for-utility-functions
test: add coverage for utility helpers
2025-10-05 15:03:20 +08:00
pixelpaws
2d36b461cf test(utils): add coverage for helper utilities 2025-10-05 14:43:21 +08:00
pixelpaws
badae2e8b3 test: cover websocket broadcasts and usage stats 2025-10-05 14:41:47 +08:00
pixelpaws
fdec8d283c test(example-images): expand coverage for routes and utilities 2025-10-05 14:40:48 +08:00
pixelpaws
d43d992362 feat(example-images): namespace storage by library 2025-10-04 20:29:49 +08:00
pixelpaws
8e4927600f fix(embeddings): replace spaces in relative paths 2025-09-26 09:02:46 +08:00