Compare commits

...

241 Commits

Author SHA1 Message Date
Will Miao
0968698804 feat: add v0.9.8 release notes and update version 2025-10-15 19:47:13 +08:00
Will Miao
a5b2e9b0bf feat: add update service dependency and has_update filter
- Pass ModelUpdateService to CheckpointService, EmbeddingService, and LoraService constructors
- Add has_update query parameter filter to model listing handler
- Update BaseModelService to accept optional update_service parameter

These changes enable model update functionality across different model types and provide filtering capability for models with available updates.
2025-10-15 17:25:16 +08:00
pixelpaws
5a6ff444b9 Merge pull request #572 from willmiao/codex/design-ui-for-model-update-notifications
refactor: tighten civitai update endpoints
2025-10-15 16:01:20 +08:00
pixelpaws
3bb240d3c1 fix(updates): avoid caching failed civitai lookups 2025-10-15 16:00:23 +08:00
pixelpaws
ee0d241c75 refactor(routes): limit update endpoints to essentials 2025-10-15 15:37:35 +08:00
Will Miao
321ff72953 feat: remove delay from bulk auto-organize completion 2025-10-15 10:32:59 +08:00
Will Miao
412f1e62a1 feat(i18n): add model name display option and improve localization, fixes #440
- Add new model name display setting with options to show model name or file name
- Implement helper function to determine display name based on user preference
- Update model card footer to use dynamic display name
- Include model name display setting in settings modal and state management
- Remove redundant labels from display density descriptions in multiple locales
- Simplify card info display descriptions by removing duplicate text

The changes provide cleaner UI text and add flexibility for users to choose between displaying model names or file names in card footers.
2025-10-15 10:23:39 +08:00
Will Miao
8901b32a55 Merge branch 'main' of https://github.com/willmiao/ComfyUI-Lora-Manager 2025-10-15 09:19:05 +08:00
Will Miao
8ab6cc72ad feat: add project documentation in IFLOW.md 2025-10-15 09:18:59 +08:00
pixelpaws
52e671638b feat(example-images): add stop control for download panel 2025-10-15 08:46:03 +08:00
Will Miao
a3070f8d82 feat: add rate limit error handling to CivArchive client
- Add RateLimitError import and exception handling in API methods
- Create _make_request wrapper to surface rate limit errors from downloader
- Add test case to verify rate limit error propagation
- Set default provider as "civarchive_api" for rate limit errors

This ensures rate limit errors are properly propagated and handled throughout the CivArchive client, improving error reporting and allowing callers to implement appropriate retry logic.
2025-10-14 21:38:24 +08:00
Will Miao
3fde474583 feat(civitai): add rate limiting support and error handling
- Add RateLimitError import and _make_request wrapper method to handle rate limiting
- Update API methods to use _make_request wrapper instead of direct downloader calls
- Add explicit RateLimitError handling in API methods to properly propagate rate limit errors
- Add _extract_retry_after method to parse Retry-After headers
- Improve error handling by surfacing rate limit information to callers

These changes ensure that rate limiting from the Civitai API is properly detected and handled, allowing callers to implement appropriate backoff strategies when rate limits are encountered.
2025-10-14 21:38:24 +08:00
Will Miao
1454991d6d feat(i18n): update bulk action labels to reflect selected items
Change bulk action labels from "All" to "Selected" in both English and Chinese locales to accurately reflect that these actions apply only to selected items rather than all items. This improves user interface clarity and prevents potential confusion about the scope of bulk operations.
2025-10-14 21:36:11 +08:00
Will Miao
4398851bb9 feat: reorder and update context menu items
- Remove 'clear' action from context menu
- Reorder menu items to prioritize common operations
- Move destructive operations (delete, move) to bottom section
- Add visual separation between action groups
- Maintain all existing functionality with improved organization
2025-10-14 21:29:09 +08:00
Will Miao
5173aa6c20 feat(model-scanner): add logging for file processing, fixes #566 2025-10-14 19:44:59 +08:00
Will Miao
3d98572a62 feat: improve civitai data handling and type safety, fixes #565
- Replace setdefault with get and explicit dict initialization in MetadataUpdater
- Change civitai field type from Optional[Dict] to Dict[str, Any] with default_factory
- Add None check and dict initialization in BaseModelMetadata.__post_init__
- Ensures civitai data is always a dictionary, preventing type errors and improving code reliability
2025-10-14 16:03:33 +08:00
Will Miao
c48095d9c6 feat: replace IO type imports with string literals
Remove direct imports of IO type constants from comfy.comfy_types and replace them with string literals "STRING" in input type definitions and return types. This improves code portability and reduces dependency on external type definitions.

Changes made across multiple files:
- Remove `from comfy.comfy_types import IO` imports
- Replace `IO.STRING` with "STRING" in INPUT_TYPES and RETURN_TYPES
- Move CLIPTextEncode import to function scope in prompt.py for better dependency management

This refactor maintains the same functionality while making the code more self-contained and reducing external dependencies.
2025-10-14 09:12:55 +08:00
Will Miao
1e4d1b8f15 feat(nodes): add Promp (LoraManager) node and autocomplete support 2025-10-13 23:23:32 +08:00
pixelpaws
8c037465ba Merge pull request #564 from willmiao/codex/design-apis-for-pause-and-resume-download
test: add coverage for download pause and resume controls
2025-10-13 21:39:47 +08:00
pixelpaws
055c1ca0d4 test(downloads): cover pause and resume flows 2025-10-13 21:30:23 +08:00
Will Miao
27370df93a feat(download): add support to download models from civarchive, fixes #381 2025-10-13 19:27:56 +08:00
Will Miao
60d23aa238 feat(download): enhance download progress ui with transfer stats 2025-10-13 19:06:51 +08:00
pixelpaws
5e441d9c4f Merge pull request #563 from willmiao/codex/add-download-speed-info-to-progress
feat(downloads): expose throughput metrics in progress APIs
2025-10-13 18:11:32 +08:00
pixelpaws
eb76468280 feat(downloads): expose throughput metrics in progress APIs 2025-10-13 14:39:31 +08:00
Will Miao
01bbaa31a8 fix(ModelTags): fix performance and UX issues in ModelTags 2025-10-12 22:31:10 +08:00
Will Miao
bddf023dc4 Merge branch 'main' of https://github.com/willmiao/ComfyUI-Lora-Manager 2025-10-12 17:43:31 +08:00
Will Miao
8e69a247ed feat(i18n): Update priority tags translations for better localization 2025-10-12 17:43:26 +08:00
pixelpaws
97141b01e1 Merge pull request #562 from willmiao/incremental-cache, see #561
feat(model-scanner): add metadata tracking and improve cache management
2025-10-12 17:09:19 +08:00
Will Miao
acf610ddff feat(model-scanner): add metadata tracking and improve cache management
- Add metadata_source field to track origin of model metadata
- Define MODEL_COLUMNS constants for consistent column management
- Refactor SQL queries to use dynamic column selection
- Improve Civitai data detection to include creator_username and trained_words
- Update database operations to handle new metadata field and tag management
2025-10-12 16:54:39 +08:00
Will Miao
a9a6f66035 feat(api): enhance model API creation with validation and default fallback
Refactored `createModelApiClient` to pass the specific model type as a parameter to each client constructor. Introduced `isValidModelType` for validation and added logic to set a default model type if provided type is invalid or not specified. Updated `getModelApiClient` function to utilize these improvements, ensuring robust model API instantiation.
2025-10-12 15:28:30 +08:00
Will Miao
0040863a03 feat(tests): introduce ROUTE_CALLS_KEY for organizing route calls
Addressed the aiohttp warnings by aligning the test scaffolding with current best practices. Added an AppKey constant and stored the route tracking list through it to satisfy aiohttp’s NotAppKeyWarning expectations. Swapped the websocket lambdas for async no-op handlers so the registered routes now point to coroutine callables, clearing the deprecation warning about bare functions.
2025-10-12 09:12:57 +08:00
Will Miao
4ab86b4ae2 feat(locale): add drag drop error message in locales 2025-10-12 09:07:36 +08:00
Will Miao
b32b4b4042 feat: enhance model scanning to include creator username
Updated the `ModelScanner` class to extract and format the creator username from Civitai data. This enhancement ensures that the creator information is properly included in slim model data.
2025-10-12 08:51:42 +08:00
Will Miao
4e552dcf3e feat: Add drag-and-drop support with visual feedback for sidebar nodes
This commit implements drag-and-drop functionality for sidebar nodes,
adding visual feedback via highlight styling when dragging over
valid drop targets. The CSS introduces new classes to style
drop indicators using the lora-accent color scheme, while the JS
adds event handlers to manage drag operations and update the UI
state accordingly. This improves user interaction by providing
clear visual cues for valid drop areas during file operations.
2025-10-12 06:55:01 +08:00
Will Miao
8f4c02efdc Merge branch 'main' of https://github.com/willmiao/ComfyUI-Lora-Manager 2025-10-12 05:44:07 +08:00
Will Miao
b77c596f3a Fix error message to improve clarity in DownloadManager 2025-10-12 05:43:59 +08:00
pixelpaws
181f0b5626 Merge pull request #560 from willmiao/codex/analyze-coverage-for-backend-tests
test: add standalone bootstrap and model factory coverage
2025-10-11 22:48:35 +08:00
pixelpaws
480e5d966f test: add standalone bootstrap and model factory coverage 2025-10-11 22:42:24 +08:00
Will Miao
e8636b949d feat(ModelTags): Implemented drag-and-drop reordering for tag edit mode so users can rearrange tags directly in the UI, fixes #414 2025-10-11 20:56:22 +08:00
pixelpaws
8ea369db47 Merge pull request #559 from willmiao/codex/add-info-level-logging-in-fetch_and_update_model
feat: log metadata channel on metadata fetch
2025-10-11 20:37:08 +08:00
Will Miao
ec9b37eb53 feat: add model type context to tag suggestions
- Pass modelType parameter to setupTagEditMode function
- Implement model type aware priority tag suggestions
- Add model type normalization and resolution logic
- Handle suggestion state reset when model type changes
- Maintain backward compatibility with existing functionality

The changes enable context-aware tag suggestions based on model type, improving tag relevance and user experience when editing tags for different model types.
2025-10-11 20:36:38 +08:00
Will Miao
b0847f6b87 feat(doc): update priority tags configuration guide wiki 2025-10-11 20:08:42 +08:00
pixelpaws
84d10b1f3b feat(metadata): log metadata channel on fetch 2025-10-11 20:07:01 +08:00
Will Miao
4fdc97d062 Merge branch 'main' of https://github.com/willmiao/ComfyUI-Lora-Manager 2025-10-11 19:43:41 +08:00
Will Miao
5fe5e7ea54 feat(ui): enhance settings modal styling and add priority tags tabs
- Rename `.settings-open-location-button` to `.settings-action-link` for better semantic meaning
- Add enhanced hover/focus states with accent colors and border transitions
- Implement tabbed interface for priority tags with LoRA, checkpoint, and embedding sections
- Improve input styling with consistent error states and example code formatting
- Remove deprecated grid layout in favor of tab-based organization
- Add responsive tab navigation with proper focus management and visual feedback
2025-10-11 19:43:22 +08:00
pixelpaws
7be1a2bd65 Merge pull request #557 from willmiao/civarc-api-support
CivArchive API support
2025-10-11 18:10:06 +08:00
pixelpaws
87842385c6 Merge pull request #558 from willmiao/codex/design-custom-priority-tags-format
feat: add customizable priority tags
2025-10-11 17:46:13 +08:00
Will Miao
1dc189eb39 feat(metadata): implement fallback provider strategy for deleted models
Refactor metadata sync service to use a prioritized provider fallback system when handling deleted CivitAI models. The new approach:

1. Attempts civarchive_api provider first for deleted models
2. Falls back to sqlite provider if archive DB is enabled
3. Maintains existing default provider behavior for non-deleted models
4. Tracks provider attempts and errors for better debugging

This improves reliability when fetching metadata for deleted models by trying multiple sources before giving up, and provides clearer error messages based on which providers were attempted.
2025-10-11 17:44:38 +08:00
pixelpaws
6120922204 chore(priority-tags): add newline terminator 2025-10-11 17:38:20 +08:00
Will Miao
ddb30dbb17 Revert "feat(civarchive_client): update get_model_version_info to resolve the real model/version IDs before fetching the target metadata."
This reverts commit c3a66ecf28.
2025-10-11 16:11:17 +08:00
Will Miao
1e8bd88e28 feat(metadata): improve model ID redirect logic and provider ordering
- Fix CivArchive model ID redirect logic to only follow redirects when context points to original model
- Rename CivitaiModelMetadataProvider to CivArchiveModelMetadataProvider for consistency
- Reorder fallback metadata providers to prioritize Civitai API over CivArchive API for better metadata quality
- Remove unused asyncio import and redundant logging from metadata sync service
2025-10-11 16:11:13 +08:00
Will Miao
c3a66ecf28 feat(civarchive_client): update get_model_version_info to resolve the real model/version IDs before fetching the target metadata. 2025-10-11 15:07:42 +08:00
Will Miao
1f60160e8b feat(civarchive_client): enhance request handling and context parsing
Introduce `_request_json` method for async JSON requests and improved error handling. Add static methods `_normalize_payload`, `_split_context`, `_ensure_list`, and `_build_model_info` to parse and normalize API responses. These changes improve the robustness of the CivArchiveClient by ensuring consistent data structures and handling potential API response issues gracefully.
2025-10-11 13:07:29 +08:00
Will Miao
7d560bf07a chore: add refs 2025-10-11 12:59:13 +08:00
Will Miao
47da9949d9 feat: update recipe download URL path to include /lm prefix
Update the download URL path in RecipeSharingService to include '/lm' prefix,
aligning with the new API route structure for recipe sharing endpoints.
2025-10-10 21:46:56 +08:00
scruffynerf
68c0a5ba71 Better Civ Archive support (adds API) (#549)
* add CivArchive API

* Oops, missed committing this part when I updated codebase to latest version

* Adjust API for version fetching and solve the broken API (hash gives only files, not models - likely to be fixed but in the meantime...)

* add asyncio import to allow timeout cooldown

---------

Co-authored-by: Scruffy Nerf <Scruffynerf@duck.com>
2025-10-10 20:04:01 +08:00
pixelpaws
1aa81c803b Merge pull request #551 from willmiao/codex/refactor-model-metadata-saving-logic
fix: hydrate cached metadata before persisting updates
2025-10-10 08:56:12 +08:00
pixelpaws
8f5e134d3e fix: skip redundant hydration in metadata sync service 2025-10-10 08:49:54 +08:00
pixelpaws
ef03a2a917 fix(metadata): hydrate cached records before saving 2025-10-10 08:30:51 +08:00
Will Miao
e275968553 feat(civitai): remove debug print statement from rewrite_preview_url function 2025-10-10 08:18:23 +08:00
Will Miao
76d3aa2b5b feat(version): bump version to 0.9.7 in pyproject.toml 2025-10-09 22:15:50 +08:00
Will Miao
c9a65c7347 feat(metadata): implement model data hydration and enhance metadata handling across services, fixes #547 2025-10-09 22:15:07 +08:00
Will Miao
f542ade628 feat(civitai): implement URL rewriting for Civitai previews and enhance download handling, fixes #499 2025-10-09 17:54:37 +08:00
Will Miao
d2c2bfbe6a feat(sidebar): add recursive search functionality and toggle button 2025-10-09 17:07:10 +08:00
Will Miao
2b6910bd55 feat(misc): mark model versions in library for Civitai user models 2025-10-09 15:23:42 +08:00
Will Miao
b1dd733493 feat(civitai): enhance model version handling with cache lookup 2025-10-09 14:10:00 +08:00
pixelpaws
5dcf0a1e48 Merge pull request #545 from willmiao/codex/evaluate-sqlite-cache-indexing-necessity
feat: index cached models by version id
2025-10-09 13:54:46 +08:00
pixelpaws
cf357b57fc feat(scanner): index cached models by version id 2025-10-09 13:50:44 +08:00
pixelpaws
4e1773833f Merge pull request #544 from willmiao/codex/add-endpoint-to-fetch-civitai-user-models
Add endpoint to fetch Civitai user models
2025-10-09 11:56:57 +08:00
pixelpaws
8cf762ffd3 feat(misc): add civitai user model lookup 2025-10-09 11:49:41 +08:00
pixelpaws
d997eaa429 Merge pull request #543 from willmiao/codex/refactor-get_model_version-logic-and-add-tests, fixes #540
fix: improve Civitai model version retrieval
2025-10-09 11:07:33 +08:00
pixelpaws
8e51f0f19f fix(civitai): improve model version retrieval 2025-10-09 10:56:25 +08:00
pixelpaws
f0e246b4ac Merge pull request #542 from willmiao/codex/investigate-backend-tests-modifying-settings.json
Refactor settings manager to lazy singleton
2025-10-08 16:02:11 +08:00
pixelpaws
a232997a79 fix(utils): respect metadata sync overrides 2025-10-08 15:52:15 +08:00
pixelpaws
08a449db99 fix(metadata): refresh metadata sync settings 2025-10-08 10:38:05 +08:00
pixelpaws
0c023c9888 fix(settings): lazily resolve module aliases 2025-10-08 10:10:23 +08:00
pixelpaws
0ad92d00b3 fix(settings): restore legacy settings aliases 2025-10-08 09:54:36 +08:00
pixelpaws
a726cbea1e fix(routes): pass resolved settings to metadata sync 2025-10-08 09:32:57 +08:00
pixelpaws
c53fa8692b refactor(settings): lazily initialize manager 2025-10-08 08:56:57 +08:00
Will Miao
3118f3b43c feat(graph): enhance node handling with graph identifiers and improve metadata updates, see #408, #538 2025-10-07 23:22:38 +08:00
Will Miao
9199950b74 feat(release): update version to 0.9.6 and add release notes for critical performance optimizations and new features 2025-10-07 17:41:58 +08:00
Will Miao
4c7e31687b feat(recipe_scanner): reset cached state on active library change 2025-10-07 08:07:44 +08:00
Will Miao
75e207b520 fix(banner): remove redundant community support banner registration 2025-10-06 22:39:49 +08:00
Will Miao
631289b75e feat(banner): add community support banner with Ko-fi integration and translations 2025-10-06 22:39:21 +08:00
Will Miao
1b958d0a5d feat(modal): use CSS variables for header height and improve recipe modal layout 2025-10-06 16:28:56 +08:00
Will Miao
35fdf9020d docs(README): update instructions for settings.json file creation in Portable Edition 2025-10-06 15:32:37 +08:00
Will Miao
45926b1dca feat(constants): add CHROMA model to BASE_MODELS and update categories 2025-10-06 08:48:15 +08:00
Will Miao
686ba5024d fix(tests): add loadingManager mocks in settingsManager tests 2025-10-06 08:24:58 +08:00
pixelpaws
cf375c7c86 Merge pull request #537 from willmiao/codex/implement-video-lazy-loading-with-queue
fix: throttle model card video lazy loading
2025-10-06 08:07:24 +08:00
pixelpaws
5e53d76f44 fix(model-card): throttle preview video loading 2025-10-06 07:45:51 +08:00
Will Miao
7757f72859 Enhance test for saving paths to ensure cross-platform compatibility in folder paths 2025-10-05 22:40:43 +08:00
pixelpaws
c8cc584049 Merge pull request #536 from willmiao/codex/add-unit-tests-for-config-saving-paths
Add regression tests for Config save path handling
2025-10-05 22:24:57 +08:00
pixelpaws
2cdd269bba Merge pull request #535 from willmiao/codex/add-tests-for-migration-utility-functions
test: cover example images migration flows
2025-10-05 22:24:42 +08:00
pixelpaws
d2d97ae5bb Merge pull request #534 from willmiao/codex/add-tests-for-cache-middleware
test: add cache control middleware coverage
2025-10-05 22:24:26 +08:00
pixelpaws
d08d77c555 Merge pull request #533 from willmiao/codex/add-async-test-module-for-lifecycle
test: add LoRA manager lifecycle coverage
2025-10-05 22:24:12 +08:00
pixelpaws
92f8d2139a Merge pull request #532 from willmiao/codex/create-tests-for-statsroutes
test: add coverage for stats routes endpoints
2025-10-05 22:23:59 +08:00
pixelpaws
50f2c2dfe6 test(config): cover save path migration flows 2025-10-05 22:19:36 +08:00
pixelpaws
3539c453d3 test(utils): cover example images migrations 2025-10-05 22:19:29 +08:00
pixelpaws
1631122f95 test(middleware): add cache control coverage 2025-10-05 22:19:20 +08:00
pixelpaws
8fcb979544 test(routes): cover lora manager lifecycle 2025-10-05 22:19:10 +08:00
pixelpaws
8a5af0b7f3 test(routes): add stats routes coverage 2025-10-05 22:18:59 +08:00
Will Miao
cb1f08d556 Fix low contrast on nav-item hover 2025-10-05 21:10:01 +08:00
pixelpaws
1150267765 Merge pull request #531 from willmiao/codex/-activelibrary
fix(locales): translate library selection strings
2025-10-05 21:04:10 +08:00
pixelpaws
5c1252548d fix(locales): translate library selection strings 2025-10-05 21:03:21 +08:00
Will Miao
3c7cdf5db8 feat(header): enhance navigation and search functionality with context-aware behavior 2025-10-05 20:58:14 +08:00
Will Miao
9ac4203b1c test(example-images): allow monkeypatching os.startfile on linux 2025-10-05 16:30:44 +08:00
pixelpaws
d0800510db Merge pull request #529 from willmiao/codex/update-file-url-formatting-methods
fix: route recipe previews through preview API
2025-10-05 15:53:26 +08:00
pixelpaws
f8ba551cc4 fix(recipes): use preview endpoint for recipe images 2025-10-05 15:49:18 +08:00
Will Miao
413444500e refactor(model_scanner): normalize path comparisons for model roots
fix(example_images_download_manager): re-raise specific exception on download error

refactor(usage_stats): define constants locally to avoid conditional imports

test(example_images_download_manager): update exception handling in download tests

test(example_images_file_manager): differentiate between os.startfile and subprocess.Popen in tests

test(example_images_paths): ensure valid example images root with single-library mode

test(usage_stats): use string literals for metadata payload to avoid conditional imports
2025-10-05 15:48:50 +08:00
pixelpaws
e21d5835ec Merge pull request #524 from willmiao/codex/add-async-tests-for-websocketmanager
Add websocket broadcast and usage stats tests
2025-10-05 15:19:40 +08:00
pixelpaws
f2f354e478 Merge pull request #528 from willmiao/codex/add-bulk-action-set-content-rating, fixes #428
Add bulk content rating update action
2025-10-05 15:19:19 +08:00
pixelpaws
b195d4569c test(usage-stats): allow metadata registry monkeypatch 2025-10-05 15:13:20 +08:00
pixelpaws
3b77fed72d feat(bulk): add bulk content rating action 2025-10-05 15:09:43 +08:00
pixelpaws
fc64e97f92 Merge pull request #525 from willmiao/codex/develop-tests-for-metadatasyncservice-and-modelscanner
Add coverage for metadata sync service and scanner reconciliation
2025-10-05 15:03:36 +08:00
pixelpaws
1da0434454 Merge pull request #526 from willmiao/codex/add-tests-for-utility-functions
test: add coverage for utility helpers
2025-10-05 15:03:20 +08:00
pixelpaws
cf2fe40612 Merge pull request #527 from willmiao/codex/add-unit-tests-for-metadata-components
Add metadata collector unit tests and fixtures
2025-10-05 15:03:04 +08:00
pixelpaws
8f46433ff7 Merge pull request #523 from willmiao/codex/add-tests-for-settingsmanager-migration
Add tests for settings migrations and service registry lazy loading
2025-10-05 15:02:44 +08:00
pixelpaws
f3be3ae269 Merge pull request #522 from willmiao/codex/add-tests-for-example-images-pipeline
test: add example images route and utility coverage
2025-10-05 15:02:07 +08:00
pixelpaws
cfec5447d3 test(metadata): add collector coverage 2025-10-05 14:44:17 +08:00
pixelpaws
2d36b461cf test(utils): add coverage for helper utilities 2025-10-05 14:43:21 +08:00
pixelpaws
5e23e4b13d test(metadata): cover sync service and scanner reconciliation 2025-10-05 14:42:13 +08:00
pixelpaws
badae2e8b3 test: cover websocket broadcasts and usage stats 2025-10-05 14:41:47 +08:00
pixelpaws
9e64531de6 test(settings): cover migrations and registry lazy loading 2025-10-05 14:41:24 +08:00
pixelpaws
fdec8d283c test(example-images): expand coverage for routes and utilities 2025-10-05 14:40:48 +08:00
Will Miao
9abedbf7cb fix(metadata-sync): improve error handling for deleted CivitAI models, fixes #497 2025-10-05 11:05:52 +08:00
pixelpaws
66004c1cdc Merge pull request #520 from willmiao/codex/adjust-example-images-download-to-use-library-name
fix: keep example image downloads in initial library
2025-10-05 10:08:26 +08:00
pixelpaws
5b564cd8a3 fix(example-images): pin downloads to start library 2025-10-05 09:10:25 +08:00
pixelpaws
2e79970e6e Merge pull request #519 from willmiao/codex/update-example-images-download-flow
fix: reuse migrated example image folders before download
2025-10-05 09:04:06 +08:00
pixelpaws
67c82ba6ea fix(example-images): reuse migrated folders during downloads 2025-10-05 08:37:11 +08:00
Will Miao
98425f37b8 fix(download-manager): improve handling of civitai payloads to avoid empty dictionaries 2025-10-05 07:29:11 +08:00
Will Miao
9d22dd3465 fix(model-library): update response structure to return model versions directly 2025-10-04 22:06:33 +08:00
pixelpaws
837138db49 Merge pull request #517 from willmiao/codex/determine-example_images_path-structure
feat: namespace example image storage by library
2025-10-04 20:42:15 +08:00
pixelpaws
d43d992362 feat(example-images): namespace storage by library 2025-10-04 20:29:49 +08:00
Will Miao
16b611cb7e Simplify settings file location and configuration 2025-10-04 18:42:53 +08:00
Will Miao
8dde2d5e0d feat(websocket-manager): implement caching for initialization progress and enhance broadcast functionality 2025-10-04 18:19:20 +08:00
Will Miao
22b0b2bd24 fix(model-card): correct query parameter handling in versioned preview URL 2025-10-04 17:32:16 +08:00
Will Miao
056f727bfd feat(model-scanner): enhance page type determination for model types 2025-10-04 17:08:02 +08:00
Will Miao
0aa6c53c1f feat(initialization): add support for embeddings page type and log progress updates 2025-10-04 14:11:27 +08:00
pixelpaws
d9b0660611 Merge pull request #516 from willmiao/codex/investigate-library-switching-functionality-issue
feat: serve dynamic preview assets after library switch
2025-10-04 10:51:52 +08:00
pixelpaws
d01666f4e2 feat(previews): serve dynamic library previews 2025-10-04 10:38:06 +08:00
Will Miao
51bee87cd0 fix(persistence): improve handling of lora_info attributes in recipe persistence 2025-10-04 09:52:53 +08:00
pixelpaws
3041b443e5 Merge pull request #515 from willmiao/codex/add-library-selection-in-settings-modal
Add tests for settings library switcher
2025-10-04 09:22:11 +08:00
pixelpaws
d95e6c939b test(settings): cover library switch workflow 2025-10-04 09:07:02 +08:00
pixelpaws
fd38c63b35 Merge pull request #514 from willmiao/codex/add-multi-library-endpoints-for-frontend
test: add coverage for settings library endpoints
2025-10-04 08:43:03 +08:00
pixelpaws
b69c24ae14 test(routes): cover settings library endpoints 2025-10-04 08:38:59 +08:00
pixelpaws
65a0c00e33 Merge pull request #513 from willmiao/codex/add-link-button-to-settings-modal-header
Adjust settings modal location shortcut styling
2025-10-04 07:57:43 +08:00
pixelpaws
b12a5ef133 style(settings): restyle settings location shortcut 2025-10-04 07:52:10 +08:00
pixelpaws
9e1b92c26e Merge pull request #512 from willmiao/codex/analyze-and-add-tests-for-misc-routes
test: improve misc routes coverage
2025-10-04 05:25:58 +08:00
pixelpaws
3922aec36e test(routes): extend misc routes coverage 2025-10-04 05:10:43 +08:00
pixelpaws
41cca8e56d Merge pull request #511 from willmiao/codex/refactor-misc_routes.py-and-add-tests
refactor: modularize misc route controller
2025-10-03 22:47:22 +08:00
pixelpaws
2d37a7341a fix(routes): await trained words extraction 2025-10-03 22:45:25 +08:00
pixelpaws
40e3c6134c refactor(routes): modularize misc route handling 2025-10-03 22:19:09 +08:00
pixelpaws
edddd47a1e Merge pull request #510 from willmiao/codex/update-default-library-folder-path-handling
fix: rename legacy default library to comfyui
2025-10-03 21:50:41 +08:00
pixelpaws
4ea6f38645 fix(config): rename legacy default library 2025-10-03 21:24:17 +08:00
Will Miao
40d998a026 fix(settings): use timezone-aware datetime for current timestamp
fix(tests): normalize stored paths in library upsert test
2025-10-03 20:59:47 +08:00
pixelpaws
3af8f151ac Merge pull request #509 from willmiao/codex/implement-features-from-multi-library-design
feat: add multi-library backend support
2025-10-03 20:37:59 +08:00
pixelpaws
e066fa6873 feat(settings): add multi-library backend support 2025-10-03 20:08:35 +08:00
Will Miao
6bd94269d4 refactor(model-scanner): remove unused model scanning methods 2025-10-03 18:58:02 +08:00
Will Miao
c90edec18a feat(multi-library): add design documentation for multi-library management in standalone mode 2025-10-03 18:34:52 +08:00
Will Miao
cbb302614c Merge branch 'main' of https://github.com/willmiao/ComfyUI-Lora-Manager 2025-10-03 15:23:30 +08:00
Will Miao
c54611a11b fix(metadata-service): change log level to debug for SQLite and fallback provider registration 2025-10-03 15:23:28 +08:00
pixelpaws
88f249649a Merge pull request #508 from willmiao/codex/sync-persistent-model-cache-metadata-updates
fix: sync persistent cache with metadata updates
2025-10-03 15:06:30 +08:00
pixelpaws
fe9fbdb93c fix(cache): sync persistent metadata updates 2025-10-03 14:57:44 +08:00
Will Miao
28bc966b76 feat(model-scanner): enhance cache loading with progress reporting and fallback to full scan 2025-10-03 14:31:08 +08:00
Will Miao
77bbf85b52 feat(persistent-cache): implement SQLite-based persistent model cache with loading and saving functionality 2025-10-03 11:00:51 +08:00
Will Miao
3b1990e97a feat(scanner): enhance model scanning with cache build result and progress reporting 2025-10-02 21:25:09 +08:00
Will Miao
375b5a49f3 fix(config): update standalone mode environment variable usage 2025-10-02 09:40:24 +08:00
pixelpaws
392c157cb5 Merge pull request #503 from willmiao/codex/add-hebrew-locale-to-tests
fix(i18n): add hebrew locale coverage
2025-09-30 22:07:52 +08:00
pixelpaws
6f5bf4b582 fix(i18n): add hebrew locale coverage 2025-09-30 22:03:44 +08:00
pixelpaws
2e3f48ebb7 Merge pull request #426 from start-life/main
Adding Hebrew language
2025-09-30 21:51:50 +08:00
Will Miao
e4a2c518bb fix(preset): update filePath retrieval method in removePreset function 2025-09-30 21:41:30 +08:00
pixelpaws
f19fb68b4c Merge pull request #501 from willmiao/codex/update-downloadmanager-to-handle-multiple-download-urls
feat: retry mirror downloads sequentially
2025-09-30 17:17:34 +08:00
pixelpaws
9121c12a2c feat(download): retry mirror urls sequentially 2025-09-30 17:14:59 +08:00
Will Miao
d0fe28cfe2 fix(recipe): validate modelVersionId before fetching hash from cache or Civitai 2025-09-28 09:18:59 +08:00
Will Miao
656e3e43be fix(imports): update import paths for ensure_settings_file to use relative imports 2025-09-28 08:40:09 +08:00
pixelpaws
c2c1772371 Merge pull request #496 from willmiao/codex/find-best-practices-for-settings-file-storage
feat(settings): persist settings in user config directory
2025-09-28 07:06:02 +08:00
pixelpaws
88d5caf642 feat(settings): migrate settings to user config dir 2025-09-27 22:22:15 +08:00
pixelpaws
1684978693 Merge pull request #491 from willmiao/codex/replace-spaces-in-embedding-paths
Fix embedding relative paths by replacing spaces
2025-09-26 09:06:56 +08:00
pixelpaws
8e4927600f fix(embeddings): replace spaces in relative paths 2025-09-26 09:02:46 +08:00
pixelpaws
4d72dc57e7 Merge pull request #490 from willmiao/codex/remove-comfy-field-from-images-metadata
fix: strip comfy metadata from civitai model images
2025-09-26 08:59:30 +08:00
pixelpaws
e7316b3389 fix(civitai): strip comfy metadata from images 2025-09-26 08:55:46 +08:00
start-life
e17b374606 Update zh-CN.json 2025-09-26 02:16:58 +03:00
Will Miao
141f83065f fix(locales): update language selection text in Chinese 2025-09-25 21:14:58 +08:00
pixelpaws
6381dbafc1 Merge pull request #488 from willmiao/codex/fix-bespoke-import_from-loader-issue
test: standardize backend package imports
2025-09-25 16:51:17 +08:00
pixelpaws
fc9db4510f test: fix duplicate pytest import 2025-09-25 16:44:43 +08:00
pixelpaws
66abf736c9 Merge pull request #487 from willmiao/codex/fix-lora-information-import-issue
fix: parse civitai image LoRAs from hash metadata
2025-09-25 16:00:48 +08:00
pixelpaws
af713470c1 fix(recipes): parse loras from civitai hashes 2025-09-25 15:59:44 +08:00
pixelpaws
93a51d2bcb Merge pull request #486 from willmiao/codex/enable-test-coverage-metrics-in-ci
ci: add pytest coverage reporting
2025-09-25 15:58:47 +08:00
pixelpaws
3f3e06de8a Fix pytest command in backend tests workflow 2025-09-25 15:57:59 +08:00
pixelpaws
7315aac9d8 ci: add pytest coverage reporting 2025-09-25 15:47:07 +08:00
pixelpaws
d933308a6f Merge pull request #485 from willmiao/codex/expand-vitest-coverage-for-frontend-components
test(frontend): extend coverage for comfyui widgets and helpers
2025-09-25 15:37:48 +08:00
pixelpaws
3baf93dcc5 test(frontend): extend coverage for comfyui widgets and helpers 2025-09-25 15:32:25 +08:00
pixelpaws
6ba14bd8fe Merge pull request #484 from willmiao/codex/fix-update-check-in-offline-mode
Fix offline update logging and respect update notification toggle
2025-09-25 14:54:28 +08:00
pixelpaws
7499570766 fix(updates): avoid network stack traces offline 2025-09-25 14:50:06 +08:00
start-life
003ee55a75 Merge branch 'main' into main 2025-09-25 09:28:30 +03:00
pixelpaws
b0cc42ef1f Merge pull request #483 from willmiao/codex/introduce-integration-and-contract-tests
test: add aiohttp integration coverage for ServiceRegistry
2025-09-25 14:23:53 +08:00
pixelpaws
23679ec3f5 chore(tests): clean integration route header 2025-09-25 14:17:45 +08:00
Will Miao
da52e5b9dd fix(settings): improve logic for auto-setting default root paths based on folder presence 2025-09-25 10:56:09 +08:00
Will Miao
c4e357793f Merge branch 'main' of https://github.com/willmiao/ComfyUI-Lora-Manager 2025-09-25 10:43:14 +08:00
Will Miao
6c3424029c fix(recipe image): optimize image saving and update PNG metadata handling, fixes #481 2025-09-25 10:43:10 +08:00
pixelpaws
dd9e6a5b69 Merge pull request #482 from willmiao/codex/add-pytest-modules-for-untested-services
Add backend service and route test coverage
2025-09-25 09:48:27 +08:00
pixelpaws
095320ef72 test(routes): tidy lora route test imports 2025-09-25 09:40:25 +08:00
pixelpaws
35f7674bcd Merge pull request #480 from willmiao/codex/modularize-i18n-tests-into-smaller-cases
test(i18n): modularize translation validation
2025-09-25 09:25:06 +08:00
pixelpaws
26b36c123d test(i18n): modularize translation validation 2025-09-25 09:21:08 +08:00
Will Miao
c85e694c1d docs: update repository guidelines for clarity and consistency 2025-09-25 08:36:15 +08:00
Will Miao
ec05282db6 fix(coverage): improve Vitest CLI path handling and error checking 2025-09-25 08:11:48 +08:00
pixelpaws
3d6f9b226f Merge pull request #479 from willmiao/codex/execute-phase-5-tasks-and-update-roadmap
Add frontend coverage workflow and reporting script
2025-09-25 07:00:04 +08:00
pixelpaws
eda6df4a5d chore(ci): add frontend coverage workflow 2025-09-24 23:22:32 +08:00
Will Miao
d504f89f6a Merge branch 'main' of https://github.com/willmiao/ComfyUI-Lora-Manager 2025-09-24 21:11:43 +08:00
Will Miao
14c468f2a2 feat(video): enhance video handling in model cards with lazy loading and autoplay settings, see #446 2025-09-24 21:11:36 +08:00
pixelpaws
2a99b0e46f Merge pull request #478 from willmiao/codex/execute-phase-4-tasks-from-roadmap
Add interaction-level frontend regression tests
2025-09-24 20:38:51 +08:00
pixelpaws
ae8914f5c8 test(frontend): add interaction regression suites 2025-09-24 20:33:41 +08:00
pixelpaws
0c9f8971ce Merge pull request #477 from willmiao/codex/execute-phase-3-tasks-and-update-roadmap
test: add embeddings and recipes manager suites
2025-09-24 20:18:37 +08:00
pixelpaws
d7a75ea4e5 test(frontend): cover embeddings and recipes managers 2025-09-24 20:15:38 +08:00
pixelpaws
3ad8d8b17c Merge pull request #476 from willmiao/codex/plan-next-tasks-based-on-roadmap-juork7
test(frontend): cover filtering flows for lora and checkpoints
2025-09-24 17:54:50 +08:00
pixelpaws
39225dc204 test(frontend): add filtering coverage for model pages 2025-09-24 17:50:04 +08:00
pixelpaws
4fb69f7d89 Merge pull request #475 from willmiao/codex/plan-next-tasks-based-on-roadmap-gpw3fe
Add checkpoints page smoke tests
2025-09-24 17:22:22 +08:00
pixelpaws
0890c6ad24 test(frontend): add checkpoints manager smoke tests 2025-09-24 17:18:20 +08:00
pixelpaws
dd81809589 Merge pull request #474 from willmiao/codex/plan-next-steps-for-roadmap-tasks
test(frontend): add loras page manager suite
2025-09-24 17:09:38 +08:00
pixelpaws
f0672beb46 test(frontend): add loras page manager suite 2025-09-24 16:22:17 +08:00
pixelpaws
cc5301e710 Merge pull request #473 from willmiao/codex/plan-next-tasks-based-on-roadmap
fix(frontend): validate AppCore initialization wiring
2025-09-24 16:15:05 +08:00
pixelpaws
9d5ec43c4e fix(frontend): correct AppCore example images initialization 2025-09-24 16:10:27 +08:00
pixelpaws
6d41211b07 Merge pull request #472 from willmiao/codex/plan-and-update-frontend-testing-roadmap
Add AppCore page orchestration tests
2025-09-24 15:56:17 +08:00
pixelpaws
d58b61eed5 test(frontend): cover appcore page features 2025-09-24 15:55:50 +08:00
pixelpaws
4b53d98bfc Merge pull request #471 from willmiao/codex/organize-frontend-tests-into-new-directories
test(frontend): document dom fixture workflow
2025-09-24 15:44:23 +08:00
pixelpaws
f51f354e48 test(frontend): add dom fixture helpers 2025-09-24 15:39:52 +08:00
Will Miao
59d027181d refactor: remove obsolete JSON files and add new version metadata for LORA model 2025-09-24 10:56:06 +08:00
Will Miao
0d0988c090 feat: add functionality to attach model files to version data in SQLiteModelMetadataProvider 2025-09-24 10:55:55 +08:00
Will Miao
dc2de50924 bump: update version to 0.9.5 in pyproject.toml 2025-09-24 09:16:50 +08:00
start-life
a3c28c1003 Update zh-TW.json 2025-09-12 03:33:34 +03:00
start-life
f4b7c9a138 Update zh-CN.json 2025-09-12 03:33:09 +03:00
start-life
6b860b5f29 Update ru.json 2025-09-12 03:32:15 +03:00
start-life
37dfcd6abd Update ko.json 2025-09-12 03:31:57 +03:00
start-life
bc2fca3a4f Update ja.json 2025-09-12 03:31:35 +03:00
start-life
f8ef159656 Update fr.json 2025-09-12 03:31:12 +03:00
start-life
b2b8a9d37e Update es.json 2025-09-12 03:30:56 +03:00
start-life
15ae4031b7 Update de.json 2025-09-12 03:30:39 +03:00
start-life
688976ce3b Update en.json 2025-09-12 03:30:14 +03:00
start-life
a548af01dc Update settings_modal.html
Adding Hebrew language
2025-09-12 03:28:45 +03:00
start-life
0dd52eceb3 Update index.js
Adding Hebrew language
2025-09-12 03:26:08 +03:00
start-life
b8c6cf4ac1 Add files via upload
Adding Hebrew language
2025-09-12 03:20:38 +03:00
241 changed files with 25636 additions and 5445 deletions

69
.github/workflows/backend-tests.yml vendored Normal file
View File

@@ -0,0 +1,69 @@
name: Backend Tests
on:
push:
branches:
- main
- master
paths:
- 'py/**'
- 'standalone.py'
- 'tests/**'
- 'requirements.txt'
- 'requirements-dev.txt'
- 'pyproject.toml'
- 'pytest.ini'
- '.github/workflows/backend-tests.yml'
pull_request:
paths:
- 'py/**'
- 'standalone.py'
- 'tests/**'
- 'requirements.txt'
- 'requirements-dev.txt'
- 'pyproject.toml'
- 'pytest.ini'
- '.github/workflows/backend-tests.yml'
jobs:
pytest:
name: Run pytest with coverage
runs-on: ubuntu-latest
steps:
- name: Check out repository
uses: actions/checkout@v4
- name: Set up Python 3.11
uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: 'pip'
cache-dependency-path: |
requirements.txt
requirements-dev.txt
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements-dev.txt
- name: Run pytest with coverage
env:
COVERAGE_FILE: coverage/backend/.coverage
run: |
mkdir -p coverage/backend
python -m pytest \
--cov=py \
--cov=standalone \
--cov-report=term-missing \
--cov-report=xml:coverage/backend/coverage.xml \
--cov-report=html:coverage/backend/html \
--cov-report=json:coverage/backend/coverage.json
- name: Upload coverage artifact
if: always()
uses: actions/upload-artifact@v4
with:
name: backend-coverage
path: coverage/backend
if-no-files-found: warn

52
.github/workflows/frontend-tests.yml vendored Normal file
View File

@@ -0,0 +1,52 @@
name: Frontend Tests
on:
push:
branches:
- main
- master
paths:
- 'package.json'
- 'package-lock.json'
- 'vitest.config.js'
- 'tests/frontend/**'
- 'static/js/**'
- 'scripts/run_frontend_coverage.js'
- '.github/workflows/frontend-tests.yml'
pull_request:
paths:
- 'package.json'
- 'package-lock.json'
- 'vitest.config.js'
- 'tests/frontend/**'
- 'static/js/**'
- 'scripts/run_frontend_coverage.js'
- '.github/workflows/frontend-tests.yml'
jobs:
vitest:
name: Run Vitest with coverage
runs-on: ubuntu-latest
steps:
- name: Check out repository
uses: actions/checkout@v4
- name: Use Node.js 20
uses: actions/setup-node@v4
with:
node-version: 20
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Run frontend tests with coverage
run: npm run test:coverage
- name: Upload coverage artifact
if: always()
uses: actions/upload-artifact@v4
with:
name: frontend-coverage
path: coverage/frontend
if-no-files-found: warn

1
.gitignore vendored
View File

@@ -8,3 +8,4 @@ cache/
civitai/
node_modules/
coverage/
.coverage

View File

@@ -1,19 +1,22 @@
# Repository Guidelines
# Repository Guidelines
## Project Structure & Module Organization
ComfyUI LoRA Manager pairs a Python backend with lightweight browser scripts. Backend modules live in `py/`, organized by responsibility: HTTP entry points under `routes/`, feature logic in `services/`, reusable helpers within `utils/`, and custom nodes in `nodes/`. Front-end widgets that extend the ComfyUI interface sit in `web/comfyui/`, while static images and templates are in `static/` and `templates/`. Shared localization files are stored in `locales/`, with workflow examples under `example_workflows/`. Tests currently reside alongside the source (`test_i18n.py`) until a dedicated `tests/` folder is introduced.
ComfyUI LoRA Manager pairs a Python backend with browser-side widgets. Backend modules live in <code>py/</code> with HTTP entry points in <code>py/routes/</code>, feature logic in <code>py/services/</code>, shared helpers in <code>py/utils/</code>, and custom nodes in <code>py/nodes/</code>. UI scripts extend ComfyUI from <code>web/comfyui/</code>, while deploy-ready assets remain in <code>static/</code> and <code>templates/</code>. Localization files live in <code>locales/</code>, example workflows in <code>example_workflows/</code>, and interim tests such as <code>test_i18n.py</code> sit beside their source until a dedicated <code>tests/</code> tree lands.
## Build, Test, and Development Commands
Install dependencies with `pip install -r requirements.txt` from the repo root. Launch the standalone server for iterative work via `python standalone.py --port 8188`; ComfyUI users can also load the extension directly through ComfyUI's custom node manager. Run backend checks with `python -m pytest test_i18n.py`, and target new test files explicitly (e.g. `python -m pytest tests/test_recipes.py` once added). Use `python scripts/sync_translation_keys.py` to reconcile locale keys after updating UI strings.
- <code>pip install -r requirements.txt</code> installs backend dependencies.
- <code>python standalone.py --port 8188</code> launches the standalone server for iterative development.
- <code>python -m pytest test_i18n.py</code> runs the current regression suite; target new files explicitly, e.g. <code>python -m pytest tests/test_recipes.py</code>.
- <code>python scripts/sync_translation_keys.py</code> synchronizes locale keys after UI string updates.
## Coding Style & Naming Conventions
Follow PEP 8 with four-space indentation and descriptive snake_case module/function names, mirroring files such as `py/services/settings_manager.py`. Classes remain PascalCase, constants UPPER_SNAKE_CASE, and loggers retrieved via `logging.getLogger(__name__)`. Prefer explicit type hints for new public APIs and docstrings that clarify side effects. JavaScript in `web/comfyui/` is modern ES modules; keep imports relative, favor camelCase functions, and mirror existing file suffixes like `_widget.js` for UI components.
Follow PEP 8 with four-space indentation and descriptive snake_case file and function names such as <code>settings_manager.py</code>. Classes stay PascalCase, constants in UPPER_SNAKE_CASE, and loggers retrieved via <code>logging.getLogger(__name__)</code>. Prefer explicit type hints and docstrings on public APIs. JavaScript under <code>web/comfyui/</code> uses ES modules with camelCase helpers and the <code>_widget.js</code> suffix for UI components.
## Testing Guidelines
Extend pytest coverage by co-locating tests near the code under test or in `tests/` with names like `test_<feature>.py`. When introducing new routes or services, add regression cases that mock ComfyUI dependencies (see the standalone mocking helpers in `standalone.py`). Prioritize deterministic fixtures for filesystem interactions and ensure translations include coverage when adding new locale keys. Always run `python -m pytest` before submitting work.
Pytest powers backend tests. Name modules <code>test_<feature>.py</code> and keep them near the code or in a future <code>tests/</code> package. Mock ComfyUI dependencies through helpers in <code>standalone.py</code>, keep filesystem fixtures deterministic, and ensure translations are covered. Run <code>python -m pytest</code> before submitting changes.
## Commit & Pull Request Guidelines
Commits follow the conventional pattern seen in `git log` (`feat(scope):`, `fix(scope):`, `chore(scope):`). Keep messages imperative and scoped to a single change. Pull requests should summarize the problem, detail the solution, list manual test evidence, and link any GitHub issues. Include UI screenshots or GIFs when front-end behavior changes, and call out migration steps (e.g., settings updates) in the PR description.
Commits follow the conventional format, e.g. <code>feat(settings): add default model path</code>, and should stay focused on a single concern. Pull requests must outline the problem, summarize the solution, list manual verification steps (server run, targeted pytest), and link related issues. Include screenshots or GIFs for UI or locale updates and call out migration steps such as <code>settings.json</code> adjustments.
## Configuration & Localization Tips
Sample configuration defaults live in `settings.json.example`; copy it to `settings.json` and adjust model directories before running the standalone server. Whenever you add UI text, update `locales/<lang>.json` and run the translation sync script. Store reference assets in `civitai/` or `docs/` rather than mixing them with production templates, keeping the runtime folders (`static/`, `templates/`) deploy-ready.
Copy <code>settings.json.example</code> to <code>settings.json</code> and adapt model directories before running the standalone server. Store reference assets in <code>civitai/</code> or <code>docs/</code> to keep runtime directories deploy-ready. Whenever UI text changes, update every <code>locales/&lt;lang&gt;.json</code> file and rerun the translation sync script so ComfyUI surfaces localized strings.

103
IFLOW.md Normal file
View File

@@ -0,0 +1,103 @@
# ComfyUI LoRA Manager - iFlow 上下文
## 项目概述
ComfyUI LoRA Manager 是一个全面的工具集,用于简化 ComfyUI 中 LoRA 模型的组织、下载和应用。它提供了强大的功能,如配方管理、检查点组织和一键工作流集成,使模型操作更快、更流畅、更简单。
该项目是一个 Python 后端与 JavaScript 前端结合的 Web 应用程序,既可以作为 ComfyUI 的自定义节点运行,也可以作为独立应用程序运行。
## 项目结构
```
D:\Workspace\ComfyUI\custom_nodes\ComfyUI-Lora-Manager\
├── py/ # Python 后端代码
│ ├── config.py # 全局配置
│ ├── lora_manager.py # 主入口点
│ ├── controllers/ # 控制器
│ ├── metadata_collector/ # 元数据收集器
│ ├── middleware/ # 中间件
│ ├── nodes/ # ComfyUI 节点
│ ├── recipes/ # 配方相关
│ ├── routes/ # API 路由
│ ├── services/ # 业务逻辑服务
│ ├── utils/ # 工具函数
│ └── validators/ # 验证器
├── static/ # 静态资源 (CSS, JS, 图片)
├── templates/ # HTML 模板
├── locales/ # 国际化文件
├── tests/ # 测试代码
├── standalone.py # 独立模式入口
├── requirements.txt # Python 依赖
├── package.json # Node.js 依赖和脚本
└── README.md # 项目说明
```
## 核心组件
### 后端 (Python)
- **主入口**: `py/lora_manager.py``standalone.py`
- **配置**: `py/config.py` 管理全局配置和路径
- **路由**: `py/routes/` 目录下包含各种 API 路由
- **服务**: `py/services/` 目录下包含业务逻辑,如模型扫描、下载管理等
- **模型管理**: 使用 `ModelServiceFactory` 来管理不同类型的模型 (LoRA, Checkpoint, Embedding)
### 前端 (JavaScript)
- **构建工具**: 使用 Node.js 和 npm 进行依赖管理和测试
- **测试**: 使用 Vitest 进行前端测试
## 构建和运行
### 安装依赖
```bash
# Python 依赖
pip install -r requirements.txt
# Node.js 依赖 (用于测试)
npm install
```
### 运行 (ComfyUI 模式)
作为 ComfyUI 的自定义节点安装后,在 ComfyUI 中启动即可。
### 运行 (独立模式)
```bash
# 使用默认配置运行
python standalone.py
# 指定主机和端口
python standalone.py --host 127.0.0.1 --port 9000
```
### 测试
#### 后端测试
```bash
# 安装开发依赖
pip install -r requirements-dev.txt
# 运行测试
pytest
```
#### 前端测试
```bash
# 运行测试
npm run test
# 运行测试并生成覆盖率报告
npm run test:coverage
```
## 开发约定
- **代码风格**: Python 代码应遵循 PEP 8 规范
- **测试**: 新功能应包含相应的单元测试
- **配置**: 使用 `settings.json` 文件进行用户配置
- **日志**: 使用 Python 标准库 `logging` 模块进行日志记录

View File

@@ -34,7 +34,16 @@ Enhance your Civitai browsing experience with our companion browser extension! S
## Release Notes
### v0.9.3
### v0.9.8
* **Full CivArchive API Support** - Added complete support for the CivArchive API as a fallback metadata source beyond Civitai API. Models deleted from Civitai can now still retrieve metadata through the CivArchive API.
* **Download Models from CivArchive** - Added support for downloading models directly from CivArchive, similar to downloading from Civitai. Simply click the Download button and paste the model URL to download the corresponding model.
* **Custom Priority Tags** - Introduced Custom Priority Tags feature, allowing users to define custom priority tags. These tags will appear as suggestions when editing tags or during auto organization/download using default paths, providing more precise and controlled folder organization. [Guide](https://github.com/willmiao/ComfyUI-Lora-Manager/wiki/Priority-Tags-Configuration-Guide)
* **Drag and Drop Tag Reordering** - Added drag and drop functionality to reorder tags in the tags edit mode for improved usability.
* **Download Control in Example Images Panel** - Added stop control in the Download Example Images Panel for better download management.
* **Prompt (LoraManager) Node with Autocomplete** - Added new Prompt (LoraManager) node with autocomplete feature for adding embeddings.
* **Lora Manager Nodes in Subgraphs** - Lora Manager nodes now support being placed within subgraphs for more flexible workflow organization.
### v0.9.6
* **Metadata Archive Database Support** - Added the ability to download and utilize a metadata archive database, enabling access to metadata for models that have been deleted from CivitAI.
* **App-Level Proxy Settings** - Introduced support for configuring a global proxy within the application, making it easier to use the manager behind network restrictions.
* **Bug Fixes** - Various bug fixes for improved stability and reliability.
@@ -141,7 +150,7 @@ Enhance your Civitai browsing experience with our companion browser extension! S
1. Download the [Portable Package](https://github.com/willmiao/ComfyUI-Lora-Manager/releases/download/v0.9.2/lora_manager_portable.7z)
2. Copy the provided `settings.json.example` file to create a new file named `settings.json` in `comfyui-lora-manager` folder
3. Edit `settings.json` to include your correct model folder paths and CivitAI API key
3. Edit the new `settings.json` to include your correct model folder paths and CivitAI API key
4. Run run.bat
- To change the startup port, edit `run.bat` and modify the parameter (e.g. `--port 9001`)
@@ -209,7 +218,7 @@ You can combine multiple patterns to create detailed, organized filenames for yo
You can now run LoRA Manager independently from ComfyUI:
1. **For ComfyUI users**:
- Launch ComfyUI with LoRA Manager at least once to initialize the necessary path information in the `settings.json` file.
- Launch ComfyUI with LoRA Manager at least once to initialize the necessary path information in the `settings.json` file located in your user settings folder (see paths above).
- Make sure dependencies are installed: `pip install -r requirements.txt`
- From your ComfyUI root directory, run:
```bash
@@ -231,8 +240,37 @@ You can now run LoRA Manager independently from ComfyUI:
```
- Access the interface through your browser at: `http://localhost:8188/loras`
> **Note:** Existing installations automatically migrate the legacy `settings.json` from the plugin folder to the user settings directory the first time you launch this version.
This standalone mode provides a lightweight option for managing your model and recipe collection without needing to run the full ComfyUI environment, making it useful even for users who primarily use other stable diffusion interfaces.
## Testing & Coverage
### Backend
Install the development dependencies and run pytest with coverage reports:
```bash
pip install -r requirements-dev.txt
COVERAGE_FILE=coverage/backend/.coverage pytest \
--cov=py \
--cov=standalone \
--cov-report=term-missing \
--cov-report=html:coverage/backend/html \
--cov-report=xml:coverage/backend/coverage.xml \
--cov-report=json:coverage/backend/coverage.json
```
HTML, XML, and JSON artifacts are stored under `coverage/backend/` so you can inspect hot spots locally or from CI artifacts.
### Frontend
Run the Vitest coverage suite to analyze widget hot spots:
```bash
npm run test:coverage
```
---
## Contributing

View File

@@ -2,6 +2,7 @@ try: # pragma: no cover - import fallback for pytest collection
from .py.lora_manager import LoraManager
from .py.nodes.lora_loader import LoraManagerLoader, LoraManagerTextLoader
from .py.nodes.trigger_word_toggle import TriggerWordToggle
from .py.nodes.prompt import PromptLoraManager
from .py.nodes.lora_stacker import LoraStacker
from .py.nodes.save_image import SaveImage
from .py.nodes.debug_metadata import DebugMetadata
@@ -17,6 +18,7 @@ except ImportError: # pragma: no cover - allows running under pytest without pa
if str(package_root) not in sys.path:
sys.path.append(str(package_root))
PromptLoraManager = importlib.import_module("py.nodes.prompt").PromptLoraManager
LoraManager = importlib.import_module("py.lora_manager").LoraManager
LoraManagerLoader = importlib.import_module("py.nodes.lora_loader").LoraManagerLoader
LoraManagerTextLoader = importlib.import_module("py.nodes.lora_loader").LoraManagerTextLoader
@@ -29,6 +31,7 @@ except ImportError: # pragma: no cover - allows running under pytest without pa
init_metadata_collector = importlib.import_module("py.metadata_collector").init
NODE_CLASS_MAPPINGS = {
PromptLoraManager.NAME: PromptLoraManager,
LoraManagerLoader.NAME: LoraManagerLoader,
LoraManagerTextLoader.NAME: LoraManagerTextLoader,
TriggerWordToggle.NAME: TriggerWordToggle,

View File

@@ -0,0 +1,34 @@
# Multi-Library Management for Standalone Mode
## Requirements Summary
- **Independent libraries**: In standalone mode, users can maintain multiple libraries, where each library represents a distinct set of model folders (LoRAs, checkpoints, embeddings, etc.). Only one library is active at any given time, but users need a fast way to switch between them.
- **Library-specific settings**: The fields that vary per library are `folder_paths`, `default_lora_root`, `default_checkpoint_root`, and `default_embedding_root` inside `settings.json`.
- **Persistent caches**: Every library must have its own SQLite persistent model cache so that metadata generated for one library does not leak into another.
- **Backward compatibility**: Existing single-library setups should continue to work. When no multi-library configuration is provided, the application should behave exactly as before.
## Proposed Design
1. **Library registry**
- Extend the standalone configuration to hold a list of libraries, each identified by a unique name.
- Each entry stores the folder path configuration plus any library-scoped metadata (e.g. creation time, display name).
- The active library key is stored separately to allow quick switching without rewriting the full config.
2. **Settings management**
- Update `settings_manager` to load and persist the library registry. When a library is activated, hydrate the in-memory settings object with that library's folder configuration.
- Provide helper methods for creating, renaming, and deleting libraries, ensuring validation for duplicate names and path collisions.
- Continue writing the active library settings to `settings.json` for compatibility, while storing the registry in a new section such as `libraries`.
3. **Persistent model cache**
- Derive the SQLite file path from the active library, e.g. `model_cache_<library>.sqlite` or a nested directory structure like `model_cache/<library>/models.sqlite`.
- Update `PersistentModelCache` so it resolves the database path dynamically whenever the active library changes. Ensure connections are closed before switching to avoid locking issues.
- Migrate existing single cache files by treating them as the default library's cache.
4. **Model scanning workflow**
- Modify `ModelScanner` and related services to react to library switches by clearing in-memory caches, re-reading folder paths, and rehydrating metadata from the library-specific SQLite cache.
- Provide API endpoints in standalone mode to list libraries, activate one, and trigger a rescan.
5. **UI/UX considerations**
- In the standalone UI, introduce a library selector component that surfaces available libraries and offers quick switching.
- Offer feedback when switching libraries (e.g. spinner while rescanning) and guard destructive actions with confirmation prompts.
## Implementation Notes
- **Data migration**: On startup, detect if the old `settings.json` structure is present. If so, create a default library entry using the current folder paths and point the active library to it.
- **Thread safety**: Ensure that any long-running scans are cancelled or awaited before switching libraries to prevent race conditions in cache writes.
- **Testing**: Add unit tests for the settings manager to cover library CRUD operations and cache path resolution. Include integration tests that simulate switching libraries and verifying that the correct models are loaded.
- **Documentation**: Update user guides to explain how to define libraries, switch between them, and where the new cache files are stored.
- **Extensibility**: Keep the design open to future per-library settings (e.g. auto-refresh intervals, metadata overrides) by storing library data as objects instead of flat maps.

View File

@@ -0,0 +1,46 @@
# Custom Priority Tag Format Proposal
To support user-defined priority tags with flexible aliasing across different model types, the configuration will be stored as editable strings. The format balances readability with enough structure for parsing on both the backend and frontend.
## Format Overview
- Each model type is declared on its own line: `model_type: entries`.
- Entries are comma-separated and ordered by priority from highest to lowest.
- An entry may be a single canonical tag (e.g., `realistic`) or a canonical tag with aliases.
- Canonical tags define the final folder name that should be used when matching that entry.
- Aliases are enclosed in parentheses and separated by `|` (vertical bar).
- All matching is case-insensitive; stored canonical names preserve the user-specified casing for folder creation and UI suggestions.
### Grammar
```
priority-config := model-config { "\n" model-config }
model-config := model-type ":" entry-list
model-type := <identifier without spaces>
entry-list := entry { "," entry }
entry := canonical [ "(" alias { "|" alias } ")" ]
canonical := <tag text without parentheses or commas>
alias := <tag text without parentheses, commas, or pipes>
```
Examples:
```
lora: celebrity(celeb|celebrity), stylized, character(char)
checkpoint: realistic(realism|realistic), anime(anime-style|toon)
embedding: face, celeb(celebrity|celeb)
```
## Parsing Notes
- Whitespace around separators is ignored to make manual editing more forgiving.
- Duplicate canonical tags within the same model type collapse to a single entry; the first definition wins.
- Aliases map to their canonical tag. When generating folder names, the canonical form is used.
- Tags that do not match any alias or canonical entry fall back to the first tag in the model's tag list, preserving current behavior.
## Usage
- **Backend:** Convert each model type's string into an ordered list of canonical tags with alias sets. During path generation, iterate by priority order and match tags against both canonical names and their aliases.
- **Frontend:** Surface canonical tags as suggestions, optionally displaying aliases in tooltips or secondary text. Input validation should warn about duplicate aliases within the same model type.
This format allows users to customize priority tag handling per model type while keeping editing simple and avoiding proliferation of folder names through alias normalization.

View File

@@ -0,0 +1,51 @@
# Frontend DOM Fixture Strategy
This guide outlines how to reproduce the markup emitted by the Django templates while running Vitest in jsdom. The aim is to make it straightforward to write integration-style unit tests for managers and UI helpers without having to duplicate template fragments inline.
## Loading Template Markup
Vitest executes inside Node, so we can read the same HTML templates that ship with the extension:
1. Use the helper utilities from `tests/frontend/utils/domFixtures.js` to read files under the `templates/` directory.
2. Mount the returned markup into `document.body` (or any custom container) before importing the module under test so its query selectors resolve correctly.
```js
import { renderTemplate } from '../utils/domFixtures.js'; // adjust the relative path to your spec
beforeEach(() => {
renderTemplate('loras.html', {
dataset: { page: 'loras' }
});
});
```
The helper ensures the dataset is applied to the container, which mirrors how Django sets `data-page` in production.
## Working with Partial Components
Many features are implemented as template partials located under `templates/components/`. When a test only needs a fragment (for example, the progress panel or context menu markup), load the component file directly:
```js
const container = renderTemplate('components/progress_panel.html');
const progressPanel = container.querySelector('#progress-panel');
```
This pattern avoids hand-written fixture strings and keeps the tests aligned with the actual markup.
## Resetting Between Tests
The shared Vitest setup clears `document.body` and storage APIs before each test. If a suite adds additional DOM nodes outside of the body or needs to reset custom attributes mid-test, use `resetDom()` exported from `domFixtures.js`.
```js
import { resetDom } from '../utils/domFixtures.js';
afterEach(() => {
resetDom();
});
```
## Future Enhancements
- Provide typed helpers for injecting mock script tags (e.g., replicating ComfyUI globals).
- Compose higher-level fixtures that mimic specific pages (loras, checkpoints, recipes) once those managers receive dedicated suites.

View File

@@ -0,0 +1,44 @@
# LoRA & Checkpoints Filtering/Sorting Test Matrix
This matrix captures the scenarios that Phase 3 frontend tests should cover for the LoRA and Checkpoint managers. It focuses on how search, filter, sort, and duplicate badge toggles interact so future specs can share fixtures and expectations.
## Scope
- **Components**: `PageControls`, `FilterManager`, `SearchManager`, and `ModelDuplicatesManager` wiring invoked through `CheckpointsPageManager` and `LorasPageManager`.
- **Templates**: `templates/loras.html` and `templates/checkpoints.html` along with shared filter panel and toolbar partials.
- **APIs**: Requests issued through `baseModelApi.fetchModels` (via `resetAndReload`/`refreshModels`) and duplicates badge updates.
## Shared Setup Considerations
1. Render full page templates using `renderLorasPage` / `renderCheckpointsPage` helpers before importing modules so DOM queries resolve.
2. Stub storage helpers (`getStorageItem`, `setStorageItem`, `getSessionItem`, `setSessionItem`) to observe persistence behavior without mutating real storage.
3. Mock `sidebarManager` to capture refresh calls triggered after sort/filter actions.
4. Provide fake API implementations exposing `resetAndReload`, `refreshModels`, `fetchFromCivitai`, `toggleBulkMode`, and `clearCustomFilter` so control events remain asynchronous but deterministic.
5. Supply a minimal `ModelDuplicatesManager` mock exposing `toggleDuplicateMode`, `checkDuplicatesCount`, and `updateDuplicatesBadgeAfterRefresh` to validate duplicate badge wiring.
## Scenario Matrix
| ID | Feature | Scenario | LoRAs Expectations | Checkpoints Expectations | Notes |
| --- | --- | --- | --- | --- | --- |
| F-01 | Search filter | Typing a query updates `pageState.filters.search`, persists to session, and triggers `resetAndReload` on submit | Validate `SearchManager` writes query and reloads via API stub; confirm LoRA cards pass query downstream | Same as LoRAs | Cover `enter` press and clicking search icon |
| F-02 | Tag filter | Selecting a tag chip adds it to filters, applies active styling, and reloads results | Tag stored under `filters.tags`; `FilterManager.applyFilters` persists and triggers `resetAndReload(true)` | Same; ensure base model tag set is scoped to checkpoints dataset | Include removal path |
| F-03 | Base model filter | Toggling base model checkboxes updates `filters.baseModel`, persists, and reloads | Ensure only LoRA-supported models show; toggle multi-select | Ensure SDXL/Flux base models appear as expected | Capture UI state restored from storage on next init |
| F-04 | Favorites-only | Clicking favorites toggle updates session flag and calls `resetAndReload(true)` | Button gains `.active` class and API called | Same | Verify duplicates badge refresh when active |
| F-05 | Sort selection | Changing sort select saves preference (legacy + new format) and reloads | Confirm `PageControls.saveSortPreference` invoked with option and API called | Same with checkpoints-specific defaults | Cover `convertLegacySortFormat` branch |
| F-06 | Filter persistence | Re-initializing manager loads stored filters/sort and updates DOM | Filters pre-populate chips/checkboxes; favorites state restored | Same | Requires simulating repeated construction |
| F-07 | Combined filters | Applying search + tag + base model yields aggregated query params for fetch | Assert API receives merged filter payload | Same | Validate toast messaging for active filters |
| F-08 | Clearing filters | Using "Clear filters" resets state, storage, and reloads list | `FilterManager.clearFilters` empties `filters`, removes active class, shows toast | Same | Ensure favorites-only toggle unaffected |
| F-09 | Duplicate badge toggle | Pressing "Find duplicates" toggles duplicate mode and updates badge counts post-refresh | `ModelDuplicatesManager.toggleDuplicateMode` invoked and badge refresh called after API rebuild | Same plus checkpoint-specific duplicate badge dataset | Connects to future duplicate-specific specs |
| F-10 | Bulk actions menu | Opening bulk dropdown keeps filters intact and closes on outside click | Validate dropdown class toggling and no unintended reload | Same | Guard against regression when dropdown interacts with filters |
## Automation Coverage Status
- ✅ F-01 Search filter, F-02 Tag filter, F-03 Base model filter, F-04 Favorites-only toggle, F-05 Sort selection, and F-09 Duplicate badge toggle are covered by `tests/frontend/components/pageControls.filtering.test.js` for both LoRA and checkpoint pages.
- ⏳ F-06 Filter persistence, F-07 Combined filters, F-08 Clearing filters, and F-10 Bulk actions remain to be automated alongside upcoming bulk mode refinements.
## Coverage Gaps & Follow-Ups
- Write Vitest suites that exercise the matrix for both managers, sharing fixtures through page helpers to avoid duplication.
- Capture API parameter assertions by inspecting `baseModelApi.fetchModels` mocks rather than relying solely on state mutations.
- Add regression cases for legacy storage migrations (old filter keys) once fixtures exist for older payloads.
- Extend duplicate badge coverage with scenarios where `checkDuplicatesCount` signals zero duplicates versus pending calculations.

View File

@@ -8,16 +8,26 @@ This roadmap tracks the planned rollout of automated testing for the ComfyUI LoR
| --- | --- | --- | --- | --- |
| Phase 0 | Establish baseline tooling | Add Node test runner, jsdom environment, and seed smoke tests | ✅ Complete | Vitest + jsdom configured, example state tests committed |
| Phase 1 | Cover state management logic | Unit test selectors, derived data helpers, and storage utilities under `static/js/state` and `static/js/utils` | ✅ Complete | Storage helpers and state selectors now exercised via deterministic suites |
| Phase 2 | Test AppCore orchestration | Simulate page bootstrapping, infinite scroll hooks, and manager registration using JSDOM DOM fixtures | 🟡 In Progress | AppCore initialization specs landed; expand to additional page wiring and scroll hooks |
| Phase 3 | Validate page-specific managers | Add focused suites for `loras`, `checkpoints`, `embeddings`, and `recipes` managers covering filtering, sorting, and bulk actions | ⚪ Not Started | Consider shared helpers for mocking API modules and storage |
| Phase 4 | Interaction-level regression tests | Exercise template fragments, modals, and menus to ensure UI wiring remains intact | ⚪ Not Started | Evaluate Playwright component testing or happy-path DOM snapshots |
| Phase 5 | Continuous integration & coverage | Integrate frontend tests into CI workflow and track coverage metrics | ⚪ Not Started | Align reporting directories with backend coverage for unified reporting |
| Phase 2 | Test AppCore orchestration | Simulate page bootstrapping, infinite scroll hooks, and manager registration using JSDOM DOM fixtures | ✅ Complete | AppCore initialization + page feature suites now validate manager wiring, infinite scroll hooks, and onboarding gating |
| Phase 3 | Validate page-specific managers | Add focused suites for `loras`, `checkpoints`, `embeddings`, and `recipes` managers covering filtering, sorting, and bulk actions | ✅ Complete | LoRA/checkpoint suites expanded; embeddings + recipes managers now covered with initialization, filtering, and duplicate workflows |
| Phase 4 | Interaction-level regression tests | Exercise template fragments, modals, and menus to ensure UI wiring remains intact | ✅ Complete | Vitest DOM suites cover NSFW selector, recipe modal editing, and global context menus |
| Phase 5 | Continuous integration & coverage | Integrate frontend tests into CI workflow and track coverage metrics | ✅ Complete | CI workflow runs Vitest and aggregates V8 coverage into `coverage/frontend` via a dedicated script |
## Next Steps Checklist
- [x] Expand unit tests for `storageHelpers` covering migrations and namespace behavior.
- [ ] Document DOM fixture strategy for reproducing template structures in tests.
- [x] Document DOM fixture strategy for reproducing template structures in tests.
- [x] Prototype AppCore initialization test that verifies manager bootstrapping with stubbed dependencies.
- [ ] Evaluate integrating coverage reporting once test surface grows (> 20 specs).
- [x] Add AppCore page feature suite exercising context menu creation and infinite scroll registration via DOM fixtures.
- [x] Extend AppCore orchestration tests to cover manager wiring, bulk menu setup, and onboarding gating scenarios.
- [x] Add interaction regression suites for context menus and recipe modals to complete Phase 4.
- [x] Evaluate integrating coverage reporting once test surface grows (> 20 specs).
- [x] Create shared fixtures for the loras and checkpoints pages once dedicated manager suites are added.
- [x] Draft focused test matrix for loras/checkpoints manager filtering and sorting paths ahead of Phase 3.
- [x] Implement LoRAs manager filtering/sorting specs for scenarios F-01F-05 & F-09; queue remaining edge cases after duplicate/bulk flows stabilize.
- [x] Implement checkpoints manager filtering/sorting specs for scenarios F-01F-05 & F-09; cover remaining paths alongside bulk action work.
- [x] Implement checkpoints page manager smoke tests covering initialization and duplicate badge wiring.
- [x] Outline focused checkpoints scenarios (filtering, sorting, duplicate badge toggles) to feed into the shared test matrix.
- [ ] Add duplicate badge regression coverage for zero/pending states after API refreshes.
Maintaining this roadmap alongside code changes will make it easier to append new automated test tasks and update their progress.

28
docs/library-switching.md Normal file
View File

@@ -0,0 +1,28 @@
# Library Switching and Preview Routes
Library switching no longer requires restarting the backend. The preview
thumbnails shown in the UI are now served through a dynamic endpoint that
resolves files against the folders registered for the active library at request
time. This allows the multi-library flow to update model roots without touching
the aiohttp router, so previews remain available immediately after a switch.
## How the dynamic preview endpoint works
* `config.get_preview_static_url()` now returns `/api/lm/previews?path=<encoded>`
for any preview path. The raw filesystem location is URL encoded so that it
can be passed through the query string without leaking directory structure in
the route itself.【F:py/config.py†L398-L404】
* `PreviewRoutes` exposes the `/api/lm/previews` handler which validates the
decoded path against the directories registered for the current library. The
request is rejected if it falls outside those roots or if the file does not
exist.【F:py/routes/preview_routes.py†L5-L21】【F:py/routes/handlers/preview_handlers.py†L9-L48】
* `Config` keeps an up-to-date cache of allowed preview roots. Every time a
library is applied the cache is rebuilt using the declared LoRA, checkpoint
and embedding directories (including symlink targets). The validation logic
checks preview requests against this cache.【F:py/config.py†L51-L68】【F:py/config.py†L180-L248】【F:py/config.py†L332-L346】
Both the ComfyUI runtime (`LoraManager.add_routes`) and the standalone launcher
(`StandaloneLoraManager.add_routes`) register the new preview routes instead of
mounting a static directory per root. Switching libraries therefore works
without restarting the application, and preview URLs generated before or after a
switch continue to resolve correctly.【F:py/lora_manager.py†L21-L82】【F:standalone.py†L302-L315】

View File

@@ -0,0 +1,71 @@
# Priority Tags Configuration Guide
This guide explains how to tailor the tag priority order that powers folder naming and tag suggestions in the LoRA Manager. You only need to edit the comma-separated list of entries shown in the **Priority Tags** field for each model type.
## 1. Pick the Model Type
In the **Priority Tags** dialog you will find one tab per model type (LoRA, Checkpoint, Embedding). Select the tab you want to update; changes on one tab do not affect the others.
## 2. Edit the Entry List
Inside the textarea you will see a line similar to:
```
character, concept, style(toon|toon_style)
```
This entire line is the **entry list**. Replace it with your own ordered list.
### Entry Rules
Each entry is separated by a comma, in order from highest to lowest priority:
- **Canonical tag only:** `realistic`
- **Canonical tag with aliases:** `character(char|chars)`
Aliases live inside `()` and are separated with `|`. The canonical name is what appears in folder names and UI suggestions when any of the aliases are detected. Matching is case-insensitive.
## Use `{first_tag}` in Path Templates
When your path template contains `{first_tag}`, the app picks a folder name based on your priority list and the models own tags:
- It checks the priority list from top to bottom. If a canonical tag or any of its aliases appear in the model tags, that canonical name becomes the folder name.
- If no priority tags are found but the model has tags, the very first model tag is used.
- If the model has no tags at all, the folder falls back to `no tags`.
### Example
With a template like `/{model_type}/{first_tag}` and the priority entry list `character(char|chars), style(anime|toon)`:
| Model Tags | Folder Name | Why |
| --- | --- | --- |
| `["chars", "female"]` | `character` | `chars` matches the `character` alias, so the canonical wins. |
| `["anime", "portrait"]` | `style` | `anime` hits the `style` entry, so its canonical label is used. |
| `["portrait", "bw"]` | `portrait` | No priority match, so the first model tag is used. |
| `[]` | `no tags` | Nothing to match, so the fallback is applied. |
## 3. Save the Settings
After editing the entry list, press **Enter** to save. Use **Shift+Enter** whenever you need a new line. Clicking outside the field also saves automatically. A success toast confirms the update.
## Examples
| Goal | Entry List |
| --- | --- |
| Prefer people over styles | `character, portraits, style(anime\|toon)` |
| Group sci-fi variants | `sci-fi(scifi\|science_fiction), cyberpunk(cyber\|punk)` |
| Alias shorthand tags | `realistic(real\|realisim), photorealistic(photo_real)` |
## Tips
- Keep canonical names short and meaningful—they become folder names.
- Place the most important categories first; the first match wins.
- Avoid duplicate canonical names within the same list; only the first instance is used.
## Troubleshooting
- **Unexpected folder name?** Check that the canonical name you want is placed before other matches.
- **Alias not working?** Ensure the alias is inside parentheses and separated with `|`, e.g. `character(char|chars)`.
- **Validation error?** Look for missing parentheses or stray commas. Each entry must follow the `canonical(alias|alias)` pattern or just `canonical`.
With these basics you can quickly adapt Priority Tags to match your librarys organization style.

View File

@@ -0,0 +1,26 @@
# Backend Test Coverage Notes
## Pytest Execution
- Command: `python -m pytest`
- Result: All 283 collected tests passed in the current environment.
- Coverage tooling (``pytest-cov``/``coverage``) is unavailable in the offline sandbox, so line-level metrics could not be generated. The earlier attempt to install ``pytest-cov`` failed because the package index cannot be reached from the container.
## High-Priority Gaps to Address
### 1. Standalone server bootstrapping
* **Source:** [`standalone.py`](../../standalone.py)
* **Why it matters:** The standalone entry point wires together the aiohttp application, static asset routes, model-route registration, and configuration validation. None of these behaviours are covered by automated tests, leaving regressions in bootstrapping logic undetected.
* **Suggested coverage:** Add integration-style tests that instantiate `StandaloneServer`/`StandaloneLoraManager` with temporary settings and assert that routes (HTTP + websocket) are registered, configuration warnings fire for missing paths, and the mock ComfyUI shims behave as expected.
### 2. Model service registration factory
* **Source:** [`py/services/model_service_factory.py`](../../py/services/model_service_factory.py)
* **Why it matters:** The factory coordinates which model services and routes the API exposes, including error handling when unknown model types are requested. No current tests verify registration, memoization of route instances, or the logging path on failures.
* **Suggested coverage:** Unit tests that exercise `register_model_type`, `get_route_instance`, error branches in `get_service_class`/`get_route_class`, and `setup_all_routes` when a route setup raises. Use lightweight fakes to confirm the logger is called and state is cleared via `clear_registrations`.
### 3. Server-side i18n helper
* **Source:** [`py/services/server_i18n.py`](../../py/services/server_i18n.py)
* **Why it matters:** Template rendering relies on the `ServerI18nManager` to load locale JSON, perform key lookups, and format parameters. The fallback logic (dot-notation lookup, English fallbacks, placeholder substitution) is untested, so malformed locale files or regressions in placeholder handling would slip through.
* **Suggested coverage:** Tests that load fixture locale dictionaries, assert `set_locale` fallbacks, verify nested key resolution and placeholder substitution, and ensure missing keys return the original identifier.
## Next Steps
Prioritize creating focused unit tests around these modules, then re-run pytest once coverage tooling is available to confirm the new tests close the identified gaps.

View File

@@ -31,7 +31,8 @@
"japanese": "日本語",
"korean": "한국어",
"french": "Français",
"spanish": "Español"
"spanish": "Español",
"Hebrew": "עברית"
},
"fileSize": {
"zero": "0 Bytes",
@@ -187,11 +188,18 @@
"civitaiApiKey": "Civitai API Key",
"civitaiApiKeyPlaceholder": "Geben Sie Ihren Civitai API Key ein",
"civitaiApiKeyHelp": "Wird für die Authentifizierung beim Herunterladen von Modellen von Civitai verwendet",
"openSettingsFileLocation": {
"label": "Einstellungsordner öffnen",
"tooltip": "Den Ordner mit der settings.json öffnen",
"success": "Einstellungsordner geöffnet",
"failed": "Einstellungsordner konnte nicht geöffnet werden"
},
"sections": {
"contentFiltering": "Inhaltsfilterung",
"videoSettings": "Video-Einstellungen",
"layoutSettings": "Layout-Einstellungen",
"folderSettings": "Ordner-Einstellungen",
"priorityTags": "Prioritäts-Tags",
"downloadPathTemplates": "Download-Pfad-Vorlagen",
"exampleImages": "Beispielbilder",
"misc": "Verschiedenes",
@@ -217,9 +225,9 @@
},
"displayDensityHelp": "Wählen Sie, wie viele Karten pro Zeile angezeigt werden sollen:",
"displayDensityDetails": {
"default": "Standard: 5 (1080p), 6 (2K), 8 (4K)",
"medium": "Mittel: 6 (1080p), 7 (2K), 9 (4K)",
"compact": "Kompakt: 7 (1080p), 8 (2K), 10 (4K)"
"default": "5 (1080p), 6 (2K), 8 (4K)",
"medium": "6 (1080p), 7 (2K), 9 (4K)",
"compact": "7 (1080p), 8 (2K), 10 (4K)"
},
"displayDensityWarning": "Warnung: Höhere Dichten können bei Systemen mit begrenzten Ressourcen zu Performance-Problemen führen.",
"cardInfoDisplay": "Karten-Info-Anzeige",
@@ -229,11 +237,25 @@
},
"cardInfoDisplayHelp": "Wählen Sie, wann Modellinformationen und Aktionsschaltflächen angezeigt werden sollen:",
"cardInfoDisplayDetails": {
"always": "Immer sichtbar: Kopf- und Fußzeilen sind immer sichtbar",
"hover": "Bei Hover anzeigen: Kopf- und Fußzeilen erscheinen nur beim Darüberfahren mit der Maus"
"always": "Kopf- und Fußzeilen sind immer sichtbar",
"hover": "Kopf- und Fußzeilen erscheinen nur beim Darüberfahren mit der Maus"
},
"modelNameDisplay": "Anzeige des Modellnamens",
"modelNameDisplayOptions": {
"modelName": "Modellname",
"fileName": "Dateiname"
},
"modelNameDisplayHelp": "Wählen Sie aus, was in der Fußzeile der Modellkarte angezeigt werden soll:",
"modelNameDisplayDetails": {
"modelName": "Den beschreibenden Namen des Modells anzeigen",
"fileName": "Den tatsächlichen Dateinamen auf der Festplatte anzeigen"
}
},
"folderSettings": {
"activeLibrary": "Aktive Bibliothek",
"activeLibraryHelp": "Zwischen den konfigurierten Bibliotheken wechseln, um die Standardordner zu aktualisieren. Eine Änderung der Auswahl lädt die Seite neu.",
"loadingLibraries": "Bibliotheken werden geladen...",
"noLibraries": "Keine Bibliotheken konfiguriert",
"defaultLoraRoot": "Standard-LoRA-Stammordner",
"defaultLoraRootHelp": "Legen Sie den Standard-LoRA-Stammordner für Downloads, Importe und Verschiebungen fest",
"defaultCheckpointRoot": "Standard-Checkpoint-Stammordner",
@@ -242,6 +264,26 @@
"defaultEmbeddingRootHelp": "Legen Sie den Standard-Embedding-Stammordner für Downloads, Importe und Verschiebungen fest",
"noDefault": "Kein Standard"
},
"priorityTags": {
"title": "Prioritäts-Tags",
"description": "Passen Sie die Tag-Prioritätsreihenfolge für jeden Modelltyp an (z. B. character, concept, style(toon|toon_style))",
"placeholder": "character, concept, style(toon|toon_style)",
"helpLinkLabel": "Prioritäts-Tags-Hilfe öffnen",
"modelTypes": {
"lora": "LoRA",
"checkpoint": "Checkpoint",
"embedding": "Embedding"
},
"saveSuccess": "Prioritäts-Tags aktualisiert.",
"saveError": "Prioritäts-Tags konnten nicht aktualisiert werden.",
"loadingSuggestions": "Lade Vorschläge...",
"validation": {
"missingClosingParen": "Eintrag {index} fehlt eine schließende Klammer.",
"missingCanonical": "Eintrag {index} muss einen kanonischen Tag-Namen enthalten.",
"duplicateCanonical": "Der kanonische Tag \"{tag}\" kommt mehrfach vor.",
"unknown": "Ungültige Prioritäts-Tag-Konfiguration."
}
},
"downloadPathTemplates": {
"title": "Download-Pfad-Vorlagen",
"help": "Konfigurieren Sie Ordnerstrukturen für verschiedene Modelltypen beim Herunterladen von Civitai.",
@@ -382,6 +424,7 @@
"viewSelected": "Auswahl anzeigen",
"addTags": "Allen Tags hinzufügen",
"setBaseModel": "Basis-Modell für alle festlegen",
"setContentRating": "Inhaltsbewertung für alle festlegen",
"copyAll": "Alle Syntax kopieren",
"refreshAll": "Alle Metadaten aktualisieren",
"moveAll": "Alle in Ordner verschieben",
@@ -517,13 +560,19 @@
"title": "Embedding-Modelle"
},
"sidebar": {
"modelRoot": "Modell-Stammverzeichnis",
"modelRoot": "Stammverzeichnis",
"collapseAll": "Alle Ordner einklappen",
"pinSidebar": "Sidebar anheften",
"unpinSidebar": "Sidebar lösen",
"switchToListView": "Zur Listenansicht wechseln",
"switchToTreeView": "Zur Baumansicht wechseln",
"collapseAllDisabled": "Im Listenmodus nicht verfügbar"
"recursiveOn": "Unterordner durchsuchen",
"recursiveOff": "Nur aktuellen Ordner durchsuchen",
"recursiveUnavailable": "Rekursive Suche ist nur in der Baumansicht verfügbar",
"collapseAllDisabled": "Im Listenmodus nicht verfügbar",
"dragDrop": {
"unableToResolveRoot": "Zielpfad für das Verschieben konnte nicht ermittelt werden."
}
},
"statistics": {
"title": "Statistiken",
@@ -598,6 +647,14 @@
"downloadedPreview": "Vorschaubild heruntergeladen",
"downloadingFile": "{type}-Datei wird heruntergeladen",
"finalizing": "Download wird abgeschlossen..."
},
"progress": {
"currentFile": "Aktuelle Datei:",
"downloading": "Wird heruntergeladen: {name}",
"transferred": "Heruntergeladen: {downloaded} / {total}",
"transferredSimple": "Heruntergeladen: {downloaded}",
"transferredUnknown": "Heruntergeladen: --",
"speed": "Geschwindigkeit: {speed}"
}
},
"move": {
@@ -606,6 +663,7 @@
"contentRating": {
"title": "Inhaltsbewertung festlegen",
"current": "Aktuell",
"multiple": "Mehrere Werte",
"levels": {
"pg": "PG",
"pg13": "PG13",
@@ -1084,6 +1142,10 @@
"bulkBaseModelUpdateSuccess": "Basis-Modell erfolgreich für {count} Modell(e) aktualisiert",
"bulkBaseModelUpdatePartial": "{success} Modelle aktualisiert, {failed} fehlgeschlagen",
"bulkBaseModelUpdateFailed": "Aktualisierung des Basis-Modells für ausgewählte Modelle fehlgeschlagen",
"bulkContentRatingUpdating": "Inhaltsbewertung wird für {count} Modell(e) aktualisiert...",
"bulkContentRatingSet": "Inhaltsbewertung auf {level} für {count} Modell(e) gesetzt",
"bulkContentRatingPartial": "Inhaltsbewertung auf {level} für {success} Modell(e) gesetzt, {failed} fehlgeschlagen",
"bulkContentRatingFailed": "Inhaltsbewertung für ausgewählte Modelle konnte nicht aktualisiert werden",
"invalidCharactersRemoved": "Ungültige Zeichen aus Dateiname entfernt",
"filenameCannotBeEmpty": "Dateiname darf nicht leer sein",
"renameFailed": "Fehler beim Umbenennen der Datei: {message}",
@@ -1118,6 +1180,8 @@
"compactModeToggled": "Kompakt-Modus {state}",
"settingSaveFailed": "Fehler beim Speichern der Einstellung: {message}",
"displayDensitySet": "Anzeige-Dichte auf {density} gesetzt",
"libraryLoadFailed": "Failed to load libraries: {message}",
"libraryActivateFailed": "Failed to activate library: {message}",
"languageChangeFailed": "Fehler beim Ändern der Sprache: {message}",
"cacheCleared": "Cache-Dateien wurden erfolgreich gelöscht. Cache wird bei der nächsten Aktion neu aufgebaut.",
"cacheClearFailed": "Fehler beim Löschen des Caches: {error}",
@@ -1191,6 +1255,8 @@
"pauseFailed": "Fehler beim Pausieren des Downloads: {error}",
"downloadResumed": "Download fortgesetzt",
"resumeFailed": "Fehler beim Fortsetzen des Downloads: {error}",
"downloadStopped": "Download abgebrochen",
"stopFailed": "Download konnte nicht abgebrochen werden: {error}",
"deleted": "Beispielbild gelöscht",
"deleteFailed": "Fehler beim Löschen des Beispielbilds",
"setPreviewFailed": "Fehler beim Setzen des Vorschaubilds"
@@ -1237,6 +1303,12 @@
"refreshNow": "Jetzt aktualisieren",
"refreshingIn": "Aktualisierung in",
"seconds": "Sekunden"
},
"communitySupport": {
"title": "Keep LoRA Manager Thriving with Your Support ❤️",
"content": "LoRA Manager is a passion project maintained full-time by a solo developer. Your support on Ko-fi helps cover development costs, keeps new updates coming, and unlocks a license key for the LM Civitai Extension as a thank-you gift. Every contribution truly makes a difference.",
"supportCta": "Support on Ko-fi",
"learnMore": "LM Civitai Extension Tutorial"
}
}
}

View File

@@ -31,7 +31,8 @@
"japanese": "日本語",
"korean": "한국어",
"french": "Français",
"spanish": "Español"
"spanish": "Español",
"Hebrew": "עברית"
},
"fileSize": {
"zero": "0 Bytes",
@@ -187,11 +188,18 @@
"civitaiApiKey": "Civitai API Key",
"civitaiApiKeyPlaceholder": "Enter your Civitai API key",
"civitaiApiKeyHelp": "Used for authentication when downloading models from Civitai",
"openSettingsFileLocation": {
"label": "Open settings folder",
"tooltip": "Open the folder containing settings.json",
"success": "Opened settings.json folder",
"failed": "Failed to open settings.json folder"
},
"sections": {
"contentFiltering": "Content Filtering",
"videoSettings": "Video Settings",
"layoutSettings": "Layout Settings",
"folderSettings": "Folder Settings",
"priorityTags": "Priority Tags",
"downloadPathTemplates": "Download Path Templates",
"exampleImages": "Example Images",
"misc": "Misc.",
@@ -217,9 +225,9 @@
},
"displayDensityHelp": "Choose how many cards to display per row:",
"displayDensityDetails": {
"default": "Default: 5 (1080p), 6 (2K), 8 (4K)",
"medium": "Medium: 6 (1080p), 7 (2K), 9 (4K)",
"compact": "Compact: 7 (1080p), 8 (2K), 10 (4K)"
"default": "5 (1080p), 6 (2K), 8 (4K)",
"medium": "6 (1080p), 7 (2K), 9 (4K)",
"compact": "7 (1080p), 8 (2K), 10 (4K)"
},
"displayDensityWarning": "Warning: Higher densities may cause performance issues on systems with limited resources.",
"cardInfoDisplay": "Card Info Display",
@@ -229,11 +237,25 @@
},
"cardInfoDisplayHelp": "Choose when to display model information and action buttons:",
"cardInfoDisplayDetails": {
"always": "Always Visible: Headers and footers are always visible",
"hover": "Reveal on Hover: Headers and footers only appear when hovering over a card"
"always": "Headers and footers are always visible",
"hover": "Headers and footers only appear when hovering over a card"
},
"modelNameDisplay": "Model Name Display",
"modelNameDisplayOptions": {
"modelName": "Model Name",
"fileName": "File Name"
},
"modelNameDisplayHelp": "Choose what to display in the model card footer:",
"modelNameDisplayDetails": {
"modelName": "Display the model's descriptive name",
"fileName": "Display the actual file name on disk"
}
},
"folderSettings": {
"activeLibrary": "Active Library",
"activeLibraryHelp": "Switch between configured libraries to update default folders. Changing the selection reloads the page.",
"loadingLibraries": "Loading libraries...",
"noLibraries": "No libraries configured",
"defaultLoraRoot": "Default LoRA Root",
"defaultLoraRootHelp": "Set the default LoRA root directory for downloads, imports and moves",
"defaultCheckpointRoot": "Default Checkpoint Root",
@@ -242,6 +264,26 @@
"defaultEmbeddingRootHelp": "Set the default embedding root directory for downloads, imports and moves",
"noDefault": "No Default"
},
"priorityTags": {
"title": "Priority Tags",
"description": "Customize the tag priority order for each model type (e.g., character, concept, style(toon|toon_style))",
"placeholder": "character, concept, style(toon|toon_style)",
"helpLinkLabel": "Open priority tags help",
"modelTypes": {
"lora": "LoRA",
"checkpoint": "Checkpoint",
"embedding": "Embedding"
},
"saveSuccess": "Priority tags updated.",
"saveError": "Failed to update priority tags.",
"loadingSuggestions": "Loading suggestions...",
"validation": {
"missingClosingParen": "Entry {index} is missing a closing parenthesis.",
"missingCanonical": "Entry {index} must include a canonical tag name.",
"duplicateCanonical": "The canonical tag \"{tag}\" appears more than once.",
"unknown": "Invalid priority tag configuration."
}
},
"downloadPathTemplates": {
"title": "Download Path Templates",
"help": "Configure folder structures for different model types when downloading from Civitai.",
@@ -380,13 +422,14 @@
"selected": "{count} selected",
"selectedSuffix": "selected",
"viewSelected": "View Selected",
"addTags": "Add Tags to All",
"setBaseModel": "Set Base Model for All",
"copyAll": "Copy All Syntax",
"refreshAll": "Refresh All Metadata",
"moveAll": "Move All to Folder",
"addTags": "Add Tags to Selected",
"setBaseModel": "Set Base Model for Selected",
"setContentRating": "Set Content Rating for Selected",
"copyAll": "Copy Selected Syntax",
"refreshAll": "Refresh Selected Metadata",
"moveAll": "Move Selected to Folder",
"autoOrganize": "Auto-Organize Selected",
"deleteAll": "Delete All Models",
"deleteAll": "Delete Selected Models",
"clear": "Clear Selection",
"autoOrganizeProgress": {
"initializing": "Initializing auto-organize...",
@@ -517,13 +560,19 @@
"title": "Embedding Models"
},
"sidebar": {
"modelRoot": "Model Root",
"modelRoot": "Root",
"collapseAll": "Collapse All Folders",
"pinSidebar": "Pin Sidebar",
"unpinSidebar": "Unpin Sidebar",
"switchToListView": "Switch to List View",
"switchToTreeView": "Switch to Tree View",
"collapseAllDisabled": "Not available in list view"
"recursiveOn": "Search subfolders",
"recursiveOff": "Search current folder only",
"recursiveUnavailable": "Recursive search is available in tree view only",
"collapseAllDisabled": "Not available in list view",
"dragDrop": {
"unableToResolveRoot": "Unable to determine destination path for move."
}
},
"statistics": {
"title": "Statistics",
@@ -598,6 +647,14 @@
"downloadedPreview": "Downloaded preview image",
"downloadingFile": "Downloading {type} file",
"finalizing": "Finalizing download..."
},
"progress": {
"currentFile": "Current file:",
"downloading": "Downloading: {name}",
"transferred": "Transferred: {downloaded} / {total}",
"transferredSimple": "Transferred: {downloaded}",
"transferredUnknown": "Transferred: --",
"speed": "Speed: {speed}"
}
},
"move": {
@@ -606,6 +663,7 @@
"contentRating": {
"title": "Set Content Rating",
"current": "Current",
"multiple": "Multiple values",
"levels": {
"pg": "PG",
"pg13": "PG13",
@@ -1084,6 +1142,10 @@
"bulkBaseModelUpdateSuccess": "Successfully updated base model for {count} model(s)",
"bulkBaseModelUpdatePartial": "Updated {success} model(s), failed {failed} model(s)",
"bulkBaseModelUpdateFailed": "Failed to update base model for selected models",
"bulkContentRatingUpdating": "Updating content rating for {count} model(s)...",
"bulkContentRatingSet": "Set content rating to {level} for {count} model(s)",
"bulkContentRatingPartial": "Set content rating to {level} for {success} model(s), {failed} failed",
"bulkContentRatingFailed": "Failed to update content rating for selected models",
"invalidCharactersRemoved": "Invalid characters removed from filename",
"filenameCannotBeEmpty": "File name cannot be empty",
"renameFailed": "Failed to rename file: {message}",
@@ -1118,6 +1180,8 @@
"compactModeToggled": "Compact Mode {state}",
"settingSaveFailed": "Failed to save setting: {message}",
"displayDensitySet": "Display Density set to {density}",
"libraryLoadFailed": "Failed to load libraries: {message}",
"libraryActivateFailed": "Failed to activate library: {message}",
"languageChangeFailed": "Failed to change language: {message}",
"cacheCleared": "Cache files have been cleared successfully. Cache will rebuild on next action.",
"cacheClearFailed": "Failed to clear cache: {error}",
@@ -1191,6 +1255,8 @@
"pauseFailed": "Failed to pause download: {error}",
"downloadResumed": "Download resumed",
"resumeFailed": "Failed to resume download: {error}",
"downloadStopped": "Download cancelled",
"stopFailed": "Failed to cancel download: {error}",
"deleted": "Example image deleted",
"deleteFailed": "Failed to delete example image",
"setPreviewFailed": "Failed to set preview image"
@@ -1237,6 +1303,12 @@
"refreshNow": "Refresh Now",
"refreshingIn": "Refreshing in",
"seconds": "seconds"
},
"communitySupport": {
"title": "Keep LoRA Manager Thriving with Your Support ❤️",
"content": "LoRA Manager is a passion project maintained full-time by a solo developer. Your support on Ko-fi helps cover development costs, keeps new updates coming, and unlocks a license key for the LM Civitai Extension as a thank-you gift. Every contribution truly makes a difference.",
"supportCta": "Support on Ko-fi",
"learnMore": "LM Civitai Extension Tutorial"
}
}
}

View File

@@ -31,7 +31,8 @@
"japanese": "日本語",
"korean": "한국어",
"french": "Français",
"spanish": "Español"
"spanish": "Español",
"Hebrew": "עברית"
},
"fileSize": {
"zero": "0 Bytes",
@@ -187,11 +188,18 @@
"civitaiApiKey": "Clave API de Civitai",
"civitaiApiKeyPlaceholder": "Introduce tu clave API de Civitai",
"civitaiApiKeyHelp": "Utilizada para autenticación al descargar modelos de Civitai",
"openSettingsFileLocation": {
"label": "Abrir carpeta de ajustes",
"tooltip": "Abrir la carpeta que contiene settings.json",
"success": "Carpeta de settings.json abierta",
"failed": "No se pudo abrir la carpeta de settings.json"
},
"sections": {
"contentFiltering": "Filtrado de contenido",
"videoSettings": "Configuración de video",
"layoutSettings": "Configuración de diseño",
"folderSettings": "Configuración de carpetas",
"priorityTags": "Etiquetas prioritarias",
"downloadPathTemplates": "Plantillas de rutas de descarga",
"exampleImages": "Imágenes de ejemplo",
"misc": "Varios",
@@ -217,9 +225,9 @@
},
"displayDensityHelp": "Elige cuántas tarjetas mostrar por fila:",
"displayDensityDetails": {
"default": "Predeterminado: 5 (1080p), 6 (2K), 8 (4K)",
"medium": "Medio: 6 (1080p), 7 (2K), 9 (4K)",
"compact": "Compacto: 7 (1080p), 8 (2K), 10 (4K)"
"default": "5 (1080p), 6 (2K), 8 (4K)",
"medium": "6 (1080p), 7 (2K), 9 (4K)",
"compact": "7 (1080p), 8 (2K), 10 (4K)"
},
"displayDensityWarning": "Advertencia: Densidades más altas pueden causar problemas de rendimiento en sistemas con recursos limitados.",
"cardInfoDisplay": "Visualización de información de tarjeta",
@@ -229,11 +237,25 @@
},
"cardInfoDisplayHelp": "Elige cuándo mostrar información del modelo y botones de acción:",
"cardInfoDisplayDetails": {
"always": "Siempre visible: Los encabezados y pies de página siempre son visibles",
"hover": "Mostrar al pasar el ratón: Los encabezados y pies de página solo aparecen al pasar el ratón sobre una tarjeta"
"always": "Los encabezados y pies de página siempre son visibles",
"hover": "Los encabezados y pies de página solo aparecen al pasar el ratón sobre una tarjeta"
},
"modelNameDisplay": "Visualización del nombre del modelo",
"modelNameDisplayOptions": {
"modelName": "Nombre del modelo",
"fileName": "Nombre del archivo"
},
"modelNameDisplayHelp": "Elige qué mostrar en el pie de la tarjeta del modelo:",
"modelNameDisplayDetails": {
"modelName": "Mostrar el nombre descriptivo del modelo",
"fileName": "Mostrar el nombre real del archivo en el disco"
}
},
"folderSettings": {
"activeLibrary": "Biblioteca activa",
"activeLibraryHelp": "Alterna entre las bibliotecas configuradas para actualizar las carpetas predeterminadas. Cambiar la selección recarga la página.",
"loadingLibraries": "Cargando bibliotecas...",
"noLibraries": "No hay bibliotecas configuradas",
"defaultLoraRoot": "Raíz predeterminada de LoRA",
"defaultLoraRootHelp": "Establecer el directorio raíz predeterminado de LoRA para descargas, importaciones y movimientos",
"defaultCheckpointRoot": "Raíz predeterminada de checkpoint",
@@ -242,6 +264,26 @@
"defaultEmbeddingRootHelp": "Establecer el directorio raíz predeterminado de embedding para descargas, importaciones y movimientos",
"noDefault": "Sin predeterminado"
},
"priorityTags": {
"title": "Etiquetas prioritarias",
"description": "Personaliza el orden de prioridad de etiquetas para cada tipo de modelo (p. ej., character, concept, style(toon|toon_style))",
"placeholder": "character, concept, style(toon|toon_style)",
"helpLinkLabel": "Abrir ayuda de etiquetas prioritarias",
"modelTypes": {
"lora": "LoRA",
"checkpoint": "Checkpoint",
"embedding": "Embedding"
},
"saveSuccess": "Etiquetas prioritarias actualizadas.",
"saveError": "Error al actualizar las etiquetas prioritarias.",
"loadingSuggestions": "Cargando sugerencias...",
"validation": {
"missingClosingParen": "A la entrada {index} le falta un paréntesis de cierre.",
"missingCanonical": "La entrada {index} debe incluir un nombre de etiqueta canónica.",
"duplicateCanonical": "La etiqueta canónica \"{tag}\" aparece más de una vez.",
"unknown": "Configuración de etiquetas prioritarias no válida."
}
},
"downloadPathTemplates": {
"title": "Plantillas de rutas de descarga",
"help": "Configurar estructuras de carpetas para diferentes tipos de modelos al descargar de Civitai.",
@@ -382,6 +424,7 @@
"viewSelected": "Ver seleccionados",
"addTags": "Añadir etiquetas a todos",
"setBaseModel": "Establecer modelo base para todos",
"setContentRating": "Establecer clasificación de contenido para todos",
"copyAll": "Copiar toda la sintaxis",
"refreshAll": "Actualizar todos los metadatos",
"moveAll": "Mover todos a carpeta",
@@ -517,13 +560,19 @@
"title": "Modelos embedding"
},
"sidebar": {
"modelRoot": "Raíz del modelo",
"modelRoot": "Raíz",
"collapseAll": "Colapsar todas las carpetas",
"pinSidebar": "Fijar barra lateral",
"unpinSidebar": "Desfijar barra lateral",
"switchToListView": "Cambiar a vista de lista",
"switchToTreeView": "Cambiar a vista de árbol",
"collapseAllDisabled": "No disponible en vista de lista"
"recursiveOn": "Buscar en subcarpetas",
"recursiveOff": "Buscar solo en la carpeta actual",
"recursiveUnavailable": "La búsqueda recursiva solo está disponible en la vista en árbol",
"collapseAllDisabled": "No disponible en vista de lista",
"dragDrop": {
"unableToResolveRoot": "No se puede determinar la ruta de destino para el movimiento."
}
},
"statistics": {
"title": "Estadísticas",
@@ -598,6 +647,14 @@
"downloadedPreview": "Imagen de vista previa descargada",
"downloadingFile": "Descargando archivo de {type}",
"finalizing": "Finalizando descarga..."
},
"progress": {
"currentFile": "Archivo actual:",
"downloading": "Descargando: {name}",
"transferred": "Descargado: {downloaded} / {total}",
"transferredSimple": "Descargado: {downloaded}",
"transferredUnknown": "Descargado: --",
"speed": "Velocidad: {speed}"
}
},
"move": {
@@ -606,6 +663,7 @@
"contentRating": {
"title": "Establecer clasificación de contenido",
"current": "Actual",
"multiple": "Valores múltiples",
"levels": {
"pg": "PG",
"pg13": "PG13",
@@ -1084,6 +1142,10 @@
"bulkBaseModelUpdateSuccess": "Modelo base actualizado exitosamente para {count} modelo(s)",
"bulkBaseModelUpdatePartial": "Actualizados {success} modelo(s), fallaron {failed} modelo(s)",
"bulkBaseModelUpdateFailed": "Error al actualizar el modelo base para los modelos seleccionados",
"bulkContentRatingUpdating": "Actualizando la clasificación de contenido para {count} modelo(s)...",
"bulkContentRatingSet": "Clasificación de contenido establecida en {level} para {count} modelo(s)",
"bulkContentRatingPartial": "Clasificación de contenido establecida en {level} para {success} modelo(s), {failed} fallaron",
"bulkContentRatingFailed": "No se pudo actualizar la clasificación de contenido para los modelos seleccionados",
"invalidCharactersRemoved": "Caracteres inválidos eliminados del nombre de archivo",
"filenameCannotBeEmpty": "El nombre de archivo no puede estar vacío",
"renameFailed": "Error al renombrar archivo: {message}",
@@ -1118,6 +1180,8 @@
"compactModeToggled": "Modo compacto {state}",
"settingSaveFailed": "Error al guardar configuración: {message}",
"displayDensitySet": "Densidad de visualización establecida a {density}",
"libraryLoadFailed": "Failed to load libraries: {message}",
"libraryActivateFailed": "Failed to activate library: {message}",
"languageChangeFailed": "Error al cambiar idioma: {message}",
"cacheCleared": "Archivos de caché limpiados exitosamente. La caché se reconstruirá en la próxima acción.",
"cacheClearFailed": "Error al limpiar caché: {error}",
@@ -1191,6 +1255,8 @@
"pauseFailed": "Error al pausar descarga: {error}",
"downloadResumed": "Descarga reanudada",
"resumeFailed": "Error al reanudar descarga: {error}",
"downloadStopped": "Descarga cancelada",
"stopFailed": "Error al cancelar descarga: {error}",
"deleted": "Imagen de ejemplo eliminada",
"deleteFailed": "Error al eliminar imagen de ejemplo",
"setPreviewFailed": "Error al establecer imagen de vista previa"
@@ -1237,6 +1303,12 @@
"refreshNow": "Actualizar ahora",
"refreshingIn": "Actualizando en",
"seconds": "segundos"
},
"communitySupport": {
"title": "Keep LoRA Manager Thriving with Your Support ❤️",
"content": "LoRA Manager is a passion project maintained full-time by a solo developer. Your support on Ko-fi helps cover development costs, keeps new updates coming, and unlocks a license key for the LM Civitai Extension as a thank-you gift. Every contribution truly makes a difference.",
"supportCta": "Support on Ko-fi",
"learnMore": "LM Civitai Extension Tutorial"
}
}
}

View File

@@ -31,7 +31,8 @@
"japanese": "日本語",
"korean": "한국어",
"french": "Français",
"spanish": "Español"
"spanish": "Español",
"Hebrew": "עברית"
},
"fileSize": {
"zero": "0 Octets",
@@ -187,6 +188,12 @@
"civitaiApiKey": "Clé API Civitai",
"civitaiApiKeyPlaceholder": "Entrez votre clé API Civitai",
"civitaiApiKeyHelp": "Utilisée pour l'authentification lors du téléchargement de modèles depuis Civitai",
"openSettingsFileLocation": {
"label": "Ouvrir le dossier des paramètres",
"tooltip": "Ouvrir le dossier contenant settings.json",
"success": "Dossier settings.json ouvert",
"failed": "Impossible d'ouvrir le dossier settings.json"
},
"sections": {
"contentFiltering": "Filtrage du contenu",
"videoSettings": "Paramètres vidéo",
@@ -196,7 +203,8 @@
"exampleImages": "Images d'exemple",
"misc": "Divers",
"metadataArchive": "Base de données d'archive des métadonnées",
"proxySettings": "Paramètres du proxy"
"proxySettings": "Paramètres du proxy",
"priorityTags": "Étiquettes prioritaires"
},
"contentFiltering": {
"blurNsfwContent": "Flouter le contenu NSFW",
@@ -217,9 +225,9 @@
},
"displayDensityHelp": "Choisissez combien de cartes afficher par ligne :",
"displayDensityDetails": {
"default": "Par défaut : 5 (1080p), 6 (2K), 8 (4K)",
"medium": "Moyen : 6 (1080p), 7 (2K), 9 (4K)",
"compact": "Compact : 7 (1080p), 8 (2K), 10 (4K)"
"default": "5 (1080p), 6 (2K), 8 (4K)",
"medium": "6 (1080p), 7 (2K), 9 (4K)",
"compact": "7 (1080p), 8 (2K), 10 (4K)"
},
"displayDensityWarning": "Attention : Des densités plus élevées peuvent causer des problèmes de performance sur les systèmes avec des ressources limitées.",
"cardInfoDisplay": "Affichage des informations de carte",
@@ -229,11 +237,25 @@
},
"cardInfoDisplayHelp": "Choisissez quand afficher les informations du modèle et les boutons d'action :",
"cardInfoDisplayDetails": {
"always": "Toujours visible : Les en-têtes et pieds de page sont toujours visibles",
"hover": "Révéler au survol : Les en-têtes et pieds de page n'apparaissent qu'au survol d'une carte"
"always": "Les en-têtes et pieds de page sont toujours visibles",
"hover": "Les en-têtes et pieds de page n'apparaissent qu'au survol d'une carte"
},
"modelNameDisplay": "Affichage du nom du modèle",
"modelNameDisplayOptions": {
"modelName": "Nom du modèle",
"fileName": "Nom du fichier"
},
"modelNameDisplayHelp": "Choisissez ce qui doit être affiché dans le pied de page de la carte du modèle :",
"modelNameDisplayDetails": {
"modelName": "Afficher le nom descriptif du modèle",
"fileName": "Afficher le nom réel du fichier sur le disque"
}
},
"folderSettings": {
"activeLibrary": "Bibliothèque active",
"activeLibraryHelp": "Basculer entre les bibliothèques configurées pour mettre à jour les dossiers par défaut. Changer la sélection recharge la page.",
"loadingLibraries": "Chargement des bibliothèques...",
"noLibraries": "Aucune bibliothèque configurée",
"defaultLoraRoot": "Racine LoRA par défaut",
"defaultLoraRootHelp": "Définir le répertoire racine LoRA par défaut pour les téléchargements, imports et déplacements",
"defaultCheckpointRoot": "Racine Checkpoint par défaut",
@@ -334,6 +356,26 @@
"proxyPassword": "Mot de passe (optionnel)",
"proxyPasswordPlaceholder": "mot_de_passe",
"proxyPasswordHelp": "Mot de passe pour l'authentification proxy (si nécessaire)"
},
"priorityTags": {
"title": "Étiquettes prioritaires",
"description": "Personnalisez l'ordre de priorité des étiquettes pour chaque type de modèle (par ex. : character, concept, style(toon|toon_style))",
"placeholder": "character, concept, style(toon|toon_style)",
"helpLinkLabel": "Ouvrir l'aide sur les étiquettes prioritaires",
"modelTypes": {
"lora": "LoRA",
"checkpoint": "Checkpoint",
"embedding": "Embedding"
},
"saveSuccess": "Étiquettes prioritaires mises à jour.",
"saveError": "Échec de la mise à jour des étiquettes prioritaires.",
"loadingSuggestions": "Chargement des suggestions...",
"validation": {
"missingClosingParen": "L'entrée {index} n'a pas de parenthèse fermante.",
"missingCanonical": "L'entrée {index} doit inclure un nom d'étiquette canonique.",
"duplicateCanonical": "L'étiquette canonique \"{tag}\" apparaît plusieurs fois.",
"unknown": "Configuration d'étiquettes prioritaires invalide."
}
}
},
"loras": {
@@ -382,6 +424,7 @@
"viewSelected": "Voir la sélection",
"addTags": "Ajouter des tags à tous",
"setBaseModel": "Définir le modèle de base pour tous",
"setContentRating": "Définir la classification du contenu pour tous",
"copyAll": "Copier toute la syntaxe",
"refreshAll": "Actualiser toutes les métadonnées",
"moveAll": "Déplacer tout vers un dossier",
@@ -517,13 +560,19 @@
"title": "Modèles Embedding"
},
"sidebar": {
"modelRoot": "Racine du modèle",
"modelRoot": "Racine",
"collapseAll": "Réduire tous les dossiers",
"pinSidebar": "Épingler la barre latérale",
"unpinSidebar": "Désépingler la barre latérale",
"switchToListView": "Passer en vue liste",
"switchToTreeView": "Passer en vue arborescence",
"collapseAllDisabled": "Non disponible en vue liste"
"recursiveOn": "Rechercher dans les sous-dossiers",
"recursiveOff": "Rechercher uniquement dans le dossier actuel",
"recursiveUnavailable": "La recherche récursive n'est disponible qu'en vue arborescente",
"collapseAllDisabled": "Non disponible en vue liste",
"dragDrop": {
"unableToResolveRoot": "Impossible de déterminer le chemin de destination pour le déplacement."
}
},
"statistics": {
"title": "Statistiques",
@@ -598,6 +647,14 @@
"downloadedPreview": "Image d'aperçu téléchargée",
"downloadingFile": "Téléchargement du fichier {type}",
"finalizing": "Finalisation du téléchargement..."
},
"progress": {
"currentFile": "Fichier actuel :",
"downloading": "Téléchargement : {name}",
"transferred": "Téléchargé : {downloaded} / {total}",
"transferredSimple": "Téléchargé : {downloaded}",
"transferredUnknown": "Téléchargé : --",
"speed": "Vitesse : {speed}"
}
},
"move": {
@@ -606,6 +663,7 @@
"contentRating": {
"title": "Définir la classification du contenu",
"current": "Actuel",
"multiple": "Valeurs multiples",
"levels": {
"pg": "PG",
"pg13": "PG13",
@@ -1084,6 +1142,10 @@
"bulkBaseModelUpdateSuccess": "Modèle de base mis à jour avec succès pour {count} modèle(s)",
"bulkBaseModelUpdatePartial": "{success} modèle(s) mis à jour, {failed} modèle(s) en échec",
"bulkBaseModelUpdateFailed": "Échec de la mise à jour du modèle de base pour les modèles sélectionnés",
"bulkContentRatingUpdating": "Mise à jour de la classification du contenu pour {count} modèle(s)...",
"bulkContentRatingSet": "Classification du contenu définie sur {level} pour {count} modèle(s)",
"bulkContentRatingPartial": "Classification du contenu définie sur {level} pour {success} modèle(s), {failed} échec(s)",
"bulkContentRatingFailed": "Impossible de mettre à jour la classification du contenu pour les modèles sélectionnés",
"invalidCharactersRemoved": "Caractères invalides supprimés du nom de fichier",
"filenameCannotBeEmpty": "Le nom de fichier ne peut pas être vide",
"renameFailed": "Échec du renommage du fichier : {message}",
@@ -1118,6 +1180,8 @@
"compactModeToggled": "Mode compact {state}",
"settingSaveFailed": "Échec de la sauvegarde du paramètre : {message}",
"displayDensitySet": "Densité d'affichage définie sur {density}",
"libraryLoadFailed": "Failed to load libraries: {message}",
"libraryActivateFailed": "Failed to activate library: {message}",
"languageChangeFailed": "Échec du changement de langue : {message}",
"cacheCleared": "Les fichiers de cache ont été vidés avec succès. Le cache sera reconstruit à la prochaine action.",
"cacheClearFailed": "Échec du vidage du cache : {error}",
@@ -1191,6 +1255,8 @@
"pauseFailed": "Échec de la mise en pause du téléchargement : {error}",
"downloadResumed": "Téléchargement repris",
"resumeFailed": "Échec de la reprise du téléchargement : {error}",
"downloadStopped": "Téléchargement annulé",
"stopFailed": "Échec de l'annulation du téléchargement : {error}",
"deleted": "Image d'exemple supprimée",
"deleteFailed": "Échec de la suppression de l'image d'exemple",
"setPreviewFailed": "Échec de la définition de l'image d'aperçu"
@@ -1237,6 +1303,12 @@
"refreshNow": "Actualiser maintenant",
"refreshingIn": "Actualisation dans",
"seconds": "secondes"
},
"communitySupport": {
"title": "Keep LoRA Manager Thriving with Your Support ❤️",
"content": "LoRA Manager is a passion project maintained full-time by a solo developer. Your support on Ko-fi helps cover development costs, keeps new updates coming, and unlocks a license key for the LM Civitai Extension as a thank-you gift. Every contribution truly makes a difference.",
"supportCta": "Support on Ko-fi",
"learnMore": "LM Civitai Extension Tutorial"
}
}
}

1314
locales/he.json Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -31,7 +31,8 @@
"japanese": "日本語",
"korean": "한국어",
"french": "Français",
"spanish": "Español"
"spanish": "Español",
"Hebrew": "עברית"
},
"fileSize": {
"zero": "0バイト",
@@ -187,6 +188,12 @@
"civitaiApiKey": "Civitai APIキー",
"civitaiApiKeyPlaceholder": "Civitai APIキーを入力してください",
"civitaiApiKeyHelp": "Civitaiからモデルをダウンロードするときの認証に使用されます",
"openSettingsFileLocation": {
"label": "設定フォルダーを開く",
"tooltip": "settings.json を含むフォルダーを開きます",
"success": "settings.json フォルダーを開きました",
"failed": "settings.json フォルダーを開けませんでした"
},
"sections": {
"contentFiltering": "コンテンツフィルタリング",
"videoSettings": "動画設定",
@@ -196,7 +203,8 @@
"exampleImages": "例画像",
"misc": "その他",
"metadataArchive": "メタデータアーカイブデータベース",
"proxySettings": "プロキシ設定"
"proxySettings": "プロキシ設定",
"priorityTags": "優先タグ"
},
"contentFiltering": {
"blurNsfwContent": "NSFWコンテンツをぼかす",
@@ -217,9 +225,9 @@
},
"displayDensityHelp": "1行に表示するカード数を選択",
"displayDensityDetails": {
"default": "デフォルト:51080p、62K、84K",
"medium": "中:61080p、72K、94K",
"compact": "コンパクト:71080p、82K、104K"
"default": "51080p、62K、84K",
"medium": "61080p、72K、94K",
"compact": "71080p、82K、104K"
},
"displayDensityWarning": "警告:高密度設定は、リソースが限られたシステムでパフォーマンスの問題を引き起こす可能性があります。",
"cardInfoDisplay": "カード情報表示",
@@ -229,11 +237,25 @@
},
"cardInfoDisplayHelp": "モデル情報とアクションボタンの表示タイミングを選択:",
"cardInfoDisplayDetails": {
"always": "常に表示:ヘッダーとフッターが常に表示されます",
"hover": "ホバー時に表示:カードにホバーしたときのみヘッダーとフッターが表示されます"
"always": "ヘッダーとフッターが常に表示されます",
"hover": "カードにホバーしたときのみヘッダーとフッターが表示されます"
},
"modelNameDisplay": "モデル名表示",
"modelNameDisplayOptions": {
"modelName": "モデル名",
"fileName": "ファイル名"
},
"modelNameDisplayHelp": "モデルカードのフッターに表示する内容を選択:",
"modelNameDisplayDetails": {
"modelName": "モデルの説明的な名前を表示",
"fileName": "ディスク上の実際のファイル名を表示"
}
},
"folderSettings": {
"activeLibrary": "アクティブライブラリ",
"activeLibraryHelp": "設定済みのライブラリを切り替えてデフォルトのフォルダを更新します。選択を変更するとページが再読み込みされます。",
"loadingLibraries": "ライブラリを読み込み中...",
"noLibraries": "ライブラリが設定されていません",
"defaultLoraRoot": "デフォルトLoRAルート",
"defaultLoraRootHelp": "ダウンロード、インポート、移動用のデフォルトLoRAルートディレクトリを設定",
"defaultCheckpointRoot": "デフォルトCheckpointルート",
@@ -334,6 +356,26 @@
"proxyPassword": "パスワード(任意)",
"proxyPasswordPlaceholder": "パスワード",
"proxyPasswordHelp": "プロキシ認証用のパスワード(必要な場合)"
},
"priorityTags": {
"title": "優先タグ",
"description": "各モデルタイプのタグ優先順位をカスタマイズします (例: character, concept, style(toon|toon_style))",
"placeholder": "character, concept, style(toon|toon_style)",
"helpLinkLabel": "優先タグのヘルプを開く",
"modelTypes": {
"lora": "LoRA",
"checkpoint": "チェックポイント",
"embedding": "埋め込み"
},
"saveSuccess": "優先タグを更新しました。",
"saveError": "優先タグの更新に失敗しました。",
"loadingSuggestions": "候補を読み込み中...",
"validation": {
"missingClosingParen": "エントリ {index} に閉じ括弧がありません。",
"missingCanonical": "エントリ {index} には正規タグ名を含める必要があります。",
"duplicateCanonical": "正規タグ \"{tag}\" が複数回登場しています。",
"unknown": "無効な優先タグ設定です。"
}
}
},
"loras": {
@@ -382,6 +424,7 @@
"viewSelected": "選択中を表示",
"addTags": "すべてにタグを追加",
"setBaseModel": "すべてにベースモデルを設定",
"setContentRating": "すべてのモデルのコンテンツレーティングを設定",
"copyAll": "すべての構文をコピー",
"refreshAll": "すべてのメタデータを更新",
"moveAll": "すべてをフォルダに移動",
@@ -517,13 +560,19 @@
"title": "Embeddingモデル"
},
"sidebar": {
"modelRoot": "モデルルート",
"modelRoot": "ルート",
"collapseAll": "すべてのフォルダを折りたたむ",
"pinSidebar": "サイドバーを固定",
"unpinSidebar": "サイドバーの固定を解除",
"switchToListView": "リストビューに切り替え",
"switchToTreeView": "ツリービューに切り替え",
"collapseAllDisabled": "リストビューでは利用できません"
"switchToTreeView": "ツリー表示に切り替え",
"recursiveOn": "サブフォルダーを検索",
"recursiveOff": "現在のフォルダーのみを検索",
"recursiveUnavailable": "再帰検索はツリービューでのみ利用できます",
"collapseAllDisabled": "リストビューでは利用できません",
"dragDrop": {
"unableToResolveRoot": "移動先のパスを特定できません。"
}
},
"statistics": {
"title": "統計",
@@ -598,6 +647,14 @@
"downloadedPreview": "プレビュー画像をダウンロードしました",
"downloadingFile": "{type}ファイルをダウンロード中",
"finalizing": "ダウンロードを完了中..."
},
"progress": {
"currentFile": "現在のファイル:",
"downloading": "ダウンロード中: {name}",
"transferred": "ダウンロード済み: {downloaded} / {total}",
"transferredSimple": "ダウンロード済み: {downloaded}",
"transferredUnknown": "ダウンロード済み: --",
"speed": "速度: {speed}"
}
},
"move": {
@@ -606,6 +663,7 @@
"contentRating": {
"title": "コンテンツレーティングを設定",
"current": "現在",
"multiple": "複数の値",
"levels": {
"pg": "PG",
"pg13": "PG13",
@@ -1084,6 +1142,10 @@
"bulkBaseModelUpdateSuccess": "{count} モデルのベースモデルが正常に更新されました",
"bulkBaseModelUpdatePartial": "{success} モデルを更新、{failed} モデルは失敗しました",
"bulkBaseModelUpdateFailed": "選択したモデルのベースモデルの更新に失敗しました",
"bulkContentRatingUpdating": "{count} 件のモデルのコンテンツレーティングを更新中...",
"bulkContentRatingSet": "{count} 件のモデルのコンテンツレーティングを {level} に設定しました",
"bulkContentRatingPartial": "{success} 件のモデルのコンテンツレーティングを {level} に設定、{failed} 件は失敗しました",
"bulkContentRatingFailed": "選択したモデルのコンテンツレーティングを更新できませんでした",
"invalidCharactersRemoved": "ファイル名から無効な文字が削除されました",
"filenameCannotBeEmpty": "ファイル名を空にすることはできません",
"renameFailed": "ファイル名の変更に失敗しました:{message}",
@@ -1118,6 +1180,8 @@
"compactModeToggled": "コンパクトモード {state}",
"settingSaveFailed": "設定の保存に失敗しました:{message}",
"displayDensitySet": "表示密度が {density} に設定されました",
"libraryLoadFailed": "Failed to load libraries: {message}",
"libraryActivateFailed": "Failed to activate library: {message}",
"languageChangeFailed": "言語の変更に失敗しました:{message}",
"cacheCleared": "キャッシュファイルが正常にクリアされました。次回のアクションでキャッシュが再構築されます。",
"cacheClearFailed": "キャッシュのクリアに失敗しました:{error}",
@@ -1191,6 +1255,8 @@
"pauseFailed": "ダウンロードの一時停止に失敗しました:{error}",
"downloadResumed": "ダウンロードが再開されました",
"resumeFailed": "ダウンロードの再開に失敗しました:{error}",
"downloadStopped": "ダウンロードをキャンセルしました",
"stopFailed": "ダウンロードのキャンセルに失敗しました:{error}",
"deleted": "例画像が削除されました",
"deleteFailed": "例画像の削除に失敗しました",
"setPreviewFailed": "プレビュー画像の設定に失敗しました"
@@ -1237,6 +1303,12 @@
"refreshNow": "今すぐ更新",
"refreshingIn": "更新まで",
"seconds": "秒"
},
"communitySupport": {
"title": "Keep LoRA Manager Thriving with Your Support ❤️",
"content": "LoRA Manager is a passion project maintained full-time by a solo developer. Your support on Ko-fi helps cover development costs, keeps new updates coming, and unlocks a license key for the LM Civitai Extension as a thank-you gift. Every contribution truly makes a difference.",
"supportCta": "Support on Ko-fi",
"learnMore": "LM Civitai Extension Tutorial"
}
}
}

View File

@@ -31,7 +31,8 @@
"japanese": "日本語",
"korean": "한국어",
"french": "Français",
"spanish": "Español"
"spanish": "Español",
"Hebrew": "עברית"
},
"fileSize": {
"zero": "0 바이트",
@@ -187,6 +188,12 @@
"civitaiApiKey": "Civitai API 키",
"civitaiApiKeyPlaceholder": "Civitai API 키를 입력하세요",
"civitaiApiKeyHelp": "Civitai에서 모델을 다운로드할 때 인증에 사용됩니다",
"openSettingsFileLocation": {
"label": "설정 폴더 열기",
"tooltip": "settings.json이 있는 폴더를 엽니다",
"success": "settings.json 폴더를 열었습니다",
"failed": "settings.json 폴더를 열지 못했습니다"
},
"sections": {
"contentFiltering": "콘텐츠 필터링",
"videoSettings": "비디오 설정",
@@ -196,7 +203,8 @@
"exampleImages": "예시 이미지",
"misc": "기타",
"metadataArchive": "메타데이터 아카이브 데이터베이스",
"proxySettings": "프록시 설정"
"proxySettings": "프록시 설정",
"priorityTags": "우선순위 태그"
},
"contentFiltering": {
"blurNsfwContent": "NSFW 콘텐츠 블러 처리",
@@ -217,9 +225,9 @@
},
"displayDensityHelp": "한 줄에 표시할 카드 수를 선택하세요:",
"displayDensityDetails": {
"default": "기본: 5개 (1080p), 6개 (2K), 8개 (4K)",
"medium": "중간: 6개 (1080p), 7개 (2K), 9개 (4K)",
"compact": "조밀: 7개 (1080p), 8개 (2K), 10개 (4K)"
"default": "5개 (1080p), 6개 (2K), 8개 (4K)",
"medium": "6개 (1080p), 7개 (2K), 9개 (4K)",
"compact": "7개 (1080p), 8개 (2K), 10개 (4K)"
},
"displayDensityWarning": "경고: 높은 밀도는 리소스가 제한된 시스템에서 성능 문제를 일으킬 수 있습니다.",
"cardInfoDisplay": "카드 정보 표시",
@@ -229,11 +237,25 @@
},
"cardInfoDisplayHelp": "모델 정보 및 액션 버튼을 언제 표시할지 선택하세요:",
"cardInfoDisplayDetails": {
"always": "항상 표시: 헤더와 푸터가 항상 보입니다",
"hover": "호버 시 표시: 카드에 마우스를 올렸을 때만 헤더와 푸터가 나타납니다"
"always": "헤더와 푸터가 항상 보입니다",
"hover": "카드에 마우스를 올렸을 때만 헤더와 푸터가 나타납니다"
},
"modelNameDisplay": "모델명 표시",
"modelNameDisplayOptions": {
"modelName": "모델명",
"fileName": "파일명"
},
"modelNameDisplayHelp": "모델 카드 하단에 표시할 내용을 선택하세요:",
"modelNameDisplayDetails": {
"modelName": "모델의 설명적 이름 표시",
"fileName": "디스크의 실제 파일명 표시"
}
},
"folderSettings": {
"activeLibrary": "활성 라이브러리",
"activeLibraryHelp": "구성된 라이브러리를 전환하여 기본 폴더를 업데이트합니다. 선택을 변경하면 페이지가 다시 로드됩니다.",
"loadingLibraries": "라이브러리를 불러오는 중...",
"noLibraries": "구성된 라이브러리가 없습니다",
"defaultLoraRoot": "기본 LoRA 루트",
"defaultLoraRootHelp": "다운로드, 가져오기 및 이동을 위한 기본 LoRA 루트 디렉토리를 설정합니다",
"defaultCheckpointRoot": "기본 Checkpoint 루트",
@@ -334,6 +356,26 @@
"proxyPassword": "비밀번호 (선택사항)",
"proxyPasswordPlaceholder": "password",
"proxyPasswordHelp": "프록시 인증에 필요한 비밀번호 (필요한 경우)"
},
"priorityTags": {
"title": "우선순위 태그",
"description": "모델 유형별 태그 우선순위를 사용자 지정합니다(예: character, concept, style(toon|toon_style)).",
"placeholder": "character, concept, style(toon|toon_style)",
"helpLinkLabel": "우선순위 태그 도움말 열기",
"modelTypes": {
"lora": "LoRA",
"checkpoint": "체크포인트",
"embedding": "임베딩"
},
"saveSuccess": "우선순위 태그가 업데이트되었습니다.",
"saveError": "우선순위 태그를 업데이트하지 못했습니다.",
"loadingSuggestions": "추천을 불러오는 중...",
"validation": {
"missingClosingParen": "{index}번째 항목에 닫는 괄호가 없습니다.",
"missingCanonical": "{index}번째 항목에는 정식 태그 이름이 포함되어야 합니다.",
"duplicateCanonical": "정식 태그 \"{tag}\"가 여러 번 나타납니다.",
"unknown": "잘못된 우선순위 태그 구성입니다."
}
}
},
"loras": {
@@ -382,6 +424,7 @@
"viewSelected": "선택 항목 보기",
"addTags": "모두에 태그 추가",
"setBaseModel": "모두에 베이스 모델 설정",
"setContentRating": "모든 모델에 콘텐츠 등급 설정",
"copyAll": "모든 문법 복사",
"refreshAll": "모든 메타데이터 새로고침",
"moveAll": "모두 폴더로 이동",
@@ -517,13 +560,19 @@
"title": "Embedding 모델"
},
"sidebar": {
"modelRoot": "모델 루트",
"modelRoot": "루트",
"collapseAll": "모든 폴더 접기",
"pinSidebar": "사이드바 고정",
"unpinSidebar": "사이드바 고정 해제",
"switchToListView": "목록 보기로 전환",
"switchToTreeView": "트리 보기로 전환",
"collapseAllDisabled": "목록 보기에서는 사용할 수 없습니다"
"recursiveOn": "하위 폴더 검색",
"recursiveOff": "현재 폴더만 검색",
"recursiveUnavailable": "재귀 검색은 트리 보기에서만 사용할 수 있습니다",
"collapseAllDisabled": "목록 보기에서는 사용할 수 없습니다",
"dragDrop": {
"unableToResolveRoot": "이동할 대상 경로를 확인할 수 없습니다."
}
},
"statistics": {
"title": "통계",
@@ -598,6 +647,14 @@
"downloadedPreview": "미리보기 이미지 다운로드됨",
"downloadingFile": "{type} 파일 다운로드 중",
"finalizing": "다운로드 완료 중..."
},
"progress": {
"currentFile": "현재 파일:",
"downloading": "다운로드 중: {name}",
"transferred": "다운로드됨: {downloaded} / {total}",
"transferredSimple": "다운로드됨: {downloaded}",
"transferredUnknown": "다운로드됨: --",
"speed": "속도: {speed}"
}
},
"move": {
@@ -606,6 +663,7 @@
"contentRating": {
"title": "콘텐츠 등급 설정",
"current": "현재",
"multiple": "여러 값",
"levels": {
"pg": "PG",
"pg13": "PG13",
@@ -1084,6 +1142,10 @@
"bulkBaseModelUpdateSuccess": "{count}개의 모델에 베이스 모델이 성공적으로 업데이트되었습니다",
"bulkBaseModelUpdatePartial": "{success}개의 모델이 업데이트되었고, {failed}개의 모델이 실패했습니다",
"bulkBaseModelUpdateFailed": "선택한 모델의 베이스 모델 업데이트에 실패했습니다",
"bulkContentRatingUpdating": "{count}개 모델의 콘텐츠 등급을 업데이트하는 중...",
"bulkContentRatingSet": "{count}개 모델의 콘텐츠 등급을 {level}(으)로 설정했습니다",
"bulkContentRatingPartial": "{success}개 모델의 콘텐츠 등급을 {level}(으)로 설정했고, {failed}개는 실패했습니다",
"bulkContentRatingFailed": "선택한 모델의 콘텐츠 등급을 업데이트하지 못했습니다",
"invalidCharactersRemoved": "파일명에서 잘못된 문자가 제거되었습니다",
"filenameCannotBeEmpty": "파일 이름은 비어있을 수 없습니다",
"renameFailed": "파일 이름 변경 실패: {message}",
@@ -1118,6 +1180,8 @@
"compactModeToggled": "컴팩트 모드 {state}",
"settingSaveFailed": "설정 저장 실패: {message}",
"displayDensitySet": "표시 밀도가 {density}로 설정되었습니다",
"libraryLoadFailed": "Failed to load libraries: {message}",
"libraryActivateFailed": "Failed to activate library: {message}",
"languageChangeFailed": "언어 변경 실패: {message}",
"cacheCleared": "캐시 파일이 성공적으로 지워졌습니다. 다음 작업 시 캐시가 재구축됩니다.",
"cacheClearFailed": "캐시 지우기 실패: {error}",
@@ -1191,6 +1255,8 @@
"pauseFailed": "다운로드 일시정지 실패: {error}",
"downloadResumed": "다운로드가 재개되었습니다",
"resumeFailed": "다운로드 재개 실패: {error}",
"downloadStopped": "다운로드가 취소되었습니다",
"stopFailed": "다운로드 취소 실패: {error}",
"deleted": "예시 이미지가 삭제되었습니다",
"deleteFailed": "예시 이미지 삭제 실패",
"setPreviewFailed": "미리보기 이미지 설정 실패"
@@ -1237,6 +1303,12 @@
"refreshNow": "지금 새로고침",
"refreshingIn": "새로고침까지",
"seconds": "초"
},
"communitySupport": {
"title": "Keep LoRA Manager Thriving with Your Support ❤️",
"content": "LoRA Manager is a passion project maintained full-time by a solo developer. Your support on Ko-fi helps cover development costs, keeps new updates coming, and unlocks a license key for the LM Civitai Extension as a thank-you gift. Every contribution truly makes a difference.",
"supportCta": "Support on Ko-fi",
"learnMore": "LM Civitai Extension Tutorial"
}
}
}

View File

@@ -31,7 +31,8 @@
"japanese": "日本語",
"korean": "한국어",
"french": "Français",
"spanish": "Español"
"spanish": "Español",
"Hebrew": "עברית"
},
"fileSize": {
"zero": "0 Байт",
@@ -187,6 +188,12 @@
"civitaiApiKey": "Ключ API Civitai",
"civitaiApiKeyPlaceholder": "Введите ваш ключ API Civitai",
"civitaiApiKeyHelp": "Используется для аутентификации при загрузке моделей с Civitai",
"openSettingsFileLocation": {
"label": "Открыть папку настроек",
"tooltip": "Открыть папку, содержащую settings.json",
"success": "Папка settings.json открыта",
"failed": "Не удалось открыть папку settings.json"
},
"sections": {
"contentFiltering": "Фильтрация контента",
"videoSettings": "Настройки видео",
@@ -196,7 +203,8 @@
"exampleImages": "Примеры изображений",
"misc": "Разное",
"metadataArchive": "Архив метаданных",
"proxySettings": "Настройки прокси"
"proxySettings": "Настройки прокси",
"priorityTags": "Приоритетные теги"
},
"contentFiltering": {
"blurNsfwContent": "Размывать NSFW контент",
@@ -217,9 +225,9 @@
},
"displayDensityHelp": "Выберите количество карточек для отображения в ряду:",
"displayDensityDetails": {
"default": "По умолчанию: 5 (1080p), 6 (2K), 8 (4K)",
"medium": "Средняя: 6 (1080p), 7 (2K), 9 (4K)",
"compact": "Компактная: 7 (1080p), 8 (2K), 10 (4K)"
"default": "5 (1080p), 6 (2K), 8 (4K)",
"medium": "6 (1080p), 7 (2K), 9 (4K)",
"compact": "7 (1080p), 8 (2K), 10 (4K)"
},
"displayDensityWarning": "Предупреждение: Высокая плотность может вызвать проблемы с производительностью на системах с ограниченными ресурсами.",
"cardInfoDisplay": "Отображение информации карточки",
@@ -229,11 +237,25 @@
},
"cardInfoDisplayHelp": "Выберите когда отображать информацию о модели и кнопки действий:",
"cardInfoDisplayDetails": {
"always": "Всегда видимо: Заголовки и подписи всегда видны",
"hover": "Показать при наведении: Заголовки и подписи появляются только при наведении на карточку"
"always": "Заголовки и подписи всегда видны",
"hover": "Заголовки и подписи появляются только при наведении на карточку"
},
"modelNameDisplay": "Отображение названия модели",
"modelNameDisplayOptions": {
"modelName": "Название модели",
"fileName": "Имя файла"
},
"modelNameDisplayHelp": "Выберите, что отображать в нижней части карточки модели:",
"modelNameDisplayDetails": {
"modelName": "Отображать описательное название модели",
"fileName": "Отображать фактическое имя файла на диске"
}
},
"folderSettings": {
"activeLibrary": "Активная библиотека",
"activeLibraryHelp": "Переключайтесь между настроенными библиотеками, чтобы обновить папки по умолчанию. Изменение выбора перезагружает страницу.",
"loadingLibraries": "Загрузка библиотек...",
"noLibraries": "Библиотеки не настроены",
"defaultLoraRoot": "Корневая папка LoRA по умолчанию",
"defaultLoraRootHelp": "Установить корневую папку LoRA по умолчанию для загрузок, импорта и перемещений",
"defaultCheckpointRoot": "Корневая папка Checkpoint по умолчанию",
@@ -334,6 +356,26 @@
"proxyPassword": "Пароль (необязательно)",
"proxyPasswordPlaceholder": "пароль",
"proxyPasswordHelp": "Пароль для аутентификации на прокси (если требуется)"
},
"priorityTags": {
"title": "Приоритетные теги",
"description": "Настройте порядок приоритетов тегов для каждого типа моделей (например, character, concept, style(toon|toon_style)).",
"placeholder": "character, concept, style(toon|toon_style)",
"helpLinkLabel": "Открыть справку по приоритетным тегам",
"modelTypes": {
"lora": "LoRA",
"checkpoint": "Чекпойнт",
"embedding": "Эмбеддинг"
},
"saveSuccess": "Приоритетные теги обновлены.",
"saveError": "Не удалось обновить приоритетные теги.",
"loadingSuggestions": "Загрузка подсказок...",
"validation": {
"missingClosingParen": "В записи {index} отсутствует закрывающая скобка.",
"missingCanonical": "Запись {index} должна содержать каноническое имя тега.",
"duplicateCanonical": "Канонический тег \"{tag}\" встречается более одного раза.",
"unknown": "Недопустимая конфигурация приоритетных тегов."
}
}
},
"loras": {
@@ -382,6 +424,7 @@
"viewSelected": "Просмотреть выбранные",
"addTags": "Добавить теги ко всем",
"setBaseModel": "Установить базовую модель для всех",
"setContentRating": "Установить рейтинг контента для всех",
"copyAll": "Копировать весь синтаксис",
"refreshAll": "Обновить все метаданные",
"moveAll": "Переместить все в папку",
@@ -517,13 +560,19 @@
"title": "Модели Embedding"
},
"sidebar": {
"modelRoot": "Корень моделей",
"modelRoot": "Корень",
"collapseAll": "Свернуть все папки",
"pinSidebar": "Закрепить боковую панель",
"unpinSidebar": "Открепить боковую панель",
"switchToListView": "Переключить на вид списка",
"switchToTreeView": "Переключить на древовидный вид",
"collapseAllDisabled": "Недоступно в виде списка"
"recursiveOn": "Искать во вложенных папках",
"recursiveOff": "Искать только в текущей папке",
"recursiveUnavailable": "Рекурсивный поиск доступен только в режиме дерева",
"collapseAllDisabled": "Недоступно в виде списка",
"dragDrop": {
"unableToResolveRoot": "Не удалось определить путь назначения для перемещения."
}
},
"statistics": {
"title": "Статистика",
@@ -598,6 +647,14 @@
"downloadedPreview": "Превью изображение загружено",
"downloadingFile": "Загрузка файла {type}",
"finalizing": "Завершение загрузки..."
},
"progress": {
"currentFile": "Текущий файл:",
"downloading": "Скачивается: {name}",
"transferred": "Скачано: {downloaded} / {total}",
"transferredSimple": "Скачано: {downloaded}",
"transferredUnknown": "Скачано: --",
"speed": "Скорость: {speed}"
}
},
"move": {
@@ -606,6 +663,7 @@
"contentRating": {
"title": "Установить рейтинг контента",
"current": "Текущий",
"multiple": "Несколько значений",
"levels": {
"pg": "PG",
"pg13": "PG13",
@@ -1084,6 +1142,10 @@
"bulkBaseModelUpdateSuccess": "Базовая модель успешно обновлена для {count} моделей",
"bulkBaseModelUpdatePartial": "Обновлено {success} моделей, не удалось обновить {failed} моделей",
"bulkBaseModelUpdateFailed": "Не удалось обновить базовую модель для выбранных моделей",
"bulkContentRatingUpdating": "Обновление рейтинга контента для {count} модель(ей)...",
"bulkContentRatingSet": "Рейтинг контента установлен на {level} для {count} модель(ей)",
"bulkContentRatingPartial": "Рейтинг контента {level} установлен для {success} модель(ей), {failed} не удалось",
"bulkContentRatingFailed": "Не удалось обновить рейтинг контента для выбранных моделей",
"invalidCharactersRemoved": "Недопустимые символы удалены из имени файла",
"filenameCannotBeEmpty": "Имя файла не может быть пустым",
"renameFailed": "Не удалось переименовать файл: {message}",
@@ -1118,6 +1180,8 @@
"compactModeToggled": "Компактный режим {state}",
"settingSaveFailed": "Не удалось сохранить настройку: {message}",
"displayDensitySet": "Плотность отображения установлена на {density}",
"libraryLoadFailed": "Failed to load libraries: {message}",
"libraryActivateFailed": "Failed to activate library: {message}",
"languageChangeFailed": "Не удалось изменить язык: {message}",
"cacheCleared": "Файлы кэша успешно очищены. Кэш будет пересобран при следующем действии.",
"cacheClearFailed": "Не удалось очистить кэш: {error}",
@@ -1191,6 +1255,8 @@
"pauseFailed": "Не удалось приостановить загрузку: {error}",
"downloadResumed": "Загрузка возобновлена",
"resumeFailed": "Не удалось возобновить загрузку: {error}",
"downloadStopped": "Загрузка отменена",
"stopFailed": "Не удалось отменить загрузку: {error}",
"deleted": "Пример изображения удален",
"deleteFailed": "Не удалось удалить пример изображения",
"setPreviewFailed": "Не удалось установить превью изображение"
@@ -1237,6 +1303,12 @@
"refreshNow": "Обновить сейчас",
"refreshingIn": "Обновление через",
"seconds": "секунд"
},
"communitySupport": {
"title": "Keep LoRA Manager Thriving with Your Support ❤️",
"content": "LoRA Manager is a passion project maintained full-time by a solo developer. Your support on Ko-fi helps cover development costs, keeps new updates coming, and unlocks a license key for the LM Civitai Extension as a thank-you gift. Every contribution truly makes a difference.",
"supportCta": "Support on Ko-fi",
"learnMore": "LM Civitai Extension Tutorial"
}
}
}

View File

@@ -21,8 +21,8 @@
"disabled": "已禁用"
},
"language": {
"select": "Language",
"select_help": "Choose your preferred language for the interface",
"select": "选择语言",
"select_help": "选择你喜欢的界面语言",
"english": "English",
"chinese_simplified": "中文(简体)",
"chinese_traditional": "中文(繁体)",
@@ -31,7 +31,8 @@
"japanese": "日本語",
"korean": "한국어",
"french": "Français",
"spanish": "Español"
"spanish": "Español",
"Hebrew": "עברית"
},
"fileSize": {
"zero": "0 字节",
@@ -187,6 +188,12 @@
"civitaiApiKey": "Civitai API 密钥",
"civitaiApiKeyPlaceholder": "请输入你的 Civitai API 密钥",
"civitaiApiKeyHelp": "用于从 Civitai 下载模型时的身份验证",
"openSettingsFileLocation": {
"label": "打开设置文件夹",
"tooltip": "打开包含 settings.json 的文件夹",
"success": "已打开 settings.json 文件夹",
"failed": "无法打开 settings.json 文件夹"
},
"sections": {
"contentFiltering": "内容过滤",
"videoSettings": "视频设置",
@@ -196,7 +203,8 @@
"exampleImages": "示例图片",
"misc": "其他",
"metadataArchive": "元数据归档数据库",
"proxySettings": "代理设置"
"proxySettings": "代理设置",
"priorityTags": "优先标签"
},
"contentFiltering": {
"blurNsfwContent": "模糊 NSFW 内容",
@@ -217,9 +225,9 @@
},
"displayDensityHelp": "选择每行显示卡片数量:",
"displayDensityDetails": {
"default": "默认:51080p62K84K",
"medium": "中等:61080p72K94K",
"compact": "紧凑:71080p82K104K"
"default": "51080p62K84K",
"medium": "61080p72K94K",
"compact": "71080p82K104K"
},
"displayDensityWarning": "警告:高密度可能导致资源有限的系统性能下降。",
"cardInfoDisplay": "卡片信息显示",
@@ -229,11 +237,25 @@
},
"cardInfoDisplayHelp": "选择何时显示模型信息和操作按钮:",
"cardInfoDisplayDetails": {
"always": "始终可见:标题和底部始终显示",
"hover": "悬停时显示:仅在悬停卡片时显示标题和底部"
"always": "标题和底部始终显示",
"hover": "仅在悬停卡片时显示标题和底部"
},
"modelNameDisplay": "模型名称显示",
"modelNameDisplayOptions": {
"modelName": "模型名称",
"fileName": "文件名"
},
"modelNameDisplayHelp": "选择在模型卡片底部显示的内容:",
"modelNameDisplayDetails": {
"modelName": "显示模型的描述性名称",
"fileName": "显示磁盘上的实际文件名"
}
},
"folderSettings": {
"activeLibrary": "活动库",
"activeLibraryHelp": "在已配置的库之间切换以更新默认文件夹。更改选择将重新加载页面。",
"loadingLibraries": "正在加载库...",
"noLibraries": "尚未配置库",
"defaultLoraRoot": "默认 LoRA 根目录",
"defaultLoraRootHelp": "设置下载、导入和移动时的默认 LoRA 根目录",
"defaultCheckpointRoot": "默认 Checkpoint 根目录",
@@ -334,6 +356,26 @@
"proxyPassword": "密码 (可选)",
"proxyPasswordPlaceholder": "密码",
"proxyPasswordHelp": "代理认证的密码 (如果需要)"
},
"priorityTags": {
"title": "优先标签",
"description": "为每种模型类型自定义标签优先级顺序 (例如: character, concept, style(toon|toon_style))",
"placeholder": "character, concept, style(toon|toon_style)",
"helpLinkLabel": "打开优先标签帮助",
"modelTypes": {
"lora": "LoRA",
"checkpoint": "Checkpoint",
"embedding": "Embedding"
},
"saveSuccess": "优先标签已更新。",
"saveError": "优先标签更新失败。",
"loadingSuggestions": "正在加载建议...",
"validation": {
"missingClosingParen": "条目 {index} 缺少右括号。",
"missingCanonical": "条目 {index} 必须包含规范标签名称。",
"duplicateCanonical": "规范标签 \"{tag}\" 出现多次。",
"unknown": "优先标签配置无效。"
}
}
},
"loras": {
@@ -380,13 +422,14 @@
"selected": "已选中 {count} 项",
"selectedSuffix": "已选中",
"viewSelected": "查看已选中",
"addTags": "为所添加标签",
"setBaseModel": "为所设置基础模型",
"copyAll": "复制全部语法",
"refreshAll": "刷新全部元数据",
"moveAll": "全部移动到文件夹",
"addTags": "为所选中添加标签",
"setBaseModel": "为所选中设置基础模型",
"setContentRating": "为所选中设置内容评级",
"copyAll": "复制所选中语法",
"refreshAll": "刷新所选中元数据",
"moveAll": "移动所选中到文件夹",
"autoOrganize": "自动整理所选模型",
"deleteAll": "删除所有模型",
"deleteAll": "删除选中模型",
"clear": "清除选择",
"autoOrganizeProgress": {
"initializing": "正在初始化自动整理...",
@@ -517,13 +560,19 @@
"title": "Embedding 模型"
},
"sidebar": {
"modelRoot": "模型根目录",
"modelRoot": "根目录",
"collapseAll": "折叠所有文件夹",
"pinSidebar": "固定侧边栏",
"unpinSidebar": "取消固定侧边栏",
"switchToListView": "切换到列表视图",
"switchToTreeView": "切换到树状视图",
"collapseAllDisabled": "列表视图下不可用"
"recursiveOn": "搜索子文件夹",
"recursiveOff": "仅搜索当前文件夹",
"recursiveUnavailable": "仅在树形视图中可使用递归搜索",
"collapseAllDisabled": "列表视图下不可用",
"dragDrop": {
"unableToResolveRoot": "无法确定移动的目标路径。"
}
},
"statistics": {
"title": "统计",
@@ -598,6 +647,14 @@
"downloadedPreview": "预览图片已下载",
"downloadingFile": "正在下载 {type} 文件",
"finalizing": "正在完成下载..."
},
"progress": {
"currentFile": "当前文件:",
"downloading": "下载中:{name}",
"transferred": "已下载:{downloaded} / {total}",
"transferredSimple": "已下载:{downloaded}",
"transferredUnknown": "已下载:--",
"speed": "速度:{speed}"
}
},
"move": {
@@ -606,6 +663,7 @@
"contentRating": {
"title": "设置内容评级",
"current": "当前",
"multiple": "多个值",
"levels": {
"pg": "PG",
"pg13": "PG13",
@@ -1084,6 +1142,10 @@
"bulkBaseModelUpdateSuccess": "成功为 {count} 个模型更新基础模型",
"bulkBaseModelUpdatePartial": "更新了 {success} 个模型,{failed} 个失败",
"bulkBaseModelUpdateFailed": "为选中模型更新基础模型失败",
"bulkContentRatingUpdating": "正在为 {count} 个模型更新内容评级...",
"bulkContentRatingSet": "已将 {count} 个模型的内容评级设置为 {level}",
"bulkContentRatingPartial": "已将 {success} 个模型的内容评级设置为 {level}{failed} 个失败",
"bulkContentRatingFailed": "未能更新所选模型的内容评级",
"invalidCharactersRemoved": "文件名中的无效字符已移除",
"filenameCannotBeEmpty": "文件名不能为空",
"renameFailed": "重命名文件失败:{message}",
@@ -1118,6 +1180,8 @@
"compactModeToggled": "紧凑模式 {state}",
"settingSaveFailed": "保存设置失败:{message}",
"displayDensitySet": "显示密度已设置为 {density}",
"libraryLoadFailed": "Failed to load libraries: {message}",
"libraryActivateFailed": "Failed to activate library: {message}",
"languageChangeFailed": "切换语言失败:{message}",
"cacheCleared": "缓存文件已成功清除。下次操作将重建缓存。",
"cacheClearFailed": "清除缓存失败:{error}",
@@ -1191,6 +1255,8 @@
"pauseFailed": "暂停下载失败:{error}",
"downloadResumed": "下载已恢复",
"resumeFailed": "恢复下载失败:{error}",
"downloadStopped": "下载已取消",
"stopFailed": "取消下载失败:{error}",
"deleted": "示例图片已删除",
"deleteFailed": "删除示例图片失败",
"setPreviewFailed": "设置预览图片失败"
@@ -1237,6 +1303,12 @@
"refreshNow": "立即刷新",
"refreshingIn": "将在",
"seconds": "秒后刷新"
},
"communitySupport": {
"title": "Keep LoRA Manager Thriving with Your Support ❤️",
"content": "LoRA Manager is a passion project maintained full-time by a solo developer. Your support on Ko-fi helps cover development costs, keeps new updates coming, and unlocks a license key for the LM Civitai Extension as a thank-you gift. Every contribution truly makes a difference.",
"supportCta": "Support on Ko-fi",
"learnMore": "LM Civitai Extension Tutorial"
}
}
}

View File

@@ -31,7 +31,8 @@
"japanese": "日本語",
"korean": "한국어",
"french": "Français",
"spanish": "Español"
"spanish": "Español",
"Hebrew": "עברית"
},
"fileSize": {
"zero": "0 位元組",
@@ -187,6 +188,12 @@
"civitaiApiKey": "Civitai API 金鑰",
"civitaiApiKeyPlaceholder": "請輸入您的 Civitai API 金鑰",
"civitaiApiKeyHelp": "用於從 Civitai 下載模型時的身份驗證",
"openSettingsFileLocation": {
"label": "開啟設定資料夾",
"tooltip": "開啟包含 settings.json 的資料夾",
"success": "已開啟 settings.json 資料夾",
"failed": "無法開啟 settings.json 資料夾"
},
"sections": {
"contentFiltering": "內容過濾",
"videoSettings": "影片設定",
@@ -196,7 +203,8 @@
"exampleImages": "範例圖片",
"misc": "其他",
"metadataArchive": "中繼資料封存資料庫",
"proxySettings": "代理設定"
"proxySettings": "代理設定",
"priorityTags": "優先標籤"
},
"contentFiltering": {
"blurNsfwContent": "模糊 NSFW 內容",
@@ -217,9 +225,9 @@
},
"displayDensityHelp": "選擇每行顯示卡片數量:",
"displayDensityDetails": {
"default": "預設:51080p、62K、84K",
"medium": "中等:61080p、72K、94K",
"compact": "緊湊:71080p、82K、104K"
"default": "51080p、62K、84K",
"medium": "61080p、72K、94K",
"compact": "71080p、82K、104K"
},
"displayDensityWarning": "警告:較高密度可能導致資源有限的系統效能下降。",
"cardInfoDisplay": "卡片資訊顯示",
@@ -229,11 +237,25 @@
},
"cardInfoDisplayHelp": "選擇何時顯示模型資訊與操作按鈕:",
"cardInfoDisplayDetails": {
"always": "永遠顯示:標題與頁腳始終可見",
"hover": "滑鼠懸停顯示:標題與頁腳僅在滑鼠懸停時顯示"
"always": "標題與頁腳始終可見",
"hover": "標題與頁腳僅在滑鼠懸停時顯示"
},
"modelNameDisplay": "模型名稱顯示",
"modelNameDisplayOptions": {
"modelName": "模型名稱",
"fileName": "檔案名稱"
},
"modelNameDisplayHelp": "選擇在模型卡片底部顯示的內容:",
"modelNameDisplayDetails": {
"modelName": "顯示模型的描述性名稱",
"fileName": "顯示磁碟上的實際檔案名稱"
}
},
"folderSettings": {
"activeLibrary": "使用中的資料庫",
"activeLibraryHelp": "在已設定的資料庫之間切換以更新預設資料夾。變更選項會重新載入頁面。",
"loadingLibraries": "正在載入資料庫...",
"noLibraries": "尚未設定任何資料庫",
"defaultLoraRoot": "預設 LoRA 根目錄",
"defaultLoraRootHelp": "設定下載、匯入和移動時的預設 LoRA 根目錄",
"defaultCheckpointRoot": "預設 Checkpoint 根目錄",
@@ -334,6 +356,26 @@
"proxyPassword": "密碼(選填)",
"proxyPasswordPlaceholder": "password",
"proxyPasswordHelp": "代理驗證所需的密碼(如有需要)"
},
"priorityTags": {
"title": "優先標籤",
"description": "為每種模型類型自訂標籤的優先順序 (例如: character, concept, style(toon|toon_style))",
"placeholder": "character, concept, style(toon|toon_style)",
"helpLinkLabel": "開啟優先標籤說明",
"modelTypes": {
"lora": "LoRA",
"checkpoint": "Checkpoint",
"embedding": "Embedding"
},
"saveSuccess": "優先標籤已更新。",
"saveError": "更新優先標籤失敗。",
"loadingSuggestions": "正在載入建議...",
"validation": {
"missingClosingParen": "項目 {index} 缺少右括號。",
"missingCanonical": "項目 {index} 必須包含正規標籤名稱。",
"duplicateCanonical": "正規標籤 \"{tag}\" 出現多於一次。",
"unknown": "優先標籤設定無效。"
}
}
},
"loras": {
@@ -382,6 +424,7 @@
"viewSelected": "檢視已選取",
"addTags": "新增標籤到全部",
"setBaseModel": "設定全部基礎模型",
"setContentRating": "為全部設定內容分級",
"copyAll": "複製全部語法",
"refreshAll": "刷新全部 metadata",
"moveAll": "全部移動到資料夾",
@@ -517,13 +560,19 @@
"title": "Embedding 模型"
},
"sidebar": {
"modelRoot": "模型根目錄",
"modelRoot": "根目錄",
"collapseAll": "全部摺疊資料夾",
"pinSidebar": "固定側邊欄",
"unpinSidebar": "取消固定側邊欄",
"switchToListView": "切換至列表檢視",
"switchToTreeView": "切換樹狀檢視",
"collapseAllDisabled": "列表檢視下不可用"
"switchToTreeView": "切換樹狀檢視",
"recursiveOn": "搜尋子資料夾",
"recursiveOff": "僅搜尋目前資料夾",
"recursiveUnavailable": "遞迴搜尋僅能在樹狀檢視中使用",
"collapseAllDisabled": "列表檢視下不可用",
"dragDrop": {
"unableToResolveRoot": "無法確定移動的目標路徑。"
}
},
"statistics": {
"title": "統計",
@@ -598,6 +647,14 @@
"downloadedPreview": "已下載預覽圖片",
"downloadingFile": "正在下載 {type} 檔案",
"finalizing": "完成下載中..."
},
"progress": {
"currentFile": "目前檔案:",
"downloading": "下載中:{name}",
"transferred": "已下載:{downloaded} / {total}",
"transferredSimple": "已下載:{downloaded}",
"transferredUnknown": "已下載:--",
"speed": "速度:{speed}"
}
},
"move": {
@@ -606,6 +663,7 @@
"contentRating": {
"title": "設定內容分級",
"current": "目前",
"multiple": "多個值",
"levels": {
"pg": "PG",
"pg13": "PG13",
@@ -1084,6 +1142,10 @@
"bulkBaseModelUpdateSuccess": "已成功為 {count} 個模型更新基礎模型",
"bulkBaseModelUpdatePartial": "已更新 {success} 個模型,{failed} 個模型失敗",
"bulkBaseModelUpdateFailed": "更新所選模型的基礎模型失敗",
"bulkContentRatingUpdating": "正在為 {count} 個模型更新內容分級...",
"bulkContentRatingSet": "已將 {count} 個模型的內容分級設定為 {level}",
"bulkContentRatingPartial": "已將 {success} 個模型的內容分級設定為 {level}{failed} 個失敗",
"bulkContentRatingFailed": "無法更新所選模型的內容分級",
"invalidCharactersRemoved": "已移除檔名中的無效字元",
"filenameCannotBeEmpty": "檔案名稱不可為空",
"renameFailed": "重新命名檔案失敗:{message}",
@@ -1118,6 +1180,8 @@
"compactModeToggled": "緊湊模式已{state}",
"settingSaveFailed": "儲存設定失敗:{message}",
"displayDensitySet": "顯示密度已設為 {density}",
"libraryLoadFailed": "Failed to load libraries: {message}",
"libraryActivateFailed": "Failed to activate library: {message}",
"languageChangeFailed": "切換語言失敗:{message}",
"cacheCleared": "快取檔案已成功清除。快取將於下次操作時重建。",
"cacheClearFailed": "清除快取失敗:{error}",
@@ -1191,6 +1255,8 @@
"pauseFailed": "暫停下載失敗:{error}",
"downloadResumed": "下載已恢復",
"resumeFailed": "恢復下載失敗:{error}",
"downloadStopped": "下載已取消",
"stopFailed": "取消下載失敗:{error}",
"deleted": "範例圖片已刪除",
"deleteFailed": "刪除範例圖片失敗",
"setPreviewFailed": "設定預覽圖片失敗"
@@ -1237,6 +1303,12 @@
"refreshNow": "立即重新整理",
"refreshingIn": "將於",
"seconds": "秒後重新整理"
},
"communitySupport": {
"title": "Keep LoRA Manager Thriving with Your Support ❤️",
"content": "LoRA Manager is a passion project maintained full-time by a solo developer. Your support on Ko-fi helps cover development costs, keeps new updates coming, and unlocks a license key for the LM Civitai Extension as a thank-you gift. Every contribution truly makes a difference.",
"supportCta": "Support on Ko-fi",
"learnMore": "LM Civitai Extension Tutorial"
}
}
}

View File

@@ -5,7 +5,8 @@
"type": "module",
"scripts": {
"test": "vitest run",
"test:watch": "vitest"
"test:watch": "vitest",
"test:coverage": "node scripts/run_frontend_coverage.js"
},
"devDependencies": {
"jsdom": "^24.0.0",

View File

@@ -1,16 +1,50 @@
import os
import platform
from pathlib import Path
import folder_paths # type: ignore
from typing import List
from typing import Dict, Iterable, List, Mapping, Set
import logging
import json
import urllib.parse
from .utils.settings_paths import ensure_settings_file
# Use an environment variable to control standalone mode
standalone_mode = os.environ.get("HF_HUB_DISABLE_TELEMETRY", "0") == "0"
standalone_mode = os.environ.get("LORA_MANAGER_STANDALONE", "0") == "1" or os.environ.get("HF_HUB_DISABLE_TELEMETRY", "0") == "0"
logger = logging.getLogger(__name__)
def _normalize_folder_paths_for_comparison(
folder_paths: Mapping[str, Iterable[str]]
) -> Dict[str, Set[str]]:
"""Normalize folder paths for comparison across libraries."""
normalized: Dict[str, Set[str]] = {}
for key, values in folder_paths.items():
if isinstance(values, str):
candidate_values: Iterable[str] = [values]
else:
try:
candidate_values = iter(values)
except TypeError:
continue
normalized_values: Set[str] = set()
for value in candidate_values:
if not isinstance(value, str):
continue
stripped = value.strip()
if not stripped:
continue
normalized_values.add(os.path.normcase(os.path.normpath(stripped)))
if normalized_values:
normalized[key] = normalized_values
return normalized
class Config:
"""Global configuration for LoRA Manager"""
@@ -19,9 +53,9 @@ class Config:
self.static_path = os.path.join(os.path.dirname(os.path.dirname(__file__)), 'static')
self.i18n_path = os.path.join(os.path.dirname(os.path.dirname(__file__)), 'locales')
# Path mapping dictionary, target to link mapping
self._path_mappings = {}
# Static route mapping dictionary, target to route mapping
self._route_mappings = {}
self._path_mappings: Dict[str, str] = {}
# Normalized preview root directories used to validate preview access
self._preview_root_paths: Set[Path] = set()
self.loras_roots = self._init_lora_paths()
self.checkpoints_roots = None
self.unet_roots = None
@@ -30,45 +64,74 @@ class Config:
self.embeddings_roots = self._init_embedding_paths()
# Scan symbolic links during initialization
self._scan_symbolic_links()
self._rebuild_preview_roots()
if not standalone_mode:
# Save the paths to settings.json when running in ComfyUI mode
self.save_folder_paths_to_settings()
def save_folder_paths_to_settings(self):
"""Save folder paths to settings.json for standalone mode to use later"""
"""Persist ComfyUI-derived folder paths to the multi-library settings."""
try:
# Check if we're running in ComfyUI mode (not standalone)
# Load existing settings
settings_path = os.path.join(os.path.dirname(os.path.dirname(__file__)), 'settings.json')
settings = {}
if os.path.exists(settings_path):
with open(settings_path, 'r', encoding='utf-8') as f:
settings = json.load(f)
# Update settings with paths
settings['folder_paths'] = {
'loras': self.loras_roots,
'checkpoints': self.checkpoints_roots,
'unet': self.unet_roots,
'embeddings': self.embeddings_roots,
}
# Add default roots if there's only one item and key doesn't exist
if len(self.loras_roots) == 1 and "default_lora_root" not in settings:
settings["default_lora_root"] = self.loras_roots[0]
if self.checkpoints_roots and len(self.checkpoints_roots) == 1 and "default_checkpoint_root" not in settings:
settings["default_checkpoint_root"] = self.checkpoints_roots[0]
ensure_settings_file(logger)
from .services.settings_manager import get_settings_manager
if self.embeddings_roots and len(self.embeddings_roots) == 1 and "default_embedding_root" not in settings:
settings["default_embedding_root"] = self.embeddings_roots[0]
# Save settings
with open(settings_path, 'w', encoding='utf-8') as f:
json.dump(settings, f, indent=2)
logger.info("Saved folder paths to settings.json")
settings_service = get_settings_manager()
libraries = settings_service.get_libraries()
comfy_library = libraries.get("comfyui", {})
default_library = libraries.get("default", {})
target_folder_paths = {
'loras': list(self.loras_roots),
'checkpoints': list(self.checkpoints_roots or []),
'unet': list(self.unet_roots or []),
'embeddings': list(self.embeddings_roots or []),
}
normalized_target_paths = _normalize_folder_paths_for_comparison(target_folder_paths)
if (not comfy_library and default_library and normalized_target_paths and
_normalize_folder_paths_for_comparison(default_library.get("folder_paths", {})) ==
normalized_target_paths):
try:
settings_service.rename_library("default", "comfyui")
logger.info("Renamed legacy 'default' library to 'comfyui'")
libraries = settings_service.get_libraries()
comfy_library = libraries.get("comfyui", {})
except Exception as rename_error:
logger.debug(
"Failed to rename legacy 'default' library: %s", rename_error
)
default_lora_root = comfy_library.get("default_lora_root", "")
if not default_lora_root and len(self.loras_roots) == 1:
default_lora_root = self.loras_roots[0]
default_checkpoint_root = comfy_library.get("default_checkpoint_root", "")
if (not default_checkpoint_root and self.checkpoints_roots and
len(self.checkpoints_roots) == 1):
default_checkpoint_root = self.checkpoints_roots[0]
default_embedding_root = comfy_library.get("default_embedding_root", "")
if (not default_embedding_root and self.embeddings_roots and
len(self.embeddings_roots) == 1):
default_embedding_root = self.embeddings_roots[0]
metadata = dict(comfy_library.get("metadata", {}))
metadata.setdefault("display_name", "ComfyUI")
metadata["source"] = "comfyui"
settings_service.upsert_library(
"comfyui",
folder_paths=target_folder_paths,
default_lora_root=default_lora_root,
default_checkpoint_root=default_checkpoint_root,
default_embedding_root=default_embedding_root,
metadata=metadata,
activate=True,
)
logger.info("Updated 'comfyui' library with current folder paths")
except Exception as e:
logger.warning(f"Failed to save folder paths: {e}")
@@ -125,12 +188,65 @@ class Config:
# Keep the original mapping: target path -> link path
self._path_mappings[normalized_target] = normalized_link
logger.info(f"Added path mapping: {normalized_target} -> {normalized_link}")
self._preview_root_paths.update(self._expand_preview_root(normalized_target))
self._preview_root_paths.update(self._expand_preview_root(normalized_link))
def add_route_mapping(self, path: str, route: str):
"""Add a static route mapping"""
normalized_path = os.path.normpath(path).replace(os.sep, '/')
self._route_mappings[normalized_path] = route
# logger.info(f"Added route mapping: {normalized_path} -> {route}")
def _expand_preview_root(self, path: str) -> Set[Path]:
"""Return normalized ``Path`` objects representing a preview root."""
roots: Set[Path] = set()
if not path:
return roots
try:
raw_path = Path(path).expanduser()
except Exception:
return roots
if raw_path.is_absolute():
roots.add(raw_path)
try:
resolved = raw_path.resolve(strict=False)
except RuntimeError:
resolved = raw_path.absolute()
roots.add(resolved)
try:
real_path = raw_path.resolve()
except (FileNotFoundError, RuntimeError):
real_path = resolved
roots.add(real_path)
normalized: Set[Path] = set()
for candidate in roots:
if candidate.is_absolute():
normalized.add(candidate)
else:
try:
normalized.add(candidate.resolve(strict=False))
except RuntimeError:
normalized.add(candidate.absolute())
return normalized
def _rebuild_preview_roots(self) -> None:
"""Recompute the cache of directories permitted for previews."""
preview_roots: Set[Path] = set()
for root in self.loras_roots or []:
preview_roots.update(self._expand_preview_root(root))
for root in self.base_models_roots or []:
preview_roots.update(self._expand_preview_root(root))
for root in self.embeddings_roots or []:
preview_roots.update(self._expand_preview_root(root))
for target, link in self._path_mappings.items():
preview_roots.update(self._expand_preview_root(target))
preview_roots.update(self._expand_preview_root(link))
self._preview_root_paths = {path for path in preview_roots if path.is_absolute()}
def map_path_to_link(self, path: str) -> str:
"""Map a target path back to its symbolic link path"""
@@ -154,31 +270,93 @@ class Config:
return mapped_path
return link_path
def _dedupe_existing_paths(self, raw_paths: Iterable[str]) -> Dict[str, str]:
dedup: Dict[str, str] = {}
for path in raw_paths:
if not isinstance(path, str):
continue
if not os.path.exists(path):
continue
real_path = os.path.normpath(os.path.realpath(path)).replace(os.sep, '/')
normalized = os.path.normpath(path).replace(os.sep, '/')
if real_path not in dedup:
dedup[real_path] = normalized
return dedup
def _prepare_lora_paths(self, raw_paths: Iterable[str]) -> List[str]:
path_map = self._dedupe_existing_paths(raw_paths)
unique_paths = sorted(path_map.values(), key=lambda p: p.lower())
for original_path in unique_paths:
real_path = os.path.normpath(os.path.realpath(original_path)).replace(os.sep, '/')
if real_path != original_path:
self.add_path_mapping(original_path, real_path)
return unique_paths
def _prepare_checkpoint_paths(
self, checkpoint_paths: Iterable[str], unet_paths: Iterable[str]
) -> List[str]:
checkpoint_map = self._dedupe_existing_paths(checkpoint_paths)
unet_map = self._dedupe_existing_paths(unet_paths)
merged_map: Dict[str, str] = {}
for real_path, original in {**checkpoint_map, **unet_map}.items():
if real_path not in merged_map:
merged_map[real_path] = original
unique_paths = sorted(merged_map.values(), key=lambda p: p.lower())
checkpoint_values = set(checkpoint_map.values())
unet_values = set(unet_map.values())
self.checkpoints_roots = [p for p in unique_paths if p in checkpoint_values]
self.unet_roots = [p for p in unique_paths if p in unet_values]
for original_path in unique_paths:
real_path = os.path.normpath(os.path.realpath(original_path)).replace(os.sep, '/')
if real_path != original_path:
self.add_path_mapping(original_path, real_path)
return unique_paths
def _prepare_embedding_paths(self, raw_paths: Iterable[str]) -> List[str]:
path_map = self._dedupe_existing_paths(raw_paths)
unique_paths = sorted(path_map.values(), key=lambda p: p.lower())
for original_path in unique_paths:
real_path = os.path.normpath(os.path.realpath(original_path)).replace(os.sep, '/')
if real_path != original_path:
self.add_path_mapping(original_path, real_path)
return unique_paths
def _apply_library_paths(self, folder_paths: Mapping[str, Iterable[str]]) -> None:
self._path_mappings.clear()
self._preview_root_paths = set()
lora_paths = folder_paths.get('loras', []) or []
checkpoint_paths = folder_paths.get('checkpoints', []) or []
unet_paths = folder_paths.get('unet', []) or []
embedding_paths = folder_paths.get('embeddings', []) or []
self.loras_roots = self._prepare_lora_paths(lora_paths)
self.base_models_roots = self._prepare_checkpoint_paths(checkpoint_paths, unet_paths)
self.embeddings_roots = self._prepare_embedding_paths(embedding_paths)
self._scan_symbolic_links()
self._rebuild_preview_roots()
def _init_lora_paths(self) -> List[str]:
"""Initialize and validate LoRA paths from ComfyUI settings"""
try:
raw_paths = folder_paths.get_folder_paths("loras")
# Normalize and resolve symlinks, store mapping from resolved -> original
path_map = {}
for path in raw_paths:
if os.path.exists(path):
real_path = os.path.normpath(os.path.realpath(path)).replace(os.sep, '/')
path_map[real_path] = path_map.get(real_path, path.replace(os.sep, "/")) # preserve first seen
# Now sort and use only the deduplicated real paths
unique_paths = sorted(path_map.values(), key=lambda p: p.lower())
unique_paths = self._prepare_lora_paths(raw_paths)
logger.info("Found LoRA roots:" + ("\n - " + "\n - ".join(unique_paths) if unique_paths else "[]"))
if not unique_paths:
logger.warning("No valid loras folders found in ComfyUI configuration")
return []
for original_path in unique_paths:
real_path = os.path.normpath(os.path.realpath(original_path)).replace(os.sep, '/')
if real_path != original_path:
self.add_path_mapping(original_path, real_path)
return unique_paths
except Exception as e:
logger.warning(f"Error initializing LoRA paths: {e}")
@@ -187,52 +365,17 @@ class Config:
def _init_checkpoint_paths(self) -> List[str]:
"""Initialize and validate checkpoint paths from ComfyUI settings"""
try:
# Get checkpoint paths from folder_paths
raw_checkpoint_paths = folder_paths.get_folder_paths("checkpoints")
raw_unet_paths = folder_paths.get_folder_paths("unet")
# Normalize and resolve symlinks for checkpoints, store mapping from resolved -> original
checkpoint_map = {}
for path in raw_checkpoint_paths:
if os.path.exists(path):
real_path = os.path.normpath(os.path.realpath(path)).replace(os.sep, '/')
checkpoint_map[real_path] = checkpoint_map.get(real_path, path.replace(os.sep, "/")) # preserve first seen
# Normalize and resolve symlinks for unet, store mapping from resolved -> original
unet_map = {}
for path in raw_unet_paths:
if os.path.exists(path):
real_path = os.path.normpath(os.path.realpath(path)).replace(os.sep, '/')
unet_map[real_path] = unet_map.get(real_path, path.replace(os.sep, "/")) # preserve first seen
# Merge both maps and deduplicate by real path
merged_map = {}
for real_path, orig_path in {**checkpoint_map, **unet_map}.items():
if real_path not in merged_map:
merged_map[real_path] = orig_path
unique_paths = self._prepare_checkpoint_paths(raw_checkpoint_paths, raw_unet_paths)
# Now sort and use only the deduplicated real paths
unique_paths = sorted(merged_map.values(), key=lambda p: p.lower())
# Split back into checkpoints and unet roots for class properties
self.checkpoints_roots = [p for p in unique_paths if p in checkpoint_map.values()]
self.unet_roots = [p for p in unique_paths if p in unet_map.values()]
all_paths = unique_paths
logger.info("Found checkpoint roots:" + ("\n - " + "\n - ".join(all_paths) if all_paths else "[]"))
if not all_paths:
logger.info("Found checkpoint roots:" + ("\n - " + "\n - ".join(unique_paths) if unique_paths else "[]"))
if not unique_paths:
logger.warning("No valid checkpoint folders found in ComfyUI configuration")
return []
# Initialize path mappings
for original_path in all_paths:
real_path = os.path.normpath(os.path.realpath(original_path)).replace(os.sep, '/')
if real_path != original_path:
self.add_path_mapping(original_path, real_path)
return all_paths
return unique_paths
except Exception as e:
logger.warning(f"Error initializing checkpoint paths: {e}")
return []
@@ -241,27 +384,13 @@ class Config:
"""Initialize and validate embedding paths from ComfyUI settings"""
try:
raw_paths = folder_paths.get_folder_paths("embeddings")
# Normalize and resolve symlinks, store mapping from resolved -> original
path_map = {}
for path in raw_paths:
if os.path.exists(path):
real_path = os.path.normpath(os.path.realpath(path)).replace(os.sep, '/')
path_map[real_path] = path_map.get(real_path, path.replace(os.sep, "/")) # preserve first seen
# Now sort and use only the deduplicated real paths
unique_paths = sorted(path_map.values(), key=lambda p: p.lower())
unique_paths = self._prepare_embedding_paths(raw_paths)
logger.info("Found embedding roots:" + ("\n - " + "\n - ".join(unique_paths) if unique_paths else "[]"))
if not unique_paths:
logger.warning("No valid embeddings folders found in ComfyUI configuration")
return []
for original_path in unique_paths:
real_path = os.path.normpath(os.path.realpath(original_path)).replace(os.sep, '/')
if real_path != original_path:
self.add_path_mapping(original_path, real_path)
return unique_paths
except Exception as e:
logger.warning(f"Error initializing embedding paths: {e}")
@@ -270,25 +399,62 @@ class Config:
def get_preview_static_url(self, preview_path: str) -> str:
if not preview_path:
return ""
real_path = os.path.realpath(preview_path).replace(os.sep, '/')
# Find longest matching path (most specific match)
best_match = ""
best_route = ""
for path, route in self._route_mappings.items():
if real_path.startswith(path) and len(path) > len(best_match):
best_match = path
best_route = route
if best_match:
relative_path = os.path.relpath(real_path, best_match).replace(os.sep, '/')
safe_parts = [urllib.parse.quote(part) for part in relative_path.split('/')]
safe_path = '/'.join(safe_parts)
return f'{best_route}/{safe_path}'
return ""
normalized = os.path.normpath(preview_path).replace(os.sep, '/')
encoded_path = urllib.parse.quote(normalized, safe='')
return f'/api/lm/previews?path={encoded_path}'
def is_preview_path_allowed(self, preview_path: str) -> bool:
"""Return ``True`` if ``preview_path`` is within an allowed directory."""
if not preview_path:
return False
try:
candidate = Path(preview_path).expanduser().resolve(strict=False)
except Exception:
return False
for root in self._preview_root_paths:
try:
candidate.relative_to(root)
return True
except ValueError:
continue
return False
def apply_library_settings(self, library_config: Mapping[str, object]) -> None:
"""Update runtime paths to match the provided library configuration."""
folder_paths = library_config.get('folder_paths') if isinstance(library_config, Mapping) else {}
if not isinstance(folder_paths, Mapping):
folder_paths = {}
self._apply_library_paths(folder_paths)
logger.info(
"Applied library settings with %d lora roots, %d checkpoint roots, and %d embedding roots",
len(self.loras_roots or []),
len(self.base_models_roots or []),
len(self.embeddings_roots or []),
)
def get_library_registry_snapshot(self) -> Dict[str, object]:
"""Return the current library registry and active library name."""
try:
from .services.settings_manager import get_settings_manager
settings_service = get_settings_manager()
libraries = settings_service.get_libraries()
active_library = settings_service.get_active_library_name()
return {
"active_library": active_library,
"libraries": libraries,
}
except Exception as exc: # pragma: no cover - defensive logging
logger.debug("Failed to collect library registry snapshot: %s", exc)
return {"active_library": "", "libraries": {}}
# Global config instance
config = Config()

View File

@@ -2,7 +2,6 @@ import asyncio
import sys
import os
import logging
from pathlib import Path
from server import PromptServer # type: ignore
from .config import config
@@ -11,9 +10,10 @@ from .routes.recipe_routes import RecipeRoutes
from .routes.stats_routes import StatsRoutes
from .routes.update_routes import UpdateRoutes
from .routes.misc_routes import MiscRoutes
from .routes.preview_routes import PreviewRoutes
from .routes.example_images_routes import ExampleImagesRoutes
from .services.service_registry import ServiceRegistry
from .services.settings_manager import settings
from .services.settings_manager import get_settings_manager
from .utils.example_images_migration import ExampleImagesMigration
from .services.websocket_manager import ws_manager
from .services.example_images_cleanup_service import ExampleImagesCleanupService
@@ -23,6 +23,25 @@ logger = logging.getLogger(__name__)
# Check if we're in standalone mode
STANDALONE_MODE = 'nodes' not in sys.modules
class _SettingsProxy:
def __init__(self):
self._manager = None
def _resolve(self):
if self._manager is None:
self._manager = get_settings_manager()
return self._manager
def get(self, *args, **kwargs):
return self._resolve().get(*args, **kwargs)
def __getattr__(self, item):
return getattr(self._resolve(), item)
settings = _SettingsProxy()
class LoraManager:
"""Main entry point for LoRA Manager plugin"""
@@ -50,102 +69,12 @@ class LoraManager:
asyncio_logger = logging.getLogger("asyncio")
asyncio_logger.addFilter(ConnectionResetFilter())
added_targets = set() # Track already added target paths
# Add static route for example images if the path exists in settings
example_images_path = settings.get('example_images_path')
logger.info(f"Example images path: {example_images_path}")
if example_images_path and os.path.exists(example_images_path):
app.router.add_static('/example_images_static', example_images_path)
logger.info(f"Added static route for example images: /example_images_static -> {example_images_path}")
# Add static routes for each lora root
for idx, root in enumerate(config.loras_roots, start=1):
preview_path = f'/loras_static/root{idx}/preview'
real_root = root
if root in config._path_mappings.values():
for target, link in config._path_mappings.items():
if link == root:
real_root = target
break
# Add static route for original path
app.router.add_static(preview_path, real_root)
logger.info(f"Added static route {preview_path} -> {real_root}")
# Record route mapping
config.add_route_mapping(real_root, preview_path)
added_targets.add(real_root)
# Add static routes for each checkpoint root
for idx, root in enumerate(config.base_models_roots, start=1):
preview_path = f'/checkpoints_static/root{idx}/preview'
real_root = root
if root in config._path_mappings.values():
for target, link in config._path_mappings.items():
if link == root:
real_root = target
break
# Add static route for original path
app.router.add_static(preview_path, real_root)
logger.info(f"Added static route {preview_path} -> {real_root}")
# Record route mapping
config.add_route_mapping(real_root, preview_path)
added_targets.add(real_root)
# Add static routes for each embedding root
for idx, root in enumerate(config.embeddings_roots, start=1):
preview_path = f'/embeddings_static/root{idx}/preview'
real_root = root
if root in config._path_mappings.values():
for target, link in config._path_mappings.items():
if link == root:
real_root = target
break
# Add static route for original path
app.router.add_static(preview_path, real_root)
logger.info(f"Added static route {preview_path} -> {real_root}")
# Record route mapping
config.add_route_mapping(real_root, preview_path)
added_targets.add(real_root)
# Add static routes for symlink target paths
link_idx = {
'lora': 1,
'checkpoint': 1,
'embedding': 1
}
for target_path, link_path in config._path_mappings.items():
if target_path not in added_targets:
# Determine if this is a checkpoint, lora, or embedding link based on path
is_checkpoint = any(cp_root in link_path for cp_root in config.base_models_roots)
is_checkpoint = is_checkpoint or any(cp_root in target_path for cp_root in config.base_models_roots)
is_embedding = any(emb_root in link_path for emb_root in config.embeddings_roots)
is_embedding = is_embedding or any(emb_root in target_path for emb_root in config.embeddings_roots)
if is_checkpoint:
route_path = f'/checkpoints_static/link_{link_idx["checkpoint"]}/preview'
link_idx["checkpoint"] += 1
elif is_embedding:
route_path = f'/embeddings_static/link_{link_idx["embedding"]}/preview'
link_idx["embedding"] += 1
else:
route_path = f'/loras_static/link_{link_idx["lora"]}/preview'
link_idx["lora"] += 1
try:
app.router.add_static(route_path, Path(target_path).resolve(strict=False))
logger.info(f"Added static route for link target {route_path} -> {target_path}")
config.add_route_mapping(target_path, route_path)
added_targets.add(target_path)
except Exception as e:
logger.warning(f"Failed to add static route on initialization for {target_path}: {e}")
continue
# Add static route for locales JSON files
if os.path.exists(config.i18n_path):
@@ -168,6 +97,7 @@ class LoraManager:
UpdateRoutes.setup_routes(app)
MiscRoutes.setup_routes(app)
ExampleImagesRoutes.setup_routes(app, ws_manager=ws_manager)
PreviewRoutes.setup_routes(app)
# Setup WebSocket routes that are shared across all model types
app.router.add_get('/ws/fetch-progress', ws_manager.handle_connection)

View File

@@ -1,7 +1,7 @@
import os
# Check if running in standalone mode
standalone_mode = os.environ.get("HF_HUB_DISABLE_TELEMETRY", "0") == "0"
standalone_mode = os.environ.get("LORA_MANAGER_STANDALONE", "0") == "1" or os.environ.get("HF_HUB_DISABLE_TELEMETRY", "0") == "0"
if not standalone_mode:
from .metadata_hook import MetadataHook

View File

@@ -3,7 +3,7 @@ import os
from .constants import IMAGES
# Check if running in standalone mode
standalone_mode = os.environ.get("HF_HUB_DISABLE_TELEMETRY", "0") == "0"
standalone_mode = os.environ.get("LORA_MANAGER_STANDALONE", "0") == "1" or os.environ.get("HF_HUB_DISABLE_TELEMETRY", "0") == "0"
from .constants import MODELS, PROMPTS, SAMPLING, LORAS, SIZE, IS_SAMPLER

View File

@@ -666,6 +666,7 @@ NODE_EXTRACTORS = {
"LoraManagerLoader": LoraLoaderManagerExtractor,
# Conditioning
"CLIPTextEncode": CLIPTextEncodeExtractor,
"PromptLoraManager": CLIPTextEncodeExtractor,
"CLIPTextEncodeFlux": CLIPTextEncodeFluxExtractor, # Add CLIPTextEncodeFlux
"WAS_Text_to_Conditioning": CLIPTextEncodeExtractor,
"AdvancedCLIPTextEncode": CLIPTextEncodeExtractor, # From https://github.com/BlenderNeko/ComfyUI_ADV_CLIP_emb

View File

@@ -1,7 +1,6 @@
import logging
import re
from nodes import LoraLoader
from comfy.comfy_types import IO # type: ignore
from ..utils.utils import get_lora_info
from .utils import FlexibleOptionalInputType, any_type, extract_lora_name, get_loras_list, nunchaku_load_lora
@@ -17,7 +16,7 @@ class LoraManagerLoader:
"required": {
"model": ("MODEL",),
# "clip": ("CLIP",),
"text": (IO.STRING, {
"text": ("STRING", {
"multiline": True,
"pysssss.autocomplete": False,
"dynamicPrompts": True,
@@ -28,7 +27,7 @@ class LoraManagerLoader:
"optional": FlexibleOptionalInputType(any_type),
}
RETURN_TYPES = ("MODEL", "CLIP", IO.STRING, IO.STRING)
RETURN_TYPES = ("MODEL", "CLIP", "STRING", "STRING")
RETURN_NAMES = ("MODEL", "CLIP", "trigger_words", "loaded_loras")
FUNCTION = "load_loras"
@@ -141,7 +140,7 @@ class LoraManagerTextLoader:
return {
"required": {
"model": ("MODEL",),
"lora_syntax": (IO.STRING, {
"lora_syntax": ("STRING", {
"defaultInput": True,
"forceInput": True,
"tooltip": "Format: <lora:lora_name:strength> separated by spaces or punctuation"
@@ -153,7 +152,7 @@ class LoraManagerTextLoader:
}
}
RETURN_TYPES = ("MODEL", "CLIP", IO.STRING, IO.STRING)
RETURN_TYPES = ("MODEL", "CLIP", "STRING", "STRING")
RETURN_NAMES = ("MODEL", "CLIP", "trigger_words", "loaded_loras")
FUNCTION = "load_loras_from_text"

View File

@@ -1,4 +1,3 @@
from comfy.comfy_types import IO # type: ignore
import os
from ..utils.utils import get_lora_info
from .utils import FlexibleOptionalInputType, any_type, extract_lora_name, get_loras_list
@@ -15,7 +14,7 @@ class LoraStacker:
def INPUT_TYPES(cls):
return {
"required": {
"text": (IO.STRING, {
"text": ("STRING", {
"multiline": True,
"pysssss.autocomplete": False,
"dynamicPrompts": True,
@@ -26,7 +25,7 @@ class LoraStacker:
"optional": FlexibleOptionalInputType(any_type),
}
RETURN_TYPES = ("LORA_STACK", IO.STRING, IO.STRING)
RETURN_TYPES = ("LORA_STACK", "STRING", "STRING")
RETURN_NAMES = ("LORA_STACK", "trigger_words", "active_loras")
FUNCTION = "stack_loras"

59
py/nodes/prompt.py Normal file
View File

@@ -0,0 +1,59 @@
from typing import Any, Optional
class PromptLoraManager:
"""Encodes text (and optional trigger words) into CLIP conditioning."""
NAME = "Prompt (LoraManager)"
CATEGORY = "Lora Manager/conditioning"
DESCRIPTION = (
"Encodes a text prompt using a CLIP model into an embedding that can be used "
"to guide the diffusion model towards generating specific images."
)
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"text": (
'STRING',
{
"multiline": True,
"pysssss.autocomplete": False,
"dynamicPrompts": True,
"tooltip": "The text to be encoded.",
},
),
"clip": (
'CLIP',
{"tooltip": "The CLIP model used for encoding the text."},
),
},
"optional": {
"trigger_words": (
'STRING',
{
"forceInput": True,
"tooltip": (
"Optional trigger words to prepend to the text before "
"encoding."
)
},
)
},
}
RETURN_TYPES = ('CONDITIONING', 'STRING',)
RETURN_NAMES = ('CONDITIONING', 'PROMPT',)
OUTPUT_TOOLTIPS = (
"A conditioning containing the embedded text used to guide the diffusion model.",
)
FUNCTION = "encode"
def encode(self, text: str, clip: Any, trigger_words: Optional[str] = None):
prompt = text
if trigger_words:
prompt = ", ".join([trigger_words, text])
from nodes import CLIPTextEncode # type: ignore
conditioning = CLIPTextEncode().encode(clip, prompt)[0]
return (conditioning, prompt,)

View File

@@ -1,6 +1,5 @@
import json
import re
from server import PromptServer # type: ignore
from .utils import FlexibleOptionalInputType, any_type
import logging

View File

@@ -1,4 +1,3 @@
from comfy.comfy_types import IO # type: ignore
import folder_paths # type: ignore
from ..utils.utils import get_lora_info
from .utils import FlexibleOptionalInputType, any_type, get_loras_list
@@ -16,7 +15,7 @@ class WanVideoLoraSelect:
"required": {
"low_mem_load": ("BOOLEAN", {"default": False, "tooltip": "Load LORA models with less VRAM usage, slower loading. This affects ALL LoRAs, not just the current ones. No effect if merge_loras is False"}),
"merge_loras": ("BOOLEAN", {"default": True, "tooltip": "Merge LoRAs into the model, otherwise they are loaded on the fly. Always disabled for GGUF and scaled fp8 models. This affects ALL LoRAs, not just the current one"}),
"text": (IO.STRING, {
"text": ("STRING", {
"multiline": True,
"pysssss.autocomplete": False,
"dynamicPrompts": True,
@@ -27,7 +26,7 @@ class WanVideoLoraSelect:
"optional": FlexibleOptionalInputType(any_type),
}
RETURN_TYPES = ("WANVIDLORA", IO.STRING, IO.STRING)
RETURN_TYPES = ("WANVIDLORA", "STRING", "STRING")
RETURN_NAMES = ("lora", "trigger_words", "active_loras")
FUNCTION = "process_loras"

View File

@@ -1,5 +1,4 @@
from comfy.comfy_types import IO
import folder_paths
import folder_paths # type: ignore
from ..utils.utils import get_lora_info
from .utils import any_type
import logging
@@ -20,7 +19,7 @@ class WanVideoLoraSelectFromText:
"required": {
"low_mem_load": ("BOOLEAN", {"default": False, "tooltip": "Load LORA models with less VRAM usage, slower loading. This affects ALL LoRAs, not just the current ones. No effect if merge_loras is False"}),
"merge_lora": ("BOOLEAN", {"default": True, "tooltip": "Merge LoRAs into the model, otherwise they are loaded on the fly. Always disabled for GGUF and scaled fp8 models. This affects ALL LoRAs, not just the current one"}),
"lora_syntax": (IO.STRING, {
"lora_syntax": ("STRING", {
"multiline": True,
"defaultInput": True,
"forceInput": True,
@@ -34,7 +33,7 @@ class WanVideoLoraSelectFromText:
}
}
RETURN_TYPES = ("WANVIDLORA", IO.STRING, IO.STRING)
RETURN_TYPES = ("WANVIDLORA", "STRING", "STRING")
RETURN_NAMES = ("lora", "trigger_words", "active_loras")
FUNCTION = "process_loras_from_syntax"

View File

@@ -284,7 +284,59 @@ class CivitaiApiMetadataParser(RecipeMetadataParser):
logger.error(f"Error fetching Civitai info for model ID {version_id}: {e}")
result["loras"].append(lora_entry)
# If we found LoRA hashes in the metadata but haven't already
# populated entries for them, fall back to creating LoRAs from
# the hashes section. Some Civitai image responses only include
# LoRA information here without explicit resources entries.
for lora_name, lora_hash in lora_hashes.items():
if not lora_hash:
continue
# Skip LoRAs we've already added via resources or other fields
if lora_hash in added_loras:
continue
lora_entry = {
'name': lora_name,
'type': "lora",
'weight': 1.0,
'hash': lora_hash,
'existsLocally': False,
'localPath': None,
'file_name': lora_name,
'thumbnailUrl': '/loras_static/images/no-preview.png',
'baseModel': '',
'size': 0,
'downloadUrl': '',
'isDeleted': False
}
if metadata_provider:
try:
civitai_info = await metadata_provider.get_model_by_hash(lora_hash)
populated_entry = await self.populate_lora_from_civitai(
lora_entry,
civitai_info,
recipe_scanner,
base_model_counts,
lora_hash
)
if populated_entry is None:
continue
lora_entry = populated_entry
if 'id' in lora_entry and lora_entry['id']:
added_loras[str(lora_entry['id'])] = len(result["loras"])
except Exception as e:
logger.error(f"Error fetching Civitai info for LoRA hash {lora_hash}: {e}")
added_loras[lora_hash] = len(result["loras"])
result["loras"].append(lora_entry)
# Check for LoRA info in the format "Lora_0 Model hash", "Lora_0 Model name", etc.
lora_index = 0
while f"Lora_{lora_index} Model hash" in metadata and f"Lora_{lora_index} Model name" in metadata:

View File

@@ -2,7 +2,7 @@ from __future__ import annotations
import logging
from abc import ABC, abstractmethod
from typing import Callable, Dict, Mapping
from typing import TYPE_CHECKING, Callable, Dict, Mapping
import jinja2
from aiohttp import web
@@ -17,7 +17,7 @@ from ..services.model_lifecycle_service import ModelLifecycleService
from ..services.preview_asset_service import PreviewAssetService
from ..services.server_i18n import server_i18n as default_server_i18n
from ..services.service_registry import ServiceRegistry
from ..services.settings_manager import settings as default_settings
from ..services.settings_manager import get_settings_manager
from ..services.tag_update_service import TagUpdateService
from ..services.websocket_manager import ws_manager as default_ws_manager
from ..services.use_cases import (
@@ -42,8 +42,12 @@ from .handlers.model_handlers import (
ModelMoveHandler,
ModelPageView,
ModelQueryHandler,
ModelUpdateHandler,
)
if TYPE_CHECKING:
from ..services.model_update_service import ModelUpdateService
logger = logging.getLogger(__name__)
@@ -56,14 +60,14 @@ class BaseModelRoutes(ABC):
self,
service=None,
*,
settings_service=default_settings,
settings_service=None,
ws_manager=default_ws_manager,
server_i18n=default_server_i18n,
metadata_provider_factory=get_default_metadata_provider,
) -> None:
self.service = None
self.model_type = ""
self._settings = settings_service
self._settings = settings_service or get_settings_manager()
self._ws_manager = ws_manager
self._server_i18n = server_i18n
self._metadata_provider_factory = metadata_provider_factory
@@ -90,7 +94,7 @@ class BaseModelRoutes(ABC):
self._metadata_sync_service = MetadataSyncService(
metadata_manager=MetadataManager,
preview_service=self._preview_service,
settings=settings_service,
settings=self._settings,
default_metadata_provider_factory=metadata_provider_factory,
metadata_provider_selector=get_metadata_provider,
)
@@ -99,10 +103,18 @@ class BaseModelRoutes(ABC):
ws_manager=self._ws_manager,
download_manager_factory=ServiceRegistry.get_download_manager,
)
self._model_update_service: ModelUpdateService | None = None
if service is not None:
self.attach_service(service)
def set_model_update_service(self, service: "ModelUpdateService") -> None:
"""Attach the model update tracking service."""
self._model_update_service = service
self._handler_set = None
self._handler_mapping = None
def attach_service(self, service) -> None:
"""Attach a model service and rebuild handler dependencies."""
self.service = service
@@ -127,6 +139,7 @@ class BaseModelRoutes(ABC):
def _create_handler_set(self) -> ModelHandlerSet:
service = self._ensure_service()
update_service = self._ensure_model_update_service()
page_view = ModelPageView(
template_env=self.template_env,
template_name=self.template_name or "",
@@ -186,6 +199,12 @@ class BaseModelRoutes(ABC):
ws_manager=self._ws_manager,
logger=logger,
)
updates = ModelUpdateHandler(
service=service,
update_service=update_service,
metadata_provider_selector=get_metadata_provider,
logger=logger,
)
return ModelHandlerSet(
page_view=page_view,
listing=listing,
@@ -195,6 +214,7 @@ class BaseModelRoutes(ABC):
civitai=civitai,
move=move,
auto_organize=auto_organize,
updates=updates,
)
@property
@@ -273,3 +293,8 @@ class BaseModelRoutes(ABC):
return proxy
def _ensure_model_update_service(self) -> "ModelUpdateService":
if self._model_update_service is None:
raise RuntimeError("Model update service has not been attached")
return self._model_update_service

View File

@@ -18,7 +18,7 @@ from ..services.recipes import (
)
from ..services.server_i18n import server_i18n
from ..services.service_registry import ServiceRegistry
from ..services.settings_manager import settings
from ..services.settings_manager import get_settings_manager
from ..utils.constants import CARD_PREVIEW_WIDTH
from ..utils.exif_utils import ExifUtils
from .handlers.recipe_handlers import (
@@ -48,7 +48,7 @@ class BaseRecipeRoutes:
self.recipe_scanner = None
self.lora_scanner = None
self.civitai_client = None
self.settings = settings
self.settings = get_settings_manager()
self.server_i18n = server_i18n
self.template_env = jinja2.Environment(
loader=jinja2.FileSystemLoader(config.templates_path),
@@ -134,7 +134,7 @@ class BaseRecipeRoutes:
recipe_scanner_getter = lambda: self.recipe_scanner
civitai_client_getter = lambda: self.civitai_client
standalone_mode = os.environ.get("HF_HUB_DISABLE_TELEMETRY", "0") == "0"
standalone_mode = os.environ.get("LORA_MANAGER_STANDALONE", "0") == "1" or os.environ.get("HF_HUB_DISABLE_TELEMETRY", "0") == "0"
if not standalone_mode:
from ..metadata_collector import get_metadata # type: ignore[import-not-found]
from ..metadata_collector.metadata_processor import ( # type: ignore[import-not-found]

View File

@@ -20,8 +20,10 @@ class CheckpointRoutes(BaseModelRoutes):
async def initialize_services(self):
"""Initialize services from ServiceRegistry"""
checkpoint_scanner = await ServiceRegistry.get_checkpoint_scanner()
self.service = CheckpointService(checkpoint_scanner)
update_service = await ServiceRegistry.get_model_update_service()
self.service = CheckpointService(checkpoint_scanner, update_service=update_service)
self.set_model_update_service(update_service)
# Attach service dependencies
self.attach_service(self.service)
@@ -93,4 +95,4 @@ class CheckpointRoutes(BaseModelRoutes):
return web.json_response({
"success": False,
"error": str(e)
}, status=500)
}, status=500)

View File

@@ -19,8 +19,10 @@ class EmbeddingRoutes(BaseModelRoutes):
async def initialize_services(self):
"""Initialize services from ServiceRegistry"""
embedding_scanner = await ServiceRegistry.get_embedding_scanner()
self.service = EmbeddingService(embedding_scanner)
update_service = await ServiceRegistry.get_model_update_service()
self.service = EmbeddingService(embedding_scanner, update_service=update_service)
self.set_model_update_service(update_service)
# Attach service dependencies
self.attach_service(self.service)

View File

@@ -22,6 +22,7 @@ ROUTE_DEFINITIONS: tuple[RouteDefinition, ...] = (
RouteDefinition("GET", "/api/lm/example-images-status", "get_example_images_status"),
RouteDefinition("POST", "/api/lm/pause-example-images", "pause_example_images"),
RouteDefinition("POST", "/api/lm/resume-example-images", "resume_example_images"),
RouteDefinition("POST", "/api/lm/stop-example-images", "stop_example_images"),
RouteDefinition("POST", "/api/lm/open-example-images-folder", "open_example_images_folder"),
RouteDefinition("GET", "/api/lm/example-image-files", "get_example_image_files"),
RouteDefinition("GET", "/api/lm/has-example-images", "has_example_images"),

View File

@@ -68,6 +68,13 @@ class ExampleImagesDownloadHandler:
except DownloadNotRunningError as exc:
return web.json_response({'success': False, 'error': str(exc)}, status=400)
async def stop_example_images(self, request: web.Request) -> web.StreamResponse:
try:
result = await self._download_manager.stop_download(request)
return web.json_response(result)
except DownloadNotRunningError as exc:
return web.json_response({'success': False, 'error': str(exc)}, status=400)
async def force_download_example_images(self, request: web.Request) -> web.StreamResponse:
try:
payload = await request.json()
@@ -149,6 +156,7 @@ class ExampleImagesHandlerSet:
"get_example_images_status": self.download.get_example_images_status,
"pause_example_images": self.download.pause_example_images,
"resume_example_images": self.download.resume_example_images,
"stop_example_images": self.download.stop_example_images,
"force_download_example_images": self.download.force_download_example_images,
"import_example_images": self.management.import_example_images,
"delete_example_image": self.management.delete_example_image,

View File

@@ -0,0 +1,980 @@
"""Handlers for miscellaneous routes.
The legacy :mod:`py.routes.misc_routes` module bundled HTTP wiring and
business logic in a single class. This module mirrors the model route
architecture by splitting the responsibilities into dedicated handler
objects that can be composed by the route controller.
"""
from __future__ import annotations
import asyncio
import logging
import os
import subprocess
import sys
from dataclasses import dataclass
from typing import Awaitable, Callable, Dict, Mapping, Protocol
from aiohttp import web
from ...config import config
from ...services.metadata_service import (
get_metadata_archive_manager,
update_metadata_providers,
)
from ...services.service_registry import ServiceRegistry
from ...services.settings_manager import get_settings_manager
from ...services.websocket_manager import ws_manager
from ...services.downloader import get_downloader
from ...utils.constants import (
CIVITAI_USER_MODEL_TYPES,
DEFAULT_NODE_COLOR,
NODE_TYPES,
SUPPORTED_MEDIA_EXTENSIONS,
VALID_LORA_TYPES,
)
from ...utils.civitai_utils import rewrite_preview_url
from ...utils.example_images_paths import is_valid_example_images_root
from ...utils.lora_metadata import extract_trained_words
from ...utils.usage_stats import UsageStats
logger = logging.getLogger(__name__)
class PromptServerProtocol(Protocol):
"""Subset of PromptServer used by the handlers."""
instance: "PromptServerProtocol"
def send_sync(self, event: str, payload: dict) -> None: # pragma: no cover - protocol
...
class DownloaderProtocol(Protocol):
async def refresh_session(self) -> None: # pragma: no cover - protocol
...
class UsageStatsFactory(Protocol):
def __call__(self) -> UsageStats: # pragma: no cover - protocol
...
class MetadataProviderProtocol(Protocol):
async def get_model_versions(self, model_id: int) -> dict | None: # pragma: no cover - protocol
...
class MetadataArchiveManagerProtocol(Protocol):
async def download_and_extract_database(
self, progress_callback: Callable[[str, str], None]
) -> bool: # pragma: no cover - protocol
...
async def remove_database(self) -> bool: # pragma: no cover - protocol
...
def is_database_available(self) -> bool: # pragma: no cover - protocol
...
def get_database_path(self) -> str | None: # pragma: no cover - protocol
...
class NodeRegistry:
"""Thread-safe registry for tracking LoRA nodes in active workflows."""
def __init__(self) -> None:
self._lock = asyncio.Lock()
self._nodes: Dict[str, dict] = {}
self._registry_updated = asyncio.Event()
async def register_nodes(self, nodes: list[dict]) -> None:
async with self._lock:
self._nodes.clear()
for node in nodes:
node_id = node["node_id"]
graph_id = str(node["graph_id"])
unique_id = f"{graph_id}:{node_id}"
node_type = node.get("type", "")
type_id = NODE_TYPES.get(node_type, 0)
bgcolor = node.get("bgcolor") or DEFAULT_NODE_COLOR
self._nodes[unique_id] = {
"id": node_id,
"graph_id": graph_id,
"graph_name": node.get("graph_name"),
"unique_id": unique_id,
"bgcolor": bgcolor,
"title": node.get("title"),
"type": type_id,
"type_name": node_type,
}
logger.debug("Registered %s nodes in registry", len(nodes))
self._registry_updated.set()
async def get_registry(self) -> dict:
async with self._lock:
return {
"nodes": dict(self._nodes),
"node_count": len(self._nodes),
}
async def wait_for_update(self, timeout: float = 1.0) -> bool:
self._registry_updated.clear()
try:
await asyncio.wait_for(self._registry_updated.wait(), timeout=timeout)
return True
except asyncio.TimeoutError:
return False
class HealthCheckHandler:
async def health_check(self, request: web.Request) -> web.Response:
return web.json_response({"status": "ok"})
class SettingsHandler:
"""Sync settings between backend and frontend."""
_SYNC_KEYS = (
"civitai_api_key",
"default_lora_root",
"default_checkpoint_root",
"default_embedding_root",
"base_model_path_mappings",
"download_path_templates",
"enable_metadata_archive_db",
"language",
"proxy_enabled",
"proxy_type",
"proxy_host",
"proxy_port",
"proxy_username",
"proxy_password",
"example_images_path",
"optimize_example_images",
"auto_download_example_images",
"blur_mature_content",
"autoplay_on_hover",
"display_density",
"card_info_display",
"include_trigger_words",
"show_only_sfw",
"compact_mode",
"priority_tags",
"model_name_display",
)
_PROXY_KEYS = {"proxy_enabled", "proxy_host", "proxy_port", "proxy_username", "proxy_password", "proxy_type"}
def __init__(
self,
*,
settings_service=None,
metadata_provider_updater: Callable[[], Awaitable[None]] = update_metadata_providers,
downloader_factory: Callable[[], Awaitable[DownloaderProtocol]] = get_downloader,
) -> None:
self._settings = settings_service or get_settings_manager()
self._metadata_provider_updater = metadata_provider_updater
self._downloader_factory = downloader_factory
async def get_libraries(self, request: web.Request) -> web.Response:
"""Return the registered libraries and the active selection."""
try:
snapshot = config.get_library_registry_snapshot()
libraries = snapshot.get("libraries", {})
active_library = snapshot.get("active_library", "")
return web.json_response(
{
"success": True,
"libraries": libraries,
"active_library": active_library,
}
)
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Error getting library registry: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def get_settings(self, request: web.Request) -> web.Response:
try:
response_data = {}
for key in self._SYNC_KEYS:
value = self._settings.get(key)
if value is not None:
response_data[key] = value
return web.json_response({"success": True, "settings": response_data})
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Error getting settings: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def get_priority_tags(self, request: web.Request) -> web.Response:
try:
suggestions = self._settings.get_priority_tag_suggestions()
return web.json_response({"success": True, "tags": suggestions})
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Error getting priority tags: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def activate_library(self, request: web.Request) -> web.Response:
"""Activate the selected library."""
try:
data = await request.json()
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Error parsing activate library request: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": "Invalid JSON payload"}, status=400)
library_name = data.get("library") or data.get("library_name")
if not isinstance(library_name, str) or not library_name.strip():
return web.json_response(
{"success": False, "error": "Library name is required"}, status=400
)
try:
normalized_name = library_name.strip()
self._settings.activate_library(normalized_name)
snapshot = config.get_library_registry_snapshot()
libraries = snapshot.get("libraries", {})
active_library = snapshot.get("active_library", "")
return web.json_response(
{
"success": True,
"active_library": active_library,
"libraries": libraries,
}
)
except KeyError as exc:
logger.debug("Attempted to activate unknown library '%s'", library_name)
return web.json_response({"success": False, "error": str(exc)}, status=404)
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Error activating library '%s': %s", library_name, exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def update_settings(self, request: web.Request) -> web.Response:
try:
data = await request.json()
proxy_changed = False
for key, value in data.items():
if value == self._settings.get(key):
continue
if key == "example_images_path" and value:
validation_error = self._validate_example_images_path(value)
if validation_error:
return web.json_response({"success": False, "error": validation_error})
if value == "__DELETE__" and key in ("proxy_username", "proxy_password"):
self._settings.delete(key)
else:
self._settings.set(key, value)
if key == "enable_metadata_archive_db":
await self._metadata_provider_updater()
if key in self._PROXY_KEYS:
proxy_changed = True
if proxy_changed:
downloader = await self._downloader_factory()
await downloader.refresh_session()
return web.json_response({"success": True})
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Error updating settings: %s", exc, exc_info=True)
return web.Response(status=500, text=str(exc))
def _validate_example_images_path(self, folder_path: str) -> str | None:
if not os.path.exists(folder_path):
return f"Path does not exist: {folder_path}"
if not os.path.isdir(folder_path):
return "Please set a dedicated folder for example images."
if not self._is_dedicated_example_images_folder(folder_path):
return "Please set a dedicated folder for example images."
return None
def _is_dedicated_example_images_folder(self, folder_path: str) -> bool:
return is_valid_example_images_root(folder_path)
class UsageStatsHandler:
def __init__(self, usage_stats_factory: UsageStatsFactory = UsageStats) -> None:
self._usage_stats_factory = usage_stats_factory
async def update_usage_stats(self, request: web.Request) -> web.Response:
try:
data = await request.json()
prompt_id = data.get("prompt_id")
if not prompt_id:
return web.json_response({"success": False, "error": "Missing prompt_id"}, status=400)
usage_stats = self._usage_stats_factory()
await usage_stats.process_execution(prompt_id)
return web.json_response({"success": True})
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Failed to update usage stats: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def get_usage_stats(self, request: web.Request) -> web.Response:
try:
usage_stats = self._usage_stats_factory()
stats = await usage_stats.get_stats()
stats_response = {"success": True, "data": stats, "format_version": 2}
return web.json_response(stats_response)
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Failed to get usage stats: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
class LoraCodeHandler:
def __init__(self, prompt_server: type[PromptServerProtocol]) -> None:
self._prompt_server = prompt_server
async def update_lora_code(self, request: web.Request) -> web.Response:
try:
data = await request.json()
node_ids = data.get("node_ids")
lora_code = data.get("lora_code", "")
mode = data.get("mode", "append")
if not lora_code:
return web.json_response({"success": False, "error": "Missing lora_code parameter"}, status=400)
results = []
if node_ids is None:
try:
self._prompt_server.instance.send_sync(
"lora_code_update", {"id": -1, "lora_code": lora_code, "mode": mode}
)
results.append({"node_id": "broadcast", "success": True})
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Error broadcasting lora code: %s", exc)
results.append({"node_id": "broadcast", "success": False, "error": str(exc)})
else:
for entry in node_ids:
node_identifier = entry
graph_identifier = None
if isinstance(entry, dict):
node_identifier = entry.get("node_id")
graph_identifier = entry.get("graph_id")
if node_identifier is None:
results.append(
{
"node_id": node_identifier,
"graph_id": graph_identifier,
"success": False,
"error": "Missing node_id parameter",
}
)
continue
try:
parsed_node_id = int(node_identifier)
except (TypeError, ValueError):
parsed_node_id = node_identifier
payload = {
"id": parsed_node_id,
"lora_code": lora_code,
"mode": mode,
}
if graph_identifier is not None:
payload["graph_id"] = str(graph_identifier)
try:
self._prompt_server.instance.send_sync(
"lora_code_update",
payload,
)
results.append(
{
"node_id": parsed_node_id,
"graph_id": payload.get("graph_id"),
"success": True,
}
)
except Exception as exc: # pragma: no cover - defensive logging
logger.error(
"Error sending lora code to node %s (graph %s): %s",
parsed_node_id,
graph_identifier,
exc,
)
results.append(
{
"node_id": parsed_node_id,
"graph_id": payload.get("graph_id"),
"success": False,
"error": str(exc),
}
)
return web.json_response({"success": True, "results": results})
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Failed to update lora code: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
class TrainedWordsHandler:
async def get_trained_words(self, request: web.Request) -> web.Response:
try:
file_path = request.query.get("file_path")
if not file_path:
return web.json_response({"success": False, "error": "Missing file_path parameter"}, status=400)
if not os.path.exists(file_path):
return web.json_response({"success": False, "error": "File not found"}, status=404)
if not file_path.endswith(".safetensors"):
return web.json_response({"success": False, "error": "File must be a safetensors file"}, status=400)
trained_words, class_tokens = await extract_trained_words(file_path)
return web.json_response(
{
"success": True,
"trained_words": trained_words,
"class_tokens": class_tokens,
}
)
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Failed to get trained words: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
class ModelExampleFilesHandler:
async def get_model_example_files(self, request: web.Request) -> web.Response:
try:
model_path = request.query.get("model_path")
if not model_path:
return web.json_response({"success": False, "error": "Missing model_path parameter"}, status=400)
model_dir = os.path.dirname(model_path)
if not os.path.exists(model_dir):
return web.json_response({"success": False, "error": "Model directory not found"}, status=404)
base_name = os.path.splitext(os.path.basename(model_path))[0]
files = []
pattern = f"{base_name}.example."
for file in os.listdir(model_dir):
if not file.startswith(pattern):
continue
file_full_path = os.path.join(model_dir, file)
if not os.path.isfile(file_full_path):
continue
file_ext = os.path.splitext(file)[1].lower()
if file_ext not in SUPPORTED_MEDIA_EXTENSIONS["images"] and file_ext not in SUPPORTED_MEDIA_EXTENSIONS["videos"]:
continue
try:
index = int(file[len(pattern) :].split(".")[0])
except (ValueError, IndexError):
index = float("inf")
static_url = config.get_preview_static_url(file_full_path)
files.append(
{
"name": file,
"path": static_url,
"extension": file_ext,
"is_video": file_ext in SUPPORTED_MEDIA_EXTENSIONS["videos"],
"index": index,
}
)
files.sort(key=lambda item: item["index"])
for file in files:
file.pop("index", None)
return web.json_response({"success": True, "files": files})
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Failed to get model example files: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
@dataclass
class ServiceRegistryAdapter:
get_lora_scanner: Callable[[], Awaitable]
get_checkpoint_scanner: Callable[[], Awaitable]
get_embedding_scanner: Callable[[], Awaitable]
class ModelLibraryHandler:
def __init__(self, service_registry: ServiceRegistryAdapter, metadata_provider_factory: Callable[[], Awaitable[MetadataProviderProtocol | None]]) -> None:
self._service_registry = service_registry
self._metadata_provider_factory = metadata_provider_factory
async def check_model_exists(self, request: web.Request) -> web.Response:
try:
model_id_str = request.query.get("modelId")
model_version_id_str = request.query.get("modelVersionId")
if not model_id_str:
return web.json_response({"success": False, "error": "Missing required parameter: modelId"}, status=400)
try:
model_id = int(model_id_str)
except ValueError:
return web.json_response({"success": False, "error": "Parameter modelId must be an integer"}, status=400)
lora_scanner = await self._service_registry.get_lora_scanner()
checkpoint_scanner = await self._service_registry.get_checkpoint_scanner()
embedding_scanner = await self._service_registry.get_embedding_scanner()
if model_version_id_str:
try:
model_version_id = int(model_version_id_str)
except ValueError:
return web.json_response({"success": False, "error": "Parameter modelVersionId must be an integer"}, status=400)
exists = False
model_type = None
if await lora_scanner.check_model_version_exists(model_version_id):
exists = True
model_type = "lora"
elif checkpoint_scanner and await checkpoint_scanner.check_model_version_exists(model_version_id):
exists = True
model_type = "checkpoint"
elif embedding_scanner and await embedding_scanner.check_model_version_exists(model_version_id):
exists = True
model_type = "embedding"
return web.json_response({"success": True, "exists": exists, "modelType": model_type if exists else None})
lora_versions = await lora_scanner.get_model_versions_by_id(model_id)
checkpoint_versions = []
embedding_versions = []
if not lora_versions and checkpoint_scanner:
checkpoint_versions = await checkpoint_scanner.get_model_versions_by_id(model_id)
if not lora_versions and not checkpoint_versions and embedding_scanner:
embedding_versions = await embedding_scanner.get_model_versions_by_id(model_id)
model_type = None
versions = []
if lora_versions:
model_type = "lora"
versions = lora_versions
elif checkpoint_versions:
model_type = "checkpoint"
versions = checkpoint_versions
elif embedding_versions:
model_type = "embedding"
versions = embedding_versions
return web.json_response({"success": True, "modelType": model_type, "versions": versions})
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Failed to check model existence: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def get_model_versions_status(self, request: web.Request) -> web.Response:
try:
model_id_str = request.query.get("modelId")
if not model_id_str:
return web.json_response({"success": False, "error": "Missing required parameter: modelId"}, status=400)
try:
model_id = int(model_id_str)
except ValueError:
return web.json_response({"success": False, "error": "Parameter modelId must be an integer"}, status=400)
metadata_provider = await self._metadata_provider_factory()
if not metadata_provider:
return web.json_response({"success": False, "error": "Metadata provider not available"}, status=503)
response = await metadata_provider.get_model_versions(model_id)
if not response or not response.get("modelVersions"):
return web.json_response({"success": False, "error": "Model not found"}, status=404)
versions = response.get("modelVersions", [])
model_name = response.get("name", "")
model_type = response.get("type", "").lower()
scanner = None
normalized_type = None
if model_type in {"lora", "locon", "dora"}:
scanner = await self._service_registry.get_lora_scanner()
normalized_type = "lora"
elif model_type == "checkpoint":
scanner = await self._service_registry.get_checkpoint_scanner()
normalized_type = "checkpoint"
elif model_type == "textualinversion":
scanner = await self._service_registry.get_embedding_scanner()
normalized_type = "embedding"
else:
return web.json_response({"success": False, "error": f'Model type "{model_type}" is not supported'}, status=400)
if not scanner:
return web.json_response({"success": False, "error": f'Scanner for type "{normalized_type}" is not available'}, status=503)
local_versions = await scanner.get_model_versions_by_id(model_id)
local_version_ids = {version["versionId"] for version in local_versions}
enriched_versions = []
for version in versions:
version_id = version.get("id")
enriched_versions.append(
{
"id": version_id,
"name": version.get("name", ""),
"thumbnailUrl": version.get("images")[0]["url"] if version.get("images") else None,
"inLibrary": version_id in local_version_ids,
}
)
return web.json_response(
{
"success": True,
"modelId": model_id,
"modelName": model_name,
"modelType": model_type,
"versions": enriched_versions,
}
)
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Failed to get model versions status: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def get_civitai_user_models(self, request: web.Request) -> web.Response:
try:
username = request.query.get("username")
if not username:
return web.json_response({"success": False, "error": "Missing required parameter: username"}, status=400)
metadata_provider = await self._metadata_provider_factory()
if not metadata_provider:
return web.json_response({"success": False, "error": "Metadata provider not available"}, status=503)
try:
models = await metadata_provider.get_user_models(username)
except NotImplementedError:
return web.json_response({"success": False, "error": "Metadata provider does not support user model queries"}, status=501)
if models is None:
return web.json_response({"success": False, "error": "Failed to fetch user models"}, status=502)
if not isinstance(models, list):
models = []
lora_scanner = await self._service_registry.get_lora_scanner()
checkpoint_scanner = await self._service_registry.get_checkpoint_scanner()
embedding_scanner = await self._service_registry.get_embedding_scanner()
normalized_allowed_types = {model_type.lower() for model_type in CIVITAI_USER_MODEL_TYPES}
lora_type_aliases = {model_type.lower() for model_type in VALID_LORA_TYPES}
type_scanner_map: Dict[str, object | None] = {
**{alias: lora_scanner for alias in lora_type_aliases},
"checkpoint": checkpoint_scanner,
"textualinversion": embedding_scanner,
}
versions: list[dict] = []
for model in models:
if not isinstance(model, dict):
continue
model_type = str(model.get("type", "")).lower()
if model_type not in normalized_allowed_types:
continue
scanner = type_scanner_map.get(model_type)
if scanner is None:
return web.json_response({"success": False, "error": f'Scanner for type "{model_type}" is not available'}, status=503)
tags_value = model.get("tags")
tags = tags_value if isinstance(tags_value, list) else []
model_id = model.get("id")
try:
model_id_int = int(model_id)
except (TypeError, ValueError):
continue
model_name = model.get("name", "")
versions_data = model.get("modelVersions")
if not isinstance(versions_data, list):
continue
for version in versions_data:
if not isinstance(version, dict):
continue
version_id = version.get("id")
try:
version_id_int = int(version_id)
except (TypeError, ValueError):
continue
images = version.get("images") or []
thumbnail_url = None
if images and isinstance(images, list):
first_image = images[0]
if isinstance(first_image, dict):
raw_url = first_image.get("url")
media_type = first_image.get("type")
rewritten_url, _ = rewrite_preview_url(raw_url, media_type)
thumbnail_url = rewritten_url
in_library = await scanner.check_model_version_exists(version_id_int)
versions.append(
{
"modelId": model_id_int,
"versionId": version_id_int,
"modelName": model_name,
"versionName": version.get("name", ""),
"type": model.get("type"),
"tags": tags,
"baseModel": version.get("baseModel"),
"thumbnailUrl": thumbnail_url,
"inLibrary": in_library,
}
)
return web.json_response({"success": True, "username": username, "versions": versions})
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Failed to get Civitai user models: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
class MetadataArchiveHandler:
def __init__(
self,
*,
metadata_archive_manager_factory: Callable[[], Awaitable[MetadataArchiveManagerProtocol]] = get_metadata_archive_manager,
settings_service=None,
metadata_provider_updater: Callable[[], Awaitable[None]] = update_metadata_providers,
) -> None:
self._metadata_archive_manager_factory = metadata_archive_manager_factory
self._settings = settings_service or get_settings_manager()
self._metadata_provider_updater = metadata_provider_updater
async def download_metadata_archive(self, request: web.Request) -> web.Response:
try:
archive_manager = await self._metadata_archive_manager_factory()
download_id = request.query.get("download_id")
def progress_callback(stage: str, message: str) -> None:
data = {"stage": stage, "message": message, "type": "metadata_archive_download"}
if download_id:
asyncio.create_task(ws_manager.broadcast_download_progress(download_id, data))
else:
asyncio.create_task(ws_manager.broadcast(data))
success = await archive_manager.download_and_extract_database(progress_callback)
if success:
self._settings.set("enable_metadata_archive_db", True)
await self._metadata_provider_updater()
return web.json_response({"success": True, "message": "Metadata archive database downloaded and extracted successfully"})
return web.json_response({"success": False, "error": "Failed to download and extract metadata archive database"}, status=500)
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Error downloading metadata archive: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def remove_metadata_archive(self, request: web.Request) -> web.Response:
try:
archive_manager = await self._metadata_archive_manager_factory()
success = await archive_manager.remove_database()
if success:
self._settings.set("enable_metadata_archive_db", False)
await self._metadata_provider_updater()
return web.json_response({"success": True, "message": "Metadata archive database removed successfully"})
return web.json_response({"success": False, "error": "Failed to remove metadata archive database"}, status=500)
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Error removing metadata archive: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def get_metadata_archive_status(self, request: web.Request) -> web.Response:
try:
archive_manager = await self._metadata_archive_manager_factory()
is_available = archive_manager.is_database_available()
is_enabled = self._settings.get("enable_metadata_archive_db", False)
db_size = 0
if is_available:
db_path = archive_manager.get_database_path()
if db_path and os.path.exists(db_path):
db_size = os.path.getsize(db_path)
return web.json_response(
{
"success": True,
"isAvailable": is_available,
"isEnabled": is_enabled,
"databaseSize": db_size,
"databasePath": archive_manager.get_database_path() if is_available else None,
}
)
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Error getting metadata archive status: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
class FileSystemHandler:
async def open_file_location(self, request: web.Request) -> web.Response:
try:
data = await request.json()
file_path = data.get("file_path")
if not file_path:
return web.json_response({"success": False, "error": "Missing file_path parameter"}, status=400)
file_path = os.path.abspath(file_path)
if not os.path.isfile(file_path):
return web.json_response({"success": False, "error": "File does not exist"}, status=404)
if os.name == "nt":
subprocess.Popen(["explorer", "/select,", file_path])
elif os.name == "posix":
if sys.platform == "darwin":
subprocess.Popen(["open", "-R", file_path])
else:
folder = os.path.dirname(file_path)
subprocess.Popen(["xdg-open", folder])
return web.json_response({"success": True, "message": f"Opened folder and selected file: {file_path}"})
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Failed to open file location: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
class NodeRegistryHandler:
def __init__(
self,
node_registry: NodeRegistry,
prompt_server: type[PromptServerProtocol],
*,
standalone_mode: bool,
) -> None:
self._node_registry = node_registry
self._prompt_server = prompt_server
self._standalone_mode = standalone_mode
async def register_nodes(self, request: web.Request) -> web.Response:
try:
data = await request.json()
nodes = data.get("nodes", [])
if not isinstance(nodes, list):
return web.json_response({"success": False, "error": "nodes must be a list"}, status=400)
for index, node in enumerate(nodes):
if not isinstance(node, dict):
return web.json_response({"success": False, "error": f"Node {index} must be an object"}, status=400)
node_id = node.get("node_id")
if node_id is None:
return web.json_response({"success": False, "error": f"Node {index} missing node_id parameter"}, status=400)
graph_id = node.get("graph_id")
if graph_id is None:
return web.json_response({"success": False, "error": f"Node {index} missing graph_id parameter"}, status=400)
graph_name = node.get("graph_name")
try:
node["node_id"] = int(node_id)
except (TypeError, ValueError):
return web.json_response({"success": False, "error": f"Node {index} node_id must be an integer"}, status=400)
node["graph_id"] = str(graph_id)
if graph_name is None:
node["graph_name"] = None
elif isinstance(graph_name, str):
node["graph_name"] = graph_name
else:
node["graph_name"] = str(graph_name)
await self._node_registry.register_nodes(nodes)
return web.json_response({"success": True, "message": f"{len(nodes)} nodes registered successfully"})
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Failed to register nodes: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def get_registry(self, request: web.Request) -> web.Response:
try:
if self._standalone_mode:
logger.warning("Registry refresh not available in standalone mode")
return web.json_response(
{
"success": False,
"error": "Standalone Mode Active",
"message": "Cannot interact with ComfyUI in standalone mode.",
},
status=503,
)
try:
self._prompt_server.instance.send_sync("lora_registry_refresh", {})
logger.debug("Sent registry refresh request to frontend")
except Exception as exc:
logger.error("Failed to send registry refresh message: %s", exc)
return web.json_response(
{
"success": False,
"error": "Communication Error",
"message": f"Failed to communicate with ComfyUI frontend: {exc}",
},
status=500,
)
registry_updated = await self._node_registry.wait_for_update(timeout=1.0)
if not registry_updated:
logger.warning("Registry refresh timeout after 1 second")
return web.json_response(
{
"success": False,
"error": "Timeout Error",
"message": "Registry refresh timeout - ComfyUI frontend may not be responsive",
},
status=408,
)
registry_info = await self._node_registry.get_registry()
return web.json_response({"success": True, "data": registry_info})
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Failed to get registry: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": "Internal Error", "message": str(exc)}, status=500)
class MiscHandlerSet:
"""Aggregate handlers into a lookup compatible with the registrar."""
def __init__(
self,
*,
health: HealthCheckHandler,
settings: SettingsHandler,
usage_stats: UsageStatsHandler,
lora_code: LoraCodeHandler,
trained_words: TrainedWordsHandler,
model_examples: ModelExampleFilesHandler,
node_registry: NodeRegistryHandler,
model_library: ModelLibraryHandler,
metadata_archive: MetadataArchiveHandler,
filesystem: FileSystemHandler,
) -> None:
self.health = health
self.settings = settings
self.usage_stats = usage_stats
self.lora_code = lora_code
self.trained_words = trained_words
self.model_examples = model_examples
self.node_registry = node_registry
self.model_library = model_library
self.metadata_archive = metadata_archive
self.filesystem = filesystem
def to_route_mapping(self) -> Mapping[str, Callable[[web.Request], Awaitable[web.StreamResponse]]]:
return {
"health_check": self.health.health_check,
"get_settings": self.settings.get_settings,
"update_settings": self.settings.update_settings,
"get_priority_tags": self.settings.get_priority_tags,
"get_settings_libraries": self.settings.get_libraries,
"activate_library": self.settings.activate_library,
"update_usage_stats": self.usage_stats.update_usage_stats,
"get_usage_stats": self.usage_stats.get_usage_stats,
"update_lora_code": self.lora_code.update_lora_code,
"get_trained_words": self.trained_words.get_trained_words,
"get_model_example_files": self.model_examples.get_model_example_files,
"register_nodes": self.node_registry.register_nodes,
"get_registry": self.node_registry.get_registry,
"check_model_exists": self.model_library.check_model_exists,
"get_civitai_user_models": self.model_library.get_civitai_user_models,
"download_metadata_archive": self.metadata_archive.download_metadata_archive,
"remove_metadata_archive": self.metadata_archive.remove_metadata_archive,
"get_metadata_archive_status": self.metadata_archive.get_metadata_archive_status,
"get_model_versions_status": self.model_library.get_model_versions_status,
"open_file_location": self.filesystem.open_file_location,
}
def build_service_registry_adapter() -> ServiceRegistryAdapter:
return ServiceRegistryAdapter(
get_lora_scanner=ServiceRegistry.get_lora_scanner,
get_checkpoint_scanner=ServiceRegistry.get_checkpoint_scanner,
get_embedding_scanner=ServiceRegistry.get_embedding_scanner,
)

View File

@@ -6,7 +6,7 @@ import json
import logging
import os
from dataclasses import dataclass
from typing import Awaitable, Callable, Dict, Iterable, Mapping, Optional
from typing import Awaitable, Callable, Dict, Iterable, List, Mapping, Optional
from aiohttp import web
import jinja2
@@ -29,7 +29,9 @@ from ...services.use_cases import (
)
from ...services.websocket_manager import WebSocketManager
from ...services.websocket_progress_callback import WebSocketProgressCallback
from ...services.errors import RateLimitError
from ...utils.file_utils import calculate_sha256
from ...utils.metadata_manager import MetadataManager
class ModelPageView:
@@ -164,6 +166,11 @@ class ModelListingHandler:
except (json.JSONDecodeError, TypeError):
pass
has_update = request.query.get("has_update", "false")
has_update_filter = (
has_update.lower() in {"1", "true", "yes"} if isinstance(has_update, str) else False
)
return {
"page": page,
"page_size": page_size,
@@ -176,6 +183,7 @@ class ModelListingHandler:
"search_options": search_options,
"hash_filters": hash_filters,
"favorites_only": favorites_only,
"has_update": has_update_filter,
**self._parse_specific_params(request),
}
@@ -244,6 +252,8 @@ class ModelManagementHandler:
if not model_data.get("sha256"):
return web.json_response({"success": False, "error": "No SHA256 hash found"}, status=400)
await MetadataManager.hydrate_model_data(model_data)
success, error = await self._metadata_sync.fetch_and_update_model(
sha256=model_data["sha256"],
file_path=file_path,
@@ -755,6 +765,30 @@ class ModelDownloadHandler:
self._logger.error("Error cancelling download via GET: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def pause_download_get(self, request: web.Request) -> web.Response:
try:
download_id = request.query.get("download_id")
if not download_id:
return web.json_response({"success": False, "error": "Download ID is required"}, status=400)
result = await self._download_coordinator.pause_download(download_id)
status = 200 if result.get("success") else 400
return web.json_response(result, status=status)
except Exception as exc:
self._logger.error("Error pausing download via GET: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def resume_download_get(self, request: web.Request) -> web.Response:
try:
download_id = request.query.get("download_id")
if not download_id:
return web.json_response({"success": False, "error": "Download ID is required"}, status=400)
result = await self._download_coordinator.resume_download(download_id)
status = 200 if result.get("success") else 400
return web.json_response(result, status=status)
except Exception as exc:
self._logger.error("Error resuming download via GET: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
async def get_download_progress(self, request: web.Request) -> web.Response:
try:
download_id = request.match_info.get("download_id")
@@ -763,7 +797,23 @@ class ModelDownloadHandler:
progress_data = self._ws_manager.get_download_progress(download_id)
if progress_data is None:
return web.json_response({"success": False, "error": "Download ID not found"}, status=404)
return web.json_response({"success": True, "progress": progress_data.get("progress", 0)})
response_payload = {
"success": True,
"progress": progress_data.get("progress", 0),
"bytes_downloaded": progress_data.get("bytes_downloaded"),
"total_bytes": progress_data.get("total_bytes"),
"bytes_per_second": progress_data.get("bytes_per_second", 0.0),
}
status = progress_data.get("status")
if status and status != "progress":
response_payload["status"] = status
if "message" in progress_data:
response_payload["message"] = progress_data["message"]
elif status is None and "message" in progress_data:
response_payload["message"] = progress_data["message"]
return web.json_response(response_payload)
except Exception as exc:
self._logger.error("Error getting download progress: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
@@ -825,18 +875,30 @@ class ModelCivitaiHandler:
status=400,
)
cache = await self._service.scanner.get_cached_data()
version_index = cache.version_index
for version in versions:
model_file = self._find_model_file(version.get("files", [])) if isinstance(version.get("files"), Iterable) else None
if model_file:
hashes = model_file.get("hashes", {}) if isinstance(model_file, Mapping) else {}
sha256 = hashes.get("SHA256") if isinstance(hashes, Mapping) else None
if sha256:
version["existsLocally"] = self._service.has_hash(sha256)
if version["existsLocally"]:
version["localPath"] = self._service.get_path_by_hash(sha256)
version["modelSizeKB"] = model_file.get("sizeKB") if isinstance(model_file, Mapping) else None
version_id = None
version_id_raw = version.get("id")
if version_id_raw is not None:
try:
version_id = int(str(version_id_raw))
except (TypeError, ValueError):
version_id = None
cache_entry = version_index.get(version_id) if (version_id is not None and version_index) else None
version["existsLocally"] = cache_entry is not None
if cache_entry and isinstance(cache_entry, Mapping):
local_path = cache_entry.get("file_path")
if local_path:
version["localPath"] = local_path
else:
version["existsLocally"] = False
version.pop("localPath", None)
model_file = self._find_model_file(version.get("files", [])) if isinstance(version.get("files"), Iterable) else None
if model_file and isinstance(model_file, Mapping):
version["modelSizeKB"] = model_file.get("sizeKB")
return web.json_response(versions)
except Exception as exc:
self._logger.error("Error fetching %s model versions: %s", self._service.model_type, exc)
@@ -962,6 +1024,156 @@ class ModelAutoOrganizeHandler:
return web.json_response({"success": False, "error": str(exc)}, status=500)
class ModelUpdateHandler:
"""Handle update tracking requests."""
def __init__(
self,
*,
service,
update_service,
metadata_provider_selector,
logger: logging.Logger,
) -> None:
self._service = service
self._update_service = update_service
self._metadata_provider_selector = metadata_provider_selector
self._logger = logger
async def refresh_model_updates(self, request: web.Request) -> web.Response:
payload = await self._read_json(request)
force_refresh = self._parse_bool(request.query.get("force")) or self._parse_bool(
payload.get("force")
)
provider = await self._get_civitai_provider()
if provider is None:
return web.json_response(
{"success": False, "error": "Civitai provider not available"}, status=503
)
try:
records = await self._update_service.refresh_for_model_type(
self._service.model_type,
self._service.scanner,
provider,
force_refresh=force_refresh,
)
except RateLimitError as exc:
return web.json_response(
{"success": False, "error": str(exc) or "Rate limited"}, status=429
)
except Exception as exc: # pragma: no cover - defensive logging
self._logger.error("Failed to refresh model updates: %s", exc, exc_info=True)
return web.json_response({"success": False, "error": str(exc)}, status=500)
return web.json_response(
{
"success": True,
"records": [self._serialize_record(record) for record in records.values()],
}
)
async def set_model_update_ignore(self, request: web.Request) -> web.Response:
payload = await self._read_json(request)
model_id = self._normalize_model_id(payload.get("modelId"))
if model_id is None:
return web.json_response({"success": False, "error": "modelId is required"}, status=400)
should_ignore = self._parse_bool(payload.get("shouldIgnore"))
record = await self._update_service.set_should_ignore(
self._service.model_type, model_id, should_ignore
)
return web.json_response({"success": True, "record": self._serialize_record(record)})
async def get_model_update_status(self, request: web.Request) -> web.Response:
model_id = self._normalize_model_id(request.match_info.get("model_id"))
if model_id is None:
return web.json_response({"success": False, "error": "model_id must be an integer"}, status=400)
refresh = self._parse_bool(request.query.get("refresh"))
force = self._parse_bool(request.query.get("force"))
try:
record = await self._get_or_refresh_record(model_id, refresh=refresh, force=force)
except RateLimitError as exc:
return web.json_response(
{"success": False, "error": str(exc) or "Rate limited"}, status=429
)
if record is None:
return web.json_response(
{"success": False, "error": "Model not tracked"}, status=404
)
return web.json_response({"success": True, "record": self._serialize_record(record)})
async def _get_or_refresh_record(
self, model_id: int, *, refresh: bool, force: bool
) -> Optional[object]:
record = await self._update_service.get_record(self._service.model_type, model_id)
if record and not refresh and not force:
return record
provider = await self._get_civitai_provider()
if provider is None:
return record
return await self._update_service.refresh_single_model(
self._service.model_type,
model_id,
self._service.scanner,
provider,
force_refresh=force or refresh,
)
async def _get_civitai_provider(self):
try:
return await self._metadata_provider_selector("civitai_api")
except Exception as exc: # pragma: no cover - defensive log
self._logger.error("Failed to acquire civitai provider: %s", exc, exc_info=True)
return None
async def _read_json(self, request: web.Request) -> Dict:
if not request.can_read_body:
return {}
try:
return await request.json()
except Exception:
return {}
@staticmethod
def _parse_bool(value) -> bool:
if isinstance(value, bool):
return value
if isinstance(value, str):
return value.lower() in {"1", "true", "yes"}
if isinstance(value, (int, float)):
return bool(value)
return False
@staticmethod
def _normalize_model_id(value) -> Optional[int]:
try:
if value is None:
return None
return int(value)
except (TypeError, ValueError):
return None
@staticmethod
def _serialize_record(record) -> Dict:
return {
"modelType": record.model_type,
"modelId": record.model_id,
"largestVersionId": record.largest_version_id,
"versionIds": record.version_ids,
"inLibraryVersionIds": record.in_library_version_ids,
"lastCheckedAt": record.last_checked_at,
"shouldIgnore": record.should_ignore,
"hasUpdate": record.has_update(),
}
@dataclass
class ModelHandlerSet:
"""Aggregate concrete handlers into a flat mapping."""
@@ -974,6 +1186,7 @@ class ModelHandlerSet:
civitai: ModelCivitaiHandler
move: ModelMoveHandler
auto_organize: ModelAutoOrganizeHandler
updates: ModelUpdateHandler
def to_route_mapping(self) -> Dict[str, Callable[[web.Request], Awaitable[web.Response]]]:
return {
@@ -1002,6 +1215,8 @@ class ModelHandlerSet:
"download_model": self.download.download_model,
"download_model_get": self.download.download_model_get,
"cancel_download_get": self.download.cancel_download_get,
"pause_download_get": self.download.pause_download_get,
"resume_download_get": self.download.resume_download_get,
"get_download_progress": self.download.get_download_progress,
"get_civitai_versions": self.civitai.get_civitai_versions,
"get_civitai_model_by_version": self.civitai.get_civitai_model_by_version,
@@ -1016,5 +1231,8 @@ class ModelHandlerSet:
"get_model_metadata": self.query.get_model_metadata,
"get_model_description": self.query.get_model_description,
"get_relative_paths": self.query.get_relative_paths,
"refresh_model_updates": self.updates.refresh_model_updates,
"set_model_update_ignore": self.updates.set_model_update_ignore,
"get_model_update_status": self.updates.get_model_update_status,
}

View File

@@ -0,0 +1,56 @@
"""Handlers responsible for serving preview assets dynamically."""
from __future__ import annotations
import logging
import urllib.parse
from pathlib import Path
from aiohttp import web
from ...config import config as global_config
logger = logging.getLogger(__name__)
class PreviewHandler:
"""Serve preview assets for the active library at request time."""
def __init__(self, *, config=global_config) -> None:
self._config = config
async def serve_preview(self, request: web.Request) -> web.StreamResponse:
"""Return the preview file referenced by the encoded ``path`` query."""
raw_path = request.query.get("path", "")
if not raw_path:
raise web.HTTPBadRequest(text="Missing 'path' query parameter")
try:
decoded_path = urllib.parse.unquote(raw_path)
except Exception as exc: # pragma: no cover - defensive guard
logger.debug("Failed to decode preview path %s: %s", raw_path, exc)
raise web.HTTPBadRequest(text="Invalid preview path encoding") from exc
normalized = decoded_path.replace("\\", "/")
candidate = Path(normalized)
try:
resolved = candidate.expanduser().resolve(strict=False)
except Exception as exc:
logger.debug("Failed to resolve preview path %s: %s", normalized, exc)
raise web.HTTPBadRequest(text="Unable to resolve preview path") from exc
resolved_str = str(resolved)
if not self._config.is_preview_path_allowed(resolved_str):
logger.debug("Rejected preview outside allowed roots: %s", resolved_str)
raise web.HTTPForbidden(text="Preview path is not within an allowed directory")
if not resolved.is_file():
logger.debug("Preview file not found at %s", resolved_str)
raise web.HTTPNotFound(text="Preview file not found")
# aiohttp's FileResponse handles range requests and content headers for us.
return web.FileResponse(path=resolved, chunk_size=256 * 1024)
__all__ = ["PreviewHandler"]

View File

@@ -206,18 +206,16 @@ class RecipeListingHandler:
def format_recipe_file_url(self, file_path: str) -> str:
try:
recipes_dir = os.path.join(config.loras_roots[0], "recipes").replace(os.sep, "/")
normalized_path = file_path.replace(os.sep, "/")
if normalized_path.startswith(recipes_dir):
relative_path = os.path.relpath(file_path, config.loras_roots[0]).replace(os.sep, "/")
return f"/loras_static/root1/preview/{relative_path}"
file_name = os.path.basename(file_path)
return f"/loras_static/root1/preview/recipes/{file_name}"
normalized_path = os.path.normpath(file_path)
static_url = config.get_preview_static_url(normalized_path)
if static_url:
return static_url
except Exception as exc: # pragma: no cover - logging path
self._logger.error("Error formatting recipe file URL: %s", exc, exc_info=True)
return "/loras_static/images/no-preview.png"
return "/loras_static/images/no-preview.png"
class RecipeQueryHandler:
"""Provide read-only insights on recipe data."""

View File

@@ -23,8 +23,10 @@ class LoraRoutes(BaseModelRoutes):
async def initialize_services(self):
"""Initialize services from ServiceRegistry"""
lora_scanner = await ServiceRegistry.get_lora_scanner()
self.service = LoraService(lora_scanner)
update_service = await ServiceRegistry.get_model_update_service()
self.service = LoraService(lora_scanner, update_service=update_service)
self.set_model_update_service(update_service)
# Attach service dependencies
self.attach_service(self.service)
@@ -229,11 +231,27 @@ class LoraRoutes(BaseModelRoutes):
trigger_words_text = ",, ".join(all_trigger_words) if all_trigger_words else ""
# Send update to all connected trigger word toggle nodes
for node_id in node_ids:
PromptServer.instance.send_sync("trigger_word_update", {
"id": node_id,
for entry in node_ids:
node_identifier = entry
graph_identifier = None
if isinstance(entry, dict):
node_identifier = entry.get("node_id")
graph_identifier = entry.get("graph_id")
try:
parsed_node_id = int(node_identifier)
except (TypeError, ValueError):
parsed_node_id = node_identifier
payload = {
"id": parsed_node_id,
"message": trigger_words_text
})
}
if graph_identifier is not None:
payload["graph_id"] = str(graph_identifier)
PromptServer.instance.send_sync("trigger_word_update", payload)
return web.json_response({"success": True})

View File

@@ -0,0 +1,71 @@
"""Route registrar for miscellaneous endpoints.
This module mirrors the model route registrar architecture so that
miscellaneous endpoints share a consistent registration flow.
"""
from dataclasses import dataclass
from typing import Callable, Iterable, Mapping
from aiohttp import web
@dataclass(frozen=True)
class RouteDefinition:
"""Declarative definition for a HTTP route."""
method: str
path: str
handler_name: str
MISC_ROUTE_DEFINITIONS: tuple[RouteDefinition, ...] = (
RouteDefinition("GET", "/api/lm/settings", "get_settings"),
RouteDefinition("POST", "/api/lm/settings", "update_settings"),
RouteDefinition("GET", "/api/lm/priority-tags", "get_priority_tags"),
RouteDefinition("GET", "/api/lm/settings/libraries", "get_settings_libraries"),
RouteDefinition("POST", "/api/lm/settings/libraries/activate", "activate_library"),
RouteDefinition("GET", "/api/lm/health-check", "health_check"),
RouteDefinition("POST", "/api/lm/open-file-location", "open_file_location"),
RouteDefinition("POST", "/api/lm/update-usage-stats", "update_usage_stats"),
RouteDefinition("GET", "/api/lm/get-usage-stats", "get_usage_stats"),
RouteDefinition("POST", "/api/lm/update-lora-code", "update_lora_code"),
RouteDefinition("GET", "/api/lm/trained-words", "get_trained_words"),
RouteDefinition("GET", "/api/lm/model-example-files", "get_model_example_files"),
RouteDefinition("POST", "/api/lm/register-nodes", "register_nodes"),
RouteDefinition("GET", "/api/lm/get-registry", "get_registry"),
RouteDefinition("GET", "/api/lm/check-model-exists", "check_model_exists"),
RouteDefinition("GET", "/api/lm/civitai/user-models", "get_civitai_user_models"),
RouteDefinition("POST", "/api/lm/download-metadata-archive", "download_metadata_archive"),
RouteDefinition("POST", "/api/lm/remove-metadata-archive", "remove_metadata_archive"),
RouteDefinition("GET", "/api/lm/metadata-archive-status", "get_metadata_archive_status"),
RouteDefinition("GET", "/api/lm/model-versions-status", "get_model_versions_status"),
)
class MiscRouteRegistrar:
"""Bind miscellaneous route definitions to an aiohttp router."""
_METHOD_MAP = {
"GET": "add_get",
"POST": "add_post",
"PUT": "add_put",
"DELETE": "add_delete",
}
def __init__(self, app: web.Application) -> None:
self._app = app
def register_routes(
self,
handler_lookup: Mapping[str, Callable[[web.Request], object]],
*,
definitions: Iterable[RouteDefinition] = MISC_ROUTE_DEFINITIONS,
) -> None:
for definition in definitions:
self._bind(definition.method, definition.path, handler_lookup[definition.handler_name])
def _bind(self, method: str, path: str, handler: Callable) -> None:
add_method_name = self._METHOD_MAP[method.upper()]
add_method = getattr(self._app.router, add_method_name)
add_method(path, handler)

File diff suppressed because it is too large Load Diff

View File

@@ -55,9 +55,14 @@ COMMON_ROUTE_DEFINITIONS: tuple[RouteDefinition, ...] = (
RouteDefinition("GET", "/api/lm/{prefix}/civitai/versions/{model_id}", "get_civitai_versions"),
RouteDefinition("GET", "/api/lm/{prefix}/civitai/model/version/{modelVersionId}", "get_civitai_model_by_version"),
RouteDefinition("GET", "/api/lm/{prefix}/civitai/model/hash/{hash}", "get_civitai_model_by_hash"),
RouteDefinition("POST", "/api/lm/{prefix}/updates/refresh", "refresh_model_updates"),
RouteDefinition("POST", "/api/lm/{prefix}/updates/ignore", "set_model_update_ignore"),
RouteDefinition("GET", "/api/lm/{prefix}/updates/status/{model_id}", "get_model_update_status"),
RouteDefinition("POST", "/api/lm/download-model", "download_model"),
RouteDefinition("GET", "/api/lm/download-model-get", "download_model_get"),
RouteDefinition("GET", "/api/lm/cancel-download-get", "cancel_download_get"),
RouteDefinition("GET", "/api/lm/pause-download", "pause_download_get"),
RouteDefinition("GET", "/api/lm/resume-download", "resume_download_get"),
RouteDefinition("GET", "/api/lm/download-progress/{download_id}", "get_download_progress"),
RouteDefinition("GET", "/{prefix}", "handle_models_page"),
)

View File

@@ -0,0 +1,25 @@
"""Route controller for preview asset delivery."""
from __future__ import annotations
from aiohttp import web
from .handlers.preview_handlers import PreviewHandler
class PreviewRoutes:
"""Register routes that expose preview assets."""
def __init__(self, *, handler: PreviewHandler | None = None) -> None:
self._handler = handler or PreviewHandler()
@classmethod
def setup_routes(cls, app: web.Application) -> None:
controller = cls()
controller.register(app)
def register(self, app: web.Application) -> None:
app.router.add_get('/api/lm/previews', self._handler.serve_preview)
__all__ = ["PreviewRoutes"]

View File

@@ -8,13 +8,32 @@ from collections import defaultdict, Counter
from typing import Dict, List, Any
from ..config import config
from ..services.settings_manager import settings
from ..services.settings_manager import get_settings_manager
from ..services.server_i18n import server_i18n
from ..services.service_registry import ServiceRegistry
from ..utils.usage_stats import UsageStats
logger = logging.getLogger(__name__)
class _SettingsProxy:
def __init__(self):
self._manager = None
def _resolve(self):
if self._manager is None:
self._manager = get_settings_manager()
return self._manager
def get(self, *args, **kwargs):
return self._resolve().get(*args, **kwargs)
def __getattr__(self, item):
return getattr(self._resolve(), item)
settings = _SettingsProxy()
class StatsRoutes:
"""Route handlers for Statistics page and API endpoints"""
@@ -66,7 +85,9 @@ class StatsRoutes:
is_initializing = lora_initializing or checkpoint_initializing or embedding_initializing
# 获取用户语言设置
user_language = settings.get('language', 'en')
settings_object = settings
user_language = settings_object.get('language', 'en')
settings_manager = settings_object if not isinstance(settings_object, _SettingsProxy) else settings_object._resolve()
# 设置服务端i18n语言
server_i18n.set_locale(user_language)
@@ -79,7 +100,7 @@ class StatsRoutes:
template = self.template_env.get_template('statistics.html')
rendered = template.render(
is_initializing=is_initializing,
settings=settings,
settings=settings_manager,
request=request,
t=server_i18n.get_translation,
)

View File

@@ -5,12 +5,18 @@ import git
import zipfile
import shutil
import tempfile
from aiohttp import web
import asyncio
from aiohttp import web, ClientError
from typing import Dict, List
from ..utils.settings_paths import ensure_settings_file
from ..services.downloader import get_downloader
logger = logging.getLogger(__name__)
NETWORK_EXCEPTIONS = (ClientError, OSError, asyncio.TimeoutError)
class UpdateRoutes:
"""Routes for handling plugin update checks"""
@@ -63,6 +69,12 @@ class UpdateRoutes:
'nightly': nightly
})
except NETWORK_EXCEPTIONS as e:
logger.warning("Network unavailable during update check: %s", e)
return web.json_response({
'success': False,
'error': 'Network unavailable for update check'
})
except Exception as e:
logger.error(f"Failed to check for updates: {e}", exc_info=True)
return web.json_response({
@@ -111,7 +123,7 @@ class UpdateRoutes:
current_dir = os.path.dirname(os.path.abspath(__file__))
plugin_root = os.path.dirname(os.path.dirname(current_dir))
settings_path = os.path.join(plugin_root, 'settings.json')
settings_path = ensure_settings_file(logger)
settings_backup = None
if os.path.exists(settings_path):
with open(settings_path, 'r', encoding='utf-8') as f:
@@ -283,6 +295,9 @@ class UpdateRoutes:
return version, changelog
except NETWORK_EXCEPTIONS as e:
logger.warning("Unable to reach GitHub for nightly version: %s", e)
return "main", []
except Exception as e:
logger.error(f"Error fetching nightly version: {e}", exc_info=True)
return "main", []
@@ -448,6 +463,9 @@ class UpdateRoutes:
return version, changelog
except NETWORK_EXCEPTIONS as e:
logger.warning("Unable to reach GitHub for release info: %s", e)
return "v0.0.0", []
except Exception as e:
logger.error(f"Error fetching remote version: {e}", exc_info=True)
return "v0.0.0", []

View File

@@ -1,14 +1,19 @@
from abc import ABC, abstractmethod
from typing import Dict, List, Optional, Type
import asyncio
from typing import Dict, List, Optional, Type, TYPE_CHECKING
import logging
import os
from ..utils.models import BaseModelMetadata
from ..utils.metadata_manager import MetadataManager
from .model_query import FilterCriteria, ModelCacheRepository, ModelFilterSet, SearchStrategy, SettingsProvider
from .settings_manager import settings as default_settings
from .settings_manager import get_settings_manager
logger = logging.getLogger(__name__)
if TYPE_CHECKING:
from .model_update_service import ModelUpdateService
class BaseModelService(ABC):
"""Base service class for all model types"""
@@ -22,6 +27,7 @@ class BaseModelService(ABC):
filter_set: Optional[ModelFilterSet] = None,
search_strategy: Optional[SearchStrategy] = None,
settings_provider: Optional[SettingsProvider] = None,
update_service: Optional["ModelUpdateService"] = None,
):
"""Initialize the service.
@@ -33,14 +39,16 @@ class BaseModelService(ABC):
filter_set: Filter component controlling folder/tag/favorites logic.
search_strategy: Search component for fuzzy/text matching.
settings_provider: Settings object; defaults to the global settings manager.
update_service: Service used to determine whether models have remote updates available.
"""
self.model_type = model_type
self.scanner = scanner
self.metadata_class = metadata_class
self.settings = settings_provider or default_settings
self.settings = settings_provider or get_settings_manager()
self.cache_repository = cache_repository or ModelCacheRepository(scanner)
self.filter_set = filter_set or ModelFilterSet(self.settings)
self.search_strategy = search_strategy or SearchStrategy()
self.update_service = update_service
async def get_paginated_data(
self,
@@ -55,6 +63,7 @@ class BaseModelService(ABC):
search_options: dict = None,
hash_filters: dict = None,
favorites_only: bool = False,
has_update: bool = False,
**kwargs,
) -> Dict:
"""Get paginated and filtered model data"""
@@ -84,6 +93,9 @@ class BaseModelService(ABC):
filtered_data = await self._apply_specific_filters(filtered_data, **kwargs)
if has_update:
filtered_data = await self._apply_update_filter(filtered_data)
return self._paginate(filtered_data, page, page_size)
@@ -143,6 +155,59 @@ class BaseModelService(ABC):
async def _apply_specific_filters(self, data: List[Dict], **kwargs) -> List[Dict]:
"""Apply model-specific filters - to be overridden by subclasses if needed"""
return data
async def _apply_update_filter(self, data: List[Dict]) -> List[Dict]:
"""Filter models to those with remote updates available when requested."""
if not data:
return []
if self.update_service is None:
logger.warning(
"Requested has_update filter for %s models but update service is unavailable",
self.model_type,
)
return []
candidates: List[tuple[Dict, int]] = []
for item in data:
model_id = self._extract_model_id(item)
if model_id is not None:
candidates.append((item, model_id))
if not candidates:
return []
tasks = [
self.update_service.has_update(self.model_type, model_id)
for _, model_id in candidates
]
results = await asyncio.gather(*tasks, return_exceptions=True)
filtered: List[Dict] = []
for (item, model_id), result in zip(candidates, results):
if isinstance(result, Exception):
logger.error(
"Failed to resolve update status for model %s (%s): %s",
model_id,
self.model_type,
result,
)
continue
if result:
filtered.append(item)
return filtered
@staticmethod
def _extract_model_id(item: Dict) -> Optional[int]:
civitai = item.get('civitai') if isinstance(item, dict) else None
if not isinstance(civitai, dict):
return None
try:
value = civitai.get('modelId')
if value is None:
return None
return int(value)
except (TypeError, ValueError):
return None
def _paginate(self, data: List[Dict], page: int, page_size: int) -> Dict:
"""Apply pagination to filtered data"""
@@ -313,24 +378,24 @@ class BaseModelService(ABC):
return {'civitai_url': None, 'model_id': None, 'version_id': None}
async def get_model_metadata(self, file_path: str) -> Optional[Dict]:
"""Get filtered CivitAI metadata for a model by file path"""
cache = await self.scanner.get_cached_data()
for model in cache.raw_data:
if model.get('file_path') == file_path:
return self.filter_civitai_data(model.get("civitai", {}))
return None
"""Load full metadata for a single model.
Listing/search endpoints return lightweight cache entries; this method performs
a lazy read of the on-disk metadata snapshot when callers need full detail.
"""
metadata, should_skip = await MetadataManager.load_metadata(file_path, self.metadata_class)
if should_skip or metadata is None:
return None
return self.filter_civitai_data(metadata.to_dict().get("civitai", {}))
async def get_model_description(self, file_path: str) -> Optional[str]:
"""Get model description by file path"""
cache = await self.scanner.get_cached_data()
for model in cache.raw_data:
if model.get('file_path') == file_path:
return model.get('modelDescription', '')
return None
"""Return the stored modelDescription field for a model."""
metadata, should_skip = await MetadataManager.load_metadata(file_path, self.metadata_class)
if should_skip or metadata is None:
return None
return metadata.modelDescription or ''
async def search_relative_paths(self, search_term: str, limit: int = 15) -> List[str]:
"""Search model relative file paths for autocomplete functionality"""
@@ -372,4 +437,4 @@ class BaseModelService(ABC):
x.lower() # Then alphabetically
))
return matching_paths[:limit]
return matching_paths[:limit]

View File

@@ -11,13 +11,14 @@ logger = logging.getLogger(__name__)
class CheckpointService(BaseModelService):
"""Checkpoint-specific service implementation"""
def __init__(self, scanner):
def __init__(self, scanner, update_service=None):
"""Initialize Checkpoint service
Args:
scanner: Checkpoint scanner instance
update_service: Optional service for remote update tracking.
"""
super().__init__("checkpoint", scanner, CheckpointMetadata)
super().__init__("checkpoint", scanner, CheckpointMetadata, update_service=update_service)
async def format_response(self, checkpoint_data: Dict) -> Dict:
"""Format Checkpoint data for API response"""
@@ -46,4 +47,4 @@ class CheckpointService(BaseModelService):
def find_duplicate_filenames(self) -> Dict:
"""Find Checkpoints with conflicting filenames"""
return self.scanner._hash_index.get_duplicate_filenames()
return self.scanner._hash_index.get_duplicate_filenames()

View File

@@ -0,0 +1,554 @@
import os
import json
import logging
import asyncio
from copy import deepcopy
from typing import Optional, Dict, Tuple, List
from .model_metadata_provider import CivArchiveModelMetadataProvider, ModelMetadataProviderManager
from .downloader import get_downloader
from .errors import RateLimitError
try:
from bs4 import BeautifulSoup
except ImportError as exc:
BeautifulSoup = None # type: ignore[assignment]
_BS4_IMPORT_ERROR = exc
else:
_BS4_IMPORT_ERROR = None
def _require_beautifulsoup():
if BeautifulSoup is None:
raise RuntimeError(
"BeautifulSoup (bs4) is required for CivArchive client. "
"Install it with 'pip install beautifulsoup4'."
) from _BS4_IMPORT_ERROR
return BeautifulSoup
logger = logging.getLogger(__name__)
class CivArchiveClient:
_instance = None
_lock = asyncio.Lock()
@classmethod
async def get_instance(cls):
"""Get singleton instance of CivArchiveClient"""
async with cls._lock:
if cls._instance is None:
cls._instance = cls()
# Register this client as a metadata provider
provider_manager = await ModelMetadataProviderManager.get_instance()
provider_manager.register_provider('civarchive', CivArchiveModelMetadataProvider(cls._instance), False)
return cls._instance
def __init__(self):
# Check if already initialized for singleton pattern
if hasattr(self, '_initialized'):
return
self._initialized = True
self.base_url = "https://civarchive.com/api"
async def _request_json(
self,
path: str,
params: Optional[Dict[str, str]] = None
) -> Tuple[Optional[Dict], Optional[str]]:
"""Call CivArchive API and return JSON payload"""
success, payload = await self._make_request(path, params=params)
if not success:
error = payload if isinstance(payload, str) else "Request failed"
return None, error
if not isinstance(payload, dict):
return None, "Invalid response structure"
return payload, None
async def _make_request(
self,
path: str,
*,
params: Optional[Dict[str, str]] = None,
) -> Tuple[bool, Dict | str]:
"""Wrapper around downloader.make_request that surfaces rate limits."""
downloader = await get_downloader()
kwargs: Dict[str, Dict[str, str]] = {}
if params:
safe_params = {str(key): str(value) for key, value in params.items() if value is not None}
if safe_params:
kwargs["params"] = safe_params
success, payload = await downloader.make_request(
"GET",
f"{self.base_url}{path}",
use_auth=False,
**kwargs,
)
if not success and isinstance(payload, RateLimitError):
if payload.provider is None:
payload.provider = "civarchive_api"
raise payload
return success, payload
@staticmethod
def _normalize_payload(payload: Dict) -> Dict:
"""Unwrap CivArchive responses that wrap content under a data key"""
if not isinstance(payload, dict):
return {}
data = payload.get("data")
if isinstance(data, dict):
return data
return payload
@staticmethod
def _split_context(payload: Dict) -> Tuple[Dict, Dict, List[Dict]]:
"""Separate version payload from surrounding model context"""
data = CivArchiveClient._normalize_payload(payload)
context: Dict = {}
fallback_files: List[Dict] = []
version: Dict = {}
for key, value in data.items():
if key in {"version", "model"}:
continue
context[key] = value
if isinstance(data.get("version"), dict):
version = data["version"]
model_block = data.get("model")
if isinstance(model_block, dict):
for key, value in model_block.items():
if key == "version":
if not version and isinstance(value, dict):
version = value
continue
context.setdefault(key, value)
fallback_files = fallback_files or model_block.get("files") or []
fallback_files = fallback_files or data.get("files") or []
return context, version, fallback_files
@staticmethod
def _ensure_list(value) -> List:
if isinstance(value, list):
return value
if value is None:
return []
return [value]
@staticmethod
def _build_model_info(context: Dict) -> Dict:
tags = context.get("tags")
if not isinstance(tags, list):
tags = list(tags) if isinstance(tags, (set, tuple)) else ([] if tags is None else [tags])
return {
"name": context.get("name"),
"type": context.get("type"),
"nsfw": bool(context.get("is_nsfw", context.get("nsfw", False))),
"description": context.get("description"),
"tags": tags,
}
@staticmethod
def _build_creator_info(context: Dict) -> Dict:
username = context.get("creator_username") or context.get("username") or ""
image = context.get("creator_image") or context.get("creator_avatar") or ""
creator: Dict[str, Optional[str]] = {
"username": username,
"image": image,
}
if context.get("creator_name"):
creator["name"] = context["creator_name"]
if context.get("creator_url"):
creator["url"] = context["creator_url"]
return creator
@staticmethod
def _transform_file_entry(file_data: Dict) -> Dict:
mirrors = file_data.get("mirrors") or []
if not isinstance(mirrors, list):
mirrors = [mirrors]
available_mirror = next(
(mirror for mirror in mirrors if isinstance(mirror, dict) and mirror.get("deletedAt") is None),
None
)
download_url = file_data.get("downloadUrl")
if not download_url and available_mirror:
download_url = available_mirror.get("url")
name = file_data.get("name")
if not name and available_mirror:
name = available_mirror.get("filename")
transformed: Dict = {
"id": file_data.get("id"),
"sizeKB": file_data.get("sizeKB"),
"name": name,
"type": file_data.get("type"),
"downloadUrl": download_url,
"primary": True,
# TODO: for some reason is_primary is false in CivArchive response, need to figure this out,
# "primary": bool(file_data.get("is_primary", file_data.get("primary", False))),
"mirrors": mirrors,
}
sha256 = file_data.get("sha256")
if sha256:
transformed["hashes"] = {"SHA256": str(sha256).upper()}
elif isinstance(file_data.get("hashes"), dict):
transformed["hashes"] = file_data["hashes"]
if "metadata" in file_data:
transformed["metadata"] = file_data["metadata"]
if file_data.get("modelVersionId") is not None:
transformed["modelVersionId"] = file_data.get("modelVersionId")
elif file_data.get("model_version_id") is not None:
transformed["modelVersionId"] = file_data.get("model_version_id")
if file_data.get("modelId") is not None:
transformed["modelId"] = file_data.get("modelId")
elif file_data.get("model_id") is not None:
transformed["modelId"] = file_data.get("model_id")
return transformed
def _transform_files(
self,
files: Optional[List[Dict]],
fallback_files: Optional[List[Dict]] = None
) -> List[Dict]:
candidates: List[Dict] = []
if isinstance(files, list) and files:
candidates = files
elif isinstance(fallback_files, list):
candidates = fallback_files
transformed_files: List[Dict] = []
for file_data in candidates:
if isinstance(file_data, dict):
transformed_files.append(self._transform_file_entry(file_data))
return transformed_files
def _transform_version(
self,
context: Dict,
version: Dict,
fallback_files: Optional[List[Dict]] = None
) -> Optional[Dict]:
if not version:
return None
version_copy = deepcopy(version)
version_copy.pop("model", None)
version_copy.pop("creator", None)
if "trigger" in version_copy:
triggers = version_copy.pop("trigger")
if isinstance(triggers, list):
version_copy["trainedWords"] = triggers
elif triggers is None:
version_copy["trainedWords"] = []
else:
version_copy["trainedWords"] = [triggers]
if "trainedWords" in version_copy and isinstance(version_copy["trainedWords"], str):
version_copy["trainedWords"] = [version_copy["trainedWords"]]
if "nsfw_level" in version_copy:
version_copy["nsfwLevel"] = version_copy.pop("nsfw_level")
elif "nsfwLevel" not in version_copy and context.get("nsfw_level") is not None:
version_copy["nsfwLevel"] = context.get("nsfw_level")
stats_keys = ["downloadCount", "ratingCount", "rating"]
stats = {key: version_copy.pop(key) for key in stats_keys if key in version_copy}
if stats:
version_copy["stats"] = stats
version_copy["files"] = self._transform_files(version_copy.get("files"), fallback_files)
version_copy["images"] = self._ensure_list(version_copy.get("images"))
version_copy["model"] = self._build_model_info(context)
version_copy["creator"] = self._build_creator_info(context)
version_copy["source"] = "civarchive"
version_copy["is_deleted"] = bool(context.get("deletedAt")) or bool(version.get("deletedAt"))
return version_copy
async def _resolve_version_from_files(self, payload: Dict) -> Optional[Dict]:
"""Fallback to fetch version data when only file metadata is available"""
data = self._normalize_payload(payload)
files = data.get("files") or payload.get("files") or []
if not isinstance(files, list):
files = [files]
for file_data in files:
if not isinstance(file_data, dict):
continue
model_id = file_data.get("model_id") or file_data.get("modelId")
version_id = file_data.get("model_version_id") or file_data.get("modelVersionId")
if model_id is None or version_id is None:
continue
resolved = await self.get_model_version(model_id, version_id)
if resolved:
return resolved
return None
async def get_model_by_hash(self, model_hash: str) -> Tuple[Optional[Dict], Optional[str]]:
"""Find model by SHA256 hash value using CivArchive API"""
try:
payload, error = await self._request_json(f"/sha256/{model_hash.lower()}")
if error:
if "not found" in error.lower():
return None, "Model not found"
return None, error
context, version_data, fallback_files = self._split_context(payload)
transformed = self._transform_version(context, version_data, fallback_files)
if transformed:
return transformed, None
resolved = await self._resolve_version_from_files(payload)
if resolved:
return resolved, None
logger.error("Error fetching version of CivArchive model by hash %s", model_hash[:10])
return None, "No version data found"
except RateLimitError:
raise
except Exception as e:
logger.error(f"Error fetching CivArchive model by hash {model_hash[:10]}: {e}")
return None, str(e)
async def get_model_versions(self, model_id: str) -> Optional[Dict]:
"""Get all versions of a model using CivArchive API"""
try:
payload, error = await self._request_json(f"/models/{model_id}")
if error or payload is None:
if error and "not found" in error.lower():
return None
logger.error(f"Error fetching CivArchive model versions for {model_id}: {error}")
return None
data = self._normalize_payload(payload)
context, version_data, fallback_files = self._split_context(payload)
versions_meta = data.get("versions") or []
transformed_versions: List[Dict] = []
for meta in versions_meta:
if not isinstance(meta, dict):
continue
version_id = meta.get("id")
if version_id is None:
continue
target_model_id = meta.get("modelId") or model_id
version = await self.get_model_version(target_model_id, version_id)
if version:
transformed_versions.append(version)
# Ensure the primary version is included even if versions list was empty
primary_version = self._transform_version(context, version_data, fallback_files)
if primary_version:
transformed_versions.insert(0, primary_version)
ordered_versions: List[Dict] = []
seen_ids = set()
for version in transformed_versions:
version_id = version.get("id")
if version_id in seen_ids:
continue
seen_ids.add(version_id)
ordered_versions.append(version)
return {
"modelVersions": ordered_versions,
"type": context.get("type", ""),
"name": context.get("name", ""),
}
except RateLimitError:
raise
except Exception as e:
logger.error(f"Error fetching CivArchive model versions for {model_id}: {e}")
return None
async def get_model_version(self, model_id: int = None, version_id: int = None) -> Optional[Dict]:
"""Get specific model version using CivArchive API
Args:
model_id: The model ID (required)
version_id: Optional specific version ID to filter to
Returns:
Optional[Dict]: The model version data or None if not found
"""
if model_id is None:
return None
try:
params = {"modelVersionId": version_id} if version_id is not None else None
payload, error = await self._request_json(f"/models/{model_id}", params=params)
if error or payload is None:
if error and "not found" in error.lower():
return None
logger.error(f"Error fetching CivArchive model version via API {model_id}/{version_id}: {error}")
return None
context, version_data, fallback_files = self._split_context(payload)
if not version_data:
return await self._resolve_version_from_files(payload)
if version_id is not None:
raw_id = version_data.get("id")
if raw_id != version_id:
logger.warning(
"Requested version %s doesn't match default version %s for model %s",
version_id,
raw_id,
model_id,
)
return None
actual_model_id = version_data.get("modelId")
context_model_id = context.get("id")
# CivArchive can respond with data for a different model id while already
# returning the fully resolved model context. Only follow the redirect when
# the context itself still points to the original (wrong) model.
if (
actual_model_id is not None
and str(actual_model_id) != str(model_id)
and (context_model_id is None or str(context_model_id) != str(actual_model_id))
):
return await self.get_model_version(actual_model_id, version_id)
return self._transform_version(context, version_data, fallback_files)
except RateLimitError:
raise
except Exception as e:
logger.error(f"Error fetching CivArchive model version via API {model_id}/{version_id}: {e}")
return None
async def get_model_version_info(self, version_id: str) -> Tuple[Optional[Dict], Optional[str]]:
""" Fetch model version metadata using a known bogus model lookup
CivArchive lacks a direct version lookup API, this uses a workaround (which we handle in the main model request now)
Args:
version_id: The model version ID
Returns:
Tuple[Optional[Dict], Optional[str]]: (version_data, error_message)
"""
version = await self.get_model_version(1, version_id)
if version is None:
return None, "Model not found"
return version, None
async def get_model_by_url(self, url) -> Optional[Dict]:
"""Get specific model version by parsing CivArchive HTML page (legacy method)
This is the original HTML scraping implementation, kept for reference and new sites added not in api.
The primary get_model_version() now uses the API instead.
"""
try:
# Construct CivArchive URL
url = f"https://civarchive.com/{url}"
downloader = await get_downloader()
session = await downloader.session
async with session.get(url) as response:
if response.status != 200:
return None
html_content = await response.text()
# Parse HTML to extract JSON data
soup_parser = _require_beautifulsoup()
soup = soup_parser(html_content, 'html.parser')
script_tag = soup.find('script', {'id': '__NEXT_DATA__', 'type': 'application/json'})
if not script_tag:
return None
# Parse JSON content
json_data = json.loads(script_tag.string)
model_data = json_data.get('props', {}).get('pageProps', {}).get('model')
if not model_data or 'version' not in model_data:
return None
# Extract version data as base
version = model_data['version'].copy()
# Restructure stats
if 'downloadCount' in version and 'ratingCount' in version and 'rating' in version:
version['stats'] = {
'downloadCount': version.pop('downloadCount'),
'ratingCount': version.pop('ratingCount'),
'rating': version.pop('rating')
}
# Rename trigger to trainedWords
if 'trigger' in version:
version['trainedWords'] = version.pop('trigger')
# Transform files data to expected format
if 'files' in version:
transformed_files = []
for file_data in version['files']:
# Find first available mirror (deletedAt is null)
available_mirror = None
for mirror in file_data.get('mirrors', []):
if mirror.get('deletedAt') is None:
available_mirror = mirror
break
# Create transformed file entry
transformed_file = {
'id': file_data.get('id'),
'sizeKB': file_data.get('sizeKB'),
'name': available_mirror.get('filename', file_data.get('name')) if available_mirror else file_data.get('name'),
'type': file_data.get('type'),
'downloadUrl': available_mirror.get('url') if available_mirror else None,
'primary': file_data.get('is_primary', False),
'mirrors': file_data.get('mirrors', [])
}
# Transform hash format
if 'sha256' in file_data:
transformed_file['hashes'] = {
'SHA256': file_data['sha256'].upper()
}
transformed_files.append(transformed_file)
version['files'] = transformed_files
# Add model information
version['model'] = {
'name': model_data.get('name'),
'type': model_data.get('type'),
'nsfw': model_data.get('is_nsfw', False),
'description': model_data.get('description'),
'tags': model_data.get('tags', [])
}
version['creator'] = {
'username': model_data.get('username'),
'image': ''
}
# Add source identifier
version['source'] = 'civarchive'
version['is_deleted'] = json_data.get('query', {}).get('is_deleted', False)
return version
except RateLimitError:
raise
except Exception as e:
logger.error(f"Error fetching CivArchive model version (scraping) {url}: {e}")
return None

View File

@@ -1,10 +1,11 @@
import os
import asyncio
import copy
import logging
import asyncio
import os
from typing import Optional, Dict, Tuple, List
from .model_metadata_provider import CivitaiModelMetadataProvider, ModelMetadataProviderManager
from .downloader import get_downloader
from .errors import RateLimitError
logger = logging.getLogger(__name__)
@@ -32,6 +33,47 @@ class CivitaiClient:
self._initialized = True
self.base_url = "https://civitai.com/api/v1"
async def _make_request(
self,
method: str,
url: str,
*,
use_auth: bool = False,
**kwargs,
) -> Tuple[bool, Dict | str]:
"""Wrapper around downloader.make_request that surfaces rate limits."""
downloader = await get_downloader()
success, result = await downloader.make_request(
method,
url,
use_auth=use_auth,
**kwargs,
)
if not success and isinstance(result, RateLimitError):
if result.provider is None:
result.provider = "civitai_api"
raise result
return success, result
@staticmethod
def _remove_comfy_metadata(model_version: Optional[Dict]) -> None:
"""Remove Comfy-specific metadata from model version images."""
if not isinstance(model_version, dict):
return
images = model_version.get("images")
if not isinstance(images, list):
return
for image in images:
if not isinstance(image, dict):
continue
meta = image.get("meta")
if isinstance(meta, dict) and "comfy" in meta:
meta.pop("comfy", None)
async def download_file(self, url: str, save_dir: str, default_filename: str, progress_callback=None) -> Tuple[bool, str]:
"""Download file with resumable downloads and retry mechanism
@@ -61,8 +103,7 @@ class CivitaiClient:
async def get_model_by_hash(self, model_hash: str) -> Tuple[Optional[Dict], Optional[str]]:
try:
downloader = await get_downloader()
success, result = await downloader.make_request(
success, result = await self._make_request(
'GET',
f"{self.base_url}/model-versions/by-hash/{model_hash}",
use_auth=True
@@ -72,7 +113,7 @@ class CivitaiClient:
model_id = result.get('modelId')
if model_id:
# Fetch additional model metadata
success_model, data = await downloader.make_request(
success_model, data = await self._make_request(
'GET',
f"{self.base_url}/models/{model_id}",
use_auth=True
@@ -81,10 +122,11 @@ class CivitaiClient:
# Enrich version_info with model data
result['model']['description'] = data.get("description")
result['model']['tags'] = data.get("tags", [])
# Add creator from model data
result['creator'] = data.get("creator")
self._remove_comfy_metadata(result)
return result, None
# Handle specific error cases
@@ -94,6 +136,8 @@ class CivitaiClient:
# Other error cases
logger.error(f"Failed to fetch model info for {model_hash[:10]}: {result}")
return None, str(result)
except RateLimitError:
raise
except Exception as e:
logger.error(f"API Error: {str(e)}")
return None, str(e)
@@ -119,8 +163,7 @@ class CivitaiClient:
async def get_model_versions(self, model_id: str) -> List[Dict]:
"""Get all versions of a model with local availability info"""
try:
downloader = await get_downloader()
success, result = await downloader.make_request(
success, result = await self._make_request(
'GET',
f"{self.base_url}/models/{model_id}",
use_auth=True
@@ -133,144 +176,167 @@ class CivitaiClient:
'name': result.get('name', '')
}
return None
except RateLimitError:
raise
except Exception as e:
logger.error(f"Error fetching model versions: {e}")
return None
async def get_model_version(self, model_id: int = None, version_id: int = None) -> Optional[Dict]:
"""Get specific model version with additional metadata
Args:
model_id: The Civitai model ID (optional if version_id is provided)
version_id: Optional specific version ID to retrieve
Returns:
Optional[Dict]: The model version data with additional fields or None if not found
"""
"""Get specific model version with additional metadata."""
try:
downloader = await get_downloader()
# Case 1: Only version_id is provided
if model_id is None and version_id is not None:
# First get the version info to extract model_id
success, version = await downloader.make_request(
'GET',
f"{self.base_url}/model-versions/{version_id}",
use_auth=True
)
if not success:
return None
model_id = version.get('modelId')
if not model_id:
logger.error(f"No modelId found in version {version_id}")
return None
# Now get the model data for additional metadata
success, model_data = await downloader.make_request(
'GET',
f"{self.base_url}/models/{model_id}",
use_auth=True
)
if success:
# Enrich version with model data
version['model']['description'] = model_data.get("description")
version['model']['tags'] = model_data.get("tags", [])
version['creator'] = model_data.get("creator")
return version
# Case 2: model_id is provided (with or without version_id)
elif model_id is not None:
# Step 1: Get model data to find version_id if not provided and get additional metadata
success, data = await downloader.make_request(
'GET',
f"{self.base_url}/models/{model_id}",
use_auth=True
)
if not success:
return None
return await self._get_version_by_id_only(version_id)
model_versions = data.get('modelVersions', [])
if not model_versions:
logger.warning(f"No model versions found for model {model_id}")
return None
if model_id is not None:
return await self._get_version_with_model_id(model_id, version_id)
# Step 2: Determine the target version entry to use
target_version = None
if version_id is not None:
target_version = next(
(item for item in model_versions if item.get('id') == version_id),
None
)
if target_version is None:
logger.warning(
f"Version {version_id} not found for model {model_id}, defaulting to first version"
)
if target_version is None:
target_version = model_versions[0]
logger.error("Either model_id or version_id must be provided")
return None
target_version_id = target_version.get('id')
# Step 3: Get detailed version info using the SHA256 hash
model_hash = None
for file_info in target_version.get('files', []):
if file_info.get('type') == 'Model' and file_info.get('primary'):
model_hash = file_info.get('hashes', {}).get('SHA256')
if model_hash:
break
version = None
if model_hash:
success, version = await downloader.make_request(
'GET',
f"{self.base_url}/model-versions/by-hash/{model_hash}",
use_auth=True
)
if not success:
logger.warning(
f"Failed to fetch version by hash for model {model_id} version {target_version_id}: {version}"
)
version = None
else:
logger.warning(
f"No primary model hash found for model {model_id} version {target_version_id}"
)
if version is None:
version = copy.deepcopy(target_version)
version.pop('index', None)
version['modelId'] = model_id
version['model'] = {
'name': data.get('name'),
'type': data.get('type'),
'nsfw': data.get('nsfw'),
'poi': data.get('poi')
}
# Step 4: Enrich version_info with model data
# Add description and tags from model data
model_info = version.get('model')
if not isinstance(model_info, dict):
model_info = {}
version['model'] = model_info
model_info['description'] = data.get("description")
model_info['tags'] = data.get("tags", [])
# Add creator from model data
version['creator'] = data.get("creator")
return version
# Case 3: Neither model_id nor version_id provided
else:
logger.error("Either model_id or version_id must be provided")
return None
except RateLimitError:
raise
except Exception as e:
logger.error(f"Error fetching model version: {e}")
return None
async def _get_version_by_id_only(self, version_id: int) -> Optional[Dict]:
version = await self._fetch_version_by_id(version_id)
if version is None:
return None
model_id = version.get('modelId')
if not model_id:
logger.error(f"No modelId found in version {version_id}")
return None
model_data = await self._fetch_model_data(model_id)
if model_data:
self._enrich_version_with_model_data(version, model_data)
self._remove_comfy_metadata(version)
return version
async def _get_version_with_model_id(self, model_id: int, version_id: Optional[int]) -> Optional[Dict]:
model_data = await self._fetch_model_data(model_id)
if not model_data:
return None
target_version = self._select_target_version(model_data, model_id, version_id)
if target_version is None:
return None
target_version_id = target_version.get('id')
version = await self._fetch_version_by_id(target_version_id) if target_version_id else None
if version is None:
model_hash = self._extract_primary_model_hash(target_version)
if model_hash:
version = await self._fetch_version_by_hash(model_hash)
else:
logger.warning(
f"No primary model hash found for model {model_id} version {target_version_id}"
)
if version is None:
version = self._build_version_from_model_data(target_version, model_id, model_data)
self._enrich_version_with_model_data(version, model_data)
self._remove_comfy_metadata(version)
return version
async def _fetch_model_data(self, model_id: int) -> Optional[Dict]:
success, data = await self._make_request(
'GET',
f"{self.base_url}/models/{model_id}",
use_auth=True
)
if success:
return data
logger.warning(f"Failed to fetch model data for model {model_id}")
return None
async def _fetch_version_by_id(self, version_id: Optional[int]) -> Optional[Dict]:
if version_id is None:
return None
success, version = await self._make_request(
'GET',
f"{self.base_url}/model-versions/{version_id}",
use_auth=True
)
if success:
return version
logger.warning(f"Failed to fetch version by id {version_id}")
return None
async def _fetch_version_by_hash(self, model_hash: Optional[str]) -> Optional[Dict]:
if not model_hash:
return None
success, version = await self._make_request(
'GET',
f"{self.base_url}/model-versions/by-hash/{model_hash}",
use_auth=True
)
if success:
return version
logger.warning(f"Failed to fetch version by hash {model_hash}")
return None
def _select_target_version(self, model_data: Dict, model_id: int, version_id: Optional[int]) -> Optional[Dict]:
model_versions = model_data.get('modelVersions', [])
if not model_versions:
logger.warning(f"No model versions found for model {model_id}")
return None
if version_id is not None:
target_version = next(
(item for item in model_versions if item.get('id') == version_id),
None
)
if target_version is None:
logger.warning(
f"Version {version_id} not found for model {model_id}, defaulting to first version"
)
return model_versions[0]
return target_version
return model_versions[0]
def _extract_primary_model_hash(self, version_entry: Dict) -> Optional[str]:
for file_info in version_entry.get('files', []):
if file_info.get('type') == 'Model' and file_info.get('primary'):
hashes = file_info.get('hashes', {})
model_hash = hashes.get('SHA256')
if model_hash:
return model_hash
return None
def _build_version_from_model_data(self, version_entry: Dict, model_id: int, model_data: Dict) -> Dict:
version = copy.deepcopy(version_entry)
version.pop('index', None)
version['modelId'] = model_id
version['model'] = {
'name': model_data.get('name'),
'type': model_data.get('type'),
'nsfw': model_data.get('nsfw'),
'poi': model_data.get('poi')
}
return version
def _enrich_version_with_model_data(self, version: Dict, model_data: Dict) -> None:
model_info = version.get('model')
if not isinstance(model_info, dict):
model_info = {}
version['model'] = model_info
model_info['description'] = model_data.get("description")
model_info['tags'] = model_data.get("tags", [])
version['creator'] = model_data.get("creator")
async def get_model_version_info(self, version_id: str) -> Tuple[Optional[Dict], Optional[str]]:
"""Fetch model version metadata from Civitai
@@ -283,11 +349,10 @@ class CivitaiClient:
- An error message if there was an error, or None on success
"""
try:
downloader = await get_downloader()
url = f"{self.base_url}/model-versions/{version_id}"
logger.debug(f"Resolving DNS for model version info: {url}")
success, result = await downloader.make_request(
success, result = await self._make_request(
'GET',
url,
use_auth=True
@@ -295,6 +360,7 @@ class CivitaiClient:
if success:
logger.debug(f"Successfully fetched model version info for: {version_id}")
self._remove_comfy_metadata(result)
return result, None
# Handle specific error cases
@@ -306,6 +372,8 @@ class CivitaiClient:
# Other error cases
logger.error(f"Failed to fetch model info for {version_id}: {result}")
return None, str(result)
except RateLimitError:
raise
except Exception as e:
error_msg = f"Error fetching model version info: {e}"
logger.error(error_msg)
@@ -313,7 +381,7 @@ class CivitaiClient:
async def get_image_info(self, image_id: str) -> Optional[Dict]:
"""Fetch image information from Civitai API
Args:
image_id: The Civitai image ID
@@ -321,11 +389,10 @@ class CivitaiClient:
Optional[Dict]: The image data or None if not found
"""
try:
downloader = await get_downloader()
url = f"{self.base_url}/images?imageId={image_id}&nsfw=X"
logger.debug(f"Fetching image info for ID: {image_id}")
success, result = await downloader.make_request(
success, result = await self._make_request(
'GET',
url,
use_auth=True
@@ -340,7 +407,44 @@ class CivitaiClient:
logger.error(f"Failed to fetch image info for ID: {image_id}: {result}")
return None
except RateLimitError:
raise
except Exception as e:
error_msg = f"Error fetching image info: {e}"
logger.error(error_msg)
return None
async def get_user_models(self, username: str) -> Optional[List[Dict]]:
"""Fetch all models for a specific Civitai user."""
if not username:
return None
try:
url = f"{self.base_url}/models?username={username}"
success, result = await self._make_request(
'GET',
url,
use_auth=True
)
if not success:
logger.error("Failed to fetch models for %s: %s", username, result)
return None
items = result.get("items") if isinstance(result, dict) else None
if not isinstance(items, list):
return []
for model in items:
versions = model.get("modelVersions")
if not isinstance(versions, list):
continue
for version in versions:
self._remove_comfy_metadata(version)
return items
except RateLimitError:
raise
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Error fetching models for %s: %s", username, exc)
return None

View File

@@ -5,6 +5,8 @@ from __future__ import annotations
import logging
from typing import Any, Awaitable, Callable, Dict, Optional
from .downloader import DownloadProgress
logger = logging.getLogger(__name__)
@@ -29,14 +31,40 @@ class DownloadCoordinator:
download_id = payload.get("download_id") or self._ws_manager.generate_download_id()
payload.setdefault("download_id", download_id)
async def progress_callback(progress: Any) -> None:
async def progress_callback(progress: Any, snapshot: Optional[DownloadProgress] = None) -> None:
percent = 0.0
metrics: Optional[DownloadProgress] = None
if isinstance(progress, DownloadProgress):
metrics = progress
percent = progress.percent_complete
elif isinstance(snapshot, DownloadProgress):
metrics = snapshot
percent = snapshot.percent_complete
else:
try:
percent = float(progress)
except (TypeError, ValueError):
percent = 0.0
payload: Dict[str, Any] = {
"status": "progress",
"progress": round(percent),
"download_id": download_id,
}
if metrics is not None:
payload.update(
{
"bytes_downloaded": metrics.bytes_downloaded,
"total_bytes": metrics.total_bytes,
"bytes_per_second": metrics.bytes_per_second,
}
)
await self._ws_manager.broadcast_download_progress(
download_id,
{
"status": "progress",
"progress": progress,
"download_id": download_id,
},
payload,
)
model_id = self._parse_optional_int(payload.get("model_id"), "model_id")
@@ -81,6 +109,56 @@ class DownloadCoordinator:
return result
async def pause_download(self, download_id: str) -> Dict[str, Any]:
"""Pause an active download and notify listeners."""
download_manager = await self._download_manager_factory()
result = await download_manager.pause_download(download_id)
if result.get("success"):
cached_progress = self._ws_manager.get_download_progress(download_id) or {}
payload: Dict[str, Any] = {
"status": "paused",
"progress": cached_progress.get("progress", 0),
"download_id": download_id,
"message": "Download paused by user",
}
for field in ("bytes_downloaded", "total_bytes", "bytes_per_second"):
if field in cached_progress:
payload[field] = cached_progress[field]
payload["bytes_per_second"] = 0.0
await self._ws_manager.broadcast_download_progress(download_id, payload)
return result
async def resume_download(self, download_id: str) -> Dict[str, Any]:
"""Resume a paused download and notify listeners."""
download_manager = await self._download_manager_factory()
result = await download_manager.resume_download(download_id)
if result.get("success"):
cached_progress = self._ws_manager.get_download_progress(download_id) or {}
payload: Dict[str, Any] = {
"status": "downloading",
"progress": cached_progress.get("progress", 0),
"download_id": download_id,
"message": "Download resumed by user",
}
for field in ("bytes_downloaded", "total_bytes"):
if field in cached_progress:
payload[field] = cached_progress[field]
payload["bytes_per_second"] = cached_progress.get("bytes_per_second", 0.0)
await self._ws_manager.broadcast_download_progress(download_id, payload)
return result
async def list_active_downloads(self) -> Dict[str, Any]:
"""Return the active download map from the underlying manager."""

View File

@@ -1,17 +1,20 @@
import logging
import os
import asyncio
import inspect
from collections import OrderedDict
import uuid
from typing import Dict
from typing import Dict, List, Optional, Tuple
from urllib.parse import urlparse
from ..utils.models import LoraMetadata, CheckpointMetadata, EmbeddingMetadata
from ..utils.constants import CARD_PREVIEW_WIDTH, VALID_LORA_TYPES, CIVITAI_MODEL_TAGS
from ..utils.constants import CARD_PREVIEW_WIDTH, VALID_LORA_TYPES
from ..utils.civitai_utils import rewrite_preview_url
from ..utils.exif_utils import ExifUtils
from ..utils.metadata_manager import MetadataManager
from .service_registry import ServiceRegistry
from .settings_manager import settings
from .settings_manager import get_settings_manager
from .metadata_service import get_default_metadata_provider
from .downloader import get_downloader
from .downloader import get_downloader, DownloadProgress
# Download to temporary file first
import tempfile
@@ -40,6 +43,7 @@ class DownloadManager:
self._active_downloads = OrderedDict() # download_id -> download_info
self._download_semaphore = asyncio.Semaphore(5) # Limit concurrent downloads
self._download_tasks = {} # download_id -> asyncio.Task
self._pause_events: Dict[str, asyncio.Event] = {}
async def _get_lora_scanner(self):
"""Get the lora scanner from registry"""
@@ -80,13 +84,20 @@ class DownloadManager:
'model_id': model_id,
'model_version_id': model_version_id,
'progress': 0,
'status': 'queued'
'status': 'queued',
'bytes_downloaded': 0,
'total_bytes': None,
'bytes_per_second': 0.0,
}
pause_event = asyncio.Event()
pause_event.set()
self._pause_events[task_id] = pause_event
# Create tracking task
download_task = asyncio.create_task(
self._download_with_semaphore(
task_id, model_id, model_version_id, save_dir,
task_id, model_id, model_version_id, save_dir,
relative_path, progress_callback, use_default_paths, source
)
)
@@ -105,9 +116,10 @@ class DownloadManager:
# Clean up task reference
if task_id in self._download_tasks:
del self._download_tasks[task_id]
self._pause_events.pop(task_id, None)
async def _download_with_semaphore(self, task_id: str, model_id: int, model_version_id: int,
save_dir: str, relative_path: str,
save_dir: str, relative_path: str,
progress_callback=None, use_default_paths: bool = False,
source: str = None):
"""Execute download with semaphore to limit concurrency"""
@@ -117,15 +129,30 @@ class DownloadManager:
# Wrap progress callback to track progress in active_downloads
original_callback = progress_callback
async def tracking_callback(progress):
async def tracking_callback(progress, metrics=None):
progress_value, snapshot = self._normalize_progress(progress, metrics)
if task_id in self._active_downloads:
self._active_downloads[task_id]['progress'] = progress
info = self._active_downloads[task_id]
info['progress'] = round(progress_value)
if snapshot is not None:
info['bytes_downloaded'] = snapshot.bytes_downloaded
info['total_bytes'] = snapshot.total_bytes
info['bytes_per_second'] = snapshot.bytes_per_second
if original_callback:
await original_callback(progress)
await self._dispatch_progress(original_callback, snapshot, progress_value)
# Acquire semaphore to limit concurrent downloads
try:
async with self._download_semaphore:
pause_event = self._pause_events.get(task_id)
if pause_event is not None and not pause_event.is_set():
if task_id in self._active_downloads:
self._active_downloads[task_id]['status'] = 'paused'
self._active_downloads[task_id]['bytes_per_second'] = 0.0
await pause_event.wait()
# Update status to downloading
if task_id in self._active_downloads:
self._active_downloads[task_id]['status'] = 'downloading'
@@ -147,12 +174,14 @@ class DownloadManager:
self._active_downloads[task_id]['status'] = 'completed' if result['success'] else 'failed'
if not result['success']:
self._active_downloads[task_id]['error'] = result.get('error', 'Unknown error')
self._active_downloads[task_id]['bytes_per_second'] = 0.0
return result
except asyncio.CancelledError:
# Handle cancellation
if task_id in self._active_downloads:
self._active_downloads[task_id]['status'] = 'cancelled'
self._active_downloads[task_id]['bytes_per_second'] = 0.0
logger.info(f"Download cancelled for task {task_id}")
raise
except Exception as e:
@@ -161,6 +190,7 @@ class DownloadManager:
if task_id in self._active_downloads:
self._active_downloads[task_id]['status'] = 'failed'
self._active_downloads[task_id]['error'] = str(e)
self._active_downloads[task_id]['bytes_per_second'] = 0.0
return {'success': False, 'error': str(e)}
finally:
# Schedule cleanup of download record after delay
@@ -172,9 +202,17 @@ class DownloadManager:
if task_id in self._active_downloads:
del self._active_downloads[task_id]
async def _execute_original_download(self, model_id, model_version_id, save_dir,
relative_path, progress_callback, use_default_paths,
download_id=None, source=None):
async def _execute_original_download(
self,
model_id,
model_version_id,
save_dir,
relative_path,
progress_callback,
use_default_paths,
download_id=None,
source=None,
):
"""Wrapper for original download_from_civitai implementation"""
try:
# Check if model version already exists in library
@@ -195,13 +233,8 @@ class DownloadManager:
# Check embedding scanner
if await embedding_scanner.check_model_version_exists(model_version_id):
return {'success': False, 'error': 'Model version already exists in embedding library'}
# Get metadata provider based on source parameter
if source == 'civarchive':
from .metadata_service import get_metadata_provider
metadata_provider = await get_metadata_provider('civarchive')
else:
metadata_provider = await get_default_metadata_provider()
metadata_provider = await get_default_metadata_provider()
# Get version info based on the provided identifier
version_info = await metadata_provider.get_model_version(model_id, model_version_id)
@@ -241,23 +274,24 @@ class DownloadManager:
# Handle use_default_paths
if use_default_paths:
settings_manager = get_settings_manager()
# Set save_dir based on model type
if model_type == 'checkpoint':
default_path = settings.get('default_checkpoint_root')
default_path = settings_manager.get('default_checkpoint_root')
if not default_path:
return {'success': False, 'error': 'Default checkpoint root path not set in settings'}
save_dir = default_path
elif model_type == 'lora':
default_path = settings.get('default_lora_root')
default_path = settings_manager.get('default_lora_root')
if not default_path:
return {'success': False, 'error': 'Default lora root path not set in settings'}
save_dir = default_path
elif model_type == 'embedding':
default_path = settings.get('default_embedding_root')
default_path = settings_manager.get('default_embedding_root')
if not default_path:
return {'success': False, 'error': 'Default embedding root path not set in settings'}
save_dir = default_path
# Calculate relative path using template
relative_path = self._calculate_relative_path(version_info, model_type)
@@ -291,11 +325,22 @@ class DownloadManager:
await progress_callback(0)
# 2. Get file information
file_info = next((f for f in version_info.get('files', []) if f.get('primary')), None)
file_info = next((f for f in version_info.get('files', []) if f.get('primary') and f.get('type') == 'Model'), None)
if not file_info:
return {'success': False, 'error': 'No primary file found in metadata'}
if not file_info.get('downloadUrl'):
return {'success': False, 'error': 'No download URL found for primary file'}
mirrors = file_info.get('mirrors') or []
download_urls = []
if mirrors:
for mirror in mirrors:
if mirror.get('deletedAt') is None and mirror.get('url'):
download_urls.append(mirror['url'])
else:
download_url = file_info.get('downloadUrl')
if download_url:
download_urls.append(download_url)
if not download_urls:
return {'success': False, 'error': 'No mirror URL found'}
# 3. Prepare download
file_name = file_info['name']
@@ -314,14 +359,14 @@ class DownloadManager:
# 6. Start download process
result = await self._execute_download(
download_url=file_info.get('downloadUrl', ''),
download_urls=download_urls,
save_dir=save_dir,
metadata=metadata,
version_info=version_info,
relative_path=relative_path,
progress_callback=progress_callback,
model_type=model_type,
download_id=download_id
download_id=download_id,
)
# If early_access_msg exists and download failed, replace error message
@@ -349,7 +394,8 @@ class DownloadManager:
Relative path string
"""
# Get path template from settings for specific model type
path_template = settings.get_download_path_template(model_type)
settings_manager = get_settings_manager()
path_template = settings_manager.get_download_path_template(model_type)
# If template is empty, return empty path (flat structure)
if not path_template:
@@ -366,35 +412,36 @@ class DownloadManager:
author = 'Anonymous'
# Apply mapping if available
base_model_mappings = settings.get('base_model_path_mappings', {})
base_model_mappings = settings_manager.get('base_model_path_mappings', {})
mapped_base_model = base_model_mappings.get(base_model, base_model)
# Get model tags
model_tags = version_info.get('model', {}).get('tags', [])
# Find the first Civitai model tag that exists in model_tags
first_tag = ''
for civitai_tag in CIVITAI_MODEL_TAGS:
if civitai_tag in model_tags:
first_tag = civitai_tag
break
# If no Civitai model tag found, fallback to first tag
if not first_tag and model_tags:
first_tag = model_tags[0]
first_tag = settings_manager.resolve_priority_tag_for_model(model_tags, model_type)
# Format the template with available data
formatted_path = path_template
formatted_path = formatted_path.replace('{base_model}', mapped_base_model)
formatted_path = formatted_path.replace('{first_tag}', first_tag)
formatted_path = formatted_path.replace('{author}', author)
if model_type == 'embedding':
formatted_path = formatted_path.replace(' ', '_')
return formatted_path
async def _execute_download(self, download_url: str, save_dir: str,
metadata, version_info: Dict,
relative_path: str, progress_callback=None,
model_type: str = "lora", download_id: str = None) -> Dict:
async def _execute_download(
self,
download_urls: List[str],
save_dir: str,
metadata,
version_info: Dict,
relative_path: str,
progress_callback=None,
model_type: str = "lora",
download_id: str = None,
) -> Dict:
"""Execute the actual download process including preview images and model files"""
try:
# Extract original filename details
@@ -425,6 +472,8 @@ class DownloadManager:
part_path = save_path + '.part'
metadata_path = os.path.splitext(save_path)[0] + '.metadata.json'
pause_event = self._pause_events.get(download_id) if download_id else None
# Store file paths in active_downloads for potential cleanup
if download_id and download_id in self._active_downloads:
@@ -434,102 +483,157 @@ class DownloadManager:
# Download preview image if available
images = version_info.get('images', [])
if images:
# Report preview download progress
if progress_callback:
await progress_callback(1) # 1% progress for starting preview download
# Check if it's a video or an image
is_video = images[0].get('type') == 'video'
if (is_video):
# For videos, use .mp4 extension
preview_ext = '.mp4'
preview_path = os.path.splitext(save_path)[0] + preview_ext
# Download video directly using downloader
downloader = await get_downloader()
success, result = await downloader.download_file(
images[0]['url'],
preview_path,
use_auth=False # Preview images typically don't need auth
)
if success:
metadata.preview_url = preview_path.replace(os.sep, '/')
metadata.preview_nsfw_level = images[0].get('nsfwLevel', 0)
else:
# For images, use WebP format for better performance
with tempfile.NamedTemporaryFile(suffix='.png', delete=False) as temp_file:
temp_path = temp_file.name
# Download the original image to temp path using downloader
downloader = await get_downloader()
success, content, headers = await downloader.download_to_memory(
images[0]['url'],
use_auth=False
)
if success:
# Save to temp file
with open(temp_path, 'wb') as f:
f.write(content)
# Optimize and convert to WebP
preview_path = os.path.splitext(save_path)[0] + '.webp'
# Use ExifUtils to optimize and convert the image
optimized_data, _ = ExifUtils.optimize_image(
image_data=temp_path,
target_width=CARD_PREVIEW_WIDTH,
format='webp',
quality=85,
preserve_metadata=False
)
# Save the optimized image
with open(preview_path, 'wb') as f:
f.write(optimized_data)
# Update metadata
metadata.preview_url = preview_path.replace(os.sep, '/')
metadata.preview_nsfw_level = images[0].get('nsfwLevel', 0)
# Remove temporary file
try:
os.unlink(temp_path)
except Exception as e:
logger.warning(f"Failed to delete temp file: {e}")
first_image = images[0] if isinstance(images[0], dict) else None
preview_url = first_image.get('url') if first_image else None
media_type = (first_image.get('type') or '').lower() if first_image else ''
nsfw_level = first_image.get('nsfwLevel', 0) if first_image else 0
def _extension_from_url(url: str, fallback: str) -> str:
try:
parsed = urlparse(url)
except ValueError:
return fallback
ext = os.path.splitext(parsed.path)[1]
return ext or fallback
preview_downloaded = False
preview_path = None
if preview_url:
downloader = await get_downloader()
if media_type == 'video':
preview_ext = _extension_from_url(preview_url, '.mp4')
preview_path = os.path.splitext(save_path)[0] + preview_ext
rewritten_url, rewritten = rewrite_preview_url(preview_url, media_type='video')
attempt_urls: List[str] = []
if rewritten:
attempt_urls.append(rewritten_url)
attempt_urls.append(preview_url)
seen_attempts = set()
for attempt in attempt_urls:
if not attempt or attempt in seen_attempts:
continue
seen_attempts.add(attempt)
success, _ = await downloader.download_file(
attempt,
preview_path,
use_auth=False
)
if success:
preview_downloaded = True
break
else:
rewritten_url, rewritten = rewrite_preview_url(preview_url, media_type='image')
if rewritten:
preview_ext = _extension_from_url(preview_url, '.png')
preview_path = os.path.splitext(save_path)[0] + preview_ext
success, _ = await downloader.download_file(
rewritten_url,
preview_path,
use_auth=False
)
if success:
preview_downloaded = True
if not preview_downloaded:
temp_path: str | None = None
try:
with tempfile.NamedTemporaryFile(suffix='.png', delete=False) as temp_file:
temp_path = temp_file.name
success, content, _ = await downloader.download_to_memory(
preview_url,
use_auth=False
)
if success:
with open(temp_path, 'wb') as temp_file_handle:
temp_file_handle.write(content)
preview_path = os.path.splitext(save_path)[0] + '.webp'
optimized_data, _ = ExifUtils.optimize_image(
image_data=temp_path,
target_width=CARD_PREVIEW_WIDTH,
format='webp',
quality=85,
preserve_metadata=False
)
with open(preview_path, 'wb') as preview_file:
preview_file.write(optimized_data)
preview_downloaded = True
finally:
if temp_path and os.path.exists(temp_path):
try:
os.unlink(temp_path)
except Exception as e:
logger.warning(f"Failed to delete temp file: {e}")
if preview_downloaded and preview_path:
metadata.preview_url = preview_path.replace(os.sep, '/')
metadata.preview_nsfw_level = nsfw_level
if download_id and download_id in self._active_downloads:
self._active_downloads[download_id]['preview_path'] = preview_path
# Report preview download completion
if progress_callback:
await progress_callback(3) # 3% progress after preview download
# Download model file with progress tracking using downloader
downloader = await get_downloader()
# Determine if the download URL is from Civitai
use_auth = download_url.startswith("https://civitai.com/api/download/")
success, result = await downloader.download_file(
download_url,
save_path, # Use full path instead of separate dir and filename
progress_callback=lambda p: self._handle_download_progress(p, progress_callback),
use_auth=use_auth # Only use authentication for Civitai downloads
)
last_error = None
for download_url in download_urls:
use_auth = download_url.startswith("https://civitai.com/api/download/")
download_kwargs = {
"progress_callback": lambda progress, snapshot=None: self._handle_download_progress(
progress,
progress_callback,
snapshot,
),
"use_auth": use_auth, # Only use authentication for Civitai downloads
}
if not success:
if pause_event is not None:
download_kwargs["pause_event"] = pause_event
success, result = await downloader.download_file(
download_url,
save_path, # Use full path instead of separate dir and filename
**download_kwargs,
)
if success:
break
last_error = result
if os.path.exists(save_path):
try:
os.remove(save_path)
except Exception as e:
logger.warning(f"Failed to remove incomplete file {save_path}: {e}")
else:
# Clean up files on failure, but preserve .part file for resume
cleanup_files = [metadata_path]
if metadata.preview_url and os.path.exists(metadata.preview_url):
cleanup_files.append(metadata.preview_url)
preview_path_value = getattr(metadata, 'preview_url', None)
if preview_path_value and os.path.exists(preview_path_value):
cleanup_files.append(preview_path_value)
for path in cleanup_files:
if path and os.path.exists(path):
try:
os.remove(path)
except Exception as e:
logger.warning(f"Failed to cleanup file {path}: {e}")
# Log but don't remove .part file to allow resume
if os.path.exists(part_path):
logger.info(f"Preserving partial download for resume: {part_path}")
return {'success': False, 'error': result}
return {'success': False, 'error': last_error or 'Failed to download file'}
# 4. Update file information (size and modified time)
metadata.update_file_info(save_path)
@@ -578,21 +682,37 @@ class DownloadManager:
return {'success': False, 'error': str(e)}
async def _handle_download_progress(self, file_progress: float, progress_callback):
"""Convert file download progress to overall progress
Args:
file_progress: Progress of file download (0-100)
progress_callback: Callback function for progress updates
"""
if progress_callback:
# Scale file progress to 3-100 range (after preview download)
overall_progress = 3 + (file_progress * 0.97) # 97% of progress for file download
await progress_callback(round(overall_progress))
async def _handle_download_progress(
self,
progress_update,
progress_callback,
snapshot=None,
):
"""Convert file download progress to overall progress."""
if not progress_callback:
return
file_progress, original_snapshot = self._normalize_progress(progress_update, snapshot)
overall_progress = 3 + (file_progress * 0.97)
overall_progress = max(0.0, min(overall_progress, 100.0))
rounded_progress = round(overall_progress)
normalized_snapshot: Optional[DownloadProgress] = None
if original_snapshot is not None:
normalized_snapshot = DownloadProgress(
percent_complete=overall_progress,
bytes_downloaded=original_snapshot.bytes_downloaded,
total_bytes=original_snapshot.total_bytes,
bytes_per_second=original_snapshot.bytes_per_second,
timestamp=original_snapshot.timestamp,
)
await self._dispatch_progress(progress_callback, normalized_snapshot, rounded_progress)
async def cancel_download(self, download_id: str) -> Dict:
"""Cancel an active download by download_id
Args:
download_id: The unique identifier of the download task
@@ -606,10 +726,15 @@ class DownloadManager:
# Get the task and cancel it
task = self._download_tasks[download_id]
task.cancel()
pause_event = self._pause_events.get(download_id)
if pause_event is not None:
pause_event.set()
# Update status in active downloads
if download_id in self._active_downloads:
self._active_downloads[download_id]['status'] = 'cancelling'
self._active_downloads[download_id]['bytes_per_second'] = 0.0
# Wait briefly for the task to acknowledge cancellation
try:
@@ -650,7 +775,15 @@ class DownloadManager:
except Exception as e:
logger.error(f"Error deleting metadata file: {e}")
# Delete preview file if exists (.webp or .mp4)
preview_path_value = download_info.get('preview_path')
if preview_path_value and os.path.exists(preview_path_value):
try:
os.unlink(preview_path_value)
logger.debug(f"Deleted preview file: {preview_path_value}")
except Exception as e:
logger.error(f"Error deleting preview file: {e}")
# Delete preview file if exists (.webp or .mp4) for legacy paths
for preview_ext in ['.webp', '.mp4']:
preview_path = os.path.splitext(file_path)[0] + preview_ext
if os.path.exists(preview_path):
@@ -664,7 +797,99 @@ class DownloadManager:
except Exception as e:
logger.error(f"Error cancelling download: {e}", exc_info=True)
return {'success': False, 'error': str(e)}
finally:
self._pause_events.pop(download_id, None)
async def pause_download(self, download_id: str) -> Dict:
"""Pause an active download without losing progress."""
if download_id not in self._download_tasks:
return {'success': False, 'error': 'Download task not found'}
pause_event = self._pause_events.get(download_id)
if pause_event is None:
pause_event = asyncio.Event()
pause_event.set()
self._pause_events[download_id] = pause_event
if not pause_event.is_set():
return {'success': False, 'error': 'Download is already paused'}
pause_event.clear()
download_info = self._active_downloads.get(download_id)
if download_info is not None:
download_info['status'] = 'paused'
download_info['bytes_per_second'] = 0.0
return {'success': True, 'message': 'Download paused successfully'}
async def resume_download(self, download_id: str) -> Dict:
"""Resume a previously paused download."""
pause_event = self._pause_events.get(download_id)
if pause_event is None:
return {'success': False, 'error': 'Download task not found'}
if pause_event.is_set():
return {'success': False, 'error': 'Download is not paused'}
pause_event.set()
download_info = self._active_downloads.get(download_id)
if download_info is not None:
if download_info.get('status') == 'paused':
download_info['status'] = 'downloading'
download_info.setdefault('bytes_per_second', 0.0)
return {'success': True, 'message': 'Download resumed successfully'}
@staticmethod
def _coerce_progress_value(progress) -> float:
try:
return float(progress)
except (TypeError, ValueError):
return 0.0
@classmethod
def _normalize_progress(
cls,
progress,
snapshot: Optional[DownloadProgress] = None,
) -> Tuple[float, Optional[DownloadProgress]]:
if isinstance(progress, DownloadProgress):
return progress.percent_complete, progress
if isinstance(snapshot, DownloadProgress):
return snapshot.percent_complete, snapshot
if isinstance(progress, dict):
if 'percent_complete' in progress:
return cls._coerce_progress_value(progress['percent_complete']), snapshot
if 'progress' in progress:
return cls._coerce_progress_value(progress['progress']), snapshot
return cls._coerce_progress_value(progress), None
async def _dispatch_progress(
self,
callback,
snapshot: Optional[DownloadProgress],
progress_value: float,
) -> None:
try:
if snapshot is not None:
result = callback(snapshot, snapshot)
else:
result = callback(progress_value)
except TypeError:
result = callback(progress_value)
if inspect.isawaitable(result):
await result
elif asyncio.iscoroutine(result):
await result
async def get_active_downloads(self) -> Dict:
"""Get information about all active downloads
@@ -679,8 +904,11 @@ class DownloadManager:
'model_version_id': info.get('model_version_id'),
'progress': info.get('progress', 0),
'status': info.get('status', 'unknown'),
'error': info.get('error', None)
'error': info.get('error', None),
'bytes_downloaded': info.get('bytes_downloaded', 0),
'total_bytes': info.get('total_bytes'),
'bytes_per_second': info.get('bytes_per_second', 0.0),
}
for task_id, info in self._active_downloads.items()
]
}
}

View File

@@ -14,13 +14,28 @@ import os
import logging
import asyncio
import aiohttp
from datetime import datetime
from typing import Optional, Dict, Tuple, Callable, Union
from ..services.settings_manager import settings
from collections import deque
from dataclasses import dataclass
from datetime import datetime, timedelta
from email.utils import parsedate_to_datetime
from typing import Optional, Dict, Tuple, Callable, Union, Awaitable
from ..services.settings_manager import get_settings_manager
from .errors import RateLimitError
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class DownloadProgress:
"""Snapshot of a download transfer at a moment in time."""
percent_complete: float
bytes_downloaded: int
total_bytes: Optional[int]
bytes_per_second: float
timestamp: float
class Downloader:
"""Unified downloader for all HTTP/HTTPS downloads in the application."""
@@ -94,12 +109,13 @@ class Downloader:
# Check for app-level proxy settings
proxy_url = None
if settings.get('proxy_enabled', False):
proxy_host = settings.get('proxy_host', '').strip()
proxy_port = settings.get('proxy_port', '').strip()
proxy_type = settings.get('proxy_type', 'http').lower()
proxy_username = settings.get('proxy_username', '').strip()
proxy_password = settings.get('proxy_password', '').strip()
settings_manager = get_settings_manager()
if settings_manager.get('proxy_enabled', False):
proxy_host = settings_manager.get('proxy_host', '').strip()
proxy_port = settings_manager.get('proxy_port', '').strip()
proxy_type = settings_manager.get('proxy_type', 'http').lower()
proxy_username = settings_manager.get('proxy_username', '').strip()
proxy_password = settings_manager.get('proxy_password', '').strip()
if proxy_host and proxy_port:
# Build proxy URL
@@ -146,7 +162,8 @@ class Downloader:
if use_auth:
# Add CivitAI API key if available
api_key = settings.get('civitai_api_key')
settings_manager = get_settings_manager()
api_key = settings_manager.get('civitai_api_key')
if api_key:
headers['Authorization'] = f'Bearer {api_key}'
headers['Content-Type'] = 'application/json'
@@ -157,10 +174,11 @@ class Downloader:
self,
url: str,
save_path: str,
progress_callback: Optional[Callable[[float], None]] = None,
progress_callback: Optional[Callable[..., Awaitable[None]]] = None,
use_auth: bool = False,
custom_headers: Optional[Dict[str, str]] = None,
allow_resume: bool = True
allow_resume: bool = True,
pause_event: Optional[asyncio.Event] = None,
) -> Tuple[bool, str]:
"""
Download a file with resumable downloads and retry mechanism
@@ -172,6 +190,7 @@ class Downloader:
use_auth: Whether to include authentication headers (e.g., CivitAI API key)
custom_headers: Additional headers to include in request
allow_resume: Whether to support resumable downloads
pause_event: Optional event that, when cleared, will pause streaming until set again
Returns:
Tuple[bool, str]: (success, save_path or error message)
@@ -246,7 +265,16 @@ class Downloader:
if allow_resume:
os.rename(part_path, save_path)
if progress_callback:
await progress_callback(100)
await self._dispatch_progress_callback(
progress_callback,
DownloadProgress(
percent_complete=100.0,
bytes_downloaded=part_size,
total_bytes=actual_size,
bytes_per_second=0.0,
timestamp=datetime.now().timestamp(),
),
)
return True, save_path
# Remove corrupted part file and restart
os.remove(part_path)
@@ -274,6 +302,8 @@ class Downloader:
current_size = resume_offset
last_progress_report_time = datetime.now()
progress_samples: deque[tuple[datetime, int]] = deque()
progress_samples.append((last_progress_report_time, current_size))
# Ensure directory exists
os.makedirs(os.path.dirname(save_path), exist_ok=True)
@@ -283,18 +313,41 @@ class Downloader:
mode = 'ab' if (allow_resume and resume_offset > 0) else 'wb'
with open(part_path, mode) as f:
async for chunk in response.content.iter_chunked(self.chunk_size):
if pause_event is not None and not pause_event.is_set():
await pause_event.wait()
if chunk:
# Run blocking file write in executor
await loop.run_in_executor(None, f.write, chunk)
current_size += len(chunk)
# Limit progress update frequency to reduce overhead
now = datetime.now()
time_diff = (now - last_progress_report_time).total_seconds()
if progress_callback and total_size and time_diff >= 1.0:
progress = (current_size / total_size) * 100
await progress_callback(progress)
if progress_callback and time_diff >= 1.0:
progress_samples.append((now, current_size))
cutoff = now - timedelta(seconds=5)
while progress_samples and progress_samples[0][0] < cutoff:
progress_samples.popleft()
percent = (current_size / total_size) * 100 if total_size else 0.0
bytes_per_second = 0.0
if len(progress_samples) >= 2:
first_time, first_bytes = progress_samples[0]
last_time, last_bytes = progress_samples[-1]
elapsed = (last_time - first_time).total_seconds()
if elapsed > 0:
bytes_per_second = (last_bytes - first_bytes) / elapsed
progress_snapshot = DownloadProgress(
percent_complete=percent,
bytes_downloaded=current_size,
total_bytes=total_size or None,
bytes_per_second=bytes_per_second,
timestamp=now.timestamp(),
)
await self._dispatch_progress_callback(progress_callback, progress_snapshot)
last_progress_report_time = now
# Download completed successfully
@@ -329,7 +382,15 @@ class Downloader:
# Ensure 100% progress is reported
if progress_callback:
await progress_callback(100)
final_snapshot = DownloadProgress(
percent_complete=100.0,
bytes_downloaded=final_size,
total_bytes=total_size or final_size,
bytes_per_second=0.0,
timestamp=datetime.now().timestamp(),
)
await self._dispatch_progress_callback(progress_callback, final_snapshot)
return True, save_path
@@ -361,7 +422,24 @@ class Downloader:
return False, str(e)
return False, f"Download failed after {self.max_retries + 1} attempts"
async def _dispatch_progress_callback(
self,
progress_callback: Callable[..., Awaitable[None]],
snapshot: DownloadProgress,
) -> None:
"""Invoke a progress callback while preserving backward compatibility."""
try:
result = progress_callback(snapshot, snapshot)
except TypeError:
result = progress_callback(snapshot.percent_complete)
if asyncio.iscoroutine(result):
await result
elif hasattr(result, "__await__"):
await result
async def download_to_memory(
self,
url: str,
@@ -511,6 +589,19 @@ class Downloader:
return False, "Access forbidden"
elif response.status == 404:
return False, "Resource not found"
elif response.status == 429:
retry_after = self._extract_retry_after(response.headers)
error_msg = "Request rate limited"
logger.warning(
"Rate limit encountered for %s %s; retry_after=%s",
method,
url,
retry_after,
)
return False, RateLimitError(
error_msg,
retry_after=retry_after,
)
else:
return False, f"Request failed with status {response.status}"
@@ -532,6 +623,38 @@ class Downloader:
await self._create_session()
logger.info("HTTP session refreshed due to settings change")
@staticmethod
def _extract_retry_after(headers) -> Optional[float]:
"""Parse the Retry-After header into seconds."""
if not headers:
return None
header_value = headers.get("Retry-After")
if not header_value:
return None
header_value = header_value.strip()
if not header_value:
return None
if header_value.isdigit():
try:
seconds = float(header_value)
except ValueError:
return None
return max(0.0, seconds)
try:
retry_datetime = parsedate_to_datetime(header_value)
except (TypeError, ValueError):
return None
if retry_datetime.tzinfo is None:
return None
delta = retry_datetime - datetime.now(tz=retry_datetime.tzinfo)
return max(0.0, delta.total_seconds())
# Global instance accessor
async def get_downloader() -> Downloader:

View File

@@ -11,13 +11,14 @@ logger = logging.getLogger(__name__)
class EmbeddingService(BaseModelService):
"""Embedding-specific service implementation"""
def __init__(self, scanner):
def __init__(self, scanner, update_service=None):
"""Initialize Embedding service
Args:
scanner: Embedding scanner instance
update_service: Optional service for remote update tracking.
"""
super().__init__("embedding", scanner, EmbeddingMetadata)
super().__init__("embedding", scanner, EmbeddingMetadata, update_service=update_service)
async def format_response(self, embedding_data: Dict) -> Dict:
"""Format Embedding data for API response"""

21
py/services/errors.py Normal file
View File

@@ -0,0 +1,21 @@
"""Common service-level exception types."""
from __future__ import annotations
from typing import Optional
class RateLimitError(RuntimeError):
"""Raised when a remote provider rejects a request due to rate limiting."""
def __init__(
self,
message: str,
*,
retry_after: Optional[float] = None,
provider: Optional[str] = None,
) -> None:
super().__init__(message)
self.retry_after = retry_after
self.provider = provider

View File

@@ -8,10 +8,11 @@ import os
import shutil
from dataclasses import dataclass
from pathlib import Path
from typing import Dict, List
from typing import Dict, List, Tuple
from .service_registry import ServiceRegistry
from .settings_manager import settings
from .settings_manager import get_settings_manager
from ..utils.example_images_paths import iter_library_roots
logger = logging.getLogger(__name__)
@@ -61,7 +62,8 @@ class ExampleImagesCleanupService:
async def cleanup_example_image_folders(self) -> Dict[str, object]:
"""Clean empty or orphaned example image folders by moving them under a deleted bucket."""
example_images_path = settings.get("example_images_path")
settings_manager = get_settings_manager()
example_images_path = settings_manager.get("example_images_path")
if not example_images_path:
logger.debug("Cleanup skipped: example images path not configured")
return {
@@ -70,9 +72,9 @@ class ExampleImagesCleanupService:
"error_code": "path_not_configured",
}
example_root = Path(example_images_path)
if not example_root.exists():
logger.debug("Cleanup skipped: example images path missing -> %s", example_root)
base_root = Path(example_images_path)
if not base_root.exists():
logger.debug("Cleanup skipped: example images path missing -> %s", base_root)
return {
"success": False,
"error": "Example images path does not exist.",
@@ -91,9 +93,6 @@ class ExampleImagesCleanupService:
"error_code": "scanner_initialization_failed",
}
deleted_bucket = example_root / self._deleted_folder_name
deleted_bucket.mkdir(exist_ok=True)
checked_folders = 0
moved_empty = 0
moved_orphaned = 0
@@ -101,45 +100,96 @@ class ExampleImagesCleanupService:
move_failures = 0
errors: List[str] = []
for entry in os.scandir(example_root):
if not entry.is_dir(follow_symlinks=False):
resolved_base = base_root.resolve()
library_paths: List[Tuple[str, Path]] = []
processed_paths = {resolved_base}
for library_name, library_path in iter_library_roots():
if not library_path:
continue
if entry.name == self._deleted_folder_name:
continue
checked_folders += 1
folder_path = Path(entry.path)
library_root = Path(library_path)
try:
if self._is_folder_empty(folder_path):
if await self._remove_empty_folder(folder_path):
moved_empty += 1
else:
move_failures += 1
continue
if not self._is_hash_folder(entry.name):
skipped_non_hash += 1
continue
hash_exists = (
lora_scanner.has_hash(entry.name)
or checkpoint_scanner.has_hash(entry.name)
or embedding_scanner.has_hash(entry.name)
resolved = library_root.resolve()
except FileNotFoundError:
continue
if resolved in processed_paths:
continue
if not library_root.exists():
logger.debug(
"Skipping cleanup for library '%s': folder missing (%s)",
library_name,
library_root,
)
continue
processed_paths.add(resolved)
library_paths.append((library_name, library_root))
if not hash_exists:
if await self._move_folder(folder_path, deleted_bucket):
moved_orphaned += 1
else:
move_failures += 1
deleted_roots: List[Path] = []
except Exception as exc: # pragma: no cover - filesystem guard
move_failures += 1
error_message = f"{entry.name}: {exc}"
errors.append(error_message)
logger.error("Error processing example images folder %s: %s", folder_path, exc, exc_info=True)
# Build list of (label, root) pairs including the base root for legacy layouts
cleanup_targets: List[Tuple[str, Path]] = [("__base__", base_root)] + library_paths
library_root_set = {root.resolve() for _, root in library_paths}
for label, root_path in cleanup_targets:
deleted_bucket = root_path / self._deleted_folder_name
deleted_bucket.mkdir(exist_ok=True)
deleted_roots.append(deleted_bucket)
for entry in os.scandir(root_path):
if not entry.is_dir(follow_symlinks=False):
continue
if entry.name == self._deleted_folder_name:
continue
entry_path = Path(entry.path)
if label == "__base__":
try:
resolved_entry = entry_path.resolve()
except FileNotFoundError:
continue
if resolved_entry in library_root_set:
# Skip library-specific folders tracked separately
continue
checked_folders += 1
try:
if self._is_folder_empty(entry_path):
if await self._remove_empty_folder(entry_path):
moved_empty += 1
else:
move_failures += 1
continue
if not self._is_hash_folder(entry.name):
skipped_non_hash += 1
continue
hash_exists = (
lora_scanner.has_hash(entry.name)
or checkpoint_scanner.has_hash(entry.name)
or embedding_scanner.has_hash(entry.name)
)
if not hash_exists:
if await self._move_folder(entry_path, deleted_bucket):
moved_orphaned += 1
else:
move_failures += 1
except Exception as exc: # pragma: no cover - filesystem guard
move_failures += 1
error_message = f"{entry.name}: {exc}"
errors.append(error_message)
logger.error(
"Error processing example images folder %s: %s",
entry_path,
exc,
exc_info=True,
)
partial_success = move_failures > 0 and (moved_empty > 0 or moved_orphaned > 0)
success = move_failures == 0 and not errors
@@ -152,11 +202,12 @@ class ExampleImagesCleanupService:
skipped_non_hash=skipped_non_hash,
move_failures=move_failures,
errors=errors,
deleted_root=str(deleted_bucket),
deleted_root=str(deleted_roots[0]) if deleted_roots else None,
partial_success=partial_success,
)
summary = result.to_dict()
summary["deleted_roots"] = [str(path) for path in deleted_roots]
if success:
logger.info(
"Example images cleanup complete: checked=%s, moved_empty=%s, moved_orphaned=%s",

View File

@@ -11,13 +11,14 @@ logger = logging.getLogger(__name__)
class LoraService(BaseModelService):
"""LoRA-specific service implementation"""
def __init__(self, scanner):
def __init__(self, scanner, update_service=None):
"""Initialize LoRA service
Args:
scanner: LoRA scanner instance
update_service: Optional service for remote update tracking.
"""
super().__init__("lora", scanner, LoraMetadata)
super().__init__("lora", scanner, LoraMetadata, update_service=update_service)
async def format_response(self, lora_data: Dict) -> Dict:
"""Format LoRA data for API response"""
@@ -178,4 +179,4 @@ class LoraService(BaseModelService):
def find_duplicate_filenames(self) -> Dict:
"""Find LoRAs with conflicting filenames"""
return self.scanner._hash_index.get_duplicate_filenames()
return self.scanner._hash_index.get_duplicate_filenames()

View File

@@ -3,7 +3,7 @@ import logging
import asyncio
from pathlib import Path
from typing import Optional
from .downloader import get_downloader
from .downloader import get_downloader, DownloadProgress
logger = logging.getLogger(__name__)
@@ -77,9 +77,15 @@ class MetadataArchiveManager:
progress_callback("download", f"Downloading from {url}")
# Custom progress callback to report download progress
async def download_progress(progress):
async def download_progress(progress, snapshot=None):
if progress_callback:
progress_callback("download", f"Downloading archive... {progress:.1f}%")
if isinstance(progress, DownloadProgress):
percent = progress.percent_complete
elif isinstance(snapshot, DownloadProgress):
percent = snapshot.percent_complete
else:
percent = float(progress or 0)
progress_callback("download", f"Downloading archive... {percent:.1f}%")
success, result = await downloader.download_file(
url=url,

View File

@@ -1,12 +1,14 @@
import os
import logging
from .model_metadata_provider import (
ModelMetadataProvider,
ModelMetadataProviderManager,
SQLiteModelMetadataProvider,
CivitaiModelMetadataProvider,
CivArchiveModelMetadataProvider,
FallbackMetadataProvider
)
from .settings_manager import settings
from .settings_manager import get_settings_manager
from .metadata_archive_manager import MetadataArchiveManager
from .service_registry import ServiceRegistry
@@ -21,7 +23,8 @@ async def initialize_metadata_providers():
provider_manager.default_provider = None
# Get settings
enable_archive_db = settings.get('enable_metadata_archive_db', False)
settings_manager = get_settings_manager()
enable_archive_db = settings_manager.get('enable_metadata_archive_db', False)
providers = []
@@ -37,7 +40,7 @@ async def initialize_metadata_providers():
sqlite_provider = SQLiteModelMetadataProvider(db_path)
provider_manager.register_provider('sqlite', sqlite_provider)
providers.append(('sqlite', sqlite_provider))
logger.info(f"SQLite metadata provider registered with database: {db_path}")
logger.debug(f"SQLite metadata provider registered with database: {db_path}")
else:
logger.warning("Metadata archive database is enabled but database file not found")
except Exception as e:
@@ -53,26 +56,27 @@ async def initialize_metadata_providers():
except Exception as e:
logger.error(f"Failed to initialize Civitai API metadata provider: {e}")
# Register CivArchive provider, but do NOT add to fallback providers
# Register CivArchive provider, and all add to fallback providers
try:
from .model_metadata_provider import CivArchiveModelMetadataProvider
civarchive_provider = CivArchiveModelMetadataProvider()
provider_manager.register_provider('civarchive', civarchive_provider)
logger.debug("CivArchive metadata provider registered (not included in fallback)")
civarchive_client = await ServiceRegistry.get_civarchive_client()
civarchive_provider = CivArchiveModelMetadataProvider(civarchive_client)
provider_manager.register_provider('civarchive_api', civarchive_provider)
providers.append(('civarchive_api', civarchive_provider))
logger.debug("CivArchive metadata provider registered (also included in fallback)")
except Exception as e:
logger.error(f"Failed to initialize CivArchive metadata provider: {e}")
# Set up fallback provider based on available providers
if len(providers) > 1:
# Always use Civitai API first, then Archive DB
ordered_providers = []
ordered_providers.extend([p[1] for p in providers if p[0] == 'civitai_api'])
ordered_providers.extend([p[1] for p in providers if p[0] == 'sqlite'])
# Always use Civitai API (it has better metadata), then CivArchive API, then Archive DB
ordered_providers: list[tuple[str, ModelMetadataProvider]] = []
ordered_providers.extend([p for p in providers if p[0] == 'civitai_api'])
ordered_providers.extend([p for p in providers if p[0] == 'civarchive_api'])
ordered_providers.extend([p for p in providers if p[0] == 'sqlite'])
if ordered_providers:
fallback_provider = FallbackMetadataProvider(ordered_providers)
provider_manager.register_provider('fallback', fallback_provider, is_default=True)
logger.info(f"Fallback metadata provider registered with {len(ordered_providers)} providers, Civitai API first")
elif len(providers) == 1:
# Only one provider available, set it as default
provider_name, provider = providers[0]
@@ -87,7 +91,8 @@ async def update_metadata_providers():
"""Update metadata providers based on current settings"""
try:
# Get current settings
enable_archive_db = settings.get('enable_metadata_archive_db', False)
settings_manager = get_settings_manager()
enable_archive_db = settings_manager.get('enable_metadata_archive_db', False)
# Reinitialize all providers with new settings
provider_manager = await initialize_metadata_providers()

View File

@@ -10,6 +10,7 @@ from typing import Any, Awaitable, Callable, Dict, Iterable, Optional
from ..services.settings_manager import SettingsManager
from ..utils.model_utils import determine_base_model
from .errors import RateLimitError
logger = logging.getLogger(__name__)
@@ -153,7 +154,12 @@ class MetadataSyncService:
model_data: Dict[str, Any],
update_cache_func: Callable[[str, str, Dict[str, Any]], Awaitable[bool]],
) -> tuple[bool, Optional[str]]:
"""Fetch metadata for a model and update both disk and cache state."""
"""Fetch metadata for a model and update both disk and cache state.
Callers should hydrate ``model_data`` via ``MetadataManager.hydrate_model_data``
before invoking this method so that the persisted payload retains all known
metadata fields.
"""
if not isinstance(model_data, dict):
error = f"Invalid model_data type: {type(model_data)}"
@@ -162,41 +168,118 @@ class MetadataSyncService:
metadata_path = os.path.splitext(file_path)[0] + ".metadata.json"
enable_archive = self._settings.get("enable_metadata_archive_db", False)
previous_source = model_data.get("metadata_source") or (model_data.get("civitai") or {}).get("source")
try:
if model_data.get("civitai_deleted") is True:
if not enable_archive or model_data.get("db_checked") is True:
return (
False,
"CivitAI model is deleted and metadata archive DB is not enabled",
)
metadata_provider = await self._get_provider("sqlite")
else:
metadata_provider = await self._get_default_provider()
provider_attempts: list[tuple[Optional[str], MetadataProviderProtocol]] = []
sqlite_attempted = False
civitai_metadata, error = await metadata_provider.get_model_by_hash(sha256)
if not civitai_metadata:
if error == "Model not found":
if model_data.get("civitai_deleted") is True:
if previous_source in (None, "civarchive"):
try:
provider_attempts.append(("civarchive_api", await self._get_provider("civarchive_api")))
except Exception as exc: # pragma: no cover - provider resolution fault
logger.debug("Unable to resolve civarchive provider: %s", exc)
if enable_archive and model_data.get("db_checked") is not True:
try:
provider_attempts.append(("sqlite", await self._get_provider("sqlite")))
except Exception as exc: # pragma: no cover - provider resolution fault
logger.debug("Unable to resolve sqlite provider: %s", exc)
if not provider_attempts:
if not enable_archive:
error_msg = "CivitAI model is deleted and metadata archive DB is not enabled"
elif model_data.get("db_checked") is True:
error_msg = "CivitAI model is deleted and not found in metadata archive DB"
else:
error_msg = "CivitAI model is deleted and no archive provider is available"
return False, error_msg
else:
provider_attempts.append((None, await self._get_default_provider()))
civitai_metadata: Optional[Dict[str, Any]] = None
metadata_provider: Optional[MetadataProviderProtocol] = None
provider_used: Optional[str] = None
last_error: Optional[str] = None
for provider_name, provider in provider_attempts:
try:
civitai_metadata_candidate, error = await provider.get_model_by_hash(sha256)
except RateLimitError as exc:
exc.provider = exc.provider or (provider_name or provider.__class__.__name__)
raise
except Exception as exc: # pragma: no cover - defensive logging
logger.error("Provider %s failed for hash %s: %s", provider_name, sha256, exc)
civitai_metadata_candidate, error = None, str(exc)
if provider_name == "sqlite":
sqlite_attempted = True
if civitai_metadata_candidate:
civitai_metadata = civitai_metadata_candidate
metadata_provider = provider
provider_used = provider_name
break
last_error = error or last_error
if civitai_metadata is None or metadata_provider is None:
if sqlite_attempted:
model_data["db_checked"] = True
if last_error == "Model not found":
model_data["from_civitai"] = False
model_data["civitai_deleted"] = True
model_data["db_checked"] = enable_archive
model_data["db_checked"] = sqlite_attempted or (enable_archive and model_data.get("db_checked", False))
model_data["last_checked_at"] = datetime.now().timestamp()
data_to_save = model_data.copy()
data_to_save.pop("folder", None)
await self._metadata_manager.save_metadata(file_path, data_to_save)
default_error = (
"CivitAI model is deleted and metadata archive DB is not enabled"
if model_data.get("civitai_deleted") and not enable_archive
else "CivitAI model is deleted and not found in metadata archive DB"
if model_data.get("civitai_deleted") and (model_data.get("db_checked") is True or sqlite_attempted)
else "No provider returned metadata"
)
error_msg = (
f"Error fetching metadata: {error} (model_name={model_data.get('model_name', '')})"
f"Error fetching metadata: {last_error or default_error} "
f"(model_name={model_data.get('model_name', '')})"
)
logger.error(error_msg)
return False, error_msg
model_data["from_civitai"] = True
model_data["civitai_deleted"] = civitai_metadata.get("source") == "archive_db"
model_data["db_checked"] = enable_archive
model_data["civitai_deleted"] = civitai_metadata.get("source") == "archive_db" or civitai_metadata.get("source") == "civarchive"
model_data["db_checked"] = enable_archive and (
civitai_metadata.get("source") == "archive_db" or sqlite_attempted
)
source = civitai_metadata.get("source") or "civitai_api"
if source == "api":
source = "civitai_api"
elif provider_used == "civarchive_api" and source != "civarchive":
source = "civarchive"
elif provider_used == "sqlite":
source = "archive_db"
model_data["metadata_source"] = source
model_data["last_checked_at"] = datetime.now().timestamp()
readable_source = {
"civitai_api": "CivitAI API",
"civarchive": "CivArchive API",
"archive_db": "Archive Database",
}.get(source, source)
logger.info(
"Fetched metadata for %s via %s",
model_data.get("model_name", ""),
readable_source,
)
local_metadata = model_data.copy()
local_metadata.pop("folder", None)
@@ -220,6 +303,16 @@ class MetadataSyncService:
error_msg = f"Error fetching metadata - Missing key: {exc} in model_data={model_data}"
logger.error(error_msg)
return False, error_msg
except RateLimitError as exc:
provider_label = exc.provider or "metadata provider"
wait_hint = (
f"; retry after approximately {int(exc.retry_after)}s"
if exc.retry_after and exc.retry_after > 0
else ""
)
error_msg = f"Rate limited by {provider_label}{wait_hint}"
logger.warning(error_msg)
return False, error_msg
except Exception as exc: # pragma: no cover - error path
error_msg = f"Error fetching metadata: {exc}"
logger.error(error_msg, exc_info=True)

View File

@@ -1,6 +1,6 @@
import asyncio
from typing import List, Dict, Tuple
from dataclasses import dataclass
from typing import Any, Dict, List, Optional, Tuple
from dataclasses import dataclass, field
from operator import itemgetter
from natsort import natsorted
@@ -17,10 +17,12 @@ SUPPORTED_SORT_MODES = [
@dataclass
class ModelCache:
"""Cache structure for model data with extensible sorting"""
"""Cache structure for model data with extensible sorting."""
raw_data: List[Dict]
folders: List[str]
version_index: Dict[int, Dict] = field(default_factory=dict)
def __post_init__(self):
self._lock = asyncio.Lock()
# Cache for last sort: (sort_key, order) -> sorted list
@@ -28,6 +30,58 @@ class ModelCache:
self._last_sorted_data: List[Dict] = []
# Default sort on init
asyncio.create_task(self.resort())
self.rebuild_version_index()
@staticmethod
def _normalize_version_id(value: Any) -> Optional[int]:
"""Normalize a potential version identifier into an integer."""
if isinstance(value, int):
return value
if isinstance(value, str):
try:
return int(value)
except ValueError:
return None
return None
def rebuild_version_index(self) -> None:
"""Rebuild the version index from the current raw data."""
self.version_index = {}
for item in self.raw_data:
self.add_to_version_index(item)
def add_to_version_index(self, item: Dict) -> None:
"""Register a cache item in the version index if possible."""
civitai_data = item.get('civitai') if isinstance(item, dict) else None
if not isinstance(civitai_data, dict):
return
version_id = self._normalize_version_id(civitai_data.get('id'))
if version_id is None:
return
self.version_index[version_id] = item
def remove_from_version_index(self, item: Dict) -> None:
"""Remove a cache item from the version index if present."""
civitai_data = item.get('civitai') if isinstance(item, dict) else None
if not isinstance(civitai_data, dict):
return
version_id = self._normalize_version_id(civitai_data.get('id'))
if version_id is None:
return
existing = self.version_index.get(version_id)
if existing is item or (
isinstance(existing, dict)
and existing.get('file_path') == item.get('file_path')
):
self.version_index.pop(version_id, None)
async def resort(self):
"""Resort cached data according to last sort mode if set"""
@@ -41,6 +95,7 @@ class ModelCache:
all_folders = set(l['folder'] for l in self.raw_data)
self.folders = sorted(list(all_folders), key=lambda x: x.lower())
self.rebuild_version_index()
def _sort_data(self, data: List[Dict], sort_key: str, order: str) -> List[Dict]:
"""Sort data by sort_key and order"""

View File

@@ -6,7 +6,7 @@ from abc import ABC, abstractmethod
from ..utils.utils import calculate_relative_path_for_model, remove_empty_dirs
from ..utils.constants import AUTO_ORGANIZE_BATCH_SIZE
from ..services.settings_manager import settings
from ..services.settings_manager import get_settings_manager
logger = logging.getLogger(__name__)
@@ -114,7 +114,8 @@ class ModelFileService:
raise ValueError('No model roots configured')
# Check if flat structure is configured for this model type
path_template = settings.get_download_path_template(self.model_type)
settings_manager = get_settings_manager()
path_template = settings_manager.get_download_path_template(self.model_type)
result.is_flat_structure = not path_template
# Initialize tracking

View File

@@ -1,8 +1,11 @@
from abc import ABC, abstractmethod
import asyncio
import json
import logging
from typing import Optional, Dict, Tuple, Any
import random
from typing import Optional, Dict, Tuple, Any, List, Sequence
from .downloader import get_downloader
from .errors import RateLimitError
try:
from bs4 import BeautifulSoup
@@ -61,6 +64,11 @@ class ModelMetadataProvider(ABC):
"""Fetch model version metadata"""
pass
@abstractmethod
async def get_user_models(self, username: str) -> Optional[List[Dict]]:
"""Fetch models owned by the specified user"""
pass
class CivitaiModelMetadataProvider(ModelMetadataProvider):
"""Provider that uses Civitai API for metadata"""
@@ -79,123 +87,30 @@ class CivitaiModelMetadataProvider(ModelMetadataProvider):
async def get_model_version_info(self, version_id: str) -> Tuple[Optional[Dict], Optional[str]]:
return await self.client.get_model_version_info(version_id)
async def get_user_models(self, username: str) -> Optional[List[Dict]]:
return await self.client.get_user_models(username)
class CivArchiveModelMetadataProvider(ModelMetadataProvider):
"""Provider that uses CivArchive HTML page parsing for metadata"""
"""Provider that uses CivArchive API for metadata"""
def __init__(self, civarchive_client):
self.client = civarchive_client
async def get_model_by_hash(self, model_hash: str) -> Tuple[Optional[Dict], Optional[str]]:
"""Not supported by CivArchive provider"""
return None, "CivArchive provider does not support hash lookup"
return await self.client.get_model_by_hash(model_hash)
async def get_model_versions(self, model_id: str) -> Optional[Dict]:
"""Not supported by CivArchive provider"""
return None
return await self.client.get_model_versions(model_id)
async def get_model_version(self, model_id: int = None, version_id: int = None) -> Optional[Dict]:
"""Get specific model version by parsing CivArchive HTML page"""
if model_id is None or version_id is None:
return None
try:
# Construct CivArchive URL
url = f"https://civarchive.com/models/{model_id}?modelVersionId={version_id}"
downloader = await get_downloader()
session = await downloader.session
async with session.get(url) as response:
if response.status != 200:
return None
html_content = await response.text()
# Parse HTML to extract JSON data
soup_parser = _require_beautifulsoup()
soup = soup_parser(html_content, 'html.parser')
script_tag = soup.find('script', {'id': '__NEXT_DATA__', 'type': 'application/json'})
if not script_tag:
return None
# Parse JSON content
json_data = json.loads(script_tag.string)
model_data = json_data.get('props', {}).get('pageProps', {}).get('model')
if not model_data or 'version' not in model_data:
return None
# Extract version data as base
version = model_data['version'].copy()
# Restructure stats
if 'downloadCount' in version and 'ratingCount' in version and 'rating' in version:
version['stats'] = {
'downloadCount': version.pop('downloadCount'),
'ratingCount': version.pop('ratingCount'),
'rating': version.pop('rating')
}
# Rename trigger to trainedWords
if 'trigger' in version:
version['trainedWords'] = version.pop('trigger')
# Transform files data to expected format
if 'files' in version:
transformed_files = []
for file_data in version['files']:
# Find first available mirror (deletedAt is null)
available_mirror = None
for mirror in file_data.get('mirrors', []):
if mirror.get('deletedAt') is None:
available_mirror = mirror
break
# Create transformed file entry
transformed_file = {
'id': file_data.get('id'),
'sizeKB': file_data.get('sizeKB'),
'name': available_mirror.get('filename', file_data.get('name')) if available_mirror else file_data.get('name'),
'type': file_data.get('type'),
'downloadUrl': available_mirror.get('url') if available_mirror else None,
'primary': True,
'mirrors': file_data.get('mirrors', [])
}
# Transform hash format
if 'sha256' in file_data:
transformed_file['hashes'] = {
'SHA256': file_data['sha256'].upper()
}
transformed_files.append(transformed_file)
version['files'] = transformed_files
# Add model information
version['model'] = {
'name': model_data.get('name'),
'type': model_data.get('type'),
'nsfw': model_data.get('is_nsfw', False),
'description': model_data.get('description'),
'tags': model_data.get('tags', [])
}
version['creator'] = {
'username': model_data.get('username'),
'image': ''
}
# Add source identifier
version['source'] = 'civarchive'
version['is_deleted'] = json_data.get('query', {}).get('is_deleted', False)
return version
except Exception as e:
logger.error(f"Error fetching CivArchive model version {model_id}/{version_id}: {e}")
return None
return await self.client.get_model_version(model_id, version_id)
async def get_model_version_info(self, version_id: str) -> Tuple[Optional[Dict], Optional[str]]:
"""Not supported by CivArchive provider - requires both model_id and version_id"""
return None, "CivArchive provider requires both model_id and version_id"
return await self.client.get_model_version_info(version_id)
async def get_user_models(self, username: str) -> Optional[List[Dict]]:
"""Not supported by CivArchive provider"""
return None
class SQLiteModelMetadataProvider(ModelMetadataProvider):
"""Provider that uses SQLite database for metadata"""
@@ -329,20 +244,24 @@ class SQLiteModelMetadataProvider(ModelMetadataProvider):
"""Fetch model version metadata from SQLite database"""
async with self._aiosqlite.connect(self.db_path) as db:
db.row_factory = self._aiosqlite.Row
# Get version details
version_query = "SELECT model_id FROM model_versions WHERE id = ?"
cursor = await db.execute(version_query, (version_id,))
version_row = await cursor.fetchone()
if not version_row:
return None, "Model version not found"
model_id = version_row['model_id']
# Build complete version data with model info
version_data = await self._get_version_with_model_data(db, model_id, version_id)
return version_data, None
async def get_user_models(self, username: str) -> Optional[List[Dict]]:
"""Listing models by username is not supported for archive database"""
return None
async def _get_version_with_model_data(self, db, model_id, version_id) -> Optional[Dict]:
"""Helper to build version data with model information"""
@@ -389,59 +308,211 @@ class SQLiteModelMetadataProvider(ModelMetadataProvider):
# Add any additional fields from version data
result.update(version_data)
# Attach files associated with this version from model_files table
files_query = """
SELECT data
FROM model_files
WHERE version_id = ? AND type = 'Model'
ORDER BY id ASC
"""
cursor = await db.execute(files_query, (version_id,))
file_rows = await cursor.fetchall()
files = []
for file_row in file_rows:
try:
file_data = json.loads(file_row['data'])
except json.JSONDecodeError:
logger.warning(
"Skipping model_files entry with invalid JSON for version_id %s", version_id
)
continue
# Remove 'modelId' and 'modelVersionId' fields if present
file_data.pop('modelId', None)
file_data.pop('modelVersionId', None)
files.append(file_data)
if 'files' in result:
existing_files = result['files']
if isinstance(existing_files, list):
existing_files.extend(files)
result['files'] = existing_files
else:
merged_files = files.copy()
if existing_files:
merged_files.insert(0, existing_files)
result['files'] = merged_files
elif files:
result['files'] = files
else:
result['files'] = []
return result
except json.JSONDecodeError:
return None
class FallbackMetadataProvider(ModelMetadataProvider):
"""Try providers in order, return first successful result."""
def __init__(self, providers: list):
self.providers = providers
def __init__(
self,
providers: Sequence[ModelMetadataProvider | Tuple[str, ModelMetadataProvider]],
*,
rate_limit_retry_limit: int = 3,
rate_limit_base_delay: float = 1.5,
rate_limit_max_delay: float = 30.0,
rate_limit_jitter_ratio: float = 0.2,
) -> None:
self.providers: List[ModelMetadataProvider] = []
self._provider_labels: List[str] = []
for entry in providers:
if isinstance(entry, tuple) and len(entry) == 2:
name, provider = entry
else:
provider = entry
name = provider.__class__.__name__
self.providers.append(provider)
self._provider_labels.append(str(name))
self._rate_limit_retry_limit = max(1, rate_limit_retry_limit)
self._rate_limit_base_delay = rate_limit_base_delay
self._rate_limit_max_delay = rate_limit_max_delay
self._rate_limit_jitter_ratio = max(0.0, rate_limit_jitter_ratio)
async def get_model_by_hash(self, model_hash: str) -> Tuple[Optional[Dict], Optional[str]]:
for provider in self.providers:
for provider, label in self._iter_providers():
try:
result, error = await provider.get_model_by_hash(model_hash)
result, error = await self._call_with_rate_limit(
label,
provider.get_model_by_hash,
model_hash,
)
if result:
return result, error
except RateLimitError as exc:
exc.provider = exc.provider or label
raise exc
except Exception as e:
logger.debug(f"Provider failed for get_model_by_hash: {e}")
logger.debug("Provider %s failed for get_model_by_hash: %s", label, e)
continue
return None, "Model not found"
async def get_model_versions(self, model_id: str) -> Optional[Dict]:
for provider in self.providers:
for provider, label in self._iter_providers():
try:
result = await provider.get_model_versions(model_id)
result = await self._call_with_rate_limit(
label,
provider.get_model_versions,
model_id,
)
if result:
return result
except RateLimitError as exc:
exc.provider = exc.provider or label
raise exc
except Exception as e:
logger.debug(f"Provider failed for get_model_versions: {e}")
logger.debug("Provider %s failed for get_model_versions: %s", label, e)
continue
return None
async def get_model_version(self, model_id: int = None, version_id: int = None) -> Optional[Dict]:
for provider in self.providers:
for provider, label in self._iter_providers():
try:
result = await provider.get_model_version(model_id, version_id)
result = await self._call_with_rate_limit(
label,
provider.get_model_version,
model_id,
version_id,
)
if result:
return result
except RateLimitError as exc:
exc.provider = exc.provider or label
raise exc
except Exception as e:
logger.debug(f"Provider failed for get_model_version: {e}")
logger.debug("Provider %s failed for get_model_version: %s", label, e)
continue
return None
async def get_model_version_info(self, version_id: str) -> Tuple[Optional[Dict], Optional[str]]:
for provider in self.providers:
for provider, label in self._iter_providers():
try:
result, error = await provider.get_model_version_info(version_id)
result, error = await self._call_with_rate_limit(
label,
provider.get_model_version_info,
version_id,
)
if result:
return result, error
except RateLimitError as exc:
exc.provider = exc.provider or label
raise exc
except Exception as e:
logger.debug(f"Provider failed for get_model_version_info: {e}")
logger.debug("Provider %s failed for get_model_version_info: %s", label, e)
continue
return None, "No provider could retrieve the data"
async def get_user_models(self, username: str) -> Optional[List[Dict]]:
for provider, label in self._iter_providers():
try:
result = await self._call_with_rate_limit(
label,
provider.get_user_models,
username,
)
if result is not None:
return result
except RateLimitError as exc:
exc.provider = exc.provider or label
raise exc
except Exception as e:
logger.debug("Provider %s failed for get_user_models: %s", label, e)
continue
return None
def _iter_providers(self):
return zip(self.providers, self._provider_labels)
async def _call_with_rate_limit(
self,
label: str,
func,
*args,
**kwargs,
):
attempt = 0
while True:
try:
return await func(*args, **kwargs)
except RateLimitError as exc:
attempt += 1
if attempt >= self._rate_limit_retry_limit:
exc.provider = exc.provider or label
raise exc
delay = self._calculate_rate_limit_delay(exc.retry_after, attempt)
logger.warning(
"Provider %s rate limited request; retrying in %.2fs (attempt %s/%s)",
label,
delay,
attempt,
self._rate_limit_retry_limit,
)
await asyncio.sleep(delay)
except Exception:
raise
def _calculate_rate_limit_delay(self, retry_after: Optional[float], attempt: int) -> float:
if retry_after is not None:
return min(self._rate_limit_max_delay, max(0.0, retry_after))
base_delay = self._rate_limit_base_delay * (2 ** max(0, attempt - 1))
jitter_span = base_delay * self._rate_limit_jitter_ratio
if jitter_span > 0:
base_delay += random.uniform(-jitter_span, jitter_span)
return min(self._rate_limit_max_delay, max(0.0, base_delay))
class ModelMetadataProviderManager:
"""Manager for selecting and using model metadata providers"""
@@ -483,6 +554,11 @@ class ModelMetadataProviderManager:
"""Fetch model version info using specified or default provider"""
provider = self._get_provider(provider_name)
return await provider.get_model_version_info(version_id)
async def get_user_models(self, username: str, provider_name: str = None) -> Optional[List[Dict]]:
"""Fetch models owned by the specified user"""
provider = self._get_provider(provider_name)
return await provider.get_user_models(username)
def _get_provider(self, provider_name: str = None) -> ModelMetadataProvider:
"""Get provider by name or default provider"""

View File

@@ -4,7 +4,8 @@ import logging
import asyncio
import time
import shutil
from typing import List, Dict, Optional, Type, Set
from dataclasses import dataclass
from typing import Any, Awaitable, Callable, Dict, List, Mapping, Optional, Set, Type, Union
from ..utils.models import BaseModelMetadata
from ..config import config
@@ -16,9 +17,20 @@ from ..utils.constants import PREVIEW_EXTENSIONS
from .model_lifecycle_service import delete_model_artifacts
from .service_registry import ServiceRegistry
from .websocket_manager import ws_manager
from .persistent_model_cache import get_persistent_cache
logger = logging.getLogger(__name__)
@dataclass
class CacheBuildResult:
"""Represents the outcome of scanning model files for cache building."""
raw_data: List[Dict]
hash_index: ModelHashIndex
tags_count: Dict[str, int]
excluded_models: List[str]
class ModelScanner:
"""Base service for scanning and managing model files"""
@@ -68,16 +80,130 @@ class ModelScanner:
self._tags_count = {} # Dictionary to store tag counts
self._is_initializing = False # Flag to track initialization state
self._excluded_models = [] # List to track excluded models
self._persistent_cache = get_persistent_cache()
self._initialized = True
# Register this service
asyncio.create_task(self._register_service())
def on_library_changed(self) -> None:
"""Reset caches when the active library changes."""
self._persistent_cache = get_persistent_cache()
self._cache = None
self._hash_index = ModelHashIndex()
self._tags_count = {}
self._excluded_models = []
self._is_initializing = False
try:
loop = asyncio.get_running_loop()
except RuntimeError:
loop = None
if loop and not loop.is_closed():
loop.create_task(self.initialize_in_background())
async def _register_service(self):
"""Register this instance with the ServiceRegistry"""
service_name = f"{self.model_type}_scanner"
await ServiceRegistry.register_service(service_name, self)
def _slim_civitai_payload(self, civitai: Optional[Mapping[str, Any]]) -> Optional[Dict[str, Any]]:
"""Return a lightweight civitai payload containing only frequently used keys."""
if not isinstance(civitai, Mapping) or not civitai:
return None
slim: Dict[str, Any] = {}
for key in ('id', 'modelId', 'name'):
value = civitai.get(key)
if value not in (None, '', []):
slim[key] = value
creator = civitai.get('creator')
if isinstance(creator, Mapping):
username = creator.get('username')
if username:
slim['creator'] = {'username': username}
trained_words = civitai.get('trainedWords')
if trained_words:
slim['trainedWords'] = list(trained_words) if isinstance(trained_words, list) else trained_words
return slim or None
def _build_cache_entry(
self,
source: Union[BaseModelMetadata, Mapping[str, Any]],
*,
folder: Optional[str] = None,
file_path_override: Optional[str] = None
) -> Dict[str, Any]:
"""Project metadata into the lightweight cache representation."""
is_mapping = isinstance(source, Mapping)
def get_value(key: str, default: Any = None) -> Any:
if is_mapping:
return source.get(key, default)
return getattr(source, key, default)
file_path = file_path_override or get_value('file_path', '') or ''
normalized_path = file_path.replace('\\', '/')
folder_value = folder if folder is not None else get_value('folder', '') or ''
normalized_folder = folder_value.replace('\\', '/')
tags_value = get_value('tags') or []
if isinstance(tags_value, list):
tags_list = list(tags_value)
elif isinstance(tags_value, (set, tuple)):
tags_list = list(tags_value)
else:
tags_list = []
preview_url = get_value('preview_url', '') or ''
if isinstance(preview_url, str):
preview_url = preview_url.replace('\\', '/')
else:
preview_url = ''
civitai_slim = self._slim_civitai_payload(get_value('civitai'))
usage_tips = get_value('usage_tips', '') or ''
if not isinstance(usage_tips, str):
usage_tips = str(usage_tips)
notes = get_value('notes', '') or ''
if not isinstance(notes, str):
notes = str(notes)
entry: Dict[str, Any] = {
'file_path': normalized_path,
'file_name': get_value('file_name', '') or '',
'model_name': get_value('model_name', '') or '',
'folder': normalized_folder,
'size': int(get_value('size', 0) or 0),
'modified': float(get_value('modified', 0.0) or 0.0),
'sha256': (get_value('sha256', '') or '').lower(),
'base_model': get_value('base_model', '') or '',
'preview_url': preview_url,
'preview_nsfw_level': int(get_value('preview_nsfw_level', 0) or 0),
'from_civitai': bool(get_value('from_civitai', True)),
'favorite': bool(get_value('favorite', False)),
'notes': notes,
'usage_tips': usage_tips,
'metadata_source': get_value('metadata_source', None),
'exclude': bool(get_value('exclude', False)),
'db_checked': bool(get_value('db_checked', False)),
'last_checked_at': float(get_value('last_checked_at', 0.0) or 0.0),
'tags': tags_list,
'civitai': civitai_slim,
'civitai_deleted': bool(get_value('civitai_deleted', False)),
}
model_type = get_value('model_type', None)
if model_type:
entry['model_type'] = model_type
return entry
async def initialize_in_background(self) -> None:
"""Initialize cache in background using thread pool"""
try:
@@ -92,7 +218,12 @@ class ModelScanner:
self._is_initializing = True
# Determine the page type based on model type
page_type = 'loras' if self.model_type == 'lora' else 'checkpoints'
page_type_map = {
'lora': 'loras',
'checkpoint': 'checkpoints',
'embedding': 'embeddings'
}
page_type = page_type_map.get(self.model_type, self.model_type)
# First, try to load from cache
await ws_manager.broadcast_init_progress({
@@ -102,8 +233,25 @@ class ModelScanner:
'scanner_type': self.model_type,
'pageType': page_type
})
# If cache loading failed, proceed with full scan
cache_loaded = await self._load_persisted_cache(page_type)
if cache_loaded:
await asyncio.sleep(0) # Yield control so the UI can process the cache hydration update
await ws_manager.broadcast_init_progress({
'stage': 'finalizing',
'progress': 100,
'status': 'complete',
'details': f"Loaded {len(self._cache.raw_data)} cached {self.model_type} files from disk.",
'scanner_type': self.model_type,
'pageType': page_type
})
logger.info(
f"{self.model_type.capitalize()} cache hydrated from persisted snapshot with {len(self._cache.raw_data)} models"
)
return
# Persistent load failed; fall back to a full scan
await ws_manager.broadcast_init_progress({
'stage': 'scan_folders',
'progress': 0,
@@ -130,13 +278,17 @@ class ModelScanner:
start_time = time.time()
# Use thread pool to execute CPU-intensive operations with progress reporting
await loop.run_in_executor(
scan_result: Optional[CacheBuildResult] = await loop.run_in_executor(
None, # Use default thread pool
self._initialize_cache_sync, # Run synchronous version in thread
total_files, # Pass the total file count for progress reporting
page_type # Pass the page type for progress reporting
)
if scan_result:
await self._apply_scan_result(scan_result)
await self._save_persistent_cache(scan_result)
# Send final progress update
await ws_manager.broadcast_init_progress({
'stage': 'finalizing',
@@ -165,6 +317,105 @@ class ModelScanner:
# Always clear the initializing flag when done
self._is_initializing = False
async def _load_persisted_cache(self, page_type: str) -> bool:
"""Attempt to hydrate the in-memory cache from the SQLite snapshot."""
if not getattr(self, '_persistent_cache', None):
return False
loop = asyncio.get_event_loop()
try:
persisted = await loop.run_in_executor(
None,
self._persistent_cache.load_cache,
self.model_type
)
except FileNotFoundError:
return False
except Exception as exc:
logger.debug("%s Scanner: Could not load persisted cache: %s", self.model_type.capitalize(), exc)
return False
if not persisted or not persisted.raw_data:
return False
hash_index = ModelHashIndex()
for sha_value, path in persisted.hash_rows:
if sha_value and path:
hash_index.add_entry(sha_value.lower(), path)
tags_count: Dict[str, int] = {}
for item in persisted.raw_data:
for tag in item.get('tags') or []:
tags_count[tag] = tags_count.get(tag, 0) + 1
scan_result = CacheBuildResult(
raw_data=list(persisted.raw_data),
hash_index=hash_index,
tags_count=tags_count,
excluded_models=list(persisted.excluded_models)
)
await self._apply_scan_result(scan_result)
await ws_manager.broadcast_init_progress({
'stage': 'loading_cache',
'progress': 1,
'details': f"Loaded cached {self.model_type} data from disk",
'scanner_type': self.model_type,
'pageType': page_type
})
return True
async def _save_persistent_cache(self, scan_result: CacheBuildResult) -> None:
if not scan_result or not getattr(self, '_persistent_cache', None):
return
hash_snapshot = self._build_hash_index_snapshot(scan_result.hash_index)
loop = asyncio.get_event_loop()
try:
await loop.run_in_executor(
None,
self._persistent_cache.save_cache,
self.model_type,
list(scan_result.raw_data),
hash_snapshot,
list(scan_result.excluded_models)
)
except Exception as exc:
logger.warning("%s Scanner: Failed to persist cache: %s", self.model_type.capitalize(), exc)
def _build_hash_index_snapshot(self, hash_index: Optional[ModelHashIndex]) -> Dict[str, List[str]]:
snapshot: Dict[str, List[str]] = {}
if not hash_index:
return snapshot
for sha_value, path in getattr(hash_index, '_hash_to_path', {}).items():
if not sha_value or not path:
continue
bucket = snapshot.setdefault(sha_value.lower(), [])
if path not in bucket:
bucket.append(path)
for sha_value, paths in getattr(hash_index, '_duplicate_hashes', {}).items():
if not sha_value:
continue
bucket = snapshot.setdefault(sha_value.lower(), [])
for path in paths:
if path and path not in bucket:
bucket.append(path)
return snapshot
async def _persist_current_cache(self) -> None:
if self._cache is None or not getattr(self, '_persistent_cache', None):
return
snapshot = CacheBuildResult(
raw_data=list(self._cache.raw_data),
hash_index=self._hash_index,
tags_count=dict(self._tags_count),
excluded_models=list(self._excluded_models)
)
await self._save_persistent_cache(snapshot)
def _count_model_files(self) -> int:
"""Count all model files with supported extensions in all roots
@@ -204,124 +455,53 @@ class ModelScanner:
return total_files
def _initialize_cache_sync(self, total_files=0, page_type='loras'):
def _initialize_cache_sync(self, total_files: int = 0, page_type: str = 'loras') -> Optional[CacheBuildResult]:
"""Synchronous version of cache initialization for thread pool execution"""
loop = asyncio.new_event_loop()
try:
# Create a new event loop for this thread
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
# Create a synchronous method to bypass the async lock
def sync_initialize_cache():
# Track progress
processed_files = 0
last_progress_time = time.time()
last_progress_percent = 0
# We need a wrapper around scan_all_models to track progress
# This is a local function that will run in our thread's event loop
async def scan_with_progress():
nonlocal processed_files, last_progress_time, last_progress_percent
# For storing raw model data
all_models = []
# Process each model root
for root_path in self.get_model_roots():
if not os.path.exists(root_path):
continue
# Track visited paths to avoid symlink loops
visited_paths = set()
# Recursively process directory
async def scan_dir_with_progress(path):
nonlocal processed_files, last_progress_time, last_progress_percent
try:
real_path = os.path.realpath(path)
if real_path in visited_paths:
return
visited_paths.add(real_path)
with os.scandir(path) as it:
entries = list(it)
for entry in entries:
try:
if entry.is_file(follow_symlinks=True):
ext = os.path.splitext(entry.name)[1].lower()
if ext in self.file_extensions:
file_path = entry.path.replace(os.sep, "/")
result = await self._process_model_file(file_path, root_path)
if result:
all_models.append(result)
# Update progress counter
processed_files += 1
# Update progress periodically (not every file to avoid excessive updates)
current_time = time.time()
if total_files > 0 and (current_time - last_progress_time > 0.5 or processed_files == total_files):
# Adjusted progress calculation
progress_percent = min(99, int(1 + (processed_files / total_files) * 98))
if progress_percent > last_progress_percent:
last_progress_percent = progress_percent
last_progress_time = current_time
# Send progress update through websocket
await ws_manager.broadcast_init_progress({
'stage': 'process_models',
'progress': progress_percent,
'details': f"Processing {self.model_type} files: {processed_files}/{total_files}",
'scanner_type': self.model_type,
'pageType': page_type
})
elif entry.is_dir(follow_symlinks=True):
await scan_dir_with_progress(entry.path)
except Exception as e:
logger.error(f"Error processing entry {entry.path}: {e}")
except Exception as e:
logger.error(f"Error scanning {path}: {e}")
# Process the root path
await scan_dir_with_progress(root_path)
return all_models
# Run the progress-tracking scan function
raw_data = loop.run_until_complete(scan_with_progress())
# Update hash index and tags count
for model_data in raw_data:
if 'sha256' in model_data and 'file_path' in model_data:
self._hash_index.add_entry(model_data['sha256'].lower(), model_data['file_path'])
# Count tags
if 'tags' in model_data and model_data['tags']:
for tag in model_data['tags']:
self._tags_count[tag] = self._tags_count.get(tag, 0) + 1
# Log duplicate filename warnings after building the index
# duplicate_filenames = self._hash_index.get_duplicate_filenames()
# if duplicate_filenames:
# logger.warning(f"Found {len(duplicate_filenames)} filename(s) with duplicates during {self.model_type} cache build:")
# for filename, paths in duplicate_filenames.items():
# logger.warning(f" Duplicate filename '{filename}': {paths}")
# Update cache
self._cache.raw_data = raw_data
loop.run_until_complete(self._cache.resort())
return self._cache
# Run our sync initialization that avoids lock conflicts
return sync_initialize_cache()
last_progress_time = time.time()
last_progress_percent = 0
async def progress_callback(processed_files: int, expected_total: int) -> None:
nonlocal last_progress_time, last_progress_percent
if expected_total <= 0:
return
current_time = time.time()
progress_percent = min(99, int(1 + (processed_files / expected_total) * 98))
if progress_percent <= last_progress_percent:
return
if current_time - last_progress_time <= 0.5 and processed_files != expected_total:
return
last_progress_percent = progress_percent
last_progress_time = current_time
await ws_manager.broadcast_init_progress({
'stage': 'process_models',
'progress': progress_percent,
'details': f"Processing {self.model_type} files: {processed_files}/{expected_total}",
'scanner_type': self.model_type,
'pageType': page_type
})
return loop.run_until_complete(
self._gather_model_data(
total_files=total_files,
progress_callback=progress_callback
)
)
except Exception as e:
logger.error(f"Error in thread-based {self.model_type} cache initialization: {e}")
return None
finally:
# Clean up the event loop
asyncio.set_event_loop(None)
loop.close()
async def get_cached_data(self, force_refresh: bool = False, rebuild_cache: bool = False) -> ModelCache:
@@ -353,45 +533,16 @@ class ModelScanner:
self._is_initializing = True # Set flag
try:
start_time = time.time()
# Clear existing hash index
self._hash_index.clear()
# Clear existing tags count
self._tags_count = {}
# Determine the page type based on model type
page_type = 'loras' if self.model_type == 'lora' else 'checkpoints'
# Scan for new data
raw_data = await self.scan_all_models()
# Build hash index and tags count
for model_data in raw_data:
if 'sha256' in model_data and 'file_path' in model_data:
self._hash_index.add_entry(model_data['sha256'].lower(), model_data['file_path'])
# Count tags
if 'tags' in model_data and model_data['tags']:
for tag in model_data['tags']:
self._tags_count[tag] = self._tags_count.get(tag, 0) + 1
# Log duplicate filename warnings after building the index
# duplicate_filenames = self._hash_index.get_duplicate_filenames()
# if duplicate_filenames:
# logger.warning(f"Found {len(duplicate_filenames)} filename(s) with duplicates during {self.model_type} cache build:")
# for filename, paths in duplicate_filenames.items():
# logger.warning(f" Duplicate filename '{filename}': {paths}")
# Update cache
self._cache = ModelCache(
raw_data=raw_data,
folders=[]
)
# Resort cache
await self._cache.resort()
scan_result = await self._gather_model_data()
await self._apply_scan_result(scan_result)
await self._save_persistent_cache(scan_result)
logger.info(f"{self.model_type.capitalize()} Scanner: Cache initialization completed in {time.time() - start_time:.2f} seconds, found {len(raw_data)} models")
logger.info(
f"{self.model_type.capitalize()} Scanner: Cache initialization completed in {time.time() - start_time:.2f} seconds, "
f"found {len(scan_result.raw_data)} models"
)
except Exception as e:
logger.error(f"{self.model_type.capitalize()} Scanner: Error initializing cache: {e}")
# Ensure cache is at least an empty structure on error
@@ -473,11 +624,16 @@ class ModelScanner:
for i in range(0, len(new_files), batch_size):
batch = new_files[i:i+batch_size]
for path in batch:
logger.info(f"{self.model_type.capitalize()} Scanner: Processing {path}")
try:
# Find the appropriate root path for this file
root_path = None
for potential_root in self.get_model_roots():
if path.startswith(potential_root):
model_roots = self.get_model_roots()
for potential_root in model_roots:
# Normalize both paths for comparison
normalized_path = os.path.normpath(path)
normalized_root = os.path.normpath(potential_root)
if normalized_path.startswith(normalized_root):
root_path = potential_root
break
@@ -486,7 +642,8 @@ class ModelScanner:
if model_data:
# Add to cache
self._cache.raw_data.append(model_data)
self._cache.add_to_version_index(model_data)
# Update hash index if available
if 'sha256' in model_data and 'file_path' in model_data:
self._hash_index.add_entry(model_data['sha256'].lower(), model_data['file_path'])
@@ -513,7 +670,9 @@ class ModelScanner:
for path in missing_files:
try:
model_to_remove = path_to_item[path]
self._cache.remove_from_version_index(model_to_remove)
# Update tags count
for tag in model_to_remove.get('tags', []):
if tag in self._tags_count:
@@ -535,70 +694,19 @@ class ModelScanner:
# Update folders list
all_folders = set(item.get('folder', '') for item in self._cache.raw_data)
self._cache.folders = sorted(list(all_folders), key=lambda x: x.lower())
self._cache.rebuild_version_index()
# Resort cache
await self._cache.resort()
await self._persist_current_cache()
logger.info(f"{self.model_type.capitalize()} Scanner: Cache reconciliation completed in {time.time() - start_time:.2f} seconds. Added {total_added}, removed {total_removed} models.")
except Exception as e:
logger.error(f"{self.model_type.capitalize()} Scanner: Error reconciling cache: {e}", exc_info=True)
finally:
self._is_initializing = False # Unset flag
async def scan_all_models(self) -> List[Dict]:
"""Scan all model directories and return metadata"""
all_models = []
# Create scan tasks for each directory
scan_tasks = []
for model_root in self.get_model_roots():
task = asyncio.create_task(self._scan_directory(model_root))
scan_tasks.append(task)
# Wait for all tasks to complete
for task in scan_tasks:
try:
models = await task
all_models.extend(models)
except Exception as e:
logger.error(f"Error scanning directory: {e}")
return all_models
async def _scan_directory(self, root_path: str) -> List[Dict]:
"""Scan a single directory for model files"""
models = []
original_root = root_path # Save original root path
async def scan_recursive(path: str, visited_paths: set):
"""Recursively scan directory, avoiding circular symlinks"""
try:
real_path = os.path.realpath(path)
if real_path in visited_paths:
logger.debug(f"Skipping already visited path: {path}")
return
visited_paths.add(real_path)
with os.scandir(path) as it:
entries = list(it)
for entry in entries:
try:
if entry.is_file(follow_symlinks=True) and any(entry.name.endswith(ext) for ext in self.file_extensions):
file_path = entry.path.replace(os.sep, "/")
result = await self._process_model_file(file_path, original_root)
# Only add to models if result is not None (skip corrupted metadata)
if result:
models.append(result)
await asyncio.sleep(0)
elif entry.is_dir(follow_symlinks=True):
await scan_recursive(entry.path, visited_paths)
except Exception as e:
logger.error(f"Error processing entry {entry.path}: {e}")
except Exception as e:
logger.error(f"Error scanning {path}: {e}")
await scan_recursive(root_path, set())
return models
def is_initializing(self) -> bool:
"""Check if the scanner is currently initializing"""
@@ -624,8 +732,18 @@ class ModelScanner:
"""Hook for subclasses: adjust metadata during scanning"""
return metadata
async def _process_model_file(self, file_path: str, root_path: str) -> Dict:
async def _process_model_file(
self,
file_path: str,
root_path: str,
*,
hash_index: Optional[ModelHashIndex] = None,
excluded_models: Optional[List[str]] = None
) -> Dict:
"""Process a single model file and return its metadata"""
hash_index = hash_index or self._hash_index
excluded_models = excluded_models if excluded_models is not None else self._excluded_models
metadata, should_skip = await MetadataManager.load_metadata(file_path, self.model_class)
if should_skip:
@@ -685,30 +803,136 @@ class ModelScanner:
# Hook: allow subclasses to adjust metadata
metadata = self.adjust_metadata(metadata, file_path, root_path)
model_data = metadata.to_dict()
# Skip excluded models
if model_data.get('exclude', False):
self._excluded_models.append(model_data['file_path'])
return None
# Check for duplicate filename before adding to hash index
filename = os.path.splitext(os.path.basename(file_path))[0]
existing_hash = self._hash_index.get_hash_by_filename(filename)
if existing_hash and existing_hash != model_data.get('sha256', '').lower():
existing_path = self._hash_index.get_path(existing_hash)
if existing_path and existing_path != file_path:
logger.warning(f"Duplicate filename detected: '{filename}' - files: '{existing_path}' and '{file_path}'")
rel_path = os.path.relpath(file_path, root_path)
folder = os.path.dirname(rel_path)
model_data['folder'] = folder.replace(os.path.sep, '/')
normalized_folder = folder.replace(os.path.sep, '/')
model_data = self._build_cache_entry(metadata, folder=normalized_folder)
# Skip excluded models
if model_data.get('exclude', False):
excluded_models.append(model_data['file_path'])
return None
# Check for duplicate filename before adding to hash index
# filename = os.path.splitext(os.path.basename(file_path))[0]
# existing_hash = hash_index.get_hash_by_filename(filename)
# if existing_hash and existing_hash != model_data.get('sha256', '').lower():
# existing_path = hash_index.get_path(existing_hash)
# if existing_path and existing_path != file_path:
# logger.warning(f"Duplicate filename detected: '{filename}' - files: '{existing_path}' and '{file_path}'")
return model_data
async def _apply_scan_result(self, scan_result: CacheBuildResult) -> None:
"""Apply scan results to the cache and associated indexes."""
if scan_result is None:
return
self._hash_index = scan_result.hash_index
self._tags_count = dict(scan_result.tags_count)
self._excluded_models = list(scan_result.excluded_models)
if self._cache is None:
self._cache = ModelCache(
raw_data=list(scan_result.raw_data),
folders=[]
)
else:
self._cache.raw_data = list(scan_result.raw_data)
self._cache.rebuild_version_index()
await self._cache.resort()
async def _gather_model_data(
self,
*,
total_files: int = 0,
progress_callback: Optional[Callable[[int, int], Awaitable[None]]] = None
) -> CacheBuildResult:
"""Collect metadata for all model files."""
raw_data: List[Dict] = []
hash_index = ModelHashIndex()
tags_count: Dict[str, int] = {}
excluded_models: List[str] = []
processed_files = 0
async def handle_progress() -> None:
if progress_callback is None:
return
try:
await progress_callback(processed_files, total_files)
except Exception as exc: # pragma: no cover - defensive logging
logger.error(f"Error reporting progress for {self.model_type}: {exc}")
async def scan_recursive(current_path: str, root_path: str, visited_paths: Set[str]) -> None:
nonlocal processed_files
try:
real_path = os.path.realpath(current_path)
if real_path in visited_paths:
return
visited_paths.add(real_path)
with os.scandir(current_path) as iterator:
entries = list(iterator)
for entry in entries:
try:
if entry.is_file(follow_symlinks=True):
ext = os.path.splitext(entry.name)[1].lower()
if ext not in self.file_extensions:
continue
file_path = entry.path.replace(os.sep, "/")
result = await self._process_model_file(
file_path,
root_path,
hash_index=hash_index,
excluded_models=excluded_models
)
processed_files += 1
if result:
raw_data.append(result)
sha_value = result.get('sha256')
model_path = result.get('file_path')
if sha_value and model_path:
hash_index.add_entry(sha_value.lower(), model_path)
for tag in result.get('tags') or []:
tags_count[tag] = tags_count.get(tag, 0) + 1
await handle_progress()
await asyncio.sleep(0)
elif entry.is_dir(follow_symlinks=True):
await scan_recursive(entry.path, root_path, visited_paths)
except Exception as entry_error:
logger.error(f"Error processing entry {entry.path}: {entry_error}")
except Exception as scan_error:
logger.error(f"Error scanning {current_path}: {scan_error}")
for model_root in self.get_model_roots():
if not os.path.exists(model_root):
continue
await scan_recursive(model_root, model_root, set())
return CacheBuildResult(
raw_data=raw_data,
hash_index=hash_index,
tags_count=tags_count,
excluded_models=excluded_models
)
async def add_model_to_cache(self, metadata_dict: Dict, folder: str = '') -> bool:
"""Add a model to the cache
Args:
metadata_dict: The model metadata dictionary
folder: The relative folder path for the model
@@ -725,7 +949,8 @@ class ModelScanner:
# Add to cache
self._cache.raw_data.append(metadata_dict)
self._cache.add_to_version_index(metadata_dict)
# Resort cache data
await self._cache.resort()
@@ -736,6 +961,7 @@ class ModelScanner:
# Update the hash index
self._hash_index.add_entry(metadata_dict['sha256'], metadata_dict['file_path'])
await self._persist_current_cache()
return True
except Exception as e:
logger.error(f"Error adding model to cache: {e}")
@@ -864,8 +1090,11 @@ class ModelScanner:
async def update_single_model_cache(self, original_path: str, new_path: str, metadata: Dict) -> bool:
"""Update cache after a model has been moved or modified"""
cache = await self.get_cached_data()
existing_item = next((item for item in cache.raw_data if item['file_path'] == original_path), None)
if existing_item:
cache.remove_from_version_index(existing_item)
if existing_item and 'tags' in existing_item:
for tag in existing_item.get('tags', []):
if tag in self._tags_count:
@@ -876,35 +1105,45 @@ class ModelScanner:
self._hash_index.remove_by_path(original_path)
cache.raw_data = [
item for item in cache.raw_data
item for item in cache.raw_data
if item['file_path'] != original_path
]
cache_modified = bool(existing_item) or bool(metadata)
if metadata:
if original_path == new_path:
existing_folder = next((item['folder'] for item in cache.raw_data
if item['file_path'] == original_path), None)
if existing_folder:
metadata['folder'] = existing_folder
else:
metadata['folder'] = self._calculate_folder(new_path)
normalized_new_path = new_path.replace(os.sep, '/')
if original_path == new_path and existing_item:
folder_value = existing_item.get('folder', self._calculate_folder(new_path))
else:
metadata['folder'] = self._calculate_folder(new_path)
cache.raw_data.append(metadata)
if 'sha256' in metadata:
self._hash_index.add_entry(metadata['sha256'].lower(), new_path)
folder_value = self._calculate_folder(new_path)
cache_entry = self._build_cache_entry(
metadata,
folder=folder_value,
file_path_override=normalized_new_path,
)
cache.raw_data.append(cache_entry)
cache.add_to_version_index(cache_entry)
sha_value = cache_entry.get('sha256')
if sha_value:
self._hash_index.add_entry(sha_value.lower(), normalized_new_path)
all_folders = set(item['folder'] for item in cache.raw_data)
cache.folders = sorted(list(all_folders), key=lambda x: x.lower())
if 'tags' in metadata:
for tag in metadata.get('tags', []):
self._tags_count[tag] = self._tags_count.get(tag, 0) + 1
for tag in cache_entry.get('tags', []):
self._tags_count[tag] = self._tags_count.get(tag, 0) + 1
cache.rebuild_version_index()
await cache.resort()
if cache_modified:
await self._persist_current_cache()
return True
def has_hash(self, sha256: str) -> bool:
@@ -1006,7 +1245,10 @@ class ModelScanner:
if self._cache is None:
return False
return await self._cache.update_preview_url(file_path, preview_url, preview_nsfw_level)
updated = await self._cache.update_preview_url(file_path, preview_url, preview_nsfw_level)
if updated:
await self._persist_current_cache()
return updated
async def bulk_delete_models(self, file_paths: List[str]) -> Dict:
"""Delete multiple models and update cache in a batch operation
@@ -1119,11 +1361,12 @@ class ModelScanner:
# Update hash index
for model in models_to_remove:
file_path = model['file_path']
self._cache.remove_from_version_index(model)
if hasattr(self, '_hash_index') and self._hash_index:
# Get the hash and filename before removal for duplicate checking
file_name = os.path.splitext(os.path.basename(file_path))[0]
hash_val = model.get('sha256', '').lower()
# Remove from hash index
self._hash_index.remove_by_path(file_path, hash_val)
@@ -1132,10 +1375,13 @@ class ModelScanner:
# Update cache data
self._cache.raw_data = [item for item in self._cache.raw_data if item['file_path'] not in file_paths]
# Resort cache
self._cache.rebuild_version_index()
await self._cache.resort()
await self._persist_current_cache()
return True
except Exception as e:
@@ -1171,16 +1417,17 @@ class ModelScanner:
Returns:
bool: True if the model version exists, False otherwise
"""
try:
normalized_id = int(model_version_id)
except (TypeError, ValueError):
return False
try:
cache = await self.get_cached_data()
if not cache or not cache.raw_data:
if not cache:
return False
for item in cache.raw_data:
if item.get('civitai') and item['civitai'].get('id') == model_version_id:
return True
return False
return normalized_id in cache.version_index
except Exception as e:
logger.error(f"Error checking model version existence: {e}")
return False

View File

@@ -0,0 +1,411 @@
"""Service for tracking remote model version updates."""
from __future__ import annotations
import asyncio
import json
import logging
import os
import sqlite3
import time
from dataclasses import dataclass
from typing import Dict, Iterable, List, Mapping, Optional, Sequence
from .errors import RateLimitError
logger = logging.getLogger(__name__)
@dataclass
class ModelUpdateRecord:
"""Representation of a persisted update record."""
model_type: str
model_id: int
largest_version_id: Optional[int]
version_ids: List[int]
in_library_version_ids: List[int]
last_checked_at: Optional[float]
should_ignore: bool
def has_update(self) -> bool:
"""Return True when remote versions exceed the local library."""
if self.should_ignore or not self.version_ids:
return False
local_versions = set(self.in_library_version_ids)
return any(version_id not in local_versions for version_id in self.version_ids)
class ModelUpdateService:
"""Persist and query remote model version metadata."""
_SCHEMA = """
CREATE TABLE IF NOT EXISTS model_update_status (
model_type TEXT NOT NULL,
model_id INTEGER NOT NULL,
largest_version_id INTEGER,
version_ids TEXT,
in_library_version_ids TEXT,
last_checked_at REAL,
should_ignore INTEGER DEFAULT 0,
PRIMARY KEY (model_type, model_id)
)
"""
def __init__(self, db_path: str, *, ttl_seconds: int = 24 * 60 * 60) -> None:
self._db_path = db_path
self._ttl_seconds = ttl_seconds
self._lock = asyncio.Lock()
self._schema_initialized = False
self._ensure_directory()
self._initialize_schema()
def _ensure_directory(self) -> None:
directory = os.path.dirname(self._db_path)
if directory:
os.makedirs(directory, exist_ok=True)
def _connect(self) -> sqlite3.Connection:
conn = sqlite3.connect(self._db_path, check_same_thread=False)
conn.row_factory = sqlite3.Row
return conn
def _initialize_schema(self) -> None:
if self._schema_initialized:
return
try:
with self._connect() as conn:
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA foreign_keys = ON")
conn.executescript(self._SCHEMA)
self._schema_initialized = True
except Exception as exc: # pragma: no cover - defensive guard
logger.error("Failed to initialize update schema: %s", exc, exc_info=True)
raise
async def refresh_for_model_type(
self,
model_type: str,
scanner,
metadata_provider,
*,
force_refresh: bool = False,
) -> Dict[int, ModelUpdateRecord]:
"""Refresh update information for every model present in the cache."""
local_versions = await self._collect_local_versions(scanner)
results: Dict[int, ModelUpdateRecord] = {}
for model_id, version_ids in local_versions.items():
record = await self._refresh_single_model(
model_type,
model_id,
version_ids,
metadata_provider,
force_refresh=force_refresh,
)
if record:
results[model_id] = record
return results
async def refresh_single_model(
self,
model_type: str,
model_id: int,
scanner,
metadata_provider,
*,
force_refresh: bool = False,
) -> Optional[ModelUpdateRecord]:
"""Refresh update information for a specific model id."""
local_versions = await self._collect_local_versions(scanner)
version_ids = local_versions.get(model_id, [])
return await self._refresh_single_model(
model_type,
model_id,
version_ids,
metadata_provider,
force_refresh=force_refresh,
)
async def update_in_library_versions(
self,
model_type: str,
model_id: int,
version_ids: Sequence[int],
) -> ModelUpdateRecord:
"""Persist a new set of in-library version identifiers."""
normalized_versions = self._normalize_sequence(version_ids)
async with self._lock:
existing = self._get_record(model_type, model_id)
record = ModelUpdateRecord(
model_type=model_type,
model_id=model_id,
largest_version_id=existing.largest_version_id if existing else None,
version_ids=list(existing.version_ids) if existing else [],
in_library_version_ids=normalized_versions,
last_checked_at=existing.last_checked_at if existing else None,
should_ignore=existing.should_ignore if existing else False,
)
self._upsert_record(record)
return record
async def set_should_ignore(
self, model_type: str, model_id: int, should_ignore: bool
) -> ModelUpdateRecord:
"""Toggle the ignore flag for a model."""
async with self._lock:
existing = self._get_record(model_type, model_id)
if existing:
record = ModelUpdateRecord(
model_type=model_type,
model_id=model_id,
largest_version_id=existing.largest_version_id,
version_ids=list(existing.version_ids),
in_library_version_ids=list(existing.in_library_version_ids),
last_checked_at=existing.last_checked_at,
should_ignore=should_ignore,
)
else:
record = ModelUpdateRecord(
model_type=model_type,
model_id=model_id,
largest_version_id=None,
version_ids=[],
in_library_version_ids=[],
last_checked_at=None,
should_ignore=should_ignore,
)
self._upsert_record(record)
return record
async def get_record(self, model_type: str, model_id: int) -> Optional[ModelUpdateRecord]:
"""Return a cached record without triggering remote fetches."""
async with self._lock:
return self._get_record(model_type, model_id)
async def has_update(self, model_type: str, model_id: int) -> bool:
"""Determine if a model has updates pending."""
record = await self.get_record(model_type, model_id)
return record.has_update() if record else False
async def _refresh_single_model(
self,
model_type: str,
model_id: int,
local_versions: Sequence[int],
metadata_provider,
*,
force_refresh: bool = False,
) -> Optional[ModelUpdateRecord]:
normalized_local = self._normalize_sequence(local_versions)
now = time.time()
async with self._lock:
existing = self._get_record(model_type, model_id)
if existing and existing.should_ignore and not force_refresh:
record = ModelUpdateRecord(
model_type=model_type,
model_id=model_id,
largest_version_id=existing.largest_version_id,
version_ids=list(existing.version_ids),
in_library_version_ids=normalized_local,
last_checked_at=existing.last_checked_at,
should_ignore=True,
)
self._upsert_record(record)
return record
should_fetch = force_refresh or not existing or self._is_stale(existing, now)
# release lock during network request
fetched_versions: List[int] | None = None
refresh_succeeded = False
if metadata_provider and should_fetch:
try:
response = await metadata_provider.get_model_versions(model_id)
except RateLimitError:
raise
except Exception as exc: # pragma: no cover - defensive log
logger.error(
"Failed to fetch versions for model %s (%s): %s",
model_id,
model_type,
exc,
exc_info=True,
)
else:
if response is not None:
extracted = self._extract_version_ids(response)
if extracted is not None:
fetched_versions = extracted
refresh_succeeded = True
async with self._lock:
existing = self._get_record(model_type, model_id)
if existing and existing.should_ignore and not force_refresh:
# Ignore state could have flipped while awaiting provider
record = ModelUpdateRecord(
model_type=model_type,
model_id=model_id,
largest_version_id=existing.largest_version_id,
version_ids=list(existing.version_ids),
in_library_version_ids=normalized_local,
last_checked_at=existing.last_checked_at,
should_ignore=True,
)
self._upsert_record(record)
return record
version_ids = (
fetched_versions
if refresh_succeeded
else (list(existing.version_ids) if existing else [])
)
largest = max(version_ids) if version_ids else None
last_checked = now if refresh_succeeded else (
existing.last_checked_at if existing else None
)
record = ModelUpdateRecord(
model_type=model_type,
model_id=model_id,
largest_version_id=largest,
version_ids=version_ids,
in_library_version_ids=normalized_local,
last_checked_at=last_checked,
should_ignore=existing.should_ignore if existing else False,
)
self._upsert_record(record)
return record
async def _collect_local_versions(self, scanner) -> Dict[int, List[int]]:
cache = await scanner.get_cached_data()
mapping: Dict[int, set[int]] = {}
if not cache or not getattr(cache, "raw_data", None):
return {}
for item in cache.raw_data:
civitai = item.get("civitai") if isinstance(item, dict) else None
if not isinstance(civitai, dict):
continue
model_id = self._normalize_int(civitai.get("modelId"))
version_id = self._normalize_int(civitai.get("id"))
if model_id is None or version_id is None:
continue
mapping.setdefault(model_id, set()).add(version_id)
return {model_id: sorted(ids) for model_id, ids in mapping.items()}
def _is_stale(self, record: ModelUpdateRecord, now: float) -> bool:
if record.last_checked_at is None:
return True
return (now - record.last_checked_at) >= self._ttl_seconds
@staticmethod
def _normalize_int(value) -> Optional[int]:
try:
if value is None:
return None
return int(value)
except (TypeError, ValueError):
return None
def _normalize_sequence(self, values: Sequence[int]) -> List[int]:
normalized = [
item
for item in (self._normalize_int(value) for value in values)
if item is not None
]
return sorted(dict.fromkeys(normalized))
def _extract_version_ids(self, response) -> Optional[List[int]]:
if not isinstance(response, Mapping):
return None
versions = response.get("modelVersions")
if versions is None:
return []
if not isinstance(versions, Iterable):
return None
normalized = []
for entry in versions:
if isinstance(entry, Mapping):
normalized_id = self._normalize_int(entry.get("id"))
else:
normalized_id = self._normalize_int(entry)
if normalized_id is not None:
normalized.append(normalized_id)
return sorted(dict.fromkeys(normalized))
def _get_record(self, model_type: str, model_id: int) -> Optional[ModelUpdateRecord]:
with self._connect() as conn:
row = conn.execute(
"""
SELECT model_type, model_id, largest_version_id, version_ids,
in_library_version_ids, last_checked_at, should_ignore
FROM model_update_status
WHERE model_type = ? AND model_id = ?
""",
(model_type, model_id),
).fetchone()
if not row:
return None
return ModelUpdateRecord(
model_type=row["model_type"],
model_id=int(row["model_id"]),
largest_version_id=self._normalize_int(row["largest_version_id"]),
version_ids=self._deserialize_json_array(row["version_ids"]),
in_library_version_ids=self._deserialize_json_array(
row["in_library_version_ids"]
),
last_checked_at=row["last_checked_at"],
should_ignore=bool(row["should_ignore"]),
)
def _upsert_record(self, record: ModelUpdateRecord) -> None:
payload = (
record.model_type,
record.model_id,
record.largest_version_id,
json.dumps(record.version_ids),
json.dumps(record.in_library_version_ids),
record.last_checked_at,
1 if record.should_ignore else 0,
)
with self._connect() as conn:
conn.execute(
"""
INSERT INTO model_update_status (
model_type, model_id, largest_version_id, version_ids,
in_library_version_ids, last_checked_at, should_ignore
) VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(model_type, model_id) DO UPDATE SET
largest_version_id = excluded.largest_version_id,
version_ids = excluded.version_ids,
in_library_version_ids = excluded.in_library_version_ids,
last_checked_at = excluded.last_checked_at,
should_ignore = excluded.should_ignore
""",
payload,
)
conn.commit()
@staticmethod
def _deserialize_json_array(value) -> List[int]:
if not value:
return []
try:
data = json.loads(value)
except (TypeError, json.JSONDecodeError):
return []
if isinstance(data, list):
normalized = []
for entry in data:
try:
normalized.append(int(entry))
except (TypeError, ValueError):
continue
return sorted(dict.fromkeys(normalized))
return []

View File

@@ -0,0 +1,569 @@
import json
import logging
import os
import re
import sqlite3
import threading
from dataclasses import dataclass
from typing import Dict, List, Optional, Sequence, Tuple
from ..utils.settings_paths import get_settings_dir
logger = logging.getLogger(__name__)
@dataclass
class PersistedCacheData:
"""Lightweight structure returned by the persistent cache."""
raw_data: List[Dict]
hash_rows: List[Tuple[str, str]]
excluded_models: List[str]
class PersistentModelCache:
"""Persist core model metadata and hash index data in SQLite."""
_DEFAULT_FILENAME = "model_cache.sqlite"
_MODEL_COLUMNS: Tuple[str, ...] = (
"model_type",
"file_path",
"file_name",
"model_name",
"folder",
"size",
"modified",
"sha256",
"base_model",
"preview_url",
"preview_nsfw_level",
"from_civitai",
"favorite",
"notes",
"usage_tips",
"metadata_source",
"civitai_id",
"civitai_model_id",
"civitai_name",
"civitai_creator_username",
"trained_words",
"civitai_deleted",
"exclude",
"db_checked",
"last_checked_at",
)
_MODEL_UPDATE_COLUMNS: Tuple[str, ...] = _MODEL_COLUMNS[2:]
_instances: Dict[str, "PersistentModelCache"] = {}
_instance_lock = threading.Lock()
def __init__(self, library_name: str = "default", db_path: Optional[str] = None) -> None:
self._library_name = library_name or "default"
self._db_path = db_path or self._resolve_default_path(self._library_name)
self._db_lock = threading.Lock()
self._schema_initialized = False
try:
directory = os.path.dirname(self._db_path)
if directory:
os.makedirs(directory, exist_ok=True)
except Exception as exc: # pragma: no cover - defensive guard
logger.warning("Could not create cache directory %s: %s", directory, exc)
if self.is_enabled():
self._initialize_schema()
@classmethod
def get_default(cls, library_name: Optional[str] = None) -> "PersistentModelCache":
name = (library_name or "default")
with cls._instance_lock:
if name not in cls._instances:
cls._instances[name] = cls(name)
return cls._instances[name]
def is_enabled(self) -> bool:
return os.environ.get("LORA_MANAGER_DISABLE_PERSISTENT_CACHE", "0") != "1"
def get_database_path(self) -> str:
"""Expose the resolved SQLite database path."""
return self._db_path
def load_cache(self, model_type: str) -> Optional[PersistedCacheData]:
if not self.is_enabled():
return None
if not self._schema_initialized:
self._initialize_schema()
if not self._schema_initialized:
return None
try:
with self._db_lock:
conn = self._connect(readonly=True)
try:
model_columns_sql = ", ".join(self._MODEL_COLUMNS[1:])
rows = conn.execute(
f"SELECT {model_columns_sql} FROM models WHERE model_type = ?",
(model_type,),
).fetchall()
if not rows:
return None
tags = self._load_tags(conn, model_type)
hash_rows = conn.execute(
"SELECT sha256, file_path FROM hash_index WHERE model_type = ?",
(model_type,),
).fetchall()
excluded = conn.execute(
"SELECT file_path FROM excluded_models WHERE model_type = ?",
(model_type,),
).fetchall()
finally:
conn.close()
except Exception as exc:
logger.warning("Failed to load persisted cache for %s: %s", model_type, exc)
return None
raw_data: List[Dict] = []
for row in rows:
file_path: str = row["file_path"]
trained_words = []
if row["trained_words"]:
try:
trained_words = json.loads(row["trained_words"])
except json.JSONDecodeError:
trained_words = []
creator_username = row["civitai_creator_username"]
civitai: Optional[Dict] = None
civitai_has_data = any(
row[col] is not None for col in ("civitai_id", "civitai_model_id", "civitai_name")
) or trained_words or creator_username
if civitai_has_data:
civitai = {}
if row["civitai_id"] is not None:
civitai["id"] = row["civitai_id"]
if row["civitai_model_id"] is not None:
civitai["modelId"] = row["civitai_model_id"]
if row["civitai_name"]:
civitai["name"] = row["civitai_name"]
if trained_words:
civitai["trainedWords"] = trained_words
if creator_username:
civitai.setdefault("creator", {})["username"] = creator_username
item = {
"file_path": file_path,
"file_name": row["file_name"],
"model_name": row["model_name"],
"folder": row["folder"] or "",
"size": row["size"] or 0,
"modified": row["modified"] or 0.0,
"sha256": row["sha256"] or "",
"base_model": row["base_model"] or "",
"preview_url": row["preview_url"] or "",
"preview_nsfw_level": row["preview_nsfw_level"] or 0,
"from_civitai": bool(row["from_civitai"]),
"favorite": bool(row["favorite"]),
"notes": row["notes"] or "",
"usage_tips": row["usage_tips"] or "",
"metadata_source": row["metadata_source"] or None,
"exclude": bool(row["exclude"]),
"db_checked": bool(row["db_checked"]),
"last_checked_at": row["last_checked_at"] or 0.0,
"tags": tags.get(file_path, []),
"civitai": civitai,
"civitai_deleted": bool(row["civitai_deleted"]),
}
raw_data.append(item)
hash_pairs = [(entry["sha256"].lower(), entry["file_path"]) for entry in hash_rows if entry["sha256"]]
if not hash_pairs:
# Fall back to hashes stored on the model rows
for item in raw_data:
sha_value = item.get("sha256")
if sha_value:
hash_pairs.append((sha_value.lower(), item["file_path"]))
excluded_paths = [row["file_path"] for row in excluded]
return PersistedCacheData(raw_data=raw_data, hash_rows=hash_pairs, excluded_models=excluded_paths)
def save_cache(self, model_type: str, raw_data: Sequence[Dict], hash_index: Dict[str, List[str]], excluded_models: Sequence[str]) -> None:
if not self.is_enabled():
return
if not self._schema_initialized:
self._initialize_schema()
if not self._schema_initialized:
return
try:
with self._db_lock:
conn = self._connect()
try:
conn.execute("PRAGMA foreign_keys = ON")
conn.execute("BEGIN")
model_rows = [self._prepare_model_row(model_type, item) for item in raw_data]
model_map: Dict[str, Tuple] = {
row[1]: row for row in model_rows if row[1] # row[1] is file_path
}
existing_models = conn.execute(
"SELECT "
+ ", ".join(self._MODEL_COLUMNS[1:])
+ " FROM models WHERE model_type = ?",
(model_type,),
).fetchall()
existing_model_map: Dict[str, sqlite3.Row] = {
row["file_path"]: row for row in existing_models
}
to_remove_models = [
(model_type, path)
for path in existing_model_map.keys()
if path not in model_map
]
if to_remove_models:
conn.executemany(
"DELETE FROM models WHERE model_type = ? AND file_path = ?",
to_remove_models,
)
conn.executemany(
"DELETE FROM model_tags WHERE model_type = ? AND file_path = ?",
to_remove_models,
)
conn.executemany(
"DELETE FROM hash_index WHERE model_type = ? AND file_path = ?",
to_remove_models,
)
conn.executemany(
"DELETE FROM excluded_models WHERE model_type = ? AND file_path = ?",
to_remove_models,
)
insert_rows: List[Tuple] = []
update_rows: List[Tuple] = []
for file_path, row in model_map.items():
existing = existing_model_map.get(file_path)
if existing is None:
insert_rows.append(row)
continue
existing_values = tuple(
existing[column] for column in self._MODEL_COLUMNS[1:]
)
current_values = row[1:]
if existing_values != current_values:
update_rows.append(row[2:] + (model_type, file_path))
if insert_rows:
conn.executemany(self._insert_model_sql(), insert_rows)
if update_rows:
set_clause = ", ".join(
f"{column} = ?"
for column in self._MODEL_UPDATE_COLUMNS
)
update_sql = (
f"UPDATE models SET {set_clause} WHERE model_type = ? AND file_path = ?"
)
conn.executemany(update_sql, update_rows)
existing_tags_rows = conn.execute(
"SELECT file_path, tag FROM model_tags WHERE model_type = ?",
(model_type,),
).fetchall()
existing_tags: Dict[str, set] = {}
for row in existing_tags_rows:
existing_tags.setdefault(row["file_path"], set()).add(row["tag"])
new_tags: Dict[str, set] = {}
for item in raw_data:
file_path = item.get("file_path")
if not file_path:
continue
tags = set(item.get("tags") or [])
if tags:
new_tags[file_path] = tags
tag_inserts: List[Tuple[str, str, str]] = []
tag_deletes: List[Tuple[str, str, str]] = []
all_tag_paths = set(existing_tags.keys()) | set(new_tags.keys())
for path in all_tag_paths:
existing_set = existing_tags.get(path, set())
new_set = new_tags.get(path, set())
to_add = new_set - existing_set
to_remove = existing_set - new_set
for tag in to_add:
tag_inserts.append((model_type, path, tag))
for tag in to_remove:
tag_deletes.append((model_type, path, tag))
if tag_deletes:
conn.executemany(
"DELETE FROM model_tags WHERE model_type = ? AND file_path = ? AND tag = ?",
tag_deletes,
)
if tag_inserts:
conn.executemany(
"INSERT INTO model_tags (model_type, file_path, tag) VALUES (?, ?, ?)",
tag_inserts,
)
existing_hash_rows = conn.execute(
"SELECT sha256, file_path FROM hash_index WHERE model_type = ?",
(model_type,),
).fetchall()
existing_hash_map: Dict[str, set] = {}
for row in existing_hash_rows:
sha_value = (row["sha256"] or "").lower()
if not sha_value:
continue
existing_hash_map.setdefault(sha_value, set()).add(row["file_path"])
new_hash_map: Dict[str, set] = {}
for sha_value, paths in hash_index.items():
normalized_sha = (sha_value or "").lower()
if not normalized_sha:
continue
bucket = new_hash_map.setdefault(normalized_sha, set())
for path in paths:
if path:
bucket.add(path)
hash_inserts: List[Tuple[str, str, str]] = []
hash_deletes: List[Tuple[str, str, str]] = []
all_shas = set(existing_hash_map.keys()) | set(new_hash_map.keys())
for sha_value in all_shas:
existing_paths = existing_hash_map.get(sha_value, set())
new_paths = new_hash_map.get(sha_value, set())
for path in existing_paths - new_paths:
hash_deletes.append((model_type, sha_value, path))
for path in new_paths - existing_paths:
hash_inserts.append((model_type, sha_value, path))
if hash_deletes:
conn.executemany(
"DELETE FROM hash_index WHERE model_type = ? AND sha256 = ? AND file_path = ?",
hash_deletes,
)
if hash_inserts:
conn.executemany(
"INSERT OR IGNORE INTO hash_index (model_type, sha256, file_path) VALUES (?, ?, ?)",
hash_inserts,
)
existing_excluded_rows = conn.execute(
"SELECT file_path FROM excluded_models WHERE model_type = ?",
(model_type,),
).fetchall()
existing_excluded = {row["file_path"] for row in existing_excluded_rows}
new_excluded = {path for path in excluded_models if path}
excluded_deletes = [
(model_type, path)
for path in existing_excluded - new_excluded
]
excluded_inserts = [
(model_type, path)
for path in new_excluded - existing_excluded
]
if excluded_deletes:
conn.executemany(
"DELETE FROM excluded_models WHERE model_type = ? AND file_path = ?",
excluded_deletes,
)
if excluded_inserts:
conn.executemany(
"INSERT OR IGNORE INTO excluded_models (model_type, file_path) VALUES (?, ?)",
excluded_inserts,
)
conn.commit()
finally:
conn.close()
except Exception as exc:
logger.warning("Failed to persist cache for %s: %s", model_type, exc)
# Internal helpers -------------------------------------------------
def _resolve_default_path(self, library_name: str) -> str:
override = os.environ.get("LORA_MANAGER_CACHE_DB")
if override:
return override
try:
settings_dir = get_settings_dir(create=True)
except Exception as exc: # pragma: no cover - defensive guard
logger.warning("Falling back to project directory for cache: %s", exc)
settings_dir = os.path.dirname(os.path.dirname(self._db_path)) if hasattr(self, "_db_path") else os.getcwd()
safe_name = re.sub(r"[^A-Za-z0-9_.-]", "_", library_name or "default")
if safe_name.lower() in ("default", ""):
legacy_path = os.path.join(settings_dir, self._DEFAULT_FILENAME)
if os.path.exists(legacy_path):
return legacy_path
return os.path.join(settings_dir, "model_cache", f"{safe_name}.sqlite")
def _initialize_schema(self) -> None:
with self._db_lock:
if self._schema_initialized:
return
try:
with self._connect() as conn:
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA foreign_keys = ON")
conn.executescript(
"""
CREATE TABLE IF NOT EXISTS models (
model_type TEXT NOT NULL,
file_path TEXT NOT NULL,
file_name TEXT,
model_name TEXT,
folder TEXT,
size INTEGER,
modified REAL,
sha256 TEXT,
base_model TEXT,
preview_url TEXT,
preview_nsfw_level INTEGER,
from_civitai INTEGER,
favorite INTEGER,
notes TEXT,
usage_tips TEXT,
metadata_source TEXT,
civitai_id INTEGER,
civitai_model_id INTEGER,
civitai_name TEXT,
civitai_creator_username TEXT,
trained_words TEXT,
civitai_deleted INTEGER,
exclude INTEGER,
db_checked INTEGER,
last_checked_at REAL,
PRIMARY KEY (model_type, file_path)
);
CREATE TABLE IF NOT EXISTS model_tags (
model_type TEXT NOT NULL,
file_path TEXT NOT NULL,
tag TEXT NOT NULL,
PRIMARY KEY (model_type, file_path, tag)
);
CREATE TABLE IF NOT EXISTS hash_index (
model_type TEXT NOT NULL,
sha256 TEXT NOT NULL,
file_path TEXT NOT NULL,
PRIMARY KEY (model_type, sha256, file_path)
);
CREATE TABLE IF NOT EXISTS excluded_models (
model_type TEXT NOT NULL,
file_path TEXT NOT NULL,
PRIMARY KEY (model_type, file_path)
);
"""
)
self._ensure_additional_model_columns(conn)
conn.commit()
self._schema_initialized = True
except Exception as exc: # pragma: no cover - defensive guard
logger.warning("Failed to initialize persistent cache schema: %s", exc)
def _ensure_additional_model_columns(self, conn: sqlite3.Connection) -> None:
try:
existing_columns = {
row["name"]
for row in conn.execute("PRAGMA table_info(models)").fetchall()
}
except Exception: # pragma: no cover - defensive guard
return
required_columns = {
"metadata_source": "TEXT",
"civitai_creator_username": "TEXT",
"civitai_deleted": "INTEGER DEFAULT 0",
}
for column, definition in required_columns.items():
if column not in existing_columns:
conn.execute(f"ALTER TABLE models ADD COLUMN {column} {definition}")
def _connect(self, readonly: bool = False) -> sqlite3.Connection:
uri = False
path = self._db_path
if readonly:
if not os.path.exists(path):
raise FileNotFoundError(path)
path = f"file:{path}?mode=ro"
uri = True
conn = sqlite3.connect(path, check_same_thread=False, uri=uri, detect_types=sqlite3.PARSE_DECLTYPES)
conn.row_factory = sqlite3.Row
return conn
def _prepare_model_row(self, model_type: str, item: Dict) -> Tuple:
civitai = item.get("civitai") or {}
trained_words = civitai.get("trainedWords")
if isinstance(trained_words, str):
trained_words_json = trained_words
elif trained_words is None:
trained_words_json = None
else:
trained_words_json = json.dumps(trained_words)
metadata_source = item.get("metadata_source") or None
creator_username = None
creator_data = civitai.get("creator") if isinstance(civitai, dict) else None
if isinstance(creator_data, dict):
creator_username = creator_data.get("username") or None
return (
model_type,
item.get("file_path"),
item.get("file_name"),
item.get("model_name"),
item.get("folder"),
int(item.get("size") or 0),
float(item.get("modified") or 0.0),
(item.get("sha256") or "").lower() or None,
item.get("base_model"),
item.get("preview_url"),
int(item.get("preview_nsfw_level") or 0),
1 if item.get("from_civitai", True) else 0,
1 if item.get("favorite") else 0,
item.get("notes"),
item.get("usage_tips"),
metadata_source,
civitai.get("id"),
civitai.get("modelId"),
civitai.get("name"),
creator_username,
trained_words_json,
1 if item.get("civitai_deleted") else 0,
1 if item.get("exclude") else 0,
1 if item.get("db_checked") else 0,
float(item.get("last_checked_at") or 0.0),
)
def _insert_model_sql(self) -> str:
columns = ", ".join(self._MODEL_COLUMNS)
placeholders = ", ".join(["?"] * len(self._MODEL_COLUMNS))
return f"INSERT INTO models ({columns}) VALUES ({placeholders})"
def _load_tags(self, conn: sqlite3.Connection, model_type: str) -> Dict[str, List[str]]:
tag_rows = conn.execute(
"SELECT file_path, tag FROM model_tags WHERE model_type = ?",
(model_type,),
).fetchall()
result: Dict[str, List[str]] = {}
for row in tag_rows:
result.setdefault(row["file_path"], []).append(row["tag"])
return result
def get_persistent_cache() -> PersistentModelCache:
from .settings_manager import get_settings_manager # Local import to avoid cycles
library_name = get_settings_manager().get_active_library_name()
return PersistentModelCache.get_default(library_name)

View File

@@ -5,8 +5,10 @@ from __future__ import annotations
import logging
import os
from typing import Awaitable, Callable, Dict, Optional, Sequence
from urllib.parse import urlparse
from ..utils.constants import CARD_PREVIEW_WIDTH, PREVIEW_EXTENSIONS
from ..utils.civitai_utils import rewrite_preview_url
logger = logging.getLogger(__name__)
@@ -45,23 +47,59 @@ class PreviewAssetService:
base_name = os.path.splitext(os.path.splitext(os.path.basename(metadata_path))[0])[0]
preview_dir = os.path.dirname(metadata_path)
is_video = first_preview.get("type") == "video"
preview_url = first_preview.get("url")
if not preview_url:
return
def extension_from_url(url: str, fallback: str) -> str:
try:
parsed = urlparse(url)
except ValueError:
return fallback
ext = os.path.splitext(parsed.path)[1]
return ext or fallback
downloader = await self._downloader_factory()
if is_video:
extension = ".mp4"
extension = extension_from_url(preview_url, ".mp4")
preview_path = os.path.join(preview_dir, base_name + extension)
downloader = await self._downloader_factory()
success, result = await downloader.download_file(
first_preview["url"], preview_path, use_auth=False
)
if success:
local_metadata["preview_url"] = preview_path.replace(os.sep, "/")
local_metadata["preview_nsfw_level"] = first_preview.get("nsfwLevel", 0)
rewritten_url, rewritten = rewrite_preview_url(preview_url, media_type="video")
attempt_urls = []
if rewritten:
attempt_urls.append(rewritten_url)
attempt_urls.append(preview_url)
seen: set[str] = set()
for candidate in attempt_urls:
if not candidate or candidate in seen:
continue
seen.add(candidate)
success, _ = await downloader.download_file(candidate, preview_path, use_auth=False)
if success:
local_metadata["preview_url"] = preview_path.replace(os.sep, "/")
local_metadata["preview_nsfw_level"] = first_preview.get("nsfwLevel", 0)
return
else:
rewritten_url, rewritten = rewrite_preview_url(preview_url, media_type="image")
if rewritten:
extension = extension_from_url(preview_url, ".png")
preview_path = os.path.join(preview_dir, base_name + extension)
success, _ = await downloader.download_file(
rewritten_url, preview_path, use_auth=False
)
if success:
local_metadata["preview_url"] = preview_path.replace(os.sep, "/")
local_metadata["preview_nsfw_level"] = first_preview.get("nsfwLevel", 0)
return
extension = ".webp"
preview_path = os.path.join(preview_dir, base_name + extension)
downloader = await self._downloader_factory()
success, content, _headers = await downloader.download_to_memory(
first_preview["url"], use_auth=False
preview_url, use_auth=False
)
if not success:
return

View File

@@ -52,6 +52,31 @@ class RecipeScanner:
if lora_scanner:
self._lora_scanner = lora_scanner
self._initialized = True
def on_library_changed(self) -> None:
"""Reset cached state when the active library changes."""
# Cancel any in-flight initialization or resorting work so the next
# access rebuilds the cache for the new library.
if self._initialization_task and not self._initialization_task.done():
self._initialization_task.cancel()
for task in list(self._resort_tasks):
if not task.done():
task.cancel()
self._resort_tasks.clear()
self._cache = None
self._initialization_task = None
self._is_initializing = False
try:
loop = asyncio.get_running_loop()
except RuntimeError:
loop = None
if loop and not loop.is_closed():
loop.create_task(self.initialize_in_background())
async def _get_civitai_client(self):
"""Lazily initialize CivitaiClient from registry"""
@@ -424,27 +449,29 @@ class RecipeScanner:
# If has modelVersionId but no hash, look in lora cache first, then fetch from Civitai
if 'modelVersionId' in lora and not lora.get('hash'):
model_version_id = lora['modelVersionId']
# Check if model_version_id is an integer and > 0
if isinstance(model_version_id, int) and model_version_id > 0:
# Try to find in lora cache first
hash_from_cache = await self._find_hash_in_lora_cache(model_version_id)
if hash_from_cache:
lora['hash'] = hash_from_cache
metadata_updated = True
else:
# If not in cache, fetch from Civitai
result = await self._get_hash_from_civitai(model_version_id)
if isinstance(result, tuple):
hash_from_civitai, is_deleted = result
if hash_from_civitai:
lora['hash'] = hash_from_civitai
metadata_updated = True
elif is_deleted:
# Mark the lora as deleted if it was not found on Civitai
lora['isDeleted'] = True
logger.warning(f"Marked lora with modelVersionId {model_version_id} as deleted")
metadata_updated = True
# Try to find in lora cache first
hash_from_cache = await self._find_hash_in_lora_cache(model_version_id)
if hash_from_cache:
lora['hash'] = hash_from_cache
metadata_updated = True
else:
logger.debug(f"Could not get hash for modelVersionId {model_version_id}")
# If not in cache, fetch from Civitai
result = await self._get_hash_from_civitai(model_version_id)
if isinstance(result, tuple):
hash_from_civitai, is_deleted = result
if hash_from_civitai:
lora['hash'] = hash_from_civitai
metadata_updated = True
elif is_deleted:
# Mark the lora as deleted if it was not found on Civitai
lora['isDeleted'] = True
logger.warning(f"Marked lora with modelVersionId {model_version_id} as deleted")
metadata_updated = True
else:
logger.debug(f"Could not get hash for modelVersionId {model_version_id}")
# If has hash but no file_name, look up in lora library
if 'hash' in lora and (not lora.get('file_name') or not lora['file_name']):
@@ -740,20 +767,17 @@ class RecipeScanner:
"""Format file path as URL for serving in web UI"""
if not file_path:
return '/loras_static/images/no-preview.png'
try:
# Format file path as a URL that will work with static file serving
recipes_dir = os.path.join(config.loras_roots[0], "recipes").replace(os.sep, '/')
if file_path.replace(os.sep, '/').startswith(recipes_dir):
relative_path = os.path.relpath(file_path, config.loras_roots[0]).replace(os.sep, '/')
return f"/loras_static/root1/preview/{relative_path}"
# If not in recipes dir, try to create a valid URL from the file name
file_name = os.path.basename(file_path)
return f"/loras_static/root1/preview/recipes/{file_name}"
normalized_path = os.path.normpath(file_path)
static_url = config.get_preview_static_url(normalized_path)
if static_url:
return static_url
except Exception as e:
logger.error(f"Error formatting file URL: {e}")
return '/loras_static/images/no-preview.png'
return '/loras_static/images/no-preview.png'
def _format_timestamp(self, timestamp: float) -> str:
"""Format timestamp for display"""

View File

@@ -279,10 +279,17 @@ class RecipePersistenceService:
os.makedirs(recipes_dir, exist_ok=True)
recipe_id = str(uuid.uuid4())
image_filename = f"{recipe_id}.png"
optimized_image, extension = self._exif_utils.optimize_image(
image_data=image_bytes,
target_width=self._card_preview_width,
format="webp",
quality=85,
preserve_metadata=True,
)
image_filename = f"{recipe_id}{extension}"
image_path = os.path.join(recipes_dir, image_filename)
with open(image_path, "wb") as file_obj:
file_obj.write(image_bytes)
file_obj.write(optimized_image)
lora_stack = metadata.get("loras", "")
lora_matches = re.findall(r"<lora:([^:]+):([^>]+)>", lora_stack)
@@ -298,9 +305,9 @@ class RecipePersistenceService:
"file_name": name,
"strength": float(strength),
"hash": (lora_info.get("sha256") or "").lower() if lora_info else "",
"modelVersionId": lora_info.get("civitai", {}).get("id") if lora_info else 0,
"modelName": lora_info.get("civitai", {}).get("model", {}).get("name") if lora_info else "",
"modelVersionName": lora_info.get("civitai", {}).get("name") if lora_info else "",
"modelVersionId": (lora_info.get("civitai") or {}).get("id", 0) if lora_info else 0,
"modelName": ((lora_info.get("civitai") or {}).get("model") or {}).get("name", name) if lora_info else "",
"modelVersionName": (lora_info.get("civitai") or {}).get("name", "") if lora_info else "",
"isDeleted": False,
"exclude": False,
}

View File

@@ -61,7 +61,7 @@ class RecipeSharingService:
safe_title = recipe.get("title", "").replace(" ", "_").lower()
filename = f"recipe_{safe_title}{ext}" if safe_title else f"recipe_{recipe_id}{ext}"
url_path = f"/api/recipe/{recipe_id}/share/download?t={timestamp}"
url_path = f"/api/lm/recipe/{recipe_id}/share/download?t={timestamp}"
return SharingResult({"success": True, "download_url": url_path, "filename": filename})
async def prepare_download(self, *, recipe_scanner, recipe_id: str) -> DownloadInfo:

View File

@@ -128,6 +128,49 @@ class ServiceRegistry:
async def get_civitai_client(cls):
"""Get or create CivitAI client instance"""
service_name = "civitai_client"
if service_name in cls._services:
return cls._services[service_name]
async with cls._get_lock(service_name):
# Double-check after acquiring lock
if service_name in cls._services:
return cls._services[service_name]
# Import here to avoid circular imports
from .civitai_client import CivitaiClient
client = await CivitaiClient.get_instance()
cls._services[service_name] = client
logger.debug(f"Created and registered {service_name}")
return client
@classmethod
async def get_model_update_service(cls):
"""Get or create the model update tracking service."""
service_name = "model_update_service"
if service_name in cls._services:
return cls._services[service_name]
async with cls._get_lock(service_name):
if service_name in cls._services:
return cls._services[service_name]
from .model_update_service import ModelUpdateService
from .persistent_model_cache import get_persistent_cache
cache = get_persistent_cache()
service = ModelUpdateService(cache.get_database_path())
cls._services[service_name] = service
logger.debug(f"Created and registered {service_name}")
return service
@classmethod
async def get_civarchive_client(cls):
"""Get or create CivArchive client instance"""
service_name = "civarchive_client"
if service_name in cls._services:
return cls._services[service_name]
@@ -138,9 +181,9 @@ class ServiceRegistry:
return cls._services[service_name]
# Import here to avoid circular imports
from .civitai_client import CivitaiClient
from .civarchive_client import CivArchiveClient
client = await CivitaiClient.get_instance()
client = await CivArchiveClient.get_instance()
cls._services[service_name] = client
logger.debug(f"Created and registered {service_name}")
return client

View File

@@ -1,7 +1,19 @@
import os
import copy
import json
import os
import logging
from typing import Any, Dict
from datetime import datetime, timezone
from threading import Lock
from typing import Any, Dict, Iterable, List, Mapping, Optional, Sequence
from ..utils.constants import DEFAULT_PRIORITY_TAG_CONFIG
from ..utils.settings_paths import ensure_settings_file
from ..utils.tag_priorities import (
PriorityTagEntry,
collect_canonical_tags,
parse_priority_tag_string,
resolve_priority_tag,
)
logger = logging.getLogger(__name__)
@@ -31,15 +43,18 @@ DEFAULT_SETTINGS: Dict[str, Any] = {
"card_info_display": "always",
"include_trigger_words": False,
"compact_mode": False,
"priority_tags": DEFAULT_PRIORITY_TAG_CONFIG.copy(),
"model_name_display": "model_name",
}
class SettingsManager:
def __init__(self):
self.settings_file = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'settings.json')
self.settings_file = ensure_settings_file(logger)
self.settings = self._load_settings()
self._migrate_setting_keys()
self._ensure_default_settings()
self._migrate_to_library_registry()
self._migrate_download_path_template()
self._auto_set_default_roots()
self._check_environment_variables()
@@ -57,6 +72,12 @@ class SettingsManager:
def _ensure_default_settings(self) -> None:
"""Ensure all default settings keys exist"""
updated = False
normalized_priority = self._normalize_priority_tag_config(
self.settings.get("priority_tags")
)
if normalized_priority != self.settings.get("priority_tags"):
self.settings["priority_tags"] = normalized_priority
updated = True
for key, value in self._get_default_settings().items():
if key not in self.settings:
if isinstance(value, dict):
@@ -67,6 +88,223 @@ class SettingsManager:
if updated:
self._save_settings()
def _migrate_to_library_registry(self) -> None:
"""Ensure settings include the multi-library registry structure."""
libraries = self.settings.get("libraries")
active_name = self.settings.get("active_library")
if not isinstance(libraries, dict) or not libraries:
library_name = active_name or "default"
library_payload = self._build_library_payload(
folder_paths=self.settings.get("folder_paths", {}),
default_lora_root=self.settings.get("default_lora_root", ""),
default_checkpoint_root=self.settings.get("default_checkpoint_root", ""),
default_embedding_root=self.settings.get("default_embedding_root", ""),
)
libraries = {library_name: library_payload}
self.settings["libraries"] = libraries
self.settings["active_library"] = library_name
self._sync_active_library_to_root(save=False)
self._save_settings()
return
sanitized_libraries: Dict[str, Dict[str, Any]] = {}
changed = False
for name, data in libraries.items():
if not isinstance(data, dict):
data = {}
changed = True
payload = self._build_library_payload(
folder_paths=data.get("folder_paths"),
default_lora_root=data.get("default_lora_root"),
default_checkpoint_root=data.get("default_checkpoint_root"),
default_embedding_root=data.get("default_embedding_root"),
metadata=data.get("metadata"),
base=data,
)
sanitized_libraries[name] = payload
if payload is not data:
changed = True
if changed:
self.settings["libraries"] = sanitized_libraries
if not active_name or active_name not in sanitized_libraries:
if sanitized_libraries:
self.settings["active_library"] = next(iter(sanitized_libraries.keys()))
else:
self.settings["active_library"] = "default"
self._sync_active_library_to_root(save=changed)
def _sync_active_library_to_root(self, *, save: bool = False) -> None:
"""Update top-level folder path settings to mirror the active library."""
libraries = self.settings.get("libraries", {})
active_name = self.settings.get("active_library")
if not libraries:
return
if active_name not in libraries:
active_name = next(iter(libraries.keys()))
self.settings["active_library"] = active_name
active_library = libraries.get(active_name, {})
folder_paths = copy.deepcopy(active_library.get("folder_paths", {}))
self.settings["folder_paths"] = folder_paths
self.settings["default_lora_root"] = active_library.get("default_lora_root", "")
self.settings["default_checkpoint_root"] = active_library.get("default_checkpoint_root", "")
self.settings["default_embedding_root"] = active_library.get("default_embedding_root", "")
if save:
self._save_settings()
def _current_timestamp(self) -> str:
return datetime.now(timezone.utc).replace(microsecond=0).isoformat()
def _build_library_payload(
self,
*,
folder_paths: Optional[Mapping[str, Iterable[str]]] = None,
default_lora_root: Optional[str] = None,
default_checkpoint_root: Optional[str] = None,
default_embedding_root: Optional[str] = None,
metadata: Optional[Mapping[str, Any]] = None,
base: Optional[Mapping[str, Any]] = None,
) -> Dict[str, Any]:
payload: Dict[str, Any] = dict(base or {})
timestamp = self._current_timestamp()
if folder_paths is not None:
payload["folder_paths"] = self._normalize_folder_paths(folder_paths)
else:
payload.setdefault("folder_paths", {})
if default_lora_root is not None:
payload["default_lora_root"] = default_lora_root
else:
payload.setdefault("default_lora_root", "")
if default_checkpoint_root is not None:
payload["default_checkpoint_root"] = default_checkpoint_root
else:
payload.setdefault("default_checkpoint_root", "")
if default_embedding_root is not None:
payload["default_embedding_root"] = default_embedding_root
else:
payload.setdefault("default_embedding_root", "")
if metadata:
merged_meta = dict(payload.get("metadata", {}))
merged_meta.update(metadata)
payload["metadata"] = merged_meta
payload.setdefault("created_at", timestamp)
payload["updated_at"] = timestamp
return payload
def _normalize_folder_paths(
self, folder_paths: Mapping[str, Iterable[str]]
) -> Dict[str, List[str]]:
normalized: Dict[str, List[str]] = {}
for key, values in folder_paths.items():
if not isinstance(values, Iterable):
continue
cleaned: List[str] = []
seen = set()
for value in values:
if not isinstance(value, str):
continue
stripped = value.strip()
if not stripped:
continue
if stripped not in seen:
cleaned.append(stripped)
seen.add(stripped)
normalized[key] = cleaned
return normalized
def _validate_folder_paths(
self,
library_name: str,
folder_paths: Mapping[str, Iterable[str]],
) -> None:
"""Ensure folder paths do not overlap with other libraries."""
libraries = self.settings.get("libraries", {})
normalized_new: Dict[str, Dict[str, str]] = {}
for key, values in folder_paths.items():
path_map: Dict[str, str] = {}
for value in values:
if not isinstance(value, str):
continue
stripped = value.strip()
if not stripped:
continue
normalized_value = os.path.normcase(os.path.normpath(stripped))
path_map[normalized_value] = stripped
if path_map:
normalized_new[key] = path_map
if not normalized_new:
return
for other_name, other in libraries.items():
if other_name == library_name:
continue
other_paths = other.get("folder_paths", {})
for key, new_paths in normalized_new.items():
existing = {
os.path.normcase(os.path.normpath(path))
for path in other_paths.get(key, [])
if isinstance(path, str) and path
}
overlap = existing.intersection(new_paths.keys())
if overlap:
collisions = ", ".join(sorted(new_paths[value] for value in overlap))
raise ValueError(
f"Folder path(s) {collisions} already assigned to library '{other_name}'"
)
def _update_active_library_entry(
self,
*,
folder_paths: Optional[Mapping[str, Iterable[str]]] = None,
default_lora_root: Optional[str] = None,
default_checkpoint_root: Optional[str] = None,
default_embedding_root: Optional[str] = None,
) -> bool:
libraries = self.settings.get("libraries", {})
active_name = self.settings.get("active_library")
if not active_name or active_name not in libraries:
return False
library = libraries[active_name]
changed = False
if folder_paths is not None:
normalized_paths = self._normalize_folder_paths(folder_paths)
if library.get("folder_paths") != normalized_paths:
library["folder_paths"] = normalized_paths
changed = True
if default_lora_root is not None and library.get("default_lora_root") != default_lora_root:
library["default_lora_root"] = default_lora_root
changed = True
if default_checkpoint_root is not None and library.get("default_checkpoint_root") != default_checkpoint_root:
library["default_checkpoint_root"] = default_checkpoint_root
changed = True
if default_embedding_root is not None and library.get("default_embedding_root") != default_embedding_root:
library["default_embedding_root"] = default_embedding_root
changed = True
if changed:
library.setdefault("created_at", self._current_timestamp())
library["updated_at"] = self._current_timestamp()
return changed
def _migrate_setting_keys(self) -> None:
"""Migrate legacy camelCase setting keys to snake_case"""
key_migrations = {
@@ -111,25 +349,36 @@ class SettingsManager:
logger.info("Migration completed")
def _auto_set_default_roots(self):
"""Auto set default root paths if only one folder is present and default is empty."""
"""Auto set default root paths when only one folder is present and the current default is unset or not among the options."""
folder_paths = self.settings.get('folder_paths', {})
updated = False
# loras
loras = folder_paths.get('loras', [])
if isinstance(loras, list) and len(loras) == 1 and not self.settings.get('default_lora_root'):
self.settings['default_lora_root'] = loras[0]
updated = True
if isinstance(loras, list) and len(loras) == 1:
current_lora_root = self.settings.get('default_lora_root')
if current_lora_root not in loras:
self.settings['default_lora_root'] = loras[0]
updated = True
# checkpoints
checkpoints = folder_paths.get('checkpoints', [])
if isinstance(checkpoints, list) and len(checkpoints) == 1 and not self.settings.get('default_checkpoint_root'):
self.settings['default_checkpoint_root'] = checkpoints[0]
updated = True
if isinstance(checkpoints, list) and len(checkpoints) == 1:
current_checkpoint_root = self.settings.get('default_checkpoint_root')
if current_checkpoint_root not in checkpoints:
self.settings['default_checkpoint_root'] = checkpoints[0]
updated = True
# embeddings
embeddings = folder_paths.get('embeddings', [])
if isinstance(embeddings, list) and len(embeddings) == 1 and not self.settings.get('default_embedding_root'):
self.settings['default_embedding_root'] = embeddings[0]
updated = True
if isinstance(embeddings, list) and len(embeddings) == 1:
current_embedding_root = self.settings.get('default_embedding_root')
if current_embedding_root not in embeddings:
self.settings['default_embedding_root'] = embeddings[0]
updated = True
if updated:
self._update_active_library_entry(
default_lora_root=self.settings.get('default_lora_root'),
default_checkpoint_root=self.settings.get('default_checkpoint_root'),
default_embedding_root=self.settings.get('default_embedding_root'),
)
self._save_settings()
def _check_environment_variables(self) -> None:
@@ -151,8 +400,56 @@ class SettingsManager:
# Ensure nested dicts are independent copies
defaults['base_model_path_mappings'] = {}
defaults['download_path_templates'] = {}
defaults['priority_tags'] = DEFAULT_PRIORITY_TAG_CONFIG.copy()
return defaults
def _normalize_priority_tag_config(self, value: Any) -> Dict[str, str]:
normalized: Dict[str, str] = {}
if isinstance(value, Mapping):
for key, raw in value.items():
if not isinstance(key, str) or not isinstance(raw, str):
continue
normalized[key] = raw.strip()
for model_type, default_value in DEFAULT_PRIORITY_TAG_CONFIG.items():
normalized.setdefault(model_type, default_value)
return normalized
def get_priority_tag_config(self) -> Dict[str, str]:
stored_value = self.settings.get("priority_tags")
normalized = self._normalize_priority_tag_config(stored_value)
if normalized != stored_value:
self.settings["priority_tags"] = normalized
self._save_settings()
return normalized.copy()
def get_priority_tag_entries(self, model_type: str) -> List[PriorityTagEntry]:
config = self.get_priority_tag_config()
raw_config = config.get(model_type, "")
return parse_priority_tag_string(raw_config)
def resolve_priority_tag_for_model(
self, tags: Sequence[str] | Iterable[str], model_type: str
) -> str:
entries = self.get_priority_tag_entries(model_type)
resolved = resolve_priority_tag(tags, entries)
if resolved:
return resolved
for tag in tags:
if isinstance(tag, str) and tag:
return tag
return ""
def get_priority_tag_suggestions(self) -> Dict[str, List[str]]:
suggestions: Dict[str, List[str]] = {}
config = self.get_priority_tag_config()
for model_type, raw_value in config.items():
entries = parse_priority_tag_string(raw_value)
suggestions[model_type] = collect_canonical_tags(entries)
return suggestions
def get(self, key: str, default: Any = None) -> Any:
"""Get setting value"""
return self.settings.get(key, default)
@@ -160,6 +457,14 @@ class SettingsManager:
def set(self, key: str, value: Any) -> None:
"""Set setting value and save"""
self.settings[key] = value
if key == 'folder_paths' and isinstance(value, Mapping):
self._update_active_library_entry(folder_paths=value) # type: ignore[arg-type]
elif key == 'default_lora_root':
self._update_active_library_entry(default_lora_root=str(value))
elif key == 'default_checkpoint_root':
self._update_active_library_entry(default_checkpoint_root=str(value))
elif key == 'default_embedding_root':
self._update_active_library_entry(default_embedding_root=str(value))
self._save_settings()
def delete(self, key: str) -> None:
@@ -177,6 +482,227 @@ class SettingsManager:
except Exception as e:
logger.error(f"Error saving settings: {e}")
def get_libraries(self) -> Dict[str, Dict[str, Any]]:
"""Return a copy of the registered libraries."""
libraries = self.settings.get("libraries", {})
return copy.deepcopy(libraries)
def get_active_library_name(self) -> str:
"""Return the currently active library name."""
libraries = self.settings.get("libraries", {})
active_name = self.settings.get("active_library")
if active_name and active_name in libraries:
return active_name
if libraries:
return next(iter(libraries.keys()))
return "default"
def get_active_library(self) -> Dict[str, Any]:
"""Return a copy of the active library configuration."""
libraries = self.settings.get("libraries", {})
active_name = self.get_active_library_name()
return copy.deepcopy(libraries.get(active_name, {}))
def activate_library(self, library_name: str) -> None:
"""Activate a library by name and refresh dependent services."""
libraries = self.settings.get("libraries", {})
if library_name not in libraries:
raise KeyError(f"Library '{library_name}' does not exist")
current_active = self.get_active_library_name()
if current_active == library_name:
# Ensure root settings stay in sync even if already active
self._sync_active_library_to_root(save=False)
self._save_settings()
self._notify_library_change(library_name)
return
self.settings["active_library"] = library_name
self._sync_active_library_to_root(save=False)
self._save_settings()
self._notify_library_change(library_name)
def upsert_library(
self,
library_name: str,
*,
folder_paths: Optional[Mapping[str, Iterable[str]]] = None,
default_lora_root: Optional[str] = None,
default_checkpoint_root: Optional[str] = None,
default_embedding_root: Optional[str] = None,
metadata: Optional[Mapping[str, Any]] = None,
activate: bool = False,
) -> Dict[str, Any]:
"""Create or update a library definition."""
name = library_name.strip()
if not name:
raise ValueError("Library name cannot be empty")
if folder_paths is not None:
self._validate_folder_paths(name, folder_paths)
libraries = self.settings.setdefault("libraries", {})
existing = libraries.get(name, {})
payload = self._build_library_payload(
folder_paths=folder_paths if folder_paths is not None else existing.get("folder_paths"),
default_lora_root=default_lora_root if default_lora_root is not None else existing.get("default_lora_root"),
default_checkpoint_root=(
default_checkpoint_root
if default_checkpoint_root is not None
else existing.get("default_checkpoint_root")
),
default_embedding_root=(
default_embedding_root
if default_embedding_root is not None
else existing.get("default_embedding_root")
),
metadata=metadata if metadata is not None else existing.get("metadata"),
base=existing,
)
libraries[name] = payload
if activate or not self.settings.get("active_library"):
self.settings["active_library"] = name
self._sync_active_library_to_root(save=False)
self._save_settings()
if self.settings.get("active_library") == name:
self._notify_library_change(name)
return payload
def create_library(
self,
library_name: str,
*,
folder_paths: Mapping[str, Iterable[str]],
default_lora_root: str = "",
default_checkpoint_root: str = "",
default_embedding_root: str = "",
metadata: Optional[Mapping[str, Any]] = None,
activate: bool = False,
) -> Dict[str, Any]:
"""Create a new library entry."""
libraries = self.settings.get("libraries", {})
if library_name in libraries:
raise ValueError(f"Library '{library_name}' already exists")
return self.upsert_library(
library_name,
folder_paths=folder_paths,
default_lora_root=default_lora_root,
default_checkpoint_root=default_checkpoint_root,
default_embedding_root=default_embedding_root,
metadata=metadata,
activate=activate,
)
def rename_library(self, old_name: str, new_name: str) -> None:
"""Rename an existing library."""
libraries = self.settings.get("libraries", {})
if old_name not in libraries:
raise KeyError(f"Library '{old_name}' does not exist")
new_name_stripped = new_name.strip()
if not new_name_stripped:
raise ValueError("New library name cannot be empty")
if new_name_stripped in libraries:
raise ValueError(f"Library '{new_name_stripped}' already exists")
libraries[new_name_stripped] = libraries.pop(old_name)
if self.settings.get("active_library") == old_name:
self.settings["active_library"] = new_name_stripped
active_name = new_name_stripped
else:
active_name = self.settings.get("active_library")
self._sync_active_library_to_root(save=False)
self._save_settings()
if active_name == new_name_stripped:
self._notify_library_change(new_name_stripped)
def delete_library(self, library_name: str) -> None:
"""Remove a library definition."""
libraries = self.settings.get("libraries", {})
if library_name not in libraries:
raise KeyError(f"Library '{library_name}' does not exist")
if len(libraries) == 1:
raise ValueError("At least one library must remain")
was_active = self.settings.get("active_library") == library_name
libraries.pop(library_name)
if was_active:
new_active = next(iter(libraries.keys()))
self.settings["active_library"] = new_active
self._sync_active_library_to_root(save=False)
self._save_settings()
if was_active:
self._notify_library_change(self.settings["active_library"])
def update_active_library_paths(
self,
folder_paths: Mapping[str, Iterable[str]],
*,
default_lora_root: Optional[str] = None,
default_checkpoint_root: Optional[str] = None,
default_embedding_root: Optional[str] = None,
) -> None:
"""Update folder paths for the active library."""
active_name = self.get_active_library_name()
self.upsert_library(
active_name,
folder_paths=folder_paths,
default_lora_root=default_lora_root,
default_checkpoint_root=default_checkpoint_root,
default_embedding_root=default_embedding_root,
activate=True,
)
def _notify_library_change(self, library_name: str) -> None:
"""Notify dependent services that the active library changed."""
libraries = self.settings.get("libraries", {})
library_config = libraries.get(library_name, {})
library_snapshot = copy.deepcopy(library_config)
try:
from ..config import config # Local import to avoid circular dependency
config.apply_library_settings(library_snapshot)
except Exception as exc: # pragma: no cover - defensive logging
logger.debug("Failed to apply library settings to config: %s", exc)
try:
from .service_registry import ServiceRegistry # type: ignore
for service_name in (
"lora_scanner",
"checkpoint_scanner",
"embedding_scanner",
"recipe_scanner",
):
service = ServiceRegistry.get_service_sync(service_name)
if service and hasattr(service, "on_library_changed"):
try:
service.on_library_changed()
except Exception as service_exc: # pragma: no cover - defensive logging
logger.debug(
"Service %s failed to handle library change: %s",
service_name,
service_exc,
)
except Exception as exc: # pragma: no cover - defensive logging
logger.debug("Failed to notify services about library change: %s", exc)
def get_download_path_template(self, model_type: str) -> str:
"""Get download path template for specific model type
@@ -226,4 +752,38 @@ class SettingsManager:
return templates.get(model_type, '{base_model}/{first_tag}')
settings = SettingsManager()
_SETTINGS_MANAGER: Optional["SettingsManager"] = None
_SETTINGS_MANAGER_LOCK = Lock()
# Legacy module-level alias for backwards compatibility with callers that
# monkeypatch ``py.services.settings_manager.settings`` during tests.
settings: Optional["SettingsManager"] = None
def get_settings_manager() -> "SettingsManager":
"""Return the lazily initialised global :class:`SettingsManager`."""
global _SETTINGS_MANAGER, settings
if settings is not None:
return settings
if _SETTINGS_MANAGER is None:
with _SETTINGS_MANAGER_LOCK:
if _SETTINGS_MANAGER is None:
_SETTINGS_MANAGER = SettingsManager()
settings = _SETTINGS_MANAGER
return _SETTINGS_MANAGER
def reset_settings_manager() -> None:
"""Reset the cached settings manager instance.
Primarily intended for tests so they can configure the settings
directory before the manager touches the filesystem.
"""
global _SETTINGS_MANAGER, settings
with _SETTINGS_MANAGER_LOCK:
_SETTINGS_MANAGER = None
settings = None

View File

@@ -6,6 +6,7 @@ import logging
from typing import Any, Dict, Optional, Protocol, Sequence
from ..metadata_sync_service import MetadataSyncService
from ...utils.metadata_manager import MetadataManager
class MetadataRefreshProgressReporter(Protocol):
@@ -70,6 +71,7 @@ class BulkMetadataRefreshUseCase:
for model in to_process:
try:
original_name = model.get("model_name")
await MetadataManager.hydrate_model_data(model)
result, _ = await self._metadata_sync.fetch_and_update_model(
sha256=model["sha256"],
file_path=model["file_path"],

View File

@@ -16,6 +16,8 @@ class WebSocketManager:
self._download_websockets: Dict[str, web.WebSocketResponse] = {} # New dict for download-specific clients
# Add progress tracking dictionary
self._download_progress: Dict[str, Dict] = {}
# Cache last initialization progress payloads
self._last_init_progress: Dict[str, Dict] = {}
# Add auto-organize progress tracking
self._auto_organize_progress: Optional[Dict] = None
self._auto_organize_lock = asyncio.Lock()
@@ -39,8 +41,10 @@ class WebSocketManager:
ws = web.WebSocketResponse()
await ws.prepare(request)
self._init_websockets.add(ws)
try:
await self._send_cached_init_progress(ws)
async for msg in ws:
if msg.type == web.WSMsgType.ERROR:
logger.error(f'Init WebSocket error: {ws.exception()}')
@@ -102,30 +106,70 @@ class WebSocketManager:
async def broadcast_init_progress(self, data: Dict):
"""Broadcast initialization progress to connected clients"""
payload = dict(data) if data else {}
if 'stage' not in payload:
payload['stage'] = 'processing'
if 'progress' not in payload:
payload['progress'] = 0
if 'details' not in payload:
payload['details'] = 'Processing...'
key = self._get_init_progress_key(payload)
self._last_init_progress[key] = dict(payload)
if not self._init_websockets:
return
# Ensure data has all required fields
if 'stage' not in data:
data['stage'] = 'processing'
if 'progress' not in data:
data['progress'] = 0
if 'details' not in data:
data['details'] = 'Processing...'
for ws in self._init_websockets:
stale_clients = []
for ws in list(self._init_websockets):
try:
await ws.send_json(data)
await ws.send_json(payload)
except Exception as e:
logger.error(f"Error sending initialization progress: {e}")
stale_clients.append(ws)
for ws in stale_clients:
self._init_websockets.discard(ws)
async def _send_cached_init_progress(self, ws: web.WebSocketResponse) -> None:
"""Send cached initialization progress payloads to a new client"""
if not self._last_init_progress:
return
for payload in list(self._last_init_progress.values()):
try:
await ws.send_json(payload)
except Exception as e:
logger.debug(f'Error sending cached initialization progress: {e}')
def _get_init_progress_key(self, data: Dict) -> str:
"""Return a stable key for caching initialization progress payloads"""
page_type = data.get('pageType')
if page_type:
return f'page:{page_type}'
scanner_type = data.get('scanner_type')
if scanner_type:
return f'scanner:{scanner_type}'
return 'global'
async def broadcast_download_progress(self, download_id: str, data: Dict):
"""Send progress update to specific download client"""
# Store simplified progress data in memory (only progress percentage)
self._download_progress[download_id] = {
progress_entry = {
'progress': data.get('progress', 0),
'timestamp': datetime.now()
'timestamp': datetime.now(),
}
for field in ('bytes_downloaded', 'total_bytes', 'bytes_per_second'):
if field in data:
progress_entry[field] = data[field]
if 'status' in data:
progress_entry['status'] = data['status']
if 'message' in data:
progress_entry['message'] = data['message']
self._download_progress[download_id] = progress_entry
if download_id not in self._download_websockets:
logger.debug(f"No WebSocket found for download ID: {download_id}")
@@ -202,4 +246,5 @@ class WebSocketManager:
return str(uuid4())
# Global instance
ws_manager = WebSocketManager()
ws_manager = WebSocketManager()

47
py/utils/civitai_utils.py Normal file
View File

@@ -0,0 +1,47 @@
"""Utilities for working with Civitai assets."""
from __future__ import annotations
from urllib.parse import urlparse, urlunparse
def rewrite_preview_url(source_url: str | None, media_type: str | None = None) -> tuple[str | None, bool]:
"""Rewrite Civitai preview URLs to use optimized renditions.
Args:
source_url: Original preview URL from the Civitai API.
media_type: Optional media type hint (e.g. ``"image"`` or ``"video"``).
Returns:
A tuple of the potentially rewritten URL and a flag indicating whether the
replacement occurred. When the URL is not rewritten, the original value is
returned with ``False``.
"""
if not source_url:
return source_url, False
try:
parsed = urlparse(source_url)
except ValueError:
return source_url, False
if parsed.netloc.lower() != "image.civitai.com":
return source_url, False
replacement = "/width=450,optimized=true"
if (media_type or "").lower() == "video":
replacement = "/transcode=true,width=450,optimized=true"
if "/original=true" not in parsed.path:
return source_url, False
updated_path = parsed.path.replace("/original=true", replacement, 1)
if updated_path == parsed.path:
return source_url, False
rewritten = urlunparse(parsed._replace(path=updated_path))
return rewritten, True
__all__ = ["rewrite_preview_url"]

View File

@@ -48,6 +48,13 @@ SUPPORTED_MEDIA_EXTENSIONS = {
# Valid Lora types
VALID_LORA_TYPES = ['lora', 'locon', 'dora']
# Supported Civitai model types for user model queries (case-insensitive)
CIVITAI_USER_MODEL_TYPES = [
*VALID_LORA_TYPES,
'textualinversion',
'checkpoint',
]
# Auto-organize settings
AUTO_ORGANIZE_BATCH_SIZE = 50 # Process models in batches to avoid overwhelming the system
@@ -57,4 +64,11 @@ CIVITAI_MODEL_TAGS = [
'realistic', 'anime', 'toon', 'furry', 'style',
'poses', 'background', 'tool', 'vehicle', 'buildings',
'objects', 'assets', 'animal', 'action'
]
]
# Default priority tag configuration strings for each model type
DEFAULT_PRIORITY_TAG_CONFIG = {
'lora': ', '.join(CIVITAI_MODEL_TAGS),
'checkpoint': ', '.join(CIVITAI_MODEL_TAGS),
'embedding': ', '.join(CIVITAI_MODEL_TAGS),
}

View File

@@ -1,18 +1,24 @@
from __future__ import annotations
import logging
import os
import asyncio
import json
import time
import logging
import os
import shutil
from typing import Any, Dict
from ..services.service_registry import ServiceRegistry
from ..utils.example_images_paths import (
ExampleImagePathResolver,
ensure_library_root_exists,
uses_library_scoped_folders,
)
from ..utils.metadata_manager import MetadataManager
from .example_images_processor import ExampleImagesProcessor
from .example_images_metadata import MetadataUpdater
from ..services.downloader import get_downloader
from ..services.settings_manager import settings
from ..services.settings_manager import get_settings_manager
class ExampleImagesDownloadError(RuntimeError):
@@ -74,6 +80,22 @@ class _DownloadProgress(dict):
snapshot['failed_models'] = list(self['failed_models'])
return snapshot
def _model_directory_has_files(path: str) -> bool:
"""Return True when the provided directory exists and contains entries."""
if not path or not os.path.isdir(path):
return False
try:
with os.scandir(path) as entries:
for _ in entries:
return True
except OSError:
return False
return False
class DownloadManager:
"""Manages downloading example images for models."""
@@ -83,6 +105,13 @@ class DownloadManager:
self._progress = _DownloadProgress()
self._ws_manager = ws_manager
self._state_lock = state_lock or asyncio.Lock()
self._stop_requested = False
def _resolve_output_dir(self, library_name: str | None = None) -> str:
base_path = get_settings_manager().get('example_images_path')
if not base_path:
return ''
return ensure_library_root_exists(library_name)
async def start_download(self, options: dict):
"""Start downloading example images for models."""
@@ -98,9 +127,10 @@ class DownloadManager:
model_types = data.get('model_types', ['lora', 'checkpoint'])
delay = float(data.get('delay', 0.2))
output_dir = settings.get('example_images_path')
settings_manager = get_settings_manager()
base_path = settings_manager.get('example_images_path')
if not output_dir:
if not base_path:
error_msg = 'Example images path not configured in settings'
if auto_mode:
logger.debug(error_msg)
@@ -110,17 +140,43 @@ class DownloadManager:
}
raise DownloadConfigurationError(error_msg)
os.makedirs(output_dir, exist_ok=True)
active_library = get_settings_manager().get_active_library_name()
output_dir = self._resolve_output_dir(active_library)
if not output_dir:
raise DownloadConfigurationError('Example images path not configured in settings')
self._progress.reset()
self._stop_requested = False
self._progress['status'] = 'running'
self._progress['start_time'] = time.time()
self._progress['end_time'] = None
progress_file = os.path.join(output_dir, '.download_progress.json')
if os.path.exists(progress_file):
progress_source = progress_file
if uses_library_scoped_folders():
legacy_root = get_settings_manager().get('example_images_path') or ''
legacy_progress = os.path.join(legacy_root, '.download_progress.json') if legacy_root else ''
if legacy_progress and os.path.exists(legacy_progress) and not os.path.exists(progress_file):
try:
os.makedirs(output_dir, exist_ok=True)
shutil.move(legacy_progress, progress_file)
logger.info(
"Migrated legacy download progress file '%s' to '%s'",
legacy_progress,
progress_file,
)
except OSError as exc:
logger.warning(
"Failed to migrate download progress file from '%s' to '%s': %s",
legacy_progress,
progress_file,
exc,
)
progress_source = legacy_progress
if os.path.exists(progress_source):
try:
with open(progress_file, 'r', encoding='utf-8') as f:
with open(progress_source, 'r', encoding='utf-8') as f:
saved_progress = json.load(f)
self._progress['processed_models'] = set(saved_progress.get('processed_models', []))
self._progress['failed_models'] = set(saved_progress.get('failed_models', []))
@@ -143,11 +199,17 @@ class DownloadManager:
output_dir,
optimize,
model_types,
delay
delay,
active_library,
)
)
snapshot = self._progress.snapshot()
except ExampleImagesDownloadError:
# Re-raise our own exception types without wrapping
self._is_downloading = False
self._download_task = None
raise
except Exception as e:
self._is_downloading = False
self._download_task = None
@@ -207,8 +269,36 @@ class DownloadManager:
'success': True,
'message': 'Download resumed'
}
async def stop_download(self, request):
"""Stop the example images download after the current model completes."""
async with self._state_lock:
if not self._is_downloading:
raise DownloadNotRunningError()
if self._progress['status'] in {'completed', 'error', 'stopped'}:
raise DownloadNotRunningError()
if self._progress['status'] != 'stopping':
self._stop_requested = True
self._progress['status'] = 'stopping'
await self._broadcast_progress(status='stopping')
return {
'success': True,
'message': 'Download stopping'
}
async def _download_all_example_images(self, output_dir, optimize, model_types, delay):
async def _download_all_example_images(
self,
output_dir,
optimize,
model_types,
delay,
library_name,
):
"""Download example images for all models."""
downloader = await get_downloader()
@@ -244,29 +334,75 @@ class DownloadManager:
# Process each model
for i, (scanner_type, model, scanner) in enumerate(all_models):
async with self._state_lock:
current_status = self._progress['status']
if current_status not in {'running', 'paused', 'stopping'}:
break
# Main logic for processing model is here, but actual operations are delegated to other classes
was_remote_download = await self._process_model(
scanner_type, model, scanner,
output_dir, optimize, downloader
scanner_type,
model,
scanner,
output_dir,
optimize,
downloader,
library_name,
)
# Update progress
self._progress['completed'] += 1
await self._broadcast_progress(status='running')
async with self._state_lock:
current_status = self._progress['status']
should_stop = self._stop_requested and current_status == 'stopping'
broadcast_status = 'running' if current_status == 'running' else current_status
await self._broadcast_progress(status=broadcast_status)
if should_stop:
break
# Only add delay after remote download of models, and not after processing the last model
if was_remote_download and i < len(all_models) - 1 and self._progress['status'] == 'running':
if (
was_remote_download
and i < len(all_models) - 1
and current_status == 'running'
):
await asyncio.sleep(delay)
# Mark as completed
self._progress['status'] = 'completed'
self._progress['end_time'] = time.time()
logger.debug(
"Example images download completed: %s/%s models processed",
self._progress['completed'],
self._progress['total'],
)
await self._broadcast_progress(status='completed')
async with self._state_lock:
if self._stop_requested and self._progress['status'] == 'stopping':
self._progress['status'] = 'stopped'
self._progress['end_time'] = time.time()
self._stop_requested = False
final_status = 'stopped'
elif self._progress['status'] not in {'error', 'stopped'}:
self._progress['status'] = 'completed'
self._progress['end_time'] = time.time()
self._stop_requested = False
final_status = 'completed'
else:
final_status = self._progress['status']
self._stop_requested = False
if self._progress['end_time'] is None:
self._progress['end_time'] = time.time()
if final_status == 'completed':
logger.debug(
"Example images download completed: %s/%s models processed",
self._progress['completed'],
self._progress['total'],
)
elif final_status == 'stopped':
logger.debug(
"Example images download stopped: %s/%s models processed",
self._progress['completed'],
self._progress['total'],
)
await self._broadcast_progress(status=final_status)
except Exception as e:
error_msg = f"Error during example images download: {str(e)}"
@@ -288,8 +424,18 @@ class DownloadManager:
async with self._state_lock:
self._is_downloading = False
self._download_task = None
self._stop_requested = False
async def _process_model(self, scanner_type, model, scanner, output_dir, optimize, downloader):
async def _process_model(
self,
scanner_type,
model,
scanner,
output_dir,
optimize,
downloader,
library_name,
):
"""Process a single model download."""
# Check if download is paused
@@ -297,7 +443,7 @@ class DownloadManager:
await asyncio.sleep(1)
# Check if download should continue
if self._progress['status'] != 'running':
if self._progress['status'] not in {'running', 'stopping'}:
logger.info(f"Download stopped: {self._progress['status']}")
return False # Return False to indicate no remote download happened
@@ -316,20 +462,35 @@ class DownloadManager:
logger.debug(f"Skipping known failed model: {model_name}")
return False
model_dir = ExampleImagePathResolver.get_model_folder(model_hash, library_name)
existing_files = _model_directory_has_files(model_dir)
# Skip if already processed AND directory exists with files
if model_hash in self._progress['processed_models']:
model_dir = os.path.join(output_dir, model_hash)
has_files = os.path.exists(model_dir) and any(os.listdir(model_dir))
if has_files:
if existing_files:
logger.debug(f"Skipping already processed model: {model_name}")
return False
else:
logger.info(f"Model {model_name} marked as processed but folder empty or missing, reprocessing")
# Remove from processed models since we need to reprocess
self._progress['processed_models'].discard(model_hash)
logger.info(f"Model {model_name} marked as processed but folder empty or missing, reprocessing")
# Remove from processed models since we need to reprocess
self._progress['processed_models'].discard(model_hash)
if existing_files and model_hash not in self._progress['processed_models']:
logger.debug(
"Model folder already populated for %s, marking as processed without download",
model_name,
)
self._progress['processed_models'].add(model_hash)
return False
if not model_dir:
logger.warning(
"Unable to resolve example images folder for model %s (%s)",
model_name,
model_hash,
)
return False
# Create model directory
model_dir = os.path.join(output_dir, model_hash)
os.makedirs(model_dir, exist_ok=True)
# First check for local example images - local processing doesn't need delay
@@ -345,14 +506,20 @@ class DownloadManager:
self._progress['processed_models'].add(model_hash)
return False # Return False to indicate no remote download happened
full_model = await MetadataUpdater.get_updated_model(
model_hash, scanner
)
civitai_payload = (full_model or {}).get('civitai') if full_model else None
civitai_payload = civitai_payload or {}
# If no local images, try to download from remote
elif model.get('civitai') and model.get('civitai', {}).get('images'):
images = model.get('civitai', {}).get('images', [])
if civitai_payload.get('images'):
images = civitai_payload.get('images', [])
success, is_stale = await ExampleImagesProcessor.download_model_images(
model_hash, model_name, images, model_dir, optimize, downloader
)
# If metadata is stale, try to refresh it
if is_stale and model_hash not in self._progress['refreshed_models']:
await MetadataUpdater.refresh_model_metadata(
@@ -363,16 +530,18 @@ class DownloadManager:
updated_model = await MetadataUpdater.get_updated_model(
model_hash, scanner
)
updated_civitai = (updated_model or {}).get('civitai') if updated_model else None
updated_civitai = updated_civitai or {}
if updated_model and updated_model.get('civitai', {}).get('images'):
if updated_civitai.get('images'):
# Retry download with updated metadata
updated_images = updated_model.get('civitai', {}).get('images', [])
updated_images = updated_civitai.get('images', [])
success, _ = await ExampleImagesProcessor.download_model_images(
model_hash, model_name, updated_images, model_dir, optimize, downloader
)
self._progress['refreshed_models'].add(model_hash)
# Mark as processed if successful, or as failed if unsuccessful after refresh
if success:
self._progress['processed_models'].add(model_hash)
@@ -381,13 +550,13 @@ class DownloadManager:
if model_hash in self._progress['refreshed_models']:
self._progress['failed_models'].add(model_hash)
logger.info(f"Marking model {model_name} as failed after metadata refresh")
return True # Return True to indicate a remote download happened
else:
# No civitai data or images available, mark as failed to avoid future attempts
self._progress['failed_models'].add(model_hash)
logger.debug(f"No civitai images available for model {model_name}, marking as failed")
# Save progress periodically
if self._progress['completed'] % 10 == 0 or self._progress['completed'] == self._progress['total'] - 1:
self._save_progress(output_dir)
@@ -452,14 +621,18 @@ class DownloadManager:
if not model_hashes:
raise DownloadConfigurationError('Missing model_hashes parameter')
output_dir = settings.get('example_images_path')
settings_manager = get_settings_manager()
base_path = settings_manager.get('example_images_path')
if not base_path:
raise DownloadConfigurationError('Example images path not configured in settings')
active_library = settings_manager.get_active_library_name()
output_dir = self._resolve_output_dir(active_library)
if not output_dir:
raise DownloadConfigurationError('Example images path not configured in settings')
os.makedirs(output_dir, exist_ok=True)
self._progress.reset()
self._stop_requested = False
self._progress['total'] = len(model_hashes)
self._progress['status'] = 'running'
self._progress['start_time'] = time.time()
@@ -475,15 +648,21 @@ class DownloadManager:
output_dir,
optimize,
model_types,
delay
delay,
active_library,
)
async with self._state_lock:
self._is_downloading = False
final_status = self._progress['status']
message = 'Force download completed'
if final_status == 'stopped':
message = 'Force download stopped'
return {
'success': True,
'message': 'Force download completed',
'message': message,
'result': result
}
@@ -494,7 +673,15 @@ class DownloadManager:
await self._broadcast_progress(status='error', extra={'error': str(e)})
raise ExampleImagesDownloadError(str(e)) from e
async def _download_specific_models_example_images_sync(self, model_hashes, output_dir, optimize, model_types, delay):
async def _download_specific_models_example_images_sync(
self,
model_hashes,
output_dir,
optimize,
model_types,
delay,
library_name,
):
"""Download example images for specific models only - synchronous version."""
downloader = await get_downloader()
@@ -533,37 +720,81 @@ class DownloadManager:
# Process each model
success_count = 0
for i, (scanner_type, model, scanner) in enumerate(models_to_process):
async with self._state_lock:
current_status = self._progress['status']
if current_status not in {'running', 'paused', 'stopping'}:
break
# Force process this model regardless of previous status
was_successful = await self._process_specific_model(
scanner_type, model, scanner,
output_dir, optimize, downloader
scanner_type,
model,
scanner,
output_dir,
optimize,
downloader,
library_name,
)
if was_successful:
success_count += 1
# Update progress
self._progress['completed'] += 1
async with self._state_lock:
current_status = self._progress['status']
should_stop = self._stop_requested and current_status == 'stopping'
broadcast_status = 'running' if current_status == 'running' else current_status
# Send progress update via WebSocket
await self._broadcast_progress(status='running')
await self._broadcast_progress(status=broadcast_status)
if should_stop:
break
# Only add delay after remote download, and not after processing the last model
if was_successful and i < len(models_to_process) - 1 and self._progress['status'] == 'running':
if (
was_successful
and i < len(models_to_process) - 1
and current_status == 'running'
):
await asyncio.sleep(delay)
# Mark as completed
self._progress['status'] = 'completed'
self._progress['end_time'] = time.time()
logger.debug(
"Forced example images download completed: %s/%s models processed",
self._progress['completed'],
self._progress['total'],
)
async with self._state_lock:
if self._stop_requested and self._progress['status'] == 'stopping':
self._progress['status'] = 'stopped'
self._progress['end_time'] = time.time()
self._stop_requested = False
final_status = 'stopped'
elif self._progress['status'] not in {'error', 'stopped'}:
self._progress['status'] = 'completed'
self._progress['end_time'] = time.time()
self._stop_requested = False
final_status = 'completed'
else:
final_status = self._progress['status']
self._stop_requested = False
if self._progress['end_time'] is None:
self._progress['end_time'] = time.time()
if final_status == 'completed':
logger.debug(
"Forced example images download completed: %s/%s models processed",
self._progress['completed'],
self._progress['total'],
)
elif final_status == 'stopped':
logger.debug(
"Forced example images download stopped: %s/%s models processed",
self._progress['completed'],
self._progress['total'],
)
# Send final progress via WebSocket
await self._broadcast_progress(status='completed')
await self._broadcast_progress(status=final_status)
return {
'total': self._progress['total'],
'processed': self._progress['completed'],
@@ -588,7 +819,16 @@ class DownloadManager:
# No need to close any sessions since we use the global downloader
pass
async def _process_specific_model(self, scanner_type, model, scanner, output_dir, optimize, downloader):
async def _process_specific_model(
self,
scanner_type,
model,
scanner,
output_dir,
optimize,
downloader,
library_name,
):
"""Process a specific model for forced download, ignoring previous download status."""
# Check if download is paused
@@ -596,7 +836,7 @@ class DownloadManager:
await asyncio.sleep(1)
# Check if download should continue
if self._progress['status'] != 'running':
if self._progress['status'] not in {'running', 'stopping'}:
logger.info(f"Download stopped: {self._progress['status']}")
return False
@@ -610,8 +850,15 @@ class DownloadManager:
self._progress['current_model'] = f"{model_name} ({model_hash[:8]})"
await self._broadcast_progress(status='running')
# Create model directory
model_dir = os.path.join(output_dir, model_hash)
model_dir = ExampleImagePathResolver.get_model_folder(model_hash, library_name)
if not model_dir:
logger.warning(
"Unable to resolve example images folder for model %s (%s)",
model_name,
model_hash,
)
return False
os.makedirs(model_dir, exist_ok=True)
# First check for local example images - local processing doesn't need delay
@@ -627,51 +874,61 @@ class DownloadManager:
self._progress['processed_models'].add(model_hash)
return False # Return False to indicate no remote download happened
full_model = await MetadataUpdater.get_updated_model(
model_hash, scanner
)
civitai_payload = (full_model or {}).get('civitai') if full_model else None
civitai_payload = civitai_payload or {}
# If no local images, try to download from remote
elif model.get('civitai') and model.get('civitai', {}).get('images'):
images = model.get('civitai', {}).get('images', [])
if civitai_payload.get('images'):
images = civitai_payload.get('images', [])
success, is_stale, failed_images = await ExampleImagesProcessor.download_model_images_with_tracking(
model_hash, model_name, images, model_dir, optimize, downloader
)
# If metadata is stale, try to refresh it
if is_stale and model_hash not in self._progress['refreshed_models']:
await MetadataUpdater.refresh_model_metadata(
model_hash, model_name, scanner_type, scanner, self._progress
)
# Get the updated model data
updated_model = await MetadataUpdater.get_updated_model(
model_hash, scanner
)
if updated_model and updated_model.get('civitai', {}).get('images'):
updated_civitai = (updated_model or {}).get('civitai') if updated_model else None
updated_civitai = updated_civitai or {}
if updated_civitai.get('images'):
# Retry download with updated metadata
updated_images = updated_model.get('civitai', {}).get('images', [])
updated_images = updated_civitai.get('images', [])
success, _, additional_failed_images = await ExampleImagesProcessor.download_model_images_with_tracking(
model_hash, model_name, updated_images, model_dir, optimize, downloader
)
# Combine failed images from both attempts
failed_images.extend(additional_failed_images)
self._progress['refreshed_models'].add(model_hash)
# For forced downloads, remove failed images from metadata
if failed_images:
# Create a copy of images excluding failed ones
await self._remove_failed_images_from_metadata(
model_hash, model_name, failed_images, scanner
)
# Mark as processed
if success or failed_images: # Mark as processed if we successfully downloaded some images or removed failed ones
self._progress['processed_models'].add(model_hash)
return True # Return True to indicate a remote download happened
else:
logger.debug(f"No civitai images available for model {model_name}")
return False
except Exception as e:

View File

@@ -3,7 +3,11 @@ import os
import sys
import subprocess
from aiohttp import web
from ..services.settings_manager import settings
from ..services.settings_manager import get_settings_manager
from ..utils.example_images_paths import (
get_model_folder,
get_model_relative_path,
)
from ..utils.constants import SUPPORTED_MEDIA_EXTENSIONS
logger = logging.getLogger(__name__)
@@ -33,7 +37,8 @@ class ExampleImagesFileManager:
}, status=400)
# Get example images path from settings
example_images_path = settings.get('example_images_path')
settings_manager = get_settings_manager()
example_images_path = settings_manager.get('example_images_path')
if not example_images_path:
return web.json_response({
'success': False,
@@ -41,8 +46,12 @@ class ExampleImagesFileManager:
}, status=400)
# Construct folder path for this model
model_folder = os.path.join(example_images_path, model_hash)
model_folder = os.path.abspath(model_folder) # Get absolute path
model_folder = get_model_folder(model_hash)
if not model_folder:
return web.json_response({
'success': False,
'error': 'Failed to resolve example images folder for this model.'
}, status=500)
# Path validation: ensure model_folder is under example_images_path
if not model_folder.startswith(os.path.abspath(example_images_path)):
@@ -101,7 +110,8 @@ class ExampleImagesFileManager:
}, status=400)
# Get example images path from settings
example_images_path = settings.get('example_images_path')
settings_manager = get_settings_manager()
example_images_path = settings_manager.get('example_images_path')
if not example_images_path:
return web.json_response({
'success': False,
@@ -109,8 +119,13 @@ class ExampleImagesFileManager:
}, status=400)
# Construct folder path for this model
model_folder = os.path.join(example_images_path, model_hash)
model_folder = get_model_folder(model_hash)
if not model_folder:
return web.json_response({
'success': False,
'error': 'Failed to resolve example images folder for this model'
}, status=500)
# Check if folder exists
if not os.path.exists(model_folder):
return web.json_response({
@@ -128,9 +143,10 @@ class ExampleImagesFileManager:
file_ext = os.path.splitext(file)[1].lower()
if (file_ext in SUPPORTED_MEDIA_EXTENSIONS['images'] or
file_ext in SUPPORTED_MEDIA_EXTENSIONS['videos']):
relative_path = get_model_relative_path(model_hash)
files.append({
'name': file,
'path': f'/example_images_static/{model_hash}/{file}',
'path': f'/example_images_static/{relative_path}/{file}',
'extension': file_ext,
'is_video': file_ext in SUPPORTED_MEDIA_EXTENSIONS['videos']
})
@@ -169,15 +185,21 @@ class ExampleImagesFileManager:
}, status=400)
# Get example images path from settings
example_images_path = settings.get('example_images_path')
settings_manager = get_settings_manager()
example_images_path = settings_manager.get('example_images_path')
if not example_images_path:
return web.json_response({
'has_images': False
})
# Construct folder path for this model
model_folder = os.path.join(example_images_path, model_hash)
model_folder = get_model_folder(model_hash)
if not model_folder:
return web.json_response({
'has_images': False,
'error': 'Failed to resolve example images folder for this model'
})
# Check if folder exists
if not os.path.exists(model_folder) or not os.path.isdir(model_folder):
return web.json_response({

View File

@@ -1,12 +1,13 @@
import logging
import os
import re
from typing import TYPE_CHECKING, Any, Dict, Optional
from ..recipes.constants import GEN_PARAM_KEYS
from ..services.metadata_service import get_default_metadata_provider, get_metadata_provider
from ..services.metadata_sync_service import MetadataSyncService
from ..services.preview_asset_service import PreviewAssetService
from ..services.settings_manager import settings
from ..services.settings_manager import get_settings_manager
from ..services.downloader import get_downloader
from ..utils.constants import SUPPORTED_MEDIA_EXTENSIONS
from ..utils.exif_utils import ExifUtils
@@ -20,13 +21,46 @@ _preview_service = PreviewAssetService(
exif_utils=ExifUtils,
)
_metadata_sync_service = MetadataSyncService(
metadata_manager=MetadataManager,
preview_service=_preview_service,
settings=settings,
default_metadata_provider_factory=get_default_metadata_provider,
metadata_provider_selector=get_metadata_provider,
)
_metadata_sync_service: MetadataSyncService | None = None
_metadata_sync_service_settings: Optional["SettingsManager"] = None
if TYPE_CHECKING: # pragma: no cover - import for type checkers only
from ..services.settings_manager import SettingsManager
def _build_metadata_sync_service(settings_manager: "SettingsManager") -> MetadataSyncService:
"""Construct a metadata sync service bound to the provided settings."""
return MetadataSyncService(
metadata_manager=MetadataManager,
preview_service=_preview_service,
settings=settings_manager,
default_metadata_provider_factory=get_default_metadata_provider,
metadata_provider_selector=get_metadata_provider,
)
def _get_metadata_sync_service() -> MetadataSyncService:
"""Return the shared metadata sync service, initialising it lazily."""
global _metadata_sync_service, _metadata_sync_service_settings
settings_manager = get_settings_manager()
if isinstance(_metadata_sync_service, MetadataSyncService):
if _metadata_sync_service_settings is not settings_manager:
_metadata_sync_service = _build_metadata_sync_service(settings_manager)
_metadata_sync_service_settings = settings_manager
elif _metadata_sync_service is None:
_metadata_sync_service = _build_metadata_sync_service(settings_manager)
_metadata_sync_service_settings = settings_manager
else:
# Tests may inject stand-ins that do not match the sync service type. Preserve
# those injections while still updating our cached settings reference so the
# next real service instantiation uses the current configuration.
_metadata_sync_service_settings = settings_manager
return _metadata_sync_service
class MetadataUpdater:
@@ -71,7 +105,8 @@ class MetadataUpdater:
async def update_cache_func(old_path, new_path, metadata):
return await scanner.update_single_model_cache(old_path, new_path, metadata)
success, error = await _metadata_sync_service.fetch_and_update_model(
await MetadataManager.hydrate_model_data(model_data)
success, error = await _get_metadata_sync_service().fetch_and_update_model(
sha256=model_hash,
file_path=file_path,
model_data=model_data,
@@ -95,21 +130,35 @@ class MetadataUpdater:
@staticmethod
async def get_updated_model(model_hash, scanner):
"""Get updated model data
Args:
model_hash: SHA256 hash of the model
scanner: Scanner instance
Returns:
dict: Updated model data or None if not found
"""
"""Load the most recent metadata for a model identified by hash."""
cache = await scanner.get_cached_data()
target = None
for item in cache.raw_data:
if item.get('sha256') == model_hash:
return item
return None
target = item
break
if not target:
return None
file_path = target.get('file_path')
if not file_path:
return target
model_cls = getattr(scanner, 'model_class', None)
if model_cls is None:
metadata, should_skip = await MetadataManager.load_metadata(file_path)
else:
metadata, should_skip = await MetadataManager.load_metadata(file_path, model_cls)
if should_skip or metadata is None:
return target
rich_metadata = metadata.to_dict()
rich_metadata.setdefault('folder', target.get('folder', ''))
return rich_metadata
@staticmethod
async def update_metadata_from_local_examples(model_hash, model, scanner_type, scanner, model_dir):
"""Update model metadata with local example image information
@@ -137,16 +186,16 @@ class MetadataUpdater:
if is_supported:
local_images_paths.append(file_path)
await MetadataManager.hydrate_model_data(model)
civitai_data = model.setdefault('civitai', {})
# Check if metadata update is needed (no civitai field or empty images)
needs_update = not model.get('civitai') or not model.get('civitai', {}).get('images')
needs_update = not civitai_data or not civitai_data.get('images')
if needs_update and local_images_paths:
logger.debug(f"Found {len(local_images_paths)} local example images for {model.get('model_name')}, updating metadata")
# Create or get civitai field
if not model.get('civitai'):
model['civitai'] = {}
# Create images array
images = []
@@ -181,16 +230,13 @@ class MetadataUpdater:
images.append(image_entry)
# Update the model's civitai.images field
model['civitai']['images'] = images
civitai_data['images'] = images
# Save metadata to .metadata.json file
file_path = model.get('file_path')
try:
# Create a copy of model data without 'folder' field
model_copy = model.copy()
model_copy.pop('folder', None)
# Write metadata to file
await MetadataManager.save_metadata(file_path, model_copy)
logger.info(f"Saved metadata for {model.get('model_name')}")
except Exception as e:
@@ -223,16 +269,18 @@ class MetadataUpdater:
tuple: (regular_images, custom_images) - Both image arrays
"""
try:
# Ensure civitai field exists in model_data
if not model_data.get('civitai'):
model_data['civitai'] = {}
# Ensure customImages array exists
if not model_data['civitai'].get('customImages'):
model_data['civitai']['customImages'] = []
# Get current customImages array
custom_images = model_data['civitai']['customImages']
await MetadataManager.hydrate_model_data(model_data)
civitai_data = model_data.get('civitai')
if not isinstance(civitai_data, dict):
civitai_data = {}
model_data['civitai'] = civitai_data
custom_images = civitai_data.get('customImages')
if not isinstance(custom_images, list):
custom_images = []
civitai_data['customImages'] = custom_images
# Add new image entry for each imported file
for path_tuple in newly_imported_paths:
@@ -290,11 +338,8 @@ class MetadataUpdater:
file_path = model_data.get('file_path')
if file_path:
try:
# Create a copy of model data without 'folder' field
model_copy = model_data.copy()
model_copy.pop('folder', None)
# Write metadata to file
await MetadataManager.save_metadata(file_path, model_copy)
logger.info(f"Saved metadata for {model_data.get('model_name')}")
except Exception as e:
@@ -305,7 +350,7 @@ class MetadataUpdater:
await scanner.update_single_model_cache(file_path, file_path, model_data)
# Get regular images array (might be None)
regular_images = model_data['civitai'].get('images', [])
regular_images = civitai_data.get('images', [])
# Return both image arrays
return regular_images, custom_images
@@ -406,4 +451,4 @@ class MetadataUpdater:
except Exception as e:
logger.error(f"Error parsing image metadata: {e}", exc_info=True)
return None
return None

View File

@@ -3,8 +3,9 @@ import logging
import os
import re
import json
from ..services.settings_manager import settings
from ..services.settings_manager import get_settings_manager
from ..services.service_registry import ServiceRegistry
from ..utils.example_images_paths import iter_library_roots
from ..utils.metadata_manager import MetadataManager
from ..utils.example_images_processor import ExampleImagesProcessor
from ..utils.constants import SUPPORTED_MEDIA_EXTENSIONS
@@ -13,35 +14,60 @@ logger = logging.getLogger(__name__)
CURRENT_NAMING_VERSION = 2 # Increment this when naming conventions change
class _SettingsProxy:
def __init__(self):
self._manager = None
def _resolve(self):
if self._manager is None:
self._manager = get_settings_manager()
return self._manager
def get(self, *args, **kwargs):
return self._resolve().get(*args, **kwargs)
def __getattr__(self, item):
return getattr(self._resolve(), item)
settings = _SettingsProxy()
class ExampleImagesMigration:
"""Handles migrations for example images naming conventions"""
@staticmethod
async def check_and_run_migrations():
"""Check if migrations are needed and run them in background"""
example_images_path = settings.get('example_images_path')
if not example_images_path or not os.path.exists(example_images_path):
root = settings.get('example_images_path')
if not root or not os.path.exists(root):
logger.debug("No example images path configured or path doesn't exist, skipping migrations")
return
# Check current version from progress file
current_version = 0
progress_file = os.path.join(example_images_path, '.download_progress.json')
if os.path.exists(progress_file):
try:
with open(progress_file, 'r', encoding='utf-8') as f:
progress_data = json.load(f)
current_version = progress_data.get('naming_version', 0)
except Exception as e:
logger.error(f"Failed to load progress file for migration check: {e}")
# If current version is less than target version, start migration
if current_version < CURRENT_NAMING_VERSION:
logger.info(f"Starting example images naming migration from v{current_version} to v{CURRENT_NAMING_VERSION}")
# Start migration in background task
asyncio.create_task(
ExampleImagesMigration.run_migrations(example_images_path, current_version, CURRENT_NAMING_VERSION)
)
for library_name, library_path in iter_library_roots():
if not library_path or not os.path.exists(library_path):
continue
current_version = 0
progress_file = os.path.join(library_path, '.download_progress.json')
if os.path.exists(progress_file):
try:
with open(progress_file, 'r', encoding='utf-8') as f:
progress_data = json.load(f)
current_version = progress_data.get('naming_version', 0)
except Exception as e:
logger.error(f"Failed to load progress file for migration check: {e}")
if current_version < CURRENT_NAMING_VERSION:
logger.info(
"Starting example images naming migration from v%s to v%s for library '%s'",
current_version,
CURRENT_NAMING_VERSION,
library_name,
)
asyncio.create_task(
ExampleImagesMigration.run_migrations(library_path, current_version, CURRENT_NAMING_VERSION)
)
@staticmethod
async def run_migrations(example_images_path, from_version, to_version):

View File

@@ -0,0 +1,226 @@
"""Utility helpers for resolving example image storage paths."""
from __future__ import annotations
import logging
import os
import re
import shutil
from typing import Iterable, List, Optional, Tuple
from ..services.settings_manager import get_settings_manager
_HEX_PATTERN = re.compile(r"[a-fA-F0-9]{64}")
logger = logging.getLogger(__name__)
def _get_configured_libraries() -> List[str]:
"""Return configured library names if multi-library support is enabled."""
settings_manager = get_settings_manager()
libraries = settings_manager.get("libraries")
if isinstance(libraries, dict) and libraries:
return list(libraries.keys())
return []
def get_example_images_root() -> str:
"""Return the root directory configured for example images."""
settings_manager = get_settings_manager()
root = settings_manager.get("example_images_path") or ""
return os.path.abspath(root) if root else ""
def uses_library_scoped_folders() -> bool:
"""Return True when example images should be separated per library."""
libraries = _get_configured_libraries()
return len(libraries) > 1
def sanitize_library_name(library_name: Optional[str]) -> str:
"""Return a filesystem safe library name."""
settings_manager = get_settings_manager()
name = library_name or settings_manager.get_active_library_name() or "default"
safe_name = re.sub(r"[^A-Za-z0-9_.-]", "_", name)
return safe_name or "default"
def get_library_root(library_name: Optional[str] = None) -> str:
"""Return the directory where a library's example images should live."""
root = get_example_images_root()
if not root:
return ""
if uses_library_scoped_folders():
return os.path.join(root, sanitize_library_name(library_name))
return root
def ensure_library_root_exists(library_name: Optional[str] = None) -> str:
"""Ensure the example image directory for a library exists and return it."""
library_root = get_library_root(library_name)
if library_root:
os.makedirs(library_root, exist_ok=True)
return library_root
def get_model_folder(model_hash: str, library_name: Optional[str] = None) -> str:
"""Return the folder path for a model's example images."""
if not model_hash:
return ""
library_root = ensure_library_root_exists(library_name)
if not library_root:
return ""
normalized_hash = (model_hash or "").lower()
resolved_folder = os.path.join(library_root, normalized_hash)
if uses_library_scoped_folders():
legacy_root = get_example_images_root()
legacy_folder = os.path.join(legacy_root, normalized_hash)
if os.path.exists(legacy_folder) and not os.path.exists(resolved_folder):
try:
os.makedirs(library_root, exist_ok=True)
shutil.move(legacy_folder, resolved_folder)
logger.info(
"Migrated legacy example images folder '%s' to '%s'", legacy_folder, resolved_folder
)
except OSError as exc:
logger.error(
"Failed to migrate example images from '%s' to '%s': %s",
legacy_folder,
resolved_folder,
exc,
)
return legacy_folder
return resolved_folder
class ExampleImagePathResolver:
"""Convenience wrapper exposing example image path helpers."""
@staticmethod
def get_model_folder(model_hash: str, library_name: Optional[str] = None) -> str:
"""Return the example image folder for a model, migrating legacy paths."""
return get_model_folder(model_hash, library_name)
@staticmethod
def get_library_root(library_name: Optional[str] = None) -> str:
"""Return the configured library root for example images."""
return get_library_root(library_name)
@staticmethod
def ensure_library_root_exists(library_name: Optional[str] = None) -> str:
"""Ensure the library root exists before writing files."""
return ensure_library_root_exists(library_name)
@staticmethod
def get_model_relative_path(model_hash: str, library_name: Optional[str] = None) -> str:
"""Return the relative path to a model folder from the static mount point."""
return get_model_relative_path(model_hash, library_name)
def get_model_relative_path(model_hash: str, library_name: Optional[str] = None) -> str:
"""Return the relative URL path from the static mount to a model folder."""
root = get_example_images_root()
folder = get_model_folder(model_hash, library_name)
if not root or not folder:
return ""
try:
relative = os.path.relpath(folder, root)
except ValueError:
return ""
return relative.replace("\\", "/")
def iter_library_roots() -> Iterable[Tuple[str, str]]:
"""Yield configured library names and their resolved filesystem roots."""
root = get_example_images_root()
if not root:
return []
libraries = _get_configured_libraries()
if uses_library_scoped_folders():
results: List[Tuple[str, str]] = []
if libraries:
for library in libraries:
results.append((library, get_library_root(library)))
else:
# Fall back to the active library to avoid skipping migrations/cleanup
settings_manager = get_settings_manager()
active = settings_manager.get_active_library_name() or "default"
results.append((active, get_library_root(active)))
return results
settings_manager = get_settings_manager()
active = settings_manager.get_active_library_name() or "default"
return [(active, root)]
def is_hash_folder(name: str) -> bool:
"""Return True if the provided name looks like a model hash folder."""
return bool(_HEX_PATTERN.fullmatch(name or ""))
def is_valid_example_images_root(folder_path: str) -> bool:
"""Check whether a folder looks like a dedicated example images root."""
try:
items = os.listdir(folder_path)
except OSError:
return False
for item in items:
item_path = os.path.join(folder_path, item)
if item == ".download_progress.json" and os.path.isfile(item_path):
continue
if os.path.isdir(item_path):
if is_hash_folder(item):
continue
if item == "_deleted":
# Allow cleanup staging folders
continue
# When multi-library mode is active we expect nested hash folders
if uses_library_scoped_folders():
if _library_folder_has_only_hash_dirs(item_path):
continue
return False
return True
def _library_folder_has_only_hash_dirs(path: str) -> bool:
"""Return True when a library subfolder only contains hash folders or metadata files."""
try:
for entry in os.listdir(path):
entry_path = os.path.join(path, entry)
if entry == ".download_progress.json" and os.path.isfile(entry_path):
continue
if entry == "_deleted" and os.path.isdir(entry_path):
continue
if not os.path.isdir(entry_path) or not is_hash_folder(entry):
return False
except OSError:
return False
return True

View File

@@ -6,7 +6,8 @@ import string
from aiohttp import web
from ..utils.constants import SUPPORTED_MEDIA_EXTENSIONS
from ..services.service_registry import ServiceRegistry
from ..services.settings_manager import settings
from ..services.settings_manager import get_settings_manager
from ..utils.example_images_paths import get_model_folder, get_model_relative_path
from .example_images_metadata import MetadataUpdater
from ..utils.metadata_manager import MetadataManager
@@ -317,7 +318,7 @@ class ExampleImagesProcessor:
try:
# Get example images path
example_images_path = settings.get('example_images_path')
example_images_path = get_settings_manager().get('example_images_path')
if not example_images_path:
raise ExampleImagesValidationError('No example images path configured')
@@ -346,7 +347,9 @@ class ExampleImagesProcessor:
)
# Create model folder
model_folder = os.path.join(example_images_path, model_hash)
model_folder = get_model_folder(model_hash)
if not model_folder:
raise ExampleImagesImportError('Failed to resolve model folder for example images')
os.makedirs(model_folder, exist_ok=True)
imported_files = []
@@ -383,7 +386,7 @@ class ExampleImagesProcessor:
# Add to imported files list
imported_files.append({
'name': new_filename,
'path': f'/example_images_static/{model_hash}/{new_filename}',
'path': f'/example_images_static/{get_model_relative_path(model_hash)}/{new_filename}',
'extension': file_ext,
'is_video': file_ext in SUPPORTED_MEDIA_EXTENSIONS['videos']
})
@@ -439,7 +442,7 @@ class ExampleImagesProcessor:
}, status=400)
# Get example images path
example_images_path = settings.get('example_images_path')
example_images_path = get_settings_manager().get('example_images_path')
if not example_images_path:
return web.json_response({
'success': False,
@@ -472,15 +475,17 @@ class ExampleImagesProcessor:
'error': f"Model with hash {model_hash} not found in cache"
}, status=404)
# Check if model has custom images
if not model_data.get('civitai', {}).get('customImages'):
await MetadataManager.hydrate_model_data(model_data)
civitai_data = model_data.setdefault('civitai', {})
custom_images = civitai_data.get('customImages')
if not isinstance(custom_images, list) or not custom_images:
return web.json_response({
'success': False,
'error': f"Model has no custom images"
}, status=404)
# Find the custom image with matching short_id
custom_images = model_data['civitai']['customImages']
matching_image = None
new_custom_images = []
@@ -497,7 +502,12 @@ class ExampleImagesProcessor:
}, status=404)
# Find and delete the actual file
model_folder = os.path.join(example_images_path, model_hash)
model_folder = get_model_folder(model_hash)
if not model_folder:
return web.json_response({
'success': False,
'error': 'Failed to resolve model folder for example images'
}, status=500)
file_deleted = False
if os.path.exists(model_folder):
@@ -519,17 +529,15 @@ class ExampleImagesProcessor:
logger.warning(f"File for custom example with id {short_id} not found, but metadata will still be updated")
# Update metadata
model_data['civitai']['customImages'] = new_custom_images
civitai_data['customImages'] = new_custom_images
model_data.setdefault('civitai', {})['customImages'] = new_custom_images
# Save updated metadata to file
file_path = model_data.get('file_path')
if file_path:
try:
# Create a copy of model data without 'folder' field
model_copy = model_data.copy()
model_copy.pop('folder', None)
# Write metadata to file
await MetadataManager.save_metadata(file_path, model_copy)
logger.debug(f"Saved updated metadata for {model_data.get('model_name')}")
except Exception as e:
@@ -543,7 +551,7 @@ class ExampleImagesProcessor:
await scanner.update_single_model_cache(file_path, file_path, model_data)
# Get regular images array (might be None)
regular_images = model_data['civitai'].get('images', [])
regular_images = civitai_data.get('images', [])
return web.json_response({
'success': True,
@@ -560,4 +568,4 @@ class ExampleImagesProcessor:
}, status=500)

View File

@@ -4,7 +4,7 @@ import logging
from typing import Optional
from io import BytesIO
import os
from PIL import Image
from PIL import Image, PngImagePlugin
logger = logging.getLogger(__name__)
@@ -86,9 +86,10 @@ class ExifUtils:
# For PNG, try to update parameters directly
if img_format == 'PNG':
# We'll save with parameters in the PNG info
info_dict = {'parameters': metadata}
img.save(image_path, format='PNG', pnginfo=info_dict)
# Use PngInfo instead of plain dictionary
png_info = PngImagePlugin.PngInfo()
png_info.add_text("parameters", metadata)
img.save(image_path, format='PNG', pnginfo=png_info)
return image_path
# For WebP format, use PIL's exif parameter directly

View File

@@ -2,7 +2,7 @@ from datetime import datetime
import os
import json
import logging
from typing import Dict, Optional, Type, Union
from typing import Any, Dict, Optional, Type, Union
from .models import BaseModelMetadata, LoraMetadata
from .file_utils import normalize_path, find_preview_file, calculate_sha256
@@ -53,6 +53,70 @@ class MetadataManager:
error_type = "Invalid JSON" if isinstance(e, json.JSONDecodeError) else "Parse error"
logger.error(f"{error_type} in metadata file: {metadata_path}. Error: {str(e)}. Skipping model to preserve existing data.")
return None, True # should_skip = True
@staticmethod
async def load_metadata_payload(file_path: str) -> Dict:
"""
Load metadata and return it as a dictionary, including any unknown fields.
Falls back to reading the raw JSON file if parsing into a model class fails.
"""
payload: Dict = {}
metadata_obj, should_skip = await MetadataManager.load_metadata(file_path)
if metadata_obj:
payload = metadata_obj.to_dict()
unknown_fields = getattr(metadata_obj, "_unknown_fields", None)
if isinstance(unknown_fields, dict):
payload.update(unknown_fields)
else:
if not should_skip:
metadata_path = (
file_path
if file_path.endswith(".metadata.json")
else f"{os.path.splitext(file_path)[0]}.metadata.json"
)
if os.path.exists(metadata_path):
try:
with open(metadata_path, "r", encoding="utf-8") as handle:
raw = json.load(handle)
if isinstance(raw, dict):
payload = raw
except json.JSONDecodeError:
logger.warning(
"Failed to parse metadata file %s while loading payload",
metadata_path,
)
except Exception as exc: # pragma: no cover - defensive logging
logger.warning("Failed to read metadata file %s: %s", metadata_path, exc)
if not isinstance(payload, dict):
payload = {}
if file_path:
payload.setdefault("file_path", normalize_path(file_path))
return payload
@staticmethod
async def hydrate_model_data(model_data: Dict[str, Any]) -> Dict[str, Any]:
"""
Replace the provided model data with the authoritative payload from disk.
Preserves the cached folder entry if present.
"""
file_path = model_data.get("file_path")
if not file_path:
return model_data
folder = model_data.get("folder")
payload = await MetadataManager.load_metadata_payload(file_path)
if folder is not None:
payload["folder"] = folder
model_data.clear()
model_data.update(payload)
return model_data
@staticmethod
async def save_metadata(path: str, metadata: Union[BaseModelMetadata, Dict]) -> bool:

View File

@@ -18,18 +18,22 @@ class BaseModelMetadata:
preview_nsfw_level: int = 0 # NSFW level of the preview image
notes: str = "" # Additional notes
from_civitai: bool = True # Whether from Civitai
civitai: Optional[Dict] = None # Civitai API data if available
civitai: Dict[str, Any] = field(default_factory=dict) # Civitai API data if available
tags: List[str] = None # Model tags
modelDescription: str = "" # Full model description
civitai_deleted: bool = False # Whether deleted from Civitai
favorite: bool = False # Whether the model is a favorite
exclude: bool = False # Whether to exclude this model from the cache
db_checked: bool = False # Whether checked in archive DB
metadata_source: Optional[str] = None # Last provider that supplied metadata
last_checked_at: float = 0 # Last checked timestamp
_unknown_fields: Dict[str, Any] = field(default_factory=dict, repr=False, compare=False) # Store unknown fields
def __post_init__(self):
# Initialize empty lists to avoid mutable default parameter issue
if self.civitai is None:
self.civitai = {}
if self.tags is None:
self.tags = []

View File

@@ -0,0 +1,90 @@
"""Utilities for locating and migrating the LoRA Manager settings file."""
from __future__ import annotations
import logging
import os
import shutil
from typing import Optional
from platformdirs import user_config_dir
APP_NAME = "ComfyUI-LoRA-Manager"
_LOGGER = logging.getLogger(__name__)
def get_project_root() -> str:
"""Return the root directory of the project repository."""
return os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
def get_legacy_settings_path() -> str:
"""Return the legacy location of ``settings.json`` within the project tree."""
return os.path.join(get_project_root(), "settings.json")
def get_settings_dir(create: bool = True) -> str:
"""Return the user configuration directory for the application.
Args:
create: Whether to create the directory if it does not already exist.
Returns:
The absolute path to the user configuration directory.
"""
config_dir = user_config_dir(APP_NAME, appauthor=False)
if create:
os.makedirs(config_dir, exist_ok=True)
return config_dir
def get_settings_file_path(create_dir: bool = True) -> str:
"""Return the path to ``settings.json`` in the user configuration directory."""
return os.path.join(get_settings_dir(create=create_dir), "settings.json")
def ensure_settings_file(logger: Optional[logging.Logger] = None) -> str:
"""Ensure the settings file resides in the user configuration directory.
If a legacy ``settings.json`` is detected in the project root it is migrated to
the platform-specific user configuration folder. The caller receives the path
to the settings file irrespective of whether a migration was needed.
Args:
logger: Optional logger used for migration messages. Falls back to a
module level logger when omitted.
Returns:
The absolute path to ``settings.json`` in the user configuration folder.
"""
logger = logger or _LOGGER
target_path = get_settings_file_path(create_dir=True)
preferred_dir = user_config_dir(APP_NAME, appauthor=False)
preferred_path = os.path.join(preferred_dir, "settings.json")
if os.path.abspath(target_path) != os.path.abspath(preferred_path):
os.makedirs(preferred_dir, exist_ok=True)
target_path = preferred_path
legacy_path = get_legacy_settings_path()
if os.path.exists(legacy_path) and not os.path.exists(target_path):
try:
os.makedirs(os.path.dirname(target_path), exist_ok=True)
shutil.move(legacy_path, target_path)
logger.info("Migrated settings.json to %s", target_path)
except Exception as exc: # pragma: no cover - defensive fallback path
logger.warning("Failed to move legacy settings.json: %s", exc)
try:
shutil.copy2(legacy_path, target_path)
logger.info("Copied legacy settings.json to %s", target_path)
except Exception as copy_exc: # pragma: no cover - defensive fallback path
logger.error("Could not migrate settings.json: %s", copy_exc)
return target_path

104
py/utils/tag_priorities.py Normal file
View File

@@ -0,0 +1,104 @@
"""Helpers for parsing and resolving priority tag configurations."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Dict, Iterable, List, Optional, Sequence, Set
@dataclass(frozen=True)
class PriorityTagEntry:
"""A parsed priority tag configuration entry."""
canonical: str
aliases: Set[str]
@property
def normalized_aliases(self) -> Set[str]:
return {alias.lower() for alias in self.aliases}
def _normalize_alias(alias: str) -> str:
return alias.strip()
def parse_priority_tag_string(config: str | None) -> List[PriorityTagEntry]:
"""Parse the user-facing priority tag string into structured entries."""
if not config:
return []
entries: List[PriorityTagEntry] = []
seen_canonicals: Set[str] = set()
for raw_entry in _split_priority_entries(config):
canonical, aliases = _parse_priority_entry(raw_entry)
if not canonical:
continue
normalized_canonical = canonical.lower()
if normalized_canonical in seen_canonicals:
# Skip duplicate canonicals while preserving first occurrence priority
continue
seen_canonicals.add(normalized_canonical)
alias_set = {canonical, *aliases}
cleaned_aliases = {_normalize_alias(alias) for alias in alias_set if _normalize_alias(alias)}
if not cleaned_aliases:
continue
entries.append(PriorityTagEntry(canonical=canonical, aliases=cleaned_aliases))
return entries
def _split_priority_entries(config: str) -> List[str]:
# Split on commas while respecting that users may add new lines for readability
parts = []
for chunk in config.split('\n'):
parts.extend(chunk.split(','))
return [part.strip() for part in parts if part.strip()]
def _parse_priority_entry(entry: str) -> tuple[str, Set[str]]:
if '(' in entry and entry.endswith(')'):
canonical, raw_aliases = entry.split('(', 1)
canonical = canonical.strip()
alias_section = raw_aliases[:-1] # drop trailing ')'
aliases = {alias.strip() for alias in alias_section.split('|') if alias.strip()}
return canonical, aliases
if '(' in entry and not entry.endswith(')'):
# Malformed entry; treat as literal canonical to avoid surprises
entry = entry.replace('(', '').replace(')', '')
canonical = entry.strip()
return canonical, set()
def resolve_priority_tag(
tags: Sequence[str] | Iterable[str],
entries: Sequence[PriorityTagEntry],
) -> Optional[str]:
"""Resolve the first matching canonical priority tag for the provided tags."""
tag_lookup: Dict[str, str] = {}
for tag in tags:
if not isinstance(tag, str):
continue
normalized = tag.lower()
if normalized not in tag_lookup:
tag_lookup[normalized] = tag
for entry in entries:
for alias in entry.normalized_aliases:
if alias in tag_lookup:
return entry.canonical
return None
def collect_canonical_tags(entries: Iterable[PriorityTagEntry]) -> List[str]:
"""Return the ordered list of canonical tags from the parsed entries."""
return [entry.canonical for entry in entries]

View File

@@ -11,11 +11,21 @@ from ..config import config
from ..services.service_registry import ServiceRegistry
# Check if running in standalone mode
standalone_mode = os.environ.get("HF_HUB_DISABLE_TELEMETRY", "0") == "0"
standalone_mode = os.environ.get("LORA_MANAGER_STANDALONE", "0") == "1" or os.environ.get("HF_HUB_DISABLE_TELEMETRY", "0") == "0"
# Define constants locally to avoid dependency on conditional imports
MODELS = "models"
LORAS = "loras"
if not standalone_mode:
from ..metadata_collector.metadata_registry import MetadataRegistry
from ..metadata_collector.constants import MODELS, LORAS
# Import constants from metadata_collector to ensure consistency, but we have fallbacks defined above
try:
from ..metadata_collector.constants import MODELS as _MODELS, LORAS as _LORAS
MODELS = _MODELS
LORAS = _LORAS
except ImportError:
pass # Use the local definitions
logger = logging.getLogger(__name__)

View File

@@ -3,8 +3,7 @@ import os
from typing import Dict
from ..services.service_registry import ServiceRegistry
from ..config import config
from ..services.settings_manager import settings
from .constants import CIVITAI_MODEL_TAGS
from ..services.settings_manager import get_settings_manager
import asyncio
def get_lora_info(lora_name):
@@ -143,7 +142,8 @@ def calculate_relative_path_for_model(model_data: Dict, model_type: str = 'lora'
Relative path string (empty string for flat structure)
"""
# Get path template from settings for specific model type
path_template = settings.get_download_path_template(model_type)
settings_manager = get_settings_manager()
path_template = settings_manager.get_download_path_template(model_type)
# If template is empty, return empty path (flat structure)
if not path_template:
@@ -166,19 +166,10 @@ def calculate_relative_path_for_model(model_data: Dict, model_type: str = 'lora'
model_tags = model_data.get('tags', [])
# Apply mapping if available
base_model_mappings = settings.get('base_model_path_mappings', {})
base_model_mappings = settings_manager.get('base_model_path_mappings', {})
mapped_base_model = base_model_mappings.get(base_model, base_model)
# Find the first Civitai model tag that exists in model_tags
first_tag = ''
for civitai_tag in CIVITAI_MODEL_TAGS:
if civitai_tag in model_tags:
first_tag = civitai_tag
break
# If no Civitai model tag found, fallback to first tag
if not first_tag and model_tags:
first_tag = model_tags[0]
first_tag = settings_manager.resolve_priority_tag_for_model(model_tags, model_type)
if not first_tag:
first_tag = 'no tags' # Default if no tags available
@@ -189,6 +180,9 @@ def calculate_relative_path_for_model(model_data: Dict, model_type: str = 'lora'
formatted_path = formatted_path.replace('{first_tag}', first_tag)
formatted_path = formatted_path.replace('{author}', author)
if model_type == 'embedding':
formatted_path = formatted_path.replace(' ', '_')
return formatted_path
def remove_empty_dirs(path):

View File

@@ -1,7 +1,7 @@
[project]
name = "comfyui-lora-manager"
description = "Revolutionize your workflow with the ultimate LoRA companion for ComfyUI!"
version = "0.9.4"
version = "0.9.8"
license = {file = "LICENSE"}
dependencies = [
"aiohttp",
@@ -13,7 +13,8 @@ dependencies = [
"toml",
"natsort",
"GitPython",
"aiosqlite"
"aiosqlite",
"platformdirs"
]
[project.urls]

View File

@@ -0,0 +1,134 @@
{
"id": 1746460,
"name": "Mixplin Style [Illustrious]",
"type": "LORA",
"description": "description",
"username": "Ty_Lee",
"downloadCount": 4207,
"favoriteCount": 0,
"commentCount": 8,
"ratingCount": 0,
"rating": 0,
"is_nsfw": true,
"nsfw_level": 31,
"createdAt": "2025-07-06T01:51:42.859Z",
"updatedAt": "2025-10-10T23:15:26.714Z",
"deletedAt": null,
"tags": [
"art",
"style",
"artist style",
"styles",
"mixplin",
"artiststyle"
],
"creator_id": "Ty_Lee",
"creator_username": "Ty_Lee",
"creator_name": "Ty_Lee",
"creator_url": "/users/Ty_Lee",
"versions": [
{
"id": 2042594,
"name": "v2.0",
"href": "/models/1746460?modelVersionId=2042594"
},
{
"id": 1976567,
"name": "v1.0",
"href": "/models/1746460?modelVersionId=1976567"
}
],
"version": {
"id": 1976567,
"modelId": 1746460,
"name": "v1.0",
"baseModel": "Illustrious",
"baseModelType": "Standard",
"description": null,
"downloadCount": 437,
"ratingCount": 0,
"rating": 0,
"is_nsfw": true,
"nsfw_level": 31,
"createdAt": "2025-07-05T10:17:28.716Z",
"updatedAt": "2025-10-10T23:15:26.756Z",
"deletedAt": null,
"files": [
{
"id": 1874043,
"name": "mxpln-illustrious-ty_lee.safetensors",
"type": "Model",
"sizeKB": 223124.37109375,
"downloadUrl": "https://civitai.com/api/download/models/1976567",
"modelId": 1746460,
"modelName": "Mixplin Style [Illustrious]",
"modelVersionId": 1976567,
"is_nsfw": true,
"nsfw_level": 31,
"sha256": "e2b7a280d6539556f23f380b3f71e4e22bc4524445c4c96526e117c6005c6ad3",
"createdAt": "2025-07-05T10:17:28.716Z",
"updatedAt": "2025-10-10T23:15:26.766Z",
"is_primary": false,
"mirrors": [
{
"filename": "mxpln-illustrious-ty_lee.safetensors",
"url": "https://civitai.com/api/download/models/1976567",
"source": "civitai",
"model_id": 1746460,
"model_version_id": 1976567,
"deletedAt": null,
"is_gated": false,
"is_paid": false
}
]
}
],
"images": [
{
"id": 86403595,
"url": "https://img.genur.art/sig/width:450/quality:85/aHR0cHM6Ly9jLmdlbnVyLmFydC9hNmE3Njc2YS0wMWQ3LTQ1YzAtOWEzYS1mNWJiYTU4MDNiMDE=",
"nsfwLevel": 1,
"width": 1560,
"height": 2280,
"hash": "U7G8Zp0w02%IA6%N00-;D]-W~VNG0nMw-.IV",
"type": "image",
"minor": false,
"poi": false,
"hasMeta": true,
"hasPositivePrompt": true,
"onSite": false,
"remixOfId": null,
"image_url": "https://img.genur.art/sig/width:450/quality:85/aHR0cHM6Ly9jLmdlbnVyLmFydC9hNmE3Njc2YS0wMWQ3LTQ1YzAtOWEzYS1mNWJiYTU4MDNiMDE=",
"link": "https://genur.art/posts/86403595"
}
],
"trigger": [
"mxpln"
],
"allow_download": true,
"download_url": "/api/download/models/1976567",
"platform_url": "https://civitai.com/models/1746460?modelVersionId=1976567",
"civitai_model_id": 1746460,
"civitai_model_version_id": 1976567,
"href": "/models/1746460?modelVersionId=1976567",
"mirrors": [
{
"platform": "tensorart",
"href": "/tensorart/models/904473536033245448/versions/904473536033245448",
"platform_url": "https://tensor.art/models/904473536033245448",
"name": "Mixplin Style MXP",
"version_name": "Mixplin",
"id": "904473536033245448",
"version_id": "904473536033245448"
}
]
},
"platform": "civitai",
"platform_name": "CivitAI",
"meta": {
"title": "Mixplin Style [Illustrious] - v1.0 - CivitAI Archive",
"description": "Mixplin Style [Illustrious] v1.0 is a Illustrious LORA AI model created by Ty_Lee for generating images of art, style, artist style, styles, mixplin, artiststyle",
"image": "https://img.genur.art/sig/width:450/quality:85/aHR0cHM6Ly9jLmdlbnVyLmFydC9hNmE3Njc2YS0wMWQ3LTQ1YzAtOWEzYS1mNWJiYTU4MDNiMDE=",
"canonical": "https://civarchive.com/models/1746460?modelVersionId=1976567"
}
}

Some files were not shown because too many files have changed in this diff Show More