Commit Graph

68 Commits

Author SHA1 Message Date
Will Miao 84146b62fd feat(other-models): announce the feature only when folders are available
Other Models management is opt-in and its folders come from
folder_paths.get_folder_paths(). In plugin mode ComfyUI registers vae,
upscale_models, text_encoders, clip_vision and controlnet out of the box, so
enabling the feature works immediately. Standalone only knows the keys present
in settings.json.folder_paths, and that file is edited by hand - there is no UI
for those keys - so a standalone user who followed the announcement banner
reached "Enable Other Models" and then an empty page.

Gate the announcement on the capability instead of on how the process was
started:

- Config.get_other_models_availability() probes every canonical other key
  (legacy clip collapses into text_encoders where the host exposes
  map_legacy) and reports which sub_types resolve to a folder that exists on
  disk. It deliberately ignores enable_other_models: the question is "could
  this work here at all?". An empty folder counts, because CivitAI downloads
  can target it.
- /api/lm/settings exposes it as the derived, non-persisted
  other_models_paths_available flag; a probe failure yields null and the
  banner fails open.
- BannerService only registers the announcement when the flag is not false.
  `=== false` (not falsy) keeps a cached/older payload working, and nothing is
  written to dismissed_banners, so the banner can return once folders exist.
- The Other page grows an "enabled but nothing to scan" empty state driven by
  config.other_roots, showing the settings.json snippet for standalone and a
  pointer to ComfyUI model paths otherwise, plus an Open Settings action. It
  also covers the corner where only a non-default sub_type has a folder.

Translate the six other.noPaths.* keys into all nine locales and record the
new "folder key" / "on disk" terminology in the i18n guidelines.

Backend tests and pytest tests/i18n could not run in this environment (no
pytest/platformdirs); the probe was exercised against a stubbed folder_paths.
Frontend: 120 files / 1101 JS tests passed.
2026-09-13 21:47:16 +08:00
Will Miao 4d87ae7637 fix(other-models): stop warning about legacy folder keys that alias
Enabling Other Models logged two warnings on a stock ComfyUI install:

  Detected the same folder '.../clip' under multiple other-model categories
  ('.../clip' is already mapped). Keeping the first category; please fix
  your path configuration.

Nothing was wrong with the configuration. ComfyUI's folder_paths rewrites
legacy names before every access (map_legacy: clip -> text_encoders,
unet -> diffusion_models) and registers both legacy directories under the
canonical key, so get_folder_paths("clip") returns exactly the same list as
get_folder_paths("text_encoders"). Both keys are in the enabled allow-list,
so the second pass hit the overlap guard for every text-encoder folder and
printed advice the user cannot act on. The path list itself was correct
(deduped), only the message was wrong.

- Config._collapse_legacy_folder_keys() drops a key when the host exposes
  map_legacy and resolves it to another queried key. That is provably
  lossless: an empty canonical list implies an empty alias list. The
  standalone MockFolderPaths has no map_legacy and its keys are independent
  settings.json entries, so every key is still queried there.
- _prepare_other_paths() now tracks the claiming sub_type alongside the
  business path and downgrades a same-sub_type duplicate to debug, keeping
  the warning for a genuine cross-category collision (and naming the other
  category in the message).

Regression tests cover the aliased-key layout (no warning, no redundant
query, both folders still managed) and the same-sub_type duplicate, and the
opt-in test is parametrized over controlnet and clip_vision.
2026-09-13 20:12:45 +08:00
Will Miao 28fbb86dce feat(backend): gate Other Models behind opt-in management toggles
Other Models management is now opt-in: enable_other_models (default false)
plus the enabled_other_sub_types allow-list replace the unreleased additive
enabled_other_folders key.

- config._get_enabled_other_folder_keys() is the single scan gate; a new
  refresh_other_roots() rebuilds roots and preview roots on toggle.
- ModelScanner gains a _should_keep_cached_entry() hydration hook and
  on_library_changed(reconcile=...) so switching a sub_type off drops its
  entries (and hash/autov3 rows) at load time and switching it on rescans.
- OtherScanner filters location-derived entries accordingly.
- Other routes reject every other type while off (or a disabled sub_type) and
  expose an "other_disabled" page flag; download routing returns a disabled
  marker instead of guessing; the download manager refuses other-type
  downloads and default-path routing for switched-off sub_types.
- Doctor / init-status / refresh-all skip the other scanner while off; the
  scanner stays registered so staged pending-deletes still merge.
- Tests updated with explicit opt-in fixtures plus new gating coverage.
2026-09-13 07:59:28 +08:00
Will Miao 27da7b3ca3 feat(backend): add Other model type (VAE/upscaler/text encoder) scanner, service and routes 2026-09-12 11:25:51 +08:00
Will Miao 8e724538bd fix(types): resolve pre-existing basedpyright errors in py/ and standalone.py
Fix ~950 basedpyright errors across the backend:
- Convert ineffective # type: ignore comments to # pyright: ignore[rule]
- Add missing generic type arguments (Dict[str, Any], list[Any], ...)
- Annotate dynamic dict literals and runtime-initialized attributes
- Widen CivitAI provider tuple signatures in recipe parsers
- Remove dead LoraRoutes handlers calling nonexistent LoraService methods
- Suppress unavoidable ServiceRegistry import cycles (basedpyright counts
  function-local imports as cycle edges)
2026-08-08 20:12:52 +08:00
Will Miao 7ee2361e87 fix(config): remove stale 'default' library entry and consolidate example images on startup 2026-07-18 17:25:36 +08:00
Will Miao 02c249917a fix(recipe): ensure custom recipes_path is added to preview allowed roots on startup 2026-07-14 18:15:20 +08:00
Will Miao b22f09bd1d fix(standalone): load extra folder paths from library settings in standalone mode 2026-07-03 19:21:56 +08:00
Will Miao cf898da193 feat(agent): add LLM-powered metadata enrichment system with AgentCLI and PostProcessor
Introduce an agent skill framework for LLM-driven metadata enrichment:

- AgentCLI (py/agent_cli/): in-process wrappers around internal services
  using standard relative imports, eliminating the need for sys.path hacks
- LLMService: centralized BYOK (bring-your-own-key) LLM client supporting
  OpenAI, Ollama, and custom OpenAI-compatible endpoints
- PostProcessor: deterministic engine that applies LLM output via AgentCLI
  (replaces old handler.py + _BASE_MODEL_ALIASES approach)
- SkillRegistry: filesystem-based skill discovery (skill.yaml + prompt.md)
- AgentService: orchestrates skill execution with WebSocket progress
- Frontend AgentManager: WebSocket listeners, skill execution, config UI
- Context menu entries (single + bulk) for "Enrich Metadata (Agent)"
- Settings UI for AI Provider configuration (BYOK)
- Full i18n support across 9 locales

Bug fixes found during review:
- aiohttp.web.json_response: status_code= -> status=
- settings_modal cancelEditApiKey: wrong argument position
- AgentManager.isLlmConfigured: allow Ollama without API key
- PostProcessor._merge_tags: lowercase all tags to match TagUpdateService
2026-07-02 21:27:01 +08:00
Will Miao d9dc0dba8d perf(startup): load extra model paths during Config init to avoid double symlink scan
Move extra folder path resolution from _initialize_services (app.on_startup)
into Config.__init__ via new _load_extra_paths_from_settings() method.
This eliminates a redundant second symlink scan and consolidates all
'Found roots' / 'Found extra roots' logs into one contiguous block
during custom node import, before the ComfyUI server starts.
2026-05-08 14:55:53 +08:00
Will Miao ef4923fd94 fix(settings): normalize default root path comparisons 2026-04-21 09:43:37 +08:00
Will Miao 34f03d6495 fix(settings): preserve extra default roots in comfyui sync 2026-04-20 15:48:30 +08:00
Will Miao db4726a961 feat(recipes): add configurable storage path migration 2026-04-09 15:57:37 +08:00
Will Miao 9bdb337962 fix(settings): enforce valid default model roots 2026-04-01 20:36:37 +08:00
Will Miao 76ee59cdb9 fix(paths): deduplicate LoRA path overlap (#871) 2026-03-27 17:35:24 +08:00
Will Miao 2dae4c1291 fix: isolate extra unet paths from checkpoints to prevent type misclassification
Refactor _prepare_checkpoint_paths() to return a tuple instead of having
side effects on instance variables. This prevents extra unet paths from
being incorrectly classified as checkpoints when processing extra paths.

- Changed return type from List[str] to Tuple[List[str], List[str], List[str]]
  (all_paths, checkpoint_roots, unet_roots)
- Updated _init_checkpoint_paths() and _apply_library_paths() callers
- Fixed extra paths processing to properly isolate main and extra roots
- Updated test_checkpoint_path_overlap.py tests for new API

This ensures models in extra unet paths are correctly identified as
diffusion_model type and don't appear in checkpoints list.
2026-03-17 22:03:57 +08:00
Will Miao c9e5ea42cb Fix null-safety issues and apply code formatting
Bug fixes:
- Add null guards for base_models_roots/embeddings_roots in backup cleanup
- Fix null-safety initialization of extra_unet_roots

Formatting:
- Apply consistent code style across Python files
- Fix line wrapping, quote consistency, and trailing commas
- Add type ignore comments for dynamic/platform-specific code
2026-02-28 21:38:41 +08:00
Will Miao d50bbe71c2 fix(extra-folder-paths): fix extra folder paths support for checkpoint and unet roots
- Fix config.py: save and restore main paths when processing extra folder paths to prevent
  _prepare_checkpoint_paths from overwriting checkpoints_roots and unet_roots
- Fix lora_manager.py: apply library settings during initialization to load extra folder paths
  in ComfyUI plugin mode
- Fix checkpoint_routes.py: merge checkpoints/unet roots with extra paths in API endpoints
- Add logging for extra folder paths

Fixes issue where extra folder paths were not recognized for checkpoints and unet models.
2026-02-27 10:27:29 +08:00
Will Miao 87b462192b feat: Add extra folder paths support for LoRA Manager
Introduce extra_folder_paths feature to allow users to add additional
model roots that are managed by LoRA Manager but not shared with ComfyUI.

Changes:
- Add extra_folder_paths support in SettingsManager (stored per library)
- Add extra path attributes in Config class (extra_loras_roots, etc.)
- Merge folder_paths with extra_folder_paths when applying library settings
- Update LoraScanner, CheckpointScanner, EmbeddingScanner to include
  extra paths in their model roots
- Add comprehensive tests for the new functionality

This enables users to manage models from additional directories without
modifying ComfyUI's model folder configuration.
2026-02-25 18:16:17 +08:00
Will Miao 7bcf4e4491 feat(config): discover deep symlinks dynamically when accessing previews 2026-02-04 00:16:59 +08:00
Will Miao 655d3cab71 fix(config): prioritize checkpoints over unet when paths overlap, #799
When checkpoints and unet folders point to the same physical location
(via symlinks), prioritize checkpoints for backward compatibility.

This prevents the 'Failed to load Checkpoint root' error that users
experience when they have incorrectly configured their ComfyUI paths.

Changes:
- Detect overlapping real paths between checkpoints and unet
- Log warning to inform users of the configuration issue
- Remove overlapping paths from unet_map, keeping checkpoints

Fixes #<issue-number>
2026-02-03 18:27:42 +08:00
Will Miao 553e1868e1 perf(config): limit symlink scan to first level for faster startup
Replace recursive directory traversal with first-level-only symlink scanning
to fix severe performance issues on large model collections (220K+ files).

- Rename _scan_directory_links to _scan_first_level_symlinks
- Only scan symlinks directly under each root directory
- Skip traversal of normal subdirectories entirely
- Update tests to reflect first-level behavior
- Add test_deep_symlink_not_scanned to document intentional limitation

Startup time reduced from 15+ minutes to seconds for affected users.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-01 12:42:35 +08:00
pixelpaws 93cd0b54dc Merge branch 'main' into fix-symlink 2026-01-26 17:29:31 +08:00
Will Miao 7b0c6c8bab refactor(cache): reorganize cache directory structure with automatic legacy cleanup
- Centralize cache path resolution in new py/utils/cache_paths.py module
- Migrate legacy cache files to organized structure: {settings_dir}/cache/{model|recipe|fts|symlink}/
- Automatically clean up legacy files after successful migration with integrity verification
- Update Config symlink cache to use new path and migrate from old location
- Simplify service classes (PersistentModelCache, PersistentRecipeCache, RecipeFTSIndex, TagFTSIndex) to use centralized migration logic
- Add comprehensive test coverage for cache paths and automatic cleanup
2026-01-26 16:12:08 +08:00
Will Miao d063d48417 feat(symlink): add deep validation for symlink cache invalidation
Detects symlink changes at any depth, not just at root level. Uses two-tier validation:
- Fingerprint check for new symlinks
- Deep mapping validation for removed/retargeted symlinks
2026-01-26 09:30:10 +08:00
Will Miao c9e305397c feat: enhance symlink detection and cache invalidation
- Add `_entry_is_symlink` method to detect symlinks and Windows junctions
- Include first-level symlinks in fingerprint for better cache invalidation
- Re-enable preview path validation for security
- Update tests to verify retargeted symlinks trigger rescan
2026-01-25 19:14:16 +08:00
Will Miao 6f74186498 feat(config): add debug logging for preview root operations, see #772 and #774
- Log preview root rebuilding with counts of different root types
- Add detailed debug output when preview paths are rejected
- Improve visibility into path mapping and validation processes
2026-01-21 09:22:42 +08:00
Will Miao eb8b95176b fix(config): return normalized path in link mapping methods
Previously, `map_path_to_link` and `map_link_to_path` returned the original input path when no mapping was found, instead of the normalized version. This could cause inconsistencies when paths with different representations (e.g., trailing slashes) were used. Now both methods consistently return the normalized path, ensuring uniform path handling throughout the application.
2026-01-21 09:09:02 +08:00
Will Miao 379e3ce2f6 feat(config): normalize paths for case-insensitive comparison on Windows, see #774 and #772
Use os.path.normcase to ensure case-insensitive path matching on Windows, addressing issues where drive letter case mismatches (e.g., 'a:/folder' vs 'A:/folder') prevented correct detection of paths under preview roots. Replace Path.relative_to() with string-based comparison for consistent behavior across platforms.
2026-01-21 08:32:22 +08:00
Will Miao edbcca9bbd refactor: simplify symlink cache invalidation by removing background rescan and noise_mtime in favor of a root-path-only fingerprint. 2026-01-03 19:29:53 +08:00
Will Miao a80380d1f0 chore: reduce symlink cache and scan log verbosity from info to debug level. 2026-01-03 14:50:05 +08:00
Will Miao 099a71b2cc feat(config): seed root symlink mappings before deep scanning
Add `_seed_root_symlink_mappings` method to ensure symlinked root folders are recorded before deep scanning, preventing them from being missed during directory traversal. This ensures that root symlinks are properly captured in the path mappings.

Additionally, normalize separators in relative paths for cross-platform consistency in `BaseModelService`, and update tests to verify root symlinks are preserved in the cache.
2025-12-16 22:05:40 +08:00
Will Miao 2494fa19a6 feat(config): add background symlink rescan and simplify cache validation
- Added threading import and optional `_rescan_thread` for background operations
- Simplified `_load_symlink_cache` to only validate path mappings, removing fingerprint checks
- Updated `_initialize_symlink_mappings` to rebuild preview roots and schedule rescan when cache is loaded
- Added `_schedule_symlink_rescan` method to perform background validation of symlinks
- Cleared `_path_mappings` at start of `_scan_symbolic_links` to prevent stale entries
- Background rescan improves performance by deferring symlink validation after cache load
2025-12-15 18:46:23 +08:00
Will Miao 5359129fad feat(config): improve symlink cache logging and add performance timing
- Add `time` import for performance measurement
- Change debug logs to info level for better visibility of cache operations
- Add detailed logging for cache validation failures and successes
- Include timing metrics for symlink initialization and scanning
- Log cache save/load operations with mapping counts
2025-12-14 15:58:58 +08:00
Will Miao 6b3a11e01a fix(config): ensure symlink mappings are recorded before duplicate check
Update symlink traversal logic to always record path mappings before checking for visited directories. This prevents valid link->target pairs from being dropped when the target directory has already been visited via another path. Also correct path mapping lookup to properly replace link paths with their actual target paths.
2025-12-09 21:46:33 +08:00
Will Miao 3fc72d6bc1 feat(config): replace symlink scanning with cached mapping system
- Add `get_settings_dir` import for cache directory resolution
- Replace `_scan_symbolic_links` and `_rebuild_preview_roots` with unified `_initialize_symlink_mappings` method
- Implement fingerprint-based cache validation using root mtimes, inodes, and noise-aware timestamps
- Add helper methods for path normalization, cache location, and symlink root aggregation
- Improve performance by avoiding redundant symlink traversal when directory structure is unchanged
2025-12-09 19:51:01 +08:00
pixelpaws ef0bcc6cf1 fix(config): remove template default library before saving paths 2025-11-02 09:09:36 +08:00
pixelpaws c53fa8692b refactor(settings): lazily initialize manager 2025-10-08 08:56:57 +08:00
pixelpaws d01666f4e2 feat(previews): serve dynamic library previews 2025-10-04 10:38:06 +08:00
pixelpaws b69c24ae14 test(routes): cover settings library endpoints 2025-10-04 08:38:59 +08:00
pixelpaws 4ea6f38645 fix(config): rename legacy default library 2025-10-03 21:24:17 +08:00
pixelpaws e066fa6873 feat(settings): add multi-library backend support 2025-10-03 20:08:35 +08:00
Will Miao 375b5a49f3 fix(config): update standalone mode environment variable usage 2025-10-02 09:40:24 +08:00
Will Miao 656e3e43be fix(imports): update import paths for ensure_settings_file to use relative imports 2025-09-28 08:40:09 +08:00
pixelpaws 88d5caf642 feat(settings): migrate settings to user config dir 2025-09-27 22:22:15 +08:00
Will Miao fec85bcc08 refactor: unify standalone mode check using environment variable 2025-09-21 22:45:11 +08:00
Will Miao 52acbd954a Add Chinese (Simplified and Traditional) localization files and implement i18n tests
- Created zh-CN.json and zh-TW.json for Simplified and Traditional Chinese translations respectively.
- Added comprehensive test suite in test_i18n.py to validate JSON structure, server-side i18n functionality, and translation completeness across multiple languages.
2025-08-30 21:41:48 +08:00
Will Miao 083f4805b2 feat: Enhance get_preview_static_url to find the longest matching route for static URLs 2025-08-26 20:41:01 +08:00
Will Miao f8ca8584ae feat: enhance URL safety in path mapping by encoding special characters 2025-08-09 16:25:55 +08:00
Will Miao bf7b07ba74 feat: deduplicate and merge checkpoint and unet paths in configuration. See #338 and #312 2025-08-04 10:48:48 +08:00