Commit Graph

524 Commits

Author SHA1 Message Date
pixelpaws
dc674098e7 Merge pull request #1050 from willmiao/fix/recipes-bulk-content-rating
fix(recipes): enable bulk content rating for selected recipes
2026-08-03 20:58:24 +08:00
Will Miao
9087b4b07c feat(example-images): add missing-only download path and skip existing files
Split the single-model and bulk context menu actions into 'Download
Missing Example Images' (regular endpoint, skips already-processed
models) and 'Re-process Example Images' (force endpoint, retries
failed models).

- start_download accepts model_hashes so a selected subset can be
  processed with the progress-aware skip logic; explicitly targeted
  models bypass the failed/processed model-level guards so per-image
  gaps are filled
- pre-download existence check in the processor skips network requests
  for image files already on disk across all download paths
- force download retries previously failed models and clears their
  failed status on success
- add i18n keys for the new menu items across all locales
2026-08-03 20:52:46 +08:00
Will Miao
8e45c22d7a fix(recipes): enable bulk content rating for selected recipes 2026-08-03 19:31:58 +08:00
Will Miao
ab4154c57d feat(ui): add seeded random sort option to model pages (#1049) 2026-08-03 15:02:49 +08:00
Will Miao
28e93d12ff fix(example-images): use in-place cache sync and bulk pending-check index for large libraries 2026-08-03 12:04:56 +08:00
Will Miao
75e63c758b feat(api): add cursor-based pagination to civitai user-models endpoint 2026-08-03 11:07:06 +08:00
Will Miao
823f71f269 feat(nodes): make Lora Stack Combiner inputs dynamic 2026-08-02 22:04:40 +08:00
Will Miao
042dd4088d fix(nodes): make Lora Stack Combiner inputs optional 2026-08-01 17:14:00 +08:00
Will Miao
4c647ad9c8 fix(update): throttle nightly update badge to once per day 2026-07-31 21:18:58 +08:00
Will Miao
8ca3e6c33f fix(ui): guard marquee bulk-mode entry against click jitter and stale drag state 2026-07-31 18:40:14 +08:00
Will Miao
dd6bdbf297 fix(update): persist update_channel via settings.json instead of hasGit
After b464fdc3 (preserve .git on release switch), the hasGit-based
channel detection is unreliable — .git now exists for both release
and nightly installs, so page refresh always reset the channel.

- Add _resolveChannelFromSettings() with migration heuristic:
  !hasGit → release (ZIP), detached HEAD → release (on tag),
  on branch → nightly. Uses gitInfo.branch from check-updates.
- Persist resolved channel to settings.json on first load
  (one-time migration) and on explicit switchChannel.
- Add update_channel validation (release|nightly) in backend
  update_settings handler.
- Remove hasGit-based guessing from initialize(); defer to
  checkForUpdates where full gitInfo is available.
- Channel resolution runs before checkForUpdates early-returns
  to avoid null channelMode on reload-within-interval.

Tests: 361 passed.
2026-07-31 13:23:54 +08:00
Will Miao
b47dde87e4 fix(settings): suppress error toasts when optional model roots are empty 2026-07-31 10:07:52 +08:00
Will Miao
3bdacb8f46 fix(test): update release channel git test to mock _perform_git_update instead of _download_and_replace_zip 2026-07-30 18:35:43 +08:00
Will Miao
c9e5e784fc fix(metadata-overwrite): use sentinel default for clip_skip to accept wired 0 2026-07-28 23:13:00 +08:00
Will Miao
cfc4903c0c fix(update): read ahead_by from GitHub compare API when status is ahead/diverged
The compare API URL format compare/{local_hash}...main returns
status='ahead' when main is ahead of the local commit. The count is
in the ahead_by field, not behind_by. The old code only read behind_by
which is always 0 in this case, causing the UI to show 'Up to date'
when actually several commits behind.

Also handle status='diverged' (both sides have unique commits) by
reading ahead_by for the remote-ahead count.

Frontend adds a hash comparison fallback: if behind_by is 0 but local
and remote commit hashes differ, show 'Behind main' instead of the
incorrect 'Up to date'.

Tests: _AheadCompareDownloader and _DivergedCompareDownloader mocks
for the two status paths.
2026-07-28 17:47:38 +08:00
Will Miao
66d1c96783 feat(update): add Release/Nightly channel switching
- Add POST /api/lm/switch-channel endpoint with git init / ZIP fallback
- Add _backup_git/_restore_git helpers with safe rollback
- Version-info endpoint now returns has_git flag for auto-detection
- Check-updates always returns releases (changelog) regardless of channel
- Nightly mode shows 'N commits behind main' with commit hash and date
- View on GitHub link points to /commits/main in nightly mode
- Channel toggle UI with pill-style buttons in update modal
- Confirmation dialog with Esc / backdrop-dismiss support
- Channel derived from has_git on every page load, no localStorage
- i18n: 11 new keys translated across 9 non-English locales
- CSS: unified card-style sections in _base.css
- Tests: 8 new tests covering switch-channel, nightly response, init_git_repo
2026-07-27 20:27:05 +08:00
Will Miao
1de0a53241 feat(grouping): version-group library cards by HuggingFace repo for non-Civitai sources (#1040) 2026-07-26 21:46:55 +08:00
Will Miao
0ec7eaf606 fix(wildcards): resolve weighted N::value syntax inside wildcard YAML lists (#1039) 2026-07-26 18:33:31 +08:00
Will Miao
f49b4ba4db fix(metadata-overwrite): rename 'checkpoint' input to 'model' 2026-07-26 10:59:38 +08:00
Will Miao
125bed3f09 feat: add Metadata Overwrite node for manual generation params override 2026-07-26 08:50:21 +08:00
Will Miao
077e70169d feat: add type-signature-based fallback for unregistered nodes
GenericNodeExtractor (previously a no-op) now inspects
RETURN_TYPES to detect MODEL loaders and CONDITIONING
encoders in nodes not registered in NODE_EXTRACTORS.

- Propagate return_types from the hook layer through the
  registry to GenericNodeExtractor.extract() and update().
- MODEL detection: scan ckpt_name/unet_name/model_path/
  model_name/gguf_name fields, validate by extension.
- CONDITIONING detection: scan text/clip_l/t5xxl/prompt
  fields, store prompt text and conditioning tensor.
- _fill_missing_metadata also checks node_cache, so
  GenericNodeExtractor-handled nodes survive cache.
2026-07-25 22:14:51 +08:00
Will Miao
f34c02756d fix(recipes): eliminate O(n) fuzzy search fallback over 42k+ recipes
Drop the SequenceMatcher-based fuzzy_match fallback that froze the server
when FTS returned empty results. FTS now returns empty set for zero results
(no fallback), and when the index is not yet ready, search returns empty
rather than scanning all items in Python.
2026-07-25 17:34:37 +08:00
Will Miao
55896669fc feat(SaveImageLM): expose webp_method and jpeg_subsampling as conditional node inputs
Add two new optional parameters to the Save Image node:

- webp_method (INT, 0-6, default 6): Controls WebP compression level.
  0=fastest/largest, 6=slowest/smallest. Previously hardcoded to 0.
- jpeg_subsampling (INT, 0-2, default 0): Controls JPEG chroma
  subsampling. 0=4:4:4 (best quality), 1=4:2:2, 2=4:2:0.

Frontend JS extension hides/disables each parameter when the
selected file_format doesn't apply (e.g., webp_method is hidden
when saving as PNG or JPEG). 7 new tests cover parameter plumbing
and default consistency across INPUT_TYPES, save_images(), and
process_image().
2026-07-24 19:32:51 +08:00
Will Miao
e341e0b9d2 fix(test): update parameters assertion to include Version: ComfyUI after metadata format upgrade 2026-07-24 18:29:07 +08:00
Will Miao
2aabd1d90e fix(ai): use json_schema instead of json_object for broader provider compatibility (#1033)
LM Studio and some other OpenAI-compatible servers reject
response_format=json_object but accept json_schema. Switch to the
equivalent json_schema format and add a fallback that retries
without response_format when the provider rejects the format type.
2026-07-23 09:17:29 +08:00
Will Miao
7c8dc57d55 fix(security): use abspath instead of realpath in containment checks to support symlinks (#1028) 2026-07-23 07:06:41 +08:00
Will Miao
cf64043f7d fix(security): add library root containment check for delete/move/rename operations (#1028) 2026-07-21 15:23:38 +08:00
Will Miao
ea80c2224c fix(download): prevent path traversal in download template resolution (#1028) 2026-07-20 21:08:43 +08:00
Will Miao
f53f859a71 feat(filter): add debounced tag search with backend search-tags endpoint 2026-07-20 17:37:47 +08:00
Will Miao
a8d742b031 feat(metadata): add CivArchive API toggle and provider fallback order settings
- Add enable_civarchive_api toggle (default on) to allow disabling
  CivArchive to avoid its rate-limit windows entirely
- Add metadata_provider_order dropdown with two presets:
  CivitAI → CivArchive → Archive DB (default) and
  CivitAI → Archive DB → CivArchive
- Wire both settings through backend (metadata_service, settings_manager,
  misc_handlers) and frontend (SettingsManager, state, settings modal)
- Reorder Metadata section in settings modal: toggles → status/management
  → fallback order, for natural top-down workflow
- Make update_metadata_providers() log the effective provider chain
  using actually-registered providers rather than settings assumptions
- Add 5 test cases covering all provider-combination paths
- Complete i18n translations for 6 new keys across all 9 non-English locales
2026-07-19 17:51:50 +08:00
Will Miao
c27e4d1bfc feat(cache): opportunistic cache sync on metadata read with in-place update
- Add PersistentModelCache.update_single_model() for lightweight targeted
  SQL update (single row + incremental tag/hash deltas, no full table scan)
- Add ModelScanner.sync_cache_from_metadata() with compare-first logic:
  skips entirely when cache is already in sync; when stale, updates the
  entry in-place (O(1) instead of O(n) remove+append), incrementally
  adjusts tag counts/hash index/version index, and resorts only when
  sort-relevant fields changed
- Wire sync_cache_from_metadata() into BaseModelService.get_model_metadata()
  via fire-and-forget asyncio.create_task — disk I/O is already paid for
- Include identity re-validation guard against concurrent cache replacement
- Add 16 tests covering _cache_entries_differ, sync_cache_from_metadata
  (no-change, in-place, fallback, conditional resort), and
  update_single_model (insert, tag delta, hash delta)
2026-07-19 08:32:21 +08:00
Will Miao
d15a8aa9a2 fix(workflow): accept non-string widget values and support GlobalSeed node in gen-params (#1026) 2026-07-18 22:57:20 +08:00
Will Miao
2f94a9773e feat(workflow): redirect gen-params updates to connected Primitive nodes (#1026)
When a KSampler marked as 'Send Gen Params Target' has widget inputs
wired to Primitive nodes (PrimitiveNode, PrimitiveInt, PrimitiveFloat,
etc.), sending gen params from the Lora Manager UI now updates the
Primitive node's value instead of the KSampler widget. This is
necessary because ComfyUI's execution engine reads from the connected
input, ignoring the widget value when a wire is present.

Also fix two minor issues found during review:
- Remove unnecessary String() wrapping on numeric gen params (seed,
  steps, cfg) to preserve native types through the JSON/WS path
- Correct misleading isNodeEnabled comment: LGraphEventMode values
  are 0=Always, 2=Never, 4=Bypass (not 'Normal/Enabled')
2026-07-18 22:10:48 +08:00
Will Miao
f0bf2728c9 fix(downloads): accept download_id in history delete/retry endpoints, add unique index 2026-07-18 21:10:42 +08:00
Will Miao
7ee2361e87 fix(config): remove stale 'default' library entry and consolidate example images on startup 2026-07-18 17:25:36 +08:00
Will Miao
2018722cc8 fix(registry): handle compound subgraph node IDs, add proactive node push from graph hooks
- Handle compound node IDs (e.g. "252:0") from expanded group subgraphs
  to fix 400 Bad Request on workflows with group nodes
- Frontend proactively pushes node data via afterConfigureGraph and
  LiteGraph hooks (onNodeAdded/onNodeRemoved/graphChanged), eliminating
  WebSocket round-trip latency for most "Send to Workflow" operations
- Add content-fingerprint dedup to skip duplicate register-nodes POSTs
- Fast-path cache returns immediately when tabs are registered (including
  0-node registrations), avoiding unnecessary WS refresh cycles
- Distinguish "Empty Registry" from other errors in standalone UI toast
- Reduce WS refresh timeout 2s→0.5s, add cooldown and lock to prevent
  concurrent refresh storms
- All [LM:Registry] logs at DEBUG level
2026-07-13 18:02:26 +08:00
Will Miao
71658ab37b feat(settings): add LORA_MANAGER_PORTABLE env var for per-instance settings isolation (#1018) 2026-07-12 07:44:31 +08:00
pixelpaws
cb4ad27813 Merge pull request #1013 from willmiao/agent
Hugging Face model metadata AI enrichment
2026-07-06 12:21:19 +08:00
Will Miao
00228deaaa fix(download): retry on Civitai 429 rate limit instead of removing images from metadata
When Civitai returns 429 (Too Many Requests) during example image
downloads, the previous behavior treated all failures identically and
permanently removed the corresponding images from model metadata —
making them impossible to retry.

This commit adds:
- 429 detection + Retry-After header parsing in download_to_memory
- Exponential backoff retry (up to 3 attempts) in
  download_model_images_with_tracking
- Separate tracking of rate-limited vs permanently failed URLs
- rate_limited_models progress tracking persisted to disk
- Rate-limited models are NOT added to failed_models/processed_models
  so they are automatically retried on subsequent download runs
- Force mode clears failed_models when rate-limited images exist
2026-07-06 11:58:19 +08:00
Will Miao
e0e1b804a7 fix(llm): require api_base for custom provider without preset default 2026-07-06 10:02:04 +08:00
Will Miao
5983eaa1ce refactor(llm): use catalog-based max_tokens, remove JSON retry, reduce Ollama num_ctx
- Parse limit.output from model catalog alongside model IDs
  for per-model max output token limits
- Use catalog lookup in chat_completion_json() to set max_tokens;
  fall back to 4096 for unknown models (e.g. local Ollama)
- Remove the JSON retry (response_format → plain text fallback);
  keep _try_salvage_json as last-resort for truncated responses
- Reduce Ollama num_ctx from 32768 to 8192 (sufficient for
  metadata enrichment, saves VRAM)
- Fix stale test comment referencing removed retry
2026-07-06 09:13:42 +08:00
Will Miao
07fa454f72 chore(tests): stop tracking HF enrichment baseline snapshots
Remove tests/enrich_hf_validation/baselines/ from git tracking
(.gitignore entry + git rm --cached). These contain README snapshots
from community HF repos that may include NSFW/sensitive content.

Local files are preserved on disk for offline reference.
2026-07-06 01:08:25 +08:00
Will Miao
4b5aa45379 chore(tests): update bash code block tests to match preserved-bash behavior
Commit 9a0d866b changed _strip_fenced_code_blocks to preserve bash/shell
code blocks (they carry CLI setup and trigger-word metadata signal).
Update the two affected tests to expect bash content in the output
instead of asserting it is stripped.

- Rename test_bash_code_block_stripped → test_bash_code_block_preserved
- Update assertions: expect 'pip install' in result
2026-07-06 01:02:04 +08:00
Will Miao
d0e8938039 fix(agent): call _format_base_models via self. to prevent NameError
The bare call  inside _build_prompt_context
would raise NameError because class methods don't close over class-level
scope. Use  instead to trigger attribute lookup.

Update enrich_hf_metadata prompt.md clue locations for better LLM accuracy.
Update baseline report to v2 (mean 69.0, 46 models, +2.2pp vs baseline 71.1%).
Consolidate README snapshots into baselines/readmes/.
2026-07-06 00:10:30 +08:00
Will Miao
13ed898b6b chore(tests): add base_model ground truth mapping for all 46 test entries 2026-07-05 20:47:30 +08:00
Will Miao
e1dfd1c2a6 chore(tests): add two test entries and their HF README snapshots 2026-07-05 20:45:01 +08:00
Will Miao
51c0135250 refactor(agent): rename agent_cli to metadata_ops, strip temp debug logs
- Rename py/agent_cli/ -> py/metadata_ops/ (module was never agent-related)
- Rename tests/agent_cli/ -> tests/metadata_ops/
- Remove 9 low-value/debug INFO log points across agent_handlers.py,
  agent_service.py, llm_service.py, and metadata_ops/__init__.py
- Keep LLM raw response at DEBUG level for diagnostics
- Consolidate per-model progress + LLM result into single concise
  log line with basename instead of full path
- Update package/class/method docstrings to clarify this is a
  pipeline infrastructure, not a true agent loop
2026-07-05 18:00:58 +08:00
Will Miao
7b19bbb14e fix(agent): preserve preview URLs for collection repo models with flat heading structure
Three-part fix for enrich_hf_metadata failing to extract correct preview_url
from HuggingFace collection repos where models share flat heading levels:

1. _strip_standalone_images() now converts <img> tags to markdown image
   syntax ![alt](src) instead of stripping the URL entirely, so the LLM
   can still extract preview URLs.

2. _extract_section() uses a line-count-based forward window (stopping at
   <a id> anchors) for non-heading matches, instead of stopping at the
   very next heading. This prevents same-level sub-headings (# Download,
   # Trigger, # Sample prompt within a single model section) from
   truncating the window before sample images are included.

3. Post-processor preview fallback now filters gallery images to the
   model-specific README section before falling back to the repo-wide
   first image.
2026-07-05 17:05:47 +08:00
Will Miao
5494a70f40 chore(tests): commit validation dataset and baseline reports into repo
Move the HF model list from ~/Documents/ into tests/enrich_hf_validation/test_data/
and commit the pipeline validation baseline artifacts (report.json,
preprocessing_audit.json, README snapshots) into baselines/.

Update config.py and run_validation.py defaults to use repo-relative paths
via os.path.dirname(__file__) instead of ~/Documents/ hardcode.

Originates from changes in 8fb00998 (validation pipeline audit).
2026-07-05 17:03:45 +08:00
Will Miao
8fb00998a7 feat(agent): fix extract_relevant_section false positives, add validation pipeline audit
- extract_relevant_section: raise token threshold >3, verify anchor
  sections contain basename, require 2+ heading token overlaps, skip
  TOC-style headings (markdown links), verify heading section size
- metadata_constructor: parse repo_id,model_name.safetensors format
  so model_path basename matches real filename
- config: replace hardcoded SUPPORTED_BASE_MODELS with dynamic
  init_supported_base_models() using production list_base_models()
- preprocessing_auditor: new Phase 1.5 audit module — fetches each
  README, runs extract_relevant_section + clean_readme_for_llm,
  records stats and flags, saves raw READMEs for cross-reference
- run_validation: integrate audit phase, add --audit-only mode,
  add LLM config consistency check, add ComfyUI root to sys.path
- report_generator: add Preprocessing Audit and Config Warnings
  sections to both markdown and JSON reports
2026-07-05 11:18:48 +08:00