Commit Graph

11 Commits

Author SHA1 Message Date
Will Miao
308d8f71b8 feat(ui): gray out enrich-hf-llm when no hf_url, add backend fast-fail, rename labels across locales, reposition menu item 2026-07-06 00:34:18 +08:00
Will Miao
d0e8938039 fix(agent): call _format_base_models via self. to prevent NameError
The bare call  inside _build_prompt_context
would raise NameError because class methods don't close over class-level
scope. Use  instead to trigger attribute lookup.

Update enrich_hf_metadata prompt.md clue locations for better LLM accuracy.
Update baseline report to v2 (mean 69.0, 46 models, +2.2pp vs baseline 71.1%).
Consolidate README snapshots into baselines/readmes/.
2026-07-06 00:10:30 +08:00
Will Miao
51c0135250 refactor(agent): rename agent_cli to metadata_ops, strip temp debug logs
- Rename py/agent_cli/ -> py/metadata_ops/ (module was never agent-related)
- Rename tests/agent_cli/ -> tests/metadata_ops/
- Remove 9 low-value/debug INFO log points across agent_handlers.py,
  agent_service.py, llm_service.py, and metadata_ops/__init__.py
- Keep LLM raw response at DEBUG level for diagnostics
- Consolidate per-model progress + LLM result into single concise
  log line with basename instead of full path
- Update package/class/method docstrings to clarify this is a
  pipeline infrastructure, not a true agent loop
2026-07-05 18:00:58 +08:00
Will Miao
26c9ade1c9 feat(agent): optimize base model prompt — grouped display, comprehensive mapping rules, filename inference
- agent_service._format_base_models: output bullet list instead of
  JSON array for cleaner LLM parsing
- prompt.md mapping section: replace 14-row HF→CivitAI table with
  compact rule set covering 14 mapping paths including new entries
  for HiDream-ai, OnomaAIResearch/Illustrious, ideogram-ai/ideogram,
  Tongyi-MAI/Z-Image-Turbo, and Wan-AI/Wan2.*
- base_model extraction instruction: add guidance to infer from
  model filename, YAML tags, and README body text when YAML
  frontmatter has no explicit base_model:
2026-07-05 15:45:17 +08:00
Will Miao
dd3aa97d0a refactor(agent): rename md_to_html to readme_processor, fix section extraction, widget parsing, and list_base_models
- Rename md_to_html.py → readme_processor.py (file no longer just HTML conversion)
- _extract_section: include YAML frontmatter, use heading-level-aware forward
  walk (sub-headings under # are included), increase walk limit past 30 lines
- _is_heading: exclude </hN> closing tags from boundary detection
- _heading_level: new helper for heading-level-aware section matching
- css: yield 0 for heading like closing tags, was unexpectedly caught by _is_heading
- extract_gallery_images: fix YAML block scalar (text: >-) prompt extraction;
  use endswith instead of == to detect the block marker
- _strip_widget_section: add to clean_readme_for_llm (widget text is handled
  by post-processor, not needed in LLM prompt)
- _strip_standalone_images: keep markdown image URLs intact for LLM preview
  extraction (was stripping to alt text only)
- list_base_models: switch from scanner-cache aggregation to
  CivitaiBaseModelService.get_base_models() - always returns full list
- Ollama: add num_ctx=32768 to payload options so thinking models have room
  to both reason and produce output
- Add tests/agent_cli/test_readme_processor.py: 59 tests covering extraction,
  cleaning, section matching, heading detection
- Update existing tests for behavioral changes
2026-07-05 06:39:54 +08:00
Will Miao
f7632a47f9 feat(agent): enrich_hf_metadata with per-model progress and in-place card update
- PostProcessor returns updates dict from enrich_hf_metadata
- AgentService includes updated_data per model in WebSocket progress events
- Convert preview_url to HTTP URL via config.get_preview_static_url()
- LoraContextMenu: showEnhancedProgress + updateSingleItem per model
- BulkContextMenu: same pattern, remove window.location.reload()
- Guard empty updated_data and clean up callbacks on HTTP error
2026-07-04 16:50:56 +08:00
Will Miao
170c8068c5 feat(agent): enrich_hf_metadata — filename-aware section matching, preview extraction for markdown/HTML/widget, JSON salvage, instance_prompt fallback, and validation suite
- extract_relevant_section(): trim README to model-filename-matching section
  for collection repos (download link, anchor ID, heading strategies)
- _strip_standalone_images(): preserve markdown image URLs so LLM can
  extract preview_url; strip only HTML <img> tags
- extract_simple_markdown_images(): extract civitai.images from ![]() body
- extract_html_img_tags(): extract from <img src="..."> (deadman44-style)
- extract_gallery_images(): fix widget parser for YAML - output: dash prefix
- _is_heading: exclude </hN> closing tags from boundary detection
- _extract_section: start at matching heading when match IS a heading line
- _try_salvage_json(): recover truncated JSON (close braces/brackets in
  LIFO order, close unterminated strings, strip trailing commas)
- PostProcessor: store _llm_confidence, add instance_prompt YAML fallback
- agent_service: pass model_basename to prompt, trim README via
  extract_relevant_section before clean_readme_for_llm
- Add tests/enrich_hf_validation/ suite: 100-model pipeline with progress
  checkpoint/resume, per-field scoring, markdown+JSON reporting
- Fix evaluation_engine: read _llm_confidence (not _llm_response)
2026-07-04 12:00:15 +08:00
Will Miao
a1fd4e150b feat(agent): optimize enrich_hf_metadata with README cleaning, Ollama native API, and expanded fields
- Add clean_readme_for_llm() to strip noise from README before LLM injection
- Keep widget section text (valuable tag signal) and unmarked code blocks (trigger words)
- Preserve standalone image alt text instead of removing entirely
- Switch Ollama to native /api/chat with think:false to fix empty content on thinking models
- Extract Sample Gallery table images and deduplicate with widget images
- Only strip code blocks with explicit language tags (bash)
- Add notes and usage_tips fields to SKILL.md output format and post-processor
- Clean up dead code, fix regex edge cases, remove double type annotation
2026-07-04 08:01:50 +08:00
Will Miao
88349bf944 feat(agent): render HF README as HTML in modelDescription, move converter to skill-local module
- Add inline convert_readme_to_html() in new skill-local md_to_html.py
  (zero external deps, handles h1-h4/bold/italic/code/lists/tables/links/hr)
- Strip YAML frontmatter, <Gallery />, badge images, HTML comments pre-conversion
- Fix indented whitespace after lists being misidentified as code blocks
- Fix HTML double-escaping in _inline_md (each pattern escapes independently)
- LLM short_description → civitai.description ("About this version" sidebar)
- raw README HTML → modelDescription (description tab, always available offline)
- Pass full readme_content from agent_service to post_processor
- 51 tests for converter + 4 updated/added post-processor tests
2026-07-02 23:34:52 +08:00
Will Miao
a8adcaf023 feat(agent): improve enrich_hf_metadata skill with priority_tags, preview_url fix, civitai.trainedWords
- Add identify_model_type() helper to determine lora/checkpoint/embedding
- Pass priority_tags from user settings to LLM prompt for tag relevance
- SKILL.md: instruct LLM to exclude technical/generic HF tags, cross-reference
  against priority_tags; forbid ['None'] placeholder for trigger words
- post_processor: fix preview_url not updated after download (now writes local
  .webp path to metadata); write trigger words to civitai.trainedWords instead
  of top-level; sanitize ['None']/'null'/'n/a' placeholder values to []
- download_preview() now returns str | None (local path) instead of bool
- Update tests for new return type and nested civitai.trainedWords structure
2026-07-02 22:14:44 +08:00
Will Miao
cf898da193 feat(agent): add LLM-powered metadata enrichment system with AgentCLI and PostProcessor
Introduce an agent skill framework for LLM-driven metadata enrichment:

- AgentCLI (py/agent_cli/): in-process wrappers around internal services
  using standard relative imports, eliminating the need for sys.path hacks
- LLMService: centralized BYOK (bring-your-own-key) LLM client supporting
  OpenAI, Ollama, and custom OpenAI-compatible endpoints
- PostProcessor: deterministic engine that applies LLM output via AgentCLI
  (replaces old handler.py + _BASE_MODEL_ALIASES approach)
- SkillRegistry: filesystem-based skill discovery (skill.yaml + prompt.md)
- AgentService: orchestrates skill execution with WebSocket progress
- Frontend AgentManager: WebSocket listeners, skill execution, config UI
- Context menu entries (single + bulk) for "Enrich Metadata (Agent)"
- Settings UI for AI Provider configuration (BYOK)
- Full i18n support across 9 locales

Bug fixes found during review:
- aiohttp.web.json_response: status_code= -> status=
- settings_modal cancelEditApiKey: wrong argument position
- AgentManager.isLlmConfigured: allow Ollama without API key
- PostProcessor._merge_tags: lowercase all tags to match TagUpdateService
2026-07-02 21:27:01 +08:00