7 Commits

Author	SHA1	Message	Date
Will Miao	d572292142	feat(download): fill model metadata from the source API on download ... A ModelScope or Hugging Face download landed as a bare filename, hash and source link; the model card stayed empty until the user ran "Enrich Metadata with AI" by hand. But everything that makes a CivitAI download useful — the display name, the description, the tags, the trigger words, the example images, the preview — is already published by those sites' public APIs, so asking for it at download time is deterministic work, not model work. Add `py/services/model_sources/hydration.py`, called by `_save_source_metadata()` once the sidecar exists and the file is in the scanner cache. It fetches the model card plus the site's card extras and hands them to the same `PostProcessor` the AI skill uses, with an empty `llm_output`, so the two paths cannot drift apart. What lands: * `model_name` from the site's own display name (ModelScope's `Name`), so the card stops showing the local filename — written only while the value still equals the file stem, since once a user renames a model that choice is theirs to keep * `civitai.name` from the matched version's label (`showName`), which the card renders as the version chip * `civitai.description` / `modelDescription` from the author summary plus the README as HTML * `civitai.images` / `preview_url` from the per-file example images * `civitai.trainedWords` from the per-file trigger words * `base_model`, `tags` and `usage_tips` as before Provenance stays honest: the pass records `metadata_source = "source:<platform>"` rather than the skill's `agent:enrich_hf_metadata`, and — because no provider ran — it no longer stamps `llm_enriched_at`; that stamp is now conditional on the LLM actually answering, which is what the field means. The five hand-rolled `civitai` dict merges in the post-processor collapse into one `_merge_civitai()` helper. Two guards keep it safe. Only a model whose stored `source_platform`/`source_url` match the repository being downloaded is updated, so a local file that merely shares a name never receives another model's card; and a file already on disk is topped up too, which back-fills models downloaded before this existed. READMEs and detail payloads describe the repository rather than the file, so a short-lived process-wide `ModelSourceCache` (300 s, 32 entries) keeps a batch over one repository to two HTTP requests. Every failure is logged and swallowed: hydration can never fail a download. Fix the hash policy while here. `_save_source_metadata()` went straight to `MetadataManager.create_default_metadata()`, bypassing the per-type factory on the owning scanner, so a checkpoint paid a full SHA256 inside the download request — `CheckpointScanner`/`OtherScanner` deliberately record `hash_status="pending"` with an empty `sha256` for their multi-GB files. Metadata is now created through `scanner._create_default_metadata()`. Hydration copes with the empty hash: `_matching_versions()` falls back to the repository basename, which is exactly what the download just wrote. Report both post-transfer stages, which advance no byte counter and so read as a stall: the bar sat at 100% showing `0 B/s` for the seconds spent hashing and fetching. `_report_phase()` broadcasts `{"status": "metadata", "stage": "indexing" | "source", "platform": ...}`, and `LoadingManager` names the stage in the status line (keeping the batch position), retitles the item line, replaces the dead speed figure and runs a sheen over the bar. `stage`/`platform` are machine-readable; the wording is localised in the frontend. Finally, `modelscope.ai` is its own catalogue rather than an alias of `modelscope.cn` — `referall13/EM1` exists only on `.ai` and `jj3550945163/Krea-2-LORA` only on `.cn` — so its URLs were rejected with "Invalid model URL format". Register it as `ModelScopeIntlSource` (`platform="modelscope-ai"`, `msai:` group prefix, its own default download directory) and derive every URL either deployment builds from a per-class `base_url`. `modelscope.com` stays an alias of `.cn`, which is what it redirects to. The frontend source table, the link dialog hints and the docs mirror the split. Verified against the live APIs: both reported `.ai` repositories list their files, read their READMEs and yield name / version / base model / trigger words / example images. Backend 3092 passed; frontend 1259 JS + 91 Vue passed. The nine locales carry the new progress copy in the next commit.	2026-09-17 07:47:07 +08:00
Will Miao	0f160e157f	fix(modelscope): identify a model file by hash before filename ... A file was matched to its published version by comparing basenames against each version's `stats.fileList`. Renaming the weights — routine once a model is filed away, and the reason the scanner records a sha256 at all — made the match fail silently, so the file lost its example images and its preview with no indication why. The detail payload's `ModelInfos.safetensor.files[]` carries a real sha256 per published file, and the local hash is already on disk, so match on that first: it is the one identifier a rename cannot invalidate. Exact basename and `showName` matching remain as fallbacks, and an unknown hash falls through to them rather than giving up, so a re-encoded file still resolves. Verified against the live repository: a renamed `c1-st1000` file with its hash yields the c1-st1000 image, the same rename without a hash yields nothing, and supplying c1-st2000's hash resolves to the c1-st2000 image even when the filename claims otherwise.	2026-09-14 21:27:09 +08:00
Will Miao	e9e9ee20c6	perf(modelscope): fetch a repository's model card once per run ... A collection repository publishes many model files under a single source id, but enrichment re-read the README and the model-detail payload for every one of them: eight checkpoints meant sixteen HTTP requests, each detail payload being 10-22 KB of JSON. Add `ModelSourceCache`, created by `execute_skill()` for the duration of a run and passed to the provider through a new optional `cache` argument on `fetch_model_card_context()`. The agent caches the README (repository-wide and provider-agnostic), and ModelScope caches its detail payload under a provider-namespaced key. Only successful reads are memoised, so a transient failure is still retried for the next file, and the per-file selection is redone from the cached payload so a checkpoint never inherits a sibling's example images. Nothing is retained across runs — a model card can change at any time — and download URLs are not routed through the cache. Measured over the eight checkpoints of one ModelScope repository: 16 requests before, 2 after. To keep the two concerns separable, `_build_card_context()` now turns a detail payload into a `ModelCardContext` as a pure function.	2026-09-14 21:24:57 +08:00
Will Miao	4064ea7d3a	refactor(agent): apply model-source data without an LLM ... `_build_prompt_context()` was only reached when the LLM was configured, so a user with no provider got nothing at all from a linked model source — no preview, no example images, no author summary, no tags — even though all of that is deterministic data from a public API. Split the model-card fetch into `_load_source_card()`, which runs for every source-backed enrichment, and have the post-processor apply its result whether or not the LLM runs. The prompt is then built from the already-fetched card rather than re-fetching it. Invoking "Enrich Metadata with AI" still always calls the provider; a model source supplying a description, images and tags is not treated as a reason to skip it, since the LLM's summary and notes are richer and an action that silently does not call out to the provider would be unpredictable. The site data acts as a fallback for the gaps the LLM leaves. Add `base_model_resolver.resolve_base_model()` to map the site's own names (`krea/Krea-2-Turbo`, `KREA_2_TURBO`) onto the canonical vocabulary, used only when the LLM returns no base model. It is strictly conservative — exact normalised matching plus a bounded set of variant suffixes, and it only ever returns a name that is already in the vocabulary — so an uncertain hint defers to the LLM instead of writing a plausible-looking wrong value.	2026-09-14 20:39:10 +08:00
Will Miao	38d4c59b4c	feat(download): support ModelScope repositories in the URL downloader ... ModelScope became a linkable source, but downloading from it was impossible: the URL picker only recognised huggingface.co, the file listing hit a huggingface-only endpoint, the resolve URL was hardcoded, and the default path template always wrote into a `huggingface/` directory. Move the download knowledge into the providers so the handlers stay generic: - `ModelSource` gains `list_files()`, `file_download_url()`, `default_revision` and `default_subdir`. `HuggingFaceSource` keeps the Hub tree API (`/api/models/{id}/tree/{rev}`, LFS-aware sizes, `main`). `ModelScopeSource` uses `/api/v1/models/{id}/repo/files?Revision=master` — which reports real byte sizes for LFS files, so no HEAD probe is needed, and which only accepts `master` (an HF-imported repo still 404s on `main`) — and downloads through `/models/{id}/resolve/{rev}/{path}`. That URL redirects to a CDN target carrying a time-limited `auth_key`, so it is rebuilt on every request and never cached, which is also what keeps resumable Range requests working. - `hf_handlers.py`/`HfHandler` become `model_source_handlers.py`/ `ModelSourceHandler` with `list_model_source_files` and `download_model_source`. New routes `/api/lm/model-source-files` and `/api/lm/download-model-source`; the old `/api/lm/hf-repo-files` and `/api/lm/download-hf-model` paths stay as aliases, and a payload without `platform` still means Hugging Face, so existing callers are unaffected. - A downloaded sidecar now records `source_platform` + `source_url` (with the `hf_url` alias only for Hugging Face) instead of always writing `hf_url`, and `use_default_paths` files ModelScope downloads under `modelscope/<owner>/<repo>`. The now-unused shared HF aiohttp session and its shutdown hook are gone; providers open short-lived sessions. - Frontend: `detectUrlType` returns the platform-neutral `model-source-repo` / `model-source-file` plus an explicit `platform`, the DownloadManager's `hf*` state and methods are renamed to `source*`, every `source === 'huggingface'` check becomes `isExternalModelSource()`, and batch groups are keyed by `platform:repo` so the same `owner/name` on two sites renders as two groups. A bare `owner/name` still means Hugging Face. - `is_valid_source_id()` centralises repo-id validation (exactly `owner/name`, no traversal, no leading dot). This also fixes the old HF download check that rejected any dot in the name, i.e. legitimate repos such as `black-forest-labs/FLUX.1-dev`. Verified against the live APIs: the example repo lists 8 weight files with correct sizes, and a ranged GET of the built resolve URL returns 206 after following the redirect to the CDN. Backend 2853 passed; frontend 1143 JS + 91 Vue passed. The nine locales carry the refreshed download copy in the next commit.	2026-09-14 07:42:51 +08:00
Will Miao	5ab0e88abc	feat(links): support ModelScope and TensorArt as model sources ... A model file could only ever be linked to huggingface.co: `set_hf_url` validated the URL with a huggingface-only regex, the agent fetched the card from a hardcoded HF URL, and the readme processor built every relative image path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the same model-card convention (README.md + YAML frontmatter, often carrying `base_model:` and `trigger_words:`) behind a public, key-less API, so the enrichment pipeline could already serve it - it was the plumbing that was HF-shaped, not the idea. Make the external source a first-class, provider-driven concept: - New `py/services/model_sources/` registry. A `ModelSource` owns URL recognition (lenient for stored values, strict for user input), the canonical page URL, model-card fetching, the asset base URL and the capability flags. `HuggingFaceSource` is the previous logic relocated; `ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md` and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is link-only on purpose: tensor.art answers plain HTTP clients with a Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud / cn.tensorart.net) rejects every /v1/model/* route with "invalid authorization header", so it declares supports_enrichment=False rather than failing silently later. - Metadata gains `source_platform` + `source_url`; `hf_url` stays as a read/write alias, written only for Hugging Face, so existing sidecars, cached rows and third-party consumers keep working. Normalisation runs at the scanner, the persistent cache (both directions, plus two new columns behind an ALTER migration) and the linking handler - which is what stops a user who switches sources from leaving a stale `hf_url` on a ModelScope model. - The agent pipeline keys off the provider instead of `hf_url`: the fast-fail gate now explains *why* a model is skipped (no source / unknown source / source without a reachable card), the prompt context exposes source_url/source_id/source_label/asset_base_url while still filling the legacy hf_url/repo aliases, and the four README image extractors take a base_url (defaulting to HF) so relative paths resolve against the right site. Version grouping generalises to hf: / ms: / ta: keys. - `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but accepts `source_url`, validates against every provider and returns the platform. `GET /api/lm/model-sources` lets the UI render the supported-site list from the server. - Frontend: a `modelSourceHelpers` mirror of the registry drives the link dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the version-group key and the enrichment gate; the versions tab no longer sends ms:/ta: keys to the CivitAI API. TensorArt stays in the list because provenance is worth keeping even when the card is unreadable - the dialog says so plainly ("Sites that don't expose one (currently TensorArt) can only be linked") and the context menu disables enrichment with a matching tooltip, instead of the user getting "Unsupported URL". Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a 1882-byte card whose frontmatter carries base_model/tags/trigger_words, and relative images resolve to .../resolve/master/.... Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest tests/i18n and a Jinja compile pass over templates/. The nine locales carry [TODO: Translate] for the new strings, completed in the next commit.	2026-09-14 07:24:08 +08:00
Will Miao	cf898da193	feat(agent): add LLM-powered metadata enrichment system with AgentCLI and PostProcessor ... Introduce an agent skill framework for LLM-driven metadata enrichment: - AgentCLI (py/agent_cli/): in-process wrappers around internal services using standard relative imports, eliminating the need for sys.path hacks - LLMService: centralized BYOK (bring-your-own-key) LLM client supporting OpenAI, Ollama, and custom OpenAI-compatible endpoints - PostProcessor: deterministic engine that applies LLM output via AgentCLI (replaces old handler.py + _BASE_MODEL_ALIASES approach) - SkillRegistry: filesystem-based skill discovery (skill.yaml + prompt.md) - AgentService: orchestrates skill execution with WebSocket progress - Frontend AgentManager: WebSocket listeners, skill execution, config UI - Context menu entries (single + bulk) for "Enrich Metadata (Agent)" - Settings UI for AI Provider configuration (BYOK) - Full i18n support across 9 locales Bug fixes found during review: - aiohttp.web.json_response: status_code= -> status= - settings_modal cancelEditApiKey: wrong argument position - AgentManager.isLlmConfigured: allow Ollama without API key - PostProcessor._merge_tags: lowercase all tags to match TagUpdateService	2026-07-02 21:27:01 +08:00