4 Commits

Author	SHA1	Message	Date
Will Miao	e9e9ee20c6	perf(modelscope): fetch a repository's model card once per run ... A collection repository publishes many model files under a single source id, but enrichment re-read the README and the model-detail payload for every one of them: eight checkpoints meant sixteen HTTP requests, each detail payload being 10-22 KB of JSON. Add `ModelSourceCache`, created by `execute_skill()` for the duration of a run and passed to the provider through a new optional `cache` argument on `fetch_model_card_context()`. The agent caches the README (repository-wide and provider-agnostic), and ModelScope caches its detail payload under a provider-namespaced key. Only successful reads are memoised, so a transient failure is still retried for the next file, and the per-file selection is redone from the cached payload so a checkpoint never inherits a sibling's example images. Nothing is retained across runs — a model card can change at any time — and download URLs are not routed through the cache. Measured over the eight checkpoints of one ModelScope repository: 16 requests before, 2 after. To keep the two concerns separable, `_build_card_context()` now turns a detail payload into a `ModelCardContext` as a pure function.	2026-09-14 21:24:57 +08:00
Will Miao	35b291ab19	feat(modelscope): read the model-detail API for card extras ... ModelScope's model card is not just README.md: the author's summary (Description), the site-curated tags (OfficialTags), the internal architecture enums (VisionFoundation/SubVisionFoundation) and — per published version — the model filenames with that file's example images (coverImages) and trigger words all live in the model-detail API. AIGC repositories there frequently ship an auto-generated boilerplate README and put the only useful text in Description, so reading just the README yielded almost nothing. Add `ModelSource.fetch_model_card_context()` returning a new `ModelCardContext`, implemented by ModelScopeSource against the public (no API key) detail endpoint. Example images are matched to the model's basename through each version's `stats.fileList`, so every checkpoint in a collection repository gets its own images rather than a sibling's. Consume the context in the post-processor: * example images seed `civitai.images` and, being per-file, take priority in the preview fallback chain * the author summary becomes a paragraph in `modelDescription` and fills `civitai.description` when the LLM returns no short description * site-curated tags are always merged in, which also fixes the official `character-enhancement` being dropped by the prompt's no-hyphen rule * per-file trigger words are used before the repo-wide YAML `instance_prompt` * an explicitly stated strength range is recovered by regex so `usage_tips` is populated even without an LLM The prompt gains a Site-Provided Metadata section so the LLM can prefer the site's first-hand data over its own guesses.	2026-09-14 20:38:56 +08:00
Will Miao	38d4c59b4c	feat(download): support ModelScope repositories in the URL downloader ... ModelScope became a linkable source, but downloading from it was impossible: the URL picker only recognised huggingface.co, the file listing hit a huggingface-only endpoint, the resolve URL was hardcoded, and the default path template always wrote into a `huggingface/` directory. Move the download knowledge into the providers so the handlers stay generic: - `ModelSource` gains `list_files()`, `file_download_url()`, `default_revision` and `default_subdir`. `HuggingFaceSource` keeps the Hub tree API (`/api/models/{id}/tree/{rev}`, LFS-aware sizes, `main`). `ModelScopeSource` uses `/api/v1/models/{id}/repo/files?Revision=master` — which reports real byte sizes for LFS files, so no HEAD probe is needed, and which only accepts `master` (an HF-imported repo still 404s on `main`) — and downloads through `/models/{id}/resolve/{rev}/{path}`. That URL redirects to a CDN target carrying a time-limited `auth_key`, so it is rebuilt on every request and never cached, which is also what keeps resumable Range requests working. - `hf_handlers.py`/`HfHandler` become `model_source_handlers.py`/ `ModelSourceHandler` with `list_model_source_files` and `download_model_source`. New routes `/api/lm/model-source-files` and `/api/lm/download-model-source`; the old `/api/lm/hf-repo-files` and `/api/lm/download-hf-model` paths stay as aliases, and a payload without `platform` still means Hugging Face, so existing callers are unaffected. - A downloaded sidecar now records `source_platform` + `source_url` (with the `hf_url` alias only for Hugging Face) instead of always writing `hf_url`, and `use_default_paths` files ModelScope downloads under `modelscope/<owner>/<repo>`. The now-unused shared HF aiohttp session and its shutdown hook are gone; providers open short-lived sessions. - Frontend: `detectUrlType` returns the platform-neutral `model-source-repo` / `model-source-file` plus an explicit `platform`, the DownloadManager's `hf*` state and methods are renamed to `source*`, every `source === 'huggingface'` check becomes `isExternalModelSource()`, and batch groups are keyed by `platform:repo` so the same `owner/name` on two sites renders as two groups. A bare `owner/name` still means Hugging Face. - `is_valid_source_id()` centralises repo-id validation (exactly `owner/name`, no traversal, no leading dot). This also fixes the old HF download check that rejected any dot in the name, i.e. legitimate repos such as `black-forest-labs/FLUX.1-dev`. Verified against the live APIs: the example repo lists 8 weight files with correct sizes, and a ranged GET of the built resolve URL returns 206 after following the redirect to the CDN. Backend 2853 passed; frontend 1143 JS + 91 Vue passed. The nine locales carry the refreshed download copy in the next commit.	2026-09-14 07:42:51 +08:00
Will Miao	5ab0e88abc	feat(links): support ModelScope and TensorArt as model sources ... A model file could only ever be linked to huggingface.co: `set_hf_url` validated the URL with a huggingface-only regex, the agent fetched the card from a hardcoded HF URL, and the readme processor built every relative image path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the same model-card convention (README.md + YAML frontmatter, often carrying `base_model:` and `trigger_words:`) behind a public, key-less API, so the enrichment pipeline could already serve it - it was the plumbing that was HF-shaped, not the idea. Make the external source a first-class, provider-driven concept: - New `py/services/model_sources/` registry. A `ModelSource` owns URL recognition (lenient for stored values, strict for user input), the canonical page URL, model-card fetching, the asset base URL and the capability flags. `HuggingFaceSource` is the previous logic relocated; `ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md` and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is link-only on purpose: tensor.art answers plain HTTP clients with a Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud / cn.tensorart.net) rejects every /v1/model/* route with "invalid authorization header", so it declares supports_enrichment=False rather than failing silently later. - Metadata gains `source_platform` + `source_url`; `hf_url` stays as a read/write alias, written only for Hugging Face, so existing sidecars, cached rows and third-party consumers keep working. Normalisation runs at the scanner, the persistent cache (both directions, plus two new columns behind an ALTER migration) and the linking handler - which is what stops a user who switches sources from leaving a stale `hf_url` on a ModelScope model. - The agent pipeline keys off the provider instead of `hf_url`: the fast-fail gate now explains *why* a model is skipped (no source / unknown source / source without a reachable card), the prompt context exposes source_url/source_id/source_label/asset_base_url while still filling the legacy hf_url/repo aliases, and the four README image extractors take a base_url (defaulting to HF) so relative paths resolve against the right site. Version grouping generalises to hf: / ms: / ta: keys. - `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but accepts `source_url`, validates against every provider and returns the platform. `GET /api/lm/model-sources` lets the UI render the supported-site list from the server. - Frontend: a `modelSourceHelpers` mirror of the registry drives the link dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the version-group key and the enrichment gate; the versions tab no longer sends ms:/ta: keys to the CivitAI API. TensorArt stays in the list because provenance is worth keeping even when the card is unreadable - the dialog says so plainly ("Sites that don't expose one (currently TensorArt) can only be linked") and the context menu disables enrichment with a matching tooltip, instead of the user getting "Unsupported URL". Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a 1882-byte card whose frontmatter carries base_model/tags/trigger_words, and relative images resolve to .../resolve/master/.... Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest tests/i18n and a Jinja compile pass over templates/. The nine locales carry [TODO: Translate] for the new strings, completed in the next commit.	2026-09-14 07:24:08 +08:00