2 Commits

Author	SHA1	Message	Date
Will Miao	38d4c59b4c	feat(download): support ModelScope repositories in the URL downloader ... ModelScope became a linkable source, but downloading from it was impossible: the URL picker only recognised huggingface.co, the file listing hit a huggingface-only endpoint, the resolve URL was hardcoded, and the default path template always wrote into a `huggingface/` directory. Move the download knowledge into the providers so the handlers stay generic: - `ModelSource` gains `list_files()`, `file_download_url()`, `default_revision` and `default_subdir`. `HuggingFaceSource` keeps the Hub tree API (`/api/models/{id}/tree/{rev}`, LFS-aware sizes, `main`). `ModelScopeSource` uses `/api/v1/models/{id}/repo/files?Revision=master` — which reports real byte sizes for LFS files, so no HEAD probe is needed, and which only accepts `master` (an HF-imported repo still 404s on `main`) — and downloads through `/models/{id}/resolve/{rev}/{path}`. That URL redirects to a CDN target carrying a time-limited `auth_key`, so it is rebuilt on every request and never cached, which is also what keeps resumable Range requests working. - `hf_handlers.py`/`HfHandler` become `model_source_handlers.py`/ `ModelSourceHandler` with `list_model_source_files` and `download_model_source`. New routes `/api/lm/model-source-files` and `/api/lm/download-model-source`; the old `/api/lm/hf-repo-files` and `/api/lm/download-hf-model` paths stay as aliases, and a payload without `platform` still means Hugging Face, so existing callers are unaffected. - A downloaded sidecar now records `source_platform` + `source_url` (with the `hf_url` alias only for Hugging Face) instead of always writing `hf_url`, and `use_default_paths` files ModelScope downloads under `modelscope/<owner>/<repo>`. The now-unused shared HF aiohttp session and its shutdown hook are gone; providers open short-lived sessions. - Frontend: `detectUrlType` returns the platform-neutral `model-source-repo` / `model-source-file` plus an explicit `platform`, the DownloadManager's `hf*` state and methods are renamed to `source*`, every `source === 'huggingface'` check becomes `isExternalModelSource()`, and batch groups are keyed by `platform:repo` so the same `owner/name` on two sites renders as two groups. A bare `owner/name` still means Hugging Face. - `is_valid_source_id()` centralises repo-id validation (exactly `owner/name`, no traversal, no leading dot). This also fixes the old HF download check that rejected any dot in the name, i.e. legitimate repos such as `black-forest-labs/FLUX.1-dev`. Verified against the live APIs: the example repo lists 8 weight files with correct sizes, and a ranged GET of the built resolve URL returns 206 after following the redirect to the CDN. Backend 2853 passed; frontend 1143 JS + 91 Vue passed. The nine locales carry the refreshed download copy in the next commit.	2026-09-14 07:42:51 +08:00
Will Miao	5ab0e88abc	feat(links): support ModelScope and TensorArt as model sources ... A model file could only ever be linked to huggingface.co: `set_hf_url` validated the URL with a huggingface-only regex, the agent fetched the card from a hardcoded HF URL, and the readme processor built every relative image path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the same model-card convention (README.md + YAML frontmatter, often carrying `base_model:` and `trigger_words:`) behind a public, key-less API, so the enrichment pipeline could already serve it - it was the plumbing that was HF-shaped, not the idea. Make the external source a first-class, provider-driven concept: - New `py/services/model_sources/` registry. A `ModelSource` owns URL recognition (lenient for stored values, strict for user input), the canonical page URL, model-card fetching, the asset base URL and the capability flags. `HuggingFaceSource` is the previous logic relocated; `ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md` and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is link-only on purpose: tensor.art answers plain HTTP clients with a Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud / cn.tensorart.net) rejects every /v1/model/* route with "invalid authorization header", so it declares supports_enrichment=False rather than failing silently later. - Metadata gains `source_platform` + `source_url`; `hf_url` stays as a read/write alias, written only for Hugging Face, so existing sidecars, cached rows and third-party consumers keep working. Normalisation runs at the scanner, the persistent cache (both directions, plus two new columns behind an ALTER migration) and the linking handler - which is what stops a user who switches sources from leaving a stale `hf_url` on a ModelScope model. - The agent pipeline keys off the provider instead of `hf_url`: the fast-fail gate now explains *why* a model is skipped (no source / unknown source / source without a reachable card), the prompt context exposes source_url/source_id/source_label/asset_base_url while still filling the legacy hf_url/repo aliases, and the four README image extractors take a base_url (defaulting to HF) so relative paths resolve against the right site. Version grouping generalises to hf: / ms: / ta: keys. - `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but accepts `source_url`, validates against every provider and returns the platform. `GET /api/lm/model-sources` lets the UI render the supported-site list from the server. - Frontend: a `modelSourceHelpers` mirror of the registry drives the link dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the version-group key and the enrichment gate; the versions tab no longer sends ms:/ta: keys to the CivitAI API. TensorArt stays in the list because provenance is worth keeping even when the card is unreadable - the dialog says so plainly ("Sites that don't expose one (currently TensorArt) can only be linked") and the context menu disables enrichment with a matching tooltip, instead of the user getting "Unsupported URL". Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a 1882-byte card whose frontmatter carries base_model/tags/trigger_words, and relative images resolve to .../resolve/master/.... Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest tests/i18n and a Jinja compile pass over templates/. The nine locales carry [TODO: Translate] for the new strings, completed in the next commit.	2026-09-14 07:24:08 +08:00