2 Commits

Author	SHA1	Message	Date
Will Miao	d572292142	feat(download): fill model metadata from the source API on download ... A ModelScope or Hugging Face download landed as a bare filename, hash and source link; the model card stayed empty until the user ran "Enrich Metadata with AI" by hand. But everything that makes a CivitAI download useful — the display name, the description, the tags, the trigger words, the example images, the preview — is already published by those sites' public APIs, so asking for it at download time is deterministic work, not model work. Add `py/services/model_sources/hydration.py`, called by `_save_source_metadata()` once the sidecar exists and the file is in the scanner cache. It fetches the model card plus the site's card extras and hands them to the same `PostProcessor` the AI skill uses, with an empty `llm_output`, so the two paths cannot drift apart. What lands: * `model_name` from the site's own display name (ModelScope's `Name`), so the card stops showing the local filename — written only while the value still equals the file stem, since once a user renames a model that choice is theirs to keep * `civitai.name` from the matched version's label (`showName`), which the card renders as the version chip * `civitai.description` / `modelDescription` from the author summary plus the README as HTML * `civitai.images` / `preview_url` from the per-file example images * `civitai.trainedWords` from the per-file trigger words * `base_model`, `tags` and `usage_tips` as before Provenance stays honest: the pass records `metadata_source = "source:<platform>"` rather than the skill's `agent:enrich_hf_metadata`, and — because no provider ran — it no longer stamps `llm_enriched_at`; that stamp is now conditional on the LLM actually answering, which is what the field means. The five hand-rolled `civitai` dict merges in the post-processor collapse into one `_merge_civitai()` helper. Two guards keep it safe. Only a model whose stored `source_platform`/`source_url` match the repository being downloaded is updated, so a local file that merely shares a name never receives another model's card; and a file already on disk is topped up too, which back-fills models downloaded before this existed. READMEs and detail payloads describe the repository rather than the file, so a short-lived process-wide `ModelSourceCache` (300 s, 32 entries) keeps a batch over one repository to two HTTP requests. Every failure is logged and swallowed: hydration can never fail a download. Fix the hash policy while here. `_save_source_metadata()` went straight to `MetadataManager.create_default_metadata()`, bypassing the per-type factory on the owning scanner, so a checkpoint paid a full SHA256 inside the download request — `CheckpointScanner`/`OtherScanner` deliberately record `hash_status="pending"` with an empty `sha256` for their multi-GB files. Metadata is now created through `scanner._create_default_metadata()`. Hydration copes with the empty hash: `_matching_versions()` falls back to the repository basename, which is exactly what the download just wrote. Report both post-transfer stages, which advance no byte counter and so read as a stall: the bar sat at 100% showing `0 B/s` for the seconds spent hashing and fetching. `_report_phase()` broadcasts `{"status": "metadata", "stage": "indexing" | "source", "platform": ...}`, and `LoadingManager` names the stage in the status line (keeping the batch position), retitles the item line, replaces the dead speed figure and runs a sheen over the bar. `stage`/`platform` are machine-readable; the wording is localised in the frontend. Finally, `modelscope.ai` is its own catalogue rather than an alias of `modelscope.cn` — `referall13/EM1` exists only on `.ai` and `jj3550945163/Krea-2-LORA` only on `.cn` — so its URLs were rejected with "Invalid model URL format". Register it as `ModelScopeIntlSource` (`platform="modelscope-ai"`, `msai:` group prefix, its own default download directory) and derive every URL either deployment builds from a per-class `base_url`. `modelscope.com` stays an alias of `.cn`, which is what it redirects to. The frontend source table, the link dialog hints and the docs mirror the split. Verified against the live APIs: both reported `.ai` repositories list their files, read their READMEs and yield name / version / base model / trigger words / example images. Backend 3092 passed; frontend 1259 JS + 91 Vue passed. The nine locales carry the new progress copy in the next commit.	2026-09-17 07:47:07 +08:00
Will Miao	38d4c59b4c	feat(download): support ModelScope repositories in the URL downloader ... ModelScope became a linkable source, but downloading from it was impossible: the URL picker only recognised huggingface.co, the file listing hit a huggingface-only endpoint, the resolve URL was hardcoded, and the default path template always wrote into a `huggingface/` directory. Move the download knowledge into the providers so the handlers stay generic: - `ModelSource` gains `list_files()`, `file_download_url()`, `default_revision` and `default_subdir`. `HuggingFaceSource` keeps the Hub tree API (`/api/models/{id}/tree/{rev}`, LFS-aware sizes, `main`). `ModelScopeSource` uses `/api/v1/models/{id}/repo/files?Revision=master` — which reports real byte sizes for LFS files, so no HEAD probe is needed, and which only accepts `master` (an HF-imported repo still 404s on `main`) — and downloads through `/models/{id}/resolve/{rev}/{path}`. That URL redirects to a CDN target carrying a time-limited `auth_key`, so it is rebuilt on every request and never cached, which is also what keeps resumable Range requests working. - `hf_handlers.py`/`HfHandler` become `model_source_handlers.py`/ `ModelSourceHandler` with `list_model_source_files` and `download_model_source`. New routes `/api/lm/model-source-files` and `/api/lm/download-model-source`; the old `/api/lm/hf-repo-files` and `/api/lm/download-hf-model` paths stay as aliases, and a payload without `platform` still means Hugging Face, so existing callers are unaffected. - A downloaded sidecar now records `source_platform` + `source_url` (with the `hf_url` alias only for Hugging Face) instead of always writing `hf_url`, and `use_default_paths` files ModelScope downloads under `modelscope/<owner>/<repo>`. The now-unused shared HF aiohttp session and its shutdown hook are gone; providers open short-lived sessions. - Frontend: `detectUrlType` returns the platform-neutral `model-source-repo` / `model-source-file` plus an explicit `platform`, the DownloadManager's `hf*` state and methods are renamed to `source*`, every `source === 'huggingface'` check becomes `isExternalModelSource()`, and batch groups are keyed by `platform:repo` so the same `owner/name` on two sites renders as two groups. A bare `owner/name` still means Hugging Face. - `is_valid_source_id()` centralises repo-id validation (exactly `owner/name`, no traversal, no leading dot). This also fixes the old HF download check that rejected any dot in the name, i.e. legitimate repos such as `black-forest-labs/FLUX.1-dev`. Verified against the live APIs: the example repo lists 8 weight files with correct sizes, and a ranged GET of the built resolve URL returns 206 after following the redirect to the CDN. Backend 2853 passed; frontend 1143 JS + 91 Vue passed. The nine locales carry the refreshed download copy in the next commit.	2026-09-14 07:42:51 +08:00