2 Commits

Author	SHA1	Message	Date
Will Miao	d572292142	feat(download): fill model metadata from the source API on download ... A ModelScope or Hugging Face download landed as a bare filename, hash and source link; the model card stayed empty until the user ran "Enrich Metadata with AI" by hand. But everything that makes a CivitAI download useful — the display name, the description, the tags, the trigger words, the example images, the preview — is already published by those sites' public APIs, so asking for it at download time is deterministic work, not model work. Add `py/services/model_sources/hydration.py`, called by `_save_source_metadata()` once the sidecar exists and the file is in the scanner cache. It fetches the model card plus the site's card extras and hands them to the same `PostProcessor` the AI skill uses, with an empty `llm_output`, so the two paths cannot drift apart. What lands: * `model_name` from the site's own display name (ModelScope's `Name`), so the card stops showing the local filename — written only while the value still equals the file stem, since once a user renames a model that choice is theirs to keep * `civitai.name` from the matched version's label (`showName`), which the card renders as the version chip * `civitai.description` / `modelDescription` from the author summary plus the README as HTML * `civitai.images` / `preview_url` from the per-file example images * `civitai.trainedWords` from the per-file trigger words * `base_model`, `tags` and `usage_tips` as before Provenance stays honest: the pass records `metadata_source = "source:<platform>"` rather than the skill's `agent:enrich_hf_metadata`, and — because no provider ran — it no longer stamps `llm_enriched_at`; that stamp is now conditional on the LLM actually answering, which is what the field means. The five hand-rolled `civitai` dict merges in the post-processor collapse into one `_merge_civitai()` helper. Two guards keep it safe. Only a model whose stored `source_platform`/`source_url` match the repository being downloaded is updated, so a local file that merely shares a name never receives another model's card; and a file already on disk is topped up too, which back-fills models downloaded before this existed. READMEs and detail payloads describe the repository rather than the file, so a short-lived process-wide `ModelSourceCache` (300 s, 32 entries) keeps a batch over one repository to two HTTP requests. Every failure is logged and swallowed: hydration can never fail a download. Fix the hash policy while here. `_save_source_metadata()` went straight to `MetadataManager.create_default_metadata()`, bypassing the per-type factory on the owning scanner, so a checkpoint paid a full SHA256 inside the download request — `CheckpointScanner`/`OtherScanner` deliberately record `hash_status="pending"` with an empty `sha256` for their multi-GB files. Metadata is now created through `scanner._create_default_metadata()`. Hydration copes with the empty hash: `_matching_versions()` falls back to the repository basename, which is exactly what the download just wrote. Report both post-transfer stages, which advance no byte counter and so read as a stall: the bar sat at 100% showing `0 B/s` for the seconds spent hashing and fetching. `_report_phase()` broadcasts `{"status": "metadata", "stage": "indexing" | "source", "platform": ...}`, and `LoadingManager` names the stage in the status line (keeping the batch position), retitles the item line, replaces the dead speed figure and runs a sheen over the bar. `stage`/`platform` are machine-readable; the wording is localised in the frontend. Finally, `modelscope.ai` is its own catalogue rather than an alias of `modelscope.cn` — `referall13/EM1` exists only on `.ai` and `jj3550945163/Krea-2-LORA` only on `.cn` — so its URLs were rejected with "Invalid model URL format". Register it as `ModelScopeIntlSource` (`platform="modelscope-ai"`, `msai:` group prefix, its own default download directory) and derive every URL either deployment builds from a per-class `base_url`. `modelscope.com` stays an alias of `.cn`, which is what it redirects to. The frontend source table, the link dialog hints and the docs mirror the split. Verified against the live APIs: both reported `.ai` repositories list their files, read their READMEs and yield name / version / base model / trigger words / example images. Backend 3092 passed; frontend 1259 JS + 91 Vue passed. The nine locales carry the new progress copy in the next commit.	2026-09-17 07:47:07 +08:00
Will Miao	5ab0e88abc	feat(links): support ModelScope and TensorArt as model sources ... A model file could only ever be linked to huggingface.co: `set_hf_url` validated the URL with a huggingface-only regex, the agent fetched the card from a hardcoded HF URL, and the readme processor built every relative image path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the same model-card convention (README.md + YAML frontmatter, often carrying `base_model:` and `trigger_words:`) behind a public, key-less API, so the enrichment pipeline could already serve it - it was the plumbing that was HF-shaped, not the idea. Make the external source a first-class, provider-driven concept: - New `py/services/model_sources/` registry. A `ModelSource` owns URL recognition (lenient for stored values, strict for user input), the canonical page URL, model-card fetching, the asset base URL and the capability flags. `HuggingFaceSource` is the previous logic relocated; `ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md` and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is link-only on purpose: tensor.art answers plain HTTP clients with a Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud / cn.tensorart.net) rejects every /v1/model/* route with "invalid authorization header", so it declares supports_enrichment=False rather than failing silently later. - Metadata gains `source_platform` + `source_url`; `hf_url` stays as a read/write alias, written only for Hugging Face, so existing sidecars, cached rows and third-party consumers keep working. Normalisation runs at the scanner, the persistent cache (both directions, plus two new columns behind an ALTER migration) and the linking handler - which is what stops a user who switches sources from leaving a stale `hf_url` on a ModelScope model. - The agent pipeline keys off the provider instead of `hf_url`: the fast-fail gate now explains *why* a model is skipped (no source / unknown source / source without a reachable card), the prompt context exposes source_url/source_id/source_label/asset_base_url while still filling the legacy hf_url/repo aliases, and the four README image extractors take a base_url (defaulting to HF) so relative paths resolve against the right site. Version grouping generalises to hf: / ms: / ta: keys. - `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but accepts `source_url`, validates against every provider and returns the platform. `GET /api/lm/model-sources` lets the UI render the supported-site list from the server. - Frontend: a `modelSourceHelpers` mirror of the registry drives the link dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the version-group key and the enrichment gate; the versions tab no longer sends ms:/ta: keys to the CivitAI API. TensorArt stays in the list because provenance is worth keeping even when the card is unreadable - the dialog says so plainly ("Sites that don't expose one (currently TensorArt) can only be linked") and the context menu disables enrichment with a matching tooltip, instead of the user getting "Unsupported URL". Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a 1882-byte card whose frontmatter carries base_model/tags/trigger_words, and relative images resolve to .../resolve/master/.... Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest tests/i18n and a Jinja compile pass over templates/. The nine locales carry [TODO: Translate] for the new strings, completed in the next commit.	2026-09-14 07:24:08 +08:00