feat(download): fill model metadata from the source API on download

A ModelScope or Hugging Face download landed as a bare filename, hash and source link; the model card stayed empty until the user ran "Enrich Metadata with AI" by hand. But everything that makes a CivitAI download useful — the display name, the description, the tags, the trigger words, the example images, the preview — is already published by those sites' public APIs, so asking for it at download time is deterministic work, not model work. Add `py/services/model_sources/hydration.py`, called by `_save_source_metadata()` once the sidecar exists and the file is in the scanner cache. It fetches the model card plus the site's card extras and hands them to the same `PostProcessor` the AI skill uses, with an empty `llm_output`, so the two paths cannot drift apart. What lands: * `model_name` from the site's own display name (ModelScope's `Name`), so the card stops showing the local filename — written only while the value still equals the file stem, since once a user renames a model that choice is theirs to keep * `civitai.name` from the matched version's label (`showName`), which the card renders as the version chip * `civitai.description` / `modelDescription` from the author summary plus the README as HTML * `civitai.images` / `preview_url` from the per-file example images * `civitai.trainedWords` from the per-file trigger words * `base_model`, `tags` and `usage_tips` as before Provenance stays honest: the pass records `metadata_source = "source:<platform>"` rather than the skill's `agent:enrich_hf_metadata`, and — because no provider ran — it no longer stamps `llm_enriched_at`; that stamp is now conditional on the LLM actually answering, which is what the field means. The five hand-rolled `civitai` dict merges in the post-processor collapse into one `_merge_civitai()` helper. Two guards keep it safe. Only a model whose stored `source_platform`/`source_url` match the repository being downloaded is updated, so a local file that merely shares a name never receives another model's card; and a file already on disk is topped up too, which back-fills models downloaded before this existed. READMEs and detail payloads describe the repository rather than the file, so a short-lived process-wide `ModelSourceCache` (300 s, 32 entries) keeps a batch over one repository to two HTTP requests. Every failure is logged and swallowed: hydration can never fail a download. Fix the hash policy while here. `_save_source_metadata()` went straight to `MetadataManager.create_default_metadata()`, bypassing the per-type factory on the owning scanner, so a checkpoint paid a full SHA256 inside the download request — `CheckpointScanner`/`OtherScanner` deliberately record `hash_status="pending"` with an empty `sha256` for their multi-GB files. Metadata is now created through `scanner._create_default_metadata()`. Hydration copes with the empty hash: `_matching_versions()` falls back to the repository basename, which is exactly what the download just wrote. Report both post-transfer stages, which advance no byte counter and so read as a stall: the bar sat at 100% showing `0 B/s` for the seconds spent hashing and fetching. `_report_phase()` broadcasts `{"status": "metadata", "stage": "indexing" | "source", "platform": ...}`, and `LoadingManager` names the stage in the status line (keeping the batch position), retitles the item line, replaces the dead speed figure and runs a sheen over the bar. `stage`/`platform` are machine-readable; the wording is localised in the frontend. Finally, `modelscope.ai` is its own catalogue rather than an alias of `modelscope.cn` — `referall13/EM1` exists only on `.ai` and `jj3550945163/Krea-2-LORA` only on `.cn` — so its URLs were rejected with "Invalid model URL format". Register it as `ModelScopeIntlSource` (`platform="modelscope-ai"`, `msai:` group prefix, its own default download directory) and derive every URL either deployment builds from a per-class `base_url`. `modelscope.com` stays an alias of `.cn`, which is what it redirects to. The frontend source table, the link dialog hints and the docs mirror the split. Verified against the live APIs: both reported `.ai` repositories list their files, read their READMEs and yield name / version / base model / trigger words / example images. Backend 3092 passed; frontend 1259 JS + 91 Vue passed. The nine locales carry the new progress copy in the next commit.
2026-09-21 03:01:27 -03:00 · 2026-09-17 07:47:07 +08:00
parent 1b1a8d63db
commit d572292142
34 changed files with 2385 additions and 135 deletions
@@ -97,6 +97,32 @@
    width: 0%;
 }

+/* The transfer is done but the backend is still indexing the file and reading
+   the model site's API. A sheen over the full bar reads as "busy" where a
+   motionless 100% bar reads as "stuck". */
+.current-item-bar.is-indeterminate {
+    position: relative;
+    overflow: hidden;
+}
+
+.current-item-bar.is-indeterminate::after {
+    content: '';
+    position: absolute;
+    inset: 0;
+    background: linear-gradient(
+        90deg,
+        transparent 0%,
+        rgba(255, 255, 255, 0.5) 50%,
+        transparent 100%
+    );
+    animation: progress-sheen 1.2s ease-in-out infinite;
+}
+
+@keyframes progress-sheen {
+    from { transform: translateX(-100%); }
+    to { transform: translateX(100%); }
+}
+
 .current-item-percent {
    font-size: 0.8rem;
    color: var(--text-color-secondary, var(--text-color));
@@ -131,4 +157,8 @@
    .current-item-bar {
        transition: none;
    }
+
+    .current-item-bar.is-indeterminate::after {
+        animation: none;
+    }
 }
@@ -340,6 +340,27 @@ export class DownloadManager {

    // ---- External repository download flow (Hugging Face / ModelScope) ----

+    /**
+     * Report a post-transfer stage frame to the progress UI.
+     *
+     * The backend keeps working after the last byte lands — it indexes the
+     * file and reads the model site's API — and announces those stages with
+     * `status: 'metadata'`. Without them the bar sits at 100% showing "0 B/s"
+     * and the download looks stuck. The stage and platform are machine
+     * readable so LoadingManager can localise the wording.
+     *
+     * @returns {boolean} `true` when the frame was a stage frame.
+     */
+    _applyMetadataStage(data, updateProgress, completed, name) {
+        if (data?.status !== 'metadata') return false;
+        updateProgress(100, completed, name, {}, {
+            phase: 'metadata',
+            stage: data.stage || '',
+            platform: data.platform || '',
+        });
+        return true;
+    }
+
    /** Rendering group key: the same repo on two sites is two groups. */
    _externalGroupKey(item) {
        return `${item.source}:${item.repo || 'unknown'}`;
@@ -1708,6 +1729,12 @@ export class DownloadManager {
                            cancelled = true;
                            return;
                        }
+                        // Indexing / site metadata: the transfer is over but the
+                        // backend is still working, so say so instead of
+                        // leaving the bar frozen at 100%.
+                        if (this._applyMetadataStage(data, updateProgress, snapshotCompleted, filename)) {
+                            return;
+                        }
                        if (data.status === 'progress') {
                            const metrics = {
                                bytesDownloaded: data.bytes_downloaded,
@@ -2331,6 +2358,9 @@ export class DownloadManager {
                        const snapshotCompleted = completedDownloads;
                        wsHf.onmessage = (event) => {
                            const data = JSON.parse(event.data);
+                            if (this._applyMetadataStage(data, updateProgress, snapshotCompleted, name)) {
+                                return;
+                            }
                            if (data.status === 'progress') {
                                const metrics = {
                                    bytesDownloaded: data.bytes_downloaded,
@@ -1,5 +1,6 @@
 import { translate } from '../utils/i18nHelpers.js';
 import { formatFileSize } from '../utils/formatters.js';
+import { getModelSource } from '../utils/modelSourceHelpers.js';

 // Loading management
 export class LoadingManager {
@@ -278,6 +279,35 @@ export class LoadingManager {
            }
        };

+        /**
+         * Describe a post-transfer stage in the status line.
+         *
+         * The byte counter stops as soon as the last byte lands, but the
+         * backend still hashes the file and reads the model site's API. Naming
+         * that work is what stops the bar looking frozen at 100%.
+         */
+        const describeMetadataStage = (stage, platform) => {
+            if (stage === 'indexing') {
+                return translate(
+                    'modals.download.progress.indexingFile',
+                    {},
+                    'Reading model file...'
+                );
+            }
+            const label = getModelSource(platform)?.label || platform || '';
+            return label
+                ? translate(
+                    'modals.download.progress.fetchingSourceMetadata',
+                    { source: label },
+                    `Fetching metadata from ${label}...`
+                )
+                : translate(
+                    'modals.download.progress.fetchingMetadata',
+                    {},
+                    'Fetching metadata...'
+                );
+        };
+
        // Initialize transfer stats with empty data
        updateTransferStats();

@@ -285,19 +315,62 @@ export class LoadingManager {
            this.loadingContent.appendChild(this.cancelButton);
        }

-        // Return update function
-        return (currentProgress, currentIndex = 0, currentName = '', metrics = {}) => {
+        /**
+         * Update the progress UI.
+         *
+         * @param {number} currentProgress Percentage of the current item.
+         * @param {number} [currentIndex] Items finished so far.
+         * @param {string} [currentName] File being processed.
+         * @param {object} [metrics] Byte counters; only meaningful while
+         *   transferring.
+         * @param {object} [phase] `{ phase: 'metadata', stage, platform }` once
+         *   the transfer has finished, so the UI can show what is still running
+         *   instead of a 0 B/s speed.
+         */
+        return (
+            currentProgress,
+            currentIndex = 0,
+            currentName = '',
+            metrics = {},
+            phase = null
+        ) => {
+            const isMetadata = phase?.phase === 'metadata';
+
            // Update current item progress
            currentItemProgress.style.width = `${currentProgress}%`;
            currentItemPercent.textContent = `${Math.floor(currentProgress)}%`;
+            currentItemProgress.classList.toggle('is-indeterminate', isMetadata);

            // Update current item label if name provided
            if (currentName) {
-                currentItemLabel.textContent = translate(
-                    'modals.download.progress.downloading',
-                    { name: currentName },
-                    `Downloading: ${currentName}`
+                currentItemLabel.textContent = isMetadata
+                    ? translate(
+                        'modals.download.progress.metadata',
+                        { name: currentName },
+                        `Metadata: ${currentName}`
+                    )
+                    : translate(
+                        'modals.download.progress.downloading',
+                        { name: currentName },
+                        `Downloading: ${currentName}`
+                    );
+            }
+
+            // No bytes are moving any more, so report the stage instead of a
+            // rate that has dropped to zero.
+            if (isMetadata) {
+                updateTransferStats({ bytesDownloaded: metrics.bytesDownloaded, totalBytes: metrics.totalBytes });
+                const stageText = describeMetadataStage(phase.stage, phase.platform);
+                speedDetail.textContent = stageText;
+                // Keep the batch position visible; the status line is the one
+                // place a caller also writes to.
+                this.setStatus(
+                    totalItems > 1
+                        ? `${Math.min(currentIndex + 1, totalItems)}/${totalItems}: ${stageText}`
+                        : stageText
                );
+            } else {
+                updateTransferStats(metrics);
            }

            // Update overall label if multiple items
@@ -311,8 +384,6 @@ export class LoadingManager {
                // Single item, just update main progress
                this.setProgress(currentProgress);
            }
-
-            updateTransferStats(metrics);
        };
    }

@@ -49,6 +49,26 @@ export const MODEL_SOURCES = [
    filePage: (id, filename) =>
      `https://modelscope.cn/models/${id}/file/view/master/${filename}`,
  },
+  {
+    // A separate catalogue from `modelscope.cn`, not an alias: a repository
+    // published on one is routinely absent from the other, so the host is part
+    // of the model's identity. Mirrors ModelScopeIntlSource in the backend.
+    platform: 'modelscope-ai',
+    label: 'ModelScope (International)',
+    groupPrefix: 'msai',
+    supportsEnrichment: true,
+    supportsDownload: true,
+    defaultRevision: 'master',
+    defaultSubdir: 'modelscope-ai',
+    exampleUrl: 'https://www.modelscope.ai/models/user/repo',
+    placeholder: 'https://www.modelscope.ai/models/user/repo',
+    pattern: /^https?:\/\/(?:www\.)?modelscope\.ai\/models\/([^/?#\s]+\/[^/?#\s]+)/i,
+    filePattern:
+      /^https?:\/\/(?:www\.)?modelscope\.ai\/models\/([^/?#\s]+\/[^/?#\s]+)\/resolve\/([^/?#\s]+)\/(.+)$/i,
+    canonical: (id) => `https://www.modelscope.ai/models/${id}`,
+    filePage: (id, filename) =>
+      `https://www.modelscope.ai/models/${id}/file/view/master/${filename}`,
+  },
  {
    platform: 'tensorart',
    label: 'TensorArt',