alphard/ComfyUI-Lora-Manager
1
0
Fork 0
mirror of https://github.com/willmiao/ComfyUI-Lora-Manager.git synced 2026-09-21 03:01:27 -03:00
Code Issues Packages Projects Releases Wiki Activity

feat(download): fill model metadata from the source API on download

Browse Source 
A ModelScope or Hugging Face download landed as a bare filename, hash and
source link; the model card stayed empty until the user ran "Enrich
Metadata with AI" by hand. But everything that makes a CivitAI download
useful — the display name, the description, the tags, the trigger words,
the example images, the preview — is already published by those sites'
public APIs, so asking for it at download time is deterministic work, not
model work.

Add `py/services/model_sources/hydration.py`, called by
`_save_source_metadata()` once the sidecar exists and the file is in the
scanner cache. It fetches the model card plus the site's card extras and
hands them to the same `PostProcessor` the AI skill uses, with an empty
`llm_output`, so the two paths cannot drift apart. What lands:

* `model_name` from the site's own display name (ModelScope's `Name`), so
  the card stops showing the local filename — written only while the value
  still equals the file stem, since once a user renames a model that
  choice is theirs to keep
* `civitai.name` from the matched version's label (`showName`), which the
  card renders as the version chip
* `civitai.description` / `modelDescription` from the author summary plus
  the README as HTML
* `civitai.images` / `preview_url` from the per-file example images
* `civitai.trainedWords` from the per-file trigger words
* `base_model`, `tags` and `usage_tips` as before

Provenance stays honest: the pass records
`metadata_source = "source:<platform>"` rather than the skill's
`agent:enrich_hf_metadata`, and — because no provider ran — it no longer
stamps `llm_enriched_at`; that stamp is now conditional on the LLM
actually answering, which is what the field means. The five hand-rolled
`civitai` dict merges in the post-processor collapse into one
`_merge_civitai()` helper.

Two guards keep it safe. Only a model whose stored
`source_platform`/`source_url` match the repository being downloaded is
updated, so a local file that merely shares a name never receives another
model's card; and a file already on disk is topped up too, which
back-fills models downloaded before this existed. READMEs and detail
payloads describe the repository rather than the file, so a short-lived
process-wide `ModelSourceCache` (300 s, 32 entries) keeps a batch over one
repository to two HTTP requests. Every failure is logged and swallowed:
hydration can never fail a download.

Fix the hash policy while here. `_save_source_metadata()` went straight to
`MetadataManager.create_default_metadata()`, bypassing the per-type
factory on the owning scanner, so a checkpoint paid a full SHA256 inside
the download request — `CheckpointScanner`/`OtherScanner` deliberately
record `hash_status="pending"` with an empty `sha256` for their multi-GB
files. Metadata is now created through `scanner._create_default_metadata()`.
Hydration copes with the empty hash: `_matching_versions()` falls back to
the repository basename, which is exactly what the download just wrote.

Report both post-transfer stages, which advance no byte counter and so
read as a stall: the bar sat at 100% showing `0 B/s` for the seconds spent
hashing and fetching. `_report_phase()` broadcasts
`{"status": "metadata", "stage": "indexing" | "source", "platform": ...}`,
and `LoadingManager` names the stage in the status line (keeping the batch
position), retitles the item line, replaces the dead speed figure and runs
a sheen over the bar. `stage`/`platform` are machine-readable; the wording
is localised in the frontend.

Finally, `modelscope.ai` is its own catalogue rather than an alias of
`modelscope.cn` — `referall13/EM1` exists only on `.ai` and
`jj3550945163/Krea-2-LORA` only on `.cn` — so its URLs were rejected with
"Invalid model URL format". Register it as `ModelScopeIntlSource`
(`platform="modelscope-ai"`, `msai:` group prefix, its own default
download directory) and derive every URL either deployment builds from a
per-class `base_url`. `modelscope.com` stays an alias of `.cn`, which is
what it redirects to. The frontend source table, the link dialog hints and
the docs mirror the split.

Verified against the live APIs: both reported `.ai` repositories list
their files, read their READMEs and yield name / version / base model /
trigger words / example images. Backend 3092 passed; frontend 1259 JS +
91 Vue passed. The nine locales carry the new progress copy in the next
commit.
This commit is contained in:
Will Miao
2026-09-17 07:47:07 +08:00
parent 1b1a8d63db
commit d572292142
34 changed files with 2385 additions and 135 deletions
Download Patch File Download Diff File Expand all files Collapse all files
static/js/managers/DownloadManager.js
+30
View File
@@ -340,6 +340,27 @@ export class DownloadManager {
// ---- External repository download flow (Hugging Face / ModelScope) ----
/**
* Report a post-transfer stage frame to the progress UI.
*
* The backend keeps working after the last byte lands — it indexes the
* file and reads the model site's API — and announces those stages with
* `status: 'metadata'`. Without them the bar sits at 100% showing "0 B/s"
* and the download looks stuck. The stage and platform are machine
* readable so LoadingManager can localise the wording.
*
* @returns {boolean} `true` when the frame was a stage frame.
*/
_applyMetadataStage(data, updateProgress, completed, name) {
if (data?.status !== 'metadata') return false;
updateProgress(100, completed, name, {}, {
phase: 'metadata',
stage: data.stage || '',
platform: data.platform || '',
});
return true;
}
/** Rendering group key: the same repo on two sites is two groups. */
_externalGroupKey(item) {
return `${item.source}:${item.repo || 'unknown'}`;
@@ -1708,6 +1729,12 @@ export class DownloadManager {
cancelled = true;
return;
}
// Indexing / site metadata: the transfer is over but the
// backend is still working, so say so instead of
// leaving the bar frozen at 100%.
if (this._applyMetadataStage(data, updateProgress, snapshotCompleted, filename)) {
return;
}
if (data.status === 'progress') {
const metrics = {
bytesDownloaded: data.bytes_downloaded,
@@ -2331,6 +2358,9 @@ export class DownloadManager {
const snapshotCompleted = completedDownloads;
wsHf.onmessage = (event) => {
const data = JSON.parse(event.data);
if (this._applyMetadataStage(data, updateProgress, snapshotCompleted, name)) {
return;
}
if (data.status === 'progress') {
const metrics = {
bytesDownloaded: data.bytes_downloaded,
static/js/managers/LoadingManager.js
+79 -8
View File
@@ -1,5 +1,6 @@
import { translate } from '../utils/i18nHelpers.js';
import { formatFileSize } from '../utils/formatters.js';
import { getModelSource } from '../utils/modelSourceHelpers.js';
// Loading management
export class LoadingManager {
@@ -278,6 +279,35 @@ export class LoadingManager {
}
};
/**
* Describe a post-transfer stage in the status line.
*
* The byte counter stops as soon as the last byte lands, but the
* backend still hashes the file and reads the model site's API. Naming
* that work is what stops the bar looking frozen at 100%.
*/
const describeMetadataStage = (stage, platform) => {
if (stage === 'indexing') {
return translate(
'modals.download.progress.indexingFile',
{},
'Reading model file...'
);
}
const label = getModelSource(platform)?.label || platform || '';
return label
? translate(
'modals.download.progress.fetchingSourceMetadata',
{ source: label },
`Fetching metadata from ${label}...`
)
: translate(
'modals.download.progress.fetchingMetadata',
{},
'Fetching metadata...'
);
};
// Initialize transfer stats with empty data
updateTransferStats();
@@ -285,19 +315,62 @@ export class LoadingManager {
this.loadingContent.appendChild(this.cancelButton);
}
// Return update function
return (currentProgress, currentIndex = 0, currentName = '', metrics = {}) => {
/**
* Update the progress UI.
*
* @param {number} currentProgress Percentage of the current item.
* @param {number} [currentIndex] Items finished so far.
* @param {string} [currentName] File being processed.
* @param {object} [metrics] Byte counters; only meaningful while
* transferring.
* @param {object} [phase] `{ phase: 'metadata', stage, platform }` once
* the transfer has finished, so the UI can show what is still running
* instead of a 0 B/s speed.
*/
return (
currentProgress,
currentIndex = 0,
currentName = '',
metrics = {},
phase = null
) => {
const isMetadata = phase?.phase === 'metadata';
// Update current item progress
currentItemProgress.style.width = `${currentProgress}%`;
currentItemPercent.textContent = `${Math.floor(currentProgress)}%`;
currentItemProgress.classList.toggle('is-indeterminate', isMetadata);
// Update current item label if name provided
if (currentName) {
currentItemLabel.textContent = translate(
'modals.download.progress.downloading',
{ name: currentName },
`Downloading: ${currentName}`
currentItemLabel.textContent = isMetadata
? translate(
'modals.download.progress.metadata',
{ name: currentName },
`Metadata: ${currentName}`
)
: translate(
'modals.download.progress.downloading',
{ name: currentName },
`Downloading: ${currentName}`
);
}
// No bytes are moving any more, so report the stage instead of a
// rate that has dropped to zero.
if (isMetadata) {
updateTransferStats({ bytesDownloaded: metrics.bytesDownloaded, totalBytes: metrics.totalBytes });
const stageText = describeMetadataStage(phase.stage, phase.platform);
speedDetail.textContent = stageText;
// Keep the batch position visible; the status line is the one
// place a caller also writes to.
this.setStatus(
totalItems > 1
? `${Math.min(currentIndex + 1, totalItems)}/${totalItems}: ${stageText}`
: stageText
);
} else {
updateTransferStats(metrics);
}
// Update overall label if multiple items
@@ -311,8 +384,6 @@ export class LoadingManager {
// Single item, just update main progress
this.setProgress(currentProgress);
}
updateTransferStats(metrics);
};
}