mirror of
https://github.com/willmiao/ComfyUI-Lora-Manager.git
synced 2026-09-20 18:51:26 -03:00
d572292142
A ModelScope or Hugging Face download landed as a bare filename, hash and
source link; the model card stayed empty until the user ran "Enrich
Metadata with AI" by hand. But everything that makes a CivitAI download
useful — the display name, the description, the tags, the trigger words,
the example images, the preview — is already published by those sites'
public APIs, so asking for it at download time is deterministic work, not
model work.
Add `py/services/model_sources/hydration.py`, called by
`_save_source_metadata()` once the sidecar exists and the file is in the
scanner cache. It fetches the model card plus the site's card extras and
hands them to the same `PostProcessor` the AI skill uses, with an empty
`llm_output`, so the two paths cannot drift apart. What lands:
* `model_name` from the site's own display name (ModelScope's `Name`), so
the card stops showing the local filename — written only while the value
still equals the file stem, since once a user renames a model that
choice is theirs to keep
* `civitai.name` from the matched version's label (`showName`), which the
card renders as the version chip
* `civitai.description` / `modelDescription` from the author summary plus
the README as HTML
* `civitai.images` / `preview_url` from the per-file example images
* `civitai.trainedWords` from the per-file trigger words
* `base_model`, `tags` and `usage_tips` as before
Provenance stays honest: the pass records
`metadata_source = "source:<platform>"` rather than the skill's
`agent:enrich_hf_metadata`, and — because no provider ran — it no longer
stamps `llm_enriched_at`; that stamp is now conditional on the LLM
actually answering, which is what the field means. The five hand-rolled
`civitai` dict merges in the post-processor collapse into one
`_merge_civitai()` helper.
Two guards keep it safe. Only a model whose stored
`source_platform`/`source_url` match the repository being downloaded is
updated, so a local file that merely shares a name never receives another
model's card; and a file already on disk is topped up too, which
back-fills models downloaded before this existed. READMEs and detail
payloads describe the repository rather than the file, so a short-lived
process-wide `ModelSourceCache` (300 s, 32 entries) keeps a batch over one
repository to two HTTP requests. Every failure is logged and swallowed:
hydration can never fail a download.
Fix the hash policy while here. `_save_source_metadata()` went straight to
`MetadataManager.create_default_metadata()`, bypassing the per-type
factory on the owning scanner, so a checkpoint paid a full SHA256 inside
the download request — `CheckpointScanner`/`OtherScanner` deliberately
record `hash_status="pending"` with an empty `sha256` for their multi-GB
files. Metadata is now created through `scanner._create_default_metadata()`.
Hydration copes with the empty hash: `_matching_versions()` falls back to
the repository basename, which is exactly what the download just wrote.
Report both post-transfer stages, which advance no byte counter and so
read as a stall: the bar sat at 100% showing `0 B/s` for the seconds spent
hashing and fetching. `_report_phase()` broadcasts
`{"status": "metadata", "stage": "indexing" | "source", "platform": ...}`,
and `LoadingManager` names the stage in the status line (keeping the batch
position), retitles the item line, replaces the dead speed figure and runs
a sheen over the bar. `stage`/`platform` are machine-readable; the wording
is localised in the frontend.
Finally, `modelscope.ai` is its own catalogue rather than an alias of
`modelscope.cn` — `referall13/EM1` exists only on `.ai` and
`jj3550945163/Krea-2-LORA` only on `.cn` — so its URLs were rejected with
"Invalid model URL format". Register it as `ModelScopeIntlSource`
(`platform="modelscope-ai"`, `msai:` group prefix, its own default
download directory) and derive every URL either deployment builds from a
per-class `base_url`. `modelscope.com` stays an alias of `.cn`, which is
what it redirects to. The frontend source table, the link dialog hints and
the docs mirror the split.
Verified against the live APIs: both reported `.ai` repositories list
their files, read their READMEs and yield name / version / base model /
trigger words / example images. Backend 3092 passed; frontend 1259 JS +
91 Vue passed. The nine locales carry the new progress copy in the next
commit.
231 lines
8.2 KiB
JavaScript
231 lines
8.2 KiB
JavaScript
import { describe, it, expect } from 'vitest';
|
|
import { DownloadManager } from '../../../static/js/managers/DownloadManager.js';
|
|
|
|
describe('DownloadManager.detectUrlType — external model source URLs', () => {
|
|
|
|
it('detects HF resolve URL with file', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://huggingface.co/dx8152/Flux2-Klein-9B-Consistency/resolve/main/Flux2-Klein-9B-consistency-V2.safetensors'
|
|
);
|
|
expect(result).toEqual({
|
|
type: 'model-source-file',
|
|
platform: 'huggingface',
|
|
repo: 'dx8152/Flux2-Klein-9B-Consistency',
|
|
revision: 'main',
|
|
filename: 'Flux2-Klein-9B-consistency-V2.safetensors',
|
|
});
|
|
});
|
|
|
|
it('detects HF resolve URL with subdirectory file', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://huggingface.co/user/repo/resolve/main/subdir/model.safetensors'
|
|
);
|
|
expect(result).toEqual({
|
|
type: 'model-source-file',
|
|
platform: 'huggingface',
|
|
repo: 'user/repo',
|
|
revision: 'main',
|
|
filename: 'subdir/model.safetensors',
|
|
});
|
|
});
|
|
|
|
it('detects HF repo URL (full URL)', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://huggingface.co/dx8152/Flux2-Klein-9B-Consistency'
|
|
);
|
|
expect(result).toEqual({
|
|
type: 'model-source-repo',
|
|
platform: 'huggingface',
|
|
repo: 'dx8152/Flux2-Klein-9B-Consistency',
|
|
});
|
|
});
|
|
|
|
it('detects HF repo URL (bare user/repo)', () => {
|
|
const result = DownloadManager.detectUrlType('dx8152/Flux2-Klein-9B-Consistency');
|
|
expect(result).toEqual({
|
|
type: 'model-source-repo',
|
|
platform: 'huggingface',
|
|
repo: 'dx8152/Flux2-Klein-9B-Consistency',
|
|
});
|
|
});
|
|
|
|
it('detects HF repo URL with trailing slash', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://huggingface.co/user/repo/'
|
|
);
|
|
expect(result).toEqual({
|
|
type: 'model-source-repo',
|
|
platform: 'huggingface',
|
|
repo: 'user/repo',
|
|
});
|
|
});
|
|
|
|
it('detects HF blob (web preview) URL as resolve', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://huggingface.co/Comfy-Org/z_image_turbo/blob/main/split_files/diffusion_models/z_image_turbo_bf16.safetensors'
|
|
);
|
|
expect(result).toEqual({
|
|
type: 'model-source-file',
|
|
platform: 'huggingface',
|
|
repo: 'Comfy-Org/z_image_turbo',
|
|
revision: 'main',
|
|
filename: 'split_files/diffusion_models/z_image_turbo_bf16.safetensors',
|
|
});
|
|
});
|
|
|
|
it('detects CivitAI URL', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://civitai.com/models/123/some-model'
|
|
);
|
|
expect(result).toEqual({ type: 'civitai' });
|
|
});
|
|
|
|
it('detects CivitAI URL on civitai.red domain', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://civitai.red/models/12345/my-model'
|
|
);
|
|
expect(result).toEqual({ type: 'civitai' });
|
|
});
|
|
|
|
it('detects CivitAI URL on civitai.green domain', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://civitai.green/models/67890/another-model'
|
|
);
|
|
expect(result).toEqual({ type: 'civitai' });
|
|
});
|
|
|
|
it('detects CivArchive URL', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://civarchive.com/models/456'
|
|
);
|
|
expect(result).toEqual({ type: 'civitai' });
|
|
});
|
|
|
|
it('detects direct HTTP URL', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://example.com/file.zip'
|
|
);
|
|
expect(result).toEqual({ type: 'direct-http' });
|
|
});
|
|
|
|
it('returns null for invalid input', () => {
|
|
expect(DownloadManager.detectUrlType('')).toBeNull();
|
|
expect(DownloadManager.detectUrlType(' ')).toBeNull();
|
|
});
|
|
|
|
it('returns null for unrecognized path', () => {
|
|
expect(DownloadManager.detectUrlType('justrandomtext')).toBeNull();
|
|
});
|
|
|
|
it('prefers HF resolve over repo when both match', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://huggingface.co/user/repo/resolve/main/file.safetensors'
|
|
);
|
|
expect(result?.type).toBe('model-source-file');
|
|
});
|
|
|
|
it('prefers CivitAI over HF when both match', () => {
|
|
// CivitAI check comes first in detectUrlType
|
|
// This URL should be detected as CivitAI, not HF
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://civitai.com/models/123?huggingface.co/test/repo'
|
|
);
|
|
expect(result?.type).toBe('civitai');
|
|
});
|
|
|
|
it('detects a ModelScope repo URL', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://modelscope.cn/models/jj3550945163/Krea-2-LORA'
|
|
);
|
|
expect(result).toEqual({
|
|
type: 'model-source-repo',
|
|
platform: 'modelscope',
|
|
repo: 'jj3550945163/Krea-2-LORA',
|
|
});
|
|
});
|
|
|
|
it('detects a ModelScope file URL with revision and subdirectory', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://modelscope.cn/models/AI-ModelScope/stable-diffusion-v1-5/resolve/master/vae/diffusion_pytorch_model.bin'
|
|
);
|
|
expect(result).toEqual({
|
|
type: 'model-source-file',
|
|
platform: 'modelscope',
|
|
repo: 'AI-ModelScope/stable-diffusion-v1-5',
|
|
revision: 'master',
|
|
filename: 'vae/diffusion_pytorch_model.bin',
|
|
});
|
|
});
|
|
|
|
it('detects a ModelScope view sub-page as a repo URL', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://www.modelscope.cn/models/user/repo/summary'
|
|
);
|
|
expect(result).toEqual({
|
|
type: 'model-source-repo',
|
|
platform: 'modelscope',
|
|
repo: 'user/repo',
|
|
});
|
|
});
|
|
|
|
it('does not treat a bare owner/name as ModelScope', () => {
|
|
// The shorthand has always meant Hugging Face; ModelScope needs its host.
|
|
const result = DownloadManager.detectUrlType('user/repo');
|
|
expect(result.platform).toBe('huggingface');
|
|
});
|
|
|
|
// modelscope.ai is a separate catalogue from modelscope.cn, not an alias,
|
|
// so it carries its own platform id all the way to the backend.
|
|
it('detects a ModelScope International repo URL', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://www.modelscope.ai/models/referall13/EM1'
|
|
);
|
|
expect(result).toEqual({
|
|
type: 'model-source-repo',
|
|
platform: 'modelscope-ai',
|
|
repo: 'referall13/EM1',
|
|
});
|
|
});
|
|
|
|
it('detects a ModelScope International repo URL without the www prefix', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://modelscope.ai/models/ErLubu/krea2_style_260911_02'
|
|
);
|
|
expect(result).toEqual({
|
|
type: 'model-source-repo',
|
|
platform: 'modelscope-ai',
|
|
repo: 'ErLubu/krea2_style_260911_02',
|
|
});
|
|
});
|
|
|
|
it('detects a ModelScope International file URL', () => {
|
|
const result = DownloadManager.detectUrlType(
|
|
'https://www.modelscope.ai/models/referall13/EM1/resolve/master/EM1_c1-st1000.safetensors'
|
|
);
|
|
expect(result).toEqual({
|
|
type: 'model-source-file',
|
|
platform: 'modelscope-ai',
|
|
repo: 'referall13/EM1',
|
|
revision: 'master',
|
|
filename: 'EM1_c1-st1000.safetensors',
|
|
});
|
|
});
|
|
|
|
it('keeps the two ModelScope deployments distinct', () => {
|
|
const mainland = DownloadManager.detectUrlType(
|
|
'https://modelscope.cn/models/referall13/EM1'
|
|
);
|
|
const intl = DownloadManager.detectUrlType(
|
|
'https://www.modelscope.ai/models/referall13/EM1'
|
|
);
|
|
expect(mainland.platform).toBe('modelscope');
|
|
expect(intl.platform).toBe('modelscope-ai');
|
|
});
|
|
|
|
it('rejects path traversal in either platform', () => {
|
|
expect(
|
|
DownloadManager.detectUrlType('https://modelscope.cn/models/../etc/passwd')
|
|
).toBeNull();
|
|
});
|
|
});
|