Codex review on #1124:
- P1: mirror layout root component is now <basename>-<roothash>
(sha256 of the normalized root path), so two roots sharing a basename
no longer map to the same mirror directory and overwrite each other's
sidecars
- P1: changing sidecar_storage_path while centralized no longer strands
assets in the old root — new relocate_root migration direction moves
the whole mirror tree, rewrites preview_url prefixes inside sidecars,
reconciles scanner caches, and prunes the emptied old tree; the
settings UI detects the path change and offers the relocation
- P2: migration enumerates the same preview candidates as
find_preview_file — case-insensitive variants (model.WEBP) and the
legacy .example.0.jpeg suffix — instead of exact lowercase
PREVIEW_EXTENSIONS only
- P2: _rollback_model_staging restores staged files with the
EXDEV-tolerant mover, so a failed undoable-delete staging no longer
strands a cross-filesystem centralized sidecar copy
Tests: same-basename root injectivity, mixed-case/example preview
migration, relocate_root happy path + guards + route 400, frontend
relocation prompt flow. Verified end-to-end in a sandboxed standalone
server: uppercase/legacy previews migrate, root relocation moves the
tree and the list API serves the new locations immediately without a
rescan.
Sandbox E2E showed that after a migration the list API kept serving
pre-migration preview_url values; the first request to a stale URL made
the preview route's stale-URL cleanup wipe the reference from the cache
entirely, recoverable only by a full rebuild rescan.
The use case now records each migrated model's final preview location
(from the destination directory, covering conflict-keep cases), updates
the owning scanner's cache entries via ModelCache.update_preview_url,
and persists the cache. Per-scanner reconcile failures are logged and
skipped; per-model migration errors no longer prevent reconciliation of
the healthy models.
Verified end-to-end in a sandboxed standalone server: after
to_centralized and to_alongside migrations the list endpoint immediately
returns the correct preview URLs with no rescan, previews serve with
HTTP 200 in both layouts, and the mirror tree is empty after migrating
back.
Add an opt-in 'centralized' sidecar storage mode alongside the default
'alongside' layout. In centralized mode, .metadata.json sidecars and
preview assets live under a configurable root (sidecar_storage_path,
default <settings_dir>/sidecars), mirroring the library-relative
directory structure: <root>/<library>/<root_basename>/<rel_dir>/.
Backend:
- settings: sidecar_storage_mode / sidecar_storage_path with validation;
changing either refreshes the preview allowlist
- config: centralized root added to preview-serving allowlist
- lifecycle: delete / move / rename / folder-rename / folder-delete and
undoable-delete staging all operate on the mirror tree in centralized
mode (model files themselves never move); EXDEV-tolerant cross-
filesystem moves
- scanners: pending-hash filesystem scan walks the mirror tree in
centralized mode; preview discovery reads from the sidecar dir;
.civitai.info stays co-located in both modes
- migration: SidecarMigrationUseCase moves sidecars+previews between
layouts both directions (keep-newer conflict resolution, preview_url
rewriting, WebSocket progress), exposed as POST+GET
/api/lm/sidecars/migrate with a mode guard (force=true for the
settings-first flow)
Frontend:
- settings modal: sidecar storage section (mode select + path input with
browse/validation), mode-change confirmation offering immediate
migration (force=true), and a 'Migrate Sidecars Now' action
- i18n keys synced to all locales ([TODO: Translate] placeholders)
Docs: metadata-json-schema.md gains a storage-location section;
AGENTS.md records the sidecar_paths helper convention.
Phase 1 of #1045 (optional centralized sidecar storage): introduce
py/utils/sidecar_paths.py as the single place that resolves .metadata.json
and preview locations, and replace all inline splitext-based derivations
across scanners, services, download manager, and route handlers.
No behavior change: the default 'alongside' storage mode resolves every
path exactly as before. .civitai.info (third-party sidecar) derivation is
intentionally left co-located.
Redefine the empty download filename template from a no-op to a bulk
revert: FilenameTemplateUseCase resolves the target from each model's
recorded original_file_name sidecar entry (skipping models without one),
which resolves follow-ups 1 and 2 with a single coherent semantic shared
by the download and bulk-apply paths.
Also replace the browser-native confirm() with a self-managed
confirmation modal (filenameTemplateConfirmModal) that stacks above the
settings modal, since ModalManager would close the settings modal when
opening a registered one.
Add per-model-type filename templates ({model_name}, {version_name},
{base_model}, {author}, {first_tag}, {hash_short}, {original_name}) so
downloaded files get informative names instead of e.g. V1.safetensors.
Empty template keeps the current filename (opt-in, off by default).
- apply template automatically after downloads; rename conflicts keep
the original name and never fail the download
- record original_file_name in metadata on rename for traceability
- bulk apply via GET|POST /api/lm/{prefix}/apply-filename-template with
WebSocket progress, sharing the auto-organize lock
- settings UI lives in the new Organization tab with validation, live
preview, and per-type 'apply to library' actions
A model file could only ever be linked to huggingface.co: `set_hf_url`
validated the URL with a huggingface-only regex, the agent fetched the card
from a hardcoded HF URL, and the readme processor built every relative image
path off `https://huggingface.co/{repo}/resolve/main`. ModelScope publishes the
same model-card convention (README.md + YAML frontmatter, often carrying
`base_model:` and `trigger_words:`) behind a public, key-less API, so the
enrichment pipeline could already serve it - it was the plumbing that was
HF-shaped, not the idea.
Make the external source a first-class, provider-driven concept:
- New `py/services/model_sources/` registry. A `ModelSource` owns URL
recognition (lenient for stored values, strict for user input), the
canonical page URL, model-card fetching, the asset base URL and the
capability flags. `HuggingFaceSource` is the previous logic relocated;
`ModelScopeSource` reads `/models/{o}/{n}/resolve/{master|main}/README.md`
and falls back to `/api/v1/models/{o}/{n}/repo`. `TensorArtSource` is
link-only on purpose: tensor.art answers plain HTTP clients with a
Cloudflare challenge and its internal API (ap-east-1.tensorart.cloud /
cn.tensorart.net) rejects every /v1/model/* route with "invalid
authorization header", so it declares supports_enrichment=False rather than
failing silently later.
- Metadata gains `source_platform` + `source_url`; `hf_url` stays as a
read/write alias, written only for Hugging Face, so existing sidecars,
cached rows and third-party consumers keep working. Normalisation runs at
the scanner, the persistent cache (both directions, plus two new columns
behind an ALTER migration) and the linking handler - which is what stops a
user who switches sources from leaving a stale `hf_url` on a ModelScope
model.
- The agent pipeline keys off the provider instead of `hf_url`: the fast-fail
gate now explains *why* a model is skipped (no source / unknown source /
source without a reachable card), the prompt context exposes
source_url/source_id/source_label/asset_base_url while still filling the
legacy hf_url/repo aliases, and the four README image extractors take a
base_url (defaulting to HF) so relative paths resolve against the right
site. Version grouping generalises to hf: / ms: / ta: keys.
- `POST /api/lm/set-hf-url` keeps its path and its legacy payload keys but
accepts `source_url`, validates against every provider and returns the
platform. `GET /api/lm/model-sources` lets the UI render the supported-site
list from the server.
- Frontend: a `modelSourceHelpers` mirror of the registry drives the link
dialog, the card/modal globe (branded "View on ModelScope/TensorArt"), the
version-group key and the enrichment gate; the versions tab no longer sends
ms:/ta: keys to the CivitAI API.
TensorArt stays in the list because provenance is worth keeping even when the
card is unreadable - the dialog says so plainly ("Sites that don't expose one
(currently TensorArt) can only be linked") and the context menu disables
enrichment with a matching tooltip, instead of the user getting
"Unsupported URL".
Verified against the real ModelScope API: jj3550945163/Krea-2-LORA returns a
1882-byte card whose frontmatter carries base_model/tags/trigger_words, and
relative images resolve to .../resolve/master/....
Tests: backend 2815 passed; frontend 1130 JS + 91 Vue passed; pytest
tests/i18n and a Jinja compile pass over templates/. The nine locales carry
[TODO: Translate] for the new strings, completed in the next commit.
hydrate_model_data replaces model_data with .metadata.json content which
may lack sha256 (corrupted file, concurrent write, etc.). Restore the
cached sha256 after hydration and persist the fix back to disk so
subsequent lookups don't hit the same error.
Also improve error log to include file_path for debugging.
- Bulk refresh filter now excludes models with hf_url
- Individual refresh for HF models only checks CivitAI API
- CivArchive client validates model IDs before querying
When CivArchive returns HTTP 429 with a large retry_after, the bulk
metadata refresh would block for hours because:
1. FallbackMetadataProvider raised RateLimitError instead of continuing
to the next provider (e.g., SQLite archive was never reached).
2. _RateLimitRetryHelper retried long-rate-limit 429s 3 times — all
futile since the hourly cap hasn't reset.
3. The batch loop had no awareness of persistent rate-limiting,
causing 192+ models to each hammer the same rate-limited endpoint.
Changes:
- FallbackMetadataProvider: all 6 methods now continue to next provider
on RateLimitError instead of raising (model_metadata_provider.py)
- fetch_and_update_model: deleted-model path also continues on
RateLimitError so sqlite provider gets a chance (metadata_sync_service.py)
- _RateLimitRetryHelper: when retry_after >= 120s, only 1 attempt is
made — retries are futile for hour-scale rate limits
- BulkMetadataRefreshUseCase: tracks consecutive rate-limit failures
and aborts early after 3 (bulk_metadata_refresh_use_case.py)
Tests: updated test_fallback_respects_retry_limit for new continue
behavior; added tests for large/small retry_after thresholds.
- Allow empty sha256 when hash_status is 'pending' in cache entry validator
- Add on-demand hash calculation during bulk metadata refresh for checkpoints
with pending hash status
- Add comprehensive tests for both fixes
Fixes issue where checkpoints in extra paths were not visible in UI and
not processed during bulk metadata refresh due to empty sha256.
When enable_metadata_archive_db=True, the previous filter logic would
repeatedly try to fetch metadata for models that were already confirmed
to not exist on CivitAI (from_civitai=False, civitai_deleted=True).
The fix adds a skip condition to exclude models that:
1. Are confirmed not from CivitAI (from_civitai=False)
2. Are marked as deleted/not found on CivitAI (civitai_deleted=True)
3. Either have no archive DB enabled, or have already been checked (db_checked=True)
This prevents unnecessary API calls to CivArchive for user-trained models
or models from non-CivitAI sources.
Fixes repeated "Error fetching version of CivArchive model by hash" logs
for models that will never be found on CivitAI/CivArchive.
Will Miao