mirror of
https://github.com/willmiao/ComfyUI-Lora-Manager.git
synced 2026-10-08 10:32:13 -03:00
feat(scanner): report and parallelize the reconcile walk
A regular Refresh walked every configured root sequentially, so a full 25 TB drive delayed the roots behind it, and the dialog sat on "Checking for changes..." at 0 % for the whole walk with no way to tell it was working. On a cold external drive the walk itself dominates the cost, so the fix is to overlap the drives and to show what the walk is doing. Backend (py/services/model_scanner.py): * Extract the per-root walk into the synchronous _walk_root_for_reconcile() worker and merge its results on the event loop afterwards, in configured root order: which business path wins a file reachable through several roots must not depend on the order the workers happened to finish in. * Group roots by device (_root_device_key: drive letter on Windows, st_dev on POSIX) and run one worker per device. Roots sharing a device stay sequential, so directory claims and the overlap dedup (#871, #1041) keep their configured-order semantics; different devices run in parallel. * Track walk progress per root (_ReconcileWalkTracker), weighted by each root's cached entry count because the real file count is only known once the walk ends. The bar splits into walk (0-50 %) and new-file (50-99 %) phases so it never jumps backwards, and the walk broadcasts files seen, active roots and the ETA counters. * Replace the Windows case-insensitive fallback -- a scan of every cached path per miss, i.e. O(files x cached) -- with a lazily built lower-cased index (_CachedPathLookups, also now guarding the realpath alias index for worker threads), and lift the os.name == "nt" gate into the module-level _CASE_INSENSITIVE_PATHS so the branch is testable off Windows. * Excluded-model membership is a set lookup instead of a list scan. Frontend: * render the walk phase as "Checking for changes... <roots> (N files)" with the ETA, and reset the ETA tracker when the stage changes: the per-file rate of counting files says nothing about processing them. * add common.scanProgress.walkFiles; the other locales keep the sanctioned [TODO: Translate] placeholder until the translation pass. Verified: no-change reconcile over 10k files/500 dirs 101 ms and 10k/5000 dirs 230 ms (was 93/229 ms, within noise); a two-device sandbox walk runs both roots concurrently, names them in the progress messages and finishes with added=15, removed=0; 3665 passed, 7 skipped; frontend 1456 passed, vue widgets 96 passed.
This commit is contained in:
@@ -532,17 +532,48 @@ export class BaseModelApiClient {
|
||||
);
|
||||
const etaTracker = createScanEtaTracker();
|
||||
let ws = null;
|
||||
let lastProgressStage = null;
|
||||
|
||||
const handleScanProgress = (data) => {
|
||||
if (typeof data.progress === 'number') {
|
||||
state.loadingManager.setProgress(data.progress);
|
||||
}
|
||||
// The per-file rate of one stage says nothing about the next one:
|
||||
// the walk phase counts files while the new-file pass processes
|
||||
// them, so a carried-over average would produce a nonsense ETA.
|
||||
if (data.stage && data.stage !== lastProgressStage) {
|
||||
lastProgressStage = data.stage;
|
||||
etaTracker.reset();
|
||||
}
|
||||
let statusText = translate(
|
||||
`common.scanProgress.stages.${data.stage}`,
|
||||
{ total: data.total },
|
||||
data.stage || ''
|
||||
);
|
||||
if (data.status === 'processing' && data.total > 0) {
|
||||
if (data.status === 'processing' && data.stage === 'reconcile_scan') {
|
||||
// Walk phase: the real file count is only known once the walk
|
||||
// finishes, so report the files checked so far and the roots
|
||||
// being walked instead of a processed/total ratio.
|
||||
const filesSeen = Number(data.files_seen) || 0;
|
||||
if (filesSeen > 0) {
|
||||
const roots = Array.isArray(data.active_roots)
|
||||
? data.active_roots.filter(Boolean)
|
||||
: [];
|
||||
const formattedFiles = filesSeen.toLocaleString();
|
||||
const filesText = translate(
|
||||
'common.scanProgress.walkFiles',
|
||||
{ count: formattedFiles },
|
||||
`${formattedFiles} files`
|
||||
);
|
||||
statusText += roots.length
|
||||
? ` ${roots.join(', ')} (${filesText})`
|
||||
: ` (${filesText})`;
|
||||
const etaText = etaTracker.update(data.processed, data.total);
|
||||
if (etaText) {
|
||||
statusText += ` | ${etaText}`;
|
||||
}
|
||||
}
|
||||
} else if (data.status === 'processing' && data.total > 0) {
|
||||
statusText += ` (${data.processed}/${data.total})`;
|
||||
if (data.current_name) {
|
||||
statusText += ` ${data.current_name}`;
|
||||
|
||||
Reference in New Issue
Block a user