mirror of
https://github.com/willmiao/ComfyUI-Lora-Manager.git
synced 2026-10-08 02:22:12 -03:00
feat(scanner): report and parallelize the reconcile walk
A regular Refresh walked every configured root sequentially, so a full 25 TB drive delayed the roots behind it, and the dialog sat on "Checking for changes..." at 0 % for the whole walk with no way to tell it was working. On a cold external drive the walk itself dominates the cost, so the fix is to overlap the drives and to show what the walk is doing. Backend (py/services/model_scanner.py): * Extract the per-root walk into the synchronous _walk_root_for_reconcile() worker and merge its results on the event loop afterwards, in configured root order: which business path wins a file reachable through several roots must not depend on the order the workers happened to finish in. * Group roots by device (_root_device_key: drive letter on Windows, st_dev on POSIX) and run one worker per device. Roots sharing a device stay sequential, so directory claims and the overlap dedup (#871, #1041) keep their configured-order semantics; different devices run in parallel. * Track walk progress per root (_ReconcileWalkTracker), weighted by each root's cached entry count because the real file count is only known once the walk ends. The bar splits into walk (0-50 %) and new-file (50-99 %) phases so it never jumps backwards, and the walk broadcasts files seen, active roots and the ETA counters. * Replace the Windows case-insensitive fallback -- a scan of every cached path per miss, i.e. O(files x cached) -- with a lazily built lower-cased index (_CachedPathLookups, also now guarding the realpath alias index for worker threads), and lift the os.name == "nt" gate into the module-level _CASE_INSENSITIVE_PATHS so the branch is testable off Windows. * Excluded-model membership is a set lookup instead of a list scan. Frontend: * render the walk phase as "Checking for changes... <roots> (N files)" with the ETA, and reset the ETA tracker when the stage changes: the per-file rate of counting files says nothing about processing them. * add common.scanProgress.walkFiles; the other locales keep the sanctioned [TODO: Translate] placeholder until the translation pass. Verified: no-change reconcile over 10k files/500 dirs 101 ms and 10k/5000 dirs 230 ms (was 93/229 ms, within noise); a two-device sandbox walk runs both roots concurrently, names them in the progress messages and finishes with added=15, removed=0; 3665 passed, 7 skipped; frontend 1456 passed, vue widgets 96 passed.
This commit is contained in:
@@ -304,6 +304,94 @@ describe('BaseModelApiClient.refreshModels scan progress', () => {
|
||||
await promise;
|
||||
});
|
||||
|
||||
it('shows the walked roots and file count during the reconcile walk', async () => {
|
||||
const fetchControl = mockFetchPending();
|
||||
const client = await createClient();
|
||||
const { promise, socket } = await startRefresh(client);
|
||||
|
||||
socket.emit({
|
||||
type: 'scan_progress',
|
||||
status: 'processing',
|
||||
stage: 'reconcile_scan',
|
||||
model_type: 'lora',
|
||||
pageType: 'loras',
|
||||
full_rebuild: false,
|
||||
progress: 12,
|
||||
processed: 1234,
|
||||
total: 2000,
|
||||
files_seen: 1234,
|
||||
roots_total: 3,
|
||||
roots_done: 0,
|
||||
active_roots: ['G:', 'Y:'],
|
||||
current_name: 'G:',
|
||||
});
|
||||
|
||||
expect(setProgressMock).toHaveBeenCalledWith(12);
|
||||
const walkStatus = setStatusMock.mock.calls.at(-1)[0];
|
||||
expect(walkStatus).toContain('G:, Y:');
|
||||
expect(walkStatus).toContain('1,234 files');
|
||||
// No processed/total ratio: the real file count is unknown mid-walk.
|
||||
expect(walkStatus).not.toContain('(1234/2000)');
|
||||
expect(walkStatus).toContain('Estimating time...');
|
||||
|
||||
// Walk finished: no ETA once the counters meet the estimate.
|
||||
socket.emit({
|
||||
type: 'scan_progress',
|
||||
status: 'processing',
|
||||
stage: 'reconcile_scan',
|
||||
model_type: 'lora',
|
||||
full_rebuild: false,
|
||||
progress: 50,
|
||||
processed: 2000,
|
||||
total: 2000,
|
||||
files_seen: 2000,
|
||||
active_roots: [],
|
||||
current_name: 'Y:',
|
||||
});
|
||||
|
||||
const finalWalkStatus = setStatusMock.mock.calls.at(-1)[0];
|
||||
expect(finalWalkStatus).toContain('2,000 files');
|
||||
expect(finalWalkStatus).not.toContain('Estimating time...');
|
||||
|
||||
fetchControl.resolveOk();
|
||||
await promise;
|
||||
});
|
||||
|
||||
it('drops the ETA samples when the scan moves to another stage', async () => {
|
||||
const fetchControl = mockFetchPending();
|
||||
let now = 1000;
|
||||
vi.spyOn(Date, 'now').mockImplementation(() => now);
|
||||
|
||||
const client = await createClient();
|
||||
const { promise, socket } = await startRefresh(client);
|
||||
|
||||
const emit = (stage, processed, total, extra = {}) => socket.emit({
|
||||
type: 'scan_progress',
|
||||
status: 'processing',
|
||||
stage,
|
||||
model_type: 'lora',
|
||||
full_rebuild: false,
|
||||
progress: 50,
|
||||
processed,
|
||||
total,
|
||||
...extra,
|
||||
});
|
||||
|
||||
emit('process_models', 1, 10);
|
||||
now = 101000;
|
||||
emit('process_models', 2, 10);
|
||||
expect(setStatusMock.mock.calls.at(-1)[0]).toContain('~7 min remaining');
|
||||
|
||||
// Same counters on the walk stage: without the reset the old per-file rate
|
||||
// (50s/file for 2 files) would be reused and produce a huge ETA.
|
||||
now = 102000;
|
||||
emit('reconcile_scan', 2, 100, { files_seen: 2, active_roots: ['G:'] });
|
||||
expect(setStatusMock.mock.calls.at(-1)[0]).toContain('Estimating time...');
|
||||
|
||||
fetchControl.resolveOk();
|
||||
await promise;
|
||||
});
|
||||
|
||||
it('shows the cancelled toast when the server reports cancellation', async () => {
|
||||
const fetchControl = mockFetchPending();
|
||||
const client = await createClient();
|
||||
|
||||
Reference in New Issue
Block a user