feat(scanner): report and parallelize the reconcile walk

A regular Refresh walked every configured root sequentially, so a full
25 TB drive delayed the roots behind it, and the dialog sat on "Checking
for changes..." at 0 % for the whole walk with no way to tell it was
working. On a cold external drive the walk itself dominates the cost, so
the fix is to overlap the drives and to show what the walk is doing.

Backend (py/services/model_scanner.py):
* Extract the per-root walk into the synchronous _walk_root_for_reconcile()
  worker and merge its results on the event loop afterwards, in configured
  root order: which business path wins a file reachable through several
  roots must not depend on the order the workers happened to finish in.
* Group roots by device (_root_device_key: drive letter on Windows, st_dev
  on POSIX) and run one worker per device. Roots sharing a device stay
  sequential, so directory claims and the overlap dedup (#871, #1041) keep
  their configured-order semantics; different devices run in parallel.
* Track walk progress per root (_ReconcileWalkTracker), weighted by each
  root's cached entry count because the real file count is only known once
  the walk ends. The bar splits into walk (0-50 %) and new-file (50-99 %)
  phases so it never jumps backwards, and the walk broadcasts files seen,
  active roots and the ETA counters.
* Replace the Windows case-insensitive fallback -- a scan of every cached
  path per miss, i.e. O(files x cached) -- with a lazily built lower-cased
  index (_CachedPathLookups, also now guarding the realpath alias index for
  worker threads), and lift the os.name == "nt" gate into the module-level
  _CASE_INSENSITIVE_PATHS so the branch is testable off Windows.
* Excluded-model membership is a set lookup instead of a list scan.

Frontend:
* render the walk phase as "Checking for changes... <roots> (N files)" with
  the ETA, and reset the ETA tracker when the stage changes: the per-file
  rate of counting files says nothing about processing them.
* add common.scanProgress.walkFiles; the other locales keep the sanctioned
  [TODO: Translate] placeholder until the translation pass.

Verified: no-change reconcile over 10k files/500 dirs 101 ms and 10k/5000
dirs 230 ms (was 93/229 ms, within noise); a two-device sandbox walk runs
both roots concurrently, names them in the progress messages and finishes
with added=15, removed=0; 3665 passed, 7 skipped; frontend 1456 passed,
vue widgets 96 passed.
This commit is contained in:
Will Miao
2026-10-06 12:40:57 +08:00
parent 8cd53c20f8
commit 14e660e876
16 changed files with 1059 additions and 187 deletions
+32 -1
View File
@@ -532,17 +532,48 @@ export class BaseModelApiClient {
);
const etaTracker = createScanEtaTracker();
let ws = null;
let lastProgressStage = null;
const handleScanProgress = (data) => {
if (typeof data.progress === 'number') {
state.loadingManager.setProgress(data.progress);
}
// The per-file rate of one stage says nothing about the next one:
// the walk phase counts files while the new-file pass processes
// them, so a carried-over average would produce a nonsense ETA.
if (data.stage && data.stage !== lastProgressStage) {
lastProgressStage = data.stage;
etaTracker.reset();
}
let statusText = translate(
`common.scanProgress.stages.${data.stage}`,
{ total: data.total },
data.stage || ''
);
if (data.status === 'processing' && data.total > 0) {
if (data.status === 'processing' && data.stage === 'reconcile_scan') {
// Walk phase: the real file count is only known once the walk
// finishes, so report the files checked so far and the roots
// being walked instead of a processed/total ratio.
const filesSeen = Number(data.files_seen) || 0;
if (filesSeen > 0) {
const roots = Array.isArray(data.active_roots)
? data.active_roots.filter(Boolean)
: [];
const formattedFiles = filesSeen.toLocaleString();
const filesText = translate(
'common.scanProgress.walkFiles',
{ count: formattedFiles },
`${formattedFiles} files`
);
statusText += roots.length
? ` ${roots.join(', ')} (${filesText})`
: ` (${filesText})`;
const etaText = etaTracker.update(data.processed, data.total);
if (etaText) {
statusText += ` | ${etaText}`;
}
}
} else if (data.status === 'processing' && data.total > 0) {
statusText += ` (${data.processed}/${data.total})`;
if (data.current_name) {
statusText += ` ${data.current_name}`;