mirror of
https://github.com/willmiao/ComfyUI-Lora-Manager.git
synced 2026-10-07 18:12:12 -03:00
feat(scanner): scan a single root and keep unreachable entries (#1108)
Refreshing had no way to say "scan only this drive": a user with three external
drives had to spin all of them up for every refresh, and switching a drive off
made the next refresh treat its whole library as deleted (rows pruned from the
memory cache and the SQLite cache, preview_url stripped on the next scroll).
Backend (py/services/model_scanner.py, py/config.py):
* ReconcileScope(roots, folder) + _reconcile_cache(scope=...): files inside the
scope reconcile normally, everything outside is neither re-read nor removed.
The folder half is plumbing for the sidebar entry in the next change.
* Path-level pruning guard: cached entries under a path this walk could not
read are kept and reported instead of removed. Sources: a configured root that
is not reachable (drive switched off while LM runs), a directory os.walk
failed to enter (permissions / I/O error / Windows junction to an offline
drive), and a known first-level symlink whose target is gone
(Config.iter_path_mappings()).
* The recorded folder list is unioned instead of replaced whenever the scan did
not verify every root, so a scoped scan cannot empty the sidebar.
* _reconcile_cache returns a summary (added / removed / repaired /
scanned_roots / skipped_roots / unavailable_paths / kept_unreachable),
exposed as ModelScanner.last_reconcile_summary, returned by
BaseModelService.scan_models() and broadcast in the completed WS payload.
* _root_display_labels(): set-aware labels ("G: loras", "usb/loras") grown
leftwards with real parent segments until unique, shared by the walk-progress
line and the roots API.
* GET /scan accepts repeated `roots` (400 for unknown roots, 400 combined with
full_rebuild=true); GET /roots gains root_details (label / reachable / cached
count) while `roots` stays a plain path list for existing callers.
* serve_preview: a 404 no longer clears the cached preview_url when the file's
own directory is unreachable - browsing the grid with a drive off used to
strip preview references from the persistent cache.
Frontend:
* Refresh ▾ gains a "Scan one folder" section listing the page's roots with
their cached counts; offline roots stay clickable and explain themselves; rows
are wired by delegation (new static/js/components/controls/ScanScopeMenu.js).
* A scoped scan reports "Scanned <root>: N new, M removed"; a scan that kept
entries reports "<N> models kept: <paths> not reachable".
* registerAPI() now injects the two cross-page passthroughs (fetchModelRoots and
an argument-forwarding refreshModels) so a page facade cannot drop them: the
first version rendered an empty menu and would have run a full refresh.
* createToastElement whitelists toast types, so a wrong `type` argument degrades
to the info style instead of rendering an unstyled box.
Verified in a sandbox instance with three roots: a scoped scan walks only the
requested root (progress roots=0/1, 240 files); a full refresh with one root
offline reports kept_unreachable=60 and leaves all 420 models cached; /roots
reports the offline root with its cached count. 3699 passed, 7 skipped;
frontend 1488 passed (148 files); vue widgets 96 passed.
This commit is contained in:
Will Miao
@@ -153,8 +153,8 @@ describe('BaseModelApiClient.refreshModels scan progress', () => {
|
||||
};
|
||||
}
|
||||
|
||||
async function startRefresh(client, fullRebuild = false) {
|
||||
const promise = client.refreshModels(fullRebuild);
|
||||
async function startRefresh(client, fullRebuild = false, options = {}) {
|
||||
const promise = client.refreshModels(fullRebuild, options);
|
||||
await vi.waitFor(() => {
|
||||
expect(FakeWebSocket.instances.length).toBe(1);
|
||||
});
|
||||
@@ -392,6 +392,75 @@ describe('BaseModelApiClient.refreshModels scan progress', () => {
|
||||
await promise;
|
||||
});
|
||||
|
||||
it('requests a scoped scan when roots are passed', async () => {
|
||||
global.fetch = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({ status: 'success' }),
|
||||
});
|
||||
|
||||
const client = await createClient();
|
||||
await client.refreshModels(false, { roots: ['/mnt/a/loras', '/mnt/b/loras'] });
|
||||
|
||||
const [url] = global.fetch.mock.calls[0];
|
||||
expect(url.searchParams.getAll('roots')).toEqual(['/mnt/a/loras', '/mnt/b/loras']);
|
||||
expect(url.searchParams.get('full_rebuild')).toBe('false');
|
||||
});
|
||||
|
||||
it('reports the scoped scan summary and the entries kept unreachable', async () => {
|
||||
const fetchControl = mockFetchPending();
|
||||
const client = await createClient();
|
||||
const { promise } = await startRefresh(client, false, { roots: ['/mnt/a/loras'] });
|
||||
|
||||
fetchControl.resolveOk({
|
||||
status: 'success',
|
||||
scanned_roots: ['a/loras'],
|
||||
added: 2,
|
||||
removed: 1,
|
||||
kept_unreachable: 5,
|
||||
unavailable_paths: [
|
||||
{ path: '/mnt/g/loras', reason: 'root_unreachable', kept: 5 },
|
||||
],
|
||||
skipped_roots: [{ path: '/mnt/g/loras', label: 'g/loras' }],
|
||||
});
|
||||
await promise;
|
||||
|
||||
expect(showToastMock).toHaveBeenCalledWith(
|
||||
'toast.api.refreshCompleteScoped',
|
||||
{ scope: 'a/loras', added: 2, removed: 1 },
|
||||
'success'
|
||||
);
|
||||
expect(showToastMock).toHaveBeenCalledWith(
|
||||
'toast.api.refreshKeptUnreachable',
|
||||
{ count: '5', paths: 'g/loras' },
|
||||
'info'
|
||||
);
|
||||
});
|
||||
|
||||
it('keeps the generic completion toast for a full-library scan', async () => {
|
||||
const fetchControl = mockFetchPending();
|
||||
const client = await createClient();
|
||||
const { promise } = await startRefresh(client);
|
||||
|
||||
fetchControl.resolveOk({
|
||||
status: 'success',
|
||||
scanned_roots: ['a/loras', 'b/loras'],
|
||||
added: 0,
|
||||
removed: 0,
|
||||
});
|
||||
await promise;
|
||||
|
||||
expect(showToastMock).toHaveBeenCalledWith(
|
||||
'toast.api.refreshComplete',
|
||||
{ action: 'Refresh' },
|
||||
'success'
|
||||
);
|
||||
expect(showToastMock).not.toHaveBeenCalledWith(
|
||||
'toast.api.refreshCompleteScoped',
|
||||
expect.anything(),
|
||||
expect.anything()
|
||||
);
|
||||
});
|
||||
|
||||
it('shows the cancelled toast when the server reports cancellation', async () => {
|
||||
const fetchControl = mockFetchPending();
|
||||
const client = await createClient();
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||
import { readFileSync } from 'fs';
|
||||
import path from 'path';
|
||||
|
||||
const {
|
||||
PAGE_CONTROLS_MODULE,
|
||||
STATE_MODULE,
|
||||
STORAGE_MODULE,
|
||||
UI_HELPERS_MODULE,
|
||||
EVENT_MANAGER_MODULE,
|
||||
UPDATE_CHECK_MODULE,
|
||||
SIDEBAR_MODULE,
|
||||
SORT_DROPDOWN_MODULE,
|
||||
API_FACTORY_MODULE,
|
||||
I18N_MODULE,
|
||||
} = vi.hoisted(() => ({
|
||||
PAGE_CONTROLS_MODULE: new URL('../../../../static/js/components/controls/PageControls.js', import.meta.url).pathname,
|
||||
STATE_MODULE: new URL('../../../../static/js/state/index.js', import.meta.url).pathname,
|
||||
STORAGE_MODULE: new URL('../../../../static/js/utils/storageHelpers.js', import.meta.url).pathname,
|
||||
UI_HELPERS_MODULE: new URL('../../../../static/js/utils/uiHelpers.js', import.meta.url).pathname,
|
||||
EVENT_MANAGER_MODULE: new URL('../../../../static/js/utils/EventManager.js', import.meta.url).pathname,
|
||||
UPDATE_CHECK_MODULE: new URL('../../../../static/js/utils/updateCheckHelpers.js', import.meta.url).pathname,
|
||||
SIDEBAR_MODULE: new URL('../../../../static/js/components/SidebarManager.js', import.meta.url).pathname,
|
||||
SORT_DROPDOWN_MODULE: new URL('../../../../static/js/components/controls/SortDropdown.js', import.meta.url).pathname,
|
||||
API_FACTORY_MODULE: new URL('../../../../static/js/api/modelApiFactory.js', import.meta.url).pathname,
|
||||
I18N_MODULE: new URL('../../../../static/js/utils/i18nHelpers.js', import.meta.url).pathname,
|
||||
}));
|
||||
|
||||
const CONTROLS_TEMPLATE = path.resolve(
|
||||
__dirname,
|
||||
'../../../../templates/components/controls.html'
|
||||
);
|
||||
|
||||
const showToastMock = vi.fn();
|
||||
const modelClient = {
|
||||
fetchModelRoots: vi.fn(),
|
||||
refreshModels: vi.fn(),
|
||||
};
|
||||
|
||||
vi.mock(STATE_MODULE, () => ({
|
||||
state: {},
|
||||
getCurrentPageState: vi.fn(() => ({ filters: {} })),
|
||||
setCurrentPageType: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock(STORAGE_MODULE, () => ({
|
||||
getStorageItem: vi.fn(),
|
||||
setStorageItem: vi.fn(),
|
||||
removeStorageItem: vi.fn(),
|
||||
getSessionItem: vi.fn(),
|
||||
setSessionItem: vi.fn(),
|
||||
removeSessionItem: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock(UI_HELPERS_MODULE, () => ({
|
||||
showToast: showToastMock,
|
||||
openCivitaiByMetadata: vi.fn(),
|
||||
isTypingContext: vi.fn(() => false),
|
||||
}));
|
||||
|
||||
vi.mock(EVENT_MANAGER_MODULE, () => ({
|
||||
eventManager: { on: vi.fn(), off: vi.fn(), emit: vi.fn() },
|
||||
}));
|
||||
|
||||
vi.mock(UPDATE_CHECK_MODULE, () => ({
|
||||
performModelUpdateCheck: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock(SIDEBAR_MODULE, () => ({
|
||||
sidebarManager: {
|
||||
setHostPageControls: vi.fn(),
|
||||
initialize: vi.fn(async () => {}),
|
||||
refresh: vi.fn(async () => {}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock(SORT_DROPDOWN_MODULE, () => ({
|
||||
initSortDropdown: vi.fn(),
|
||||
applySortToSelect: vi.fn(),
|
||||
randomizeSortValue: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock(API_FACTORY_MODULE, () => ({
|
||||
getModelApiClient: () => modelClient,
|
||||
}));
|
||||
|
||||
vi.mock(I18N_MODULE, () => ({
|
||||
translate: vi.fn((key, params, fallback) => {
|
||||
if (!fallback) {
|
||||
return key;
|
||||
}
|
||||
return Object.entries(params || {}).reduce(
|
||||
(text, [name, value]) => text.replaceAll(`{${name}}`, value),
|
||||
fallback
|
||||
);
|
||||
}),
|
||||
}));
|
||||
|
||||
const { PageControls } = await import(PAGE_CONTROLS_MODULE);
|
||||
|
||||
const ROOT_DETAILS = [
|
||||
{ path: '/mnt/a/loras', label: 'a/loras', reachable: true, models: 12 },
|
||||
{ path: '/mnt/b/loras', label: 'b/loras', reachable: false, models: 3 },
|
||||
];
|
||||
|
||||
function buildControlsDom() {
|
||||
document.body.innerHTML = `
|
||||
<div class="dropdown-group">
|
||||
<button data-action="refresh" class="dropdown-main"></button>
|
||||
<button class="dropdown-toggle"></button>
|
||||
<div class="dropdown-menu">
|
||||
<div class="dropdown-item" data-action="full-rebuild"></div>
|
||||
<div class="dropdown-separator"></div>
|
||||
<div class="dropdown-section-title">Scan one folder</div>
|
||||
<div id="refreshScopeMenu" class="dropdown-scope-list"></div>
|
||||
</div>
|
||||
</div>`;
|
||||
}
|
||||
|
||||
function bareControls(api = {}) {
|
||||
const controls = Object.create(PageControls.prototype);
|
||||
controls.pageType = 'loras';
|
||||
controls.api = api;
|
||||
controls._scanScopeDetails = null;
|
||||
controls._scanScopeLoading = false;
|
||||
controls.refreshModels = vi.fn();
|
||||
controls.sidebarManager = {
|
||||
setHostPageControls: vi.fn(),
|
||||
initialize: vi.fn(async () => {}),
|
||||
};
|
||||
return controls;
|
||||
}
|
||||
|
||||
describe('PageControls refresh scope menu', () => {
|
||||
beforeEach(() => {
|
||||
showToastMock.mockReset();
|
||||
modelClient.fetchModelRoots.mockReset();
|
||||
modelClient.refreshModels.mockReset();
|
||||
modelClient.fetchModelRoots.mockResolvedValue({ root_details: ROOT_DETAILS });
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
document.body.innerHTML = '';
|
||||
});
|
||||
|
||||
it('keeps the scope container in the controls template (non-recipes pages)', () => {
|
||||
const html = readFileSync(CONTROLS_TEMPLATE, 'utf-8');
|
||||
const rebuildIndex = html.indexOf('data-action="full-rebuild"');
|
||||
const scopeIndex = html.indexOf('id="refreshScopeMenu"');
|
||||
|
||||
expect(rebuildIndex).toBeGreaterThan(-1);
|
||||
// Same dropdown, after the rebuild entry.
|
||||
expect(scopeIndex).toBeGreaterThan(rebuildIndex);
|
||||
expect(html).toContain('loras.controls.refresh.scopeSection');
|
||||
// The recipes page has no model roots, so the section is gated.
|
||||
expect(html.slice(rebuildIndex, scopeIndex)).toContain("{% if page_id != 'recipes' %}");
|
||||
});
|
||||
|
||||
it('registers the scan-scope API for every page through registerAPI', async () => {
|
||||
const controls = bareControls();
|
||||
controls.registerAPI({});
|
||||
|
||||
expect(typeof controls.api.fetchModelRoots).toBe('function');
|
||||
expect(typeof controls.api.refreshModels).toBe('function');
|
||||
|
||||
await controls.api.fetchModelRoots();
|
||||
expect(modelClient.fetchModelRoots).toHaveBeenCalled();
|
||||
|
||||
// The page facades used to drop the second argument, which silently turned
|
||||
// a scoped scan into a full refresh.
|
||||
await controls.api.refreshModels(false, { roots: ['/mnt/a/loras'] });
|
||||
expect(modelClient.refreshModels).toHaveBeenCalledWith(false, { roots: ['/mnt/a/loras'] });
|
||||
});
|
||||
|
||||
it('renders the root rows when the dropdown is opened', async () => {
|
||||
buildControlsDom();
|
||||
const controls = bareControls({
|
||||
fetchModelRoots: vi.fn(async () => ({ root_details: ROOT_DETAILS })),
|
||||
refreshModels: vi.fn(),
|
||||
});
|
||||
|
||||
controls.initDropdowns();
|
||||
document.querySelector('.dropdown-toggle').click();
|
||||
|
||||
await vi.waitFor(() => {
|
||||
expect(document.querySelectorAll('.scan-root-item').length).toBe(2);
|
||||
});
|
||||
expect(document.querySelector('.scan-root-item').dataset.root).toBe('/mnt/a/loras');
|
||||
expect(document.querySelector('.scan-root-count').textContent).toBe('12 models');
|
||||
expect(document.querySelectorAll('.scan-root-item')[1].classList.contains('is-offline')).toBe(true);
|
||||
});
|
||||
|
||||
it('scans the clicked root only', async () => {
|
||||
buildControlsDom();
|
||||
const controls = bareControls({
|
||||
fetchModelRoots: vi.fn(async () => ({ root_details: ROOT_DETAILS })),
|
||||
refreshModels: vi.fn(),
|
||||
});
|
||||
|
||||
controls.initDropdowns();
|
||||
document.querySelector('.dropdown-toggle').click();
|
||||
await vi.waitFor(() => {
|
||||
expect(document.querySelectorAll('.scan-root-item').length).toBe(2);
|
||||
});
|
||||
|
||||
document.querySelectorAll('.scan-root-item')[0].click();
|
||||
|
||||
expect(controls.refreshModels).toHaveBeenCalledWith(false, { roots: ['/mnt/a/loras'] });
|
||||
expect(showToastMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('explains an offline root instead of scanning it', async () => {
|
||||
buildControlsDom();
|
||||
const controls = bareControls({
|
||||
fetchModelRoots: vi.fn(async () => ({ root_details: ROOT_DETAILS })),
|
||||
refreshModels: vi.fn(),
|
||||
});
|
||||
|
||||
controls.initDropdowns();
|
||||
document.querySelector('.dropdown-toggle').click();
|
||||
await vi.waitFor(() => {
|
||||
expect(document.querySelectorAll('.scan-root-item').length).toBe(2);
|
||||
});
|
||||
|
||||
document.querySelectorAll('.scan-root-item')[1].click();
|
||||
|
||||
expect(controls.refreshModels).not.toHaveBeenCalled();
|
||||
// Exactly three arguments: key, params, type. A 4th "fallback" argument
|
||||
// pushed the sentence into the type slot and rendered an unstyled toast.
|
||||
expect(showToastMock).toHaveBeenCalledWith(
|
||||
'toast.api.scanRootUnreachable',
|
||||
{ scope: 'b/loras' },
|
||||
'info'
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,77 @@
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest';
|
||||
|
||||
const { I18N_MODULE, MODULE } = vi.hoisted(() => ({
|
||||
I18N_MODULE: new URL('../../../../static/js/utils/i18nHelpers.js', import.meta.url).pathname,
|
||||
MODULE: new URL('../../../../static/js/components/controls/ScanScopeMenu.js', import.meta.url).pathname,
|
||||
}));
|
||||
|
||||
vi.mock(I18N_MODULE, () => ({
|
||||
translate: vi.fn((key, params, fallback) => {
|
||||
if (!fallback) {
|
||||
return key;
|
||||
}
|
||||
return Object.entries(params || {}).reduce(
|
||||
(text, [name, value]) => text.replaceAll(`{${name}}`, value),
|
||||
fallback
|
||||
);
|
||||
}),
|
||||
}));
|
||||
|
||||
const { renderScanScopeMenu, resolveScanScopeTarget } = await import(MODULE);
|
||||
|
||||
describe('ScanScopeMenu', () => {
|
||||
let menu;
|
||||
|
||||
beforeEach(() => {
|
||||
document.body.innerHTML = '<div id="refreshScopeMenu"></div>';
|
||||
menu = document.getElementById('refreshScopeMenu');
|
||||
});
|
||||
|
||||
it('renders one row per root with label, count and offline state', () => {
|
||||
renderScanScopeMenu(menu, [
|
||||
{ path: '/mnt/a/loras', label: 'a/loras', reachable: true, models: 7 },
|
||||
{ path: '/mnt/b/loras', label: 'b/loras', reachable: false, models: 0 },
|
||||
]);
|
||||
|
||||
const rows = menu.querySelectorAll('.scan-root-item');
|
||||
expect(rows.length).toBe(2);
|
||||
|
||||
expect(rows[0].dataset.action).toBe('scan-root');
|
||||
expect(rows[0].dataset.root).toBe('/mnt/a/loras');
|
||||
expect(rows[0].title).toBe('/mnt/a/loras');
|
||||
expect(rows[0].querySelector('.scan-root-label').textContent).toBe('a/loras');
|
||||
expect(rows[0].querySelector('.scan-root-count').textContent).toBe('7 models');
|
||||
expect(rows[0].classList.contains('is-offline')).toBe(false);
|
||||
|
||||
expect(rows[1].classList.contains('is-offline')).toBe(true);
|
||||
expect(rows[1].querySelector('.scan-root-offline').textContent).toBe('Offline');
|
||||
// Offline rows stay clickable so the click can explain the state.
|
||||
expect(rows[1].dataset.action).toBe('scan-root');
|
||||
});
|
||||
|
||||
it('resolveScanScopeTarget reads path, label and offline state', () => {
|
||||
renderScanScopeMenu(menu, [
|
||||
{ path: '/mnt/b/loras', label: 'b/loras', reachable: false, models: 0 },
|
||||
]);
|
||||
|
||||
expect(resolveScanScopeTarget(menu.querySelector('.scan-root-item'))).toEqual({
|
||||
rootPath: '/mnt/b/loras',
|
||||
label: 'b/loras',
|
||||
offline: true,
|
||||
});
|
||||
});
|
||||
|
||||
it('skips malformed entries and replaces the previous list', () => {
|
||||
renderScanScopeMenu(menu, [{ path: '/mnt/a' }, null, {}]);
|
||||
expect(menu.querySelectorAll('.scan-root-item').length).toBe(1);
|
||||
|
||||
renderScanScopeMenu(menu, []);
|
||||
expect(menu.querySelectorAll('.scan-root-item').length).toBe(0);
|
||||
});
|
||||
|
||||
it('falls back to the path when a root has no label', () => {
|
||||
renderScanScopeMenu(menu, [{ path: '/mnt/a/loras', reachable: true, models: 0 }]);
|
||||
|
||||
expect(menu.querySelector('.scan-root-label').textContent).toBe('/mnt/a/loras');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,36 @@
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest';
|
||||
|
||||
const { UI_HELPERS_MODULE, I18N_MODULE } = vi.hoisted(() => ({
|
||||
UI_HELPERS_MODULE: new URL('../../../static/js/utils/uiHelpers.js', import.meta.url).pathname,
|
||||
I18N_MODULE: new URL('../../../static/js/utils/i18nHelpers.js', import.meta.url).pathname,
|
||||
}));
|
||||
|
||||
vi.mock(I18N_MODULE, () => ({
|
||||
translate: vi.fn((key, params, fallback) => fallback || key),
|
||||
}));
|
||||
|
||||
const { showToast } = await import(UI_HELPERS_MODULE);
|
||||
|
||||
describe('toast type styling', () => {
|
||||
beforeEach(() => {
|
||||
document.body.innerHTML = '';
|
||||
});
|
||||
|
||||
it('applies the requested type class', () => {
|
||||
showToast('plain message', {}, 'success');
|
||||
expect(document.querySelector('.toast').className).toBe('toast toast-success');
|
||||
});
|
||||
|
||||
it('coerces an unknown type to info instead of rendering an unstyled toast', () => {
|
||||
// Regression guard: a call site that passed its fallback sentence as the
|
||||
// `type` produced `toast toast-drive-Z is not reachable ...`, which matched
|
||||
// no rule at all (no icon, no accent border).
|
||||
showToast('plain message', {}, 'drive-Z is not reachable right now');
|
||||
expect(document.querySelector('.toast').className).toBe('toast toast-info');
|
||||
});
|
||||
|
||||
it('defaults to info when no type is given', () => {
|
||||
showToast('plain message');
|
||||
expect(document.querySelector('.toast').className).toBe('toast toast-info');
|
||||
});
|
||||
});
|
||||
@@ -196,3 +196,126 @@ async def test_get_folder_tree_threads_include_empty():
|
||||
|
||||
assert payload["success"] is True
|
||||
assert service.received_include_empty is True
|
||||
|
||||
|
||||
class QueryParams(dict):
|
||||
"""Minimal stand-in for aiohttp's MultiDict query (supports getall)."""
|
||||
|
||||
def getall(self, key, default=None):
|
||||
value = self.get(key)
|
||||
if value is None:
|
||||
return list(default) if default else []
|
||||
return value if isinstance(value, list) else [value]
|
||||
|
||||
|
||||
class ScopedScanService:
|
||||
"""Stub service recording the scan scope the handler resolves."""
|
||||
|
||||
model_type = "lora"
|
||||
|
||||
def __init__(self, roots=None, summary=None, cancelled=False):
|
||||
self._roots = list(roots or [])
|
||||
self._summary = summary
|
||||
self.cancelled = cancelled
|
||||
self.scanner = SimpleNamespace(is_cancelled=lambda: self.cancelled)
|
||||
self.received_scope = "not-called"
|
||||
self.received_rebuild = None
|
||||
|
||||
def get_model_roots(self):
|
||||
return list(self._roots)
|
||||
|
||||
def describe_model_roots(self):
|
||||
return [
|
||||
{"path": root, "label": root.rsplit("/", 1)[-1], "reachable": True, "models": 7}
|
||||
for root in self._roots
|
||||
]
|
||||
|
||||
async def scan_models(self, force_refresh=False, rebuild_cache=False, scope=None):
|
||||
self.received_scope = scope
|
||||
self.received_rebuild = rebuild_cache
|
||||
return self._summary
|
||||
|
||||
|
||||
SUMMARY = {
|
||||
"added": 3,
|
||||
"removed": 0,
|
||||
"repaired": 1,
|
||||
"scanned_roots": ["a"],
|
||||
"skipped_roots": [],
|
||||
"unavailable_paths": [],
|
||||
"unavailable_paths_total": 0,
|
||||
"kept_unreachable": 0,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_scan_models_accepts_roots_param():
|
||||
service = ScopedScanService(roots=["/mnt/a", "/mnt/b"], summary=SUMMARY)
|
||||
handler = ModelQueryHandler(service=service, logger=logging.getLogger(__name__))
|
||||
|
||||
response = await handler.scan_models(
|
||||
SimpleNamespace(query=QueryParams({"roots": ["/mnt/a"]}))
|
||||
)
|
||||
payload = json.loads(response.text)
|
||||
|
||||
assert response.status == 200
|
||||
assert payload["status"] == "success"
|
||||
assert payload["added"] == 3
|
||||
assert payload["scanned_roots"] == ["a"]
|
||||
assert service.received_scope is not None
|
||||
assert service.received_scope.roots == ("/mnt/a",)
|
||||
assert service.received_rebuild is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_scan_models_without_roots_scans_every_root():
|
||||
service = ScopedScanService(roots=["/mnt/a"], summary=SUMMARY)
|
||||
handler = ModelQueryHandler(service=service, logger=logging.getLogger(__name__))
|
||||
|
||||
await handler.scan_models(SimpleNamespace(query=QueryParams({})))
|
||||
|
||||
assert service.received_scope is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_scan_models_rejects_unknown_root():
|
||||
service = ScopedScanService(roots=["/mnt/a"], summary=SUMMARY)
|
||||
handler = ModelQueryHandler(service=service, logger=logging.getLogger(__name__))
|
||||
|
||||
response = await handler.scan_models(
|
||||
SimpleNamespace(query=QueryParams({"roots": ["/mnt/nope"]}))
|
||||
)
|
||||
payload = json.loads(response.text)
|
||||
|
||||
assert response.status == 400
|
||||
assert payload["roots"] == ["/mnt/nope"]
|
||||
assert service.received_scope == "not-called"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_scan_models_rejects_roots_with_full_rebuild():
|
||||
service = ScopedScanService(roots=["/mnt/a"], summary=SUMMARY)
|
||||
handler = ModelQueryHandler(service=service, logger=logging.getLogger(__name__))
|
||||
|
||||
response = await handler.scan_models(
|
||||
SimpleNamespace(
|
||||
query=QueryParams({"roots": ["/mnt/a"], "full_rebuild": "true"})
|
||||
)
|
||||
)
|
||||
|
||||
assert response.status == 400
|
||||
assert service.received_scope == "not-called"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_model_roots_reports_details():
|
||||
service = ScopedScanService(roots=["/mnt/a", "/mnt/b"])
|
||||
handler = ModelQueryHandler(service=service, logger=logging.getLogger(__name__))
|
||||
|
||||
response = await handler.get_model_roots(SimpleNamespace(query=QueryParams({})))
|
||||
payload = json.loads(response.text)
|
||||
|
||||
# `roots` stays a plain list of paths for the existing callers.
|
||||
assert payload["roots"] == ["/mnt/a", "/mnt/b"]
|
||||
assert [detail["label"] for detail in payload["root_details"]] == ["a", "b"]
|
||||
assert all(detail["models"] == 7 for detail in payload["root_details"])
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import asyncio
|
||||
import os
|
||||
import urllib.parse
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
import pytest
|
||||
from aiohttp import web
|
||||
@@ -348,3 +349,67 @@ async def test_deep_symlink_discovered_on_first_access(tmp_path):
|
||||
|
||||
# Deep symlink should now be in mappings
|
||||
assert normalized_external in config._path_mappings
|
||||
|
||||
|
||||
async def test_preview_404_keeps_cache_when_parent_dir_missing(tmp_path):
|
||||
"""A preview whose directory is unreachable (drive off) must not clear the
|
||||
cached preview_url: nothing was deleted, the file just cannot be read."""
|
||||
library_root = tmp_path / "library"
|
||||
library_root.mkdir()
|
||||
unreachable_parent = library_root / "gone"
|
||||
preview_path = unreachable_parent / "model.webp"
|
||||
|
||||
config = Config()
|
||||
config.apply_library_settings(
|
||||
{
|
||||
"folder_paths": {
|
||||
"loras": [str(library_root)],
|
||||
"checkpoints": [],
|
||||
"unet": [],
|
||||
"embeddings": [],
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
handler = PreviewHandler(config=config)
|
||||
encoded_path = urllib.parse.quote(str(preview_path), safe="")
|
||||
request = make_mocked_request("GET", f"/api/lm/previews?path={encoded_path}")
|
||||
|
||||
with patch.object(handler, "_cleanup_stale_preview_url", new=AsyncMock()) as cleanup:
|
||||
with pytest.raises(web.HTTPNotFound):
|
||||
await handler.serve_preview(request)
|
||||
await asyncio.sleep(0)
|
||||
|
||||
cleanup.assert_not_called()
|
||||
|
||||
|
||||
async def test_preview_404_clears_cache_when_file_really_deleted(tmp_path):
|
||||
"""A preview deleted from a reachable directory is genuinely stale."""
|
||||
library_root = tmp_path / "library"
|
||||
library_root.mkdir()
|
||||
preview_path = library_root / "model.webp"
|
||||
preview_path.write_bytes(b"preview")
|
||||
preview_path.unlink()
|
||||
|
||||
config = Config()
|
||||
config.apply_library_settings(
|
||||
{
|
||||
"folder_paths": {
|
||||
"loras": [str(library_root)],
|
||||
"checkpoints": [],
|
||||
"unet": [],
|
||||
"embeddings": [],
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
handler = PreviewHandler(config=config)
|
||||
encoded_path = urllib.parse.quote(str(preview_path), safe="")
|
||||
request = make_mocked_request("GET", f"/api/lm/previews?path={encoded_path}")
|
||||
|
||||
with patch.object(handler, "_cleanup_stale_preview_url", new=AsyncMock()) as cleanup:
|
||||
with pytest.raises(web.HTTPNotFound):
|
||||
await handler.serve_preview(request)
|
||||
await asyncio.sleep(0)
|
||||
|
||||
cleanup.assert_awaited_once()
|
||||
|
||||
@@ -20,9 +20,11 @@ from py.services.model_hash_index import ModelHashIndex
|
||||
from py.services.model_scanner import (
|
||||
CacheBuildResult,
|
||||
ModelScanner,
|
||||
ReconcileScope,
|
||||
_ReconcileWalkTracker,
|
||||
_count_cached_entries_per_root,
|
||||
_group_roots_by_device,
|
||||
_root_display_labels,
|
||||
)
|
||||
from py.services.pending_delete_service import (
|
||||
PENDING_DELETE_DIR_NAME,
|
||||
@@ -2479,10 +2481,9 @@ async def test_reconcile_walk_broadcasts_per_root_progress(tmp_path: Path, monke
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reconcile_prunes_entries_of_an_offline_root(tmp_path: Path):
|
||||
"""A root that is gone (drive switched off) is skipped by the walk, so its
|
||||
cached entries are reported as missing. Locked in because it is the
|
||||
documented consequence of refreshing with a drive powered down."""
|
||||
async def test_reconcile_keeps_entries_of_an_offline_root(tmp_path: Path):
|
||||
"""A configured root that is gone (drive switched off) must not be treated
|
||||
as deleted: its entries are kept and reported as unreachable instead."""
|
||||
root_a = tmp_path / "a"
|
||||
root_b = tmp_path / "b"
|
||||
root_a.mkdir()
|
||||
@@ -2498,9 +2499,217 @@ async def test_reconcile_prunes_entries_of_an_offline_root(tmp_path: Path):
|
||||
|
||||
root_b.rename(tmp_path / "b_offline")
|
||||
|
||||
await scanner._reconcile_cache()
|
||||
summary = await scanner._reconcile_cache()
|
||||
|
||||
remaining = {item["file_path"] for item in scanner._cache.raw_data}
|
||||
assert remaining == {_normalize_path(file_a)}
|
||||
assert scanner._hash_index.get_path("hash-b0") is None
|
||||
assert remaining == {_normalize_path(file_a), _normalize_path(file_b)}
|
||||
assert scanner._hash_index.get_path("hash-b0") == _normalize_path(file_b)
|
||||
assert scanner._hash_index.get_path("hash-a0") == _normalize_path(file_a)
|
||||
|
||||
assert summary is not None
|
||||
assert summary["removed"] == 0
|
||||
assert summary["kept_unreachable"] == 1
|
||||
assert summary["unavailable_paths_total"] == 1
|
||||
assert [entry["path"] for entry in summary["unavailable_paths"]] == [
|
||||
_normalize_path(root_b)
|
||||
]
|
||||
assert summary["unavailable_paths"][0]["reason"] == "root_unreachable"
|
||||
assert [entry["path"] for entry in summary["skipped_roots"]] == [
|
||||
_normalize_path(root_b)
|
||||
]
|
||||
|
||||
|
||||
# --- scoped scans (issue #1108) --------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reconcile_scoped_scan_leaves_other_roots_untouched(tmp_path: Path):
|
||||
"""Scanning one root must not add, remove or re-read anything in another."""
|
||||
root_a = tmp_path / "a"
|
||||
root_b = tmp_path / "b"
|
||||
root_a.mkdir()
|
||||
root_b.mkdir()
|
||||
(root_a / "a0.txt").write_text("a", encoding="utf-8")
|
||||
(root_b / "b0.txt").write_text("b", encoding="utf-8")
|
||||
|
||||
scanner = MultiRootDummyScanner([root_a, root_b])
|
||||
await scanner._initialize_cache()
|
||||
|
||||
# Changes in B stay invisible while only A is scanned.
|
||||
(root_b / "b0.txt").unlink()
|
||||
(root_b / "b_new.txt").write_text("new", encoding="utf-8")
|
||||
new_a = root_a / "a_new.txt"
|
||||
new_a.write_text("new", encoding="utf-8")
|
||||
|
||||
summary = await scanner._reconcile_cache(scope=ReconcileScope(roots=(str(root_a),)))
|
||||
|
||||
assert summary is not None
|
||||
assert summary["added"] == 1
|
||||
assert summary["removed"] == 0
|
||||
assert summary["scanned_roots"] == ["a"]
|
||||
|
||||
cached_paths = {item["file_path"] for item in scanner._cache.raw_data}
|
||||
assert cached_paths == {
|
||||
_normalize_path(root_a / "a0.txt"),
|
||||
_normalize_path(new_a),
|
||||
_normalize_path(root_b / "b0.txt"),
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reconcile_scoped_scan_removes_deleted_files_in_scope(tmp_path: Path):
|
||||
"""Deletions inside the scanned root still reconcile normally."""
|
||||
root_a = tmp_path / "a"
|
||||
root_b = tmp_path / "b"
|
||||
root_a.mkdir()
|
||||
root_b.mkdir()
|
||||
deleted_a = root_a / "a0.txt"
|
||||
deleted_a.write_text("a", encoding="utf-8")
|
||||
(root_b / "b0.txt").write_text("b", encoding="utf-8")
|
||||
|
||||
scanner = MultiRootDummyScanner([root_a, root_b])
|
||||
await scanner._initialize_cache()
|
||||
|
||||
deleted_a.unlink()
|
||||
|
||||
summary = await scanner._reconcile_cache(scope=ReconcileScope(roots=(str(root_a),)))
|
||||
|
||||
assert summary is not None
|
||||
assert summary["removed"] == 1
|
||||
assert scanner._hash_index.get_path("hash-a0") is None
|
||||
assert {item["file_path"] for item in scanner._cache.raw_data} == {
|
||||
_normalize_path(root_b / "b0.txt")
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reconcile_folder_scope_keeps_folder_tree_outside_scope(tmp_path: Path):
|
||||
"""A folder-scoped scan must not collapse the recorded folder list to the
|
||||
scanned subtree (the sidebar would lose every other folder)."""
|
||||
root = tmp_path / "loras"
|
||||
(root / "alpha").mkdir(parents=True)
|
||||
(root / "beta").mkdir()
|
||||
(root / "alpha" / "m.txt").write_text("m", encoding="utf-8")
|
||||
|
||||
scanner = DummyScanner(root)
|
||||
await scanner._initialize_cache()
|
||||
folders_before = set(scanner._cache.all_folders or [])
|
||||
assert "beta" in folders_before
|
||||
|
||||
(root / "alpha" / "nested").mkdir()
|
||||
(root / "alpha" / "m.txt").unlink()
|
||||
|
||||
summary = await scanner._reconcile_cache(scope=ReconcileScope(folder="alpha"))
|
||||
|
||||
assert summary is not None
|
||||
assert summary["removed"] == 1
|
||||
folders_after = set(scanner._cache.all_folders or [])
|
||||
assert "beta" in folders_after
|
||||
assert "alpha/nested" in folders_after
|
||||
assert {item["file_path"] for item in scanner._cache.raw_data} == set()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reconcile_keeps_entries_under_unreadable_dir(tmp_path: Path, monkeypatch):
|
||||
"""A directory os.walk cannot enter (permissions, I/O error, offline
|
||||
junction) keeps its cached entries instead of losing them."""
|
||||
root = tmp_path / "loras"
|
||||
sub = root / "sub"
|
||||
sub.mkdir(parents=True)
|
||||
(sub / "m.txt").write_text("m", encoding="utf-8")
|
||||
(root / "top.txt").write_text("t", encoding="utf-8")
|
||||
|
||||
scanner = DummyScanner(root)
|
||||
await scanner._initialize_cache()
|
||||
|
||||
real_scandir = os.scandir
|
||||
|
||||
def _failing_scandir(path=".", *args, **kwargs):
|
||||
if os.path.normpath(str(path)) == os.path.normpath(str(sub)):
|
||||
raise PermissionError(13, "Permission denied", str(sub))
|
||||
return real_scandir(path, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(model_scanner.os, "scandir", _failing_scandir)
|
||||
|
||||
summary = await scanner._reconcile_cache()
|
||||
|
||||
assert summary is not None
|
||||
assert summary["removed"] == 0
|
||||
assert summary["kept_unreachable"] == 1
|
||||
assert summary["unavailable_paths"][0]["reason"] == "unreadable_dir"
|
||||
assert _normalize_path(sub / "m.txt") in {
|
||||
item["file_path"] for item in scanner._cache.raw_data
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reconcile_keeps_entries_under_offline_first_level_symlink(
|
||||
tmp_path: Path, monkeypatch
|
||||
):
|
||||
"""A known first-level symlink whose target is gone protects the entries
|
||||
that live under the link path (the linked drive is switched off)."""
|
||||
root = tmp_path / "loras"
|
||||
linked = root / "linked"
|
||||
linked.mkdir(parents=True)
|
||||
(linked / "m.txt").write_text("m", encoding="utf-8")
|
||||
|
||||
scanner = DummyScanner(root)
|
||||
await scanner._initialize_cache()
|
||||
|
||||
# The linked folder goes away with its drive; config still knows the link.
|
||||
linked.rename(tmp_path / "linked_offline")
|
||||
monkeypatch.setattr(
|
||||
model_scanner.config,
|
||||
"iter_path_mappings",
|
||||
lambda: [(str(tmp_path / "gone_target"), str(linked))],
|
||||
)
|
||||
|
||||
summary = await scanner._reconcile_cache()
|
||||
|
||||
assert summary is not None
|
||||
assert summary["removed"] == 0
|
||||
assert summary["kept_unreachable"] == 1
|
||||
assert summary["unavailable_paths"][0]["reason"] == "symlink_target_offline"
|
||||
assert _normalize_path(linked / "m.txt") in {
|
||||
item["file_path"] for item in scanner._cache.raw_data
|
||||
}
|
||||
|
||||
|
||||
def test_root_display_labels_dedupe_by_parent_segments(monkeypatch):
|
||||
labels = _root_display_labels(["/mnt/usb/loras", "/mnt/ssd/loras"])
|
||||
assert labels == {"/mnt/usb/loras": "usb/loras", "/mnt/ssd/loras": "ssd/loras"}
|
||||
|
||||
# Still colliding after one parent segment: keep growing leftwards.
|
||||
labels = _root_display_labels(["/mnt/a/models/loras", "/mnt/b/models/loras"])
|
||||
assert labels == {
|
||||
"/mnt/a/models/loras": "a/models/loras",
|
||||
"/mnt/b/models/loras": "b/models/loras",
|
||||
}
|
||||
|
||||
# A single root keeps the short form.
|
||||
assert _root_display_labels(["/mnt/usb/loras"]) == {"/mnt/usb/loras": "loras"}
|
||||
|
||||
# Windows drive prefixes disambiguate on their own.
|
||||
def _fake_splitdrive(path):
|
||||
for drive in ("G:", "H:"):
|
||||
if path.startswith(drive):
|
||||
return drive, path[len(drive):]
|
||||
return "", path
|
||||
|
||||
monkeypatch.setattr(model_scanner.os.path, "splitdrive", _fake_splitdrive)
|
||||
labels = _root_display_labels(["G:\\x\\loras", "H:\\y\\loras"])
|
||||
assert labels == {"G:\\x\\loras": "G: loras", "H:\\y\\loras": "H: loras"}
|
||||
|
||||
# Identical renderings (trailing separator / duplicated config entry) grow to
|
||||
# the full path and then fall back to a deterministic suffix instead of
|
||||
# silently sharing a label.
|
||||
labels = _root_display_labels(["/mnt/x/loras", "/mnt/x/loras/"])
|
||||
assert labels == {
|
||||
"/mnt/x/loras": "mnt/x/loras (1)",
|
||||
"/mnt/x/loras/": "mnt/x/loras (2)",
|
||||
}
|
||||
|
||||
long_root = "/mnt/" + "d" * 60 + "/loras"
|
||||
label = _root_display_labels([long_root])[long_root]
|
||||
assert len(label) <= 40
|
||||
assert label.endswith("loras")
|
||||
|
||||
Reference in New Issue
Block a user