fix(download): retry on Civitai 429 rate limit instead of removing images from metadata

When Civitai returns 429 (Too Many Requests) during example image
downloads, the previous behavior treated all failures identically and
permanently removed the corresponding images from model metadata —
making them impossible to retry.

This commit adds:
- 429 detection + Retry-After header parsing in download_to_memory
- Exponential backoff retry (up to 3 attempts) in
  download_model_images_with_tracking
- Separate tracking of rate-limited vs permanently failed URLs
- rate_limited_models progress tracking persisted to disk
- Rate-limited models are NOT added to failed_models/processed_models
  so they are automatically retried on subsequent download runs
- Force mode clears failed_models when rate-limited images exist
This commit is contained in:
Will Miao
2026-07-06 11:58:19 +08:00
parent 8bee8f4069
commit 00228deaaa
5 changed files with 213 additions and 66 deletions

View File

@@ -164,7 +164,7 @@ async def test_pause_resume_blocks_processing(
await first_release.wait()
else:
second_call_started.set()
return True, False, []
return True, False, [], []
async def fake_get_downloader():
class _Downloader:
@@ -288,7 +288,7 @@ async def test_legacy_folder_migrated_and_skipped(
async def fake_download_model_images(*_args, **_kwargs):
nonlocal download_called
download_called = True
return True, False, []
return True, False, [], []
async def fake_get_downloader():
class _Downloader: