When CivArchive returns HTTP 429 with a large retry_after, the bulk
metadata refresh would block for hours because:
1. FallbackMetadataProvider raised RateLimitError instead of continuing
to the next provider (e.g., SQLite archive was never reached).
2. _RateLimitRetryHelper retried long-rate-limit 429s 3 times — all
futile since the hourly cap hasn't reset.
3. The batch loop had no awareness of persistent rate-limiting,
causing 192+ models to each hammer the same rate-limited endpoint.
Changes:
- FallbackMetadataProvider: all 6 methods now continue to next provider
on RateLimitError instead of raising (model_metadata_provider.py)
- fetch_and_update_model: deleted-model path also continues on
RateLimitError so sqlite provider gets a chance (metadata_sync_service.py)
- _RateLimitRetryHelper: when retry_after >= 120s, only 1 attempt is
made — retries are futile for hour-scale rate limits
- BulkMetadataRefreshUseCase: tracks consecutive rate-limit failures
and aborts early after 3 (bulk_metadata_refresh_use_case.py)
Tests: updated test_fallback_respects_retry_limit for new continue
behavior; added tests for large/small retry_after thresholds.
Add RateLimitRetryingProvider and _RateLimitRetryHelper classes to handle rate limiting with exponential backoff retries. Update get_metadata_provider function to automatically wrap providers with rate limit handling. This improves reliability when external APIs return rate limit errors by implementing automatic retries with configurable delays and jitter.
- Add RateLimitError import and _make_request wrapper method to handle rate limiting
- Update API methods to use _make_request wrapper instead of direct downloader calls
- Add explicit RateLimitError handling in API methods to properly propagate rate limit errors
- Add _extract_retry_after method to parse Retry-After headers
- Improve error handling by surfacing rate limit information to callers
These changes ensure that rate limiting from the Civitai API is properly detected and handled, allowing callers to implement appropriate backoff strategies when rate limits are encountered.
