feat(services): add per-destination rate-limit gate for API traffic (#1085)

Implement Phase 1 of docs/plans/issue-1085-rate-limit-design.md:

- New RateLimitCoordinator: per-host shared Retry-After gate with
  exponential backoff (30s base, 1800s cap), minimum inter-request pacing
  (default 0.75s), herd-free waiter serialization via per-destination
  locks, and a bounded wait (default 300s) that raises instead of parking.
- Downloader.make_request: connectivity-guard fail-fast first, then gate
  pacing; on 429 register the cooldown and wait-and-resend (bounded);
  errors that passed through the gate are marked gate_handled.
- FallbackMetadataProvider / MetadataSyncService: a network provider 429
  no longer fails over to other network providers (stops the CivArchive
  flood); sqlite stays as local last resort. Rate-limited lookups now
  report "Rate limited" instead of "Model not found", so transient 429s
  no longer mark models civitai_deleted.
- _RateLimitRetryHelper skips its own sleep for gate_handled errors,
  removing the double wait.
- New settings: rate_limit_gate_enabled, rate_limit_max_wait_seconds,
  rate_limit_min_interval_seconds.
This commit is contained in:
Will Miao
2026-08-27 09:53:07 +08:00
parent 1e1921cabb
commit c2a2048c8b
9 changed files with 909 additions and 104 deletions
+19 -4
View File
@@ -101,7 +101,9 @@ async def test_fallback_retries_same_provider_on_rate_limit(monkeypatch):
@pytest.mark.asyncio
async def test_fallback_continues_to_next_provider_on_rate_limit(monkeypatch):
"""After exhausting retries on primary, fallback should continue to secondary."""
"""#1085: a rate-limited network provider no longer fails over to another
network provider (that just spreads the flood); local providers such as
sqlite remain as a last resort."""
sleep_mock = AsyncMock()
monkeypatch.setattr(provider_module.asyncio, "sleep", sleep_mock)
monkeypatch.setattr(provider_module.random, "uniform", lambda *_: 0.0)
@@ -114,13 +116,26 @@ async def test_fallback_continues_to_next_provider_on_rate_limit(monkeypatch):
rate_limit_retry_limit=2,
)
# After Change A: no longer raises; falls through to secondary
result, error = await fallback.get_model_by_hash("abc")
# Secondary is a network provider: it must NOT be consulted after the 429.
assert result is None
assert error == "Rate limited"
assert primary.calls == 2 # retry_limit exhausted on primary
assert secondary.calls == 0 # no network failover
# A local sqlite provider behind the rate-limited one is still allowed.
sqlite = TrackingProvider()
fallback = FallbackMetadataProvider(
[("primary", AlwaysRateLimitedProvider()), ("sqlite", sqlite)],
rate_limit_retry_limit=2,
)
result, error = await fallback.get_model_by_hash("abc")
assert error is None
assert result == {"id": "secondary"}
assert primary.calls == 2 # retry_limit exhausted on primary
assert secondary.calls == 1 # secondary IS called now
assert sqlite.calls == 1
@pytest.mark.asyncio