Guide · API reliability

LLM API rate limits: why 429 happens and how to recover

A 429 Too Many Requests response means the request hit a rate limit; it does not mean the API key is invalid or that every model provider is down. Recover safely by identifying the exhausted dimension, honoring Retry-After when available, reducing concurrency and retrying only bounded, idempotent work.

HTTP 429Rate limitsRetriesAPI reliability

Reviewed by OpenLux Engineering Team · Last verified 2026-11-07

Create an API key Read the docs

Abstract orange glass sculpture illustrating LLM API rate limits on OpenLux — LLM API rate limits + OpenLux — one key, one endpoint, one bill.

Which LLM API error are you handling?

Retry only transient failures. Authentication, balance and most request-validation failures need a different response.

Status	Meaning	Retry automatically?	First response
401	Invalid, revoked or missing API key	No	Fix credentials or key scope
402	Credit or balance problem	No	Review the account or invoice error
429	Rate or capacity limit reached	Only with bounded backoff	Read Retry-After and identify the limit
5xx	Upstream or gateway failure	Sometimes	Retry safe work; inspect request ID

How we evaluated this

This guide is based on OpenRouter's published rate-limit documentation, OpenAI's rate-limit guidance, Anthropic's API error documentation, Google Gemini API limits and OpenLux's documented error behavior reviewed on 7 November 2026.

Rate-limit values are model- and account-specific and can change. The guide does not claim one universal quota; use the response headers, console and current provider documentation for the exact route.

429 recovery checklist

Work from evidence instead of adding an unlimited retry loop.

Read the response status, error body, Retry-After and request identifier.
Identify whether the limit is requests, tokens, concurrency or spend.
Check whether the limit belongs to your application, the gateway or the upstream provider.
Cap retries and use interruption-aware exponential backoff with jitter.
Retry only idempotent calls or operations protected by an idempotency key.
Reduce concurrency before increasing model fallback complexity.
Keep fallbacks explicit and visible in logs rather than silently substituting quality.
Alert on 429 rate, retry amplification and cost per accepted request.

Set it up in four steps

Capture the actual error

Log status, error type, Retry-After, request ID, model and route without storing prompt content. A generic “request failed” hides whether the issue is quota, tokens or concurrency.

Stop the burst

Lower client concurrency and batch fan-out. Retries from many workers can keep a system above the limit even when each individual retry has backoff.

Retry within a budget

Use a small maximum attempt count, deadline, jitter and Retry-After. Abort cleanly when the deadline expires rather than retrying forever.

Measure the recovery

Track lambda, first-attempt success, retry count, time to success and cost per accepted task. A recovered call that required four retries may be more expensive than a stable fallback.

What does 429 Too Many Requests mean?

429 means the request exceeded a rate, token, concurrency or capacity policy. The exact dimension depends on the provider and endpoint. Some systems limit requests per minute while others enforce tokens per minute, requests per day or the estimated spend of requests currently in flight.

The status alone is not enough to fix the problem. Read the error body and headers, then reduce the specific pressure that caused the response. Increasing the retry count without reducing concurrency usually moves the failure later rather than solving it.

Which limit did you hit: RPM, TPM, RPD or concurrency?

RPM limits requests per minute. TPM limits input or total tokens per minute. RPD limits requests per day. Concurrency limits the number of simultaneous calls. A batch job can respect its daily total and still fail because it sends too many large prompts in one burst.

Large prompts make TPM the likely failure even at modest request rates. Agent fan-out and parallel code review make concurrency the likely issue. Read the provider's rate-limit documentation and client headers to identify which counter moved.

Is the limit in your app, gateway or upstream provider?

A hosted gateway may expose its own account limits while also forwarding to an upstream provider with separate capacity constraints. Your client can apply a third limit. All three layers should use the same retry and observability vocabulary.

Record whether a 429 came from local admission control, the gateway or the upstream model. A gateway response should preserve enough request metadata to support that diagnosis. Retrying inside several layers can multiply the original request count and make the limit worse.

How should you use Retry-After?

When the server returns Retry-After, treat it as the minimum wait before another attempt. Add small random jitter so many workers do not retry at the same instant. If the header is absent, use bounded exponential backoff with a maximum delay.

Retry-After does not replace a request deadline. The caller still needs a maximum elapsed time and a maximum attempt count. In a synchronous user path, returning a clean temporary error may be better than holding a request open for several minutes.

What retry algorithm should you use?

Use exponential backoff with full jitter for transient failures, interruption-aware sleep and a small retry budget. Base delays might start below one second and grow to a capped delay, but the correct values depend on the endpoint and latency target.

Add a circuit breaker when repeated 429 or 5xx responses show that retrying is not helping. Admission control and queue shaping often reduce failures more effectively than another retry layer, especially for batch and agent workloads.

Which LLM requests are safe to retry?

Read-only generation and retrieval calls are usually safe if duplicate work is acceptable. Tool calls that create tickets, send messages, mutate databases or charge customers require idempotency keys or an application-level operation record before retrying.

A timeout is especially dangerous for side-effecting tool loops: the model call may have succeeded even when the client stopped waiting. Use stable operation IDs, check server state and make each external effect idempotent where the provider supports it.

How can you prevent 429 errors before they happen?

Benchmark the workload at expected concurrency rather than one request at a time. Shape traffic with queue limits, split large jobs into windows and reduce prompt size before adding more retries. For agents, cap parallel sub-agents and repeated tool loops.

Track success and failure by route, tenant, key and model. A single aggregate 429 rate hides whether one batch application is saturating the account while interactive traffic remains healthy. Budget alerts should cover retry amplification, not only first-attempt spend.

When should you fail over to another model?

Fail over only when the application defines an acceptable substitute and the fallback is measured against the same acceptance rules. A different model can have different tools, context limits, data handling and output behavior even when the request contract looks compatible.

Do not let fallback hide a capacity problem. Log the substitution, its reason and the resulting model. If a cheaper fallback fails quality or increases retries, the system may cost more and produce worse outcomes than waiting within a safe retry window.

What should you monitor after fixing a 429 incident?

Monitor 429 rate by endpoint and model, first-attempt success, attempts per accepted request, p95 time to success and estimated retry spend. Add alert thresholds tied to user impact rather than arbitrary totals.

Verify the fix under a representative burst, not only with a single successful request. Keep the test traffic isolated, record the request IDs and compare the retry rate and cost with the pre-incident baseline.

Does OpenLux guarantee a fixed rate limit?

OpenLux documents standard error classes including 429 for rate limits, but this page does not invent a universal numeric quota for every account or model. The applicable limit can depend on the current service and route.

Use the current console, API response and support documentation for the exact account. Build the client so it handles a 429 response without assuming that a permanent limit is always the cause; temporary capacity and upstream conditions can produce the same status.

Bounded retry example

Use the response status and Retry-After instead of retrying every exception. The example is intentionally small; production code should also propagate cancellation and emit metrics.

Python with the OpenAI SDK

import random
import time

from openai import APIStatusError, OpenAI

client = OpenAI(
    base_url="https://openlux.lol/v1",
    api_key="sk-openlux-...",
)

def create_with_retry(payload, max_attempts=4, max_delay=20.0):
    for attempt in range(max_attempts):
        try:
            return client.chat.completions.create(**payload)
        except APIStatusError as error:
            if error.status_code != 429 or attempt == max_attempts - 1:
                raise

            retry_after = error.response.headers.get("retry-after")
            delay = min(max_delay, 2 ** attempt)
            if retry_after:
                delay = max(delay, float(retry_after))
            time.sleep(delay + random.uniform(0, 0.5))

    raise RuntimeError("retry budget exhausted")

Key takeaways

429 means a rate or capacity limit, not automatically an invalid key.
Identify whether RPM, TPM, RPD, concurrency or spend caused the response.
Honor Retry-After when provided, add jitter and enforce a retry deadline.
Retry only safe work and protect side effects with idempotency.
Fallback must preserve the task's quality and data terms, not only availability.

Frequently asked questions

What does 429 Too Many Requests mean in an LLM API?

It means a request or account exceeded a rate, token, concurrency or capacity policy. Inspect the response body, headers, account state and provider documentation to identify the exact limit.

Should I retry a 429 immediately?

No. Honor Retry-After when present, otherwise use bounded exponential backoff with jitter. Reduce concurrency and stop when the retry deadline or attempt budget is exhausted.

What is the difference between RPM and TPM?

RPM limits requests per minute. TPM limits tokens per minute. A workload that sends few but very large prompts can hit TPM even when its request count looks low.

Can I increase the rate limit?

It depends on the provider and account. Some systems raise limits after verified spend, increased tiers or support review. Do not claim a higher limit until it appears in the account or provider response.

Should a 429 trigger a fallback model?

Only when the fallback is tested, visible and acceptable for the task. A silent fallback may change quality, tools, data handling or cost even though it fixes availability temporarily.

How do I monitor retry cost?

Track attempts per accepted request, 429 rate, added tokens, time to success and estimated spend. A retry loop can increase cost while lowering the first-attempt success rate.

Sources and official documentation

OpenAI rate limits guideOfficial rate-limit concepts and headers.
OpenAI API error codesOfficial error classification.
Anthropic API errorsOfficial Claude API error and retry guidance.
Google Gemini API rate limitsRPM, TPM, RPD and project-level limit behavior.
OpenRouter rate limitsCompetitor documentation for free-model and provider limits.
OpenLux API documentationOpenLux error classes and compatible endpoint details.

Test the retry path before production traffic

Run a controlled 429 and timeout test against a non-critical key, then verify retry, fallback and cost telemetry.

Create an API key See pricing