Guide · API reliability

LLM API timeout: why requests time out and how to recover safely

An LLM API timeout means a client, gateway or upstream service stopped waiting before the request completed. It does not prove the model call failed. Recover safely by identifying the timeout stage, separating connection, first-token and total deadlines, checking whether side effects committed and logging enough evidence to distinguish cancellation from an upstream failure.

LLM timeoutsRequest deadlinesStreamingRetries

Reviewed by OpenLux Engineering Team · Last verified 2026-11-13

Create an API key Read the docs

Abstract orange glass sculpture illustrating LLM API timeout on OpenLux — LLM API timeout + OpenLux — one key, one endpoint, one bill.

Which timeout stage are you actually seeing?

“Request timed out” is too broad to diagnose. Record where elapsed time was spent and which deadline fired.

Stage	Typical symptom	Evidence to capture	Safe response
TCP connect or pool	No response bytes are received	DNS, connect and pool wait time; local or gateway caller	Check reachability and pool pressure; retry only safe requests
Request upload or write	Large multimodal or tool payload stalls	Request bytes accepted, upload duration and payload size	Reduce payload or reconnect; protect side effects before retrying
First token or response header	Stream has not started after the user deadline	Time to first byte or first token and model call start	Cancel cleanly and return a bounded timeout path
Inter-token idle	A stream starts, then stops emitting chunks	Last chunk time, bytes or tokens received and finish reason	Use an idle timeout separate from the total deadline
Total completion	Tools, reasoning or a long response exceed the job deadline	Total elapsed time, attempts, tool spans and accepted partial output	Return partial or queued work, then retry only after state checks
Gateway 504	The gateway cannot get a timely upstream response	Request ID, upstream status, gateway timing and retry history	Inspect the request before deciding whether to retry

How we evaluated this

We reviewed OpenAI’s error guidance, Anthropic’s API error documentation, Google Gemini API troubleshooting, OpenRouter’s error guide, Portkey’s documented timeout errors, LiteLLM’s routing behavior and OpenLux’s API documentation on 13 November 2026.

Timeout defaults and maximum durations depend on the client, SDK, gateway, endpoint and upstream model. This guide defines a diagnostic and deadline framework rather than presenting one universal timeout value.

LLM timeout production checklist

Treat deadlines as a hierarchy across browser, application, gateway and provider.

Set separate connect, pool, write, first-token or read, idle and total deadlines.
Keep the application deadline shorter than the user, ingress and job deadlines it controls.
Calculate a total deadline budget before starting retries or fallbacks.
Record the timeout stage, elapsed time, request ID and bytes or chunks received.
Assume a timeout may have completed remotely until operation state proves otherwise.
Use idempotency keys or operation records before retrying side-effecting agent tools.
Cancel and release streaming responses, sockets and background work on timeout.
Retry only transient connection, timeout or upstream failures with bounded backoff and jitter.
Do not retry invalid keys, malformed payloads, context overflow or policy rejections.
Test slow first token, a stalled stream, a slow non-streaming response and a cancelled request.

Set it up in four steps

Map the deadline hierarchy

Document every deadline from browser request to ingress, application, gateway and upstream provider. The innermost timeout must fire before the layer waiting on it.

Instrument each stage

Capture connect, upload, first byte, first token, last chunk and total elapsed time under one request ID without logging prompt content.

Classify the failure

Separate local cancellation, SDK timeout, gateway 504, upstream timeout and a non-timeout validation error before choosing retry or fallback.

Test recovery and side effects

Inject slow and stalled responses, then verify cancellation, retry budget, fallback behavior, cost and duplicate-effect protection.

What is an LLM API timeout?

An LLM API timeout occurs when one layer stops waiting before the response reaches a terminal state. The client may time out before connecting, while uploading a request, before the first streamed token, between tokens or after the total task deadline expires.

The failure belongs to the deadline that fired, not simply to “the API.” A browser, SDK, application server, gateway and model provider can all enforce different timeouts. Production diagnosis starts by identifying the innermost timeout and everything that layer was waiting for.

Where do LLM API timeouts happen?

Connection and pool timeouts happen before a usable socket exists. Write timeouts occur while sending a large request. Read timeouts occur while waiting for response data, and streaming systems add first-token and inter-token idle deadlines. A total deadline covers the complete task, including retries and tool calls.

These stages fail differently. A slow first token can be normal for a reasoning model, while a stalled stream after partial output usually indicates transport or upstream interruption. Combine stage timings with request IDs rather than recording one ambiguous total elapsed value.

What causes a 504 gateway timeout?

A 504 means an intermediary, commonly a reverse proxy or API gateway, did not receive an upstream response before its own deadline. The upstream call may still be running. A non-streaming long-running request is especially likely to produce a 504 even though the model eventually finishes.

Use streaming or asynchronous jobs for workloads that legitimately exceed the synchronous gateway window. If a 504 persists for normal requests, compare provider latency, queueing, tool spans, retry history and ingress timeout settings before changing the model.

Why is a timeout not proof that the request failed?

A timeout is a local observation: the caller stopped waiting. The request may have reached the provider, completed a tool call or committed a side effect. Retrying without checking state can create duplicate tickets, messages, charges or database writes.

Use a stable operation ID, idempotency key or server-side status endpoint. For read-only generation, duplicate work may only add cost. For agent tools and customer-visible actions, verify whether the first operation committed before starting another attempt.

How should streaming LLM timeouts work?

Streaming needs at least three deadlines: time to response headers or first token, maximum idle time between chunks and total completion time. A first-token deadline protects user-perceived responsiveness; an idle deadline detects a stalled stream; a total deadline prevents an endless agent or output loop.

When a deadline expires, close the response and release the HTTP connection. Record whether any text, tool call or finish reason arrived. A partial stream is user-visible output and must not be mistaken for an untouched request when retry or continuation is considered.

What deadline hierarchy should you set?

Start with the experience or job contract. An interactive chat may need a first token within seconds and a complete answer within tens of seconds; a batch extraction job may allow minutes. Derive inner timeouts from that budget so the application can return a controlled error before a browser or ingress cuts the connection.

The total budget must include connection, upload, model time, tool calls, backoff and fallback. If each retry receives the full timeout again, one user request can occupy several minutes and multiply cost. Keep one outer deadline and pass its remaining time to each inner attempt.

Should LLM timeouts be retried?

Retry a connection failure or upstream timeout only when the operation is safe to repeat and the remaining deadline can fit another meaningful attempt. Add jitter, cap attempts and classify the error. A client-side total deadline that expires with no bytes may still have reached the upstream service.

Do not retry authentication, malformed requests, context overflow, safety rejections or non-transient 4xx responses. For 429 responses, reduce concurrency first and honor Retry-After where available instead of treating every timeout as a provider outage.

How do you diagnose a timeout with request IDs?

Log the request ID, operation ID, SDK timeout configuration, stage reached, elapsed milliseconds, model, provider, attempt, bytes or chunks received and cancellation reason. The same request ID should appear in gateway and application evidence without requiring prompt bodies.

Compare successful and timed-out requests by model, input size, stream mode, tool count and route. If only long tool workflows fail, adjust the task architecture. If every request fails before headers, inspect connectivity and gateway health. If only streams stall, inspect idle handling and upstream completion.

How do timeouts interact with retries and fallbacks?

Retry, fallback and timeout share one budget. Retry the same route for a short transient failure, then use an approved fallback only if the primary remains unavailable and the remaining deadline permits it. A fallback started after the user deadline expires produces cost without serving the request.

Record the final route and why it changed. A timeout that silently selects a different model can alter quality, tools, context and data terms even when the fallback returns a valid HTTP response.

Where does OpenLux fit in a timeout architecture?

OpenLux provides one OpenAI-compatible endpoint, multi-region routing and automatic retries for managed infrastructure. These controls reduce provider integration work, but the caller still owns the user deadline, SDK timeout settings, cancellation and idempotency of side effects.

Stream high-latency workloads and keep total job deadlines explicit. Inspect request and usage logs for the model, latency, status and cost evidence needed to distinguish gateway, upstream and application behavior. This page does not claim one universal timeout for every model or endpoint.

A staged streaming timeout example

This example records first-event, idle and total deadlines with one HTTP request. Add operation-state or idempotency checks before applying it to side-effecting tools.

Python with httpx

import json
import time

import httpx

FIRST_EVENT_SECONDS = 8.0
IDLE_SECONDS = 5.0
TOTAL_SECONDS = 30.0

timeout = httpx.Timeout(
    connect=3.0,
    read=20.0,
    write=10.0,
    pool=2.0,
)

def stream_completion(prompt: str) -> str:
    started = time.monotonic()
    last_event = started
    first_event = None
    chunks: list[str] = []

    with httpx.Client(timeout=timeout) as client:
        with client.stream(
            "POST",
            "https://openlux.lol/v1/chat/completions",
            headers={"Authorization": "Bearer sk-openlux-..."},
            json={
                "model": "gpt-5.6-sol",
                "stream": True,
                "messages": [{"role": "user", "content": prompt}],
            },
        ) as response:
            response.raise_for_status()

            for line in response.iter_lines():
                now = time.monotonic()
                if first_event is None and now - started > FIRST_EVENT_SECONDS:
                    raise TimeoutError("first streaming event deadline exceeded")
                if now - last_event > IDLE_SECONDS:
                    raise TimeoutError("stream idle deadline exceeded")
                if now - started > TOTAL_SECONDS:
                    raise TimeoutError("total request deadline exceeded")

                if not line.startswith("data: "):
                    continue
                data = line.removeprefix("data: ").strip()
                if data == "[DONE]":
                    break

                event = json.loads(data)
                first_event = first_event or now
                last_event = now
                delta = event.get("choices", [{}])[0].get("delta", {})
                chunks.append(delta.get("content") or "")

    return "".join(chunks)

Key takeaways

A timeout reports where a caller stopped waiting; it does not prove the upstream request failed.
Keep connection, first-token, idle and total deadlines separate.
Use one outer budget across retries, fallbacks and tool calls.
Check operation state or idempotency before retrying side effects.
Cancel and release streaming responses when a deadline expires.
Log request ID, timeout stage, elapsed time and bytes received without storing prompt payloads.

Frequently asked questions

What does an LLM API timeout mean?

It means a client, SDK, gateway or upstream system stopped waiting before the request completed. The timeout stage matters because the request may still have reached the model or committed a side effect.

Is an LLM API timeout the same as a 504 error?

No. A client or SDK can time out without receiving any HTTP response. A 504 specifically means an intermediary did not receive a timely response from an upstream service.

Should I retry a timed-out LLM request?

Retry only when the work is safe to repeat, the failure is transient and the remaining deadline allows a meaningful attempt. Check operation state before replaying side-effecting tools.

What timeout should I set for an LLM API?

Derive it from the user or job deadline, model latency and expected output. Set separate connect, first-token, idle and total deadlines rather than copying one universal number.

Why does a streaming LLM response stall after the first token?

The transport, gateway or upstream model can stop emitting chunks. Use an idle timeout, retain what was already delivered and record the last event time before deciding whether to retry.

How do I stop a timeout from creating duplicate work?

Use stable operation IDs or idempotency keys, check whether the upstream operation committed and make customer-visible tool actions idempotent whenever the provider supports it.

Does OpenLux provide automatic timeout handling?

OpenLux provides managed routing and automatic retries for gateway infrastructure, but callers still set client deadlines, handle cancellation and protect side-effecting retries. Use the current endpoint behavior and request logs rather than assuming one universal timeout.

Sources and official documentation

OpenAI API error codesOfficial timeout, SDK and API error guidance.
Anthropic API errorsOfficial Claude API error and retry behavior.
Google Gemini API troubleshootingOfficial Gemini API error and timeout troubleshooting.
OpenRouter error handlingCompetitor documentation for provider, stream and gateway errors.
Portkey timeout error guideCompetitor timeout-error documentation and diagnostic context.
LiteLLM routing documentationSelf-hosted routing, retries, cooldowns and timeout behavior.
OpenLux API documentationEndpoints, errors, streaming and request visibility.

Measure the whole deadline, not just the model call

Run a controlled slow-first-token and stalled-stream test, then verify cancellation, retry budget and final operation state.

Create an API key See pricing