Timeouts

Every timeout is a deployment setting. A request cannot change it. An application policy cannot change it.

Request timeout map

The diagram shows each timeout on the hop that it bounds.

Settings

Variable Default What it bounds

INFERENCE_TIMEOUT

180 s

One call to the inference backend.

TIMEOUT_MAX_SECONDS

540 s

The ceiling for INFERENCE_TIMEOUT and for the API wait.

TIMEOUT_REPLY_MARGIN_SECONDS

30 s

The time the API waits after the inference call budget.

TIMEOUT_GUARDRAIL_CHECK_SECONDS

30 s

One guardrail check call.

TIMEOUT_TOOL_EXECUTION_SECONDS

60 s

One MCP tool call, sandbox calls included, or one Sherlock call.

WORKER_JOB_TIMEOUT

600 s

One worker job, with all of its calls.

HTTP_CONNECT_TIMEOUT_SEC

5 s

Opening a connection on the shared HTTP client.

HTTP_POOL_TIMEOUT_SEC

5 s

Waiting for a free connection in the shared HTTP client pool.

HTTP_READ_TIMEOUT_SEC

unset

Reading a response on the shared HTTP client. Unset means no limit.

HTTP_WRITE_TIMEOUT_SEC

unset

Sending a request on the shared HTTP client. Unset means no limit.

A call with its own timeout replaces all four HTTP_* values for that call. The inference call and the guardrail check have their own timeout. Every Sherlock call has its own timeout too. GET /v1/models uses the HTTP_* values only, so it has no read limit.

The gateway route in front of the service stops every connection after 600 seconds. That value is not part of this service.

What the client sees

A timeout that reaches the client returns HTTP 504. The error message names the operation:

Operation Cause

inference

The API waited INFERENCE_TIMEOUT + TIMEOUT_REPLY_MARGIN_SECONDS (210 s) for the worker. The API cancels the worker job.

guardrail_check

The guardrail check took longer than TIMEOUT_GUARDRAIL_CHECK_SECONDS.

worker_job

The worker job took longer than WORKER_JOB_TIMEOUT.

list_models

GET /v1/models did not get an answer in time.

sherlock

An image upload, an image fetch, or a search store call took longer than TIMEOUT_TOOL_EXECUTION_SECONDS.

An MCP tool call timeout does not return 504. The model gets the timeout as the tool error and the request continues.

Treat a 504 as retryable. Keep your client wait above 210 seconds. For a request that can run longer, use async jobs.

Sherlock calls

Sherlock call timeouts

TIMEOUT_TOOL_EXECUTION_SECONDS bounds every call to Sherlock:

  • The API creates a search store and uploads an image. A timeout returns HTTP 504 with operation sherlock.

  • The worker fetches image bytes before the inference call. A timeout returns HTTP 504 with operation sherlock.

  • The file_search and fetch_document tools search and read pages. A fetch_document timeout reaches the model as a tool error.

  • The shell tool and the skills API use the Sherlock SDK with the same timeout. An SDK timeout is not mapped to 504.

Limits to know

  • The agentic loop applies INFERENCE_TIMEOUT to each call. It can make up to 10 calls. When the calls are slow, the 210-second API wait stops the request first.

  • A streaming request gets two 210-second windows: one for the chunks and one for the final reply.

  • A sandbox run longer than TIMEOUT_TOOL_EXECUTION_SECONDS fails as a tool error.