Guardrails
The Responses API includes built-in content safety checks. Every request is screened before reaching the LLM backend. What happens on a violation depends on the policy: a category can be blocked, labelled (the request proceeds with a notice naming the categories), or allowed.
Three provider types ship out of the box:
-
llama-guard-3, content-safety classifier based on LlamaGuard 3. LLM-based, ~500ms–2s per check. -
regex, fast pattern matcher for deterministic rules (PII patterns, blocklists, malicious URLs). Microseconds per check. -
llm-judge, LLM-based judge against a free-form natural-language policy the application manager supplies. See Providers below.
A fourth, openrouter-llama-guard, is available but off by default. It runs
the same LlamaGuard classification against a model hosted on OpenRouter, as a
temporary bridge while our own LlamaGuard is unavailable.
Content checked by this provider leaves the cluster (see Data Flows). It is registered only when
GUARDRAILS_OPENROUTER_API_KEYis set, and reached only by a stage that names it, so it stays off unless an operator opts in twice. Naming it in a policy without the key set makes every check fail closed.
Like llama-guard-3 it ignores stage.config and takes its categories from
global configuration.
Images have a provider of their own, sherlock-image-classifier, registered
by the guardrails service (never the api) when GUARDRAILS_IMAGE_SHERLOCK_BASE_URL
is set on it. It classifies a request’s
images with a vision model through the Data Platform (Sherlock) and runs in a
task of its own, beside the text stages. See
sherlock-image-classifier below.
Providers are arranged into a pipeline per checkpoint. Only two checkpoints are valid: input and output. Any other key, tool_output included, is rejected at policy load or at request validation, not silently ignored. The pipeline runs stages sequentially and short-circuits on the first stage that returns a violation, so cheap regex checks can fail fast before the expensive LLM stage even runs.
input checks the user’s message before it reaches the model. output checks the model’s own answer once generation finishes, before the caller sees it. Both Llama Guard providers frame an output check as judging an Agent turn, not the user’s message. See Framing an output check below.
Guardrails are evaluated eagerly for input, before the model generates anything. For output, a non-streaming response is checked once, after generation completes; a streaming response is checked item by item beside the live stream (see Streaming and the output checkpoint). This applies to synchronous, streaming, and background=true requests.
How It Works
-
The service resolves the application identity from the verified access token and posts to
POST /v1/checkwith thatapplication_id, acheck_type("input"or"output"), and the content to check. -
GuardrailServicelooks up the per-application policy and walks the pipeline configured for the requested checkpoint. -
For each enabled stage, the registry dispatches to the provider named in
stage.provider(e.g.regex,llama-guard-3,llm-judge), passing the stage-specificconfig. -
Violations are collected across every stage and the most severe action wins, so a category one stage only labels cannot mask one a later stage blocks. The pipeline stops early only once something is blocked, since nothing outranks it.
-
If all stages pass, the request proceeds (an
inputcheck lets the request reach the model; anoutputcheck releases the answer to the caller).
The endpoint is also callable directly by other services that need to evaluate content against an application policy.
Framing an output check
An output check scores the model’s own answer, not the user’s message. Send context: {"checkpoint": "output", "user_prompt": "<the original user message>"} on the request to get the full framing: the Llama Guard providers then classify the content as the last turn in a User → Agent exchange. Without user_prompt, the content is still classified under the agent-response template, but as the only turn, with no preceding User turn. Both providers still score it as an answer, never as if the user had written it. /v1/responses builds this context automatically from the original prompt; a direct /v1/check caller wanting the full framing must pass user_prompt itself.
Standalone Check Endpoint
The guardrail pipeline is exposed directly at POST /v1/check for callers that
want to screen content without going through /v1/responses. It uses the same
per-application policy lookup and bearer auth as the rest of the API.
Example:
curl -X POST "$BASE_URL/v1/check" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"application_id": "test-app",
"check_type": "input",
"input": "What is the capital of France?"
}'
Request:
{
"application_id": "legal-app",
"check_type": "output",
"input": "Here is how you write a contract for...",
"context": {
"checkpoint": "output",
"user_prompt": "How do I write a contract for..."
}
}
| Field | Type | Description |
|---|---|---|
|
|
Application whose policy to apply. Omit (or send |
|
|
Which checkpoint to run. Only |
|
|
Content to check. May be empty when |
|
array (optional) |
Stored images to check: |
|
integer (optional) |
How many of the input’s images cannot be named in |
|
|
|
|
|
|
Response (safe):
{
"safe": true,
"violations": [],
"action": "allow",
"evaluated": true,
"configured_checkpoints": ["input", "output"],
"streaming_allowed": false
}
Response (unsafe):
{
"safe": false,
"violations": [
{
"category": "Violent Crimes",
"provider": "llama-guard-3",
"stage": "content-safety",
"step": 0
}
],
"action": "block",
"evaluated": true,
"configured_checkpoints": ["input", "output"],
"streaming_allowed": false
}
Response fields:
| Field | Type | Description |
|---|---|---|
|
boolean |
|
|
array |
Empty when |
|
string |
|
|
boolean |
Default |
|
array of string | |
The application’s configured checkpoint keys, sorted (for example |
|
boolean |
|
Violation fields:
| Field | Type | Description |
|---|---|---|
|
string |
The violated category (e.g. |
|
string |
Registry key of the stage that produced the violation (e.g. |
|
string |
|
|
integer |
Zero-indexed position in the configured pipeline list. Disabled stages do not change |
Attribution fields let callers running multi-provider pipelines tell which stage produced the block, for example, distinguishing a regex PII match from a llama-guard-3 content-safety classification when both are configured for the same checkpoint.
Status codes:
| Status | When |
|---|---|
|
Check completed (inspect |
|
Request body failed schema validation, including a |
|
|
|
The application has no pipeline configured for the requested |
Application Identity
The application identity that selects a per-application policy comes from the verified access token. Callers cannot set or override it.
| Deployment | Source of the application identity |
|---|---|
IAM |
A claim on the access token. |
Platform (JWT) |
The Token Vending Machine (TVM) mints the token. The service derives the identity from the workload identity the TVM binds into it. |
When the token carries no application identity, the lookup resolves to application_id=null and the service routes it to the configured default policy block.
An application explicitly named "default" is a regular entry and does not collide with the default fallback block. The null case is the only way to reach the fallback, unknown application IDs are rejected rather than silently inheriting the default policy.
Configuration
Guardrails are controlled via environment variables with the GUARDRAILS_ prefix:
| Variable | Pod | Default | Description |
|---|---|---|---|
|
API |
|
Disables guardrails entirely. |
|
API |
|
Model used by the |
|
API |
- |
Enables the |
|
API |
Endpoint for the |
|
|
API |
|
Model asked by the |
|
API |
|
Request timeout for the |
|
API |
|
Most in-flight requests to the |
|
API |
- |
Inline JSON array of LlamaGuard category definitions (highest priority) |
|
API |
- |
Path to a JSON file with LlamaGuard category definitions |
|
API |
- |
Inline JSON with per-application policy config (takes priority over |
|
API |
- |
Path to a JSON file with per-application policy config |
|
API |
|
Expose the |
|
Guardrails |
- |
Enables the |
|
Guardrails |
|
Longest an image stage may take end to end: creating the classification workflow, running it and reading the labels. The workflow is deleted afterwards, off the request path. Image stages run one after another, so keep this value times the number of image stages below |
|
Guardrails |
|
Interval between polls of the classification run. |
|
Worker |
Base URL for the |
Policy resolution priority: Postgres (if database is configured) > GUARDRAILS_POLICY_JSON > GUARDRAILS_POLICY_FILE > generated default policy from GUARDRAILS_LABELS.
The DB layer is highest-precedence so a runtime override beats a baked-in policy. When the DB has no row for an application_id (or the row is disabled), the chain falls through to the next layer, existing deployments without a DB row keep their current behaviour.
Helm deployments: ConfigMap-mounted policy
For Helm deployments, the recommended way to ship a per-application policy is the guardrails.policy value. The chart renders it into a ConfigMap and mounts it at /etc/guardrails/policy.json on the API pod (GUARDRAILS_POLICY_FILE is wired automatically):
# values.yaml
guardrails:
policy:
default:
fail_mode: closed
check_types:
input:
pipeline:
- provider: llama-guard-3
name: content-safety
config: {}
applications:
legal-app:
fail_mode: closed
check_types:
input:
pipeline:
- provider: regex
name: pii-patterns
config:
patterns:
- { name: steuer_id, pattern: '\b\d{11}\b', category: PII }
- provider: llama-guard-3
name: content-safety
config: {}
Why use the ConfigMap path:
-
The policy is reviewable as native YAML in
values.yaml(no JSON-in-string-in-env-var escape gymnastics). -
A
checksum/configmap-guardrails-policyannotation rolls the API pod whenever the policy changes. -
ConfigMaps hold up to ~1 MiB; env-var size limits don’t bite.
-
Out-of-band edits to the ConfigMap (
kubectl edit configmap …-api-guardrails-policy) do not automatically restart the API, runkubectl rollout restart deployment/<api-deployment>to pick them up. Hot-reload via file-watcher is not yet implemented.
GUARDRAILS_POLICY_JSON remains supported as an escape hatch (smoke tests, single-pod overrides). When both are set, the inline env var wins.
Image stages. The chart sets GUARDRAILS_IMAGE_SHERLOCK_BASE_URL on the
guardrails pod (never on the api) from
guardrails.imageClassifier.sherlockBaseUrl, falling back to
skills.sherlockBaseUrl, whenever one of them is set. It does not depend on
guardrails.policy: image stages from the Postgres/admin API or
GUARDRAILS_POLICY_JSON need the provider too. A stage then reaches the
sherlock-image-classifier with
modality: image:
guardrails:
imageClassifier:
sherlockBaseUrl: https://sherlock.example.com
policy:
applications:
legal-app:
fail_mode: closed
check_types:
input:
pipeline:
- provider: sherlock-image-classifier
name: image-safety
modality: image
config:
provider: pharia_next
model: moonshotai/kimi-k2.6
Set config.provider and config.model to a vision model the platform’s
inference serves: Holmes' default classifier model is not served everywhere.
Postgres-backed policy + admin API
For installations that already use Postgres for history/conversations/pending requests, policies can additionally be configured at runtime through HTTP, no ConfigMap edits, no pod rollouts.
Turn it on by setting GUARDRAILS_ADMIN_API_ENABLED=true. The application_policies table is created by Alembic revision 008_application_policies (runs automatically on startup with the rest of the migrations).
The admin endpoints are scoped to admin principals only (User.is_admin == True); the feature flag is the gate of last resort, so leave it off unless your auth setup is hardened. Endpoints:
| Method | Path | Description |
|---|---|---|
|
|
List all stored policies |
|
|
Fetch one, 404 if missing |
|
|
Upsert; body is an |
|
|
Delete; 204 on success, 404 if missing |
Example: upsert a policy for legal-app:
curl -X PUT \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
https://your-api/v1/admin/policies/legal-app \
-d '{
"application_id": "legal-app",
"fail_mode": "closed",
"check_types": {
"input": {
"pipeline": [
{"provider": "regex", "name": "pii", "config": {"patterns": [{"name": "steuer_id", "pattern": "\\b\\d{11}\\b", "category": "PII"}]}}
]
}
}
}'
The path application_id is authoritative; if the JSON body carries a different application_id, the path value wins. IDs are capped at 253 characters (DNS-1123), longer values return 422.
GUARDRAILS_POLICY_JSON / GUARDRAILS_POLICY_FILE remain useful for bootstrap: ship an initial policy via env var, then migrate it into the DB through the admin API. Both env-var and file paths still serve as fallbacks for any application_id not present in the DB.
app-manager-gitops deployments
-
Platform only
app-manager-gitops descriptors set envVars.GUARDRAILS_POLICY_JSON to the inline JSON policy:
{
"config": {
"envVars": {
"GUARDRAILS_POLICY_JSON": "{\"default\":{...},\"applications\":{...}}"
}
}
}
The platform-manager API supports config.volumes.configMap[] for mounting an existing ConfigMap, but exposes no endpoint for creating one, the only resource type currently provisionable via /tenants/{tenant}/resources is redis. Until that gap closes, gitops deployments stay on the inline env var.
Per-Application Policy
The policy JSON keys applications by ID and defines a pipeline per check type:
{
"default": {
"fail_mode": "closed",
"check_types": {
"input": {
"pipeline": [
{
"provider": "llama-guard-3",
"name": "content-safety",
"enabled": true,
"config": {}
}
]
}
}
},
"applications": {
"legal-app": {
"fail_mode": "closed",
"check_types": {
"input": {
"pipeline": [
{
"provider": "regex",
"name": "pii-patterns",
"config": {
"patterns": [
{
"name": "steuer_id",
"pattern": "\\b\\d{11}\\b",
"category": "PII"
}
]
}
},
{
"provider": "llama-guard-3",
"name": "content-safety",
"config": {}
}
]
}
}
}
}
}
Top-level keys:
| Key | Description |
|---|---|
|
Fallback policy block, reachable only when the request omits |
|
Map of |
Per-application fields:
| Field | Type | Description |
|---|---|---|
|
|
What to do on provider error. |
|
object |
Map of checkpoint name to its pipeline configuration. Only |
|
boolean |
Accepts the streaming risk for this application: |
Per-stage fields:
| Field | Type | Description |
|---|---|---|
|
string |
Provider registry key, e.g. |
|
string |
Unique stage name within the pipeline. Used in logs. |
|
boolean |
Toggle a stage off without removing it from the config. Default |
|
|
What the stage checks. Default |
|
object |
Provider-specific configuration. See below. |
Providers
llama-guard-3
LLM-based content safety classifier. The config block is currently ignored, LlamaGuard category selection is configured globally via GUARDRAILS_LABELS rather than per-stage.
On an input check it classifies the content as the user’s message. On an output check it classifies the content as an Agent turn, using Llama Guard 3’s agent-response template, never the user-message template. See Framing an output check.
regex
Pattern-based provider. Config schema:
{
"patterns": [
{ "name": "steuer_id", "pattern": "\\b\\d{11}\\b", "category": "PII" },
{ "name": "javascript_url", "pattern": "javascript:", "category": "MaliciousURL" }
]
}
Each pattern has a name (used in logs only, pattern names and category names are logged, but never the matched substring), a pattern (Python regular expression), and a category (returned as the violation category). Multiple patterns sharing a category produce a single deduplicated violation.
Invalid regex patterns raise at check time and are handled by fail_mode like any other provider error.
llm-judge
LLM-based judge that applies a natural-language policy supplied by the application manager. Each judge stage takes a template (the policy text) and a model (an operator-approved Responses-API model name). The provider wraps both in a server-controlled prompt envelope and parses a strict SAFE / UNSAFE first-line verdict.
Config schema:
| Field | Type | Required | Default | Notes |
|---|---|---|---|---|
|
string |
yes |
- |
Any model the inference backend can serve. The platform does not gatekeep model selection; an unknown model surfaces as a backend error at check time. |
|
string |
yes |
- |
Free-form NL policy. Length 20..2000 chars. Cannot contain reserved envelope tokens or control characters. |
|
string |
no |
|
Returned as the |
|
int |
no |
|
Per-stage user-content cap, clamped against |
Example stage:
{
"provider": "llm-judge",
"name": "stay-on-topic",
"config": {
"model": "judge-1",
"template": "Reject any message not related to legal advice in Germany.",
"violation_category": "Off-Topic"
}
}
Templates are open-ended natural-language strings, the same schema covers stay-on-topic checks, natural-language denylists ("reject any message that asks about competitors"), tone policies ("reject aggressive or threatening messages"), and anything else expressible as a one-paragraph policy. The provider does not know which kind of check the manager wrote.
Operator setup. One env var controls the per-stage user-content cap:
-
GUARDRAILS_JUDGE_MAX_INPUT_CHARS, upper bound on per-stagemax_input_chars(default8000).
There is no operator-side allowlist for judge models. Application managers may pick any model the backend can serve; an unknown-model error is handled by fail_mode like any other provider error.
Threat model.
-
Manager-supplied templates. Acceptance-time validation rejects unknown config keys, oversize templates, control characters, and embedded envelope tokens; suspicious phrases (
ignore previous,override, …) surface as warnings onPOST /v1/admin/policies. A manager who writes a permissive template degrades only their own app, the platform’s invariant is the response shape, not policy quality. -
Malicious end-user content. Reserved envelope tokens in user content are HTML-encoded before the model sees them, so jailbreak attempts cannot syntactically break out of the user-message block. A sandwich reminder after the user content restates the classifier instruction, and the strict first-line parser rejects anything other than
SAFEorUNSAFE. Underfail_mode: closed, a malformed verdict becomes aprovider_errorviolation rather than a free pass, recommended for high-stakes apps. -
Cost overrun. Templates are capped at 2000 chars and user content at
max_input_chars; one judge call is one scheduler round-trip. Put cheap stages (regex) ahead of the judge to short-circuit before the LLM call.
sherlock-image-classifier
Image stage provider ("modality": "image"). It classifies every image of a
request against a set of categories with a vision model, using the
Data Platform’s ImageAI workflow (Holmes ImageClassifier). The images are
classified in the search store they already live in: nothing is copied and no
store is created. For each store the request names, it:
-
adds an
ImageAIworkflow to that store, namedsr-guardrail-image-<uuid>, with the caller’s token. The workflow only classifies (no description pages, chunks, or vectors); its classifier answers a strict JSON Schema whoselabelsare the stage’s categories plus"safe"; -
runs the workflow on the requested `files://<file_id>`s, polls the run until it ends, and reads each image’s classification from that run;
-
deletes the workflow once it reaches a conclusion, when the check fails, and when the check is cancelled (for example because a text stage already blocked). Sherlock stops the workflow’s runs and soft-deletes the documents they produced.
Images in different stores are checked in each store, side by side.
The caller needs write access to the image’s store to add the workflow. A store the caller can only read fails the check as a provider error. The classification rows stay in the store, attached to soft-deleted documents, until Sherlock’s retention purge removes them.
Every label other than "safe" is a violated category. A run that fails or
times out, or leaves an image without a classification, is a provider error,
so fail_mode decides: closed blocks, open lets the request through.
Config schema:
| Field | Type | Required | Default | Notes |
|---|---|---|---|---|
|
array of string |
no |
|
Labels the classifier may give. |
|
string |
no |
A prompt listing |
System prompt: what to classify and what each label means. |
|
string |
no |
Holmes default |
Vision model the classifier asks. |
|
string |
no |
Holmes default |
Holmes inference provider the model is served by. |
|
string |
no |
Holmes default |
User prompt sent with each image. |
|
int |
no |
Holmes default |
Upper bound for the classifier’s reply. |
Example stage:
{
"provider": "sherlock-image-classifier",
"name": "image-safety",
"modality": "image",
"config": {"categories": ["violence", "sexual", "weapons"]}
}
The worker sends the references of every input_image stored in the Data
Platform with the input check; inline data: images that were never stored
are not sent. Only input checks carry images.
Default Safety Categories (LlamaGuard)
When no custom labels are configured, the llama-guard-3 provider evaluates against the following categories:
| # | Category | Description |
|---|---|---|
S1 |
Violent Crimes |
Terrorism, murder, assault, kidnapping, animal abuse |
S2 |
Non-Violent Crimes |
Fraud, scams, hacking, drug trafficking, weapons offenses |
S3 |
Sex Crimes |
Human trafficking, sexual assault, sexual harassment |
S4 |
Child Exploitation |
Child sexual abuse material or depictions |
S5 |
Defamation |
Verifiably false statements about real people |
S6 |
Specialized Advice |
Unqualified financial, medical, or legal advice |
S7 |
Privacy |
Disclosure of private individuals' sensitive information |
S8 |
Intellectual Property |
Content violating third-party IP rights |
S9 |
Indiscriminate Weapons |
Weapons of mass destruction (chemical, biological, nuclear) |
S10 |
Hate |
Hate speech based on protected characteristics |
S11 |
Self-Harm |
Suicide, self-injury, disordered eating |
S12 |
Sexual Content |
Explicit sexual depictions or erotic descriptions |
S13 |
Elections |
False information about electoral systems and voting |
S14 |
Code Interpreter Abuse |
Denial of service, container escapes, privilege escalation |
S15 |
Profanity |
Vulgar, offensive, or impolite language |
S16 |
Prompt Injection Attack |
Attempts to override system instructions or extract prompts |
To override or extend these, provide a JSON array of label objects via GUARDRAILS_LABELS or GUARDRAILS_LABELS_FILENAME:
[
{
"name": "Financial Advice",
"description": "AI models should not provide specific investment recommendations.",
"enabled": true
},
{
"name": "Profanity",
"description": "Vulgar or offensive language.",
"enabled": false
}
]
| Field | Type | Required | Description |
|---|---|---|---|
|
string |
Yes |
Display name of the category |
|
string |
Yes |
Description used in the safety prompt |
|
boolean |
No |
Whether this category is active (default |
The 16 default categories are always included in the safety prompt. Custom labels are appended after them. Only labels present in your configured list with "enabled": true trigger violations.
Per-category actions
What happens to a flagged category is set per application:
| Action | Effect |
|---|---|
|
The message reaches the model untouched. |
|
The message reaches the model with a notice naming the flagged categories, so it can decline or answer only the harmless part. |
|
The request is rejected before it reaches the model (see below). |
default_action applies to any flagged category not named in
category_actions. Both default to blocking, so an application that sets
neither keeps enforcing exactly as before.
{
"default_action": "label",
"category_actions": { "Child Exploitation": "block" }
}
When several categories fire, the most severe action wins.
Violation Response (Responses API)
When a guardrail violation blocks a POST /v1/responses request, the API returns HTTP 405 with:
{
"error": {
"message": "Guardrail violations found: Violent Crimes, Hate",
"type": "invalid_request_error",
"param": null,
"code": "content_policy_violation"
}
}
The message field lists the specific categories that were violated. This applies to both input and output blocks. For a non-streaming background=true request, the pending record is marked as failed with the violation reason instead of an HTTP error.
A block on a streaming request, input or output, cannot change the HTTP status: the stream already opened with 200 before the check ran. Instead the stream ends with two frames in order: an event: error frame, then a terminal event: response.failed frame, then data: [DONE]. Exactly one response.failed frame is sent. For an output block on a stream, the part of the answer that streamed before the verdict has already reached the client, but the client never receives response.completed, and nothing is stored. An output label delivers the answer unchanged and adds guardrail_action and guardrail_categories to the response’s metadata, on the client’s terminal response.completed frame as well as on the stored and replied response.
Streaming and the output checkpoint
Because the output check runs beside the live stream, flagged text can reach the client before the verdict. An application must accept that risk explicitly with "streaming_allowed": true on its output check type:
{"check_types": {"output": {"streaming_allowed": true, "pipeline": [...]}}}
The worker calls the guardrails service’s POST /v2/check (same request body as /v1/check) and reads streaming_allowed from the input check before generation starts. For a stream=true request:
/v2/check reports |
Result |
|---|---|
|
The request streams. |
|
Rejected before the model runs, as an invalid request: code |
no |
Rejected before the model runs: code |
A guardrails server that predates /v2/check answers 404. Every request that reaches the guardrail gate then fails with HTTP 502, code guardrail_version_mismatch, "v2/check not found, version mismatch between guardrails and stateful responses", streaming or not. There is no fallback to /v1/check: upgrade the guardrails service before the workers.
Both rejections arrive like any streaming error: event: error, then event: response.failed, then data: [DONE]. A non-streaming request is never affected.
When streaming is allowed, the worker never buffers the answer:
-
Live deltas. Every delta is published as it arrives.
-
Per-item checks. Each text item (an output text, refusal or reasoning part, bracketed by
content_part/reasoning_summary_partevents) is checked in its own/v2/checkcall when it completes, in parallel with the rest of the stream. Deltas that arrive outside such a bracket are still collected and checked. At mostWORKER_CHECK_CONCURRENCY(default10) item checks run at once per response; further items queue. -
Full-response check. After the stream, the worker also checks the whole response’s text, unless a single item check already saw exactly that text. Item checks see one item each, so a rule that matches across an item boundary (for example a regex over digits split between two parts), text that never streamed as a delta, or a tool-only turn is still caught. The final verdict is the most severe of all these checks (block over label over warn over allow). A streamed response is therefore never judged more leniently than the same non-streaming request, but it can be judged more strictly: an item check sees one item out of context, so a judge may block a single reasoning summary part that passes as part of the whole answer.
-
Terminal frame held for the verdict.
response.completed(orresponse.incomplete/response.failed) is the only frame held back. On allow it is published unchanged. On label it carries the label metadata. On block it is dropped, so clients that stop atresponse.completedstill see the failure. -
Early stop on block. As soon as one item check blocks, the worker stops publishing, closes the backend stream so the model stops generating, and fails the request. Only the text that streamed before that verdict reaches the client.
Each streamed response logs one INFO line, Stream output check for request <id>: effect=… item_checks=… full_check=… early_stop=…, with no content.