Guardrails

The Responses API includes built-in content safety checks. Every request is screened before reaching the LLM backend. What happens on a violation depends on the policy: a category can be blocked, labelled (the request proceeds with a notice naming the categories), or allowed.

Three provider types ship out of the box:

  • llama-guard-3, content-safety classifier based on LlamaGuard 3. LLM-based, ~500ms–2s per check.

  • regex, fast pattern matcher for deterministic rules (PII patterns, blocklists, malicious URLs). Microseconds per check.

  • llm-judge, LLM-based judge against a free-form natural-language policy the application manager supplies. See Providers below.

A fourth, openrouter-llama-guard, is available but off by default. It runs the same LlamaGuard classification against a model hosted on OpenRouter, as a temporary bridge while our own LlamaGuard is unavailable.

Content checked by this provider leaves the cluster (see Data Flows). It is registered only when GUARDRAILS_OPENROUTER_API_KEY is set, and reached only by a stage that names it, so it stays off unless an operator opts in twice. Naming it in a policy without the key set makes every check fail closed.

Like llama-guard-3 it ignores stage.config and takes its categories from global configuration.

Images have a provider of their own, sherlock-image-classifier, registered by the guardrails service (never the api) when GUARDRAILS_IMAGE_SHERLOCK_BASE_URL is set on it. It classifies a request’s images with a vision model through the Data Platform (Sherlock) and runs in a task of its own, beside the text stages. See sherlock-image-classifier below.

Providers are arranged into a pipeline per checkpoint. Only two checkpoints are valid: input and output. Any other key, tool_output included, is rejected at policy load or at request validation, not silently ignored. The pipeline runs stages sequentially and short-circuits on the first stage that returns a violation, so cheap regex checks can fail fast before the expensive LLM stage even runs.

input checks the user’s message before it reaches the model. output checks the model’s own answer once generation finishes, before the caller sees it. Both Llama Guard providers frame an output check as judging an Agent turn, not the user’s message. See Framing an output check below.

Guardrails are evaluated eagerly for input, before the model generates anything. For output, a non-streaming response is checked once, after generation completes; a streaming response is checked item by item beside the live stream (see Streaming and the output checkpoint). This applies to synchronous, streaming, and background=true requests.


How It Works

  1. The service resolves the application identity from the verified access token and posts to POST /v1/check with that application_id, a check_type ("input" or "output"), and the content to check.

  2. GuardrailService looks up the per-application policy and walks the pipeline configured for the requested checkpoint.

  3. For each enabled stage, the registry dispatches to the provider named in stage.provider (e.g. regex, llama-guard-3, llm-judge), passing the stage-specific config.

  4. Violations are collected across every stage and the most severe action wins, so a category one stage only labels cannot mask one a later stage blocks. The pipeline stops early only once something is blocked, since nothing outranks it.

  5. If all stages pass, the request proceeds (an input check lets the request reach the model; an output check releases the answer to the caller).

The endpoint is also callable directly by other services that need to evaluate content against an application policy.

Framing an output check

An output check scores the model’s own answer, not the user’s message. Send context: {"checkpoint": "output", "user_prompt": "<the original user message>"} on the request to get the full framing: the Llama Guard providers then classify the content as the last turn in a User → Agent exchange. Without user_prompt, the content is still classified under the agent-response template, but as the only turn, with no preceding User turn. Both providers still score it as an answer, never as if the user had written it. /v1/responses builds this context automatically from the original prompt; a direct /v1/check caller wanting the full framing must pass user_prompt itself.


Standalone Check Endpoint

The guardrail pipeline is exposed directly at POST /v1/check for callers that want to screen content without going through /v1/responses. It uses the same per-application policy lookup and bearer auth as the rest of the API.

Example:

curl -X POST "$BASE_URL/v1/check" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "application_id": "test-app",
    "check_type": "input",
    "input": "What is the capital of France?"
  }'

Request:

{
  "application_id": "legal-app",
  "check_type": "output",
  "input": "Here is how you write a contract for...",
  "context": {
    "checkpoint": "output",
    "user_prompt": "How do I write a contract for..."
  }
}
Field Type Description

application_id

string | null

Application whose policy to apply. Omit (or send null) to fall back to the default policy block.

check_type

string

Which checkpoint to run. Only "input" and "output" are valid. Any other value, "tool_output" included, is rejected with a 400.

input

string

Content to check. May be empty when images is set; the text stages are then skipped.

images

array (optional)

Stored images to check: [{"search_store_id": "…​", "file_id": "…​"}]. Both ids must be plain Data Platform ids (1–128 characters of A-Z, a-z, 0-9, _, -); anything else is refused with 400, and raw image data (a data: URL or base64) with the message raw image data is not accepted: upload the image first and send its reference. Only references travel; the guardrails service fetches each image from its store with the caller’s token. Checked by the pipeline’s image stages (modality: "image") only. Omitted or empty, no image stage runs. Images sent to a checkpoint with no enabled image stage are a failure that fail_mode decides: closed blocks with a provider_error violation (reason: "images received but no image stage is configured"), open lets the text verdict stand. This holds when the checkpoint has no pipeline at all and if_unconfigured is "allow": closed returns that block with evaluated: true instead of the unevaluated placeholder.

unreferenced_image_count

integer (optional)

How many of the input’s images cannot be named in images: inline data: images that were never stored (image ingestion is not configured) or references that do not resolve. No image stage can find them, so any count above zero always blocks, whatever the fail_mode and even when image stages are configured or the checkpoint has no pipeline: the check returns a provider_error violation (reason: "images received that were never stored, so no image stage can check them"). Default 0. The worker fills it in; omitted when zero.

context

object (optional)

{"checkpoint": "…​", "user_prompt": "…​"}. checkpoint should match check_type. user_prompt is the original user message; set it on an output check to get the full Llama Guard agent-response framing (see Framing an output check). Omit context entirely for a plain input check.

if_unconfigured

string (optional)

"error" or "allow". Default "error": an unconfigured check_type returns 422. Set "allow" to ask "is this checkpoint configured?" without a 422; the response comes back with evaluated: false and safe/violations/action as placeholders.

Response (safe):

{
  "safe": true,
  "violations": [],
  "action": "allow",
  "evaluated": true,
  "configured_checkpoints": ["input", "output"],
  "streaming_allowed": false
}

Response (unsafe):

{
  "safe": false,
  "violations": [
    {
      "category": "Violent Crimes",
      "provider": "llama-guard-3",
      "stage": "content-safety",
      "step": 0
    }
  ],
  "action": "block",
  "evaluated": true,
  "configured_checkpoints": ["input", "output"],
  "streaming_allowed": false
}

Response fields:

Field Type Description

safe

boolean

true when no violation was found.

violations

array

Empty when safe is true. See violation fields below.

action

string

"allow", "label", or "block". Defaults to "block" so a caller that does not yet read this field keeps enforcing.

evaluated

boolean

Default true. false only when if_unconfigured: "allow" short-circuited an unconfigured checkpoint: no pipeline ran, and safe/violations/action are placeholders, not a real result.

configured_checkpoints

array of string | null

The application’s configured checkpoint keys, sorted (for example ["input", "output"]). Lets a caller learn what else is configured from a call it already made. A check type whose pipeline is empty or has only disabled stages is left out, because it checks nothing.

streaming_allowed

boolean

/v2/check only (served by the standalone guardrails service, and by the api when it runs checks in-process; /v1/check never sends it). Whether the application accepts the streaming risk (see Streaming). true when the output pipeline sets streaming_allowed: true, or when no output pipeline has an enabled stage. false when an output pipeline runs without the flag. Always sent by /v2/check.

Violation fields:

Field Type Description

category

string

The violated category (e.g. "Violent Crimes", "PII"). For provider failures under fail_mode: closed, the sentinel "provider_error".

provider

string

Registry key of the stage that produced the violation (e.g. "llama-guard-3", "regex").

stage

string

stage.name from the configured pipeline, useful for log correlation.

step

integer

Zero-indexed position in the configured pipeline list. Disabled stages do not change step. -1 when no stage produced it (images with no image stage configured, or images that were never stored).

Attribution fields let callers running multi-provider pipelines tell which stage produced the block, for example, distinguishing a regex PII match from a llama-guard-3 content-safety classification when both are configured for the same checkpoint.

Status codes:

Status When

200

Check completed (inspect safe and violations)

400

Request body failed schema validation, including a check_type that is not "input" or "output"

404

application_id was supplied but no policy exists for it (under fail_mode: closed)

422

The application has no pipeline configured for the requested check_type


Application Identity

The application identity that selects a per-application policy comes from the verified access token. Callers cannot set or override it.

Deployment Source of the application identity

IAM

A claim on the access token.

Platform (JWT)

The Token Vending Machine (TVM) mints the token. The service derives the identity from the workload identity the TVM binds into it.

When the token carries no application identity, the lookup resolves to application_id=null and the service routes it to the configured default policy block.

An application explicitly named "default" is a regular entry and does not collide with the default fallback block. The null case is the only way to reach the fallback, unknown application IDs are rejected rather than silently inheriting the default policy.


Configuration

Guardrails are controlled via environment variables with the GUARDRAILS_ prefix:

Variable Pod Default Description

GUARDRAILS_DISABLED

API

false

Disables guardrails entirely. /v1/check still responds (returns safe=true). In distributed deployments, setting this on the worker alone has no effect because the worker only forwards requests to the API’s /v1/check endpoint.

GUARDRAILS_MODEL_NAME

API

meta-llama/Llama-Guard-3-8B

Model used by the llama-guard-3 provider. Currently only meta-llama/Llama-Guard-3-8B is supported.

GUARDRAILS_OPENROUTER_API_KEY

API

-

Enables the openrouter-llama-guard provider. Unset, it is not registered. Content checked by it leaves the cluster.

GUARDRAILS_OPENROUTER_BASE_URL

API

https://openrouter.ai/api/v1

Endpoint for the openrouter-llama-guard provider

GUARDRAILS_OPENROUTER_MODEL

API

meta-llama/llama-guard-4-12b

Model asked by the openrouter-llama-guard provider

GUARDRAILS_OPENROUTER_TIMEOUT_SECONDS

API

30.0

Request timeout for the openrouter-llama-guard provider

GUARDRAILS_OPENROUTER_MAX_CONCURRENCY

API

8

Most in-flight requests to the openrouter-llama-guard provider per process. The provider also sits behind the CIRCUIT_BREAKER_* settings

GUARDRAILS_LABELS

API

-

Inline JSON array of LlamaGuard category definitions (highest priority)

GUARDRAILS_LABELS_FILENAME

API

-

Path to a JSON file with LlamaGuard category definitions

GUARDRAILS_POLICY_JSON

API

-

Inline JSON with per-application policy config (takes priority over GUARDRAILS_POLICY_FILE)

GUARDRAILS_POLICY_FILE

API

-

Path to a JSON file with per-application policy config

GUARDRAILS_ADMIN_API_ENABLED

API

false

Expose the /v1/admin/policies CRUD endpoints. Off by default; requires Postgres (the standard database settings).

GUARDRAILS_IMAGE_SHERLOCK_BASE_URL

Guardrails

-

Enables the sherlock-image-classifier provider against this Sherlock URL. Unset, it is not registered and an image stage is a provider error.

GUARDRAILS_IMAGE_RUN_TIMEOUT_SECONDS

Guardrails

120.0

Longest an image stage may take end to end: creating the classification workflow, running it and reading the labels. The workflow is deleted afterwards, off the request path. Image stages run one after another, so keep this value times the number of image stages below GUARDRAILS_PROXY_TIMEOUT_SECONDS and the worker’s HTTP_READ_TIMEOUT_SEC, when set.

GUARDRAILS_IMAGE_POLL_INTERVAL_SECONDS

Guardrails

1.0

Interval between polls of the classification run.

GUARDRAILS_CHECK_URL

Worker

http://localhost:8000

Base URL for the /v1/check endpoint the worker calls

Policy resolution priority: Postgres (if database is configured) > GUARDRAILS_POLICY_JSON > GUARDRAILS_POLICY_FILE > generated default policy from GUARDRAILS_LABELS.

The DB layer is highest-precedence so a runtime override beats a baked-in policy. When the DB has no row for an application_id (or the row is disabled), the chain falls through to the next layer, existing deployments without a DB row keep their current behaviour.

Helm deployments: ConfigMap-mounted policy

For Helm deployments, the recommended way to ship a per-application policy is the guardrails.policy value. The chart renders it into a ConfigMap and mounts it at /etc/guardrails/policy.json on the API pod (GUARDRAILS_POLICY_FILE is wired automatically):

# values.yaml
guardrails:
  policy:
    default:
      fail_mode: closed
      check_types:
        input:
          pipeline:
            - provider: llama-guard-3
              name: content-safety
              config: {}
    applications:
      legal-app:
        fail_mode: closed
        check_types:
          input:
            pipeline:
              - provider: regex
                name: pii-patterns
                config:
                  patterns:
                    - { name: steuer_id, pattern: '\b\d{11}\b', category: PII }
              - provider: llama-guard-3
                name: content-safety
                config: {}

Why use the ConfigMap path:

  • The policy is reviewable as native YAML in values.yaml (no JSON-in-string-in-env-var escape gymnastics).

  • A checksum/configmap-guardrails-policy annotation rolls the API pod whenever the policy changes.

  • ConfigMaps hold up to ~1 MiB; env-var size limits don’t bite.

  • Out-of-band edits to the ConfigMap (kubectl edit configmap …-api-guardrails-policy) do not automatically restart the API, run kubectl rollout restart deployment/<api-deployment> to pick them up. Hot-reload via file-watcher is not yet implemented.

GUARDRAILS_POLICY_JSON remains supported as an escape hatch (smoke tests, single-pod overrides). When both are set, the inline env var wins.

Image stages. The chart sets GUARDRAILS_IMAGE_SHERLOCK_BASE_URL on the guardrails pod (never on the api) from guardrails.imageClassifier.sherlockBaseUrl, falling back to skills.sherlockBaseUrl, whenever one of them is set. It does not depend on guardrails.policy: image stages from the Postgres/admin API or GUARDRAILS_POLICY_JSON need the provider too. A stage then reaches the sherlock-image-classifier with modality: image:

guardrails:
  imageClassifier:
    sherlockBaseUrl: https://sherlock.example.com
  policy:
    applications:
      legal-app:
        fail_mode: closed
        check_types:
          input:
            pipeline:
              - provider: sherlock-image-classifier
                name: image-safety
                modality: image
                config:
                  provider: pharia_next
                  model: moonshotai/kimi-k2.6

Set config.provider and config.model to a vision model the platform’s inference serves: Holmes' default classifier model is not served everywhere.

Postgres-backed policy + admin API

For installations that already use Postgres for history/conversations/pending requests, policies can additionally be configured at runtime through HTTP, no ConfigMap edits, no pod rollouts.

Turn it on by setting GUARDRAILS_ADMIN_API_ENABLED=true. The application_policies table is created by Alembic revision 008_application_policies (runs automatically on startup with the rest of the migrations).

The admin endpoints are scoped to admin principals only (User.is_admin == True); the feature flag is the gate of last resort, so leave it off unless your auth setup is hardened. Endpoints:

Method Path Description

GET

/v1/admin/policies

List all stored policies

GET

/v1/admin/policies/{application_id}

Fetch one, 404 if missing

PUT

/v1/admin/policies/{application_id}

Upsert; body is an ApplicationPolicy JSON

DELETE

/v1/admin/policies/{application_id}

Delete; 204 on success, 404 if missing

Example: upsert a policy for legal-app:

curl -X PUT \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://your-api/v1/admin/policies/legal-app \
  -d '{
    "application_id": "legal-app",
    "fail_mode": "closed",
    "check_types": {
      "input": {
        "pipeline": [
          {"provider": "regex", "name": "pii", "config": {"patterns": [{"name": "steuer_id", "pattern": "\\b\\d{11}\\b", "category": "PII"}]}}
        ]
      }
    }
  }'

The path application_id is authoritative; if the JSON body carries a different application_id, the path value wins. IDs are capped at 253 characters (DNS-1123), longer values return 422.

GUARDRAILS_POLICY_JSON / GUARDRAILS_POLICY_FILE remain useful for bootstrap: ship an initial policy via env var, then migrate it into the DB through the admin API. Both env-var and file paths still serve as fallbacks for any application_id not present in the DB.

app-manager-gitops deployments

  • Platform only

app-manager-gitops descriptors set envVars.GUARDRAILS_POLICY_JSON to the inline JSON policy:

{
  "config": {
    "envVars": {
      "GUARDRAILS_POLICY_JSON": "{\"default\":{...},\"applications\":{...}}"
    }
  }
}

The platform-manager API supports config.volumes.configMap[] for mounting an existing ConfigMap, but exposes no endpoint for creating one, the only resource type currently provisionable via /tenants/{tenant}/resources is redis. Until that gap closes, gitops deployments stay on the inline env var.


Per-Application Policy

The policy JSON keys applications by ID and defines a pipeline per check type:

{
  "default": {
    "fail_mode": "closed",
    "check_types": {
      "input": {
        "pipeline": [
          {
            "provider": "llama-guard-3",
            "name": "content-safety",
            "enabled": true,
            "config": {}
          }
        ]
      }
    }
  },
  "applications": {
    "legal-app": {
      "fail_mode": "closed",
      "check_types": {
        "input": {
          "pipeline": [
            {
              "provider": "regex",
              "name": "pii-patterns",
              "config": {
                "patterns": [
                  {
                    "name": "steuer_id",
                    "pattern": "\\b\\d{11}\\b",
                    "category": "PII"
                  }
                ]
              }
            },
            {
              "provider": "llama-guard-3",
              "name": "content-safety",
              "config": {}
            }
          ]
        }
      }
    }
  }
}

Top-level keys:

Key Description

default

Fallback policy block, reachable only when the request omits application_id.

applications

Map of application_id → policy.

Per-application fields:

Field Type Description

fail_mode

"closed" | "open"

What to do on provider error. closed = reject (fail-secure); open = allow + log. Default "closed".

check_types

object

Map of checkpoint name to its pipeline configuration. Only "input" and "output" are valid keys. Any other key, "tool_output" included, fails policy load.

check_types.output.streaming_allowed

boolean

Accepts the streaming risk for this application: stream=true requests are served while the output check runs beside the live stream. Unset counts as false, so streaming requests are rejected. Only valid on output; setting it on input fails policy load.

Per-stage fields:

Field Type Description

provider

string

Provider registry key, e.g. "llama-guard-3", "regex", "llm-judge", or (for an image stage) "sherlock-image-classifier".

name

string

Unique stage name within the pipeline. Used in logs.

enabled

boolean

Toggle a stage off without removing it from the config. Default true.

modality

"text" | "image"

What the stage checks. Default "text": the request’s input, in pipeline order. "image": the request’s images, in a task that runs beside the text stages. A block found by either task cancels the other, and their violations merge into one verdict. A request with images and no enabled image stage fails by fail_mode.

config

object

Provider-specific configuration. See below.


Providers

llama-guard-3

LLM-based content safety classifier. The config block is currently ignored, LlamaGuard category selection is configured globally via GUARDRAILS_LABELS rather than per-stage.

On an input check it classifies the content as the user’s message. On an output check it classifies the content as an Agent turn, using Llama Guard 3’s agent-response template, never the user-message template. See Framing an output check.

regex

Pattern-based provider. Config schema:

{
  "patterns": [
    { "name": "steuer_id",     "pattern": "\\b\\d{11}\\b",                "category": "PII" },
    { "name": "javascript_url", "pattern": "javascript:",                  "category": "MaliciousURL" }
  ]
}

Each pattern has a name (used in logs only, pattern names and category names are logged, but never the matched substring), a pattern (Python regular expression), and a category (returned as the violation category). Multiple patterns sharing a category produce a single deduplicated violation.

Invalid regex patterns raise at check time and are handled by fail_mode like any other provider error.

llm-judge

LLM-based judge that applies a natural-language policy supplied by the application manager. Each judge stage takes a template (the policy text) and a model (an operator-approved Responses-API model name). The provider wraps both in a server-controlled prompt envelope and parses a strict SAFE / UNSAFE first-line verdict.

Config schema:

Field Type Required Default Notes

model

string

yes

-

Any model the inference backend can serve. The platform does not gatekeep model selection; an unknown model surfaces as a backend error at check time.

template

string

yes

-

Free-form NL policy. Length 20..2000 chars. Cannot contain reserved envelope tokens or control characters.

violation_category

string

no

"Custom"

Returned as the Violation.category on UNSAFE. Matches ^[A-Za-z0-9 _-]{1,64}$.

max_input_chars

int

no

8000

Per-stage user-content cap, clamped against GUARDRAILS_JUDGE_MAX_INPUT_CHARS.

Example stage:

{
  "provider": "llm-judge",
  "name": "stay-on-topic",
  "config": {
    "model": "judge-1",
    "template": "Reject any message not related to legal advice in Germany.",
    "violation_category": "Off-Topic"
  }
}

Templates are open-ended natural-language strings, the same schema covers stay-on-topic checks, natural-language denylists ("reject any message that asks about competitors"), tone policies ("reject aggressive or threatening messages"), and anything else expressible as a one-paragraph policy. The provider does not know which kind of check the manager wrote.

Operator setup. One env var controls the per-stage user-content cap:

  • GUARDRAILS_JUDGE_MAX_INPUT_CHARS, upper bound on per-stage max_input_chars (default 8000).

There is no operator-side allowlist for judge models. Application managers may pick any model the backend can serve; an unknown-model error is handled by fail_mode like any other provider error.

Threat model.

  • Manager-supplied templates. Acceptance-time validation rejects unknown config keys, oversize templates, control characters, and embedded envelope tokens; suspicious phrases (ignore previous, override, …) surface as warnings on POST /v1/admin/policies. A manager who writes a permissive template degrades only their own app, the platform’s invariant is the response shape, not policy quality.

  • Malicious end-user content. Reserved envelope tokens in user content are HTML-encoded before the model sees them, so jailbreak attempts cannot syntactically break out of the user-message block. A sandwich reminder after the user content restates the classifier instruction, and the strict first-line parser rejects anything other than SAFE or UNSAFE. Under fail_mode: closed, a malformed verdict becomes a provider_error violation rather than a free pass, recommended for high-stakes apps.

  • Cost overrun. Templates are capped at 2000 chars and user content at max_input_chars; one judge call is one scheduler round-trip. Put cheap stages (regex) ahead of the judge to short-circuit before the LLM call.

sherlock-image-classifier

Image stage provider ("modality": "image"). It classifies every image of a request against a set of categories with a vision model, using the Data Platform’s ImageAI workflow (Holmes ImageClassifier). The images are classified in the search store they already live in: nothing is copied and no store is created. For each store the request names, it:

  1. adds an ImageAI workflow to that store, named sr-guardrail-image-<uuid>, with the caller’s token. The workflow only classifies (no description pages, chunks, or vectors); its classifier answers a strict JSON Schema whose labels are the stage’s categories plus "safe";

  2. runs the workflow on the requested `files://<file_id>`s, polls the run until it ends, and reads each image’s classification from that run;

  3. deletes the workflow once it reaches a conclusion, when the check fails, and when the check is cancelled (for example because a text stage already blocked). Sherlock stops the workflow’s runs and soft-deletes the documents they produced.

Images in different stores are checked in each store, side by side.

The caller needs write access to the image’s store to add the workflow. A store the caller can only read fails the check as a provider error. The classification rows stay in the store, attached to soft-deleted documents, until Sherlock’s retention purge removes them.

Every label other than "safe" is a violated category. A run that fails or times out, or leaves an image without a classification, is a provider error, so fail_mode decides: closed blocks, open lets the request through.

Config schema:

Field Type Required Default Notes

categories

array of string

no

["violence", "sexual", "hate", "self_harm"]

Labels the classifier may give. "safe" is reserved. Name them as category_actions keys to set their action.

system

string

no

A prompt listing categories

System prompt: what to classify and what each label means.

model

string

no

Holmes default

Vision model the classifier asks.

provider

string

no

Holmes default

Holmes inference provider the model is served by.

prompt

string

no

Holmes default

User prompt sent with each image.

max_tokens

int

no

Holmes default

Upper bound for the classifier’s reply.

Example stage:

{
  "provider": "sherlock-image-classifier",
  "name": "image-safety",
  "modality": "image",
  "config": {"categories": ["violence", "sexual", "weapons"]}
}

The worker sends the references of every input_image stored in the Data Platform with the input check; inline data: images that were never stored are not sent. Only input checks carry images.


Default Safety Categories (LlamaGuard)

When no custom labels are configured, the llama-guard-3 provider evaluates against the following categories:

# Category Description

S1

Violent Crimes

Terrorism, murder, assault, kidnapping, animal abuse

S2

Non-Violent Crimes

Fraud, scams, hacking, drug trafficking, weapons offenses

S3

Sex Crimes

Human trafficking, sexual assault, sexual harassment

S4

Child Exploitation

Child sexual abuse material or depictions

S5

Defamation

Verifiably false statements about real people

S6

Specialized Advice

Unqualified financial, medical, or legal advice

S7

Privacy

Disclosure of private individuals' sensitive information

S8

Intellectual Property

Content violating third-party IP rights

S9

Indiscriminate Weapons

Weapons of mass destruction (chemical, biological, nuclear)

S10

Hate

Hate speech based on protected characteristics

S11

Self-Harm

Suicide, self-injury, disordered eating

S12

Sexual Content

Explicit sexual depictions or erotic descriptions

S13

Elections

False information about electoral systems and voting

S14

Code Interpreter Abuse

Denial of service, container escapes, privilege escalation

S15

Profanity

Vulgar, offensive, or impolite language

S16

Prompt Injection Attack

Attempts to override system instructions or extract prompts

To override or extend these, provide a JSON array of label objects via GUARDRAILS_LABELS or GUARDRAILS_LABELS_FILENAME:

[
  {
    "name": "Financial Advice",
    "description": "AI models should not provide specific investment recommendations.",
    "enabled": true
  },
  {
    "name": "Profanity",
    "description": "Vulgar or offensive language.",
    "enabled": false
  }
]
Field Type Required Description

name

string

Yes

Display name of the category

description

string

Yes

Description used in the safety prompt

enabled

boolean

No

Whether this category is active (default true)

The 16 default categories are always included in the safety prompt. Custom labels are appended after them. Only labels present in your configured list with "enabled": true trigger violations.


Per-category actions

What happens to a flagged category is set per application:

Action Effect

allow

The message reaches the model untouched.

label

The message reaches the model with a notice naming the flagged categories, so it can decline or answer only the harmless part.

block

The request is rejected before it reaches the model (see below).

default_action applies to any flagged category not named in category_actions. Both default to blocking, so an application that sets neither keeps enforcing exactly as before.

{
  "default_action": "label",
  "category_actions": { "Child Exploitation": "block" }
}

When several categories fire, the most severe action wins.

Violation Response (Responses API)

When a guardrail violation blocks a POST /v1/responses request, the API returns HTTP 405 with:

{
  "error": {
    "message": "Guardrail violations found: Violent Crimes, Hate",
    "type": "invalid_request_error",
    "param": null,
    "code": "content_policy_violation"
  }
}

The message field lists the specific categories that were violated. This applies to both input and output blocks. For a non-streaming background=true request, the pending record is marked as failed with the violation reason instead of an HTTP error.

A block on a streaming request, input or output, cannot change the HTTP status: the stream already opened with 200 before the check ran. Instead the stream ends with two frames in order: an event: error frame, then a terminal event: response.failed frame, then data: [DONE]. Exactly one response.failed frame is sent. For an output block on a stream, the part of the answer that streamed before the verdict has already reached the client, but the client never receives response.completed, and nothing is stored. An output label delivers the answer unchanged and adds guardrail_action and guardrail_categories to the response’s metadata, on the client’s terminal response.completed frame as well as on the stored and replied response.

Streaming and the output checkpoint

Because the output check runs beside the live stream, flagged text can reach the client before the verdict. An application must accept that risk explicitly with "streaming_allowed": true on its output check type:

{"check_types": {"output": {"streaming_allowed": true, "pipeline": [...]}}}

The worker calls the guardrails service’s POST /v2/check (same request body as /v1/check) and reads streaming_allowed from the input check before generation starts. For a stream=true request:

/v2/check reports Result

streaming_allowed: true

The request streams.

streaming_allowed: false

Rejected before the model runs, as an invalid request: code streaming_not_allowed, param stream. Retry with stream=false.

no streaming_allowed field

Rejected before the model runs: code guardrail_version_mismatch, "mismatch versions between guardrail and responses servers". Upgrade the guardrails server.

A guardrails server that predates /v2/check answers 404. Every request that reaches the guardrail gate then fails with HTTP 502, code guardrail_version_mismatch, "v2/check not found, version mismatch between guardrails and stateful responses", streaming or not. There is no fallback to /v1/check: upgrade the guardrails service before the workers.

Both rejections arrive like any streaming error: event: error, then event: response.failed, then data: [DONE]. A non-streaming request is never affected.

When streaming is allowed, the worker never buffers the answer:

  • Live deltas. Every delta is published as it arrives.

  • Per-item checks. Each text item (an output text, refusal or reasoning part, bracketed by content_part / reasoning_summary_part events) is checked in its own /v2/check call when it completes, in parallel with the rest of the stream. Deltas that arrive outside such a bracket are still collected and checked. At most WORKER_CHECK_CONCURRENCY (default 10) item checks run at once per response; further items queue.

  • Full-response check. After the stream, the worker also checks the whole response’s text, unless a single item check already saw exactly that text. Item checks see one item each, so a rule that matches across an item boundary (for example a regex over digits split between two parts), text that never streamed as a delta, or a tool-only turn is still caught. The final verdict is the most severe of all these checks (block over label over warn over allow). A streamed response is therefore never judged more leniently than the same non-streaming request, but it can be judged more strictly: an item check sees one item out of context, so a judge may block a single reasoning summary part that passes as part of the whole answer.

  • Terminal frame held for the verdict. response.completed (or response.incomplete / response.failed) is the only frame held back. On allow it is published unchanged. On label it carries the label metadata. On block it is dropped, so clients that stop at response.completed still see the failure.

  • Early stop on block. As soon as one item check blocks, the worker stops publishing, closes the backend stream so the model stops generating, and fails the request. Only the text that streamed before that verdict reaches the client.

Each streamed response logs one INFO line, Stream output check for request <id>: effect=… item_checks=… full_check=… early_stop=…, with no content.