Context Compaction

In this article:

Long conversations eventually exceed a model’s context window. The Responses API tracks how much of each model’s window a conversation is using and lets operators configure a threshold at which older turns get summarised into a single "compaction" entry, freeing room for the conversation to continue.

Status: this release ships the configuration only

The runtime features that consume this config (an explicit /v1/responses/compact endpoint, post-response auto-compaction, and an x-aleph-context-budget response header) ship in subsequent releases. The settings below are accepted by the server today but do not yet drive any behaviour. This page is updated as each feature lands.

Configuration

Operators configure context limits and the compaction threshold via a single JSON document, delivered as one of:

  • CONTEXT_CONFIG_FILE: path to a JSON file. Preferred on Kubernetes: mount as a ConfigMap.

  • CONTEXT_CONFIG_JSON: inline JSON string. Used when _FILE is unset.

If both are set, _FILE wins. If neither is set, built-in defaults apply.

A malformed config (invalid JSON, out-of-range percent, non-positive context) fails pod startup loudly. A misconfigured operator wants to know about it at boot, not when the first request lands.

Schema

{
  "auto_trigger_enabled": true,
  "default": {
    "context": 32000,
    "autocompact_at": 80,
    "target_size_pct": 50
  },
  "models": {
    "qwen3-32b": { "context": 10000 },
    "gpt-oss-120b": { "context": 128000, "autocompact_at": 75, "target_size_pct": 40 }
  }
}
Field Type Range Description

auto_trigger_enabled

bool

—

Deployment-wide on/off for the post-response auto-trigger. Default true. The explicit /v1/responses/compact endpoint is unaffected by this flag.

default.context

int

> 0

Built-in fallback context-window size, in tokens.

default.autocompact_at

int

0–100

Percent of context at which the auto-trigger fires.

default.target_size_pct

int

0–100

Approximate target compacted size, as a percent of context. The summariser is instructed to compress to roughly this fraction.

models.<name>

object

—

Per-model override. Any subset of the three fields above. Names match the request’s model field case-sensitively.

All fields inside default and inside any models entry are optional. Omitted fields fall through the resolution chain below.

Resolution order

For each field on each request, the effective value is resolved per-field in this order:

  1. models[<request.model>][<field>]: if the operator set it.

  2. Inference backend catalog value for the request’s model, if the catalog is reachable and reports that field (only context today).

  3. default[<field>]: if the operator set it.

  4. Built-in default (context=32000, autocompact_at=80, target_size_pct=50).

The catalog comes from INFERENCE_MODELS_AVAILABLE_URL. On deployments where it’s directly reachable (Pharia AI), operators can omit per-model entries and let the catalog fill in context. On deployments where the AI gateway in front of inference does not expose the catalog (Platform), the chain skips step 2 and operators must list any model they care about; unrecognised models fall through to default.

autocompact_at and target_size_pct are policy, not backend properties, so they always come from operator config or the built-in default; the catalog has nothing to contribute there.

Worked examples

All examples assume the built-in default = {context: 32000, autocompact_at: 80, target_size_pct: 50}.

  • Platform deployment, catalog unreachable, no operator entry for the model. context = 32000, autocompact_at = 80%. Auto-trigger would fire at 25,600 tokens.

  • Pharia AI deployment, catalog reports qwen3 = 40000, no operator entry for qwen3. context = 40000 (from catalog), autocompact_at = 80% (catalog doesn’t carry this field, so the operator-default applies). Auto-trigger would fire at 32,000 tokens.

  • Operator entry qwen3: { context: 29000, autocompact_at: 50 }, catalog reports qwen3 = 40000. Operator wins per field: context = 29000, autocompact_at = 50%. Auto-trigger would fire at 14,500 tokens. (Same outcome on Platform and Pharia AI; operator entry beats the catalog.)

Operational notes

  • No hot reload. A config change requires a pod restart. This matches the existing GuardRailsConfig behaviour and is intentional: context limits change on deployment timescales, not at runtime.

  • No per-tenant or per-application overrides today. If they’re needed, they can layer on top later.

  • The catalog is consulted lazily through the same cached path the rest of the worker uses; there is no additional network call per request.