Context Compaction

Long conversations eventually exceed a model’s context window. The Responses API tracks how much of each model’s window a conversation is using and lets operators configure a threshold at which older turns get summarised into a single "compaction" entry, freeing room for the conversation to continue.

Status

This release ships the explicit POST /v1/responses/compact endpoint, the post-response auto-trigger, and the x-aleph-context-budget visibility surface, all described below.

Endpoint: POST /v1/responses/compact

Clients call this endpoint to compact an existing conversation chain. The server loads the chain anchored at previous_response_id, summarises it through the configured inference backend, persists the result, and returns the synthetic compaction object. On the next /v1/responses call with the same previous_response_id, the server splices the compaction into the chain in place of the older turns.

Request:

{
  "previous_response_id": "resp_abc123"
}
Field Type Required Description

previous_response_id

string

yes

Id of an existing response the caller owns. The chain anchored at this id is what gets summarised.

This release accepts only previous_response_id. OpenAI’s "explicit input" shape (a stateless input array) is deferred to a future release.

Response (200 OK):

{
  "id": "cmp_PYWk0...",
  "created_at": 1700000000,
  "object": "response.compaction",
  "output": [
    {
      "type": "compaction",
      "encrypted_content": "Conversation summary: …"
    }
  ],
  "usage": {
    "input_tokens": 200,
    "output_tokens": 20,
    "total_tokens": 220
  }
}

  • id is derived deterministically from the persisted row’s natural key. It is not stored as a column on the row; identical chains produce identical ids.

  • output[0].type is always "compaction". The encrypted_content field name mirrors OpenAI; the value is plaintext today (real envelope encryption is a separate future feature).

  • usage is the inference backend’s reported token count for the summarisation call.

How aggressively does it compress?

The summariser is instructed to bring the resulting context down to approximately target_size_pct of the model’s context. Both values come from operator configuration (see below); there is no request parameter to override them, as operators set the target once per deployment.

Subsequent /v1/responses calls: Once a compaction exists for the chain, any /v1/responses call with previous_response_id pointing at that chain (or any of its descendants) will reconstruct the conversation with the compaction replacing the pre-anchor turns. Two compactions on the same chain are allowed; the most recently created one wins.

Errors:

Status When

400 / 422

Malformed request body (missing previous_response_id, etc.).

404

previous_response_id is unknown or belongs to a different user. The server returns 404 in both cases so existence isn’t leaked. Admin users bypass the ownership check.

413

The summariser produced more than 10 MiB of text. Try again, or accept that this branch can’t be compacted at the current target size.

501

The deployment has no compaction storage configured. Configure a Postgres backend or accept the in-memory default for non-production deployments.

502 / 504

The inference backend failed or timed out during summarisation. This includes the case where the chain you handed in is too long for the summariser to handle in one shot.

There is no pre-flight token-budget check in this release; the request reaches the backend, and whatever the backend returns is surfaced.

Ownership and auth: identical to the rest of /v1/responses/*. Bearer token, the caller must own previous_response_id, admin override applies.

Client-supplied compaction items

Compaction isn’t only server-driven. A client can summarise a conversation on its own and hand the summary back on a regular /v1/responses call by including one or more items of type: "compaction" in the input array. This lets a client that already holds the conversation compact it without a round trip to /v1/responses/compact.

The server’s handling depends on whether the request carries a previous_response_id:

  • With previous_response_id: each compaction item is persisted against that anchor (same table and shape the /compact endpoint writes) and stripped from the input before the request reaches the backend. The next reconstruction picks it up exactly as it would a server-produced compaction. The remaining non-compaction items are forwarded as the turn’s input.

  • Without previous_response_id: there is no anchor to persist against, so nothing is stored. Each compaction item is projected inline to a system-role message so the backend receives a shape it accepts. The compaction is one-shot; it influences this request only.

Storage is append-only. Pushing the same compaction item again appends another row rather than replacing the prior one; reconstruction takes the most recently created row for the (response_id, model) pair, so a replay is semantically a no-op (latest wins) but does leave a redundant row behind. There is no client-facing dedup in this release.

Persisting requires a configured compaction backend. On a deployment with no compaction storage the persist path is unreachable by construction (the port is always wired), but the code guards it explicitly rather than relying on that invariant.

Auto-trigger: post-response compaction

In addition to the explicit endpoint above, the server checks every /v1/responses delivery against the configured threshold and enqueues a compaction job when the response’s reported token usage crosses it. Clients do not opt in or opt out per request; the trigger is deployment-wide.

What fires the trigger

After a response is delivered (whether the caller used the non-streaming, streaming, or background path), the server evaluates:

usage.total_tokens * 100 / context >= autocompact_at

  • usage.total_tokens is the value the inference backend reported on the response.

  • context is the per-model context window, resolved via the rules in the Configuration section below.

  • autocompact_at is the percent threshold, also resolved per-model.

When the comparison evaluates to true, the server enqueues one compaction job. When false, the server records nothing and emits no log line.

Anchor model

The job is anchored at the response that just finished (response.id) and uses that response’s model (response.model). The worker that picks the job up calls the same code path as POST /v1/responses/compact with is_admin=true, because the trigger fires on behalf of the deployment, not the end user.

How to disable

Set the environment variable CONTEXT_AUTO_TRIGGER_ENABLED=false, or set "auto_trigger_enabled": false inside the JSON config document. The explicit POST /v1/responses/compact endpoint is unaffected by this flag and continues to work.

A deployment that has never set CONTEXT_CONFIG_FILE or CONTEXT_CONFIG_JSON runs with the built-in default auto_trigger_enabled=true. The auto-trigger fires at 80 percent of a 32,000-token context window unless the operator overrides the configuration.

Per-request behaviour

The trigger has no effect on the response that caused it. The client receives its /v1/responses response normally; the compaction work happens asynchronously on a worker. A failure to enqueue or to summarise does not surface to the client and does not poison the response that triggered the work.

When the next /v1/responses request arrives on the same chain, the compaction may or may not be persisted yet. If persisted, the server splices it in as documented under the explicit endpoint. If not yet persisted, the server reconstructs the full chain. The retry is not coordinated; the client should not change its calling pattern based on this race.

Observability

Signal Value

Log line (at WARN, structured)

auto-trigger: enqueuing compaction response_id=<id> model=<name> context.tokens_used=<int> context.tokens_limit=<int> context.pct=<int> context.autocompact_at=<int>

Metric (OpenTelemetry counter)

compactions_created_total{trigger="auto"}, bumped once per enqueue. The explicit endpoint bumps the same counter with trigger="manual".

Broker stream

<broker.stream_name>:compaction (default stream name is work:requests, so the default compaction stream is work:requests:compaction).

Broker consumer group

<broker.consumer_group>:compaction (default workers:compaction).

Broker dead-letter queue

<broker.stream_name>:compaction:dead-letter (default work:requests:compaction:dead-letter).

The WARN log fires only when the trigger fires. Below threshold, the server emits no log line for the auto-trigger (the existing /v1/responses access log still records the request).

Failure handling

If summarisation fails (the backend returns 5xx, the chain exceeds the summariser’s own input limit, the persisted output exceeds 10 MiB), the worker dead-letters the job in the queue listed above. The next /v1/responses request on the same chain whose usage crosses the threshold re-enqueues a new job. There is no compaction-specific retry loop; the next user-driven request is the retry signal.

A worker that crashes after dequeue and before persistence loses the job. The same re-enqueue-on-next-request behaviour applies.

Backwards-compatible disable

Operators who do not want the auto-trigger should set CONTEXT_AUTO_TRIGGER_ENABLED=false before deploying this release. Existing deployments that do nothing inherit the built-in default of enabled at 80 percent, which is a behaviour change relative to the prior release.

Context-budget visibility

Every completed response reports how much of the model’s context window the conversation is using. The auto-trigger runs after the response is sent, so this surface is the client’s only synchronous signal that the window is filling up.

Non-streaming responses (POST /v1/responses with stream=false, and GET /v1/responses/{response_id}) carry a response header:

x-aleph-context-budget: used=29491; limit=32000; pct=92

  • used is usage.total_tokens from the response, the same number the auto-trigger threshold reads.

  • limit is the model’s context window, resolved through the configuration chain below.

  • pct is used * 100 / limit, floor division, not clamped. pct=104 means the conversation is over budget.

Streaming responses cannot carry the header (headers ship before the token count is known), so the same data arrives as a named SSE event immediately before response.completed:

event: aleph.context_budget
data: {"used": 29491, "limit": 32000, "pct": 92}

Clients that do not recognise the event name ignore it; named events are part of the SSE contract.

Absence means unknown. When the model has no usable context limit (no operator entry, no catalog value, and the built-in default removed), neither the header nor the event is emitted. The header also does not appear on in_progress or failed background poll responses.

The budget surface is independent of the auto-trigger: it is emitted even when CONTEXT_AUTO_TRIGGER_ENABLED=false. Both integers only; the surface never carries conversation content.

Configuration

Operators configure context limits and the compaction threshold via a single JSON document, delivered as one of:

  • CONTEXT_CONFIG_FILE: path to a JSON file. Preferred on Kubernetes: mount as a ConfigMap.

  • CONTEXT_CONFIG_JSON: inline JSON string. Used when _FILE is unset.

If both are set, _FILE wins. If neither is set, built-in defaults apply.

A malformed config (invalid JSON, out-of-range percent, non-positive context) fails pod startup loudly. A misconfigured operator wants to know about it at boot, not when the first request lands.

Schema

{
  "auto_trigger_enabled": true,
  "default": {
    "context": 32000,
    "autocompact_at": 80,
    "target_size_pct": 50
  },
  "models": {
    "qwen3-32b": { "context": 10000 },
    "gpt-oss-120b": { "context": 128000, "autocompact_at": 75, "target_size_pct": 40 }
  }
}
Field Type Range Description

auto_trigger_enabled

bool

—

Deployment-wide on/off for the post-response auto-trigger. Default true. The explicit /v1/responses/compact endpoint is unaffected by this flag.

default.context

int

> 0

Built-in fallback context-window size, in tokens.

default.autocompact_at

int

0–100

Percent of context at which the auto-trigger fires.

default.target_size_pct

int

0–100

Approximate target compacted size, as a percent of context. The summariser is instructed to compress to roughly this fraction.

models.<name>

object

—

Per-model override. Any subset of the three fields above. Names match the request’s model field case-sensitively.

All fields inside default and inside any models entry are optional. Omitted fields fall through the resolution chain below.

Resolution order

For each field on each request, the effective value is resolved per-field in this order:

  1. models[<request.model>][<field>]: if the operator set it.

  2. Inference backend catalog value for the request’s model, if the catalog is reachable and reports that field (only context today).

  3. default[<field>]: if the operator set it.

  4. Built-in default (context=32000, autocompact_at=80, target_size_pct=50).

The catalog comes from INFERENCE_MODELS_AVAILABLE_URL. On deployments where it’s directly reachable (Pharia AI), operators can omit per-model entries and let the catalog fill in context. On deployments where the AI gateway in front of inference does not expose the catalog (Platform), the chain skips step 2 and operators must list any model they care about; unrecognised models fall through to default.

autocompact_at and target_size_pct are policy, not backend properties, so they always come from operator config or the built-in default; the catalog has nothing to contribute there.

Worked examples

All examples assume the built-in default = {context: 32000, autocompact_at: 80, target_size_pct: 50}.

  • Platform deployment, catalog unreachable, no operator entry for the model. context = 32000, autocompact_at = 80%. Auto-trigger would fire at 25,600 tokens.

  • Pharia AI deployment, catalog reports qwen3 = 40000, no operator entry for qwen3. context = 40000 (from catalog), autocompact_at = 80% (catalog doesn’t carry this field, so the operator-default applies). Auto-trigger would fire at 32,000 tokens.

  • Operator entry qwen3: { context: 29000, autocompact_at: 50 }, catalog reports qwen3 = 40000. Operator wins per field: context = 29000, autocompact_at = 50%. Auto-trigger would fire at 14,500 tokens. (Same outcome on Platform and Pharia AI; operator entry beats the catalog.)

Operational notes

  • No hot reload. A config change requires a pod restart. This matches the existing GuardRailsConfig behaviour and is intentional: context limits change on deployment timescales, not at runtime.

  • No per-tenant or per-application overrides today. If they’re needed, they can layer on top later.

  • The catalog is consulted lazily through the same cached path the rest of the worker uses; there is no additional network call per request.