For workflow authors

The built-in presets cover standard document ingestion. Workflows themselves are a general pipeline that users can extend. This page describes the programming model for teams that write their own workflows and processing modules. It is intended for platform engineers: workflows are written in code inside the ingestion service, not configured through the HTTP API.


Programming model

A pipeline consists of two kinds of objects:

  • A Workflow defines orchestration as data. It is a Pydantic model whose fields declare the modules it uses. Its async run(ctx) method dispatches each stage and passes lightweight references between stages. The same workflow runs without changes either in-process for debugging or distributed across the worker pool.

  • A Module is a serializable, pure transform. It receives entities (files, documents, pages, chunks) and returns new entities. It never persists anything itself: the framework loads inputs and persists outputs when it dispatches the module.

There is no separate "Step" layer. The model consists only of workflows and modules.

The module contract

A module subclasses Module and requires three elements:

  1. A kind — a Literal field with a default value (for example, kind: Literal["word_count"] = "word_count"). Declaring it registers the class in the module registry automatically and tags the workflow history.

  2. A run method — run(self, ctx, input), which returns one entity or a flat list of entities. Use async def for I/O-bound work or a plain def for CPU-bound work. The framework infers from the signature whether to call the module per item or per batch: annotate the input as page: Page for per-item processing, or pages: list[Page] for batch processing.

  3. Parameters as Pydantic fields — every field declared with Field(default=..., description=...) becomes a configurable workflow parameter. The parameter schema is generated from the model automatically.

Class-level settings control execution: compute (threads, processes, or inline), workload (CPU or IO task queue), and the per-work-unit settings batch_size, max_in_flight, and timeout. Two flags change a module’s role in the pipeline: initial = True (the module reads the run’s input source and produces the first entities) and terminal = True (the module returns a final result instead of persisting entities).

The runtime context

run receives a RuntimeContext that provides all resources a module can access:

Attribute Purpose

ctx.db / ctx.vectors

Relational and vector storage for querying existing entities. The framework persists outputs

ctx.files

File storage: open, read, upload

ctx.inference

Model inference: completion, image description, embeddings, transcription

ctx.pdf

PDF backends: render pages, extract text and images

ctx.scope

The run’s scope: search store, workflow, run ID, and input source

Modules must not open their own connections or extend their scope. Every read and write is automatically restricted to the search store and workflow run in which the module executes.

Constraints and guarantees

  • Idempotency. Entities declare which fields their ID is derived from, so re-runs and retries produce the same rows instead of duplicates. Enrichment outputs are append-only, and saving a duplicate has no effect. As a result, modules can be retried safely by the orchestrator.

  • Error handling. Raise NonRetryableError for deterministic failures, such as invalid input or configuration, so that a run that cannot succeed fails immediately. Any other exception is retried according to the worker’s retry policy.

  • Purity. A module is a pure transform of its input and ctx. Loading inputs, persisting outputs, and enforcing scope are done by the framework, not inside the module.

Composing workflows and presets

A workflow subclasses Workflow and declares its modules as fields. A field can be a single module, or a kind-discriminated union that makes the module selectable through configuration. The workflow implements async run(ctx), which dispatches the stages in sequence. Subclassing registers the workflow under its class name. This is the name you pass as workflow_name when you create a workflow through the API (see Workflows and presets).

A preset is a workflow subclass that overrides field defaults. For example, DocumentAIFull is DocumentAI with page-level keyword extraction enabled.

Registering a custom workflow

Workflows are registered when their code is imported. At startup, the worker imports the preset package. This populates the module and workflow registries and creates one execution activity per module kind. To deploy a new workflow, its code must currently be part of the ingestion service or be imported by it. There is no external plugin mechanism yet. Plan custom workflow development with the platform team.

Where to start

Ask the platform team for the workflow example and the module and workflow developer guides. The example is a complete word-count workflow that shows the full programming model.