Configuring workflows

Every example below is shown three ways. They all drive the same REST API at https://sherlock.{ingressDomain}/api/v1 — the SDK and the CLI are wrappers over the requests in the curl tab. Every request needs a bearer token; the deployment’s IAM sidecar validates it and stamps the user identity that Sherlock authorizes against.

List your stores, the workflows on one, and a single workflow’s current configuration:

The workflow read accepts either the workflow’s id or its name.

DocumentAI is a composition of modules. Each top-level key in parameters configures one module:

Groups with more than one implementation (pdf_reader, docx_reader, xlsx_reader, chunker) select it with a "kind" field. Every module also accepts the execution fields batch_size, max_in_flight, and timeout together with its own parameters.

This is the standard configuration for digital documents. It extracts the embedded text, chunks it recursively, and embeds the chunks through the platform’s inference gateway.

Create a workflow with those parameters, plus the standard run-input schema (see The input schema):

For scanned documents or complex layouts, rasterize each PDF page and annotate it with a vision model. This is a two-module pipeline. pdf_rasterizer renders each page into an image. vision_reader turns each image into a structured page — one page in, one page out — in a separate stage that fans out per page. pdf_rasterizer only renders, so a vision_reader with an explicit provider and model is required; there is no default endpoint. A run without one fails immediately and does not retry.

Useful pdf_rasterizer fields: pdf_backend (pdf_oxide default, or pdfboss), dpi (default 150), and image_format (default png).

Useful vision_reader fields: max_tokens, temperature (see Sampling temperature; use 0 for reproducible OCR), ignore_headers_footers (default true), prompt, and extra_body. prompt replaces the built-in layout-extraction prompt; dots-OCR models automatically get a dots-format prompt, and other models get a normalized-coordinate prompt. extra_body carries vendor-specific payload fields — for example, {"chat_template_kwargs": {"thinking": false}} switches off a reasoning model’s thinking mode. The annotated pages flow into the chunker exactly like text-extracted pages.

Charts, diagrams, and photos carry meaning that neither text extraction nor OCR turns into searchable text. The figure pipeline closes that gap in two stages. A figure_extractor produces one image crop per figure. image_describer sends each crop to a vision model and stores the returned description.

Two extractors fill the figure_extractor slot. Select one with kind:

image_describer fields: provider (default pharia_next) and model (default moonshotai/kimi-k2.6) name the vision deployment. prompt replaces the built-in description prompt. max_tokens (default 2048), temperature, and extra_body shape the call. max_in_flight (default 4) limits how many describe calls run at the same time.

The platform stores descriptions append-only, keyed by page, image, model, and prompt. A re-run with the same model and prompt is a no-op. A different model or prompt adds new descriptions and does not change existing ones.

Modules that call models (vision_reader, vectorize, image_describer, …​) select their client by name. The provider parameter is a lookup key; the endpoint, credentials, and retry policy behind it are server-side deployment configuration. Workflow parameters never carry secrets or URLs.

These providers are available only when the data platform runs on the Aleph Alpha platform. On-premise installations have their own, different providers.

You can select a provider only if the deployment configures it. An unconfigured provider fails the run.

Every module that calls a chat or vision model accepts a temperature field (0.0--2.0):

Embeddings (vectorize) and non-LLM modules (chunkers, keyword enrichment) have no temperature.

If you do not set temperature (the default), the request omits the field and the model server’s own default applies. 0 is a real value, and the module sends it. Use 0 for reproducible extraction — the usual choice for OCR and structured layout parsing:

Smaller chunks are more precise but lose surrounding context. Overlap keeps context at chunk boundaries.

The vectorize module calls an OpenAI-compatible embeddings endpoint through a named provider (default pharia; see Inference providers).

The search store is the source of truth for the vector schema. When you do not set model and dimension, the module adopts them from the store’s models entry, so {"provider": "pharia"} is usually sufficient. If you set them, they must match a slot that the store provisions. If they do not match, the run fails; it does not write vectors that search cannot find.

Both groups take a list of enrichment modules, selected by kind. Enrichments add metadata rows next to pages or chunks. The page and chunk content does not change.

Stripping repeated page headers and footers is not an enrichment — it happens at read time; see Cleaning page headers and footers.

Repeated headers and footers — running titles, page numbers — pollute chunks and search hits. The readers remove them:

Affected pages record the number of dropped lines as metadata.headers_footers_removed.

To create a workflow, you currently must send schema: the JSON schema of the input that each run accepts.

For DocumentAI the standard schema declares the source property:

Only future runs use the change. The platform does not automatically reprocess content that is already indexed.

To start a run, name a source (file IDs from Ingesting files). Then stream its progress:

The event stream carries server-sent events until the run completes or fails. Deleting a workflow deletes all its runs and their derived data with it.