Configuring workflows

Each example below is shown for three clients. All three call the same REST API at https://sherlock.{ingressDomain}/api/v1; the SDK and the CLI wrap the requests shown in the curl tab. Every request requires a bearer token. The deployment’s IAM sidecar validates the token and adds the caller’s user identity to the request, and Sherlock authorizes the request against that identity.

List your stores, the workflows on a store, and the current configuration of a single workflow:

To read a single workflow, pass either its ID or its name.

DocumentAI consists of modules. Each top-level key in parameters configures one module:

Groups with more than one implementation (pdf_reader, docx_reader, xlsx_reader, chunker) select the implementation with a "kind" field. In addition to its own parameters, every module accepts the execution fields batch_size, max_in_flight, and timeout.

This is the standard configuration for digitally created documents. It extracts the embedded text, chunks it recursively, and embeds the chunks through the platform’s inference gateway.

Create a workflow with these parameters and the standard run-input schema (see The input schema):

For scanned documents or complex layouts, rasterize each PDF page and annotate it with a vision model. This pipeline uses two modules. pdf_rasterizer renders each page to an image. vision_reader converts each image into one structured page, in a separate stage that processes pages in parallel. pdf_rasterizer does not extract text, so a vision_reader with an explicit provider and model is required; there is no default endpoint. A run without a vision_reader, or with one that lacks provider or model, fails immediately and is not retried.

Other pdf_rasterizer fields: pdf_backend (pdf_oxide default, or pdfboss), dpi (default 150), and image_format (default png).

Other vision_reader fields: max_tokens, temperature (see Sampling temperature; use 0 for reproducible OCR), ignore_headers_footers (default true), prompt, and extra_body. prompt replaces the built-in layout-extraction prompt. By default, dots-OCR models receive a dots-format prompt and other models receive a normalized-coordinate prompt. extra_body contains vendor-specific payload fields; for example, {"chat_template_kwargs": {"thinking": false}} disables a reasoning model’s thinking mode. The chunker processes the annotated pages the same way as text-extracted pages.

Many PDFs mix digitally created pages with scanned pages, screenshots, or pages whose fonts do not map to readable characters. pdf_text returns empty or garbled text for those pages, and pdf_rasterizer sends every page to the vision model, including the pages that pdf_text reads correctly. smart_pdf_router decides per page. Pages with a usable text layer are extracted as text. Only the other pages are rendered and annotated by vision_reader. A mostly digital document therefore costs close to pdf_text, and its scanned pages are still read.

The router reads the structure of each page (fonts, images, and annotations) without rendering it. A page goes to the vision model when one of these rules applies:

All other pages keep their text layer. A page that draws neither fonts nor images becomes an empty page and is not rendered.

vision_reader is required, as in Recipe: vision (OCR) ingestion. A run without it fails before it processes any page and is not retried, even if no page of the document would go to the vision model.

Other smart_pdf_router fields:

pdf_router is the basic version of the router. It sends a page to the vision model only when the page draws images and has no extractable text. smart_pdf_router with min_text_chars: 1 and max_unmapped_share: 1 behaves the same way.

Each page records its routing decision in its metadata under pdf_router: route (text or raster), the fonts, images, and annotations counts, and, for each share rule whose limit is below 1, the page’s measured share (unmapped_share, rtl_share). Use these values to check why a page went to the vision model.

Text extraction and OCR do not produce searchable text for the content of charts, diagrams, and photos. The figure pipeline generates text descriptions for them in two stages. A figure_extractor produces one image crop per figure. image_describer sends each crop to a vision model and stores the returned description.

Two extractors implement figure_extractor. Select one with kind:

image_describer fields: provider (default pharia_next) and model (default moonshotai/kimi-k2.6) identify the vision deployment. prompt replaces the built-in description prompt. max_tokens (default 2048), temperature, and extra_body configure the model call. max_in_flight (default 4) limits the number of concurrent describe calls.

The platform stores descriptions append-only, keyed by page, image, model, and prompt. A re-run with the same model and prompt has no effect. A different model or prompt adds new descriptions and does not change existing ones.

Modules that call models (vision_reader, vectorize, image_describer, …​) select their client by name. The provider parameter is a lookup key. The endpoint, credentials, and retry policy for each provider are defined in the server-side deployment configuration. Workflow parameters never contain secrets or URLs.

The providers below are available only when the data platform runs on the Aleph Alpha platform. On-premise installations use different providers.

You can select a provider only if the deployment configures it. If you select an unconfigured provider, the run fails.

Every module that calls a chat or vision model accepts a temperature field (0.0--2.0):

Embeddings (vectorize) and non-LLM modules (chunkers, keyword enrichment) have no temperature.

If you do not set temperature (the default), the request omits the field and the model server’s default applies. 0 is treated as a value and sent to the model; it is not treated as unset. Use 0 for reproducible extraction, which is the usual setting for OCR and structured layout parsing:

Smaller chunks give more precise matches but contain less surrounding context. Overlap preserves context across chunk boundaries.

The vectorize module calls an OpenAI-compatible embeddings endpoint through a named provider (default pharia; see Inference providers).

The search store defines the vector schema. If you do not set model and dimension, the module takes them from the store’s models entry, so {"provider": "pharia"} is usually sufficient. If you set them, they must match a vector configured on the store. If they do not match, the run fails instead of writing vectors that search cannot find.

Both groups take a list of enrichment modules, selected by kind. Enrichments add metadata rows for pages or chunks. They do not change the page and chunk content.

Removing repeated page headers and footers is not an enrichment. vision_reader does it at read time; see Cleaning page headers and footers.

Repeated headers and footers, such as running titles and page numbers, add irrelevant text to chunks and search hits. vision_reader removes them: with ignore_headers_footers (default true), it skips the header and footer regions that the layout model classifies; see Recipe: vision (OCR) ingestion.

Affected pages record the number of dropped lines as metadata.headers_footers_removed.

To create a workflow, you currently must send schema: the JSON schema of the input that each run accepts.

For DocumentAI, the standard schema declares the source property:

The change applies only to future runs. The platform does not automatically reprocess content that is already indexed.

To start a run, specify a source (file IDs from Ingesting files). Then stream its progress:

The event stream sends server-sent events until the run completes or fails. Deleting a workflow also deletes all its runs and their derived data.