Resource requirements

This article describes the infrastructure that the Data and Search Platform depends on and the factors that determine its resource usage. Use it to plan capacity for your deployment.

In this article:

Components
External dependencies
Container inventory and data processing
What drives sizing
Best practices

Components

The Data and Search Platform consists of a retrieval service, which serves the HTTP API (search stores, uploads, search), and an ingestion worker pool, which runs document-processing workflows. Both depend on the following components:

Component	Concrete container	Purpose
Retrieval service	Sherlock	HTTP API: search-store/file/document CRUD, hybrid vector search, file upload/download, ingestion workflow runs, MCP endpoint.
Content delivery	Sherlock CDN	Content-serving deployment of the same Sherlock image; authorized file downloads streamed from object storage with ETag/Range support.
Ingestion worker pool	Holmes worker	Temporal worker running ingestion workflows: collects files, parses/OCRs, chunks, embeds, writes pages/chunks to Postgres and vectors to Qdrant.
Vector database	Qdrant	Stores chunk embeddings; queried for every search request. One collection per search store.
Relational database	PostgreSQL	Stores metadata, extracted page text, and workflow/run state.
Object storage	Object storage (S3/MinIO)	Stores uploaded files and export bundles (S3-compatible).
Identity and authorization	IAM sidecar (+ Envoy) and OpenFGA	The IAM sidecar supplies user identity on every request; OpenFGA enforces per-search-store access control.
Workflow orchestration	Temporal	Schedules and tracks ingestion workflow runs.
Distributed compute (optional)	Dask scheduler + workers	Scales out ingestion processing for high document volumes.
Document conversion (optional)	Gotenberg	Converts Office documents to PDF. Off by default.

Component

Concrete container

Purpose

Retrieval service

Sherlock

HTTP API: search-store/file/document CRUD, hybrid vector search, file upload/download, ingestion workflow runs, MCP endpoint.

Content delivery

Sherlock CDN

Content-serving deployment of the same Sherlock image; authorized file downloads streamed from object storage with ETag/Range support.

Ingestion worker pool

Holmes worker

Temporal worker running ingestion workflows: collects files, parses/OCRs, chunks, embeds, writes pages/chunks to Postgres and vectors to Qdrant.

Vector database

Qdrant

Stores chunk embeddings; queried for every search request. One collection per search store.

Relational database

PostgreSQL

Stores metadata, extracted page text, and workflow/run state.

Object storage

Object storage (S3/MinIO)

Stores uploaded files and export bundles (S3-compatible).

Identity and authorization

IAM sidecar (+ Envoy) and OpenFGA

The IAM sidecar supplies user identity on every request; OpenFGA enforces per-search-store access control.

Workflow orchestration

Temporal

Schedules and tracks ingestion workflow runs.

Distributed compute (optional)

Dask scheduler + workers

Scales out ingestion processing for high document volumes.

Document conversion (optional)

Gotenberg

Converts Office documents to PDF. Off by default.

External dependencies

A deployment must be able to reach the following services. The identity services integrate the deployment with the rest of the platform. The other services are infrastructure provisioned with the deployment. Dashed edges are optional.

The following table lists the same services with the details you need to provision them:

Dependency Needed for Operator Location Notes

Dependency	Needed for	Operator	Location	Notes
Identity provider	Authenticating every request	self	eu01	The service does not validate tokens itself. A sidecar (IAM sidecar + Envoy) checks the bearer token against the deployment’s identity provider (Pharia IAM on-prem, or Zitadel userinfo) and adds the caller’s identity (`X-User-ID` / `X-User-Is-Admin`) to the request. The service trusts these headers. Where Envoy handles incoming traffic, Envoy delegates its `ext_authz` check to the same sidecar. The bearer token is sent verbatim to the userinfo endpoint and cached in memory for 60s.
OpenFGA	Per-search-store authorization	self	eu01	Holds the owner/writer/viewer grants and service-level admin roles. Deployed only where `AUTHORIZATION_MODE=fga`. A deployment without it allows all requests, which is appropriate only for local development. Tuples contain user and group ids only.
PostgreSQL	Metadata, extracted text, workflow and run state	STACKIT (managed)	eu01	Pages and chunks store full extracted document text, not just metadata, plus workflow run inputs and error messages. Reached through the platform pgdog pooler. Open: managed-Postgres legal entity per environment still to be confirmed.
Vector database (Qdrant)	Chunk embeddings and search	self (+ Qdrant Cloud where used)	eu01	Runs in-cluster through the Qdrant operator. The payload contains `user_id`, `searchstore_id`, and filterable document metadata. Qdrant Cloud / API-key mode is supported by configuration but is not deployed.
S3-compatible object storage	Uploaded files	STACKIT (managed)	eu01	Uploaded files carry the `user_id` in their object key (`users/{user_id}/search-stores/…`), so key listings alone are personal data. Tenant export zips are the exception: they sit outside that tree, at `tenant-exports/{run_id}/{store_id}.zip`. MinIO in dev, S3-compatible object storage in production.
Temporal	Ingestion workflow orchestration	self	eu01	Needs a namespace and task queue. Histories persist run inputs and activity failure payloads, which can embed document text from stack traces. Production runs self-hosted Temporal; staging runs on Temporal Cloud via api_key auth. Planned mitigations: self-hosting and encryption.
Inference endpoints	Embeddings, vision/OCR extraction, reranking, transcription	self (Pharia)	eu01	The provider for each call is configured per search store and per workflow. A deployment needs credentials for each provider it offers. Search query text and document-derived content are sent to these endpoints. Open: whether the third-party inference provider should be listed as a separate operator.
OTLP collector	Audit events and traces	self	eu01	Administrative actions are emitted as OpenTelemetry log records to an OTLP endpoint and are not written to a database. Audit retention is therefore determined by your log pipeline, not by this service.
Prometheus and Grafana (optional)	Metrics and dashboards	self	eu01
Dask cluster (optional)	Distributed ingestion compute	self	eu01	Needed only to scale ingestion beyond the in-process capacity of the worker pool. Enabled by default in the chart. Holds the same document content as the Holmes worker in memory for each task.

Identity provider

Authenticating every request

self

eu01

The service does not validate tokens itself. A sidecar (IAM sidecar + Envoy) checks the bearer token against the deployment’s identity provider (Pharia IAM on-prem, or Zitadel userinfo) and adds the caller’s identity (X-User-ID / X-User-Is-Admin) to the request. The service trusts these headers. Where Envoy handles incoming traffic, Envoy delegates its ext_authz check to the same sidecar. The bearer token is sent verbatim to the userinfo endpoint and cached in memory for 60s.

OpenFGA

Per-search-store authorization

self

eu01

Holds the owner/writer/viewer grants and service-level admin roles. Deployed only where AUTHORIZATION_MODE=fga. A deployment without it allows all requests, which is appropriate only for local development. Tuples contain user and group ids only.

PostgreSQL

Metadata, extracted text, workflow and run state

STACKIT (managed)

eu01

Pages and chunks store full extracted document text, not just metadata, plus workflow run inputs and error messages. Reached through the platform pgdog pooler. Open: managed-Postgres legal entity per environment still to be confirmed.

Vector database (Qdrant)

Chunk embeddings and search

self (+ Qdrant Cloud where used)

eu01

Runs in-cluster through the Qdrant operator. The payload contains user_id, searchstore_id, and filterable document metadata. Qdrant Cloud / API-key mode is supported by configuration but is not deployed.

S3-compatible object storage

Uploaded files

STACKIT (managed)

eu01

Uploaded files carry the user_id in their object key (users/{user_id}/search-stores/…), so key listings alone are personal data. Tenant export zips are the exception: they sit outside that tree, at tenant-exports/{run_id}/{store_id}.zip. MinIO in dev, S3-compatible object storage in production.

Temporal

Ingestion workflow orchestration

self

eu01

Needs a namespace and task queue. Histories persist run inputs and activity failure payloads, which can embed document text from stack traces. Production runs self-hosted Temporal; staging runs on Temporal Cloud via api_key auth. Planned mitigations: self-hosting and encryption.

Inference endpoints

Embeddings, vision/OCR extraction, reranking, transcription

self (Pharia)

eu01

The provider for each call is configured per search store and per workflow. A deployment needs credentials for each provider it offers. Search query text and document-derived content are sent to these endpoints. Open: whether the third-party inference provider should be listed as a separate operator.

OTLP collector

Audit events and traces

self

eu01

Administrative actions are emitted as OpenTelemetry log records to an OTLP endpoint and are not written to a database. Audit retention is therefore determined by your log pipeline, not by this service.

Prometheus and Grafana (optional)

Metrics and dashboards

self

eu01

Dask cluster (optional)

Distributed ingestion compute

self

eu01

Needed only to scale ingestion beyond the in-process capacity of the worker pool. Enabled by default in the chart. Holds the same document content as the Holmes worker in memory for each task.

Container inventory and data processing

Container What it does Operator Location Personal data Notes

Container	What it does	Operator	Location	Personal data	Notes
Sherlock	Retrieval API: search-store/file/document CRUD, hybrid vector search, file upload/download, ingestion workflow runs and MCP endpoint	self	eu01	prompt, upload, user-id, credential, trace, ip	Query text is sent to the inference provider (Pharia) for embedding/rerank. The end-user bearer token passes through the pod, but only the iam-sidecar reads it; the app trusts the `X-User-ID` header set by the sidecar. Uploaded bytes are streamed to object storage.
Sherlock CDN	Content-serving deployment of the Sherlock image; authorized file downloads from object storage with ETag/Range support	self	eu01	upload, user-id, trace, ip	Serves raw uploaded bytes after a per-request authorization check. Read-only filesystem, no upload path, no disk cache. The container retains no data.
IAM sidecar (+ Envoy)	Auth sidecar in the Sherlock/CDN pods: validates the bearer token against Pharia IAM or Zitadel userinfo and sets `X-User-ID` / `X-User-Is-Admin`; Envoy delegates `ext_authz` to it	self	eu01	credential, user-id, ip	The bearer token is sent unchanged to the userinfo endpoint and cached in memory with the resolved identity for 60s. Userinfo responses may contain email/name claims. It is not confirmed whether anything other than ID and roles is logged.
Holmes worker	Temporal worker running ingestion workflows: collects files, parses/OCRs, chunks, embeds, writes pages/chunks to Postgres and vectors to Qdrant (cpu/io/gpu worker groups)	self	eu01	prompt, completion, upload, user-id, trace	Page images and extracted text are sent to the inference endpoint for VLM OCR and embeddings; the output is returned as page text. Credentials are service keys (inference API key / TVM-issued JWT), never the end user’s token. Activity failures can embed document snippets in Temporal histories.
Dask scheduler + workers	Distributed compute backend for Holmes modules (same Holmes image); document content passes through scheduler/worker memory during module execution	self	eu01	upload, user-id, trace	Enabled by default in the chart. Same data as the Holmes worker, held in memory per task. Open: whether workers can spill to local disk under memory pressure. Spooling currently active only for the audio modality.
Gotenberg	Office document-to-PDF conversion API: receives raw uploaded file bytes, returns the converted PDF	self	eu01	upload	Disabled by default; no workflow uses it yet. Chainguard image pinned by digest, stateless, in-cluster only.
Postgres	Metadata and content store: search stores, files, folders, documents, pages (full extracted text), chunks, workflow runs	STACKIT	eu01	upload, user-id	"upload" here means the extracted document text itself (pages/chunks store the full content), plus workflow run inputs and error messages. Reached through the platform pgdog pooler. Open: managed-Postgres legal entity per environment.
Qdrant	Vector database: chunk embeddings with a payload of `user_id`, `searchstore_id` and filterable document metadata; one collection per search store	self (sheet’s first table: "self + Qdrant")	eu01	user-id, upload	Vectors are computed from chunk text; the payload can contain document metadata fields. Runs in-cluster through the Qdrant operator. Qdrant Cloud / API-key mode is supported by configuration but not deployed.
Object storage (S3/MinIO)	Binary file storage for uploaded files and export bundles	STACKIT	eu01	upload, user-id	Uploaded files carry the `user_id` in their object key (`users/{user_id}/search-stores/…`), so key listings alone are personal data. Tenant export zips are the exception: they sit outside that tree, at `tenant-exports/{run_id}/{store_id}.zip`. MinIO in dev, S3-compatible storage in production. Open: operating legal entity per environment.
Temporal	Workflow orchestration for ingestion runs: task queues plus persisted workflow histories (run inputs, source URIs, failure payloads)	self	eu01	user-id, trace (sheet’s first table also lists "upload")	Histories persist run inputs and failure payloads, which can embed document text from stack traces. Production self-hosted; staging on Temporal Cloud via api_key auth — if that stays, Temporal Technologies Inc. needs its own operator row. Planned mitigations: self-hosting and encryption.
OpenFGA	Authorization store: owner/viewer/writer relationship tuples between user ids and search stores	self	eu01	user-id	Only deployed where `AUTHORIZATION_MODE=fga`. Platform-shared instance; tuples contain user and group ids only.

Sherlock

Retrieval API: search-store/file/document CRUD, hybrid vector search, file upload/download, ingestion workflow runs and MCP endpoint

self

eu01

prompt, upload, user-id, credential, trace, ip

Query text is sent to the inference provider (Pharia) for embedding/rerank. The end-user bearer token passes through the pod, but only the iam-sidecar reads it; the app trusts the X-User-ID header set by the sidecar. Uploaded bytes are streamed to object storage.

Sherlock CDN

Content-serving deployment of the Sherlock image; authorized file downloads from object storage with ETag/Range support

self

eu01

upload, user-id, trace, ip

Serves raw uploaded bytes after a per-request authorization check. Read-only filesystem, no upload path, no disk cache. The container retains no data.

IAM sidecar (+ Envoy)

Auth sidecar in the Sherlock/CDN pods: validates the bearer token against Pharia IAM or Zitadel userinfo and sets X-User-ID / X-User-Is-Admin; Envoy delegates ext_authz to it

self

eu01

credential, user-id, ip

The bearer token is sent unchanged to the userinfo endpoint and cached in memory with the resolved identity for 60s. Userinfo responses may contain email/name claims. It is not confirmed whether anything other than ID and roles is logged.

Holmes worker

Temporal worker running ingestion workflows: collects files, parses/OCRs, chunks, embeds, writes pages/chunks to Postgres and vectors to Qdrant (cpu/io/gpu worker groups)

self

eu01

prompt, completion, upload, user-id, trace

Page images and extracted text are sent to the inference endpoint for VLM OCR and embeddings; the output is returned as page text. Credentials are service keys (inference API key / TVM-issued JWT), never the end user’s token. Activity failures can embed document snippets in Temporal histories.

Dask scheduler + workers

Distributed compute backend for Holmes modules (same Holmes image); document content passes through scheduler/worker memory during module execution

self

eu01

upload, user-id, trace

Enabled by default in the chart. Same data as the Holmes worker, held in memory per task. Open: whether workers can spill to local disk under memory pressure. Spooling currently active only for the audio modality.

Gotenberg

Office document-to-PDF conversion API: receives raw uploaded file bytes, returns the converted PDF

self

eu01

upload

Disabled by default; no workflow uses it yet. Chainguard image pinned by digest, stateless, in-cluster only.

Postgres

Metadata and content store: search stores, files, folders, documents, pages (full extracted text), chunks, workflow runs

STACKIT

eu01

upload, user-id

"upload" here means the extracted document text itself (pages/chunks store the full content), plus workflow run inputs and error messages. Reached through the platform pgdog pooler. Open: managed-Postgres legal entity per environment.

Qdrant

Vector database: chunk embeddings with a payload of user_id, searchstore_id and filterable document metadata; one collection per search store

self (sheet’s first table: "self + Qdrant")

eu01

user-id, upload

Vectors are computed from chunk text; the payload can contain document metadata fields. Runs in-cluster through the Qdrant operator. Qdrant Cloud / API-key mode is supported by configuration but not deployed.

Object storage (S3/MinIO)

Binary file storage for uploaded files and export bundles

STACKIT

eu01

upload, user-id

Temporal

Workflow orchestration for ingestion runs: task queues plus persisted workflow histories (run inputs, source URIs, failure payloads)

self

eu01

user-id, trace (sheet’s first table also lists "upload")

Histories persist run inputs and failure payloads, which can embed document text from stack traces. Production self-hosted; staging on Temporal Cloud via api_key auth — if that stays, Temporal Technologies Inc. needs its own operator row. Planned mitigations: self-hosting and encryption.

OpenFGA

Authorization store: owner/viewer/writer relationship tuples between user ids and search stores

self

eu01

user-id

Only deployed where AUTHORIZATION_MODE=fga. Platform-shared instance; tuples contain user and group ids only.

What drives sizing

Exact CPU and memory figures depend on your workload. Confirm them with your platform team during deployment sizing. The following factors determine resource usage:

Vector database

Memory scales with the number of indexed chunks, the embedding dimension, and the number of replicas configured for availability. More chunks (from more documents, or from a smaller chunk_size that produces more chunks per document) and higher-dimensional embeddings both increase memory usage per replica.

Relational database

Storage scales approximately with the volume of extracted text. Page content, chunk metadata, and workflow/run history grow as you ingest more documents.

Object storage

Storage scales with the total size of uploaded files. Files are deduplicated by content hash, so uploading the same file to the same search store again does not increase storage.

Ingestion throughput

Ingestion worker CPU and memory usage scale with the number of concurrent workflow runs and with document complexity. OCR and vision-based extraction (see Workflows and presets) use more resources per document than plain text extraction. Per-page routing (smart_pdf_router) renders only the pages that need the vision model, so a mostly digital document uses close to the resources of text extraction.

Best practices

Start with the minimums: for a new deployment, use the minimum resources recommended by your platform team.
Monitor usage: monitor vector database memory and ingestion queue depth once production documents are being ingested.
Scale gradually: increase resources based on observed usage, not on projected peak load.
Plan for growth: as described in What is the Data and Search Platform?, the same search store and workflow configuration applies from initial experimentation through production. Only the infrastructure sizing needs to increase.