Resource requirements
This article describes the resources needed for your document processing and search infrastructure. Memory units are expressed in gibibytes (GiB), following Kubernetes.
Quick reference
| Component | Minimum CPU | Minimum memory | Recommended storage |
|---|---|---|---|
PhariaSearch |
2 CPU |
1GiB |
- |
PostgreSQL |
2 CPU |
1GiB |
128GiB |
Qdrant |
2 CPU |
64GiB |
96GiB |
PhariaData API |
800m |
1GiB |
- |
ETL Service |
500m |
1.5GiB |
- |
Detailed requirements
PhariaSearch
-
CPU: 2-4 cores
-
Memory: 1-2GiB
-
Scaling Factor:
documentBatchSize * averageDocumentSize * 2 + 256MiB
| For large documents or high concurrency, consider increasing CPU to 4 cores |
Database (PostgreSQL)
-
CPU: 2-4 cores
-
Memory: 1-2GiB
-
Storage: Calculate as
rawDocumentData * 1.1
Example: A 2.8 million document corpus (such as the German Wikipedia) needs about 10GB storage.
Vector database (Qdrant)
Resource requirements per replica (default is three replicas):
-
CPU: 2-4 cores
-
Memory: 64GiB recommended
-
Storage: 96GiB minimum
Memory usage depends on:
-
Number of documents
-
Chunk size and overlap
-
Search features enabled
Memory calculator
For each index, estimate:
-
Base:
4GiB * (512/chunkSize) * (1 + overlapPercentage) -
With BM25: Add 100% more for 512 token chunks
-
With FilterIndex: Add
numberOfChunks * chunkSize * (1 + overlapPercentage) * metadataSize
Example: For 2.8M documents with:
-
256 token chunks
-
50% overlap
-
BM25 enabled
-
String metadata (20 chars average)
Total: 4GiB * 2 * 2 + 8GiB + (6M * 2 * 2 * 20B) = ~25GiB
Real-time collection indexing
For real-time collection indexing capabilities (as used in PhariaAssistant):
-
Minimum Workers: two workers for the embedding model
-
Purpose: ensures high throughput for indexing jobs
-
Benefits:
-
parallel processing of documents
-
reduced indexing latency
-
better handling of concurrent indexing requests
-
| For production environments with heavy indexing loads, consider scaling to 2-4 workers for optimal performance. |