Creating search stores

This guide describes how to create and configure a search store, the container the Data and Search Platform uses to hold your documents' embeddings and serve search requests.

In this article:

Prerequisites
Client setup
1. Choose your embedding configuration
2. Create a search store
3. Share a search store (optional)
Validation
Troubleshooting

Prerequisites

The Data and Search Platform is deployed: https://sherlock.{ingressDomain}
You can reach it with one of: curl, the Sherlock Python SDK (sherlock-client), or the platform CLI (pharia).
You understand the search store concept — see Core concepts.

About the sherlock hostname

sherlock is the internal name of the platform’s retrieval service — the component that serves the HTTP API (search stores, uploads, search) — which is why the API is reached at https://sherlock.{ingressDomain}. A deployment also includes an ingestion worker pool that executes workflows in the background; you never call it directly. The full component list is in Resource requirements.

Client setup

Every example below is shown three ways. They all drive the same REST API at https://sherlock.{ingressDomain}/api/v1 — the SDK and the CLI are wrappers over the requests in the curl tab. Every request needs a bearer token; the deployment’s IAM sidecar validates it and stamps the user identity that Sherlock authorizes against.

curl
Python SDK
Platform CLI

pharia --env aleph-alpha iam login    # one-time OAuth login via your browser

export SHERLOCK_URL="https://sherlock.{ingressDomain}"
export AA_TOKEN="$(pharia --env aleph-alpha iam token)"

Every call then carries -H "Authorization: Bearer $AA_TOKEN". Any other source of a platform token works just as well — the API only sees the header.

Install the client from the Aleph Alpha package index:

pip install sherlock-client --extra-index-url https://alephalpha.jfrog.io/artifactory/api/pypi/holmes/simple

Then point it at the deployment:

import os

from sherlock import SyncSherlock

client = SyncSherlock(
    url="https://sherlock.{ingressDomain}",
    token=os.environ["AA_TOKEN"],
)

url and token default to the SHERLOCK_BASE_URL and SHERLOCK_TOKEN environment variables, so SyncSherlock() with no arguments works once those are set. Sherlock is the async twin with an identical API. Use either as a context manager (with SyncSherlock(...) as client:) to close the connection pool when you’re done.

The platform CLI (pharia) handles authentication for you — log in once, and every command reuses the cached token:

pharia iam login    # one-time OAuth login via your browser

Use the global --env flag to select the environment (for example, pharia --env aleph-alpha iam login); tokens are cached per environment and refreshed automatically. To reach a deployment the CLI doesn’t know, pass --base-url https://sherlock.{ingressDomain} or set PHARIA_SHERLOCK_URL. For your own tooling, print the raw token with TOKEN=$(pharia iam token).

1. Choose your embedding configuration

Each entry in a store’s models array describes one vector space. These are all the accepted values:

Field Accepted values Notes

Field	Accepted values	Notes
`embedding_model`	any model identifier your deployment exposes	Free-form string, normalised to lowercase (`BAAI/bge-m3` and `baai/bge-m3` are the same model). It does not affect which collection the store lands in, only which vectors are matched inside it.
`type`	`dense`, `bm25`	`dense` for semantic similarity; `bm25` for exact lexical matching — pair one of each for hybrid search. A `bm25` entry must be `modality: "text"` with `distance: "dot"` and no `dimension`/`embedding_model` (the lexical vector is computed in-process). At most one `bm25` entry per store. `sparse`, `multi`, and `quantized` are reserved and rejected at creation.
`distance`	`cosine`, `dot`, `euclid`, `manhattan`	`cosine` for text embeddings in almost all cases. Note the spelling `euclid`, not `euclidean`. Anything outside this set is rejected with `400`.
`modality`	`text`, `image`, `audio`, `cross-modal`	`text` for document search; the rest depend on which embedding models your deployment exposes.
`dimension`	positive integer, or omitted	The embedding model’s output size. Omit it only for genuinely variable-width vectors such as `bm25` — a dense entry without a positive `dimension` is rejected with `400`.
`qdrant`	per-vector index tuning	Optional escape hatch: `on_disk`, `hnsw_config`, `quantization_config`, `datatype`, `multivector_config`. Leave it unset unless you know the collection needs it.

embedding_model

any model identifier your deployment exposes

Free-form string, normalised to lowercase (BAAI/bge-m3 and baai/bge-m3 are the same model). It does not affect which collection the store lands in, only which vectors are matched inside it.

type

dense, bm25

dense for semantic similarity; bm25 for exact lexical matching — pair one of each for hybrid search. A bm25 entry must be modality: "text" with distance: "dot" and no dimension/embedding_model (the lexical vector is computed in-process). At most one bm25 entry per store. sparse, multi, and quantized are reserved and rejected at creation.

distance

cosine, dot, euclid, manhattan

cosine for text embeddings in almost all cases. Note the spelling euclid, not euclidean. Anything outside this set is rejected with 400.

modality

text, image, audio, cross-modal

text for document search; the rest depend on which embedding models your deployment exposes.

dimension

positive integer, or omitted

The embedding model’s output size. Omit it only for genuinely variable-width vectors such as bm25 — a dense entry without a positive dimension is rejected with 400.

qdrant

per-vector index tuning

Optional escape hatch: on_disk, hnsw_config, quantization_config, datatype, multivector_config. Leave it unset unless you know the collection needs it.

Any combination of these enums validates, so a typo in type, distance, or modality fails as an invalid value rather than silently doing something else.

Every store with the same combination of type, modality, distance, and dimension shares one vector-database collection, and that collection has to exist before the store can be created — see how search stores are stored. GET /api/v1/collections lists the ones already provisioned; picking a configuration that isn’t there yet gets you a 422 Collection Not Provisioned response (see Troubleshooting), and a platform admin can provision it on request.

2. Create a search store

curl
Python SDK
Platform CLI

curl -X POST "$SHERLOCK_URL/api/v1/search-stores" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AA_TOKEN" \
  -d '{
    "name": "my-store",
    "models": [
      {
        "embedding_model": "your-embedding-model",
        "type": "dense",
        "dimension": 1024,
        "distance": "cosine",
        "modality": "text"
      }
    ],
    "fusion_strategy": {"type": "rrf", "params": {"k": 60}},
    "access_policy": "private"
  }'

store = client.v1.search_stores.create(
    name="my-store",
    models=[
        {
            "embedding_model": "your-embedding-model",
            "type": "dense",
            "dimension": 1024,
            "distance": "cosine",
            "modality": "text",
        }
    ],
    fusion_strategy={"type": "rrf", "params": {"k": 60}},
    access_policy="private",
)

print(store.id)

create returns a store handle: store.id is the id you need later, and store.files, store.workflows, and store.documents are the scoped namespaces for ingestion and retrieval.

pharia --env aleph-alpha sherlock stores create \
  --name my-store \
  --models '[{
    "embedding_model": "your-embedding-model",
    "type": "dense",
    "dimension": 1024,
    "distance": "cosine",
    "modality": "text"
  }]' \
  --fusion-strategy '{"type": "rrf", "params": {"k": 60}}' \
  --access-policy private

JSON flag values also accept @file or @- for stdin.

The response includes the store’s id, which you use in every subsequent request. access_policy defaults to private — only you can see or use the store.

For hybrid search, add a second models entry with "type": "bm25" ("modality": "text", "distance": "dot", nothing else). fusion_strategy controls how the legs combine — rrf, convex, anchored, dbsf, or dense_only; omitted, a dense + bm25 pair derives convex when the dense leg uses cosine or dot, rrf otherwise. A non-empty weights is rejected.

Version availability

Hybrid bm25 stores are available from Data and Search Platform release 0.9.0, bundled in PhariaAI 1.260900.0. Older deployments accept a "type": "sparse" entry at creation, but no ingestion workflow populates that leg, so searches there are effectively dense-only.

models is immutable after creation — a change would orphan every vector already written, since search filters on the declared model. Create a new store instead.

Declaring filterable metadata

Your ingestion can attach arbitrary metadata to chunks, and it all lands under the metadata. prefix in the vector payload. Metadata attached at file upload lands under the user sub-namespace, so declare those fields with the user. prefix ("path": "user.site"). To filter on a metadata field efficiently, declare it as an index when you create the store:

curl
Python SDK
Platform CLI

curl -X POST "$SHERLOCK_URL/api/v1/search-stores" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AA_TOKEN" \
  -d '{
    "name": "my-store",
    "models": [
      {
        "embedding_model": "your-embedding-model",
        "type": "dense",
        "dimension": 1024,
        "distance": "cosine",
        "modality": "text"
      }
    ],
    "metadata_indexes": [
      {"path": "language",     "type": "string"},
      {"path": "year",         "type": "integer"},
      {"path": "draft",        "type": "boolean"},
      {"path": "published_at", "type": "datetime"}
    ]
  }'

store = client.v1.search_stores.create(
    name="my-store",
    models=[
        {
            "embedding_model": "your-embedding-model",
            "type": "dense",
            "dimension": 1024,
            "distance": "cosine",
            "modality": "text",
        }
    ],
    metadata_indexes=[
        {"path": "language", "type": "string"},
        {"path": "year", "type": "integer"},
        {"path": "draft", "type": "boolean"},
        {"path": "published_at", "type": "datetime"},
    ],
)

pharia --env aleph-alpha sherlock stores create \
  --name my-store \
  --models @models.json \
  --metadata-indexes '[
    {"path": "language", "type": "string"},
    {"path": "year",     "type": "integer"},
    {"path": "draft",    "type": "boolean"},
    {"path": "published_at", "type": "datetime"}
  ]'

Each entry is a path — the metadata key, resolved to metadata.<path> for both indexing and filtering — and a type. The type selects the index kind in the vector database, so it has to match the values your ingestion actually writes. These are all of them:

type Index created Filter it with Notes

`type`	Index created	Filter it with	Notes
`string`	keyword	`match.value`, `match.any`	The default choice: ids, language codes, tags, enum-like values. Declared with `"prefix": true`, the index also supports `match.prefix`.
`integer`	integer	`match.value`, `match.any`, `range`
`float`	float	`range`	Use range bounds, not `match.value`: a fractional value in `match.value` produces no match at all, and a whole one (`2.0`) is matched as an integer.
`boolean`	bool	`match.value` with `true`/`false`
`datetime`	datetime	`range` with RFC 3339 bounds	Write the values as RFC 3339 timestamps (`2026-01-01T00:00:00Z`) and filter with the same, for example `{"field": "metadata.published_at", "range": {"gte": "2026-01-01T00:00:00Z"}}`. Don’t mix timestamp and numeric bounds in one range — that’s a `400`.

string

keyword

match.value, match.any

The default choice: ids, language codes, tags, enum-like values. Declared with "prefix": true, the index also supports match.prefix.

integer

integer

match.value, match.any, range

float

float

range

Use range bounds, not match.value: a fractional value in match.value produces no match at all, and a whole one (2.0) is matched as an integer.

boolean

bool

match.value with true/false

datetime

datetime

range with RFC 3339 bounds

Write the values as RFC 3339 timestamps (2026-01-01T00:00:00Z) and filter with the same, for example {"field": "metadata.published_at", "range": {"gte": "2026-01-01T00:00:00Z"}}. Don’t mix timestamp and numeric bounds in one range — that’s a 400.

Anything else is rejected as an unsupported metadata index type, as is a blank path.

Version availability

"prefix": true and the match.prefix operator are available from release 0.9.0.

Three things worth knowing before you pick your fields:

Declaring is required to filter. A search whose filter references an undeclared metadata.<path> is rejected with 400, naming the field: "field is not filterable on this search store; declare it in metadata_indexes". The underlying vector database refuses to filter on an unindexed field, so the store validates the filter up front rather than passing an opaque rejection back to you. Declare every metadata field you intend to filter on.
The system fields are always filterable — file_id, document_id, page_id, chunk_id, workflow_id, workflow_run_id. Don’t declare those.
Indexes can only be declared at creation. Updating them is deliberately off the API, because rebuilding an index over an existing collection can be expensive — so a field you forgot means creating a new store. Declarations are additive and a path’s type is fixed once set. Since collections are shared by geometry, the indexes on the underlying collection are the union of what every store sharing it declared, but each store is still gated on its own declarations — a sibling store’s index doesn’t make the field filterable in yours.

To make a store readable by any authenticated user, update its access_policy:

curl
Python SDK
Platform CLI

curl -X PUT "$SHERLOCK_URL/api/v1/search-stores/<store-id>" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AA_TOKEN" \
  -d '{"access_policy": "public"}'

client.v1.search_stores("<store-id>").update(access_policy="public")

pharia --env aleph-alpha sherlock stores update <store-id> --access-policy public

For more selective sharing, grant viewer or writer access to individual users or groups instead:

curl
Python SDK
Platform CLI

curl -X PUT "$SHERLOCK_URL/api/v1/search-stores/<store-id>/permissions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AA_TOKEN" \
  -d '{
    "add": {
      "viewers": ["user:<user-id>"],
      "writers": ["group:<group-id>"]
    }
  }'

The response is the store’s full sharing state after the change: access_policy, owner_id, viewers, and writers. Revoke with a "remove" object of the same shape; removes are applied before adds, so a subject listed in both ends up granted.

state = client.v1.search_stores("<store-id>").update_permissions(
    add={
        "viewers": ["user:<user-id>"],
        "writers": ["group:<group-id>"],
    },
)

print(state.viewers, state.writers)

Pass remove= with the same shape to revoke. Both arguments are keyword-only.

pharia --env aleph-alpha sherlock stores permissions <store-id> \
  --add-viewer user:<user-id> \
  --add-writer group:<group-id>

Only the store’s owner can change access_policy, delete the store, or manage its permissions. A public store is readable by any authenticated user and appears in their store list — see Multi-user isolation and sharing. Set the policy back to private to stop sharing broadly, or revoke individual grants by moving the same subjects from add to remove (--remove-viewer/--remove-writer on the CLI).

Validation

curl
Python SDK
Platform CLI

curl "$SHERLOCK_URL/api/v1/search-stores" \
  -H "Authorization: Bearer $AA_TOKEN"

page = client.v1.search_stores.list(limit=20, offset=0)

for store in page:
    print(store.id, store.name, store.access_policy)
print(page.total)

pharia --env aleph-alpha sherlock stores list

This lists the stores you own, plus any public store. Confirm your new store appears with the id you intend to use.

Troubleshooting

Collection not provisioned

Error: 422 Collection Not Provisioned
Solution: The embedding configuration in models doesn’t map to a collection your administrator has provisioned yet. Ask them to provision it, or pick a configuration that’s already available.

Authorisation errors

Error: 401 Unauthorized
Solution: Verify your token is current and properly formatted.
Error: 403 Forbidden
Solution: You’re authenticated, but don’t have the required permission on this store (for example, trying to change access_policy without being the owner).