Semantic Embeddings
POST/semantic_embed
Embeds a prompt using a specific model and semantic embedding method. Resulting vectors that can be used for downstream tasks (e.g. semantic similarity) and models (e.g. classifiers). To obtain a valid model, use GET
/model-settings
.
Request
Query Parameters
Setting this to True, will signal to the API that you intend to be nice to other users by de-prioritizing your request below concurrent ones.
- application/json
Body
required
- Token ID arrays are used as as-is.
- Text prompt items are tokenized using the tokenizers specific to the model.
- Each image is converted into 144 tokens.
- Text Prompt
- Multimodal
- Array [
- Text
- Image
- Token Ids
- Array [
- 0 <= factor < 1 => Suppress the given token
- factor == 1 => identity operation, no change to attention
- factor > 1 => Amplify the given token
- ]
- Array [
- 0 <= factor < 1 => Suppress the given token
- factor == 1 => identity operation, no change to attention
- factor > 1 => Amplify the given token
- ]
- Array [
- 0 <= factor < 1 => Suppress the given token
- factor == 1 => identity operation, no change to attention
- factor > 1 => Amplify the given token
- ]
- ]
Name of the model to use. A model name refers to a model's architecture (number of parameters among others). The most recent version of the model is always used. The model output contains information as to the model version. To find out which models support semantic embeddings, please refer to the /model-settings
endpoint.
Possible values: [aleph-alpha
, null
]
Optional parameter that specifies which datacenters may process the request.
You can either set the parameter to "aleph-alpha" or omit it (defaulting to null
).
Not setting this value, or setting it to null
, gives us maximal flexibility in processing your request in our
own datacenters and on servers hosted with other providers. Choose this option for maximum availability.
Setting it to "aleph-alpha" allows us to only process the request in our own datacenters. Choose this option for maximal data privacy.
prompt object required
This field is used to send prompts to the model. A prompt can either be a text prompt or a multimodal prompt. A text prompt is a string of text. A multimodal prompt is an array of prompt items. It can be a combination of text, images, and token ID arrays.
In the case of a multimodal prompt, the prompt items will be concatenated and a single prompt will be used for the model.
Tokenization:
string
Possible values: [text
]
controls object[]
Starting character index to apply the factor to.
The amount of characters to apply the factor to.
Factor to apply to the given token in the attention matrix.
Possible values: [partial
, complete
]
Default value: partial
What to do if a control partially overlaps with a text token.
If set to "partial", the factor will be adjusted proportionally with the amount of the token it overlaps. So a factor of 2.0 of a control that only covers 2 of 4 token characters, would be adjusted to 1.5. (It always moves closer to 1, since 1 is an identity operation for control factors.)
If set to "complete", the full factor will be applied as long as the control overlaps with the token at all.
Possible values: [image
]
An image send as part of a prompt to a model. The image is represented as base64.
Note: The models operate on square images. All non-square images are center-cropped before going to the model, so portions of the image may not be visible.
You can supply specific cropping parameters if you like, to choose a different area of the image than a center-crop. Or, you can always transform the image yourself to a square before sending it.
x-coordinate of top left corner of cropping box in pixels
y-coordinate of top left corner of cropping box in pixels
Size of the cropping square in pixels
controls object[]
rect objectrequired
Bounding box in logical coordinates. From 0 to 1. With (0,0) being the upper left corner, and relative to the entire image.
Keep in mind, non-square images are center-cropped by default before going to the model. (You can specify a custom cropping if you want.). Since control coordinates are relative to the entire image, all or a portion of your control may be outside the "model visible area".
x-coordinate of top left corner of the control bounding box. Must be a value between 0 and 1, where 0 is the left corner and 1 is the right corner.
y-coordinate of top left corner of the control bounding box Must be a value between 0 and 1, where 0 is the top pixel row and 1 is the bottom row.
width of the control bounding box Must be a value between 0 and 1, where 1 means the full width of the image.
height of the control bounding box Must be a value between 0 and 1, where 1 means the full height of the image.
Factor to apply to the given token in the attention matrix.
Possible values: [partial
, complete
]
Default value: partial
What to do if a control partially overlaps with an image token.
If set to "partial", the factor will be adjusted proportionally with the amount of the token it overlaps. So a factor of 2.0 of a control that only covers half of the image "tile", would be adjusted to 1.5. (It always moves closer to 1, since 1 is an identity operation for control factors.)
If set to "complete", the full factor will be applied as long as the control overlaps with the token at all.
Possible values: [token_ids
]
controls object[]
Index of the token, relative to the list of tokens IDs in the current prompt item.
Factor to apply to the given token in the attention matrix.
Possible values: [symmetric
, document
, query
]
Type of embedding representation to embed the prompt with.
"symmetric"
: Symmetric embeddings assume that the text to be compared is interchangeable. Usage examples for symmetric embeddings are clustering, classification, anomaly detection or visualisation tasks. "symmetric" embeddings should be compared with other "symmetric" embeddings.
"document"
and "query"
: Asymmetric embeddings assume that there is a difference between queries and documents. They are used together in use cases such as search where you want to compare shorter queries against larger documents.
"query"
-embeddings are optimized for shorter texts, such as questions or keywords.
"document"
-embeddings are optimized for larger pieces of text to compare queries against.
Possible values: [128
]
The default behavior is to return the full embedding with 5120 dimensions. With this parameter you can compress the returned embedding to 128 dimensions. The compression is expected to result in a small drop in accuracy performance (4-6%), with the benefit of being much smaller, which makes comparing these embeddings much faster for use cases where speed is critical. With the compressed embedding can also perform better if you are embedding really short texts or documents.
Default value: false
Return normalized embeddings. This can be used to save on additional compute when applying a cosine similarity metric.
If set to null
, attention control parameters only apply to those tokens that have
explicitly been set in the request.
If set to a non-null value, we apply the control parameters to similar tokens as well.
Controls that have been applied to one token will then be applied to all other tokens
that have at least the similarity score defined by this parameter.
The similarity score is the cosine similarity of token embeddings.
Default value: true
true
: apply controls on prompt items by adding the log(control_factor)
to attention scores.
false
: apply controls on prompt items by (attention_scores - -attention_scores.min(-1)) * control_factor
Responses
- 200
OK
- application/json
- Schema
- Example (from schema)
Schema
- Token ID arrays are used as as-is.
- Text prompt items are tokenized using the tokenizers specific to the model.
- Each image is converted into a fixed amount of tokens that depends on the chosen model.
model name and version (if any) of the used model for inference
A list of floats that can be used to compare against other embeddings.
Number of tokens in the prompt.
Tokenization:
{
"model_version": "2021-12",
"embedding": [
-0.053497314,
0.0053749084,
0.06427002,
0.05316162,
-0.0044059753,
"..."
],
"num_tokens_prompt_total": 42
}