Tokenize

POST 
https://api.aleph-alpha.com/tokenize

Tokenize a prompt for a specific model. To obtain a valid model, use GET /model-settings.

Request

application/json

Bodyrequired

modelstringrequired
promptstringrequired
tokensbooleanrequired
token_idsbooleanrequired

Responses

200

OK

application/json

Schema

Schema

tokensstring[]
token_idsinteger[]

Example (auto)

{
  "tokens": [
    "ĠAn",
    "Ġapple",
    "Ġa",
    "Ġday",
    "Ġkeeps",
    "Ġthe",
    "Ġdoctor",
    "Ġaway",
    "."
  ],
  "token_ids": [
    560,
    34438,
    246,
    1966,
    18075,
    275,
    8809,
    3476,
    17
  ]
}

Authorization: http

name: tokentype: httpscheme: bearerdescription: Can be generated in your [Aleph Alpha profile](https://app.aleph-alpha.com/profile)

• csharp
• curl
• dart
• go
• http
• java
• javascript
• kotlin
• c
• nodejs
• objective-c
• ocaml
• php
• powershell
• python
• r
• ruby
• rust
• shell
• swift

• HTTPCLIENT
• RESTSHARP

var client = new HttpClient();
var request = new HttpRequestMessage(HttpMethod.Post, "https://api.aleph-alpha.com/tokenize");
request.Headers.Add("Accept", "application/json");
request.Headers.Add("Authorization", "Bearer <token>");
var content = new StringContent("{\n  \"model\": \"llama-3.1-8b-instruct\",\n  \"prompt\": \"An apple a day keeps the doctor away.\",\n  \"tokens\": true,\n  \"token_ids\": true\n}", null, "application/json");
request.Content = content;
var response = await client.SendAsync(request);
response.EnsureSuccessStatusCode();
Console.WriteLine(await response.Content.ReadAsStringAsync());

Request Collapse all

Base URL

https://api.aleph-alpha.com

Auth

Bearer Token

Body required

{
  "model": "llama-3.1-8b-instruct",
  "prompt": "An apple a day keeps the doctor away.",
  "tokens": true,
  "token_ids": true
}

ResponseClear

Click the Send API Request button above and see the response here!