Tokenize
POST/tokenize
Tokenize a prompt for a specific model. To obtain a valid model, use GET
/models_available
.
Request
- application/json
Body
required
model stringrequired
prompt stringrequired
tokens booleanrequired
token_ids booleanrequired
Responses
- 200
OK
- application/json
- Schema
- Example (from schema)
Schema
tokens string[]
token_ids integer[]
{
"tokens": [
"ĠAn",
"Ġapple",
"Ġa",
"Ġday",
"Ġkeeps",
"Ġthe",
"Ġdoctor",
"Ġaway",
"."
],
"token_ids": [
560,
34438,
246,
1966,
18075,
275,
8809,
3476,
17
]
}
Loading...