Skip to main content

Create Examples for an evaluation dataset

An example is the line item of a dataset for Evaluation. Each Example contains the input that the user could provide and the expected output that the AI logic should return. In real world applications, these Examples are collected from Annotation tools or from validated feedback from users.

Prerequisites for implementing a task​

Make sure you have a project with Pharia Studio SDK as a dependency as explained here.

Add the necessary dependencies​

from pharia_studio_sdk.evaluation import Example

Create a list of examples​

From How to implement a task, we have already created the input and output types for our task. Let's paste them here for simplicity.

class TellAJokeTaskInput(BaseModel):
    topic: str

class TellAJokeTaskOutput(BaseModel):
    joke: str

With these types defined, we can now create our list of examples as follows:

examples = [
    Example(
        input=TellAJokeTaskInput(topic="******"),
        expected_output=TellAJokeTaskOutput(joke="@@@@@"),
        metadata={
            "author": "Shakespeare"
        }
    ),
    # ...
]

As visible, it is possible to add any metadata to each example that can help with the process of evaluations.