Skip to main content

Project insights view for project managers

Role: project manager

As a project manager, you have access to the tasks and dataset insights view. From here, you can assign and access all tasks of a project and enter the dataset view.

Task View​

Role: project manager

As a project manager, you can see all tasks of a project and assign them to domain experts of the project’s workforce for annotating. For task assignment, you have two options: manual or automated assignment.

Manually assign a task​

TIP: Manual assignment of tasks allows you to assign domain experts to a task even if they are not part of the initial workforce of a project.

The more users you assign to a task, the more meaningful the data point annotations and the more reliable the quality metrics.

To manually assign a task:

  1. Click the user icon on the right side of the task.
    The workforce assigned to this project is displayed.

  2. Select one or more annotators.
    The task is assigned to the selected users and becomes available in the annotator’s project and task view.

Auto-assign tasks​

To avoid the manual effort of assigning individual tasks to specific users, you can use the auto-assign feature to automatically assign tasks to the project’s workforce.

All data points with fewer than three annotators assigned are assigned additional annotators randomly selected from those available workforce members who have not yet been assigned to that data point. If enough workforce members are available, each data point is assigned to three annotators, and the corresponding tasks are created.

To automatically assign a task:

  • Click Auto assign.
    The tasks are distributed and included in the annotators' project and task view.

View a task​

As a project manager, you can open a task and monitor assignment, progress status and annotation result.

To open a task:

  • Click the task in the task list view.
    The task is displayed.

On the top left, you see the task overview, listing the progress status, which can be one of the following:

  • Unassigned: no annotator is assigned to this task
  • Pending: annotators are assigned to this task but none has completed it yet
  • In progress: one or more annotators (but not all) have completed the task
  • Blocked: there is either a raised flag or an annotator disagreement
  • Completed: all annotators have completed the task and the data point has been aggregated

You can also see the source, which is the name of the uploaded file from which the data point was generated.

On the left, you see the following:

  • all the annotators who have been assigned to this data point
  • the progress status of their task, which can be "not started" or "completed", or which might highlight an issue, such as "Unclear instructions" or "Corrupt data point"
  • the label that the annotator assigned to the data point

In the middle you see the content of the data point.

On the top right you see the aggregated label, which is the final result of the aggregation method used and is used in the dataset export. Currently, we support aggregation based on majority decision.

On the right you see the task and label instructions that you defined during project creation.

Delete a task​

To delete a task:

  1. Click the task in the task overview.
    The task is displayed.

  2. Click Delete at the bottom right.
    A warning appears asking you for confirmation.

  3. Click Delete again.

The task is deleted from the project. If the delete task was assigned to annotators, it is removed from the annotators' task queues.

Dataset View​

As a project manager, you have access to the dataset view. Here, you gain a detailed insight into the data points and annotations that form the final dataset.

In the dataset view, you can view the following:

  • the creation and last modification dates
  • the number of total annotations
    The number of total annotations refers to a total count of annotations received from annotators across all data points.
  • overall dataset quality as a percentage
    This is calculated as annotations divided by number of data points.
  • label distribution over the entire dataset
  • average number of annotations per data point
    Refers to total number of annotations divided by the total number of data points.
  • average annotator agreement in percent over the entire dataset
    Refers to the average of all agreement scores across all data points
  • number of flagged data points
  • average task completion time

To get an even deeper insight into each data point and its annotations, click the tile Data points. All data points of the project are displayed in a list view.

In this view, you view for each data point:

  • its unique ID

  • the aggregated annotation

  • the inter-annotator agreement as a percentage
    The icon shown to the left of the value indicates whether there is a high enough agreement to decide for an aggregated label. The green check mark indicates a sufficiently high agreement among the annotators. The yellow warning sign indicates a medium agreement among the annotators, while the red cross indicates that the agreement between the respective annotators is low. In this case, access the data point and resolve the disagreement conflict.

  • any raised flag
    Flags can be raised for different reasons. Access the data point to inspect the flag and resolve the issue.

  • the assigned annotators

  • the last modification date

Resolving conflicts​

To resolve a conflict, click the task concerned to view the data point. The progress status of the data point is shown as “blocked”. Evaluate the annotators' decisions on the left. Refer to the task and label instructions on the right and/or collaborate with the workforce to assess what might be the correct decision. If needed, assign the task to further annotators.

  1. On the left, click the label to be enforced.
  2. Click Apply this annotation.
  3. In the Aggregated label tile on the top right, the aggregated label is highlighted.
  4. Click Resolve on the bottom right.

The enforced annotation is assigned as aggregated label to the data point in the dataset and the conflict is resolved.

Resolve a flag​

For tasks blocked by raised flags, you need to evaluate the annotators' raised flags and comments on the left. Refer to the task and label instructions on the right and/or collaborate with the workforce to assess what action is required.

In the case of a “corrupt data point flag”, assess whether the data point is indeed corrupted. This can have different reasons; for example:

  • the OCR pre-processing may have rendered the text unreadable, or
  • the chunking of text into single data points has rendered incorrect results and assigning a single label is not possible for the annotator. (In this case, delete the data point from the project.)

In case of an “unclear instructions” flag, the task or label instructions may not be clear to the annotator. In this case, rework the instructions to make them more clear and resolve the conflict.

To resolve a flag:

  1. Click the task concerned.
    The task view opens and the progress status of the task is shown as “blocked”. Assess the situation as described above.

  2. Optionally, you can click Delete on the bottom right. The data point is removed from the project.

  3. Optionally, you can click Resolve in the bottom right. The flag is removed from the data point in the dataset and the task can be further processed by the annotators.

Export datasets​

When all tasks are completed and there are no remaining conflicts or flags, the dataset can be exported as the ground truth. To do this:

  1. Navigate to the dataset view of the project you want to export.
  2. Click Download.
    The download dialog opens.
  3. Enter your preferred export name into the Export name field.
  4. Select your preferred export file format.
  5. Click Download dataset.

If you try to export even when there are pending tasks or unresolved conflicts, PhariaCatch exports only the part of the dataset that is free of conflicts and is therefore usable as ground truth.