Setting up user-defined steering concepts

Determine a good steering strength

First, you need to figure out a good steering strength for your checkpoint. This is usually done by evaluating a set of completions balancing the impact of the steering examples with the correctness of the output.

At the moment, the steering strength can be set only in the worker configuration and changing it requires restarting the worker. These limitations might get lifted in a future release.

The following table shows an example:

Checkpoint Strength

llama-3.1-8b-instruct

0.062

Setting the default steering strength for your worker

If you deployed llama-3.1-8b-instruct with our default configuration, steering is already enabled. If you use a custom , worker deployment, you need to overwrite inference-worker.checkpoints in values.yaml and set a default steering strength to enable steering.

inference-worker:
  checkpoints:
    ...
    - generator:
        type: "luminous"
        pipeline_parallel_size: 1
        tensor_parallel_size: 1
        tokenizer_path: "llama-3.1-8b-instruct/tokenizer.json"
        weight_set_directories: ["llama-3.1-8b-instruct"]
        steering:
          default_in_context_vectors_strength: 0.062
      queue: "llama-3.1-8b-instruct"
      replicas: 1
      modelVolumeClaim: "models-llama-3.1-8b-instruct"
      version: 0
      models:
        llama-3.1-8b-instruct:
          ...

The Helm chart must now be redeployed for the changes to take effect.