Skip to main content

2 posts tagged with "models"

View All Tags

· One min read
Andreas Hartel

Meta has recently released their version 3.1 of the Llama family of language models. With worker version api-worker-luminous:2024-08-15-0cdc0 of our inference stack worker, we now support these models in our inference stack as well. However, we do not provide the model weights, as usual, in our JFrog Artifactory but instead ask you to download them from huggingface where Meta provides them directly.

To make use of the new models, these are the steps you need to follow:

  1. Download the model weights from huggingface, for example using this command:
huggingface-cli download --local-dir /path/to/Meta-Llama-3.1-8B-Instruct meta-llama/Meta-Llama-3.1-8B-Instruct
  1. Configure your worker with our new configuration format:
edition = 1

[queue]
url = "<your API URL>"
token = "<your API token>"
checkpoint_name = "llama-3.1-8B-instruct"

[monitoring]
metrics_port = 4000
tcp_probes = []

[generator]
type = "luminous"
pipeline_parallel_size = 1
tensor_parallel_size = 1
huggingface_model_directory = "/path/to/Meta-Llama-3.1-8B-Instruct"
tokenizer_path = "/path/to/Meta-Llama-3.1-8B-Instruct/tokenizer.json"
weight_set_directories = []

Notice that the huggingface_model_directory is the path where you downloaded the model weights to. This field is only supported in the new configuration format, which has been introduced in this previous blogpost.

· 3 min read
Niklas Finken

We are happy to announce that we have improved our luminous-control models. These new models are more instructable and perform better across a variety of tasks.

The new model versions are:

  • luminous-base-control-20240215
  • luminous-extended-control-20240215
  • luminous-supreme-control-20240215

You can access the old models at:

  • luminous-base-control-20230501
  • luminous-extended-control-20230501
  • luminous-supreme-control-20230501

Until March 4th, the default luminous-*-control will continue to point to the old models. Thereafter, you will automatically access the updated models.

While we see improved performance across the board, you can check for your use-case by changing the model to the latest model name and trying it out. You can also experiment with a smaller model size and see if it performs just as well or even better while offering faster response times. If the performance is not as expected, you can pin to the old model name to maintain the current behavior.

What's New

These new models are even better at following instructions. We have achieved this by fine-tuning on high quality instruction samples.

Simply prompt these new models like so:

{all the content and instructions relevant to your query}

### Response:

These models are prticularly good at taking into account a given document during generation. This is particularly helpful for question-answering and summarization use-cases. They are significantly less prone to hallucinations when supplied with the proper context.

Question: Who was commander of the Russian army?
Answer the question using the Source. If there's no answer, say "NO ANSWER IN TEXT".

Source: The Battle of Waterloo was fought on Sunday 18 June 1815, near Waterloo (at that time in the United Kingdom of the Netherlands, now in Belgium). A French army under the command of Napoleon was defeated by two of the armies of the Seventh Coalition. One of these was a British-led coalition consisting of units from the United Kingdom, the Netherlands, Hanover, Brunswick, and Nassau, under the command of the Duke of Wellington (referred to by many authors as the Anglo-allied army or Wellington's army). The other was composed of three corps of the Prussian army under the command of Field Marshal von Blücher (the fourth corps of this army fought at the Battle of Wavre on the same day). The battle marked the end of the Napoleonic Wars. The battle was contemporaneously known as the Battle of Mont Saint-Jean (France) or La Belle Alliance ("the Beautiful Alliance" – Prussia).

### Response: