Travel Destination Recommender

Imagine you work at a travel agency, and you have a database full of travelers' holiday pictures and reviews. You want to recommend possible destinations for a customer's next trip, taking both types of data into account. With Luminous-explore, you can semantically embed multimodal data, which can be used for a variety of use cases such as search system, classification or recommendation engines.

In the following example, we give a rough outline of what a travel recommendation engine based on semantic embeddings could look like. First, we import the necessary dependencies to build the application and set up the client to interact with the Aleph Alpha API.

import math
import statistics
import os
from typing import List, Optional

from aleph_alpha_client import (
    Client,
    Image,
    Prompt,
    SemanticEmbeddingRequest,
    SemanticRepresentation,
)

# If you are using a Windows machine, you must install the python-dotenv package and run the two below lines as well.
# from dotenv import load_dotenv

# load_dotenv()

client = Client(token=os.getenv("AA_TOKEN"))

Using a full database would be beyond the scope of this example use case. As a result, the two examples below, each with an image and a review, serve as potential travel destinations. 

Example 1: New York

I visited New York last year and was blown away by the sheer number of different cultural activities in the city!

Example 2: Mallorca

Mallorca is a great place to visit. In the summer, you can enjoy many beautiful beaches. During the other seasons, you can go for exciting hikes.

Let's first create a class called TravelLocation. This class should do three things:

• Load the data.
• Embed the texts and images.
• Compute the cosine similarity between itself and the user input to find out how well it fits the user's preferences.

class TravelLocation:
    def __init__(
        self,
        name: str,
        texts: List[str] = [],
        image_urls: List[str] = [],
    ):
        self.name = name
        self.texts = texts
        self.image_urls = image_urls
        self.embeddings = self.embed_texts_and_images()

    def embed_texts_and_images(self):
        text_prompts = [Prompt.from_text(text) for text in self.texts]
        image_prompts = [Prompt.from_image(Image.from_image_source(url)) for url in self.image_urls]

        return [self._embed(prompt) for prompt in text_prompts + image_prompts]

    def _embed(self, prompt: Prompt):
        params = {
            "prompt": prompt,
            "representation": SemanticRepresentation.Symmetric,
            "compress_to_size": 128,
        }
        request = SemanticEmbeddingRequest(**params)
        response = client.semantic_embed(request=request, model="luminous-base")
        return response.embedding

    @staticmethod
    def compute_cosine_similarity(embedding_1, embedding_2):
        sumxx, sumxy, sumyy = 0, 0, 0
        for i in range(len(embedding_1)):
            x = embedding_1[i]
            y = embedding_2[i]
            sumxx += x * x
            sumyy += y * y
            sumxy += x * y
        return sumxy / math.sqrt(sumxx * sumyy)
    
    def score_input(self, input: str):
        if input.startswith('http'):
            embedding_input = self._embed(prompt=Prompt.from_text(Image.from_image_source(image_url)))
        else:
            embedding_input = self._embed(prompt=Prompt.from_text(input))
            
        
        scores = (
            self.compute_cosine_similarity(embedding_input, embedding_element)
            for embedding_element in self.embeddings
        )
        return statistics.mean(scores)

When instantiating a TravelLocation-object with a text and/or image to embed, we automatically embed the inputs with the embed_texts_and_images-function using the semantic_embed-endpoint from the Aleph Alpha API in the _embed-function. If you would like to know more about the parameters for semantic embeddings, please check out the HTTP API or the blog-post about Luminous-explore.

travel_locations = [
    TravelLocation(
        name="New York",
        texts=[
            "I visited New York last year and was simply blown away by the sheer number of different cultural activities the city!",
        ],
        image_urls=["https://live.staticflickr.com/4091/5013079844_b1eeea34f5_b.jpg"],
    ),
    TravelLocation(
        name="Mallorca",
        texts=[
            "Mallorca is a great place to visit. In the summer, you can enjoy many beautiful beaches. In the other seasons, you can go for exciting hikes.",
        ],
        image_urls=["https://live.staticflickr.com/5539/30697007472_186edcda49_b.jpg"],
    ),
]

Finally, we want to recommend a travel destination given the user's query. To do so, we define a recommend-function, that uses the score_input-function of the TravelLocation-class to rank and recommend a travel destination.

def recommend(travel_locations: List[TravelLocation], text: str = None, image_url: str = None):
    assert bool(text) != bool(image_url), "Must provide either text or image_url."
    results = [
        {
            "location": travel_location.name,
            "score": travel_location.score_input(input=text if text else image_url),
        }
        for travel_location in travel_locations
    ]
    sorted_results = sorted(results, key=lambda d: d["score"], reverse=True)
    print(
        "We recommend you to travel to {location}!".format(
            location=sorted_results[0]["location"]
        )
    )
    return sorted_results

preference = "I love to visit cities around the world. I love exploring shops, restaurants and museums!"
recommendation = recommend(travel_locations, text=preference)

Example Output:

We recommend you to travel to New York!

Theoretically, you could also upload an image to get a recommendation or even combine text and image inputs. You can even put a conversational layer on top of your travel recommendation engine!