Semantic Routing

LLMs handling diverse user queries require efficient routing mechanisms. Imagine a single LLM trained on a massive dataset covering various domains like finance, health, literature, and travel. While the LLM can access this information, directly feeding a user query might not always lead to the most relevant response.

Routing helps us bridge this gap by directing user queries to specific sub-models or prompts that are best equipped to handle them. This ensures a more focused and informative response for the user.

What is routing?

Routing, in the context of LLMs, is the process of directing a user query to the most appropriate sub-model or prompt within the larger LLM architecture. This sub-model or prompt is likely to have been trained on a specific domain or task, allowing it to generate a more accurate and relevant response.

There are several ways to implement routing in LLMs. We will explore two common methods:

1. Semantic routing: This method leverages semantic similarity between the user query and pre-defined sets of questions or prompts from different domains.

2. Routing with LLM-based classifier: Here, a separate LLM classifier is trained to categorize the user query into a specific domain before routing it to the corresponding sub-model.

Semantic routing

Semantic routing is a data-driven approach that utilizes the semantic similarity between the user query and pre-defined prompts or questions from various domains. Here’s a breakdown of how it works:

• Pre-defined prompts and questions: We define sets of questions or prompts specific to each domain we want to handle. For example, we might have a set of questions related to personal finance, another for book reviews, and so on.

• Embedding user query and prompts: We use an embedding model to convert the user query and pre-defined prompts from each domain into numerical representations. These embeddings capture the semantic meaning of the text.

• Similarity calculation: We calculate the cosine similarity between the user query embedding and the embeddings of each pre-defined prompt set. Cosine similarity measures how similar two vectors are in a high-dimensional space.

• Routing based on highest similarity: The user query is routed to the domain with the highest cosine similarity. This indicates the closest semantic match between the query and the prompts from that domain.

• Prompt selection and response generation: The LLM uses the selected domain-specific prompt along with the user query to generate the final response.