Understanding Retrieval-Augmented Generation (RAG)

LLMs are limited by the data they are trained on and may not always have access to the most recent information. Additionally, a lack of access to external data leads to inaccurate results and hallucinations. Techniques such as prompt engineering and model fine-tuning are used to work around these limitations. Other techniques, such as sending additional context to the LLM to help it derive the right answer, are also available.

Each has its pros and cons. For example, prompt engineering (like one-shot prompting) is cost-effective but limited in scope. We can pass additional documents or information to LLM, but these techniques might be impacted by LLM token limitations and cost.

What is retrieval-augmented generation (RAG)?

Vector databases can store data in specialized form (vectors) that LLMs can access. RAG works hand-in-hand with vector databases. It aims to overcome LLM limitations by allowing them to dynamically retrieve relevant knowledge while generating responses. The idea is to have a component that can retrieve relevant information from an external knowledge base and then feed that as additional context to the language model during the response generation process. This combines language models with retrieval systems to enhance the model’s generation capabilities with external knowledge.

Key components in a RAG solution

A RAG solution consists of multiple components working together.

• Large language model (LLM): This component generates text. LLMs are trained on massive amounts of text data.

• Embedding model: It converts textual information from the data source and the user prompt into numerical representations known as embeddings. These embeddings capture the semanticIt is the meaning and interpretation of words, phrases, or text in a given context. meaning of the text, allowing for efficient comparison and retrieval of similar information.

• Data source(s): The RAG queries this knowledge base to find relevant information for the LLM. It can be a collection of web documents, a specific domain-focused corpus (like medical journals for a healthcare application), or even an organization’s internal knowledge repository. The data source’s quality and relevance significantly impact an RAG system’s accuracy.

• Data ingestion pipeline: It takes data from external data sources, cleans and preprocesses it, and feeds it into the embedding model. This may involve removing irrelevant information, formatting the data consistently, and ensuring its quality.

• Vector database: It stores high-dimensional numerical representations (embeddings) generated by the embedding model. Augmenting semantically similar results from the vector database improves the accuracy of an RAG system.

• Application: The user-facing interface interacts with the RAG system.

Benefits of RAG

Some of the key benefits of adopting RAG include:

• Improved factual accuracy: By grounding the LLM in factual information from external sources, RAG significantly reduces the risk of the model generating incorrect or misleading information.

• Access to up-to-date knowledge: Unlike LLMs, whose knowledge is limited by their training data, RAG allows the model to access and leverage the most recent information through the knowledge base.

• Transparency and trust: RAG systems can often reveal the sources used by the LLM to generate its response. This transparency builds trust in the system and allows users to verify the information presented.

RAG workflow

Here is a high-level diagram that represents the common sequence of steps in an RAG application:

1. It starts when the user sends a query to the application. The query vector embedding is calculated using an embedding model.

2. Retrieval (R): The application uses the query vector embedding to perform a semantic search against the vector database. Instead of performing a traditional keyword-based search, the system finds stored embeddings that are semantically similar to the query.

3. Augment (A): The retrieved search results are combined with the original query. This process typically involves formatting the retrieved content into a structured prompt, ensuring the LLM receives the most relevant context and the user’s query.

4. Generation (G): The LLM processes the augmented input, incorporating the user’s original query and the additional context retrieved from the vector database. It then generates a response likely to be contextually accurate and factually relevant.