Using Embeddings

Traditional methods of collecting data didn't capture relationships between words and as a result, they left out the exploration of linguistic richness. Embeddings capture a lot that was missed before and much more. In order to understand how they are used let's begin by first understanding the models that support embeddings.

Getting text embeddings using the OpenAI API

At the center of embedding, is an embedding model. When using the OpenAI API, you should use either the text-embedding-3-small or text-embedding-3-large embedding models. Compared to previous models, these offer lower costs, higher multilingual performance, and new parameters to control the overall size.

Here are the general steps for generating embeddings.

1. The process of obtaining embeddings for your text data involves interfacing with the OpenAI API, specifically targeting the embeddings endpoint. This interaction begins by preparing the text strings that you wish to analyze. Each string should be a discrete piece of text, such as a sentence or a short paragraph, that you want to convert into a machine-readable vector.

2. Once your text is ready, you will need to select an appropriate embedding model provided by OpenAI. As we've mentioned, you should almost always use text-embedding-3-small or text-embedding-3-large since these are the latest models at the moment of writing.

3. With the text and model ID in hand, the next step is to formulate a request to the OpenAI API. This request is structured as a call to the embeddings API endpoint, where you pass in your text and specify the model ID. In the case of text-embedding-3-small, the request will instruct the API to use this model to generate your embeddings.

4. Upon issuing the request, the OpenAI service will process your text through the specified model. The model effectively digests the text and produces a numerical vector representation—this is your embedding. This embedding captures the semantic essence of your text in a form that can be readily processed by algorithms.

5. The response from the OpenAI API will include the generated embeddings within its payload. Typically, this is a JSON structure, where the embedding itself is an array of numbers within the data field. Once you receive the response, you should extract this array. This extraction can be done programmatically by parsing the JSON and accessing the relevant fields.

6. After extraction, it’s essential to store the embeddings in a suitable format for later use. Depending on your application, this could mean saving them to a database, writing them to a file, or keeping them in memory for immediate processing. The stored embeddings represent your text in a transformed state, ready for tasks like similarity comparison, clustering, or as input features for machine learning models.

Try it out: Generating embeddings for text

This task will demonstrate how to use the OpenAI API to generate embeddings for text. We will use the text-embedding-3-small model to generate embeddings for a short text.

First, we'll break it down step-by-step. At the end, we'll provide the full code for you to try.

Step 1: Import the OpenAI library and set the API key.