Sparse Arrays

Introduction

Sparse data occurs when the data is predominantly empty or contains a small number of non-zero values compared to the overall size of the data set. Here are some examples of sparse data occurring in real-world scenarios:

• Image and video processing: Images and videos often contain a significant amount of empty or black pixels. Sparse matrices or compressed formats, such as Compressed Sparse Column (CSC) or Compressed Sparse Row (CSR) representations, are employed to store and process these visual data efficiently.

• Natural language processing (NLP): In NLP, text data can be represented using sparse vectors where each dimension corresponds to a unique word in a vocabulary. Since most documents contain only a small fraction of the vocabulary, sparse representations (e.g., the Term Frequency-Inverse Document Frequency model) are used.

• Recommender systems: Sparse matrices are commonly used to model user-item interactions in recommender systems. Users typically interact with only a small subset of items in a large catalog. Therefore, sparse representations are employed to store and process this data efficiently.

Note: Sparse doesn’t necessarily refer to zero values only. It can also refer to other values, such as np.nan for floats and None for other data types.

Instead of storing all the data points, which will be inefficient and consume excessive memory, sparse datasets are typically represented using specialized data structures that can effectively handle and exploit the sparsity. This leads to significant savings in memory and processing power by avoiding unnecessary computations on empty or zero values and focusing only on the non-zero values.

Note: A common question that comes up is the difference between sparse data and missing data. While sparse and missing data can involve many non-present values, the distinction is in how these non-present values are interpreted. In sparse data, the non-present values are considered meaningful (e.g., a user hasn’t rated a movie). In contrast, missing data represents an absence of information (e.g., we don’t know whether a user has rated a movie or not).

In this lesson, we’ll look at the data structures available in pandas that enable us to work effectively with sparse datasets.

SparseArray

A SparseArray is a one-dimensional array-like object designed for storing sparse data in pandas. By using a SparseArray, we only store the non-empty data and the locations of this data. This can lead to substantial memory savings when dealing with large datasets.

A SparseArray is a columnar data structure, and therefore it can be used as a column within a regular DataFrame. By having a single DataFrame that can contain a mixture of sparse and dense columns, operations like JOIN and group-by become more efficient because there is no need to convert back and forth between sparse and dense data structures.

Note: Before pandas version 1.0, there were SparseSeries and SparseDataFrame classes to handle sparse data. However, due to the introduction of the SparseArray to store sparse data in newer versions, these two classes have been deprecated.

Suppose we’ve the following sparse dataset for movie ratings scored between 1 and 5 by several viewers: