Introduction to Similarity Maximization

We previously discussed how pretext task-based self-supervised pre-training is unsuitable for all downstream tasks. However, we also understood why pretext task-based pre-training is only sometimes suitable for all downstream jobs (because of the mismatch between what is being solved in the pretext task and what we need to be achieved by the transfer task). In this chapter, we will learn similarity maximization, a popular and commonly used self-supervised paradigm that addresses the limitations of pretext task-based self-supervised learning.

What do we want from pre-trained features?

Fundamentally, after the pre-training step, we want the trained features to satisfy two important properties:

• Capture semantics: We want them to represent how images relate to each other, such as whether a pair of images are similar and to what extent.

• Robustness: We want them to be robust or invariant to “nuisance factors” like noise, data augmentation, occlusions, etc.

A trivial solution

Given a neural network, $f(.)$, we want to learn features that are robust to data augmentation, that is, $f(T_1(X_i)) = f(T_2(X_i))$. Here, $T_j(.)$ is a data augmentation strategy, and $X_i$ is the input image.