Masked Siamese Networks: Objective and Training

Similarity metric and predictions

To train the encoder, MSNs compute a distribution based on the similarity between a set of learnable prototypes $Q = \{q_1, q_2, ..., q_K\}$ (think of each prototype, $q_i$, as $d$-dimensional vectors, hence $Q$ is a $K\times d$ matrix) and each anchor, $z_i^m$, and target view, $z_i^+$, representation pair. The encoder is then penalized if both these distributions are different. More specifically, for an anchor representation, $z_i^m$, we compute its “prediction,” $p_i^m$ , by measuring the cosine similarity with prototypes.