Backpropagation Algorithm

Neural Networks (NN) are non-linear classifiers that can be formulated as a series of matrix multiplications. Just like linear classifiers, they can be trained using the same principles we followed before, namely the gradient descent algorithm. The difficulty arises in computing the gradients.

But first things first.

Let’s start with a straightforward example of a two-layered NN, with each layer containing just one neuron.

Notations

• The superscript defines the layer that we are in.
• $o^L$ denotes the activation of layer L.
• $w^L$ is a scalar weight of the layer L.
• $b^L$ is the bias term of layer L.
• $C$ is the cost function, $t$ is our target class, and $f$ is the activation function.

Forward pass

Our lovely model would look something like this in a simple sketch: