The self-contained Deep learning course launched by experienced professional engineers and researchers of the AI Summer team.

docker.tar.gz

Pytorch

GANS

This course is an accumulation of well-grounded knowledge and experience in deep learning. It provides you with the basic concepts you need in order to start working with and training various machine learning models. 

You will cover both basic and intermediate concepts including but not limited to: convolutional neural networks, recurrent neural networks, generative adversarial networks as well as transformers.

After completing this course, you will have a comprehensive understanding of the fundamental architectural components of deep learning. Whether you’re a data and computer scientist, computer and big data engineer, solution architect, or software engineer, you will benefit from this course.

Introduction to Deep Learning & Neural Networks

## Concerns on SGD

This basic version of SGD comes with some limitations and problems that might negatively affect the training.



1. If the loss function changes quickly in one direction and slowly in another, it may result in a high oscillation of gradients making the training progress very slow.

2. If the loss function has a local minimum or a **saddle point**, it is highly likely that SGD will be stuck there without being able to “jump out” and proceed in finding a better minimum. This happens because the gradient becomes zero, so there is no update in the weight whatsoever.


> A **saddle point** is a point on the surface of the graph of a function where the slopes (derivatives) are all zero but which is not a local maximum of the function.

# Concerns on SGD

This basic version of SGD comes with some limitations and problems that might negatively affect the training.



1. If the loss function changes quickly in one direction and slowly in another, it may result in a high oscillation of gradients making the training progress very slow.

2. If the loss function has a local minimum or a **saddle point**, it is highly likely that SGD will be stuck there without being able to “jump out” and proceed in finding a better minimum. This happens because the gradient becomes zero, so there is no update in the weight whatsoever.


> A **saddle point** is a point on the surface of the graph of a function where the slopes (derivatives) are all zero but which is not a local maximum of the function.

Discover the most frequently-used alternatives of gradient descent and the intuition behind them.

Learn Deep Learning

Neural Networks

Training Neural Networks

Convolutional Neural Networks

Recurrent Neural Networks

Autoencoders

Generative Adversarial Networks

Attention and Transformers

Graph Neural Networks

Conclusion

Final Quiz

Popular Optimization Algorithms

Concerns on SGD