Deep Learning - Theory and Practice

By Sriram Ganapathy

Course Content

Basics of pattern recognition, Neural networks. Introduction to deep learning, convolutional networks, Applications in audio, image and text. All coding done in Python.

Contact Details

Instructor Sriram Ganapathy
Office Electrical Engineering C 334 (2nd Floor)
Email deeplearning doT cce2019 aT gmail doT com
Teaching Assistants Shreyas R, Purvi A (Electrical Engineering C 328 (2nd Floor))

Textbooks

  • “Pattern Recognition and Machine Learning”, C.M. Bishop, 2nd Edition, Springer, 2007.
  • “Neural Networks”, C.M. Bishop, 2nd Edition, Springer, 1995.
  • “Deep Learning”, I. Goodfellow, Y, Bengio, A. Courville, MIT Press, 2016. html

Pre-requisites

  • Random Process/Probablity and Statistics - Recommended Book "Probability and Statistics" by Stark and Woods. Video Lectures
  • Linear Algebra/Matrix Theory - Recommended book "Linear Algebra", by G. Strang - Author's webpage for Videos and Text
  • Basic Programming with Python - numpy based array and matrix operations.
  • Calculus

Announcements

  • Midterm will be on March 7, 2019 in class hours.
  • No class on the election day (April 18, 2019).
  • Final exam will be on May 2, 2019, 6-7:45 pm. Venue: Lecture Hall 10, Lecture Hall Complex, Center for Continuing Education (CCE), near Prakruti Cafe, Indian Institute of Science, Bangalore-560012.

Slides





Code for CNN




10-01-2019     		Introduction to Deep Learning Course. Examples. Roadmap of the course. slides
17-01-2019   
Basics of Machine Learning - Decision and Inference Problems, Joint probability and posterior probabilities. Likelihood and priors. Loss matrix. Rule of maximum posterior probability. Loss function for regression.

slides
31-01-2019   
Matrix Derivatives. Maximum Likelihood estimation and Gaussian Example. Linear Models for Classification. Perpendicular distance of a point from a surface. Logistic regression.
Ref - PRML, Bishop, Chap. 3. Appendix C

slides
07-02-2019    Logistic regression two class motivation. Posterior probability, sigmoid function, properties. Maximum likelihood for two class logistic regression. Cross entropy error for $K$ classes. Logistic regression for K classes, softmax function. Non-convex optimization (local and global minima), Gradient Descent - motivation and algorithm.
Ref - PRML, Bishop, Chap. 4, Sec. 4.2 		slides
14-02-2019    Summary of previous lectures. Gradient descent for $K$ class logistic regression. Implementing logistic regression for MNIST dataset. Perceptron Model and motivation. Introduction to single hidden layer neural networks. Training and Validation data sets. Logistic Regression Code Discussion.

Ref - PRML, Bishop, Sec. 4.2 and NN, Bishop, Sec. 3.5, 7.5

Code for Logistic Regression

slides

15-02-2019 First Assignment posted - Due on Feb 24, 2019. Assignment-1 (with submission instructions)
21-02-2019 Perceptron and 1 Hidden Layer Neural Networks. Non-linear separability with hidden layer network. Type of hidden layer and output layer activations - sigmoid, tanh, relu, softmax functions. Error functions in MLPs.
Ref - NN, Bishop. Sec. 3.5, Sec. 4.8
slides
28-02-2019 Backpropagation in multi-layer deep neural networks. Universal approximation properties of single hidden layer networks. Need for depth. The trade-off between depth and width of networks. Representation learning in DNNs. Hierachical data abstractions. Example in Images
Ref - "Deep Learning", I. Goodfellow. Chap. 6
https://arxiv.org/abs/1311.2901
slides
Code for Single Layer NN

07-03-2019 Midterm Exam posted - Due by 19:45 hrs (90 minutes). DL2019_Midterm1

14-03-2018 Convolutional neural networks. Kernels and convolutional operations. Maxpooling and subsampling. Backpropagation in CNN layers. CNN example for MNIST.
Ref - "Deep Learning", I. Goodfellow. Chap. 9

slides
18-03-2019 Second assignment due on March 30 (Can attempt CNN questions after next class).
HW2.pdf
21-03-2019 Solutions to Midterm 1 Posted.
DL2019_Midterm1_Solutions
21-03-2018 Back Propagation in CNNs. Choice of Kernels and convolutional operations. CNN example for MNIST. Midterm Exam Discussion.
Ref - "Deep Learning", I. Goodfellow. Chap. 9

28-03-2019 Recurrent neural networks, back propagation in recurrent neural networks. Different recurrent architectures - teacher forcing networks, encoder/decoder networks, bidirectional networks.
Ref - "Deep Learning", I. Goodfellow. Chap. 10

slides
04-04-2019 Issue of Vanishing and Exploding Gradients, Long Short Term Memory Networks (LSTM), Attention Mechanism in Neural Networks
Ref - "Deep Learning", I. Goodfellow. Chap. 10

slides
11-04-2019 Advancted Topics, Network in Network, Convolutional LSTM networks, Unsupervised learning, autoencoders, adversarial learning
Ref - "Deep Learning", I. Goodfellow. Chap. 16

slides
25-04-2019 Applications of Deep learning for natural language processing, image processing and speech processing.
slides

26-04-2019 Third assignment posted, due on May 8.
HW3.pdf