E9:309 Advanced Deep Learning

Announcements       Syllabus       Grading       References       Slides      



When MW 4:30 - 6:00 PM
Where Microsoft Teams (Link: link)
Who Sriram Ganapathy
Office C 334 (2nd Floor)
Email sriramg aT iisc doT ac doT in
Teaching Assistants Jaswanth Reddy K, Prachi Singh, Akshara Sonam
Lab C 328 (2nd Floor)
Email {jaswanthk, prachisingh, aksharas} aT iisc doT ac doT in

Announcements

  • First class on October 5, 2020 3:30 PM.
  • Guidelines for monthly projects: Kit. It contains files to help you with the format for your project submissions.
  • Monthly Project 2 final submissions can be uploaded to this folder: Link
  • Follow the same formats for report and presentations from Monthly Project 1
  • Monthly Project 2 presentations dates: December 29, 30 (and 31)
  • Upload the Monthly Project 3 abstract here: link
  • Deadline for Monthly Project 3 abstract submissions: Jan 10, 2021.
  • Monthly Project 3 presentations date: 1st week of Feb. (TBA)
  • Final Exam will be on January 23, 2021 Afternoon. Same format as Mid-Term
Top      

Syllabus

  • Visual and Time Series Modeling: Semantic Models, Recurrent neural models and LSTM models, Encoder-decoder models, Attention models.
  • Representation Learning, Causality And Explainability: t-SNE visualization, Hierarchical Representation, semantic embeddings, gradient and perturbation analysis, Topics in Explainable learning, Structural causal models.
  • Unsupervised Learning: Restricted Boltzmann Machines, Variational Autoencoders, Generative Adversarial Networks.
  • New Architectures: Capsule networks, End-to-end models, Transformer Networks.
  • Applications: Applications in in NLP, Speech, Image/Video domains in all modules.
Top      

Grading Details

3 monthly research projects from three different domains (Speech/Audio, Text, Images/Videos, Biomedical, Financial, Chemical/Physical Sciences/Mathematical Sciences) 60%
Midterm exam 10%
Final exam 30%

Pre-requisites

  • Linear Algebra
  • Random Process
  • Basic Machine Learning/Pattern Recognition course
  • Good background in Python programming.
Top      

References

  • A significant portion of the material would come from research papers/tutorials in the domain.
  • Lecture notes in pdf format.
  • “Deep Learning”, I. Goodfellow, Y, Bengio, A. Courville, MIT Press, 2016. html
Top      

Slides

05-10-2020 Introduction. Setting the stage for the course.
slides video
07-10-2020 Recap of deep learning - Notations, model parameters, feed forward networks, learning rule with stochastic gradient descent, convolutional networks. Need for recurrence networks. Types of recurrence.
slides video notes
Reading Assignments An overview of GD optimization algorithms
Layer Normalization
Batch Normalization
12-10-2020 Recurrent neural networks : Forwards and Backward pass. Gradient propagation. Backpropagation through time (BPTT) algorithm. Vanishing gradients in Recurrent networks.
slides video notes
14-10-2020 Recap of RNNs, BackPropagation Through Time (BPTT), LSTMs
slides video notes
19-10-2020 Recap of RNNs, LSTMs, Bidirectional RNNs, Encoder-Decoder Models, Attention Networks
slides video notes
21-10-2020 Recap of Encoder-Decoder Models, Visualizing Attention, Multi-head Attention, Self-Attention, Transformers
slides video notes
28-10-2020 Self and multi-head attention, issues in RNN/LSTM, Introduction to transformer networks, transformer-encoder in detail.
slides video notes
Reading Assignments Neural Machine Translation By Jointly Learning to align and translate
Attention is all you need
Interesting Blogs Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)
The Illustrated Transformer
Animated RNN, LSTM and GRU
Introduction to Neural Machine Translation with GPUs
Attention and Augmented Recurrent Neural Networks
31-10-2020 Tutorial - 1: Regularization, Optimization and Pytorch basics.
video
02-11-2020 Transformer models in detail - encoder, self-attention and positional encoding.
slides video notes
04-11-2020 Transformer models in detail - encoder, self-attention and positional encoding.
slides video notes
09-11-2020 tSNE.
slides video notes
Reading Assignments Visualizing Data using t-SNE
More Sources Visualizing the Hidden Activity of Artificial Neural Networks
11-11-2020 Unsupervised representation learning, Boltzmann machine and restricted Boltzmann machine. Model parameters, conditional independence. Issues in RBM training
slides video notes
Reading Material Deep Generative Models
18-11-2020 Restricted Boltzmann machine training, approximating the negative phase with Gibbs sampling. Gaussian Bernoulli RBM - definiton and properties. Deep belief networks (DBNs).
slides video notes
Reading Assignment Restricted Boltzmann Machines for Collaborative Filtering
23-11-2020 Restricted Boltzmann machine training, Deep belief networks (DBNs) for initialization and visualization, Data Generation using RBMs, Variational Autoencoders
slides video notes
25-11-2020 Variational autoencoders (derivations of the loss functions). The Variational lower bound. Model assumptions and approximations.
slides video notes
Reading Assignment Auto-Encoding Variational Bayes
02-12-2020 Variational autoencoders examples. Generating data using VAEs. Introduction to generative adversarial networks. GANs - loss function
slides video notes
07-12-2020 Introduction to generative adversarial networks, GANs - loss function, min-max Game, Deep Convolutional GANs, Conditional GANs, CycleGANs,
slides video notes
09-12-2020 Explainable Deep Learning - Motivation, Understanding hierarchical representations in deep learning. Transfer learning and representations.
slides video notes
14-12-2020 Explainable Deep Learning - t-SNE embeddings for visualization, Understanding Deep Networks, Representations.
slides video notes
16-12-2020 Explainable Deep Learning - Architecture updates for interpretability, Improving CAM without compromising architecture, Relation between CAM and Grad-CAM, Using attention mechanism for explainability.
slides video notes
21-12-2020 Causality, Causal modeling, Structural Causal Equations, Causal inference and deep learning, Pruning based analysis of neural networks, Adversarial examples
slides video notes
23-12-2020 Causal inference, Pruning based approach to analyzing/compressing, Criterion in involved in identifying importance, Approximating gradients, Adversarial examples and learning, Explainability with distillation, LIME model
slides video notes
28-12-2020 Knowledge distillation, Knowledge distillation for explainability, Local Interpretable Model Agnostic Representation, LIME model, Future Research Directions, Capsule networks
slides video notes
30-12-2020 Future Research Directions, Problem with current deep learning networks, Capsule networks, Capsule vs Neurons, Routing algorithm, Understanding the capsule output
slides video notes
04-01-2021 Capsule networks, From a layer of neurons to layer of capsules, Capsule network performance, Automatic Sign Language Detection Task, Comparing capsule networks with other architectures, Deep learning on graphs
slides video notes
06-01-2021 Deep learning on graphs, Graph convolutional networks, Semi-supervised learning using GCN
slides video notes
13-01-2021 Modeling uncertainty in deep learning, Bayesian Deep Learning (Basics), Introduction to Gaussian processes, Allowing for noise in the model, Dropout and its Bayesian Interpretation
slides video notes
15-01-2021 Bayesian Deep Learning, Gaussian processes for Bayesian inference, Dropout and its Bayesian Interpretation, Obtaining the model uncertainity
slides video notes
Top