| When | MW 3:30 - 5:00 pm |
| Where | EE B308 |
| Who | Sriram Ganapathy |
| Office | C 334 (2nd Floor) |
| sriramg aT iisc doT ac doT in | |
| Teaching Assistant | Prachi Singh |
| Lab | C 328 (2nd Floor) |
| prachisingh aT iisc doT ac doT in |
Announcements
- MLSP practice for Final Exam Posted
- MLSP Final Exam Nov. 29 [130pm - 430pm], B308, Classroom
- MLSP Project Final Evaluation Dec. 10 [C313 Opp to classroom]. 930am-100pm
- Prepare a 10 minute presentation containing mostly your work done for the project including novelty. Presentation should clearly describe your contributions. A maximum of 10 slides is only permitted for a single person project or 15 slides for two person project. Slides should be sent by email by Dec. 10th morning 8am.
- A two page two column report on the work done for the project should be submitted by Dec. 9th latest by 4pm. Students are encouraged to use the overleaf template of ICML. link
Syllabus
- Introduction to real world signals - text, speech, image, video.
- Feature extraction and front-end signal processing - information rich representations, robustness to noise and artifacts.
- Basics of pattern recognition, Generative modeling - Gaussian and mixture Gaussian models.
- Discriminative modeling - support vector machines, neural networks and back propagation.
- Introduction to deep learning - convolutional and recurrent networks, attention in neural networks, pre-training and practical considerations in deep learning, understanding deep networks.
- Deep generative models - Autoencoders, Boltzmann machines, Adverserial Networks, Variational Learning.
- Applications in NLP, computer vision and speech recognition.
Grading Details
| Assignments | 15% |
| Midterm exam. | 20% |
| Final exam. | 35% |
| Project | 30% |
Pre-requisites
- Must - Random Process/Probablity and Statistics
- Must - Linear Algebra/Matrix Theory
- Preferred - Basic Digital Signal Processing/Signals and Systems
Textbooks
- “Pattern Recognition and Machine Learning”, C.M. Bishop, 2nd Edition, Springer, 2011.
- “Neural Networks”, C.M. Bishop, Oxford Press, 1995.
- “Deep Learning”, I. Goodfellow, Y, Bengio, A. Courville, MIT Press, 2016. html
References
- “Deep Learning : Methods and Applications”, Li Deng, Microsoft Technical Report.
- “Automatic Speech Recognition - Deep learning approach” - D. Yu, L. Deng, Springer, 2014.
- “Machine Learning for Audio, Image and Video Analysis”, F. Camastra, Vinciarelli, Springer, 2007. pdf
- Various Published Papers and Online Material
- Python Programming Basics pdf
Slides
| 05-08-2019 | Introduction to real world signals - text, speech, image, video. Learning as a pattern recognition problem. Examples. Roadmap of the course. |
slides | ||
| 12-08-2019 | Basics of Natural Language Processing - token, document and corpus. TF-IDF features. Language modeling. Smoothing and back-off. Introduction to audio signal processing. Discrete Fourier Transform, Short Term Fourier Transform. Information Extraction Book (Chapter 6) - TF-IDF Stanford Reading Material - Language Modeling |
Weblink Weblink slides |
||
| 19-08-2019 | Short-term Fourier Transform considerations. Mel-frequency cepstral coefficient (MFCC) features. Image processing - filtering, convolutions. Matrix derivatives. Dimensionality Reduction - Principal Component Analysis Columbia Univ. STFT Tutorial PRML - Bishop (Appendix C) |
Weblink slides |
||
| 21-08-2019 | Unsupervised dimensionality reduction using Principal Component Analysis. Maximum variance formulation. Solution using eigenvectors of data covariance matrix. Minimum error formulation. Whitening and standardization. PRML - Bishop (Chapter 12.1) PRML - Bishop (Chapter 4.1.4) |
slides |
||
| 26-08-2019 | PCA for high dimensional data. Supervised dimensionality reduction using linear discriminant analysis (LDA). Fisher discriminant. Solution for 2 class LDA. Multi-class LDA. Comparison between PCA and LDA. PRML - Bishop (Chapter 4.1.4) PRML - Bishop (Chapter 1.5) |
slides |
||
| 28-08-2019 | Introduction to basics of decision theory. Inference and decision problems. Prior, likelihood and posterior. Maximum-a-posteriori decision rule for two class example. Decision theory for regression. MMSE estimation. Multi-variate Gaussian Modeling. Intrepretation of Covariance. Diagonal and Full Covariance. Maximum Likelihood estimation of mean and covariance.
PRML - Bishop (Chapter 1.6) Further Reading |
slides |
||
| 04-04-2019 | Short-comings of single Gaussian modeling. Introduction to mixture Gaussian modeling. Properties and parameters. Expectation Maximization algorithm - auxillary Function, proof of conververgence. Ref - Tutorial on GMMs Proof of EM algorithm |
slides |
||
| 09-09-2019 | Expectation Maximization Algorithm for GMMs. Initiatialization using K-means. Other Considerations in GMMs. GMM example for unsupervised clustering. Ref - EM algorithm for GMMs |
slides |
||
| 11-09-2019 | Assignment #1. Due on 23-09-2019. Analytical part submitted in class. Coding part submitted via mlsp19.iisc aT gmail doT com. |
HW2 | Audio | Text |
| 11-09-2019 | GMM Intialization. Other Considerations in GMMs. Factor Analysis. EM Algorithm for Factor Analysis. Applications. |
slides |
||
| 16-09-2019 | Non-negative Matrix Factorization. Model formulation. Learning the parameters. Applications in audio source separation. Ref - Lee Paper on NMFs Ref - Audio Applications For NMF |
slides |
||
| 23-09-2019 | Linear Regression revisited. Maximum likelihood formulation and equivalence to least squares error. Regularized least squares PRML - Bishop (Chapter 3) |
slides |
||
| 25-09-2019 | Mid-Term #1.
|
|||
| 27-09-2019 | Decomposition of total loss into bias, variance and noise. Bias variance tradeoff in Regularized linear regression. Linear models for classification. PRML - Bishop (Chapter 3,4) |
|||
| 30-09-2019 | Probablistic Linear Models for Classification - Logistic Regression. Motivation and formulation for 2-class case and K-class case. Comparison with linear models for classification. Regularized least squares revisited - Primal and dual form. Optimization in the dual space. Introduction kernel function and Gram matrix. PRML - Bishop (Chapter 4,6) |
slides | ||
| 02-10-2019 | Assignment #3. Due on 14-10-2019. Analytical part submitted in class. Coding part submitted via mlsp19.iisc aT gmail doT com. |
HW3 | Image | Text |
| 02-10-2019 | Properties of kernel functions. Rules for constructing kernels. The RBF kernel. Maximum margin classifiers - problem formulation for linearly separable case. Optimization fundamentals - primal and dual problems, strong duality, KKT conditions. PRML - Bishop (Chapter 6, 7) Introduction to convex optimization - Boyd (Chapter 5) |
Weblink to the book |
||
| 09-10-2019 | Maximum margin classifiers (non-overlapping condition), primal and dual. Definition of support vectors. Complimentary slackness and KKT conditions for SVM PRML - Bishop (Chapter 7) |
slides | ||
| 14-10-2019 | Maximum margin classifiers (Overlapping condition), slack variables KKT conditions. Applications of SVMs. Support Vector Regression. PRML - Bishop (Chapter 7) |
slides | ||
| 16-10-2019 | Introduction to artificial neural networks - extension of kernel machines. Perceptron model. Multi-layer perceptron. Activation Functions. Input-output Mapping. NNPR - Bishop (Chapter 3,4) |
slides | ||
| 18-10-2019 | Forward pass in MLP. Backpropagation algorithm - recursion. Choice of hidden layer activation function. NNPR - Bishop (Chapter 4) |
|||
| 21-10-2019 | Cross entropy for two class. Expected cross entropy loss and posterior probability estimation. General condition on error function for outputs to be posterior probability. Weight learning - gradient descent method. Properties of gradient descent using quadratic approximation. Learning rate parameter. NNPR - Bishop (Chapter 6,7) |
|||
| 23-10-2019 | Computational complexity in Gradient Descent. Definition of Jacobian and Hessian matrices. Choice of error function. Mean square and conditional expectation. Accelerating Gradient Descent Method, Momentum, Adagrad and Adam Optimizers. NNPR - Bishop (Chapter 6) Overview of Learning Algorithms |
slides | 25-10-2019 | Assignment #4. Analytical part submitted in class and codes submitted via mlsp19.iisc aT gmail doT com (04-11-2019).. |
HW4 |
| 28-10-2019 | Bias-variance tradeoff in neural networks. Improving generalization in deep learning. Regularization - weight decay, dropout strategy, training with noise. Early stopping. Committee of Neural networks. NNPR - Bishop (Chapter 9) |
slides | ||
| 30-10-2019 | Introduction to deep learning. Depth versus Width. Intuition behind deep representation learning. Folding analogy of deep learning. Convolutional operations in deep neural networks.
number of linear regions | slides | ||
| 04-11-2019 | Convolutional Neural Networks, Computation of convolutions. Number of parameters. Advantages over deep neural networks. Pooling and subsampling. Backpropagation in convolution. Introduction to recurrence operations in modeling sequence data. Deep Learning Book - Goodfellow et al. (Chapter 9) |
slides | ||
| 06-11-2019 | Recurrent Neural Networks. Backprogation in time. Problem of vanishing gradients. Long short term memory networks. Various RNN architectures and applications. Deep Learning Book - Goodfellow et al. (Chapter 10) |
slides | ||
| 06-11-2019 | Assignment #5. Analysis of coding part submitted as report in class. Codes submitted separately via mlsp19.iisc aT gmail doT com (due date of 20-11-2019). |
HW5 | ||
| 08-11-2019 | Understanding and Visualizing Neural Network Activations. Stochastic Neighborhood Embedding and t-distributed Stochastic Neighborhood Embedding (tSNE). Visualizing activations in image networks and audio networks. tSNE paper |
slides | ||
| 11-11-2019 | Deep generative modeling - Restricted Boltzmann Machines (RBMs). Conditional independence property. RBM parameter learning. Positive and negative phase of learning. Intuitions behind contrastive divergence algorithm Deep Learning Book - Goodfellow et al. (Chapter 18,19,20) |
slides | ||
| 13-11-2019 | Autoencoders (AE). Variational autoencoders. Variational lower bound derivation. KL divergence derivation. Data generation with VAEs. Kingma's paper link VAE Tutorial link |
slides | ||
| 18-11-2019 | Generative Adversarial Networks. Intuition and Model Description. Theoretical bounds on the goals of GANs. GANs for data generation. Deep learning models for text. Word2vec model. GAN paper |
slides | ||
| 20-11-2019 | Deep learning for speech. Speech recognition models. End-to-end speech modeling. Deep learning for computer vision. Image classification and segmentation. Summary of the Course. | slides | ||
| 25-11-2019 | Practice Exam for Finals.
|
practiceTest |