Detailed Syllabus and Lectures

Lecture 9: Deep Generative Models - Part 1 (slides)

unsupervised learning, sparse coding, autoencoders, autoregressive models

Please study the following material in preparation for the class:

Required Reading:

• Chapter #13 of the Deep Learning text book.
• Chapter #14 of the Deep Learning text book.

Suggested Video Material:

• Foundations of Unsupervised Deep Learning, Ruslan Salakhutdinov
• Autoregressive Generative Models with Deep Learning, Hugo Larochelle

Additional Resources:

• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models, Sam Bond-Taylor, Adam Leach, Yang Long, Chris G. Willcocks, IEEE Trans. Pattern Anal. Mach. Intel., Vol. 44, No. 11, Nov. 2022.
• Pixel Recurrent Neural Networks, Aaron van den Oord, Nal Kalchbrenner, Koray Kavukcuoglu, ICML 2016.
• Conditional Image Generation with PixelCNN Decoders, Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, Koray Kavukcuoglu, NeurIPS 2016.
• Unsupervised Feature Learning and Deep Learning, Andrew Ng.
• [Blog post] Unsupervised Sentiment Neuron, Alec Radford, Ilya Sutskever, Rafal Jozefowicz, Jack Clark and Greg.

Lecture 8: Attention, Transformers and Memory (slides)

content-based attention, location-based attention, soft vs. hard attention, self-attention, attention for image captioning, transformer networks

Please study the following material in preparation for the class:

Required Reading:

• Attention and Augmented Recurrent Neural Networks, Chris Olah and Shan Carter. Distill, 2016
• [Blog post] The Illustrated Transformer, Jay Alammar
• [Blog post] Transformers for Image Recognition at Scale, Neil Houlsby and Dirk Weissenborn

Suggested Video Material:

• Recurrent Neural Networks and Language Models, Richard Socher
• Attention and Memory in Deep Learning, Alex Graves
• Attention is all you need attentional neural network models, Łukasz Kaiser
• Vision Transformers explained, AI Coffee Break with Letitia

Additional Resources:

• Neural Machine Translation by Jointly Learning to Align and Translate, D. Bahdanau, K. Cho, Y. Bengio, ICLR 2015
• Sequence Modeling with CTC, Awni Hannun, Distill, 2017
• Recurrent Models of Visual Attention, V. Mnih, N. Heess, A. Graves, K. Kavukcuoglu, NeurIPS 2014
• Attention Is All You Need, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, NeurIPS 2017
• [Blog post] The Transformer Family, Lilian Weng

Lecture 7: Recurrent Neural Networks (slides)

sequence modeling, recurrent neural networks (RNNs), RNN applications, vanilla RNN, training RNNs, long short-term memory (LSTM), LSTM variants, gated recurrent unit (GRU)

Please study the following material in preparation for the class:

Required Reading:

• Chapter #10 of the Deep Learning text book.
• Section 5 of Generating Sequence with Recurrent Neural Networks, A. Graves, ArXiV

Suggested Video Material:

• Graham Neubig's lecture on Recurrent Networks for Sentence or Language Modeling

Additional Resources:

• [Blog post] Understanding LSTM Networks, Chris Olah.
• [Blog post] The Unreasonable Effectiveness of Recurrent Neural Networks, Andrej Karpathy.
• Learning Long-Term Dependencies with Gradient Descest is Difficult, Yoshua Bengio, Patrice Simard, and Paolo Frasconi.
• Long Short-Term Memory, Sepp Hochreiter and Jürgen Schmidhuber.

Lecture 6: Understanding and Visualizing Convolutional Neural Networks (slides)

transfer learning, interpretability, visualizing neuron activations, visualizing class activations, pre-images, adversarial examples, adversarial training

Please study the following material in preparation for the class:

Required Reading:

• Matthew D Zeiler and Rob Fergus, Visualizing and Understanding Convolutional Networks, ECCV 2014.
• Christian Szegedy et al. Intriguing properties of neural networks, arXiv preprint arXiv:1312.6199v4

Suggested Video Material:

• Andrej Karpathy's Stanford CS231n Lecture 9

Additional Resources:

• [Blog post] Understanding Neural Networks Through Deep Visualization, Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson.
• [Blog post] The Building Blocks of Interpretability, Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye and Alexander Mordvintsev.
• [Blog post] Feature Visualization, Chris Olah, Alexander Mordvintsev and Ludwin Schubert.
• [Blog post] An Overview of Early Vision in InceptionV1, Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, Shan Carter.
• [Blog post] OpenAI Microscope.
• [Blog post] Breaking Linear Classifiers on ImageNet, Andrej Karpathy.
• [Blog post] Attacking machine learning with adversarial examples, OpenAI.

Lecture 5: Convolutional Neural Networks (slides)

convolution layer, pooling layer, evolution of depth, design guidelines, residual connections, semantic segmentation networks, object detection networks, backpropagation in CNNs

Please study the following material in preparation for the class:

Required Reading:

• Chapter #9 of the Deep Learning text book.

Suggested Video Material:

• Andrej Karpathy's Stanford CS231n Lecture 7
• Justin Johnson's Stanford CS231n Lecture 8
• Kaiming He's tutorial on Deep Residual Networks

Additional Resources:

• Andrej Karpathy's CS231n notes on Convolutional Networks.
• Hiroshi Kuwajima’s Memo on Backpropagation in Convolutional Neural Networks.
• A guide to convolution arithmetic for deep learning, Vincent Dumoulin and Francesco Visin.
• Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review, Waseem Rawat and Zenghui Wang.
• [Blog post] Understanding Convolutions, Christopher Olah.
• [Blog post] Deconvolution and Checkerboard Artifacts, Augustus Odena, Vincent Dumoulin, Chris Olah.
• [Blog post] Deep Learning for Object Detection: A Comprehensive Review, Joyce Xu.
• [Blog post] A Brief History of CNNs in Image Segmentation: From R-CNN to Mask R-CNN, Dhruv Parthasarathy

Lecture 4: Training Deep Neural Networks (slides)

data preprocessing, weight initialization, normalization, regularization, model ensembles, dropout, optimization methods

Please study the following material in preparation for the class:

Required Reading:

• Chapter #7 and Chapter #8 of the Deep Learning text book.

Suggested Video Material:

• Efstratios Gavves' Lecture 3.

Additional Resources:

• Stochastic Gradient Descent Tricks, Leon Bottou.
• Section 3 of Practical Recommendations for Gradient-Based Training of Deep Architectures, Yoshua Bengio.
• Troubleshooting Deep Neural Networks: A Field Guide to Fixing Your Model, Josh Tobin.
• [Blog post] Initializing neural networks, Katanforoosh & Kunin, deeplearning.ai.
• [Blog post] Parameter optimization in neural networks, Katanforoosh et al., deeplearning.ai.
• [Blog post] The Black Magic of Deep Learning - Tips and Tricks for the practitioner, Nikolas Markou.
• [Blog post] An overview of gradient descent optimization algorithms, Sebastian Ruder.
• [Blog post] Why Momentum Really Works, Gabriel Goh

Lecture 3: Multi-layer Perceptrons (slides)

feed-forward neural networks, activation functions, chain rule, backpropagation, computational graph, automatic differentiation, distributed word representations

Please study the following material in preparation for the class:

Required Reading:

• Chapter 6 of the Deep Learning text book.
• Yoav Goldberg's A Primer on Neural Network Models for Natural Language Processing, 3 to 6

Suggested Video Material:

• Hugo Larochelle’s video lectures, 1.1 to 1.6, 2.1 to 2.7

Additional Resources:

• G. Hinton's Coursera class on Neural Networks, Lecture 1 to 3.
• A. Karpathy's Neural Networks: Zero to Hero video lectures
• [Blog post] Neural Networks, Manifolds, and Topology, Christopher Olah.
• [Blog post] Calculus on Computational Graphs: Backpropagation, Christopher Olah.
• Chapter 16 of Jurafsky and Martin's Speech and Language Processing book (3rd Edition draft)

Lecture 2: Machine Learning Overview (slides)

types of machine learning problems, linear models, loss functions, linear regression, gradient descent, overfitting and generalization, regularization, cross-validation, bias-variance tradeoff, maximum likelihood estimation

Please study the following material in preparation for the class:

Required Reading:

• Chapter 5 of the Deep Learning text book.

Suggested Video Material:

• Machine Learning, Doina Precup (Deep Learning Summer School, Montreal 2016)

Additional Resources:

• A few useful things to know about machine learning, P. Domingos. Communications of the ACM, 55 (10), 78-87, 2012.

Lecture 1: Introduction to Deep Learning (slides)

course information, what is deep learning, a brief history of deep learning, compositionality, end-to-end learning, distributed representations

Please study the following material in preparation for the class:

Required Reading:

• Chapter 1 of the Deep Learning text book.
• [Blog post] AI Winter. How Canadians contributed to end it?, Pavan Mirla.
• The Bandwagon, Claude E. Shannon. IRE Transactions on Information Theory, Vol. 2, Issue 3, 1956
• Chapter 1: The Philosophy and the Approach of David Marr's Vision, 1982.

Additional Resources:

• The unreasonable effectiveness of deep learning in artificial intelligence, Terrence J. Sejnowski, PNAS, 2020.
• Deep Learning, Yann LeCun, Yoshio Bengio, Geoffrey Hinton. Nature, Vol. 521, 2015.
• Deep Learning in Neural Networks: An Overview, Juergen Schmidhuber. Neural Networks, Vol. 61, pp. 85–117, 2015.
• On the Origin of Deep Learning, Haohan Wang and Bhiksha Raj, arXiv preprint arXiv:1702.07800v4, 2017