CMP717: Image Processing
Spring 2020
Detailed Syllabus and Lectures
Topics:
Visual attention, visual saliency prediction, eye fixations, salient object, top-down saliency
Required Reading:
- A model of saliency-based visual attention for rapid scene analysis, L. Itti et al., IEEE Trans. Pattern Anal. Mach. Intell., 1998
- Visual saliency estimation by nonlinearly integrating features using region covariances, E. Erdem and A. Erdem, Journal of Vision, 2013
- Top down saliency estimation via superpixel-based discriminative dictionaries, A. Kocak et al., In Proc. BMVC, 2014
- SALICON: Reducing the Semantic Gap in Saliency Prediction by Adapting Deep Neural Networks, X. Huang et al., In Proc. ICCV, 2015
- Spatio-Temporal Saliency Networks for Dynamic Saliency Prediction, C. Bak et al., IEEE Trans. Multimedia, to appear
Additional Reading:
- State-of-the-art in visual attention modeling, A. Borji and L. Itti, IEEE Trans. Pattern Anal. Mach. Intell., 2013
- Large-Scale Optimization of Hierarchical Features for Saliency Prediction in
Natural Images, E. Vig et al., In Proc. CVPR, 2014
- Where should saliency models look next?, Z. Bylinskii et al., In Proc. ECCV, 2016
- A Deep Multi-Level Network for Saliency Prediction, M. Cornia et al., In Proc. ICPR, 2016
- Shallow and Deep Convolutional Networks for Saliency Prediction, J. Pan et al., In Proc. CVPR, 2016
- SalGAN: SalGAN: Visual Saliency Prediction with Generative Adversarial Networks, J. Pan et al., In Proc. CVPR-workshop, 2017.
- Revisiting Video Saliency: A Large-scale Benchmark and a New ModelW. Wang et al., In Proc. CVPR, 2018
- Simple vs complex temporal recurrences for video saliency prediction, P. Linardos et al., In Proc. BMVC, 2019
- Video Saliency Prediction using Spatiotemporal Residual Attentive Networks, Q. Lai et al., IEEE Trans. Image Processing, 2019
- TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection, K. Min and J. J. Corso, In Proc. ICCV, 2019
May 29: Semantic Segmentation Lecture slides
Topics:
Semantic segmentation, instance segmentation
Required Reading:
- TextonBoost for Image Understanding: Multi-Class Object Recognition and Segmentation by Jointly Modeling Texture, Layout, and Context, J. Shotton et al., Int. Journal Comp. Vis., 2007
- Simultaneous Detection and Segmentation, B. Hariharan et al., In Proc. ECCV, 2014
- Fully Convolutional Networks for Semantic Segmentation, J. Long et al., In Proc. CVPR, 2015
- Hypercolumns for Object Segmentation and Fine-grained Localization, B. Hariharan et al., In Proc. CVPR, 2015
- Mask R-CNN, K. He et al., In Proc. ICCV, 2017
Additional Reading:
Topics:
Image deblurring, blind deconvolution, non-blind deconvolution, MAP based formulations, variational Bayesian based models, edge based methods, deep learning based methods
Required Reading:
- Removing camera shake from a single image, R. Fergus et al., In Proc. SIGGRAPH, 2006
- Fast motion deblurring, S. Cho and S. Lee, ACM Trans. Graph., 2009
- Two-Phase Kernel Estimation for Robust Motion Deblurring, L. Xu and J. Jia, In Proc. ECCV, 2010
- Learning a Convolutional Neural Network for Non-uniform Motion Blur Removal, J. Sun et al., In Proc. CVPR, 2015
- From Motion Blur to Motion Flow: a Deep Learning Solution for Removing Heterogeneous Motion Blur, D. Gong et al., In Proc. CVPR, 2017
- Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring
, S. Nah et al., In Proc. CVPR, 2017
- DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks, O. Kupyn et al., In Proc. CVPR, 2018
Additional Reading:
- Image Deblurring with Blurred/Noisy Image Pairs, L. Yuan et al., SIGGRAPH, 2007
- Understanding and evaluating blind deconvolution algorithms, A. Levin et al., In Proc. CVPR, 2009
- From Learning Models of Natural Image Patches to Whole Image Restoration, D. Zoran and Y. Weiss, In Proc. ICCV, 2011
- Handling Outliers in Non-blind Image Deconvolution, S. Cho et al., In Proc. ICCV, 2011
May 15: Image to Image Translation Lecture slides
Topics:
Required Reading:
Additional Reading:
- Unsupervised Image-to-Image Translation Networks, M.-Y. Liu et al., In Proc. NIPS, 2017
- Toward Multimodal Image-to-Image Translation, J.-Y. Zhu et al., In Proc. NIPS, 2017
- Manipulating Attributes of Natural Scenes via Hallucination, L. Karacan et al., ACM Tran. Graphics, 2020
- Image Synthesis in Multi-Contrast MRI with Conditional Generative Adversarial Networks, S.U.H. Dar et al., IEEE Tran. Medical Imaging, 2019
May 8: Deep Generative Models Lecture slides
Topics:
Generative models, autoregressive models, variational autoencoders, generative adversarial networks
Required Reading:
- Auto-Encoding Variational Bayes, D. P. Kingma and M. Welling, In Proc. ICLR, 2014
- Generative Adversarial Networks, I. J. Goodfellow et al., In Proc. NIPS, 2014
- Conditional Image Generation with PixelCNN Decoders, A. van den Oord et al., In Proc. NIPS, 2016
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, A. Radford et al., In Proc. ICLR, 2016
Additional Reading:
- Pixel Recurrent Neural Networks, A. van den Oord et al., In Proc. ICML, 2016
- Progressive Growing of GANs for Improved Quality, Stability, and Variation, T. Karras et al., In Proc. ICLR, 2019
- A Style-Based Generator Architecture for Generative Adversarial Networks, T. Karras et al., arXiv preprint arXiv:1812.04948, 2018
Apr 24: Convolutional Neural Networks Lecture slides
Topics:
Convolutional neural networks
Required Reading:
Additional Reading:
Apr 17: Deep learning basics Lecture slides
Topics:
Deep learning, perceptron, multi layer perceptron, backprogation, activation functions
Required Reading:
Additional Reading:
Topics:
Sparse coding, dictionary learning, K-SVD algorithm, L0-smoothing
Paper presentation:
Required Reading:
Topics:
Nonlinear filtering, Perona-Malik diffusion, Total variation, Mumford-Shah model, Bilateral filtering, Non-local means denoising, image smoothing via region covariance (RegCov smoothing)
Required Material:
- Notes on nonlinear diffusion
- Notes on Mumford-Shah formulation
- Variational Methods: A Short Intro by Daniel Cremers
- Scale-Space and Edge Detection Using Anisotropic Diffusion, P. Perona and J. Malik, IEEE Trans. Pattern Anal. Mach. Intell., 1990
- Nonlinear Total Variation Based Noise Removal Algorithms, L. Rudin et al., Phys. D., 1992
- Mumford-Shah Regularizer with Contextual Feedback, E. Erdem and S. Tari, J. Math. Imaging and Vision, 2009
- Bilateral Filtering for Gray and Color Images, C. Tomasi and R. Manduchi, In Proc. ICCV, 1998
- A non-local algorithm for image denoising, A. Buades et al., In Proc. CVPR, 2005
- Structure Preserving Image Smoothing via Region Covariances, L. Karacan et al., ACM Trans. Graph., 2013
Additional Reading:
Mar 6: Linear Filtering, Edge/Boundary Detection, Image Segmentation Lecture slides
Topics:
Linear filtering, linear diffusion, derivative filters, Laplacian of Gaussian, Canny edge detector, pb detector, sketch tokens, k-means, normalized cut
Required Reading:
- Notes on linear diffusion
- Scale-space filtering: A New Approach to Multi-Scale Description, A. P. Witkin, In Proc. IJCAI, 1983
- Theory of Edge Detection, D. Marr and E. Hildreth
. Proc. R. Soc. Lond. B, 1980.
- Normalized Cuts and Image Segmentation, J. Shi and J. Malik, IEEE Trans. Pattern Anal. Mach. Intell., 2000
- Contour Detection and Hierarchical Image Segmentation, P. Arbelaez et al., IEEE Trans. Pattern Anal. Mach. Intell., 2011
- Sketch Tokens: A Learned Mid-level Representation for Contour and Object Detection, J. J. Lim et al., In Proc. CVPR, 2013
Additional Reading:
- Chapter 4 (Edges) and Chapter 5 (Segmentation) from R. Szeliski's book
Feb 28: Introduction to Image Processing Lecture slides
Topics:
Course information, what is image processing
Required Reading:
- Chapter 3 (Image processing) from R. Szeliski's book
Additional Reading: